Metal materials play an important role in many domains, which are significant to the national economy. However, different kinds of metal defects, such as cracks, contraction cavities, impurities, will be generated in the process of production and service. These defects will affect the metal service life and mechanical properties directly, and even cause serious economic loss. Therefore, it is vital to detect the metal defects. Numerous nondestructive testing (NDT) methods have been proposed for detecting metal defects, such as ultrasonic (US) testing, eddy current testing, photoacoustic (PA) testing, magnetic particle testing, etc. However, each of them uses a single modal signal, which leads to a limited detection range. A nondestructive detecting method for metal material defects based on multimodal signals is proposed to expand the scope of detection and obtain more complete information. Specifically, optical signal, PA signal and US signal are combined together in this method, with the consideration of their complementarities. Simulation and experiments are conducted to validate the effectiveness of the proposed method. Firstly, finite element simulation is employed to analyze the relationship between material parameters and the absorption of laser energy. Meanwhile, the influence of defect size on PA surface wave is simulated and analyzed. Then, a multimodal NDT platform is established to collect and process optical, PA and US signals of the metal defects. These three modal signals contain information about metal surface, shallow surface and internal defects respectively. Eventually, the information, including the location, appearance on the surface, depth, extension path in the material, is obtained. As demonstrated in the results, the nondestructive detecting method based on multimodal signals can detect the metal defects accurately and comprehensively. This method improves the existing methods in terms of detection range and quantitative detection. Additionally, it provides a new way for the quantitative detection and comprehensive diagnosis of metal defects.