In this work, we present a quantitative method of predicting defect depth within carbon fiber reinforced plastics (CFRP) composite with unknown thermal property of specimen. The method discussed in this paper is derived from theoretical model of ideal one-dimensional heat conduction in a flat plate sample. The principle of depth prediction using pulsed infrared thermography is presented; the time based on the peak second-derivative of the temperature difference in the logarithmic scale is selected as the characteristic time; the calibration curve based on data of single defect depth can be easily affected by random error. The calibration curve is put forward in the paper and obtained by designing a stepped reference sample, and it can be obtained by a curve fit of different depths and corresponding characteristic time using polynomial function. The polyfitting curve is used as the calibration curve for depth prediction when the relative error square is minimum. The experimental results show that the depth measured by the calibration curve based on many reference defect depths is more accurate than by single reference defect depth. Defect depth in CFRP composite can be measured by the method without knowing thermal property of specimen.