Laser heterodyne is a kind of technique based on coherent detection with high sensitivity and spectral resolution for spectrum measurements. For these reasons, it has been widely used in many research fields, such as trace gases’ detection of earth’s or terrestrial planets’ atmosphere. However, when the laser heterodyne spectrometer is used for measuring the spectrum, the instrument line shape (ILS) function usually smooth the spectrum, which affects the inversion results of the gas column density. In previous researches, the radio frequency (RF) filter response function was usually used as the ILS, but recent studies indicated that the ILS without consideration of the influence of lock-in amplifier was not precise enough. In order to obtain the ILS function of the laser heterodyne spectrometer, the main factors which influence the ILS are analyzed, including the RF filter bandwidth, integral time and low-pass filter of lock-in amplifier, and the process is based on the principle of laser heterodyne technology and the flow of heterodyne signal processing. The presented ILS is the convolution of RF filter, wavelength variation in the integral time and the low-pass filter. In addition, for testing the effectiveness of the ILS in this paper, the laser heterodyne spectrometer which was built in our laboratory is used for the multiple measurement of the absorption of water vapor and methane in the band of 3.53 ${\text{μm}}$ and the column densities are retrieved with different ILS. The experimental results show that the actual resolutions of the laser heterodyne spectrometer are about 0.005 cm ^–1and 0.025 cm ^–1when the integral times are set to be 10 ms and 100 ms respectively. Furthermore, the RF filter response function and the ILS function presented in the paper are respectively used in the procedure of water vapor and methane inversion. The results show that when the ILS function used for the retrieval, the sum of squared residual reduces about 16% and the residuals at the peak of methane absorption reduces almost 100% compared with the scenario when using the RF filter function. Above all, the comprehensive analysis of the laser heterodyne spectrometer in this paper indicates that the ILS function is more precise than pioneering studies and this work will be helpful for retrieving the precise profiles of trace gases.