In the paper we study the method of reducing environmental influence in broadband laser frequency scanning interferometer. Target displacement caused by vibration will result in Doppler shift in measurement beat frequency. The extent of frequency shift is usually much larger than the actual target displacement. So the direct calculating of the target distance will cause ranging precision to decrease. In this paper, we establish a model for the influence of environmental vibration on the measurement and analyze the influence of the vibration on ranging result. To suppress the vibration effect, the Kalman filter is combined with the overlapping Chirp Z transform to estimate the measured data. The general process is described as follows. Firstly, the tuning nonlinearity will lead to the frequency spectrum broadening, so this paper we use the frequency sampling method to correct the frequency modulation nonlinearity of the laser. The frequency sampling method has the advantages of high speed and high precision. Secondly, the measurement system has the dispersion mismatch effect due to the use of broadband frequency swept laser. To solve this problem, the influence of the dispersion on the measurement is reduced by using the method of dispersion chirp slope calibration. Thirdly, because of the long frequency sweep period of the external cavity swept frequency laser, the vibration process of the target cannot be recorded in real time by single sweep, so in this paper we propose segmenting the measurement signal of single sweep and conducting Chirp Z transform to calculate target distance at different times. Compared with FFT algorithm, Chirp Z transform can achieve arbitrary narrow band spectrum subdivision, with the advantages of high accuracy and fast frequency measurement. Lastly, the Chirp Z ranging result is further combined with the method of Kalman filter to estimate the state of the target distance information. The experimental results indicate that the measurement standard is reduced from 185.4 μm to 9 μm by the proposed method. Without changing the absolute distance measuring device of broadband laser frequency scanning interferometer, this method provides a solution for further improving the ranging accuracy in the vibration environment, and reduces the complexity and cost of the device.