An accurate aerosol optical property can be obtained by a high spectral resolution lidar (HSRL) technique, which employs a narrow spectral filter to suppress Mie scattering in the lidar return signal. The ability for filter to suppress Rayleigh scattering is critical for the HSRL. In the HSRL system, Rayleigh scattering signal is obtained and aerosol scattering is suppressed at least by a factor of 10-5 through using the narrow filter. Usually, an atomic absorption filter can reach this level. While, the gaseous absorption lines do not exist at many convenient laser wavelengths, thus restricting the development of multi-wavelength HSRL instrument. A new and practical filtering method is proposed to realize the precise detection of atmospheric optical parameters by using the reflection field of Fabry-Perot (FP) interferometer. An optical splitting system with high spectral resolution is designed and its spectral characteristics are analyzed. Based on the characteristic of hyper-spectral lidar detection signal, the variations of spectral separation ratio and Rayleigh signal transmittance with reflectivity and cavity length are discussed. Spectral separation ratio is the transmittance ratio of aerosol scattering signal to molecular scattering signal through the spectral filter. With the increases of FP cavity length and surface reflectivity, the spectral separation ratio decreases and the Rayleigh signal transmission increases. The high spectral separation ratio and Rayleigh signal transmittance can be achieved by the reflection field of FP interferometer when the FP cavity length and reflectivity parameter can be chosen reasonably. We design an FP interferometer with a cavity length of 36 mm and reflectivity of 0.4. Its spectral separation ratio is affected by the echo divergence and incidence angle. The spectral separation ratio keeps unchanged when the beam divergence angle is within 3 mrad and the incident angle of the beam is within 0.5 mrad. In addition, a simulation analysis model is established based on the error propagation. An observed actual Mie-scattering profile is used for analyzing the errors. Moreover, the influences of the divergence angle and the incident angle of the echo beam on detection results are also discussed. The results show that the proposed FP interferometer can achieve fine spectral separation of Mie and Rayleigh scattering signal, and the error of detection result is not sensitive to laser divergence angle. Fine aerosol optical parameters can be achieved when the divergence and incidence angles are controlled within 10 mrad and 1.5 mrad, respectively.