We construct a nonlinear equation between the return signal and the boundary value of extinction coefficient according to the lidar equation. And according to the nonlinear equation, we put forward a new method to solve the nonlinear equation by using Broyden algorithm. The Broyden algorithm is a concrete application of the quasi-Newton method. It has faster convergence and less iteration times, and does not need to calculate the derivative value. After choosing a suitable initial value, the boundary value can be obtained through the algorithm. A 532 nm single-band Mie scattering imaging lidar system is developed in Hefei, Southern China, for real-time atmospheric aerosol/particle remote sensing. Atmospheric measurement has been performed in Science Island during night time, and the time-range distribution of atmospheric backscattering signal was recorded on April 6, 2017, by employing the imaging lidar system. Then, the boundary values are achieved based on the Broyden algorithm and the least square algorithm. It adopts the Klett backward integration method to retrieve the horizontal distribution of extinction coefficients in a range of 1 km after the acquisition of the signal by changing the distance, then the horizontal atmospheric transmittance can be achieved based on the path integral. We also conduct a contrast experiment with the one-way transmission of the horizontal light near the ground within the range of 1 km at the same time. The initial site is situated in the experimental room besides the Dongpu reservoir and the end site is located on the second floor of our office building. The important things in this experiment are that the light reaching the target surface must be fully received and the laser power should be monitored at the double-end. Then we can obtain the transmittance by the direct method. By comparing the transmittance from the direct method with the transmittance from imaging lidar between the two different ways, i.e., Broyden algorithm and least square algorithm, then the correlation coefficients are obtained to be both over 0.95 in the period. And the method introduced in this paper is a little better than the least square algorithm with a value of 0.968. Besides, the average relative errors between the two inverse methods and the direct method are 4.66% and 9.10%, respectively. The average relative errors obtained by using the least square algorithm is about twice that by using the Broyden algorithm. It can be concluded that the algorithm introduced in this paper is effective and has certain advantages for the inverse problem.