The adjacency effect, the contribution of the neighboring pixels to the radiance of the line of sight pixel, is caused by the Rayleigh scattering of atmospheric molecules and Mie scattering of aerosol particles. The adjacency effect will cause the reflectance of each pixel in the apparent reflectance satellite image to be between the real reflectance and the average background reflectance, reducing the accuracy of the surface reflectance inversion. Therefore, it is very important to remove the adjacency effect to improve the accuracy of retrieving the surface reflectance from satellite images. The most critical issue of the adjacency effect is to accurately calculate the weight of the contribution of each background pixel to the adjacency effect. The weight value of the contribution of each background pixel to the adjacency effect mainly depends on the spatial distance between the target pixel and the background pixel, the difference in reflectance between the target pixel and the background pixel, and the optical thickness of atmospheric molecules and the optical thickness of aerosol. At present, the commonly used weight function for calculating the weight value considers only the influence of optical thickness and spatial distance on the weight value. These weight functions are applied to a relatively uniform surface. However, when these weight functions are applied to an inhomogeneous surface, they will greatly reduce the accuracy of the adjacency effect correction. The combination of ground features in satellite images with the sub-meter spatial resolution is complex, so the influence of the difference in reflectance between the target pixel and the background pixel on the adjacency effect must be considered. The adaptive atmospheric correction algorithm proposed in this paper can adjust the weight value of the contribution of background pixels to the adjacency effect according to the spatial distance between the target pixel and the background pixel, the difference in reflectance between the target pixel and the background pixel, and the difference between the atmospheric molecules’ optical thickness and aerosol optical thickness. The adaptive atmospheric correction algorithm is used to correct the adjacency effect on GF-2 panchromatic satellite images. The results show that the adaptive atmospheric correction algorithm can effectively remove the adjacency effect in sub-meter spatial resolution optical satellite images, improve both the accuracy of quantitative study and the satellite image quality.