With the wide applications of high-resolution imaging technology in topographic mapping, astronomical observation, and military reconnaissance and other fields, the requirements for imaging resolution of optical system are becoming higher and higher . According to the diffraction limit and Rayleigh criterion, the imaging resolution of the optical system is proportional to the size of the aperture of the system, but affected by the material and the processing of the optical component: the single aperture of the optical system cannot be infinitely enlarged. Therefore the synthetic aperture technology is proposed to replace the single large aperture optical system. Owing to the effect of sub-aperture arrangement and light scattering, the imaging of synthetic aperture optical system will be degraded because of insufficient light area and phase distortion. The traditional imaging restoration algorithm of synthetic aperture optical system is sensitive to noise, overly relies on degraded model, requires a lot of manually designed models, and has poor adaptability. To solve this problem, a multi-scale feature enhancement method of restoring the synthetic aperture optical image is proposed in this work. U-Net is used to obtain multi-scale feature, and self-attention in mixed domain is used to improve the ability of of the network to extract the features in space and channel. Multi-scale feature fusion module and feature enhancement module are constructed to fuse the information between features on different scales. The information interaction mode of the codec layer is optimized, the attention of the whole network to the real structure of the original image is enhanced, and the artifact interference caused by ringing is avoided in the process of restoration. The final experimental results are 1.51%, 4.42% and 5.22% higher than those from the advanced deep learning algorithms in the evaluation indexes of peak signal-to-noise ratio, structural similarity and perceived similarity, respectively. In addition, the method presented in this work has a good restoration effect on the degraded images to different degrees of synthetic aperture, and can effectively restore the degraded images and the images with abnormal light, so as to solve the problem of imaging degradation of synthetic aperture optical system. The feasibility of deep learning method in synthetic aperture optical image restoration is proved.