Image quality is seriously degraded when propagating through the turbid atmosphere. It is practical to characterize the degradation process in terms of modulation transfer function (MTF). The MTF can describe the effect of the turbid medium on imaging quantitatively in spatial frequency domain, including attenuation and multiple scattering. It is inherent property of the turbid medium. The whole spatial frequency characteristic of the turbid atmosphere MTF can be acquired through the equivalence principle, i, e., the equivalence between the MTF of a turbid medium and the transmitted radiance from the medium under isotropic diffuse illumination. In practice, the image quality is not only affected by the turbid medium MTF but also related tightly to the background radiation. The influence of scattered background radiation on imaging was almost not considered in the past when dealing with the imaging problem in the turbid atmosphere. In this paper, this issue is considered in detail. The analysis results demonstrate that the scattered background radiation increases the zero frequency component of image in spatial frequency domain. As a result, it degrades the image contrast seriously in spatial domain. Based on the optical model of image degradation in the atmosphere, the theoretical analysis is carried out to study the image quality degradation process in spatial frequency domain. The formalized MTF is proposed, which considers the effects of attenuation, multiple scattering and scattered background radiation by the turbid medium on image quality. The quantitative relation among the formalized MTF, turbid medium MTF and background radiation is confirmed. Image blur simulations show that the results from the formalized MTF are more consistent with actual scenes than results only from turbid medium MTF. Thus, the formalized MTF can describe the image degradation process through atmosphere comprehensively. The image restoration results indicate that the formalized MTF method performs better than dark channel prior method. In order to evaluate different image restoration methods effectively in spatial frequency domain, spectrum area (AS) is proposed. The AS is the area of middle-high frequency information of the region of interest in restored image. So AS can represent the scene details in the restored image. The higher the AS, the better the image quality is, which is demonstrated in this paper. In conclusion, the formalized MTF provides a more effective method for image quality analysis and assessment. Additionally, it also supplies a new standpoint for researching atmospheric degradation mechanism and correction method for imaging in turbid atmosphere. Then, AS can be an effective reference for correction to the method evaluation.