There is serious noise interference in the decryption process of the joint transform correlator (JTC) optical encryption system, so the quality of the decrypted image cannot meet the accuracy requirements in most cases. The quality of decrypted image can be improved to a certain extent when the phase key is designed by the Gerchberg-Saxton algorithm and the iterative algorithm fuzzy control algorithm, but the complexity of the design process is inevitable and the quality of the decrypted image still needs improving. Recently, the in depth learning technology has attracted the attention of scholars in the fields of computer vision, natural language processing and optical information processing. In order to deal with the noise interference in the JTC optical encryption system, combining the current deep learning method, in this paper we propose a new denoising method for JTC optical image encryption system based on in depth learning, the dense modules are added into the generated network to enhance the reuse of feature information and improve the performance of the network. The latest self-attention mechanism area is added into the network to distinguish the weights of different channels and learn the relationship between channel and channel, so that the network can selectively strengthen the useful feature information but suppress useless feature information. The density module and the channel attention module are integrated into a DCAB synthesis module, which can effectively extract the image feature information and improve the performance of the network. The receptive field of the convolution kernel is enlarged by two down-sampling and the feature map is restored to its original size by two up-sampling. The VGG-19 is used to extract high-frequency details and texture features of images, meanwhile, the non-adversarial loss and mean-square error (MSE) loss are added into the loss function to reduce the difference among the image samples. The quality of noise-reduced images in this method are obviously better than that of the existing denoising algorithms by evaluating intuitive visual observation or SSIM (structural similarity), PSNR (peak signal to noise ratio) and MSE. The results of numerical calculation and simulation experiments show that this method can greatly eliminate the influence of noise on the JTC optical image encryption system, and effectively improve the effectiveness and feasibility of JTC optical image encryption system for high-quality image encryption.