Most of optical encryption systems are symmetric cryptosystems. The plaintext and the ciphertext in optical image encryption are related linearly. The security of the system needs to be strengthened. The asymmetric cryptosystem based on phase truncated Fourier transforms (PTFT) makes the security of the encryption system greatly improved by its nonlinear phase truncation. Deep learning (DL) as a method of machine learning was proposed decades ago. With the development of computer’s performance, the practicality of deep learning proves to be more and more obvious. Recently, deep learning has been effectively used in many fields such as biomedicine, object detection, etc. The good results have been achieved. In this article proposed is the attack to the PTFT encryption system by deep learning. Through the PTFT encryption system, we construct a plaintext-ciphertext paired image dataset and then train it by residual network (ResNet). There are two problems encountered by the traditional neural network model. One is vanishing or named exploding gradient, which makes training effect difficult to converge and the other is a degradation phenomenon. When continuing to increase the number of layers for a suitable depth model, the model accuracy will decline which is not caused by overfitting. This problem can be solved by the ResNet to a certain extent by directly bypassing and then taking the input information to the output to protect the integrity of the information. The biggest difference between ordinary directly connected convolutional neural networks and ResNet is that the ResNet has many bypass branches that directly connect the input to the subsequent layers, so that the subsequent layers can directly learn the residuals. The ResNet can automatically learn the decryption characteristics of the encryption system. Finally, the test set is used to test the decryption performance of the trained model. The data show that the model can restore the image with high quality and the model has a certain anti-noise ability. Compared with the two-step iterative amplitude recovery algorithm, the the method proposed in this paper can recover high quality image.