In order to ensure the secure and effective transmission of image information, a new method of optical image encryption using the multimode fiber (MMF) specklegram based compressive sensing combined with the double random phase encoding (DRPE) is proposed in this paper. The specklegrams obtained from the facet of the multimode fiber are used as the measurement matrix of compressive sensing (CS), and the compression and the first-stage encryption of the image are completed by compressive sensing, in which the specklegram also functions as the first secret key. Then, the second-stage encryption is implemented by using the double random phase encoding technology, in which the random phase mask acts as the second secret key. All of the specklegrams used in this paper are obtained from the facet of a 5 m-long and 105-μm-diameter-MMF and offset launching technique. Then the fiber specklegrams are proposed in several steps to provide the measurement matrix in CS. By performing an encryption and decryption test on a standard Lena image of 256 × 256 size, it is found that the decrypted image and the original image are visually consistent, and the encryption is also realized in the process of compression, which indicates the method proposed in this paper is feasible. Furthermore, the comparison studies of the performances of specklegram based measurement matrix and some classic measurement matrices show that the decrypted image quality using the specklegram matrix is better. And at the same time, comparing with the high hardware implementation complexity and high cost of other measurement matrices, specklegram based matrix can be easily realized by simple optical device, and the corresponding secret key can be easily changed by the working wavelength, which is helpful for enlarging the secret key space. It is further proved that the encryption method be able to effectively resist the statistical analysis attacks, cropping attacks and noise interference, and also have high sensitivity to the secret key, which shows good robustness and high security. Therefore, the image encryption method combined with the specklegram matrix based compression sensing with the optical DRPE can obtain good encryption effect and has a great secret key space, which may provide a good candidate scheme for the pure optical realization of image encryption.