To research the encryption algorithm for color joint photographic expert group (JPEG) image, by comprehensively choosing the selective encryption and the joint compression and encryption, an encryption algorithm combining with spatial domain and frequent domain for color JPEG image is proposed. The 8×8 blocks are first diffused on spatial domain, then the edge-detection method is utilized to find out the significant blocks containing abundant details. After encrypting all the direct current coefficients, part of alternating current (AC) coefficients in significant blocks are chosen to be encrypted. Finally the information about marking the significance of the blocks is embedded into AC coefficients for transmission. Theoretical analyses and experimental results show that the proposed algorithm is compatible with JPEG format. The cipher image has good visual quality and uniform color distribution. The algorithm possesses huge key space, strong key-sensitivity and good security.