According to the Kerckhoff principle, a kind of image encryption algorithm based on hyper-chaos is discussed through choosing plaintext attack and ciphertext attack. The result shows that the key stream is independent of plaintext and one plain-text word is correlated with single cipher-text word for the algorithm, which makes the ciphertext decrypted easily with little computing by choosing plaintext and ciphertext attack. Considering the above problems, an improved algorithm based on hyper-chaos is proposed and the performance analysis is conducted, including statistical analysis, differential analysis, correlation analysis and key sensitivity testing. Theoretical analysis and simulation results show that the improved algorithm not only can resist the chosen plaintext attack and chosen ciphertext attack, but also can obtain better cryptographic properties, such as statistical characteristics, difference characteristics and so on.