The complex structure of hyperchaos and its complex dynamic behavior have a good application prospect in the fields of image encryption, digital watermarking and information security. Therefore, it has become very important to generate chaotic attractors with multi-vortex and multi-winged multi-rings with complex topologies. In this paper, we propose a new five-dimensional hyperchaotic system capable of generating multi-ring and multi-wing, and carry out theoretical analysis and numerical simulation experiments on some basic dynamic characteristics of the chaotic system. Such as equilibrium point, dissipation, Lyapunov exponent, bifurcation diagram, phase diagram and so on. In the process of encryption, first, we decompose the plaintext image matrix and the five chaotic sequences into an orthogonal matrix and an upper triangular matrix by QR decomposition. The five chaotic sequences generated by the chaotic system are respectively decomposed into an upper triangular matrix and a lower triangular matrix by the LU decomposition method. The upper triangular matrix decomposed by the QR decomposition method and the lower triangular matrix decomposed by the LU decomposition method are respectively added to obtain five discrete chaotic sequences. At the same time, the five discrete chaotic sequences are added to the upper triangular matrix decomposed by the LU decomposition method to obtain the final five discrete chaotic sequences. Secondly, the orthogonal matrix decomposed by the plaintext image matrix is multiplied by five orthogonal matrices decomposed by five chaotic sequences. At the same time, the elements in the upper triangular matrix decomposed by the plaintext image matrix are chaotically arranged by the chaotic sequence, and then the two matrices after the operation are multiplied. Finally, the multiplied matrix is chaotically placed on the bit by a chaotic sequence. Then use the chaotic sequence to perform a bitwise XOR operation to obtain the final encrypted image. The theoretical analysis and simulation results show that the algorithm has large key space and strong key sensitivity. It can effectively resist the attacks of statistical analysis and gray value analysis, and has good encryption effect on digital image encryption. This image encryption algorithm using a combination of conventional encryption and chaotic encryption does not have a defined plaintext ciphertext mapping relationship.