Ship magnetic field modeling is not only beneficial to understanding the characteristics of ship magnetic field, but also can predict the space distribution of ship magnetic field, which has an important application in ship protection and underwater weapons. Aiming at the problems of low modeling accuracy and poor stability in establishing the ship magnetic field hybrid model, a method of establishing a high precision stablity model is proposed in this paper. A hybrid model of magnetic field of a ship is established by using a uniformly magnetized rotating ellipsoid and a magnetic dipole array. Since the number and positions of magnetic dipoles in the hybrid model have an important effect on the modeling accuracy and stability, the fitting error function representing the modeling accuracy and the coefficient matrix condition number function representing the stability of the model are constructed by taking the magnetic dipole parameters as unknown variables. The multi-objective function is constructed by combining the fitting error function with the coefficient matrix conditional number function, which indirectly transforms the modeling problem into a multi-objective optimization problem. The multi-objective function is solved by using the multi-objective particle swarm optimization algorithm, and an optional set of modeling solution results is obtained. In order to select the best results from the optional set, the corresponding selection rules are designed based on the modeling accuracy. The proposed method is validated by the measured data of three kinds of ship models, the modeling results show that the relative error of the model is less than 3%, and the conversion error is less than 6%, which verifies that the proposed method can effectively model the ship magnetic field. Though the measurement data error exists, the modeling solution results from the proposed method have the best stability, which verifies that the modeling method proposed in this work has good stability. Compared with the two existing modeling methods, the proposed method has very good modeling accuracy and stability. Finally, the actual data of a ship on the sea are used for modeling, and the modeling results further verify that the proposed method has high modeling accuracy and conversion accuracy, and can be effectively applied to the relevant projects.