A hybrid optimization approach with a combination of linear programming and nonlinear programming algorithm for designing a compact self-shielded magnetic resonance imaging (MRI) superconducting magnet system is presented. The designed coils possess advantages of low construction costs, simple coil structure and the maximum magnetic strength within coils, current margin and electromagnetic stress easy to control. Firstly, in the stage of linear programming optimization, the feasible rectangular region can be divided into two-dimensional meshes, and a current map is calculated for meeting the magnetic field constraints over the surfaces of DSV sphere and 5 gauss stray field ellipse; Secondly, the current map has many nonzero current clusters and each cluster can be discretized into a solenoid. A nonlinear programming algorithm is employed to optimize the positions of all solenoids for minimizing the total coil volume and meeting all constraints including magnetic field which is the same as linear programming stage, and maximum magnetic strength, current margin and the gap between neighborhood inner coils. A 1.5 T compact self-shielded MRI superconducting magnet system is studied, the total coil length is only 1.32 m and the peak-peak homogeneity over 50 cm DSV is 10 ppm. The design approach is flexible and efficient for designing symmetrical and asymmetrical horizontal MRI and also open bi-planar MRI system.