In this paper, we present a novel nonlinear optimization algorithm for designing a shim coil system, especially a high-order axial shim coil, for a magnetic resonance imaging（MRI) system. In an MRI equipment, in order to eliminate higher-order harmonic components of the magnetic field within the volume of interest（VOI), passive shimming（PS) is adopted in traditional methods. However, such a method is suitable for the global shimming with low accuracy and poor target. Active shimming（AS) makes up for the shortcomings from PS with a set of shim coils which are designed to generate a specific magnetic fields to improve magnetic field homogeneity within the VOI. Because the complexity of wire pattern increases with the order of AS coil increasing, conventional optimization model cannot meet the design requirements for producing the complicated magnetic field. In this paper, we propose a nonlinear optimization method of designing the axial shim coils for an open-style bi-planar MRI system, based on boundary element method. The optimization model is built in light of influence extents of the various parameters on the coil characteristics for different shim coils. In such a new method, the field error between the magnetic field produced by designed shim coil and the desired target value is selected to be an optimal value subjected to some constraints including line spacing and coil radius, which makes it possible to realize the manufacture process. Meanwhile, the more design parameters, which involve not only the stream function values at each node, but also the compensation parameters and/or the number of grid nodes, are regarded as optimized variables to control the magnetic deviation and characteristics of designed coil. By using some designed shim coils for a 0.5 T open style bi-planar superconducting MRI, including Z1, Z2, Z3 and Z4, the efficiency of such a numerical design method is displayed. Especially for high-order shim coils, more optimized parameters are involved to control the magnetic deviation of the coils, thereby providing a more flexible and straightforward method of designing the axial shim coils.