Optical nonlinear effect plays an important role in optical communication, optical detection, quantum information and other areas. However, it is constrained by the weakness of the nonlinear optical response of the common materials. The enhancement of the optical nonlinear response on a nanoscale becomes a critical challenge. Over the years, several ways to enhance the optical nonlinear effects have been suggested. In fact, these technologies can slightly enhance the optical nonlinear response. Recently, some research groups focused on the materials with vanished permittivity, which is called epsilon-near-zero (ENZ) material, showing that it can exhibit large optical nonlinearity due to the field enhancement in the material of this type. However, the ENZ material only holds a large optical nonlinear response in a limited spectral range. In order to overcome this limitation, here in this paper we report the ENZ mode which is excited by the ITO film and strongly coupled to the gap surface plasmons excited by the metal-dielectric-metal structure. To acquire the nonlinear refractive index
n
2, we first calculate the ITO permittivity through the Drude-Lorentz model and find the wavelength of the ENZ material. Then we calculate the time-dependent electron temperature and lattice temperature of ITO by the two-temperature model. According to the elevated electron temperature, we can calculate the plasma frequency
${\omega _{\rm p}}$
, and by taking it into the Drude-Lorentz model, we can obtain a new permittivity of ITO compared with the initial one. Finally, we can calculate the variation of the refractive index
$ \Delta n $
, and the nonlinear refractive index
$ {n_2} = \Delta n/{I_0} $
. In this paper, our coupled structure exhibits a broadband (~1000 nm bandwidth) enhancement of the nonlinear optical effect in the near-infrared spectrum, a maximum nonlinear refractive index
n
2as large as 3.02 cm
2·GW
–1, which is nearly 3 orders larger than the previously reported nonlinear refractive index of bare ITO film. As a result, it is possible to realize a dramatically large variation of nonlinear refractive index under a low-power optical field. It is expected to be used in the nano photonic devices such as optical storage, all-optical switches, etc.