The propagation characteristics of nonlinear dust acoustic solitary waves in a complex plasma system with nonthermal electrons and trapped ions are investigate in this work. The nonlinear dispersion relation of dust acoustic waves is obtained by using the linear method, and the two-dimensional autonomous system governing the motion of nonlinear dust acoustic waves is derived by using the Sagdeev potential method. At the same time, the specific expression of the Sagdeev potential function is obtained based on the Sagdeev potential equation. The numerical simulations are used to analyze the phase portraits of the two-dimensional autonomous system, revealing the linear periodic wave orbits, nonlinear periodic wave orbits, and homoclinic orbits co-existing in the complex dusty plasma system with nonthermal electrons and trapped ions. Furthermore, from the variations of the Sagdeev potential function with different system parameters it follows that only the compressive solitary waves exist in this complex plasma system. The significant influences of various system parameters on the amplitude, width, and waveform of the nonlinear dust acoustic solitary wave in the complex plasma system are discussed in detail. The results demonstrate that the Mach number, the nonthermal electrons and trapped ions, undisturbed dust particle number density, temperature, and charge have important effects on the propagating characteristics of the nonlinear dust acoustic solitary waves in a complex plasma with nonthermal electrons and trapped ions.