From the beginning of the twentieth century the development of nonlinear theory has guided us into a new field of exploration; chaos and fractals are important tools widely used in studying the nonlinear system. Fractal focuses on the description of the physical geometry while Chaos emphasizes the research on the developing process of dynamics coupled with geometry. We have studied the nonlinear behavior of the particles in non-integrable system by means of chaos and fractals.#br#There are fractal structures in a Hénon-Heiles system, by which we can investigate the general escape law of particles in chaotic systems. The motion of particles is traced by dynamical algorithm within the framework of Chin and Chen (2005 Celestial Mechanics and Dynamics Astronomy 91 301). For the first time, we study the escape property of particles in Hénon-Heiles system and make a comparison with it in its hexagon-shaped form. Particles with energy higher than the threshold (1/6) can escape from the Hénon-Heiles system. The self-similar structures are found by calculating the escape time of particles with an energy higher than the threshold for different exit angles. We calculate the escape rate of particles with different energies and make a statistical analysis on the fractal dimensions by ‘box-counting' method. It is that the escape rate and fractal dimensions change with the energy of particles in the Hénon-Heiles system. Moreover, we find that the fractal dimensions are strongly linear with the escape rate.#br#To testify the universality of the conclusion, we calculate the escape rate and fractal dimensions of particles in the hexagon-shaped Hénon-Heiles system. Unlike the motion of particles in the Hénon-Heiles system, the escape of particles is divided into two ranges of energy. In low energy range, the escape rate and fractal dimensions of particles change with energy, which is similar to that in the Hénon-Heiles system. However, the escape rate and fractal dimensions tend to become stable in high energy range. But the general law is still valid–the fractal dimensions are strongly linear with the escape rate.#br#Therefore, the fractal dimensions can be served as a tool to study the escape features of particles in a chaotic system. We can characterize the transport behaviors in a chaotic electronic equipment by investigating the fractal dimensions in the design of mesoscopic devices.