In order to investigate the multiple scattering characteristics of energy transmission in a random particle layer on a typical rough surface, a high-order iterative method based on radiative transfer equation was proposed in this paper. The particles are placed above the rough surface and the high dimension (Z axis) is divided into several layers. Then the high-order iterative solution can be obtained by using the first-order iteration solution of radiative transfer equation and the rough surface theory. Moreover, the multiple scattering characteristics between particles can be gotten, as well as particles and rough surface. Numerical results show that the multiple scattering characteristics can be better explored by the proposed high-order iterative method when compared with the first-order iteration method. All these interactions can be interpreted by the energy transfer theory. The proposed high-order iterative method can be used as an efficient tool to explain the multiple scattering characteristics and predict the change of the scattering coefficients.