The chaotic motion of a relative rotation nonlinear dynamic system possessing both homoclinic and heteroclinic orbits is investigated. Firstly, the dynamics equation of relative rotation nonlinear dynamics system with nonlinear stiffness and nonlinear damping and forcing excitation is deduced. Secondly, a global bifurcation of the system and a probable route leading to chaos have been discussed by using Melnikov method, and the necessary condition of chaotic motion of system is presented. The chaotic motion of system is complemented by top Lyapunov exponents maps, bifurcation maps, Poincare maps and phase plane plots.