The traditional opto-mechanical coupling in an opto-mechanical system is a linear coupling which is proportional to the field intensity I and oscillator displacement x. The nonlinear spatial coupling effect becomes obvious and important in a strong cavity field with large oscillating amplitude, and then the nonlinear effect with quadratic coupling in opt-mechanical device is also significant. In this article, we find that a general opto-mechanical system with quadratic coupling will produce a stable self-sustained oscillation when the energy injected by external driving equals that of dissipations in certain parametric regions. We numerically solve the semi-classical equation of motion of the system and find high-dimensional limit circles in its phase space under the control of driving and damping. We verify the high-dimensional limit circles by the closed orbits in all the projective three-dimensional phase space and show a highly controllable topological structure of the phase orbit which is very similar to Lissajous figures formed in a two-dimensional case. The self-sustained oscillations of the driving resonator with controllable amplitudes and frequencies demonstrate a reliable physical application of opto-mechanical system under quadratic coupling.