Along with the variation of the feedback amplify coefficient, V2 controlled Buck converter exhibits abundant nonlinear dynamical behaviors. By establishing the discrete-time model of the system, this paper has studied the instability phenomena based on the monodromy matrix method. With increasing feedback factor, the analysis indicated that the converter entered from a stable period-one statue into a period-doubling statue. Finally, it showed chaos. Mechanism of the bifurcation generated by the system was fully analyzed based on the monodromy matrix, which showed that as the increase of the feedback coefficient, an eigenvalue of the monodromy matrix went out of the unit circle; this was the reason why the system generated period-doubling bifurcation. Also presented was the sinusoidal voltage compensation method to extend the stability margin based on the monodromy matrix theory, by which the instability behavior was effectively handled. Simulation and experimental results confirmed the analytical method.