Mode competition induces non-stationary oscillations during the operation of a gyrotron backward-wave oscillator (gyro-BWO), which severely reduces its tunable bandwidth and output power. Self-consistent nonlinear theory is used to study the modes-competition mechanism of a W-band fundamental TE01 mode gyro-BWO. Tapered non-resonant interaction circuit structure and loading lossy ceramic are employed to suppress the competing modes, as a way of preventing non-stationary oscillation in the circuit. Systematically optimized interaction circuit is capable of suppressing all the competing modes and can stably operate in the fundamental axial mode of the TE01 mode. Calculation indicates that a peak power of 105 kW and a -3 dB tunable bandwidth of 5.4% are attainable. This is meaningful and provides a theoretical foundation for developing broadband millimeter gyro-BWOs in the applications of counter-measure system, non-destructive detection, plasma diagnosis, material processing, and so on.