Gyrotrons are the most promising microwave source devices that can be used in the International Thermonuclear Experimental Reactor, but there are many difficulties to be solved in study and design of gyrotrons to meet the requirements. In this paper, the beam-wave interactions of a 170 GHz megawatt-level smooth-wall coaxial gyrotron are studied numerically. In order to attain high efficiency and stable radiation, TE31,12 mode that lies in a relative sparse spectrum is selected as the operating mode, and the beam-wave coupling coefficient and start oscillation current are calculated by a set of source codes developed by Matlab. Taking into account the electronic velocity spread and cavity wall resistivity, and based on a single-mode approximation, the optimization design and simulation of beam-wave interaction of a 170 GHz megawatt smooth-wall coaxial gyrotron have been fulfilled. The relationships between efficiency and magnetic field, and the voltage, current, taper angle of insert, and other parameters are presented. Results show that the voltage and magnetic field have great influence on efficiency; however, the current and velocity spread do change slightly, thus reduce the requirements of electron gun design. In addition, the optimized taper angle of insert and coaxial cavity geometry parameters can improve the efficiency, reduce the impact of velocity spread on efficiency, and can achieve an electronic efficiency around 50% and an output power 1.7 MW.