Coupled mode theory is an effective tool for analysis and synthesis of overmoded waveguides, but the inverse problem has not been solved yet. This paper completed the iterative procedure to solve the inverse problem. The new method can design automatically and fast various mode converters, mode transducers and horn antennas with special radial pattern. Compared with conventional methods, the structure design using the new method has more advantages in electromagnetic and structural properties. Two design examples are given: dual band TM01-TE11 mode converter and smooth-wall feed horn antenna. The two working frequencies of the dual band TM01-TE11 mode converter are 8.75 GHz and 10.3 GHz, and the radius is 16 mm. The converter efficiencies exceed 99% at the two working frequencies. The smooth-wall feed horn antenna converts the TE11 mode to Gaussian beam effectively. Simulation results agree well with the theoretical predictions.