Relativistic magnetron is a kind of compact cross-field high power microwave source. It has the virtues of wide frequency tunability and ability to operate with relative lower external magnetic field. To improve the compactness and reduce the size and weight of the relativistic magnetron further, a novel relativistic magnetron using all-cavity output and semi-transparent cathode is investigated theoretically and numerically. By using the all-cavity output structure, the radial dimension is reduced markedly (from 10.5 cm to 6.6 cm) and the axial dimension is also shortened considerably (from larger than 40 cm to less than 30 cm). Since the radiation fields in the interaction cavity are coupled through the coupling hole to the output fan waveguide, the cutoff frequencies of the fundamental mode and three higher order modes in the fan waveguide with different outer radii are calculated. The calculation results show that the mode separation is wide enough for the single mode operation on the fundamental mode. And by using the semi-transparent cathode, the high output efficiency can be obtained and the output characteristics are insensitive to the depth and width of each cathode slot. To verify the characteristic of the proposed magnetron, numerical simulations are carried out by using the three-dimensional particle-in-cell code. After careful optimization, simulations show that with a beam voltage of 395 kV and beam current of 5.6 kA, 1.15 GW output microwave with an efficiency of about 50% can be obtained at S-band with purer mode. The corresponding applied magnetic field is 4.75 kGs (1 Gs=1-4 T). In a relatively large range, both radiation power and the optimal magnetic field increase with the beam voltage. But the output efficiency keeps almost unchanged. The effects of the depth, width and length of the coupling hole, width of the fan waveguide and the distance from the beginning position of the fan waveguide to the coupling hole center Lsc on the output characteristics are also analyzed. Simulation results show that when the dimension of the coupling hole is small, the output power is low. But there is no mode competition and the device works on the up mode. With the increase of the coupling hole, the output power increases accordingly. When the coupling hole is large enough, the mode competition between the up mode and /3 mode becomes so serious that the mode cannot win any more. At the same time, the output power decreases markedly. There also exist optimal values of both the fan width and the beginning position of the fan waveguide (Lsc) for maximal output power.