Aiming at the application of multilayer dielectric gratings (MDGs) in spectral beam combining (SBC) systems, a theoretical model of rectangular MDGs is built up and a beam propagation model of SBC systems based on the rectangular MDGs is further developed. The phase modulation introduced by the rectangular MDG is composed of the optical path difference on the top surface of reliefs and that of the relief structure itself, and is affected by the MDG parameters such as the grating groove depth, the grating duty cycle, the incidence angle of the central beam, etc. By the diffraction integral method and the principle of incoherent superposition, the intensity distribution of the combined beam of the SBC system based on the rectangular MDGs is numerically calculated and analyzed. Additionally, the beam quality of the combined beam is also obtained by the intensity second-order moments method, and the effects of the MDG parameters such as the groove depth, the duty cycle, the incidence angle of the central beam, and the fabrication errors of the MDG on the characteristics of the combined beam of the SBC systems are simulated and discussed in detail. The simulation results show that the beam quality of the combined beam after passing through the SBC systems is significantly better than that of the laser array. Since the quality of the combined beam is almost the same as that of an individual laser beam, for a SBC system without fabrication error, changing the groove depth, the duty cycle of the rectangular MDG or the incidence angle of the central beam does not affect the beam quality while it has obvious influence on the energy of the combined beam. This is mainly because the diffraction efficiency of the rectangular MDG depends on both the parameters of the MDG and the incidence angle of the central beam. However, fabrication error of MDG is unavoidable, and the fabrication error has a significant effect on both the beam quality and the energy of the combined beam. Compared with the effect generated by the groove depth error on beam quality, the influence introduced by the duty cycle error is more obvious. It is worth mentioning that the theoretical model of the SBC system based on the rectangular MDG can be applied to some other high-power laser systems due to its advantages such as low absorption and high damage threshold.