In recent decades, as a kind of critical material in spintronics field, the diluted magnetic semiconductor with high temperature and intrinsic ferromagnetism has attracted extensive attention. In order to explore the approach to enhancing Curie temperature ( T _C), the LDA+ Umethod of the first-principles calculation is adopted to study the effect of strain on electronic structure, magnetic and optical properties in Mo doped GaSb system. The results indicate that the structure of GaSb is stable with strain in a range of –6%—2.5%. Plasticity and toughness of GaSb increase under compressive strains, which is conducive to the improvement of the mechanical properties. The strain affects the electronic structure of Mo _Gagreatly. In a range from –3% to –1.2%, Mo _Gais in the low spin state (LSS) with a 1 ${\mu _{{\rm{B}}}}$ local magnetic moment, while in a range of –1.1%—2%, Mo _Gais in high spin state (HSS) with a 3 ${\mu _{{\rm{B}}}}$ moment. The magnetic interactions between Mo _Gaand Mo _Gaare all ferromagnetic for LSS and so is the case for HSS, although they are different in coupling intensity and mechanism. In particular, appropriate compressive strains can improve the strength of ferromagnetic coupling effectively and are favorable for the preparation of the GaSb-based diluted magnetic semiconductors with high Curie temperatures and inherent ferromagnetism. Moreover, we find that Mo can greatly improve the polarization capability of GaSb and play a vital role in forming and separating the electron-hole pairs, and thus further improving the photoelectric conversion capability for long wave photons. The energy required to absorb photons for inter-band transition of electrons decreases because of the impurity levels induced by Mo, which leads the absorption edge to be red-shifted. The optical properties of (Ga,Mo)Sb in infrared region are further enhanced by the tensile strain.