A series of Ge-Se chalcogenide glasses incorporated with same molar percentage of Ga, Sn, Sb and Te are synthesized by melt-quenching method. The variations of optical band gaps doped with different elements are investigated by absorption spectra, and the relationship of optical band gap with glass network structure is studied by Raman spectra The results show that the doping of heavy metallic elements (except Ga) could reduce the optical band gap of the Ge-Se glass due to the decrease of the number of Se-Se chains or ring bonds. Third-order optical nonlinearities of the glasses are studied by femtosecond Z-scan method at a telecom wavelength of 1550nm. The results show that the performance of third-order optical nonlinearity of the Ge-Se glass could be improved by doping the above-mentioned elements. By comparison, the Sn-doped Ge-Se glass has a maximum nonlinear refraction index (n2) of 6.36× 10-17 m2/W and a figure of merit of over 23. By combining the experimental results from Raman spectra, the enhancement of third-order optical nonlinearity after the introduction of Sn can be ascribed to the formation of Sn(Se1/2)4 tetrahedra that enters into the main frame of Ge-Se glass and results in a stable Ge-Sn-Se network. Te doping could also remarkably enhance the n2 value of the Ge-Se glass, however, it could cause large two-photon absorption, leading to a poor value of figure of merit. The research result shows that chalcogenide glass in Ge-Sn-Se ternary system is an ideal candidate material for designing and fabricating infrared devices with high performance and environmental friendness.