Apart from neutrons, the fusion core produces gamma rays during fusion reaction. The spectrum of gamma ray can provide very important information for fusion diagnosis. However, due to the gamma energy and yield in one fusion pulse being both lower, the gamma spectrometer used should have high detection efficiency and energy resolution. The concept of a Gamma-to-electron magnetic spectrometer(GEMS) provides the idea to build up such a spectrometer to meet this requirement. Based on this concept design, four important parts of this facility are investigated. The first part is the gamma-electron converter. The main physics processes include Compton scattering of gamma ray with converter material generating electron, the electron multiple Coulomb scattering(MCS) inside the converter and the electron attenuation. Affected by the thickness of convector, these processes give a complex influence on the detection efficiency and angular-energy distribution of the electrons which are emitted from the downstream face of the convector. The Monte Carlo code Geant4 is employed to investigate theeffects of Compton scattering, MCS and converter thick on the angular-energy distribution. The second one is the collimation. The collimation is used to select the forward direction, the performances of cutoff angle of the collimator on the detection efficiency and resolutions, the correlation between electron transportation direction and energy, are also studied using Geant4 code. The third part is the dipole magnetic field. There are several geometric and magnetic parameters, therefore, a multi-thread parallelized genetic algorithm is developed to obtain the best result. Both the irregular geometric shape and dipole magnetic field strength are optimized to achieve the best energy resolution and detection efficiency. The obtained magnetic field has an intensity of less than 100 Gauss, and its performance on gathering elections is also verified by Geant4 code. The last one is the location of electron detectors. The study shows that all the electron detectors should be located not in a straight line but a quadratic curve. Then the optimized spectrometer is simulated by Geant4 to obtain the responses of gamma rays with various energies. For the gammas provided by fusion reaction, the simulation shows that when the neutron yields are about 2.5×1015 and 1.2×1016, the energy resolutions reach 0.5 MeV and 0.25 MeV, respectively, provided that different thick Be converters are employed. All in all, this optimized GEMS can be employed to measure the spectrum of gamma rays generated fom the fusion reaction.