The method of optimally designing experiments is introduced to study the properties of up-conversion luminescence in Tm3+/Yb3+ co-doped NaY(MoO4)2 phosphors. The design of experiment, data analysis and objective are all optimized. Uniform design is used to seek for the doping concentration range, and quadratic general rotary unitized design is included to build the equation between the luminescent intensity and the Tm3+/Yb3+doping concentration. In order to obtain the best luminescent intensity, genetic algorithm is introduced into our calculation. The optimal Tm3+/Yb3+ co-doped NaY(MoO4)2 phosphor is synthesized by high temperature solid state method. The up-conversion emission spectrum under the 980 nm laser excitation is detected and the luminescence mechanism is proposed. We observe intense blue light (476 nm) and week red light (649 nm) at room temperature, which are emitted from the Tm3+ transition of 1G4→3H6 and 1G4→3F4, respectively. In this up-conversion uminescence system, the up-conversion emission of 1G4 is due to the cooperation energy transfer of two photons. We also discuss the temperature effect of the sample, and as the temperature increases its luminescent intensity decreases.