The molecular dynamics method was used to investigate the effect of the impurities N,O,Si,P and S on the Young's moduli of armchair (5, 5) and zigzag (9, 0) single-walled carbon nanotubes. The results show that the Young's moduli of armchair (5, 5) and zigzag (9, 0) single-walled carbon nanotubes are 948 and 804 GPa, respectively. When the impurity concentration is less than 10%, the Young's moduli are approximately linearly decreasing with increasing of the impurity concentration, and the greatest decreasing ratio is induced by the impurity Si and the smallest by the impurity N. The decreasing rate of the Young's modulus increases with increase of the impurity atomic number when the impurity element is of the same period with the element C. The effect of the impurities on the Young's modulus of carbon nanotubes is the stronger when the period of the impurity element is different with the element C, and the decreasing rate of the Young's modulus decreases slightly with increasing of the impurity atomic number. The reasons are analyzed by the laws of the Young's potential energy variation of carbon nanotubes with impure and the electron cloud coupling between two atoms from the theory of the local density approximation based on the density functional theory.