This paper discusses the influence of the polarization of incident X-vay on the reconstructed atomic image by numerical simulation and reconstruction, using a crystalstructure composed of 27 Fe atoms arranged in a cubic lattice as the model. Calculated results show that the choice of polarization is crucial for the atom observability. This paper proposes a new reconstructing algorithm which can eliminate the polarization effect of incident X-ray source and twin images inherent to holography at the same time. Numerical simulation and reconstruction results show that the reconstructing algorithm is effective and can be used to eliminate the influence of polarization of incident light source and twin image in experiments.