The interaction of extreme ultraviolet (XUV) photon with matter is a meaningful way to understand the electronic structure of microscopic particles. In this paper, the electron angular distributions of single ionization and double ionization of Ne and Xe atoms interacting with XUV photons are investigated by utilizing a reaction microscope. The β-asymmetric parameters of 2p electrons of Ne atom, and 5p, 5s electrons of Xe atom combined with the reported experimental data are compared with those from different theoretical models. The result shows that the electron correlation effect can be ignored in the ionization of 2p electron of Ne atom. While the ionization of 5p electron of Xe atom is strongly influenced by the electron correlation effect, but not by the relativistic effect. These two effects play an important role in ionizing the 5s shell of Xe atom. In addition, this study finds that both direct double ionization and indirect double ionization exist simultaneously during the ionization of Xe atom, and gives the photoelectron angular distributions and the β-asymmetric parameters of the first step and the second step of indirect double ionization.