In this paper, we investigate the ground state phase transition of the spin-1/2 frustrated model on a stacked square lattice by the linear spin wave method. The effects of the anisotropy and the interplay of neighboring couplings on phase diagram and phase transformation between the Nèel state and collinear state are explored. Our results show that both the Nèel state and collinear state can exist due to the interplay of interlayer coupling $ {J_{\text{c}}} \geqslant 0.21 $ or $ {J_{\text{c}}} \leqslant - 0.175 $ for an isotropic system. For an anisotropic system, both the Nèel state and collinear state may also exist. In this case, for the weak anisotropy, the Nèel state is more stable. For the strong anisotropy, the system will undergo a first-order phase transition from the Nèel state to the collinear state.