In this paper, we investigate the ground state phase transition of the spin-1/2 frustrated model on a stacked square lattice by means of the linear spin wave method. The effects of the anisotropy and the interplay of neighboring couplings on phase diagram and phase transformation between the Nèel state and collinear state are explored. Our results show that both the Nèel state and collinear state can exist at the interplay of interlayer coupling Jc=0.21 or Jc=-0.175 for isotropic system. For anisotropic system, both the Nèel state and collinear state also may exist. In this case, for weak anisotropy, the Nèel state is more stable. For strong anisotropy, the system will undergo a first-order phase transition from the Nèel state to the collinear state.