In this paper, the modal space detector (MSD) is investigated in shallow water environment when utilizing a vertical linear array. The processing gain of the MSD is derived, and the result demonstrates that the processing gain of the MSD degrades when the number of the propagated normal modes excited by the underwater acoustic source increases, and therefore the detection performance of the MSD decreases. By exploiting the orthogonality among the modal depth functions, the MSD can be decomposed into a group of modal subspace detectors (MSSDs). The processing gains of these MSSDs are derived as well and it is found out that the processing gain of a MSSD is in direct proportion to its corresponding modal attenuation coefficients. By designing a group of weighting coefficients based on the mode attenuation law, a weighted modal subspace detector (WMSSD) is proposed to alleviate the degradation of the processing gain processing of the MSD. We analyze the influences of acoustic source locations and sound velocity profiles (SVPs) on the detection performance of the WMSSD theoretically, and verify the theoretical analyses by comparing its processing gain with the MSD in simulation experiments. The results show that the WMSSD presents various processing gains versus different acoustic source locations. In the waveguide having a negatively-gradient SVP, there exists a ‘weak detection area’ for the WMSSD, that is, the processing gain of the WMSSD is smaller than that of the MSD when the acoustic source locations are close to sea surface. The reason is because there are inversion points on the lower-order modal depth functions and the depths of the inversion points are close to sea surface. In other most areas, the processing gain of the WMSSD is larger (even remarkably larger) than that of the MSD. In the waveguide having a positively-gradient SVP, due to the phenomenon that the modal inversion points of the lower-order modal depth functions are near sea floor, there is a contrary consequence, that is, the ‘weak detection area’ is close to sea floor. And meanwhile the WMSSD outperforms the MSD in other most areas as well. There are no modal inversion points in the waveguide having a constant SVP, and therefore the WMSSD always outperforms the MSD.