The generalized likelihood ratio test (GLRT) detector and its theoretical detection performance for an underwater narrowband source with an unknown position are both given in this paper. Via the eigenvalue decomposition of the mode correlation matrix (MCM), the GLRT detector is decomposed into different spectrum components corresponding to the eigenvalues of the MCM. Based on the derived statistical property of each component, the spatial processing gain of each spectrum component with respect to the input signal is obtained, which is proportional to the corresponding eigenvalue. As there are several eigenvalues of the MCM approaching to zero when the modal information is incompletely sampled, the components corresponding to these small eigenvalues contribute much less spatial processing gain to the input signal than other components. By discarding the components corresponding to these small eigenvalues, the effective spectrum detector (ESD) is proposed, of which the target signal component in the output is approximately identical to that of the GLRT detector, and the noise in the ESD output is much less. Therefore, a much more robust detection performance is obtained by ESD than by the GLRT detector. Numerical simulations in a typical shallow water environment demonstrate that 1) the theoretical analyses, derivations and the effectiveness of the proposed ESD are verified; 2) the more incomplete the modal information sampling is, the more significant performance improvement of ESD over the GLRT detector can be acquired; 3) the numerical stability of the ESD is better than that of the GLRT detector.