Aiming at the problem of passive detection of broadband sources in underwater acoustic vector signal processing, a novel detection algorithm based on beam-space transformation is proposed. The principle of spatial spectrum detection with human eyes is employed for reflerence, and the generalized likelihood ratio test (GLRT) is applied to the beam-space. First, the design criterion of beam-space transformation matrix is studied for the compreflensive consideration of the environment of multiple targets and the characteristic of vector ambient noise field, so that the analytical solution is obtained. Second, assuming that the number of beams not containing the target signal is given, the probability density function (PDF) model of beam-space data is constructed, and the new GLR test is made by calculating the maximum likelihood estimate of the unknown variables in PDF. Finally, the information of theoretical criterion is adopted in order to estimate the number of beams not containing target signals. The processing gain and the threshold value of this test statistics are also discussed, and the specific implement is explained in detail. Theoretical analysis and simulation results show that under the complex conditions of strong target interference and ambient noise with undulated and time-variant power spectrum, the proposed algorithm can give the processing result with higher gain and detection threshold at constant false alarm rate (CFAR); the results of lake experiment further prove the favorable and robust detection performance.