The αη protocol is the random cipher protocol and hides information by the quantum phase noise. In this paper, the security criterion of the protocol is deduced by developing a quantum channel model and calculating the amount of information obtained by the eavesdropper. The results show the security of the protocol depends mainly on the average photon number of the signal quantum states and the number of cipher-text symbols. Considering the above factors, the secure value region of the average photon number and cipher-text symbol number is calculated, which is against the beam splitter attack. Furthermore, the effective communication distance is simulated.