The navigation mechanism of obtaining phase parameters based on quantum theory can break through the limitation of classical physical limit to navigation accuracy. In order to achieve the accurate estimation of the coherent state phase, it is usually assumed that the local oscillator phase must be orthogonal to the coherent state phase in the method of quantum homodyne detection. However, the coherent state phase is unknown and the hypothesis cannot be guaranteed to be correct in practice. In this paper we design a nonlinear phase-locked loop to solve the problem. Firstly, in order to obtain the Wigner distribution for a coherent state, we start with the Wigner distribution for the vacuum state and analyze the noise characteristics according to Wigner distribution of coherent state, then the output of homodyne detection is derived. Secondly, in order to avoid introducing errors in theory, caused by linearization and cope with the limiting requirement between local oscillator phase and coherent state phase in the phase tracking of coherent state, we design an orthogonal simplex cubature Kalman filter (OSCKF) algorithm to achieve the function of the nonlinear phase-locked loop. The algorithm converges by updating the state of the local oscillator phase multiple times, and then, the accurate coherent phase is obtained. Finally, according to the design of the phase-locked loop, we observe the data of homodyne detection and then verify the correctness of the OSCKF algorithm. The simulation results show that the OSCKF can converge to the real phase after observing 200 sampled data and the accuracy is higher than extended Kalmn filter (EKF) and cubature Kalman filter (CKF), and the real phase can be obtained under different local oscillator phase. In conclusion, the nonlinear phase-locked loop based on OSCKF algorithm breaks the limitation of traditional way in which the initial local oscillator phase is required to be orthogonal to the coherent state phase, and effectively avoid the linearization error and improve the anti-nonlinear ability. It is very significant in theory and application .