One of the key technologies of interferometric optical fiber sensing system is to overcome polarization-induced signal fading in interferometric fiber sensors. The theoretical and the experimental investigations of the residual polarization-induced phase noise (PIN) in single-mode optical fiber Michelson interferometer based on Faraday rotation mirror (FRM) are conducted in this paper. A theoretical model of the residual PIN based on Jones matrix is developed. Three leading influencing factors of the residual PIN are educed: the rotation deviation angle of FRM, the state of polarization of input light, and the birefrigent effect of the optical fiber. And three methods are proposed to overcome the residual PIN. The maximal PIN with its amplitude of 0.0815 rad possibly exists in the sensing system when the polarization modulation degree equals 1.84 rad. The correctness of the theoretical model is proved by the experimental results.