In quantum key distribution (QKD), the users need to share the same reference frame. If their reference frames are inconsistent, the QKD system will not function properly. The most widely used method today is the active real-time calibration of both communication reference frames by using classical communication. In order to get rid of the real-time calibration operation of the reference frames, a QKD protocol independent of reference frame is proposed, called reference-frame-independent QKD (RFI-QKD). The RFI-QKD protocol is immune to the effects of slowly changing reference frame drift, requiring that only one set of bases should be aligned by Alice and Bob, and the remaining two sets of bases can slowly change in the channel. In the real QKD system, a set of basis vectors can always be found to maintain a stable alignment. However, some assumptions are made for the sources in most reported researches, i.e. with a trusted and fixed photon-number distribution (PND), which usually cannot be satisfied in practical implementations. Those unreasonable assumptions will inevitably compromise the security of practical QKD systems. To solve the problem, in this work, we present a passive light source monitoring (PLSM) scheme for RFI-QKD, which is accomplished by a passive monitoring module consisting of a beam splitter and two detectors on the source side. Through the PLSM module, we can have four monitoring events by using two local detectors and then precisely estimate the bounds of source distributions. Specifically, we take the three-intensity decoy-state-based RFI-QKD for example for illustraing the events. Compared with the original RFI-QKD, our PLSM method can passively monitor the PND and has many advantages, with light source fluctuations, finite-size effects and reference frame deflection angles taken into account.