Plasmonic nanocavities can effectively modulate the upconversion luminescence properties of lanthanide doped upconversion nanocrystals (UCNCs), which not only enhance the luminescence intensity, but also modify the luminescence spectrum. However, currently reported studies on upconversion luminescence spectrum modulation using nanocavities are mainly based on ensemble experiments. Compared with ensemble experiments, single-particle experiments facilitate comparative studies for the same upconversion nanocrystal and therefore the influence of inhomogeneity in ensemble samples can be avoided. Here we couple a single particle of Yb3+/Tm3+ co-doped nanocrystal with a plasmonic nanocavity composed of a single gold nanorod using in-situ nano-manipulation technique based on an atomic force microscope. Experimentally, we compare the upconversion luminescence spectra, upconversion luminescence lifetimes and excitation-power dependent upconversion luminescence intensities of the same single nanocrystal before and after coupling with the single gold nanorod. The experimental measurements are consistent with the theoretical calculations using rate equations combined with electromagnetic simulations. The results indicate that the modulation of the nanocrystal upconversion luminescence spectrum with plasmonic nanocavity is the combined result of three effects: the excitation field enhancement effect, the Purcell effect and the change of radiation efficiency. The study may help to gain a deeper understanding of the mechanism in upconversion luminescence modulations based on coupling with plasmonic nanocavities.