Combined with the microstructure evolution in amorphous alloys under the external load, a fractional order viscoelastic constitutive model is first derived by replacing a Newtonian dashpot in the classical Zener model with the fractional derivative Abel dashpot. Based on the Hertzian theory and the fractional order viscoelastic constitutive model, a relationship between displacement and load (or time) for an instrumental nanoindentation test with a spherical indenter is then proposed. Finally, a series of nanoindentation test data for an Fe-base bulk amorphous alloy are employed to verify the derived model, and its viscoelastic behavior in the apparent elastic region is analyzed in detail. Results show that the fractional order rheological model has higher fitting accuracy than that of the integer order model, and the fitting parameters of the proposed model are more suitable to reflect the effect of the loading rate on the viscoelastic behavior in the alloy studied. Variation of the above-mentioned fitting parameters exhibits a strong correlation with the microstructure evolution during the loading of this Fe-base amorphous alloy.