Graphene thin films are often used to manufacture various optoelectronic nanodevices owing to their advantages such as light weight, small size, high quality factor, and good conductivity. So far, there have been few studies of the four-wave mixing characteristics in a bilayer graphene nanosystem, especially theoretical research. In this work, we study theoretically the four-wave mixing properties in a bilayer graphene nanosystem. Firstly, the analytical formula of the four-wave mixing signal is derived by quantum mechanical method, which is divided into three steps. 1) Total Hamiltonian of the system is written in the rotating wave approximation. 2) By using the Heisenberg equation of motion and the commutation relations between different operators, the corresponding density matrix equations are obtained. 3) To solve these density matrix equations, we make an ansatz and obtain the analytical formula of the four-wave mixing signal. Secondly, we explore the dependence of the four-wave mixing signal on the phonon-exciton coupling strength, pumping intensity and the detuning between the exciton and the pump field. The calculated results show that the lineshape of four-wave mixing spectrum can be switched among two-peaked, three-peaked, four-peaked, five-peaked and six-peaked by adjusting the phonon-exciton coupling strength, the pumping intensity, and the detuning between the exciton and the pump field. In a weak phonon-exciton coupling regime (i.e. phonon-exciton coupling strength
g< dephasing rate of exciton
Γ
2), the intensity of the left peak and right peak of four-wave mixing signal first increase and then decrease with the increase of the pumping intensity
$ {\varOmega }_{{\text{pu}}}^{\text{2}} $
. In the intermediate and strong phonon-exciton coupling regime (i.e.
g=
Γ
2and
g>
Γ
2), the four-wave mixing spectrum exhibits a two-peaked structure. The maximum values of these two peaks increase as
$ {\varOmega }_{{\text{pu}}}^{\text{2}} $
increases, and their spacing is equal to 2
g. Especially, for a given pumping intensity
$ {\varOmega }_{{\text{pu}}}^{\text{2}} $
(= 10 THz
2), the maximum value of the peak for
g= 4 THz becomes 0.4% of that for
g= 1 THz, indicating that the phonon-exciton coupling inhibits the enhancement of the four-wave mixing signal to a certain extent. Our findings can not only offer an efficient way to measure the phonon-exciton coupling strength in the bilayer graphene system, but also help one to further explore the underlying physical mechanism in such a nanosystem.