A novel signal processing method named adaptive variational mode decomposition with the fractal (AFVMD), which is based on variational mode decomposition and fractal theory, is proposed in this paper for solving a problem that it is easy to misjudge the working conditions of the centrifugal compressor. The measured signal of a compressor is unstable, so a traditional method is used to analyze the nonlinear phenomenon of the stall flutter. Owing to the fact that the robustness of VMD method is strong and its combination with the fractal dimension can accurately describe self-similarity and fractal characteristics of a measured signal, the proposed AFVMD method can not only achieve noise reduction, but also extract nonlinear feature from a complex signal. Taking the dynamic pressure data of the impeller during the instability of a centrifugal compressor as an object to verify the effectiveness and superiority of the proposed AFVMD method, the results are obtained as follows. Firstly, compared with the wavelet noise reduction method, the proposed AFVMD method has both noise reduction and feature extraction functions, and the compressor pressure pulsation spectrum has more significant stall characteristics. Secondly, none of the traditional nonlinear analysis methods can reflect the stall process, so the chaotic phase space attractor is used to visualize the flow field changes. Due to the reasonable choice of the delay time and the embedding dimension, the physical information originally mixed in the signal is separated, so that the attractor phase diagram method has a better process of judging the flow stall than the frequency spectrum method. The results show that the proposed AFVMD method can judge the compressor about to enter into the deep surge earlier. Thirdly, In order to quantify the superiority of the proposed method, if the process of surging and the occurrence of deep wheezing can be predicted in advance, the largest Lyapunov exponent is used as an evaluation index. The above results show that the largest Lyapunov exponent of the proposed AFVMD is smallest for illustrating that the signal has more accurate flow field nonlinear information, which improves the predictability of the signal.