In an adaptive optical system, the optimal modal control method refers to applying proportional integral control of different bandwidths to the wavefront aberrations after modal decomposition to achieve better closed-loop results than the unified bandwidth modal control. The optimal modal gain usually needs to be obtained by ergodic solution based on the transfer function model of the adaptive optical system, the measured disturbance power spectral density, and the noise power spectral density, which usually takes a long time. Owing to the time-varying statistical characteristics of atmospheric turbulence, it is difficult to ensure the timeliness of the optimal modal gain. Therefore, we propose a method of fast estimating optimal modal gain based on quadratic polynomial fitting. In the method, it is only necessary to choose three reasonable gain coefficients and calculate their corresponding closed-loop residual errors respectively in order to estimate the optimal gain of single mode. The simulated slope data used in this work are cited from Lijiang 1.8 m adaptive telescope system, which consists of a 241-unit deformable secondary mirror and a Shaker-Hartmann wavefront sensor with 192 sub-apertures, with the first 135-order modes corrected by modal method. Our experiment is to test directly on-line on this system. The results show that under the same atmospheric environment, the proposed method can accurately estimate the optimal modal gain in a very short time and effectively suppress the high-order wavefront aberration. At the same time, owing to the reduced time complexity of the algorithm, the improved optimal modal gain estimation method takes only 0.33 s. Comparatively, it will take 7.08 s to obtain the optimal modal gain coefficient by using the parameter traversal method. Therefore the time spent on obtaining the optimal modal gain is shortened by about 95.3%, which is easier to meet the real-time requirements of the telescope, and beneficial to the adaptive optics system with more high-order modes. For the future adaptive optics system with more than one-thousand units, the proposed method can update the optimal gain to the second level, while the traversal method can only reach the minute level.