Residual stresses will be formed around the mitigated site after the damaged site is irradiated by 10.6 m CO2 laser. Using those mitigated sites can improve the damage resistance ability in optics, and once the reinitiating damage occurs, the damaged site will grow under the subsequence irradiation and large fracture may form around the mitigated site. In this study, the annealing temperatures 650, 750 and 850 ℃, and time durations 6, 8, 10 and 12 h are used to anneal the samples. The sample annealed at 750 ℃ is the main research object of this study, while the sample annealed at 650 ℃ or 850 ℃ is only treated for 10 h. The differences of damage growth morphology and velocity of mitigated site on fused silica treated under those annealing conditions are investigated when it is damaged once again. Results are also compared with the damage growth behaviors of the unannealed substrate and mitigated site. It is indicated that the damage growth data still fit to an exponential curve even for the unannealed mitigated site. However, for the unannealed mitigated site, a more serious and larger size of damage site will be formed when the reinitiating damage occurs. It is mainly attributed to the fast propagation of crack under the effect of residual stress around the mitigated site. This behavior can be effectively controlled by the annealing treatment. Results show that the crack propagation behavior can be avoided when the retardation of mitigated sites is controlled in the range of 25 nm; moreover, the damage growth velocity and coefficient will gradually decrease with the increase of the annealing duration and annealing temperatures. A notable result indicates that there is no difference between the mitigated site and substrate when the retardation of mitigated sites is controlled below 10 nm, especially for the samples treated at 750 ℃ for 12 h and 850 ℃ for 10 h. Moreover, the reported investigation indicates that the stresses can still improve the damage resistance ability in optics. This is the most desirable outcome of the annealing treatment. Thus, the investigation results can provide a reference on how to analyze the effect of stress on damage growth of mitigated site and optimize the annealing parameters.