In this paper, the influence of opinion dynamics on rumor propagation is studied by introducing the mechanism of stifler’s comments on rumors into the susceptible-infected-recovered (SIR) model. In this model, individuals can form a group together with their direct neighbors, through which the spreader can spread rumors, and the stiflers can express opinions and refute the rumors. The mechanism of rumor refuting can not only reduce the acceptance rate of the ignorant to rumor, but also increase the transition probability of the spreader to be a stifler. In this paper, we use the Erdös-Rényi (ER) random network, scale-free network and real social network as the underlying interaction structure to study the influence of stifler’s silence probability on the rumor spreading dynamics. First of all, we find that the process of rumor propagation can be roughly divided into two stages, i.e., the early stage of free propagation of rumors and the later stage of checks and balances between the stiflers and the spreaders, respectively. Secondly, it is found that the rumor will break out with the increase of stifler’s silence probability. Under a threshold of rumor outbreak, the increase of silence probability will not lead the number of spreaders to significantly increase, but will cause more ignorance to perceive the rumor and quickly turn into spreaders, and then change into stiflers under the guidance of other stiflers. When the silence probability reaches a threshold, the stiflers will not be able to control the spread of rumors, which will lead the number of stiflers to decrease and the rumors to break out. Finally, the early stage of rumor propagation in scale-free networks is shorter than that of random network, which makes rumor more difficult to break out. Our model comprehensively considers the influence of opinion dynamics on the spreading of rumors and more realistically simulates the rumor diffusion process, which provides a useful insight for the rumor control in real-world social networks.