Based on the one-dimensional tight-binding Su-Schrieffer-Heeger (SSH) model, and using the molecular dynamics method, we discuss the dynamics of electron and hole polarons under the influence of impurity potentials and the distance between impurities. Under an external electric field, the electron or hole polaron can move along the polymer chain with a steady velocity. When the polarons collide with impurities, the velocities of the polarons would be affected by the impurity potentials and the distance between the impurities. 1) Firstly, at a fixed impurity potential strength, the average velocities of the electron and hole polarons as a function of the distance (2-16 times the lattice constant) between impurities have been discussed in polymers. It is found that the average velocities of the electron and hole polarons increase with increasing distance between impurities. It is worth noting that the average velocities of the electron polarons are greater than those of the hole polarons, which results from the fact that the electron and hole polarons have different coulomb interactions with the impurity ions. That is to say, the coulomb repulsion is shown between the electron polarons and impurity ions, which is similar to the potential barriers; while the coulomb attraction appears between the hole polaron and impurity ions, which is similar to a potential well. However, as the distance between the impurity ions becomes large enough, the average speeds of the electron and hole polarons almost remain the same, and show just a few small oscillation. This is due to the different distances between impurity ions which generate the different superposition effects of barrier or potential well on the electron and hole polarons. 2) Next, with a fixed distance between the two impurity ions, we find that with the increase of impurity potential strength, the average velocities of the electron and hole polarons decrease. And the decrease of the average speed of the hole polaron is more obvious. It can be explained as follows: the coulomb attraction interactions between the hole polaron and impurity ions can obviously enhance the localization of the hole polaron. While the coulomb repulsion interactions between electron polaron and impurity ions can only make the electron polaron undergo a small shift in the polymer chain, so that the localization of it is almost unchanged. In view of the average speed of the polaron being closely related to the localization of the polaron, the change of the average speed of the hole polaron is more obvious. The results above may provide some theoretical basis for understanding the conduction properties in doped polymers.