Due to the exciton migration dynamics playing an important role in the photovoltaic process of organic solar cells, which are usually composed of polymer donor and fullerene (or non-fullerene) acceptor, in this paper we propose a new strategy to achieve the ultrafast exciton migration in polymers. Here, the effects of some nonuniform fields on the exciton migration dynamics in polymers are emphasized, such as the nonuniform electric field and the nonuniform polymer packing configuration field. Both of the two kinds of nonuniform fields can be intrinsically existent or modulated in an actual photovoltaic system. In this work, the nonuniform electric field and the nonuniform configuration field are assumed to be separately created by a confined charge and a linear polymer packing, therefore, their model Hamiltonian is established. In dynamical simulations of the exciton migration dynamics in polymers, an extended version of one-dimensional Su-Schrieffer-Heeger tight-binding model combined with a nonadiabatic evolution method is employed. It is found that the nonuniform electric field and the nonuniform configuration field both can drive exciton to an ultrafast migration process. Compared with the exciton migration speed dominated by the traditional Förster or Dexter mechanism, the exciton migration speed dominated by the nonuniform electric field and that by the nonuniform configuration field can be increased by one and two orders of magnitude, respectively. In addition, the driving mechanisms of the two kinds of nonuniform fields for the exciton migration dynamics are separately clarified, where the corresponding driving forces are also quantitatively calculated. Finally, in view of the factors affecting the distributions of the two kinds of nonuniform fields (such as the distance dbetween confined charge and polymer, and the linear packing slope kbetween polymers), we discuss their effects on the exciton migration dynamics. It is found that the exciton migration in polymer can be apparently accelerated by shortening the distance dbetween confined charge and polymer, and there exists a critical value of d, beyond which the exciton will be dissociated into free charges in its migration process. For the linear packing slope kbetween polymers, we find that there exists an optimal value, at which the exciton has the highest migration speed in polymers.