To simulate the atmospheric polarization pattern for small solar elevation angle, we develop a the vector radiative transfer model VSPART (vector pseudo-spherical radiative transfer model considering refraction), and use it to calculate the polarization state of downwelling diffuse light. In this model, the propagation trajectory, transmittance rate and polarization states of directly transmitted light are tracked by ray-tracing method for spherical refractive atmosphere. Based on the matrix algorithm, an improved method to solve the radiative transfer equation is proposed. Output of this model includes not only the Stokes vector and degree of polarization of diffuse light, but also the polarized irradiance. The precision of VSPART is validated against the benchmark results, literature results and SPDISORT, and excellent agreement is achieved. DOP (degree of polarization) and AOP (angle of polarization) are simulated for pure Rayleigh scattering atmosphere and atmosphere with aerosol, and the characteristics of their angular distributions are analyzed. In addition, the influences of atmospheric spherical geometry and refraction effect on the sky DOP are discussed as well. Simulation results show that for low solar elevation angle, with the increasing of wavelength, DOP increases gradually, and the Arago and Babinet neutral points move towards the horizon when Rayleigh scattering atmosphere is considered. Although the existence of aerosol does not change the basic distribution of DOP, it has a significant influence on AOP. With the increasing of aerosol optical depth, DOP decreases gradually, and the distribution of AOP changes dramatically. By comparing the sky distribution of DOP, it could also be concluded that the neutral points might arise from low order scattering. The area affected by atmospheric spherical geometry and atmospheric refraction effect mainly includes the area near horizontal directions, the area near the neutral points and the area perpendicular to the ground. For pure Rayleigh scattering atmosphere, the influence is reduced with the increasing of the wavelength of incident light, especially for the areas near the neutral points, where the influence gradually disappears as wavelength increases. For atmosphere with aerosol, with their optical depth increasing, the effects of atmospheric spherical geometry and atmospheric refraction are gradually enhanced.