A static, compact, high optical throughput polarization-difference interference imaging spectrometer is presented. It is based on the combination of the angular shear of Wollaston prism and the lateral shear of Savart polariscope. Its remarkable characteristic is the simultaneous acquisition of the interferograms and two-dimensional images for the orthogonal polarization components of targets. Polarization spectral images of the components can be reconstructed using the algorithms of Fourier transformation and image fusion. A summation of the orthogonal polarization spectral images is equivalent to that of the conventional intensity spectral image. The difference is that the orthogonal polarization spectral images is named as polarization-difference spectral image. A higher contrast of the polarization-difference spectral image can not only provide more detailed texture information for the targets, but also can derive the size, concentration, and refractive index of elastic-scattered particles. The principle of polarization-difference interference imaging spectrometer is demonstrated in this paper. Expressions for interference intensities and theoretical simulation are presented. The mode for acquiring data is described. Approximate paraxial conditions of Fourier imaging lens for the formation of straight fringes are analyzed by using the Young's interference mode; and the influence factors for the visibility of fringes are analyzed. For the realization of a more compact system, the relationship between the collimating lens and imaging lens is analyzed in terms of optical field diffraction theory. The feasibility of the system is demonstrated by numerical simulation. This research can provide a novel development strategy for imaging spectropolarimeter.