The static polarization-difference interference imaging spectrometer, based on the combination of Wollaston prism and Savart polariscope, can simultaneously acquire hyperspectral images for orthogonal polarization components. Optical specifications are determined according to the principle of interference spectroscopy and the technical parameters of an actual detector. A design scheme is demonstrated. The Savart polariscope, Wollaston prism and Glan-Taylor prism are designed according to the birefringent features of uniaxial crystal and the wave normal tracing. More emphasis is put on the influences of the dispersion of uniaxial crystal on the incident angle, thickness, and apex angle. This research can provide a theoretical guide in the engineering of the polarization-difference interference imaging spectrometer.