Fresnel incoherent correlation holography (FINCH) has attracted much attention because it is able to record the holograms of three-dimensional (3D) samples under incoherent illumination with just a charge coupled device (CCD) and spatial light modulator (SLM). The FINCH technology achieves the splitting and phase shifting of the incident beam by loading a phase mask on an SLM. Three holograms, whose phase factors are different from each other, are recorded sequentially by a CCD. After the three holograms are superposed in the computer, the zero order image and a twin image are eliminated, and a complex hologram is obtained. The 3D properties of the object are revealed when the complex hologram is reconstructed in the computer. Spiral phase filters (SPFs) are commonly used to produce optical vortices, which can enhance and recognize image edges. In this paper, the spiral phase modulated FINCH system illuminated by Xenon lamp is built, in which the phase-only SLM is space-division multiplexed by a helical lens (superposed by an SPF and a lens) and a conventional lens. The mathematical model of spiral phase modulated FINCH system is established based on wave optics theory. The specific forms of the point spread function (PSF) and the reconstruction distance of the system are given for the first time. Experiments are conducted by using a small aperture with a diameter of 20 nm as a point source, the point source hologram recorded by CCD and the reconstructed image are consistent with the simulated ones. When the system is used for imaging resolution target and unstained onion cells, the edge contrast enhancement effects are obtained without the loss of resolution. The results show that the spiral phase modulated FINCH system can not only improve the edge contrast of the amplitude object, but also extract the edge information or recognition of the phase objects. This method has an important application prospect in the quantitative imaging of phase objects such as in real-time monitoring cell division and deformation of living cells.