As a new technique of photomicrography of complex optical field, the Fresnel incoherent correlation holography (FINCH) is particularly attractive in recent years because of its incoherent optical recording characteristics. For a new image recording and reconstruction system, a key concern is how to configure the experimental layout of FINCH by using available optical elements to achieve optimal resolution. However, in previous reports, there exist different viewpoints about this issue, and the imaging conditions of the best resolution remain to be clarified. As is well known, the imaging resolution is affected by the effective aperture of hologram and the change of the recording distance between spatial light modulator (SLM) and image sensor (CCD) can cause the hologram aperture to change. In the FINCH system the effective aperture of hologram is related not only to the aperture influence of each element used in the recording system, but also to the overlapping area of interference between the signal and reference wave and the pixel spacing of the image sensor. In previous reports, the researchers mainly used the ray-tracing method to discuss the effective aperture radius of hologram by ignoring the influences of the diffraction of light wave and the pixel spacing size of image sensor on the aperture of hologram. Based on the theories of wave optics we carry out a thorough investigation into the effective aperture of FINCH. We find that the pixelization of the image sensor, e.g. CCD, is a decisive factor influencing the resolution of FINCH, and we adopt numerical simulations and optical experiments to further verify the theoretical conclusions that the optimal lateral resolution of FINCH is achieved only if the recording distance ( Z _h) is equal to the focal length ( f _d) of diffractive lens displayed on a spatial light modulator; the resolution is deteriorated with the increase of $\left| {{Z_{\rm{h}}} - {f_{\rm{d}}}} \right|$ . From the viewpoint of Fourier optics, the smaller the imaging distance $\left| {{Z_{\rm{h}}} - {f_{\rm{d}}}} \right|$ , the larger the aperture angle of hologram ( $ \approx {{{R_{\rm{h}}}} / {\left| {{Z_{\rm{h}}} - {f_{\rm d}}} \right|}}$ ), the higher the collected spatial frequency is, hence, the higher the lateral resolution is. On the other hand, although the FINCH overcomes the spatial coherence limitation, it requires temporally coherent or quasi-monochromatic light. Our study also indicates that the requirements for the spatiotemporal coherence can be eased when the CCD is located at the focal plane of diffractive lens.