Fresnel incoherent correlation holography (FINCH) is a unique scanning-free three-dimensional imaging technique which enables holograms to be created from incoherent light illumination. However, the image quality is inclined to be destroyed by various optical aberrations, in the practical application of microscopic imaging. In order to solve this problem, some kinds of adaptive optics are combined with imaging technologies to detect the distorted wavefront and compensate for the aberrations. Phase diversity is an image-based adaptive optics method where two intensity images with a certain phase difference are used for wavefront sensing. In this paper, we develop an adaptive imaging technique by Fresnel incoherent digital holography combined with phase diversity (PD-FINCH). Two recorded phase-shift holograms are applied to wavefront sensing, and the phase of aberration is further extracted by phase diversity reconstruction algorithm. The compensation phase is uploaded on SLM in turn, thus the aberrations are corrected while recording holograms. Both the simulation and experimental results verify the validity of phase diversity in FINCH. All the results show the improvement of reconstructed image quality after wavefront aberration compensation.