X-ray ghost imaging is a low-dose, non-localized imaging method, which is of great significance in medical diagnosis and biological imaging. In crystal diffraction based X-ray ghost imaging, the blurring patterns in the diffracted beam, caused by the crystal vibration, can result in a reduction in the contrast and spatial resolution of the reconstructed imaged by ensemble average. In the paper, we systematically analyze the influence of the blurring degree of the speckle patterns from the diffracted beam on the normalized second-order intensity correlation function $ {g}^{\left(2\right)} $ numerically and theoretically. Both demonstrates that as the blurring degree increases, the maximum value of $ {g}^{\left(2\right)} $ decreases and the full width at half maximum broadens, which theoretically proves the blurring degree relating to image quality. In order to solve the above problem, in the paper we propose a $ {G}_{\mathrm{L}\mathrm{H}} $ enhanced ( $ {G}_{\mathrm{L}\mathrm{H}}\mathrm{E} $ ) method to optimize the image quality based on the scheme ( $ {G}_{\mathrm{L}\mathrm{H}} $ ) which directly correlates the bucket signals in diffracted beam with the high-definition patterns in transmitted beam. The simulation experiments exhibit that the $ {G}_{\mathrm{L}\mathrm{H}}\mathrm{E} $ method can improve both the image contrast and the spatial resolution simultaneously. As the blurring degree increases, the difference between the peak signal-to noise ratio of the reconstructed image by the iterative method and that by the scheme $ {(G}_{\mathrm{L}\mathrm{L}}) $ which preprocess the speckle patterns in the transmitted beam through Gaussian filtering, becomes greater. Furthermore, the ${G}_{\mathrm{L}\mathrm{H}}\mathrm{E} $ is almost immune to the additive noise. In summary, the present study provides a feasible idea for the practical application of X-ray ghost imaging based on crystal diffraction.