The signal-to-noise ratio and resolution of the reconstructed image is seriously influenced by speckle noise in digital holography, so it is essential to reduce the speckle noise and improve the image quality. Intensity correlation between two speckle patterns with the rectangle speckle spot is analyzed and deduced, and the minimal angular difference of illumination beams is given quantitatively under a special situation. A lensless Fourier transform holographic imaging system with angular diversity is designed based on the lens property, in which the direction of the wave can be changed by shifting the fiber instead of conventionally rotating the mirror, and the formor can change the direction of illumination with a fixed illumination size and location. Eighty one holograms with uncorrelated speckle patterns are recorded at different illumination angles by shifting the fiber. Then a digital reconstruction is achieved only by a fast Fourier transform, and the multiple reconstructed images are averaged. Experimental results show that the speckle contrast of the averaged reconstructed image can be reduced to 14.6% that from a single reconstructed image, and the signal-to-noise ratio is improved 6.9 times. This proposed digital holographic imaging method can suppress the speckle noise greatly, has a simple setup, and is easy to operate.