Using a combination of imaging and spectroscopic capabilities, neutron scatter imaging is a novel method of detecting neutrons in an energy range from 1 to 20 MeV. The technique can be applied to measurements in a variety of areas, including solar and atmospheric physics, radiation therapy, and nuclear materials monitoring. Angular resolution is an important parameter for a neutron scatter imaging system. There are some factors causing the uncertainty in the reconstructed image due to the imperfection of the detector system and natures of neutron scattering. These factors mainly are the uncertainties of the position and the energy. In this paper, the contributions of these factors to the angular resolution are discussed. The results show that the angular resolution varies with scatter angle; the position uncertainty not only directly affects the angular resolution, but also indirectly contributes to the angular uncertainty by influencing energy uncertainty; when the detector dimension is less than 5 cm, the energy uncertainty becomes a dominating factor for angular resolution. The prototype is designed based on the above analysis results. The angular resolution of the designed prototype is tested on Cf252 source. The experimental results are basically consistent with the simulation results.