The high-speed deep space communication is one of the key technologies for deep space exploration. Laser communication system equipped with sensitivity of single photon will improve existing deep space communication speed. However, laser communication at single photon level needs to consider not only the effect of transmission environment, but also the performance of used single photon detector and the photon number distribution. As a new single photon detector, superconducting nanowire single photon detector (SNSPD) outperforms the traditional semiconducting SPDs at near infrared wavelengths, and has high detection efficiency, low dark count rate, low timing jitter, high counting rate, etc. The SNSPD can be used for detecting single photons efficiently, rapidly and accurately. In this paper, we introduce the system detection efficiency and dark count rate of SNSPD based on the photoelectric detecting model without considering the effect of atmospheric turbulence, establish the mathematical model of bit error, and put forward the formula of system bit error rate. What should be emphasized is that the bit error rate is an important parameter for measuring the performance of laser communication system. Error is partly from background thermal radiation and circuit electromagnetic interference; in addition, error appears when photons reach the surface of device without being absorbed to successfully produce resistance area or photons are absorbed but there occurs no response. As a result, the calculation of bit error rate includes the whole process of photoelectric conversion. In order to analyze how to affect the size of system bit error rate, first we simulate two factors of the formula, i.e., light intensity and laser pulse repetition frequency. The results show that the light intensity has the greatest influence on error bit rate. With the light intensity increasing from 0.01 to 1000 photon/pulse, the error bit rate significantly decreases from 10-1 to 10-7 level. The influence of laser pulse repetition frequency is restricted by the light intensity, which declines with the increase of pulse repetition frequency. Then we measure the error bit rate experimentally, which validates the simulation model. However, when increasing light intensity or speed, experimental bit error rate is about 10-4 times higher than simulation result. The reason may be that the insufficiency of actual communication modulation extinction ratio of optical signal to the background noise through optical fiber increases the dark count rate. The above model and experimental results could be the foundation of high-speed deep space laser communication such as moon-earth and Mars-earth based on SNSPD.