Thermoelectric materials, which can convert heat energy into electric energy and also from electric energy into heat energy, have aroused widespread interest of both theoretical and technological researches recently. Graphene is a typical two-dimensional carbon nanomaterial and regarded as a competitive candidate for the next-generation micro/nano-devices. Unfortunately, graphene is an inefficient thermoelectric material due to the extremely high thermal conductivity. To overcome this drawback, exploring an effective way to improve the thermoelectric performance is of critical importance. In this paper, using the nonequilibrium Green’s function approach, we systematically investigate the effects of grain boundary on the thermoelectric properties of graphene nanoribbons. The results show that owing to the existence of grain boundary, the phonons and electrons encounter great scatterings when they transmit through the polycrystalline graphene nanoribbons. These scatterings cause the phononic and electronic transmission coefficient to decrease dramatically, and thus leading the thermal conductance (including both electron and phonon parts) of graphene nanoribbons to be evidently suppressed. Meanwhile, such scatterings induce more intense transmission peaks and pits in the electronic transmission spectrum of polycrystalline graphene nanoribbons. Generally, the Seebeck coefficient depends on the derivative of electronic transmission coefficient. The larger the logarithmic derivative of transmission, the higher the Seebeck coefficient can be obtained. Therefore Seebeck coefficient is improved obviously in the polycrystalline graphene nanoribbons. Based on such two positive effects, the thermoelectric performance of polycrystalline graphene nanoribbons is significantly enhanced. At room temperature, the thermoelectric figure of merit of polycrystalline graphene nanoribbons can approach to 0.3, which is about 6 times larger than that of pristine graphene nanoribbon (figure of merit is about 0.05). It is also found that the quantity of grain boundaries and length of system can further improve the thermoelectric properties of the polycrystalline graphene nanoribbons, while the width of system has a limited influence on it. This is because the quantity of grain boundaries and length of polycrystalline graphene nanoribbons can give rise to more intense phonon and electron scatterings and further decreasing of thermal conductance and enhancement of Seebeck coefficient. The results presented in this paper demonstrate that polycrystalline structure is indeed an effective way to improve the thermoelectric conversion efficiency of graphene nanoribbons, and provide a theoretical guideline for designing and preparing thermoelectric devices based on graphene nanoribbons.