The application of thermophotovoltaic energy conversion device to recovery and utilization of high-grade thermal energy are limited by its irreversible loss. In this work, we reveal the source of irreversible loss and provide a strategy for improving the performance of thermophotovoltaic energy conversion device. The maximum efficiency of thermophotovoltaic energy conversion device under ideal condition is determined by using the theory of semiconductor physics and Planck thermal radiation. Moreover, the effects of non-radiative recombination and irreversible heat transfer loss on the electrical, optical, and thermal characteristics of thermophotovoltaic device are considered to predict the optimal performance of thermophotovoltaic device. The optimal region of power density, efficiency, and photon cut-off energy are determined. The obtained results show that the open-circuit voltage, short-circuit current density and efficiency of non-ideal device are lower than those of ideal device. The voltage output and photon cut-off energy of thermophotovoltaic device and heat source temperature can be optimized to improve the power density and efficiency of the device. It is found that the theoretical results are in good agreement with the experimental results, which can provide some guidances fordeveloping the practical thermophotovoltaic devices.