Buckling behavior of boron nitride nanotubes under combined axial compression and torsion is presented by using molecular dynamics simulation. In order to study the effect of helicity and nanotube size, three groups of nanotubes are considered. The first group is a pair of boron nitride nanotubes with a similar geometry but different helicities, the second group includes three armchair naotubes having equal length but different radii, and three armchair (8, 8)-nanotubes with different lengths form the third group. The simulation is conducted by applying Nose-Hoover thermostat in a temperature range from 50 K to 1200 K. Based on the interatomic interactions given by Tersoff-type potentials, the molecular dynamics method is used to study variations of atomic interaction in initial linear deformation and postbuckling stages with various load-proportional parameters, and to determine the interactive buckling loads relationship. By comparing typical buckling modes under different loads, it is found that the boron nitride nanotube experiences complex micro-deformation processes, resulting in different variations of atomic interaction and strain energies. When the axial compressive load is relatively large, the change of atomic interaction for boron nitride nanotubes under combined loads is similar to that found under the pure axial compression. The onset of buckling leads to the abrupt releasing of strain energy. As the torsional load is relatively large, the nanotube shows torsion-like buckling behavior, no obvious reduction of strain energy is observed after the critical point. The present simulation results show that both the armchair and zigzag nanotubes exhibit nonlinear interactive buckling load relationships. Rise in temperature results in the decrease of interactive buckling load, and the effect of temperature varies with the value of load-proportional parameter. That is, the axial compressive load is relatively large, and the effect of temperature is more significant. It is found that the buckling behavior in the case of combined loading is strongly size dependent. The interactive critical axial and shear stress decrease as nanotube radius or length increases. The studies also reveal that under both simple loading and combined load condition, carbon nanotubes possess higher buckling loads than those of boron nitride nanotubes with a similar geometry, which provides valuable guidance for forming carbon and boron nitride hybrid nanotubes as well as coaxial nanotubes with superior mechanical properties.