In this paper, a strict and general instability criterion is obtained by analysing the geometric properties of the dispersion relation of electromagnetic waves in plasma with anisotropic velocity distribution. The criterion is applied to the cases of anisotropic temperature and current-carrying plasma. Finally, the relativistic correction of Weibel's formula is obtained.