Particle segregation processes in different vibration modes are simulated based on 3D discrete element method (DEM). The phenomena of wave motion of large particles in upper layer that appears in the segregation processes of linear vibration mode, accumulation and circulation that appear in circular and elliptical vibration mode are analyzed. And the influences of vibration intensity on the segregation pattern of the circular and elliptical mode were also discussed. The results show that the segregation behaviors in different modes can be well explained by a combination of three mechanisms: void filling, sidewall-driven transport of particles and nonequipartition of energy, and the distribution of particle velocity vectors. The vibration intensity has a great effect on the segregation pattern of circular and elliptical mode. For each vibration mode, fine particle segregation effect and stable particle moving state is obtained when the value of vibration intensity is about 3.