Shear test samples of different grain sizes are prepared by using mineral particles of soil, and a series of tests of quick direct shear and tri-axial shear are performed to study the size effect of granular media. Deformation curves and shear stress strength are given of test samples with particles of different size and volume fraction. On the basis of the ratio of micro-acting forces between particles to gravity and the cell element model, physical mechanism of grain size effect is, for the first time as far as we know, explained on the micro-level and mecro-level respectively. Test results show that the deformation characteristic of granular media is enhanced and its shear stress strength increases with increasing volume fraction and decreasing of particle size, and the effect of volume fraction on the deformation characteristics and strength is more notable than that of grain size. According to mechanism analysis on size effect, parameter ratio of micro-acting forces to gravity is suggested to assess aggregation and friction effects of particles in the media, and mecro-mechanism is interpreted as strain gradient and micro-cracks of deformation coordination leading to grain size effect. The cell element model presented in this paper can greatly reduce the degrees of freedom of granular media and provides an available way for calculation modeling in industry and engineering design.