In the ray-optics (RO) model of optical tweezers, tracing refractive and reflected rays with vectors play important roles in calculating the trapping forces. Traditional ray-tracing method with solid geometry, to some extent, is complicated in determining the orientations of those refractive and reflected rays according to spatial incident rays. It is difficult to calculate the trapping forces for irregular particles. In this paper, quaternion is proposed to rotate ray vectors for simplifying the traces of all kinds of spatial rays. Then, it is appropriate to calculate the trapping force of an ellipsoid bead. Based on the algorithm of quaternion and the convention between the interface normal and angular directions, the direction of normal always points from optically denser medium to thinner medium. The rotation axis is the cross product of the incident ray and the interface normal. And the positive angular direction can be determined by right-hand rule based on the orientation of the rotation axis. According to Snell' law, the rotation angle between the incident ray and refractive/reflected ray can be determined. The quaternion for rotation consists of rotation axis and angle. So the refractive and reflected rays are both determined by quaternions of incident ray and rotation based on rotation rules. Furthermore, the force on interface can also be calculated according to momentum changes of the photon before and after the interface refraction and reflection. The quaternion method is used to analyze the effects of coverslip position and deformation ratio on the trapping efficiency of ellipsoid particles. Our simulative results show that the lateral and axial trapping efficiencies are obviously affected by the deformation of the ellipsoid itself. No matter whether the bead deforms transversely or axially, the transverse and axial trapping efficiencies both become larger at a specific deformation. Meantime, the increase of the spherical aberration reduces the maximum axial trapping efficiency, and the equilibrium position of the bead becomes farther away from the center. Using quaternion method, the calculation of refractive lightvector can be simplified in comparison with by using the method of Euclidean geometry or transformation matrix. Theoretically, this quaternion can be used to trace rays on any irregular geometric surfaces. In conclusion, the method of quaternion can make ray tracing easier and extend the applications of RO model.