A new three-dimensional quasi-vectorial beam propagation method based on the series expansion (SE-QV-BPM) is proposed for simulating the optical rib waveguide and directional coupler based on InGaAs/InAlAs multiple quantum wells. For the same etched-depth, the confinement of TM mode in horizontal direction is shown to be stronger than that of TE mode, and the singular behaviors for TM modes at the corners are clearly observed. For the same gap width, the coupling length of TM mode for directional coupler is larger than that of TE mode. The effective indices and modal profiles of the quasi-vectorial fundamental TE and TM modes for shallow- and deep-etched rib waveguides and directional couplers are obtained. Besides, the evolution of the injected field with propagation distance through the directional coupler is demonstrated. The calculation has good efficiency due to the small size of the resulting matrix, and the rib structures can be conveniently dealt with since the sinusoidal functions are selected as the basis functions. Moreover, the variable transformation is introduced to avoid the boundary truncation.