The accurate computation of effective refractive index of optical fiber core and the cladding mode are needed for the fiber structure design, mode dispersion analysis and fiber grating mode coupling research. In this paper, based on three layered optical fiber waveguide model and its accurate eigenvalue equation, by using the cut string algorithm the effective refractive index of core mode is solved. At the same time the computation results are verified by using COMSOL software. By using interval traversal algorithm the effective refractive index of cladding mode is computed. Compared with the existing algorithms, this method can avoid the loss of root during the calculation. It can also avoid the generation of odd point and can keep the normality of each mode. The simulation is based on Mathematica software. The curves relating effective refractive index of core and cladding mode to wavelength are obtained.