In this paper, we present a smoothed particle hydrodynamics (SPH) method for modeling multiphase flows. The multiphase SPH method includes a corrective discretization scheme for density approximation around the fluid interface to treat large density ratio, a small repulsive force between particles from different phases to prevent particles from unphysically penetrating fluid interface, and a newly-developed hyperbolic-shaped kernel function to remove possible stress instability. This multiphase SPH method is then used to study the single-and multi-mode Rayleigh-Taylor instability problems. A comparison between our results with the results from existing literature shows that our results are obviously better than most available results from other SPH simulations. The present results are close to those by Grenier et al. while the present multiphase SPH method is simpler and easier to implement than that in the work by Grenier et al. (Grenier, et al. 2009 J. Comput. Phys. 228 8380). For the single-mode Rayleigh-Taylor instability, the evolutions of the interface pattern and vortex structures, and the penetration depth each as a function of time are investigated. For the multi-mode Rayleigh-Taylor instability, the merging of small structures into a large structure during the evolution of the interface is studied. The horizontal average density and the penetration each as a function of height are also studied.