When an atomic ensemble is coupled with a cavity field governed by the well-known Dicke model , an important quantum phase transition (QPT) from the normal phase to the superradiant phase occurs, which was predicted more than 30 years ago. In this paper, both the atom-photon nonlinear interaction and the driving field are added to the Dicke's Hamiltonian. With the method of Holstein-Primafoff transformation, the ground state energy is given by the theoretical calculation, and the rich phase figures are presented. Meanwhile, these properties are observed experimentally. We mainly dicuss the effects of the atom-photon nonlinear interaction and the driving field on QPT.