Motivated by the square-octagon lattice which supports topological phases over a wide range of parameters and a number of interesting quantum phase transitions in the phase diagram when considering the intrinsic spin-orbit coupling, we investigate the topological phase transitions in the isotropic square-octagon lattice combining the effects of both spin-orbit couplings and exchange field. The inversion symmetry and time-reversal symmetry are broken when both Rashba spin-orbit coupling and exchange field are present. The Z ₂index is not applicable for quantum spin Hall systems without time-reversal symmetry, but the spin Chern number remains valid even in the absence of time-reversal symmetry. Therefore, we use the Chern number and spin Chern number to describe the topological properties of the system. We explore that a variety of topologically nontrivial states appear with changing the exchange field, including time-reversal-symmetry-broken quantum spin Hall states and quantum anomalous Hall states. The phase transition between these topological phases is accompanied by the closing of band gaps. Interestingly, the quantum spin Hall effect described by nonzero spin Chern number is found to remain intact when the time-reversal symmetry is broken. Furthermore, the variation of the amplitude of the exchange field and filling factor drive interesting topological phase transitions from the time-reversal-symmetry-broken quantum spin Hall phase to spin-filtered quantum anomalous Hall phase. A spin-filtered quantum anomalous Hall phase is characterized by the presence of edge states with only one spin component, which provides an interesting route towards quantum spin manipulation. We also present the band structures, edge state wave functions, and spin polarizations of the different topological phases in the system. It is demonstrated that the energy spectra of edge states are in good agreement with the topological characterization based on the Chern number and spin Chern number. In particular, we observe that gapless edge states can appear in a time-reversal-symmetry-broken quantum spin Hall system, but the corresponding spin spectrum gap remains open on the edges. Recently, an important functional material ZnO with quasi square-octagon lattice has been found experimentally. Consequently, the results found in our work are helpful for understanding the property of square-octagon lattice and studying the real materials with square-octagon structure.