Optical cavity is a fundamental device of modern optics and has a wide range of applications in the fields of laser generation, nonlinear optical conversion, and optical sensors. A major aspect of the properties of optical cavity is the stability analysis. According to different geometric losses, these optical cavities can be divided into three types: stable cavity, critical cavity, and unstable cavity. The determination of the stability of the optical cavity is the basic problem of a classic system, but the research and analysis of this point have been much insufficient in the past. In this paper, by extending the definition domain of the inverse trigonometric function, the propagation matrices of the symmetric confocal cavity and the asymmetric confocal cavity are solved. The sudden change of stability with the change of geometric parameters is explained by algebraic analysis and optical ray topology.The mathematical analysis shows that the stability catastrophe of confocal cavity is due to the sudden change in the value of inverse cosine function at the critical point of the traditional domain of definition. From the perspective of geometric topology, we define the topological charge of the cavities according to the geometric propagation path of light in the cavity. Only the cavities with zero topological charge are found to be stable, and the change of topological charge is quantized, which explains the sudden change of confocal cavity stability. Finally, we build a coupled stable cavity consisting of two unstable cavities with the same parameters. The quality factors of the coupled stable cavity and the unstable cavity are analyzed by the finite difference time domain method, which further verifies the origin of the sudden change in the stability of the confocal cavity. We propose that the coupled unstable dual cavities with opposite topological charges are able to be stable, and we also find that there are new modes in the coupled cavities which are not found in the corresponding single cavity. These findings suggest a new method for controlling microcavity loss, which has a certain value for studying the new micro-nano lasers, on-chip nonlinear devices, and non-Hermitian optical sensors.