Non-equilibrium quantum many-body systems have attracted considerable attention in the past decades. The scope of the research of this kind of novel system involves interdisciplinary research of condensed matter, atomic and molecular physics, quantum optics, quantum information and quantum computation, as well as the non-equilibrium statistical physics. The non-equilibrium phenomena emerging from the aforementioned quantum systems can exhibit rich and universal behaviors, which have far from being well understood due to the novelties and complexities of these systems, and hence the quantum many-body physics becomes the research highlight. At the same time, with the rapid development of quantum techniques, the understanding of these complex systems is of important practical significance due to their potential applications in quantum computation and quantum manipulation. In this paper, we show our recent progress of non-equilibrium quantum many-body systems. We focus on the novel phenomena closely related to the temporary symmetry breaking, including the exotic quantum matter, quasi-particles as well as the dynamical universality classes in non-equilibrium quantum many-body systems.