The granular system has complicated force chain network and multiple relaxation mechanisms. The different relaxation mechanisms have largely effects on others. The force chains divide the whole system into many soft zones which dominate the main dissipation process. The system evolves into lower potential energy state gradually and forms directional arrangement under an external load. During the evolution, the complex relaxation behaviors such as transport and migrant processes, make it difficult to distinguish different dissipated mechanisms. Each single physical mechanism stripping from multiple mechanisms should be studied in depth. While among all the mechanisms, the structure evolution plays a crucial role and needs to be paid more attention to. From the view of potential energy, the detailed energy transformation is illustrated. The granular system is often at a metastable state. When the external disturbance is large enough, the system would step over the energy barrier to a new state. The height of energy barrier is related to the packing structure and grain property. In energy landscape, there exist many energy valleys which correspond to different metastable states. The grain rearrangement and structure reorganization are two main evolution processes at a quasi-static state. The former brings about major potential energy change because of friction and forms certain contact relations. While the latter evolves on the basis of the skeleton formed by grain rearrangement and reaches lower energy state. The conversion among different energy valleys can be used to explain stress relaxation process. In a complex granular system, the choosing of appropriate internal state variables becomes important, which can reflect specific relaxation process and internal characteristics. The energy fluctuation in the system has a huge influence on dissipation process and macroscopic response and is an effective internal variable to have an insight into the structure evolution. Then granular temperature rooted from gas kinetics is introduced to model the macroscopic behaviors. For loose and rapid granular flow, the kinetic granular temperature itself is the root to affect the flow process. While in a dense granular system, the granular temperature at a quasi-static state is referred to as elastic energy fluctuation. The structure can be kept stable when granular temperature exists on account of the mutual confinement among particles. And the granular temperature at a stable state is just a representation of internal structure of granular assembly. When the granular temperature stimulated by the external disturbance exceeds the stable value, the irreversible process happens and the difference between the excited state and stationary state is the driving force for evolution. The stress relaxations under different surface properties and confining pressures are simulated using non-equilibrium theory with new change for granular temperature. It can be found that the granular temperature difference triggers elastic relaxation and force chains reorganization. And the larger the temperature difference, the further away from the steady state the system is and the larger the stress change is. The more smooth the surface and the smaller the confining pressure, the lower resistance is generated, so that the initial granular temperature difference is larger and the stress change is larger during stress relaxation. The granular temperature decreases as time goes by because of its own relaxation. When the difference is equal to zero, the process of stress relaxation finishes and the system evolves into a global minimum of potential energy.