Based on the phase-conjugation polarization interference between two two-photon processes, we theoretically investigated the attosecond scale asymmetry sum-frequency polarization beat in four-level system (FASPB). The field correlation has weak influence on the FASPB signal when the laser has narrow bandwidth. Conversely, when the laser has broadband linewidth, the FASPB signal shows resonance-nonresonance cross correlation. The two-photon signal exhibits hybrid radiation-matter detuning terahertz damping oscillation, i. e., when the laser frequency is off resonance from the two-photon transition, the signal exhibits damping oscillation and the profile of the two-photon self-correlation signal also exhibits zero time-delay asymmetry of the maxima. We have also investigated the asymmetry of attosecond polarization beat caused by the shift of the two-photon self-correlation zero time-delay phenomenon, in which the maxima of the two two-photon signals are shifted from zero time-delay point to opposite directions. As an attosecond ultrafast modulation process，FASPB can be intrinsically extended to any level-summation systems of two dipolar forbidden excited states.