A common phenomenon of polymer solar cells with metal oxide electron-transport layers (ETLs), known as “light-soaking” issue, is that the as-prepared device exhibits an anomalous S-shaped J-V characteristic, resulting in an extremely low fill factor (FF) and thus a poor power conversion efficiency. However, the S-shape disappears upon white light illumination with UV spectral components, meanwhile the performance parameters of the device recover the normal values eventually. This behavior appears to be of general validity for various metal oxide layers regardless of the synthesis and fabricating processes. Its origin is still under debate, while the ETL interface problems have generally been claimed to be the underlying reason so far. In this paper, both conventional and inverted cells with using ZnO nanoparticles (NPs) as ETL are fabricated to clarify the interface effect of the ETL on the light soaking procedure. The inverted device shows a typical light-soaking issue with an initial FF less than 20% as expected, whereas the J-V curves of the conventional cell remain regular shapes throughout the test. This result indicates that the ITO/ZnO interface is a key reason of S-shaped J-V characteristics, which is further verified via the use of Cs2CO3/ZnO ETL. The insert of Cs2CO3 layer isolates the ITO electrode from contacting with ZnO layer, and the kink disappears in the as-prepared device with this bi-layered ETL inverted structure. Our explanation for the result above is that the oxygen impurities absorbed onto the surface of ZnO NPs during fabrication process, behave as strong electron traps, and thus increasing the width of the energy barrier (EB) at the interface of ITO/ZnO. Subsequently, photogenerated electrons accumulate in the ZnO layer adjacent to the interface, resulting in extremely poor performance. Upon white light illumination, however, the trap sites are filled by photogenerated carriers within the ZnO layer, and therefore narrowing the EB. As the barrier width becomes thin enough to be freely tunneled through, a good selectivity behavior of ZnO ETL is reached, leading to a fully remarkable recovery in device performances.