Instability of the continuous-wave states in a dual-core optical fiber system is investigated based on the complex Ginzburg-Landau equation. The dual-core optical fiber system consists of an active nonlinear dispersive core and a linear passive core. The modulation instability (MI) conditions are found from linearized equations for small perturbations. Simulations of the full system demonstrate that the development of the MI in the former regime leads to the establishment of a regular or chaotic array of pulses if the MI saturates, or a chain of well-separated peaks with continuously growing amplitudes if the instability does not saturate. It indicates that the peak value of multiple return-to-zero (RZ) pulses or a single RZ pulse will be amplified and the RZ pulse sources emerge in the optical fiber in the anomalous group velocity dispersion regime. This research can be used as the resource of the optical fiber telecommunications and will be useful for the study of the fibre optics and physics and also in some other fields.