In this paper, an asymmetric transmission scheme is proposed by cascading an acousto-optic device and a coupled whispering gallery mode (WGM) microsphere cavity, and it is demonstrated theoretically and experimentally. With the acousto-optic interaction in a fiber, the vector modes of the fundamental core mode can be converted into the different vector modes of a cladding (linear polarization, LP) mode, and because of the optical path difference between the cladding vector modes, the polarization of the cladding mode will be changed. The cladding mode can be converted back into the core fundamental mode by coupling a WGM microcavity. By calculating the overlapping of the mode fields in the tapered fiber and the microcavity at the resonance wavelength, the coupling coefficients between different LP modes and WGM will be solved. And, the transmitivities and conversion coefficients of the two fiber modes can be obtained on condition that the polarization of the incident light does not coincide with the polarization orientation of the WGM. The transmission spectra of the coupled WGM microcavity are calculated by using Matlab program for eight states, including the states at different incident directions, different incident polarizations of input, whether the acoustic wave is on or off. The results show that the conversion coefficient from the cladding mode to the core mode is completely different from that of the contrary process when the acoustic wave is working. And the forward incident light and backward incident light have completely different transmission characteristics, thus resulting in the asymmetric transmission. The transmittances of forward incidence and reverse incidence at different polarizations are also studied, both of them change periodically with the polarization angle, and their phase difference is equal to the polarization change caused by acousto-optic interaction in the fiber. In the experiment, a two-stage tapered fiber is used to realize the acousto-optic interaction and the coupling of whispering gallery mode at the same time. By controlling the working states of the system, the same 8 states as in the calculation are studied experimentally. The results show that due to the polarization-selection effect of the WGM, the light energy incident from the opposite directions will show different transmission characteristics. While the forward transmittance reaches a maximum value of about 0.505, the reverse transmittance reaches a minimum value of about 0.010, and the transmission isolation reaches about 17 dB. The transmittances in two directions are measured at different incident polarization angles, the transmission isolation is analyzed, and the polarization change of cladding mode in the fiber is verified to be about ${80^ \circ }$ . The measured results coincide with the calculations from the developed theory well. Finally, the shortcomings and optimization method of the scheme are discussed. The asymmetric transmission scheme in this paper inherits the advantages of rapid response and good tuning of acousto-optic device, and has an all-fiber structure, which has important application potential in optical switch and isolator.