A novel frequency selective surface(FSS) with stable performance is designed and applied to microstrip antenna. The FSS cell is composed of two metallic layers separated by a dielectric spacer. The top and bottom layers connected with metallic cylinders form the fractal dendritic model. By optimizing the geometric parameters of the model, an ultra-thin FSS spatial filter is obtained, whose thickness is only 0.017λ, with wideband, polarization insensitive, wide-incident angle, and miniaturization properties. When applying the radome to microstrip antenna, the bandwidth of the antenna is improved to 40%, the gain is enhanced in the whole operating frequency band, especially at 9.6 GHz, the gain is increased by 6.7 dB; at the same time, the in-band radar cross section(RCS) is reduced significantly, and the largest reduction exceeds 12.7 dB. Experimental results are in good agreement with the simulated ones, which verifies that the novel FSS spatial filter can be used to increase the gain of broadband antenna, enhance the directivity, improve the bandwidth, and reduce the in-band RCS; to sum up, it can be applied to broadband antennas to improve their radiation as well as scattering performance at one time.