Plasmonic color filters comprised of metal films can be adjusted easily. Periodic arrays of subwavelength size holes with circle, square and rectangular shape are fabricated on 200 nm thick Au films by using focused ion beam. The structures are utilized as plasmonic color filters for exhibiting color variation under the illumination of white light in different polarization directions. We find that for rectangular hole arrays, the color of the transmitted light changes greatly with polarization direction. However, for arrays of circle and square holes, color of the transmitted light is not sensitive to the polarization direction of incident light. We conclude that localized surface plasmon around a single hole is a key factor in influencing the color of transmission light, rather than surface plasmon polaritons induced by periodic arrays. If incident light does not excite the localized surface plasmon around the holes, the effect of surface plasmon polariton will disappear, Based on different display colors of transmitted light of the subwavelength holes with polarization, a composite color display sample is prepared, which will exhibit different color patterns as the polarization direction of incident light changes.