Full polarization imaging can obtain more information about target, which has a broad application prospect in the target detection, researches of atmospheric characteristics, and medical diagnosis. This paper develops an imaging system of polarized skylight pattern of full polarization for obtaining the information about full polarization rapidly. Meanwhile, aiming at the problem that the error of the light intensity image obtained by the system due to the different “behavior” of the system transmission matrix is brought into the solution of the target Stokes vector, this paper analyzes the condition number and determinant of the system transmission matrix. Firstly, an objective function is established by combining the three sets of condition numbers and the determinant. Therefore, the problem of solving the optimal transmission matrix is transformed into a multi-condition extremal problem. And then the objective function is minimized to determine the optimal angle of the transmission matrix when the 1 norm condition number, 2 norm condition number and ∞ norm condition number reach the minimum value and the determinant reaches the maximum value. In addition, in order to improve the measurement accuracy, the delay components of quarter wave plate, extinction ratio of polarizer, and the transmission matrix of the system are calibrated. Optimization contrast experiment and outfield experiment are performed. The entropy, mean, and standard deviation are used to quantify the optimized results of the angle of polarization, degree of polarization, and degree of linear polarization. ∆Aop is defined as the difference in absolute value of angle of polarization between the two sides of the symmetry axis to verify the optimization performance of angle of polarization. Experimental results show that the polarization angle error after optimization is reduced by more than 10% compared with that before optimization; the error of the band of maximum polarization and the error of the neutral zone in the degree of polarization and linear polarization also decline to different degrees compared with before optimization. On this basis, an experiment on measuring external field full polarization information is carried out. The results show that the system meets the design requirements and can effectively obtain the sky full polarization information.