In the infrared and visible light image fusion, the noise interference always exists. There is also the disadvantage that image fusion is easy to produce artifacts which cause blurred edge and low contrast. In order to solve these problems, in this study we propose an image fusion method based on deep model segmentation. First of all, deep Bolzmann machine is adopted to learn prior target and background contour and construct a contour deep segmentation model. After the optimal energy segmentation, Split Bregman iteration is used to obtain the infrared and visible image contour. Then non-subsampled contourlet transform is adopted to decompose the source images. The segmented background contour coefficients are fused by the structure similarity rule. Finally, the fused image is reconstructed by the non-subsampled contourlet inverse transform. The experimental results show that this algorithm can effectively obtain fused images with clear target contour and background contour. The fused images also have high contrast and low noise. The results show that it is an effective method of achieving the infrared and visible image fusion.