Research Progress of Encoder–Decoder Network in Image Fusion

Image fusion is a research hotspot in the field of computer vision, which obtains complementary information from the different modal images and then presents it in a comprehensive, highly information-cohesive fused image. Encoder–decoder networks are an important technical method in the field of image fusion. This paper reviewed the research status and development of the encoder–decoder networks in the field of image fusion from the following aspects: Firstly, the 21-image fusion-related datasets and the 27 open-source codes of deep learning-based image fusion are summarized. Secondly, the model structure of encoder–decoder networks and the basic principles for image fusion based on the encoder–decoder network are described. Thirdly, encoder–decoder networks fusion models are summarized into four aspects: convolutional auto-encoder network (CAE Net), sparse auto-encoder network (SAE Net), U-shape encoder–decoder network (U-net), and variational auto-encoder network (VAE Net). In U-net, there are five types: U-net incorporating residual connection mechanism (Res U-net), U-net incorporating dense connection mechanism (Dens U-net), U-net incorporating multiscale connection mechanism (MS U-net), U-net incorporating attention mechanism (Att U-net), and U-shaped encoder–decoder networks (U-net) combined with Generative Adversarial Network (GAN U-net). Fourthly, the existing image fusion evaluation metrics are summarized from both subjective and objective aspects. Fifthly, the application of encoder–decoder network in five kinds of image fusion, such as multi-exposure image fusion, multi-focus image fusion, infrared and visible image fusion, medical image fusion, and remote sensing image fusion, are summarized. Finally, there is a discussion of the main challenges faced by encoder–decoder networks in the image fusion domain and an outlook on future directions. This paper systematically analyzed the application of encoder–decoder networks in the image fusion areas, which has positive implications for the progress of image fusion technology.