Rethinking CNN-Based Pansharpening: Guided Colorization of Panchromatic Images via GANs

Ozcelik F., Algancı U., Sertel E., Ünal G.

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, vol.59, no.4, pp.3486-3501, 2021 (SCI-Expanded) identifier identifier identifier

  • Publication Type: Article / Article
  • Volume: 59 Issue: 4
  • Publication Date: 2021
  • Doi Number: 10.1109/tgrs.2020.3010441
  • Journal Name: IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Academic Search Premier, PASCAL, Aerospace Database, Applied Science & Technology Source, Aquatic Science & Fisheries Abstracts (ASFA), Business Source Elite, Business Source Premier, CAB Abstracts, Communication Abstracts, Compendex, Computer & Applied Sciences, Geobase, INSPEC, Metadex, Pollution Abstracts, Civil Engineering Abstracts
  • Page Numbers: pp.3486-3501
  • Keywords: Task analysis, Spatial resolution, Training, Standards, Sensors, Multiresolution analysis, AI, colorization, convolutional neural networks (CNNs), deep learning, generative adversarial networks (GANs), image fusion, PanColorization generative adversarial network (PanColorGAN), pansharpening, self-supervised learning, super-resolution (SR), FUSION, QUALITY, REGRESSION, MS
  • Istanbul Technical University Affiliated: Yes

Abstract

Convolutional neural network (CNN)-based approaches have shown promising results in the pansharpening of the satellite images in recent years. However, they still exhibit limitations in producing high-quality pansharpening outputs. To that end, we propose a new self-supervised learning framework, where we treat pansharpening as a colorization problem, which brings an entirely novel perspective and solution to the problem compared with the existing methods that base their solution solely on producing a super-resolution version of the multispectral image. Whereas the CNN-based methods provide a reduced-resolution panchromatic image as the input to their model along with the reduced-resolution multispectral images and, hence, learn to increase their resolution together, we instead provide the grayscale transformed multispectral image as the input and train our model to learn the colorization of the grayscale input. We further address the fixed downscale ratio assumption during training, which does not generalize well to the full-resolution scenario. We introduce a noise injection into the training by randomly varying the downsampling ratios. Those two critical changes, along with the addition of adversarial training in the proposed PanColorization generative adversarial network (PanColorGAN) framework, help overcome the spatial-detail loss and blur problems that are observed in CNN-based pansharpening. The proposed approach outperforms the previous CNN-based and traditional methods, as demonstrated in our experiments.