Change detection (CD) is among the most important applications in remote sensing (RS) that allows identifying the changes that occurred in a given geographical area across different times. Even though CD systems have seen a lot of progress in RS, their output is either a binary map highlighting the changing area or a semantic change map that indicates the type of change for each pixel. The change maps are often difficult to interpret by end users, and they omit important information such as relationships and attributes of the changed areas. Motivated by the recent advancement of image captioning in the RS community, in this article, we propose to describe the changes over bitemporal images through change sentence descriptions. The aim of this article is to provide a user-friendly interpretation of the occurred changes. To this end, we propose two change captioning (CC) systems that take bitemporal images as input and generate coherent sentence descriptions of the occurred changes. Convolutional neural networks (CNNs) are used to extract discriminative features from the bitemporal images and recurrent neural networks (RNNs) or support vector machines (SVMs) are exploited to generate coherent change descriptions. Furthermore, in the absence of a CC dataset to test our systems, we propose two new datasets. One is based on very high-resolution RGB images, and the other one is based on multispectral RS images. The obtained experimental results show promising capabilities of the proposed systems to generate coherent change descriptions from the bitemporal images. The datasets are available at the following link: https://disi.unitn.it/melgani/datasets.html.

Change Captioning: A New Paradigm for Multitemporal Remote Sensing Image Analysis / Hoxha, G.; Chouaf, S.; Melgani, F.; Smara, Y.. - In: IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING. - ISSN 0196-2892. - ELETTRONICO. - 60:(2022), pp. 562741401-562741414. [10.1109/TGRS.2022.3195692]

Change Captioning: A New Paradigm for Multitemporal Remote Sensing Image Analysis

Hoxha G.;Melgani F.;
2022-01-01

Abstract

Change detection (CD) is among the most important applications in remote sensing (RS) that allows identifying the changes that occurred in a given geographical area across different times. Even though CD systems have seen a lot of progress in RS, their output is either a binary map highlighting the changing area or a semantic change map that indicates the type of change for each pixel. The change maps are often difficult to interpret by end users, and they omit important information such as relationships and attributes of the changed areas. Motivated by the recent advancement of image captioning in the RS community, in this article, we propose to describe the changes over bitemporal images through change sentence descriptions. The aim of this article is to provide a user-friendly interpretation of the occurred changes. To this end, we propose two change captioning (CC) systems that take bitemporal images as input and generate coherent sentence descriptions of the occurred changes. Convolutional neural networks (CNNs) are used to extract discriminative features from the bitemporal images and recurrent neural networks (RNNs) or support vector machines (SVMs) are exploited to generate coherent change descriptions. Furthermore, in the absence of a CC dataset to test our systems, we propose two new datasets. One is based on very high-resolution RGB images, and the other one is based on multispectral RS images. The obtained experimental results show promising capabilities of the proposed systems to generate coherent change descriptions from the bitemporal images. The datasets are available at the following link: https://disi.unitn.it/melgani/datasets.html.
2022
Hoxha, G.; Chouaf, S.; Melgani, F.; Smara, Y.
Change Captioning: A New Paradigm for Multitemporal Remote Sensing Image Analysis / Hoxha, G.; Chouaf, S.; Melgani, F.; Smara, Y.. - In: IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING. - ISSN 0196-2892. - ELETTRONICO. - 60:(2022), pp. 562741401-562741414. [10.1109/TGRS.2022.3195692]
File in questo prodotto:
File Dimensione Formato  
2022_TGRS-Change Captioning.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 3.01 MB
Formato Adobe PDF
3.01 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/373007
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 19
  • ???jsp.display-item.citation.isi??? 9
social impact