Remote sensing change captioning meets large language and vision models

Change captioning (CC) aims at automatically recognizing and describing in natural language the changes between two or more images of the same location acquired at different times. Automatically translating visual changes into natural language format can benefit several practical applications such as urban planning, disaster management, environmental monitoring, and surveillance. Information in natural language is inherently more descriptive, enabling the analysis of relations between the changes, the various classes of changes, and the attributes of the objects subject to the change. Usually, change captioning is tackled by collecting large amounts of examples to train a neural network to solve the task, which is expensive and time-consuming. This paper evaluates task-agnostic large language models (LLMs) and vision-language models (VLMs) in performing zero-shot change captioning. We design different approaches to leverage the existing pre-trained capabilities of those models, covering a broad range of use cases, discussing the respective strengths and drawbacks. Specifically, we design three approaches leveraging the capabilities of Otter, a large vision-language model, and Vicuna, a large language model. We integrate the SECOND dataset, used in our experiments, with synthetic change descriptions obtained using GPT-3.5 to foster research in this promising area. Finally, we propose, analyze, and discuss a novel evaluation criterion, FMScore, that uses a set of ground truth “facts” and a large language model to evaluate the coherence of a generated description using fact matching. Experiments demonstrate that while our methods do not yield outstanding results on conventional metrics, they achieve competitive results with the leading GPT-4 model when evaluated using FMScore. We release our enriched SECOND dataset and all the code to reproduce the results at this link.