n this paper, we define and evaluate a methodology for extracting history-dependent spatial questions from visual dialogues. We say that a question is history-dependent if it requires (parts of) its dialogue history to be interpreted. We argue that some kinds of visual questions define a context upon which a follow-up spatial question relies. We call the question that restricts the context: trigger, and we call the spatial question that requires the trigger question to be answered: zoomer. We automatically extract different trigger and zoomer pairs based on the visual property that the questions rely on (e.g. color, number). We manually annotate the automatically extracted trigger and zoomer pairs to verify which zoomers require their trigger. We implement a simple baseline architecture based on a SOTA multimodal encoder. Our results reveal that there is much room for improvement for answering history-dependent questions.

Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History / Dong, Tianai; Testoni, Alberto; Benotti, Luciana; Bernardi, Raffaella. - ELETTRONICO. - (2021), pp. 22-31. (Intervento presentato al convegno Splurobonlp tenutosi a online nel 5-6 August 2021) [10.18653/v1/2021.splurobonlp-1.3].

Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History

Testoni, Alberto;Bernardi, Raffaella
2021-01-01

Abstract

n this paper, we define and evaluate a methodology for extracting history-dependent spatial questions from visual dialogues. We say that a question is history-dependent if it requires (parts of) its dialogue history to be interpreted. We argue that some kinds of visual questions define a context upon which a follow-up spatial question relies. We call the question that restricts the context: trigger, and we call the spatial question that requires the trigger question to be answered: zoomer. We automatically extract different trigger and zoomer pairs based on the visual property that the questions rely on (e.g. color, number). We manually annotate the automatically extracted trigger and zoomer pairs to verify which zoomers require their trigger. We implement a simple baseline architecture based on a SOTA multimodal encoder. Our results reveal that there is much room for improvement for answering history-dependent questions.
2021
Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics
Online
ACL
978-1-954085-78-7
Dong, Tianai; Testoni, Alberto; Benotti, Luciana; Bernardi, Raffaella
Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History / Dong, Tianai; Testoni, Alberto; Benotti, Luciana; Bernardi, Raffaella. - ELETTRONICO. - (2021), pp. 22-31. (Intervento presentato al convegno Splurobonlp tenutosi a online nel 5-6 August 2021) [10.18653/v1/2021.splurobonlp-1.3].
File in questo prodotto:
File Dimensione Formato  
2021.splurobonlp-1.3.pdf

accesso aperto

Descrizione: articolo principale
Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 5.7 MB
Formato Adobe PDF
5.7 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/328644
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? 1
  • OpenAlex ND
social impact