When training a model on referential dialogue guessing games, the best model is usually chosen based on its task success. We show that in the popular end-to-end approach, this choice prevents the model from learning to generate linguistically richer dialogues, since the acquisition of language proficiency takes longer than learning the guessing task. By comparing models playing different games (GuessWhat, GuessWhich, and Mutual Friends), we show that this discrepancy is model- and task-agnostic. We investigate whether and when better language quality could lead to higher task success. We show that in GuessWhat, models could increase their accuracy if they learn to ground, encode, and decode also words that do not occur frequently in the training set.

The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues / Testoni, Alberto; Bernardi, Raffaella. - ELETTRONICO. - (2021), pp. 2071-2082. (Intervento presentato al convegno 16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021 tenutosi a Online nel 19-23 April 2021) [10.18653/v1/2021.eacl-main.178].

The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues

Testoni, Alberto;Bernardi, Raffaella
2021-01-01

Abstract

When training a model on referential dialogue guessing games, the best model is usually chosen based on its task success. We show that in the popular end-to-end approach, this choice prevents the model from learning to generate linguistically richer dialogues, since the acquisition of language proficiency takes longer than learning the guessing task. By comparing models playing different games (GuessWhat, GuessWhich, and Mutual Friends), we show that this discrepancy is model- and task-agnostic. We investigate whether and when better language quality could lead to higher task success. We show that in GuessWhat, models could increase their accuracy if they learn to ground, encode, and decode also words that do not occur frequently in the training set.
2021
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics
209 N EIGHTH STREET, STROUDSBURG, PA 18360 USA
ACL
9781954085022
Testoni, Alberto; Bernardi, Raffaella
The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues / Testoni, Alberto; Bernardi, Raffaella. - ELETTRONICO. - (2021), pp. 2071-2082. (Intervento presentato al convegno 16th Conference of the European Chapter of the Associationfor Computational Linguistics, EACL 2021 tenutosi a Online nel 19-23 April 2021) [10.18653/v1/2021.eacl-main.178].
File in questo prodotto:
File Dimensione Formato  
2021.eacl-main.178.pdf

accesso aperto

Descrizione: articolo principale
Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 2.29 MB
Formato Adobe PDF
2.29 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/328630
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 4
  • ???jsp.display-item.citation.isi??? 3
social impact