We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.

The LAMBADA dataset: Word prediction requiring a broad discourse context / Paperno, Denis; Kruszewski Martel, German David; Lazaridou, Angeliki; Quan, Ngoc Pham; Bernardi, Raffaella; Pezzelle, Sandro; Baroni, Marco; Boleda Torrent, Gemma; Raquel, Fernández. - 3:(2016), pp. 1525-1534. (Intervento presentato al convegno ACL 2016 tenutosi a Berlin nel 7th-12th August 2016).

The LAMBADA dataset: Word prediction requiring a broad discourse context

Paperno, Denis;Kruszewski Martel, German David;Lazaridou, Angeliki;Bernardi, Raffaella;Pezzelle, Sandro;Baroni, Marco;Boleda Torrent, Gemma;
2016-01-01

Abstract

We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.
2016
The 54th Annual Meeting of the Association for Computational Linguistics Proceedings of the Conference: Vol. 1 Long Papers
Stroudsburg, PA
ACL
978-151082758-5
978-1-945626-00-5
Paperno, Denis; Kruszewski Martel, German David; Lazaridou, Angeliki; Quan, Ngoc Pham; Bernardi, Raffaella; Pezzelle, Sandro; Baroni, Marco; Boleda Torrent, Gemma; Raquel, Fernández
The LAMBADA dataset: Word prediction requiring a broad discourse context / Paperno, Denis; Kruszewski Martel, German David; Lazaridou, Angeliki; Quan, Ngoc Pham; Bernardi, Raffaella; Pezzelle, Sandro; Baroni, Marco; Boleda Torrent, Gemma; Raquel, Fernández. - 3:(2016), pp. 1525-1534. (Intervento presentato al convegno ACL 2016 tenutosi a Berlin nel 7th-12th August 2016).
File in questo prodotto:
File Dimensione Formato  
lambda.pdf

accesso aperto

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 151.65 kB
Formato Adobe PDF
151.65 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/168578
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 116
  • ???jsp.display-item.citation.isi??? 49
social impact