We report the results of the SemEval 2022 Task 3, PreTENS, on evaluation the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation. The task featured two sub-tasks articulated as: (i) binary prediction task and (ii) regression task, predicting the acceptability in a continuous scale. The sentences were artificially generated in three languages (English, Italian and French). 21 systems, with 8 system papers were submitted for the task, all based on various types of fine-tuned transformer systems, often with ensemble methods and various data augmentation techniques. The best systems reached an F1-macro score of 94.49 (sub-task1) and a Spearman correlation coefficient of 0.80 (sub-task2), with interesting variations in specific constructions and/or languages.

SemEval-2022 Task 3: PreTENS – Evaluating Neural Networks on Presuppositional Semantic Knowledge / Zamparelli, Roberto; Chowdhury, Shammur; Brunato, Dominique; Chesi, Cristiano; Dell’Orletta, Felice; Hasan, Md. Arid; Venturi, Giulia. - ELETTRONICO. - (2022), pp. 228-238. (Intervento presentato al convegno SemEval-2022 tenutosi a Seattle, United States nel 10-15 luglio 2022) [10.18653/v1/2022.semeval-1.29].

SemEval-2022 Task 3: PreTENS – Evaluating Neural Networks on Presuppositional Semantic Knowledge

Zamparelli, Roberto;Chowdhury, Shammur;
2022-01-01

Abstract

We report the results of the SemEval 2022 Task 3, PreTENS, on evaluation the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation. The task featured two sub-tasks articulated as: (i) binary prediction task and (ii) regression task, predicting the acceptability in a continuous scale. The sentences were artificially generated in three languages (English, Italian and French). 21 systems, with 8 system papers were submitted for the task, all based on various types of fine-tuned transformer systems, often with ensemble methods and various data augmentation techniques. The best systems reached an F1-macro score of 94.49 (sub-task1) and a Spearman correlation coefficient of 0.80 (sub-task2), with interesting variations in specific constructions and/or languages.
2022
Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)
Seattle, United States
Association for Computational Linguistics
Zamparelli, Roberto; Chowdhury, Shammur; Brunato, Dominique; Chesi, Cristiano; Dell’Orletta, Felice; Hasan, Md. Arid; Venturi, Giulia
SemEval-2022 Task 3: PreTENS – Evaluating Neural Networks on Presuppositional Semantic Knowledge / Zamparelli, Roberto; Chowdhury, Shammur; Brunato, Dominique; Chesi, Cristiano; Dell’Orletta, Felice; Hasan, Md. Arid; Venturi, Giulia. - ELETTRONICO. - (2022), pp. 228-238. (Intervento presentato al convegno SemEval-2022 tenutosi a Seattle, United States nel 10-15 luglio 2022) [10.18653/v1/2022.semeval-1.29].
File in questo prodotto:
File Dimensione Formato  
SemEval_2022_PreTENS_final_report.pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Creative commons
Dimensione 356.03 kB
Formato Adobe PDF
356.03 kB Adobe PDF Visualizza/Apri
2022.semeval-1.29.pdf

accesso aperto

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 360.78 kB
Formato Adobe PDF
360.78 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/353301
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact