This paper presents the design and results of a crowdsourcing experiment on the recognition of Italian event nominals. The aim of the experiment was to assess the feasibility of crowdsourcing methods for a complex semantic task such as distinguishing the eventive interpretation of polysemous nominals taking into consideration various types of syntagmatic cues. Details on the theoretical background and on the experiment set up are provided together with the final results in terms of accuracy and inter-annotator agreement. These results are compared with the ones obtained by expert annotators on the same task. The low values in accuracy and Fleiss’ kappa of the crowdsourcing experiment demonstrate that crowdsourcing is not always optimal for complex linguistic tasks. On the other hand, the use of non-expert contributors allows to understand what are the most ambiguous patterns of polysemy and the most useful syntagmatic cues to be used to identify the eventive reading of nominals.

Crowdsourcing for the identification of event nominals: an experiment / Sprugnoli, Rachele; Lenci, Alessandro. - (2014). (Intervento presentato al convegno LREC 2014 tenutosi a Reykjavik, Iceland nel 26-31 May 2014).

Crowdsourcing for the identification of event nominals: an experiment

Sprugnoli, Rachele;
2014-01-01

Abstract

This paper presents the design and results of a crowdsourcing experiment on the recognition of Italian event nominals. The aim of the experiment was to assess the feasibility of crowdsourcing methods for a complex semantic task such as distinguishing the eventive interpretation of polysemous nominals taking into consideration various types of syntagmatic cues. Details on the theoretical background and on the experiment set up are provided together with the final results in terms of accuracy and inter-annotator agreement. These results are compared with the ones obtained by expert annotators on the same task. The low values in accuracy and Fleiss’ kappa of the crowdsourcing experiment demonstrate that crowdsourcing is not always optimal for complex linguistic tasks. On the other hand, the use of non-expert contributors allows to understand what are the most ambiguous patterns of polysemy and the most useful syntagmatic cues to be used to identify the eventive reading of nominals.
2014
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Reykjavik
European Language Resources Association (ELRA)
Sprugnoli, Rachele; Lenci, Alessandro
Crowdsourcing for the identification of event nominals: an experiment / Sprugnoli, Rachele; Lenci, Alessandro. - (2014). (Intervento presentato al convegno LREC 2014 tenutosi a Reykjavik, Iceland nel 26-31 May 2014).
File in questo prodotto:
File Dimensione Formato  
1157_Paper.pdf

Solo gestori archivio

Descrizione: Paper
Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 454.92 kB
Formato Adobe PDF
454.92 kB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/164432
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact