One expensive step when defining crowd- sourcing tasks is to define the examples and control questions for instructing the crowd workers. In this paper, we intro- duce a self-training strategy for crowd- sourcing. The main idea is to use an au- tomatic classifier, trained on weakly su- pervised data, to select examples associ- ated with high confidence. These are used by our automatic agent to explain the task to crowd workers with a question answer- ing approach. We compared our relation extraction system trained with data anno- tated (i) with distant supervision and (ii) by workers instructed with our approach. The analysis shows that our method rela- tively improves the relation extraction sys- tem by about 11% in F1.

Self-Crowdsourcing Training for Relation Extraction / Abad, Azad; Nabi, Moin; Moschitti, Alessandro. - ELETTRONICO. - (2017), pp. 518-523. ((Intervento presentato al convegno The 55th Annual Meeting of the Association for Computational Linguistics tenutosi a Vancouver, Canada nel 30 July - 4 August, 2017 [10.18653/v1/P17-2082].

Self-Crowdsourcing Training for Relation Extraction

Azad Abad;Moin Nabi;Alessandro Moschitti
2017-01-01

Abstract

One expensive step when defining crowd- sourcing tasks is to define the examples and control questions for instructing the crowd workers. In this paper, we intro- duce a self-training strategy for crowd- sourcing. The main idea is to use an au- tomatic classifier, trained on weakly su- pervised data, to select examples associ- ated with high confidence. These are used by our automatic agent to explain the task to crowd workers with a question answer- ing approach. We compared our relation extraction system trained with data anno- tated (i) with distant supervision and (ii) by workers instructed with our approach. The analysis shows that our method rela- tively improves the relation extraction sys- tem by about 11% in F1.
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Volume 2: Short Papers
Vancouver, Canada
Association for Computational Linguistics (ACL)
978-1-945626-76-0
Abad, Azad; Nabi, Moin; Moschitti, Alessandro
Self-Crowdsourcing Training for Relation Extraction / Abad, Azad; Nabi, Moin; Moschitti, Alessandro. - ELETTRONICO. - (2017), pp. 518-523. ((Intervento presentato al convegno The 55th Annual Meeting of the Association for Computational Linguistics tenutosi a Vancouver, Canada nel 30 July - 4 August, 2017 [10.18653/v1/P17-2082].
File in questo prodotto:
File Dimensione Formato  
2017_ACL_Self-Crowdsourcing.pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.75 MB
Formato Adobe PDF
1.75 MB Adobe PDF Visualizza/Apri
P17-2082.pdf

accesso aperto

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 1.76 MB
Formato Adobe PDF
1.76 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/195366
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 11
  • ???jsp.display-item.citation.isi??? 7
social impact