In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: a classifier (i.e., the machine) selects the highest quality examples for training crowdsource workers (i.e., the humans). Then, the latter annotate the lower quality examples such that the classifier can be re-trained with more accurate examples. This process can be iterated several times. We tested our approach on two different tasks, Relation Extraction and Community Question Answering, which are also in two different languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. Additionally, our method can approach the performance of the state-of-the-art methods that use expensive Gold Standard for training workers.

Autonomous Crowdsourcing through Human-Machine Collaborative Learning / Abad, Azad; Nabi, Moin; Moschitti, Alessandro. - ELETTRONICO. - (2017), pp. 873-876. ((Intervento presentato al convegno SIGIR 2017 tenutosi a Shinjuku, Tokyo, Japan nel 07 - 11 August , 2017 [10.1145/3077136.3080666].

Autonomous Crowdsourcing through Human-Machine Collaborative Learning

Azad Abad;Moin Nabi;Alessandro Moschitti
2017-01-01

Abstract

In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: a classifier (i.e., the machine) selects the highest quality examples for training crowdsource workers (i.e., the humans). Then, the latter annotate the lower quality examples such that the classifier can be re-trained with more accurate examples. This process can be iterated several times. We tested our approach on two different tasks, Relation Extraction and Community Question Answering, which are also in two different languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. Additionally, our method can approach the performance of the state-of-the-art methods that use expensive Gold Standard for training workers.
Proceedings of the 40th International ACM SIGIR Conference onResearch and Development in Information Retrieval, Shinjuku, Tokyo,Japan, August 7-11, 2017
New York NY, United States
ACM
978-1-4503-5022-8
Abad, Azad; Nabi, Moin; Moschitti, Alessandro
Autonomous Crowdsourcing through Human-Machine Collaborative Learning / Abad, Azad; Nabi, Moin; Moschitti, Alessandro. - ELETTRONICO. - (2017), pp. 873-876. ((Intervento presentato al convegno SIGIR 2017 tenutosi a Shinjuku, Tokyo, Japan nel 07 - 11 August , 2017 [10.1145/3077136.3080666].
File in questo prodotto:
File Dimensione Formato  
2017_SIGIR_Autonomous_Crowdsourcing.pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.54 MB
Formato Adobe PDF
1.54 MB Adobe PDF Visualizza/Apri
3077136.3080666.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.92 MB
Formato Adobe PDF
1.92 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/195415
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 4
  • ???jsp.display-item.citation.isi??? 1
social impact