Autonomous Crowdsourcing through Human-Machine Collaborative Learning

In this paper, we introduce a general iterative human-machine collaborative method for training crowdsource workers: a classifier (i.e., the machine) selects the highest quality examples for training crowdsource workers (i.e., the humans). Then, the latter annotate the lower quality examples such that the classifier can be re-trained with more accurate examples. This process can be iterated several times. We tested our approach on two different tasks, Relation Extraction and Community Question Answering, which are also in two different languages, English and Arabic, respectively. Our experimental results show a significant improvement for creating Gold Standard data over distant supervision or just crowdsourcing without worker training. Additionally, our method can approach the performance of the state-of-the-art methods that use expensive Gold Standard for training workers.