Self-Crowdsourcing Training for Relation Extraction

One expensive step when defining crowd- sourcing tasks is to define the examples and control questions for instructing the crowd workers. In this paper, we intro- duce a self-training strategy for crowd- sourcing. The main idea is to use an au- tomatic classifier, trained on weakly su- pervised data, to select examples associ- ated with high confidence. These are used by our automatic agent to explain the task to crowd workers with a question answer- ing approach. We compared our relation extraction system trained with data anno- tated (i) with distant supervision and (ii) by workers instructed with our approach. The analysis shows that our method rela- tively improves the relation extraction sys- tem by about 11% in F1.