This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.

On the Impact of Predicate Complexity in Crowdsourced Classification Tasks / Ramírez, Jorge; Baez, Marcos; Casati, Fabio; Cernuzzi, Luca; Benatallah, Boualem; Taran, Ekaterina A.; Malanina, Veronika A.. - (2021), pp. 67-75. ((Intervento presentato al convegno International Conference on Web Search and Data Mining tenutosi a Virtual Event Israel nel March, 2021 [10.1145/3437963.3441831].

On the Impact of Predicate Complexity in Crowdsourced Classification Tasks

Ramírez, Jorge;Casati, Fabio;
2021

Abstract

This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.
Proceedings of the 14th ACM International Conference on Web Search and Data Mining
New York
Association for Computing Machinery
9781450382977
Ramírez, Jorge; Baez, Marcos; Casati, Fabio; Cernuzzi, Luca; Benatallah, Boualem; Taran, Ekaterina A.; Malanina, Veronika A.
On the Impact of Predicate Complexity in Crowdsourced Classification Tasks / Ramírez, Jorge; Baez, Marcos; Casati, Fabio; Cernuzzi, Luca; Benatallah, Boualem; Taran, Ekaterina A.; Malanina, Veronika A.. - (2021), pp. 67-75. ((Intervento presentato al convegno International Conference on Web Search and Data Mining tenutosi a Virtual Event Israel nel March, 2021 [10.1145/3437963.3441831].
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/11572/295868
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact