In many practical applications, machine learning models are embedded into a pipeline involving a human actor that decides whether to trust the machine prediction or take a default route (e.g., classify the example herself). Selective classifiers have the option to abstain from making a prediction on an example they do not feel confident about. Recently, the notion of the value of a machine learning model has been introduced as a way to jointly consider the benefit of a correct prediction, the cost of an error, and that of abstaining. In this paper, we study how active learning of selective classifiers is affected by the focus on value. We show that the performance of the state-of-the-art active learning strategies drops significantly when we evaluate them based on value rather than accuracy. Finally, we propose a novel value-aware active learning strategy that outperforms the state-of-the-art ones when the cost of incorrect predictions substantially outweighs that of abstaining.

Value-Aware Active Learning / Sayin, Burcu; Yang, Jie; Passerini, Andrea; Casati, Fabio. - ELETTRONICO. - 368:(2023), pp. 215-223. (Intervento presentato al convegno Hybrid Human Artificial Intelligence (HHAI2023) tenutosi a Munich, Germany nel June 26-30, 2023) [10.3233/FAIA230085].

Value-Aware Active Learning

Sayin, Burcu
Primo
;
Passerini, Andrea;Casati, Fabio
2023-01-01

Abstract

In many practical applications, machine learning models are embedded into a pipeline involving a human actor that decides whether to trust the machine prediction or take a default route (e.g., classify the example herself). Selective classifiers have the option to abstain from making a prediction on an example they do not feel confident about. Recently, the notion of the value of a machine learning model has been introduced as a way to jointly consider the benefit of a correct prediction, the cost of an error, and that of abstaining. In this paper, we study how active learning of selective classifiers is affected by the focus on value. We show that the performance of the state-of-the-art active learning strategies drops significantly when we evaluate them based on value rather than accuracy. Finally, we propose a novel value-aware active learning strategy that outperforms the state-of-the-art ones when the cost of incorrect predictions substantially outweighs that of abstaining.
2023
Frontiers in Artificial Intelligence and Applications
The Netherlands
IOS Press
9781643683942
9781643683959
Sayin, Burcu; Yang, Jie; Passerini, Andrea; Casati, Fabio
Value-Aware Active Learning / Sayin, Burcu; Yang, Jie; Passerini, Andrea; Casati, Fabio. - ELETTRONICO. - 368:(2023), pp. 215-223. (Intervento presentato al convegno Hybrid Human Artificial Intelligence (HHAI2023) tenutosi a Munich, Germany nel June 26-30, 2023) [10.3233/FAIA230085].
File in questo prodotto:
File Dimensione Formato  
FAIA-368-FAIA230085.pdf

accesso aperto

Descrizione: Value-aware active learning
Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 283.32 kB
Formato Adobe PDF
283.32 kB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/383689
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? 0
social impact