In many practical applications, machine learning models are embedded into a pipeline involving a human actor that decides whether to trust the machine prediction or take a default route (e.g., classify the example herself). Selective classifiers have the option to abstain from making a prediction on an example they do not feel confident about. Recently, the notion of the value of a machine learning model has been introduced as a way to jointly consider the benefit of a correct prediction, the cost of an error, and that of abstaining. In this paper, we study how active learning of selective classifiers is affected by the focus on value. We show that the performance of the state-of-the-art active learning strategies drops significantly when we evaluate them based on value rather than accuracy. Finally, we propose a novel value-aware active learning strategy that outperforms the state-of-the-art ones when the cost of incorrect predictions substantially outweighs that of abstaining.
Value-Aware Active Learning / Sayin, Burcu; Yang, Jie; Passerini, Andrea; Casati, Fabio. - ELETTRONICO. - 368:(2023), pp. 215-223. (Intervento presentato al convegno Hybrid Human Artificial Intelligence (HHAI2023) tenutosi a Munich, Germany nel June 26-30, 2023) [10.3233/FAIA230085].
Value-Aware Active Learning
Sayin, Burcu
Primo
;Passerini, Andrea;Casati, Fabio
2023-01-01
Abstract
In many practical applications, machine learning models are embedded into a pipeline involving a human actor that decides whether to trust the machine prediction or take a default route (e.g., classify the example herself). Selective classifiers have the option to abstain from making a prediction on an example they do not feel confident about. Recently, the notion of the value of a machine learning model has been introduced as a way to jointly consider the benefit of a correct prediction, the cost of an error, and that of abstaining. In this paper, we study how active learning of selective classifiers is affected by the focus on value. We show that the performance of the state-of-the-art active learning strategies drops significantly when we evaluate them based on value rather than accuracy. Finally, we propose a novel value-aware active learning strategy that outperforms the state-of-the-art ones when the cost of incorrect predictions substantially outweighs that of abstaining.File | Dimensione | Formato | |
---|---|---|---|
FAIA-368-FAIA230085.pdf
accesso aperto
Descrizione: Value-aware active learning
Tipologia:
Versione editoriale (Publisher’s layout)
Licenza:
Creative commons
Dimensione
283.32 kB
Formato
Adobe PDF
|
283.32 kB | Adobe PDF | Visualizza/Apri |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione