Zero-shot learning models achieve remarkable results on image classification for samples from classes that were not seen during training. However, such models must be trained from scratch with specialised methods: therefore, access to a training dataset is required when the need for zero-shot classification arises. In this paper, we aim to equip pre-trained models with zero-shot classification capabilities without the use of image data. We achieve this with our proposed Image-free Classifier Injection with Semantics (ICIS) that injects classifiers for new, unseen classes into pre-trained classification models in a post-hoc fashion without relying on image data. Instead, the existing classifier weights and simple class-wise descriptors, such as class names or attributes, are used. ICIS has two encoder-decoder networks that learn to reconstruct classifier weights from descriptors (and vice versa), exploiting (cross-)reconstruction and cosine losses to regularise the decoding process. Notably, ICIS can be cheaply trained and applied directly on top of pre-trained classification models. Experiments on benchmark ZSL datasets show that ICIS produces unseen classifier weights that achieve strong (generalised) zero-shot classification performance. Code is available at https://github.com/ExplainableML/ImageFreeZSL.

Image-free Classifier Injection for Zero-Shot Classification / Christensen, Anders; Mancini, Massimiliano; Koepke, A. Sophia; Winther, Ole; Akata, Zeynep. - (2023), pp. 19026-19035. (Intervento presentato al convegno ICCV tenutosi a Paris, France nel 2nd - 6th October 2023) [10.1109/ICCV51070.2023.01748].

Image-free Classifier Injection for Zero-Shot Classification

Mancini, Massimiliano
Secondo
;
2023-01-01

Abstract

Zero-shot learning models achieve remarkable results on image classification for samples from classes that were not seen during training. However, such models must be trained from scratch with specialised methods: therefore, access to a training dataset is required when the need for zero-shot classification arises. In this paper, we aim to equip pre-trained models with zero-shot classification capabilities without the use of image data. We achieve this with our proposed Image-free Classifier Injection with Semantics (ICIS) that injects classifiers for new, unseen classes into pre-trained classification models in a post-hoc fashion without relying on image data. Instead, the existing classifier weights and simple class-wise descriptors, such as class names or attributes, are used. ICIS has two encoder-decoder networks that learn to reconstruct classifier weights from descriptors (and vice versa), exploiting (cross-)reconstruction and cosine losses to regularise the decoding process. Notably, ICIS can be cheaply trained and applied directly on top of pre-trained classification models. Experiments on benchmark ZSL datasets show that ICIS produces unseen classifier weights that achieve strong (generalised) zero-shot classification performance. Code is available at https://github.com/ExplainableML/ImageFreeZSL.
2023
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Piscataway, NJ USA
IEEE Computer Society
979-8-3503-0718-4
Christensen, Anders; Mancini, Massimiliano; Koepke, A. Sophia; Winther, Ole; Akata, Zeynep
Image-free Classifier Injection for Zero-Shot Classification / Christensen, Anders; Mancini, Massimiliano; Koepke, A. Sophia; Winther, Ole; Akata, Zeynep. - (2023), pp. 19026-19035. (Intervento presentato al convegno ICCV tenutosi a Paris, France nel 2nd - 6th October 2023) [10.1109/ICCV51070.2023.01748].
File in questo prodotto:
File Dimensione Formato  
Christensen_Image-Free_Classifier_Injection_for_Zero-Shot_Classification_ICCV_2023_paper.pdf

accesso aperto

Descrizione: ICCV paper Open Access version
Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.53 MB
Formato Adobe PDF
1.53 MB Adobe PDF Visualizza/Apri
Image-free_Classifier_Injection_for_Zero-Shot_Classification.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 2.07 MB
Formato Adobe PDF
2.07 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/400794
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 4
  • ???jsp.display-item.citation.isi??? 0
social impact