Fairness is crucial when training a deep-learning discriminative model, especially in the facial domain. Models tend to correlate specific characteristics (such as age and skin color) with unrelated attributes (downstream tasks), resulting in biases which do not correspond to reality. It is common knowledge that these correlations are present in the data and are then transferred to the models during training (e.g., [35]). This paper proposes a method to mitigate these correlations to improve fairness. To do so, we learn interpretable and meaningful paths lying in the semantic space of a pre-trained diffusion model (DiffAE) [27] -such paths being supervised by contrastive text dipoles. That is, we learn to edit protected characteristics (age and skin color). These paths are then applied to augment images to improve the fairness of a given dataset. We test the proposed method on CelebA-HQ and UTKFace on several downstream tasks with age and skin color as protected characteristics. As a proxy for fairness, we compute the difference in accuracy with respect to the protected characteristics. Quantitative results show how the augmented images help the model improve the overall accuracy, the aforementioned metric, and the disparity of equal opportunity. Code is available at: https://github.com/Moreno98/Vision-Language-Bias-Control.

Improving Fairness using Vision-Language Driven Image Augmentation / D'Inca, Moreno; Tzelepis, Christos; Patras, Ioannis; Sebe, Nicu. - (2024), pp. 4683-4692. (Intervento presentato al convegno 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024 tenutosi a Waikoloa, HI, USA nel 03-08 January, 2024) [10.1109/WACV57701.2024.00463].

Improving Fairness using Vision-Language Driven Image Augmentation

D'Inca, Moreno;Sebe, Nicu
2024-01-01

Abstract

Fairness is crucial when training a deep-learning discriminative model, especially in the facial domain. Models tend to correlate specific characteristics (such as age and skin color) with unrelated attributes (downstream tasks), resulting in biases which do not correspond to reality. It is common knowledge that these correlations are present in the data and are then transferred to the models during training (e.g., [35]). This paper proposes a method to mitigate these correlations to improve fairness. To do so, we learn interpretable and meaningful paths lying in the semantic space of a pre-trained diffusion model (DiffAE) [27] -such paths being supervised by contrastive text dipoles. That is, we learn to edit protected characteristics (age and skin color). These paths are then applied to augment images to improve the fairness of a given dataset. We test the proposed method on CelebA-HQ and UTKFace on several downstream tasks with age and skin color as protected characteristics. As a proxy for fairness, we compute the difference in accuracy with respect to the protected characteristics. Quantitative results show how the augmented images help the model improve the overall accuracy, the aforementioned metric, and the disparity of equal opportunity. Code is available at: https://github.com/Moreno98/Vision-Language-Bias-Control.
2024
Proceedings - 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024
Piscataway, NJ USA
Institute of Electrical and Electronics Engineers Inc.
979-8-3503-1892-0
979-8-3503-1893-7
D'Inca, Moreno; Tzelepis, Christos; Patras, Ioannis; Sebe, Nicu
Improving Fairness using Vision-Language Driven Image Augmentation / D'Inca, Moreno; Tzelepis, Christos; Patras, Ioannis; Sebe, Nicu. - (2024), pp. 4683-4692. (Intervento presentato al convegno 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024 tenutosi a Waikoloa, HI, USA nel 03-08 January, 2024) [10.1109/WACV57701.2024.00463].
File in questo prodotto:
File Dimensione Formato  
DInca_Improving_Fairness_Using_Vision-Language_Driven_Image_Augmentation_WACV_2024_paper (1).pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 9.23 MB
Formato Adobe PDF
9.23 MB Adobe PDF Visualizza/Apri
Improving_Fairness_using_Vision-Language_Driven_Image_Augmentation.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 2.02 MB
Formato Adobe PDF
2.02 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/409350
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
social impact