Emotion recognition is involved in several real-world applications. With an increase in available modalities, automatic understanding of emotions is being performed more accurately. The success in Multimodal Emotion Recognition (MER), primarily relies on the supervised learning paradigm. However, data annotation is expensive, time-consuming, and as emotion expression and perception depends on several factors (e.g., age, gender, culture) obtaining labels with a high reliability is hard. Motivated by these, we focus on unsupervised feature learning for MER. We consider discrete emotions, and as modalities text, audio and vision are used. Our method, as being based on contrastive loss between pairwise modalities, is the first attempt in MER literature. Our end-to-end feature learning approach has several differences (and advantages) compared to existing MER methods: i) it is unsupervised, so the learning is lack of data labelling cost; ii) it does not require data spatial augmentation, modality alignment, large number of batch size or epochs; iii) it applies data fusion only at inference; and iv) it does not require backbones pre-trained on emotion recognition task. The experiments on benchmark datasets show that our method outperforms several baseline approaches and unsupervised learning methods applied in MER. Particularly, it even surpasses a few supervised MER state-of-the-art.

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss / Franceschini, Riccardo; Fini, Enrico; Beyan, Cigdem; Conti, Alessandro; Arrigoni, Federica; Ricci, Elisa. - ELETTRONICO. - (2022), pp. 2589-2596. (Intervento presentato al convegno ICPR tenutosi a Montreal, Canada nel 21th - 25 August 2022) [10.1109/ICPR56361.2022.9956589].

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Franceschini, Riccardo;Fini Enrico;Beyan Cigdem;Conti, Alessandro;Arrigoni, Federica;Ricci, Elisa
2022-01-01

Abstract

Emotion recognition is involved in several real-world applications. With an increase in available modalities, automatic understanding of emotions is being performed more accurately. The success in Multimodal Emotion Recognition (MER), primarily relies on the supervised learning paradigm. However, data annotation is expensive, time-consuming, and as emotion expression and perception depends on several factors (e.g., age, gender, culture) obtaining labels with a high reliability is hard. Motivated by these, we focus on unsupervised feature learning for MER. We consider discrete emotions, and as modalities text, audio and vision are used. Our method, as being based on contrastive loss between pairwise modalities, is the first attempt in MER literature. Our end-to-end feature learning approach has several differences (and advantages) compared to existing MER methods: i) it is unsupervised, so the learning is lack of data labelling cost; ii) it does not require data spatial augmentation, modality alignment, large number of batch size or epochs; iii) it applies data fusion only at inference; and iv) it does not require backbones pre-trained on emotion recognition task. The experiments on benchmark datasets show that our method outperforms several baseline approaches and unsupervised learning methods applied in MER. Particularly, it even surpasses a few supervised MER state-of-the-art.
2022
26st International Conference on Pattern Recognition (ICPR) 2022
New York, USA
IEEE/ IAPR
978-1-6654-9062-7
978-1-6654-9063-4
Franceschini, Riccardo; Fini, Enrico; Beyan, Cigdem; Conti, Alessandro; Arrigoni, Federica; Ricci, Elisa
Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss / Franceschini, Riccardo; Fini, Enrico; Beyan, Cigdem; Conti, Alessandro; Arrigoni, Federica; Ricci, Elisa. - ELETTRONICO. - (2022), pp. 2589-2596. (Intervento presentato al convegno ICPR tenutosi a Montreal, Canada nel 21th - 25 August 2022) [10.1109/ICPR56361.2022.9956589].
File in questo prodotto:
File Dimensione Formato  
IC23_MultimodalEmotionRecognitionwithModalityPairwise.pdf

Solo gestori archivio

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.05 MB
Formato Adobe PDF
1.05 MB Adobe PDF   Visualizza/Apri
Multimodal_Emotion_Recognition_with_Modality-Pairwise_Unsupervised_Contrastive_Loss.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 1.76 MB
Formato Adobe PDF
1.76 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/352681
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? 7
social impact