We address the challenging Voice Activity Detection (VAD) problem, which determines “Who is Speaking and When?” in audiovisual recordings. The typical audiobased VAD systems can be ineffective in the presence of ambient noise or noise variations. Moreover, due to technical or privacy reasons, audio might not be always available. In such cases, the use of video modality to perform VAD is desirable. Almost all existing visual VAD methods rely on body part detection, e.g., face, lips, or hands. In contrast, we propose a novel visual VAD method operating directly on the entire video frame, without the explicit need of detecting a person or his/her body parts. Our method, named S-VVAD, learns body motion cues associated with speech activity within a weakly supervised segmentation framework. Therefore, it not only detects the speakers/not-speakers but simultaneously localizes the image positions of them. It is an end-to-end pipeline, personindependent and it does not require any prior knowledge nor pre-processing. S-VVAD performs well in various challenging conditions and demonstrates the state-of-the-art results on multiple datasets. Moreover, the better generalization capability of S-VVAD is confirmed for cross-dataset and person-independent scenarios.

S-VVAD: Visual Voice Activity Detection by Motion Segmentation / Shahid, Muhammad; Beyan, Cigdem; Murino, Vittorio. - (2021), pp. 2331-2340. (Intervento presentato al convegno WACV tenutosi a Online nel 5-9 January 2021) [10.1109/WACV48630.2021.00238].

S-VVAD: Visual Voice Activity Detection by Motion Segmentation

Beyan, Cigdem;
2021-01-01

Abstract

We address the challenging Voice Activity Detection (VAD) problem, which determines “Who is Speaking and When?” in audiovisual recordings. The typical audiobased VAD systems can be ineffective in the presence of ambient noise or noise variations. Moreover, due to technical or privacy reasons, audio might not be always available. In such cases, the use of video modality to perform VAD is desirable. Almost all existing visual VAD methods rely on body part detection, e.g., face, lips, or hands. In contrast, we propose a novel visual VAD method operating directly on the entire video frame, without the explicit need of detecting a person or his/her body parts. Our method, named S-VVAD, learns body motion cues associated with speech activity within a weakly supervised segmentation framework. Therefore, it not only detects the speakers/not-speakers but simultaneously localizes the image positions of them. It is an end-to-end pipeline, personindependent and it does not require any prior knowledge nor pre-processing. S-VVAD performs well in various challenging conditions and demonstrates the state-of-the-art results on multiple datasets. Moreover, the better generalization capability of S-VVAD is confirmed for cross-dataset and person-independent scenarios.
2021
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2021
New York
IEEE
978-1-6654-0477-8
978-1-6654-4640-2
Shahid, Muhammad; Beyan, Cigdem; Murino, Vittorio
S-VVAD: Visual Voice Activity Detection by Motion Segmentation / Shahid, Muhammad; Beyan, Cigdem; Murino, Vittorio. - (2021), pp. 2331-2340. (Intervento presentato al convegno WACV tenutosi a Online nel 5-9 January 2021) [10.1109/WACV48630.2021.00238].
File in questo prodotto:
File Dimensione Formato  
S-VVAD_Visual_Voice_Activity_Detection_by_Motion_Segmentation.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 6.64 MB
Formato Adobe PDF
6.64 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/304317
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 19
  • ???jsp.display-item.citation.isi??? 16
  • OpenAlex ND
social impact