Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals’ personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.

SALSA: A novel dataset for multimodal group behavior analysis / Alameda Pineda, Xavier; Staiano, Jacopo; Subramanian, Ramanathan; Batrinca, Ligia Maria; Ricci, Elisa; Lepri, Bruno; Lanz, Oswald; Sebe, Niculae. - In: IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. - ISSN 0162-8828. - 38:8(2016), pp. 1707-1720. [10.1109/TPAMI.2015.2496269]

SALSA: A novel dataset for multimodal group behavior analysis

Alameda Pineda, Xavier;Staiano, Jacopo;Subramanian, Ramanathan;Batrinca, Ligia Maria;Ricci, Elisa;Lepri, Bruno;Lanz, Oswald;Sebe, Niculae
2016-01-01

Abstract

Studying free-standing conversational groups (FCGs) in unstructured social settings (e.g., cocktail party) is gratifying due to the wealth of information available at the group (mining social networks) and individual (recognizing native behavioral and personality traits) levels. However, analyzing social scenes involving FCGs is also highly challenging due to the difficulty in extracting behavioral cues such as target locations, their speaking activity and head/body pose due to crowdedness and presence of extreme occlusions. To this end, we propose SALSA, a novel dataset facilitating multimodal and Synergetic sociAL Scene Analysis, and make two main contributions to research on automated social interaction analysis: (1) SALSA records social interactions among 18 participants in a natural, indoor environment for over 60 minutes, under the poster presentation and cocktail party contexts presenting difficulties in the form of low-resolution images, lighting variations, numerous occlusions, reverberations and interfering sound sources; (2) To alleviate these problems we facilitate multimodal analysis by recording the social interplay using four static surveillance cameras and sociometric badges worn by each participant, comprising the microphone, accelerometer, bluetooth and infrared sensors. In addition to raw data, we also provide annotations concerning individuals’ personality as well as their position, head, body orientation and F-formation information over the entire event duration. Through extensive experiments with state-of-the-art approaches, we show (a) the limitations of current methods and (b) how the recorded multiple cues synergetically aid automatic analysis of social interactions. SALSA is available at http://tev.fbk.eu/salsa.
2016
8
Alameda Pineda, Xavier; Staiano, Jacopo; Subramanian, Ramanathan; Batrinca, Ligia Maria; Ricci, Elisa; Lepri, Bruno; Lanz, Oswald; Sebe, Niculae...espandi
SALSA: A novel dataset for multimodal group behavior analysis / Alameda Pineda, Xavier; Staiano, Jacopo; Subramanian, Ramanathan; Batrinca, Ligia Maria; Ricci, Elisa; Lepri, Bruno; Lanz, Oswald; Sebe, Niculae. - In: IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. - ISSN 0162-8828. - 38:8(2016), pp. 1707-1720. [10.1109/TPAMI.2015.2496269]
File in questo prodotto:
File Dimensione Formato  
PAMI2016_salsa.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 848.67 kB
Formato Adobe PDF
848.67 kB Adobe PDF   Visualizza/Apri
1506.06882.pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 5.82 MB
Formato Adobe PDF
5.82 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/148069
Citazioni
  • ???jsp.display-item.citation.pmc??? 3
  • Scopus 104
  • ???jsp.display-item.citation.isi??? 77
social impact