Multimodal systems extract and convey meanings through different I/O interfaces, such as voice, writing, gestures, gaze movements, and facial expressions. Increasing the communication bandwidth between humans and computers, current technology has the potential of introducing a major shift in the usability of future systems: the interaction becomes more natural, flexible, and robust. The visual/spatial domain is the ideal ground for multimodal systems. Referring to objects in space is strongly simplified by the synergistic usage of natural language and gestures. Despite the importance of visual cues to resolve ambiguities, traditional multimodal interfaces are blind. References are resolved mainly by considering the dialogue context. To match spontaneous user behaviour, we propose an architecture in which gesture recognition depends on anthropomorphic perceptual features. The ecological validity of our approach is confirmed by results from a 'Wizard of Oz' experiment. Users communicated with a simulated multimodal system and moved groups of objects into appropriate boxes. Speech was mediated by a microphone and gestures by an electronic pen. Visual-field organisation was manipulated according to Gestalt principles. Two conditions were tested: high vs low group salience. Results showed that both gesture trajectories and linguistic behaviour are influenced by perceptual-field organisation.

Perception, language, and gesture: towards a natural human - computer interaction

De Angeli, Antonella;
1999-01-01

Abstract

Multimodal systems extract and convey meanings through different I/O interfaces, such as voice, writing, gestures, gaze movements, and facial expressions. Increasing the communication bandwidth between humans and computers, current technology has the potential of introducing a major shift in the usability of future systems: the interaction becomes more natural, flexible, and robust. The visual/spatial domain is the ideal ground for multimodal systems. Referring to objects in space is strongly simplified by the synergistic usage of natural language and gestures. Despite the importance of visual cues to resolve ambiguities, traditional multimodal interfaces are blind. References are resolved mainly by considering the dialogue context. To match spontaneous user behaviour, we propose an architecture in which gesture recognition depends on anthropomorphic perceptual features. The ecological validity of our approach is confirmed by results from a 'Wizard of Oz' experiment. Users communicated with a simulated multimodal system and moved groups of objects into appropriate boxes. Speech was mediated by a microphone and gestures by an electronic pen. Visual-field organisation was manipulated according to Gestalt principles. Two conditions were tested: high vs low group salience. Results showed that both gesture trajectories and linguistic behaviour are influenced by perceptual-field organisation.
1999
ECVP Abstract Supplement
London
Pion Press
De Angeli, Antonella; W., Gerbino; L., Romary; F., Wolff
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/90424
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact