A multimodal architecture for simulating natural interactive walking in virtual environments

We describe a multimodal system that exploits the use of footwear-based interaction in virtual environments. We developed a pair of shoes enhanced with pressure sensors, actuators, and markers. These shoes control a multichannel surround sound system and drive a physically based audio-haptic synthesis engine that simulates the act of walking on different surfaces. We present the system in all its components, and explain its ability to simulate natural interactive walking in virtual environments. We describe two experiments where the possibilities offered by the system are tested. In the first experiment, blindfolded subjects are asked to walk on a virtual rope, guided only by auditory, haptic and audio-haptic feedback provided at feet level. In the second experiment, subjects are overlooking a virtual canyon, while wearing a head mounted display and the developed shoes. Results of the experiments provide some preliminary indications on the role of multimodal feedback delivered at feet ...