Spatiotemporal aspects in audiovisual interaction

How humans perceive everyday reality is one of the most fascinating and enduring interest of different scientific disciplines. The aim of the present dissertation is to investigate - from both a theoretical and an empirical perspective - some aspects concerning the crossmodal interactions between hearing and vision. In the first part of the introduction special attention is given to the cortical and subcortical neural substrate involved in integrating different sensory modalities, and more specifically, audiovisual stimuli. Experimental studies in the literature designed to empirically investigate different aspects of audiovisual interaction and the potential existence of a sensory dominance between hearing and vision will be presented. In the last part, the introduction will be focalized to discuss some of the principal models aimed to predict the outcome of audiovisual integration and its relation with sensory dominance. The following chapters present the experimental studies designed to empirically investigate different aspects of audiovisual interactions and the role of eye movements in auditory cognition. In a first study, the effects of eye movements on auditory spatial representation will be explored. The aim is to disentangle controversial results in the literature emerged in two studies that used different type of sounds (i.e., free field sounds provided through loudspeakers vs. sounds provided intracranially through headphones) and different tasks. In a second study, the disputed relation between perception and action will be investigated in presence of a crossmodal audiovisual illusion. The aim is to verify whether participants’ visuo-motor behaviour might be biased by the visual illusion as emerged for perception, and eventually, whether perceptive and motor biases are correlated. The last study presented in this dissertation will explore the effects of crossmodal audiovisual stimulation in low vision patients and the relation between the visual pathology. More precisely, the possible visual detection enhancement provided by a sound coupled with a visual stimulus will be investigated. To this purpose, in a first experiment, the effect of spatial disparity between audiovisual stimuli will be deepened while the last two experiments will be focalized on the effects of temporal audiovisual disparities of crossmodal stimuli. The results of the studies described in the present dissertation provide evidence of an effect of eye movements in the auditory spatial cognition and a relation between the perceptive and visuo-motor systems in presence of an illusion induced by a sound. Moreover the presented findings report for the first time a significant crossmodal effect of audition on visual perception in low vision patients.