Perception of intensity incongruence in synthesized multimodal expressions of laughter

In this paper, we study perception of intensity incongruence between auditory and visual modalities of synthesized expressions of laughter. In particular, we investigate whether incongruent expressions are perceived as 1) regulated, and 2) unsuccessful in terms of animation synthesis. For this purpose, we conducted a perceptive study with the use of a virtual agent. Congruent and incongruent multimodal expressions of laughter were synthesized from natural audiovisual laughter episodes, using machine learning algorithms. Next, the intensity of facial expressions and body movements were systematically manipulated to check whether the resulting incongruent expressions are perceived differently compared to the corresponding congruent expressions. Results show that 1) intensity incongruence lowers the perception of believability and plausibility, and 2) the incongruent laughter expressions displaying high intensity in the audio modality and low intensity in the body movement and facial expression are perceived as more fake than the corresponding congruent expressions. Such results have implications for both animation synthesis as well as expression regulation research.