Understanding Audiovisual Affective Processing: Behavioral Effects and Neural Underpinnings

Abstract

Emotional signals are typically conveyed jointly through vision and audition. This article reviews behavioral, electrophysiological, and neuroimaging evidence on audiovisual affective processing, addressing what its core effects are, when they unfold, and where they occur in the brain. Congruent audiovisual cues often facilitate emotion recognition, although benefits vary across emotions, tasks, measures, and unimodal comparisons. Audiovisual input can also shape affective appraisals and subjective emotional experience. Modality dominance is context-dependent, varying with cue reliability and the affective dimension judged. Cross-modal influences can persist when one modality is task-irrelevant, suggesting limited goal dependence. Electrophysiological findings reveal audiovisual affective interactions at both early and later stages of neural processing. Neuroimaging work implicates posterior superior temporal cortex (pSTC) within a distributed network of sensory, subcortical, and frontal regions. Building on this evidence, we consider Anderson's information integration theory as a computational framework for characterizing modality-related and value-dependent contributions to affective judgments. We propose a hypothesis-generating dual-pathway framework predicting that reliability-related changes in cue influence will covary with sensory-pSTC interactions, whereas value-dependent changes will additionally involve amygdala and prefrontal systems. These hypotheses concern relative rather than exclusive network contributions and require direct testing through independent manipulations of cue reliability and affective value.