Multimodal language processing in school-aged Mandarin-speaking children: the role of beat gesture in enhancing memory for discourse information

Abstract

Communication is typically multimodal, meaning that communication involves the utilization of various cues from different modalities. While spoken words in the auditory modality primarily convey semantic information, gestures from the visual modality complement and enhance the communication process. However, the role of visual cues, specifically beat gestures, a type of non-verbal co-speech gesture used to emphasize certain information, in language processing remains largely underexplored. The present study, using a memory task where 90 Mandarin-speaking schoolchildren aged 6–9 viewed stories individually and were later asked about what happened in the stories, investigates the role of beat gestures in enhancing the memory for discourse information in Mandarin. The results show that words with beat gestures were generally recalled better for children of all grades, indicating that by age 6, Mandarin children have already acquired the ability to utilize beat gestures to encode a discourse. This study contributes significantly to our limited understanding of the significance of visual cues in multimodal language processing.