From Modalities to Meaning: The Mediating Role of Integrative Saccades in L2 Reading of Picture Books

Abstract

Picture books, which feature the display of both text and images, have been widely used in second language (L2) learning and teaching. During picture book reading, learners need to split their attention between text and images. This behavior is believed to be crucial in cross-modal integration of information that supports comprehension. Despite its importance, integrative saccades, which refer to rapid eye movements between information presented in different modalities, have received little attention in previous L2 multimodal input research. Moreover, studies that capture attention-based learning processes often overlook the ongoing learning processes as a mediator in the treatment-outcome relationship (Jia, Ruirui, and Bronson Hui. 2025. “Modeling Relationships Between Learning Conditions, Processes, and Outcomes: An Introduction to Mediation Analysis in SLA Research.” Studies in Second Language Acquisition 47 (3): 912–47). To fill these gaps, the present brief report reanalyzed the eye-tracking data from Tytko, Tetiana, Bronson Hui, and Nick Pandža (2026. “The Effect of Multimodal Input on L2 Learners’ Reading Comprehension: A Pre-registered, Eye-Tracking Study.” Modern Language Journal 110 (1): 210–39), which examined the effect of different combinations of multimodal input reading conditions: reading-only (RO) versus reading with images (RI) versus reading with images and audio (RIA) on comprehension. With their data, we aim to explore the extent to which the effect of reading conditions on reading comprehension was mediated by the number of integrative saccades. To address this intermediary relationship, we conducted multilevel mediation analysis. It was found that, through integrative saccades, the RIA condition led to better reading comprehension than the RI condition, confirming the hypothesized mechanism that the inclusion of the audio in the RIA condition allows L2 learners to go back and forth between text and images more frequently, and this increased integration of verbal and pictorial information, in turn, led to better reading comprehension.