Temporal alignment of gesture with discourse structure in naturalistic communication

Abstract

How discourse is reorganized in real time remains difficult to observe directly because discourse transitions are typically identified from linguistic content rather than continuous behavior. Although gesture is a fundamental component of human communication, it remains unclear whether the timing and intensity of hand movement systematically change around such transitions in naturalistic settings. We addressed this question by using automated pose estimation to extract continuous hand-movement signals from mathematics and literature classroom instruction and aligning them with independently annotated discourse boundaries. Brief bursts of unusually strong hand movement occurred more often near high-confidence discourse boundaries than during other analyzed portions of the lessons, showing that manual activity clustered around moments of discourse reorganization. This boundary-centered pattern was reproducible within individual teachers across multiple lessons, although its precise shape differed across teachers and was similar in mathematics and literature instruction despite differences in communicative content. Collectively, these findings suggest that the timing of hand movement may provide an observable behavioral correlate of discourse organization and show that automated continuous-movement analysis offers a scalable way to investigate how naturalistic communication unfolds over time. More broadly, the results are consistent with theoretical accounts that view human communication as multimodal, embodied, and dynamically coordinated.