AI-powered multimodal extraction of affective and attention-related behavioral features from structured early-childhood interaction videos
Yu-Chun Chen, Yi-Ling Fan, Yuhling Wang, Hui-Ju Chen, Guan-Lin Wu, Ching-Han Hsu, Hung-Yi Chiou, Lun-De LiaoBackground and Objective
Assessing observable affective and visual-orienting behaviors in young children is challenging because behavioral expressions are rapid, context-dependent, and difficult to characterize using conventional intermittent observation alone. This study introduces a multimodal framework for extracting affective and visual-orienting behavioral indicators from structured early-childhood interaction recordings.
Methods
A CNN-based three-class facial expression model was developed using merged public and self-labeled facial expression datasets. Internal image-level performance was evaluated using a held-out test set after duplicate and near-duplicate images were removed. Subject-level partitioning could not be performed because participant identifiers were unavailable. Source-stratified and child-only validation analyses were not prespecified in the present evaluation and are therefore not reported. Rule-based head pose, gaze, and eye state descriptors were extracted as visual-orienting indicators.
Results
Under the tested internal three-class image-level evaluation setting, the selected CNN configuration achieved a reported macro-average class accuracy of 84% (positive: 89%, neutral: 100%, negative: 62%; micro accuracy: approximately 86.4%, 95% CI approximately 84.0%–88.8%), whereas the tested YOLOv8 configuration achieved a reported macro-average class accuracy of 39%. These results were obtained during final internal image-level evaluation after model selection and should be interpreted as a technical feasibility benchmark only. Compared with MTCNN, MediaPipe with landmarks reduced the processing time from 476 s to 9 s under the same benchmark processing conditions. The visual-orienting module generated rule-based head pose, gaze, and eye state descriptors under the tested recording conditions. These outputs were used as descriptive indicators rather than as validated classifiers of attention, engagement, or developmental status. In a small exploratory sample of 12 male children, the system generated descriptive behavioral profiles on the basis of affective and visual-orienting indicators. These pilot comparisons were intended to demonstrate the feasibility of feature extraction only and were not designed to estimate diagnostic, screening, clinical assessment, or educational monitoring performance.
Conclusions
The system supports efficient offline or near-real-time-capable extraction and visualization of predefined affective and visual-orienting descriptors from structured child–clinician interaction recordings. These outputs are exploratory and technical in nature; the corresponding validation limitations are detailed in the Limitations section.