Game-Based Interaction for Older Adults in Care Settings: Field Deployment and Multimodal Behavioral Data Processing

Care settings need short, accessible digital activities that encourage older adults to participate while generating process data during ordinary interaction. We developed a game-based platform with four activities: reflective choice, guided breathing, reminiscence, and artistic expression. Session video, interaction logs, and field notes were organized in a context-preserving multimodal framework. This exploratory seven-module field study included 12 older adults aged 65–90 years in a hospital-affiliated care setting. Ten participants completed all seven scheduled modules, and 81 of 84 planned module sessions were completed; some participants completed multiple modules on the same calendar day. Seventy-two videos were analyzable, representing 88.9% of completed module sessions (72/81). Across 251 activity selections, reminiscence was selected most often; mouse control and navigation were the main operating difficulties. Two raters independently annotated forty 10 s field-video clips. Target-face tracking was stable or basically stable in 90% of clips. Gaze-deviation detection achieved 87.5% accuracy and a macro-F1 of 0.871, while macro-expression classification achieved 67.5% accuracy and a macro-F1 of 0.610. The findings demonstrate a context-preserving field method for structured process records. The algorithms are auxiliary tools, and facial-expression outputs should not be interpreted as direct measures of emotion, cognition, or clinical status.