Development and temporal validation of a machine learning model to estimate total MoCA scores from telephone-based items in patients with mild cognitive impairment
Chan-Young KwonBackground
Periodic cognitive monitoring for mild cognitive impairment (MCI) is crucial. However, standard assessments like the Montreal Cognitive Assessment (MoCA) require in-person administration due to visual-based items, creating significant accessibility barriers.
Objective
This study aims to develop and rigorously validate a machine learning model to accurately estimate total MoCA scores using only telephone-based (remote) items and demographic information.
Methods
Data from three independent MCI cohorts (2022–2024) were used. Data from 2022 to 2023 (N=425) formed the training set; 2024 data (N=356) were the temporal test set. Models were trained to predict the 8-point visual-based score using 37 telephonic/demographic features. Fivefold cross-validation (80% training/20% validation per fold) was performed exclusively on the training set to compare three heterogeneous machine learning models. Random Forest, Linear Regression and XGBoost models were compared. The final score was reconstructed by summing actual telephonic scores with the predicted visual score.
Findings
In the fivefold cross-validation on the training set, the Random Forest model achieved the lowest mean Root Mean Squared Error (RMSE=1.2034) of 1.2034 points (SD=0.0407; 95% CI 1.1528 to 1.2540) for Visual Score prediction, though pairwise differences between models did not reach statistical significance, likely reflecting the limited power of fivefold comparisons. When applied to the 2024 temporal test set, the reconstructed total score showed high concordance with the actual total MoCA score (RMSE=1.29 points; mean absolute error=1.01 points; R 2 =0.9248). The most important features were education (28.7% importance) and age (13.8%), followed by sustained attention (4.1%).
Conclusions
Our machine learning model accurately and reliably predicts total 30-point MoCA scores using only telephone-based information. The model demonstrated satisfactory generalisation performance on a temporally separated cohort, supporting its potential clinical applicability pending external validation.