A Leakage‐Aware Comparative Evaluation of Machine Learning Models for Voice‐Based Parkinson's Disease Screening, With External Validation on an Independent Cohort

ABSTRACT

Voice‐based machine learning models for Parkinson's disease (PD) screening routinely exceed 95% accuracy on the UCI Oxford dysphonia benchmark, but its ∼6 recordings per participant mean a random split places the same speaker's recordings in both training and test sets, rewarding speaker recognition rather than disease detection. To quantify performance under a leakage‐free protocol, identify competitive classifiers, and test transfer to an independent cohort. A total of 9 classifiers were evaluated on 195 phonations from 32 subjects using repeated stratified group nested cross‐validation (10 × 5 × 3, 50 outer folds), subjects grouped in both loops, predictions aggregated per participant, and fold assignment made deterministic for exact reproducibility. Models were compared via paired effect sizes, bootstrap confidence intervals, and Holm‐adjusted values. External validation used 756 recordings from 252 participants (Sakar Istanbul cohort) under three separate threshold policies, with feature equivalence assessed via standardized mean differences among controls. Logistic regression and random forest were statistically indistinguishable at the top (AUC = 0.856 [0.832–0.880] and 0.850 [0.809–0.892]), both exceeding every gradient‐boosting method (CatBoost: 0.761; difference 0.095, ). A total of 12 of the 16 nominally shared features differed by  1 SD between cohorts among controls. Discrimination transferred only partially externally (AUC = 0.685–0.707), while calibration failed entirely: a threshold fixed from training labeled every external participant a patient under logistic regression (specificity = 0.000) and almost none under random forest (sensitivity = 0.096). Leakage‐free evaluation supports 0.86 AUC rather than the 0.95–0.99 accuracy widely reported, consistent with independent claims‐based prodromal studies. This is retrospective classification, not early detection; the reproducible protocol generalizes to other small repeated‐measures biomedical datasets.