Revisiting EEG Biomarkers in Psychiatry: The Role of Sample Size in Reproducibility
A. Ebadi, N. Le Guern, S. Kaderi, B. Rodríguez-Herreros, N. Chabane, S. Allouch, M. HassanIntroduction
Efforts to identify EEG biomarkers in psychiatry have been hampered by poor reproducibility and inconsistent findings, limiting their clinical translation. Underpowered studies, characterized by small sample sizes and unstable effect estimates, are thought to be a major contributor to this issue.
Objectives
Here, we aim to systematically investigate how sample size affects the stability and replicability of group comparisons and brain–behavior associations in psychiatric EEG research.
Methods
We analyzed a large cohort of ~2,900 participants, including individuals with neurodevelopmental or psychiatric conditions (ADHD, ASD, Anxiety, and Learning disorders) and healthy controls. Using iterative subsampling within each group, we conducted two analyses: (1) group comparisons with healthy controls via ANCOVA, controlling for age and sex, and (2) Spearman correlations between EEG features and clinical scores within each group. We examined trends in effect sizes and the significance of group differences across a wide range of sample sizes, from 10 participants to several hundred. The analyses covered 103 EEG features measured across 19 channels, with additional metrics averaged across all channels and 152 clinical scores.
Results
At smaller sample sizes, significance detection was highly unstable, with results fluctuating across iterations (Fig. 1). We identified four distinct patterns in the evolution of significance across subsamples (Fig. 2a) and observed that overall stability improved with increasing sample size. Across disorders, the minimum sample size required for 95% reproducibility was substantially higher than the typical sample sizes used in EEG studies (Fig. 2b). Smaller samples exhibited high false-negative rates and inflated effect sizes, often overestimating and failing to detect true effects (Fig. 2c). These patterns were consistent across both group comparisons (ANCOVA) and within-group EEG–behavior correlations, highlighting the influence of sample size and clinical heterogeneity on the reliability of brain–behavior associations.
Image 1: Long description.