AI-based algorithms trained with real-world data for suicide risk prevention in adult mental health patients: The IDICIUS Project
M. Fradera, C. Peña-Gómez, X. Sánchez Corrales, M. Caravaca, D. Roche, C. Pontes, E. Barberia, J. Giraldo, D. PalaoIntroduction
Suicidal behaviour is a major public health challenge, characterized by heterogeneity and limited predictability in clinical practice. Approximately 50-80% of suicide victims had prior healthcare contact, with 60% within the previous month. Electronic health records (EHRs) provide longitudinal data that, when analysed with artificial intelligence (AI), can uncover complex patterns preceding suicidal behaviour. The IDICIUS project aims to translate this potential into a hospital-based early warning system to support clinical decision-making.
Objectives
To develop machine learning models for suicide risk prediction using real-world data from EHRs, creating a hospital-based early warning system to complement clinical assessment.
Methods
We conducted a retrospective, population-based study including 41,557 adult patients followed in Parc Taulí Mental Health Service between January 1, 2018, and June 1, 2024. Four RWD sources were integrated: mental health EHRs, Catalonia Suicide Risk Code registry, forensic suicide death records, and outpatient medication data. The final dataset comprised 112 variables and 32,661 adult patients (age range 18–95 years, mean 47.08±19.39, 55.9% female), of whom 2,764 exhibited suicidal behaviour (suicide attempt, active ideation, or death). Models were trained using multiple machine learning algorithms, including logistic regression, support vector machines, ensemble methods (Gradient Boosting, XGBoost, CatBoost, Random Forest, AdaBoost), and deep learning. Performance was assessed on a separate test set using ROC-AUC, precision, recall, accuracy, and F1-score. Hyperparameters were tuned with cross-validation, prioritising recall to minimise false negatives. To address class imbalance, under-sampling was applied.
Results
As shown in Image 1, ensemble methods outperformed traditional statistical models. Gradient Boosting and XGBoost achieved the best overall performance with ROC-AUC >0.95, accuracy >0.91, recall 0.84–0.85, and F1-scores around 0.68. While precision was moderate (0.51–0.58), high recall ensured that most true positive cases were captured. Deep learning models showed acceptable performance (ROC-AUC 0.90, recall 0.69), but did not surpass ensemble methods. Logistic regression and elastic net yielded lower ROC-AUC (<0.85) and F1-scores (<0.40). The most important features of the two best-performing models are presented in Images 2 and 3.
Image 1: Long description.