Integrating longitudinal mixed-effects modeling and XGBoost for chronic disease progression prediction
Rehana ParvinObjective
Accurate prediction of chronic disease progression is essential for improving clinical decision-making, personalized treatment, and healthcare resource allocation. However, longitudinal healthcare data are characterized by temporal dependencies, patient-specific heterogeneity, and complex nonlinear relationships that often limit the effectiveness of conventional predictive models. To address these challenges, this study proposes a hybrid Linear Mixed Model (LMM)–XGBoost framework for predicting chronic disease progression using simulated longitudinal healthcare data.
Methods
A large-scale synthetic dataset comprising 10,000 patients observed over 10 follow-up visits under slow, moderate, and rapid disease progression scenarios was generated through Monte Carlo simulation, yielding 300,000 longitudinal observations. Missing Completely at Random (MCAR) and Missing at Random (MAR) mechanisms were incorporated and handled using Multiple Imputation by Chained Equations (MICE). The proposed framework integrates LMM’s ability to capture longitudinal and patient-specific effects with Extreme Gradient Boosting (XGBoost)’s nonlinear predictive capability. Its performance was compared with Generalized Estimating Equations (GEE), Random Forest (RF), Light Gradient Boosting Machine (LightGBM) and XGBoost models.
Results
demonstrated that the proposed Hybrid LMM–XGBoost framework achieved the best predictive performance among all competing models, yielding an Root Mean Squared Error (RMSE) of 2.84, Mean Absolute Error (MAE) of 2.00, and coefficient of determination (R²) of 0.980. For disease-stage classification, the model attained an accuracy of 0.93, precision of 0.92, recall of 0.92, F1-score of 0.92, and Area Under the Curve (AUC) of 0.981. SHapley Additive exPlanations (SHAP) analysis identified previous disease severity (DS_Lag), Body Mass Index (BMI), glucose level (GLU), and blood pressure (BP) as the most influential predictors. Monte Carlo validation (B = 1000) confirmed the model’s robustness, with a mean RMSE of 2.85 (95% CI: 2.67–3.03).
Conclusion
The findings demonstrate the effectiveness of integrating longitudinal statistical modeling, machine learning, and explainable artificial intelligence for chronic disease progression prediction.