Evaluating multimodal fusion strategies for early postoperative delirium prediction using structured electronic health record data and clinical narratives
Lena Ara, Zina Ben Miled, Malaz Boustani, Sanjay MohantyAbstract
Objectives
To evaluate multimodal fusion strategies for postoperative delirium prediction using structured electronic health record (EHR) data and clinical notes.
Materials and Methods
Using EHR data from patients undergoing major surgery (2013-2021), we compared unimodal and multimodal models. Structured data were modeled using XGBoost and Self-Attention and Intersample Attention Transformer (SAINT), and clinical notes using a fine-tuned Bioformer, a biomedical transformer language model. Multimodal approaches included feature-level, decision-level, gated, cross-attention, and representation-level fusion.
Results
Representation-level fusion achieved the highest area under the receiver operating characteristic curve (AUC-ROC = 0.791), followed by decision-level fusion (0.780), gated fusion (0.774), feature-level fusion (0.772), and cross-attention fusion (0.765). Among unimodal models, Bioformer achieved an AUC-ROC of 0.761, while SAINT and XGBoost each achieved an AUC-ROC of 0.753.
Discussion
Multimodal approaches consistently improved performance, suggesting complementary predictive information from structured variables and clinical narratives. Notably, representation-level fusion outperformed more complex fusion strategies despite its simpler architecture.
Conclusion
Representation-level fusion achieved the best performance, supporting multimodal learning for postoperative delirium prediction.