Automating data extraction from discharge reports via multimodal LLMs in the REBOOT trial

Abstract

Clinical research increasingly relies on Real-World Data locked within unstructured discharge reports inaccessible to conventional analysis (1). Among the valuable information included in discharge reports are pharmacological treatment and details on the cause of hospitalization.

To develop and validate a fully self-contained multi-pipeline LLM architecture for automated pharmacological data extraction and infrequent secondary diagnoses detection (SCAD as a role model) from anonymized clinical discharge reports from the randomized pragmatic REBOOT trial (2).

We deployed a modular three-stage pipeline (figure 1) across eight NVIDIA A-100 GPUs using open-source Vision-Language Models (VLMs) and LLMs (Qwen2.5-VL 72B, Gemma-3 27B) (3, 4): We first performed multimodal text extraction, followed by "intelligent repair" using image enhancement and cross-model validation to ensure document integrity. After that, we used a five-step process with double-inference validation to extract discharge dosages for six key pharmacological classes: antiplatelets, ACE inhibitors, beta-blockers, statins, ARBs, and MRAs. When multiple discharge reports existed, one was selected through comparison with an index case to ensure standardized data extraction. Finally, for the identification of SCAD events, we used amultilingual detection module, generating structured data for lesion type, artery, confidence, and evidence, from eight LLM replicas. To aggregate these detections, we applied a Multi-Model Weighted Consensus (WC) strategy, which assign weights inversely proportional to each model’s global detection rate, prioritizing more conservative models. Documents reaching a certain consensus threshold of 0.37 underwent manual confirmation by cardiologists.

The system processed 13,874 anonymized documents from 8,452 patients. Scaling from 1 to 8 NVIDIA A-100 GPUs reduced processing time from 2,472 to 334 hours, yielding a cost saving of $1,210–$1,516 per batch compared to on-demand cloud services. Image extraction success reached 98.8% post-repair, with pharmacological data recovery achieving an F1 score of 89.1% (figure 2a). The weighted consensus metric outperformed all individual models, achieving a sensitivity of 94.3%, specificity of 79.8%, and an AUC-ROC of 0.887 vs. 0.838 for the best individual model (figure 2b). In high-consensus cases (WCd ≥0.75), clinicians' adjudication confirmed SCAD in 72.9% of instances, capturing 96% of all SCAD cases within the entire cohort.

This multi-pipeline LLM architecture enables accurate, scalable data extraction and rare disease detection from real-world clinical documents without reliance on proprietary interfaces (APIs), ensuring privacy and enhancing performance. The weighted consensus strategy substantially reduced manual curation burden while improving precision. This approach provides a reproducible framework for high-fidelity, large-scale real-world evidence generation in clinical research.Figure 1.Multi-pipeline architectureFigure 2.Performance evaluation