Manual versus natural language processing versus large language model screening for preoperative identification of cardiovascular diseases: a prospective cohort study
K Durak, C Puelacher, D M Gualandro, N Glarner, E Ergin, F Haziri, E Kaplan, K Burri-Winkler, G Hure, D Bolliger, L Steiner, E Mujagic, D Lardinois, F Mahfoud, C MuellerAbstract
Background
Guideline-recommended surveillance for perioperative myocardial infarction/injury (PMI) depends on systematic preoperative identification of cardiovascular diseases and ordering of cardiac troponin tests. Implementation, however, is often limited by resource constraints and labour-intensive manual chart review, creating a need for scalable digital solutions.
Purpose
To evaluate the diagnostic accuracy of manual screening, natural language processing (NLP), and large language model (LLM)-based approaches for preoperative identification of cardiovascular diseases.
Methods
Consecutive patients scheduled for major non-cardiac surgery were prospectively included. At 4 PM on the preceding working day, experienced clinical staff performed structured manual screening of available medical records. Simultaneously, the complete electronic health record at that time point was extracted and frozen for automated analysis using NLP and LLM-based algorithms. Discrepant classifications between methods were adjudicated by an independent board-certified cardiologist blinded to the source of identification, generating a reference standard. Diagnostic endpoints included sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for detection of predefined cardiovascular diseases.
Results
Among 1104 patients (median age 62 years [IQR 46–73]; 47% women), 150 (13.6%) had ≥1 cardiovascular disease according to the reference standard. Manual screening identified cardiovascular disease with a sensitivity of 76.0% (95% CI 68.6–82.1) and specificity of 98.6% (97.7–99.2), yielding a PPV of 89.8% and NPV of 96.3%. NLP-based screening achieved the highest sensitivity at 99.3% (96.3–99.9) with a specificity of 97.3% (96.0–98.1), corresponding to a PPV of 85.1%, NPV of 99.9%, and F1 score of 91.7%. LLM-based screening demonstrated a sensitivity of 90.0% (84.2–93.8) and the highest specificity of 99.5% (98.8–99.8), with a PPV of 96.4%, NPV of 98.4%, and F1 score of 93.1%.
Conclusions
Both NLP- and LLM-based screening outperformed structured manual chart review by clinical staff. NLP-based screening achieved the highest sensitivity, supporting its use as a rule-out strategy. In contrast, LLM-based screening demonstrated the highest specificity, suggesting utility for rule-in identification.Figure