Artificial intelligence for automated baseline cardiovascular condition identification: accuracy, confidence, and discordance
R Aggarwal, A Rodman, A K Manrai, D L Bhatt, A Haimovich, R K Wadhera, R W YehAbstract
Background
Accurate identification of baseline cardiovascular conditions is essential for clinical registries and trials but relies on labor-intensive manual abstraction. Large language models (LLMs) offer a potential solution, yet their real-world performance and reliability remain uncertain.
Purpose
We evaluated LLM accuracy, model confidence, and areas of disagreement with registry classifications for common cardiovascular conditions.
Methods
Data were obtained from patients undergoing percutaneous coronary intervention (PCI) at a tertiary academic center who were entered into a regional cardiovascular registry in 2024. Among 889 patients, analyses were restricted to 279 with available linked admission notes after excluding repeat admissions and those without note linkage. Registry comorbidities were entered by a nurse overseeing data quality. A locally hosted LLM (Llama 3.3 70B) identified hypertension, diabetes, dyslipidemia, and heart failure, while also assigning high or low confidence based on admission notes. Performance was evaluated overall and by confidence, with review of disagreements by a board-certified cardiologist.
Results
The prevalence of baseline conditions was high, including hypertension (85.3%), diabetes (44.8%), dyslipidemia (86.7%), and heart failure (42.7%). AI demonstrated high overall accuracy in identifying cardiovascular conditions: hypertension 87.5% (95% CI 83.0–91.1), diabetes 96.1% (95% CI 93.1–98.0), dyslipidemia 90.7% (95% CI 86.6–93.8), and heart failure 87.1% (95% CI 82.6–90.8). Sensitivity and specificity were as follows: hypertension 89.1%/78.1%, diabetes 92.0%/99.4%, dyslipidemia 96.3%/54.1%, and heart failure 75.6%/95.6%. Accuracy was higher for high-confidence compared with low-confidence predictions across all conditions, including hypertension (94.6% vs 58.2%), diabetes (99.2% vs 93.7%), dyslipidemia (91.9% vs 55.6%), and heart failure (96.3% vs 83.3%) (Figure 1). Across all condition determinations (n=1116), 108 (9.7%) were discordant between AI and registry classifications. Manual review by a board-certified cardiologist found that the AI classification was correct in 85 (78.7%) discordant cases while original human determination was correct in 23 (21.3%) cases (Figure 2).
Conclusion
AI accurately identified cardiovascular conditions in this real-world registry cohort, and model-derived uncertainty may help prioritize manual review. Discordance analyses suggest a role for AI in supporting data quality in registries and trials.