Artificial Intelligence Versus Physician Decision-Making in Acute Heart Failure: A Comparative Evaluation of Four Contemporary AI Systems
Georgios E Zakynthinos, Evangelos Oikonomou, Iason- Daniil Zakynthinos, George Dimeas, Panteleimon Pantelidis, Theodore G Papaioannou, Gerasimos SiasosAbstract
Background
Large language models (LLMs) are increasingly used by the public and healthcare professionals for clinical reasoning. However, they were developed as general-purpose language models rather than dedicated clinical decision-support tools, and their performance in real-world acute heart failure (AHF) management remains insufficiently evaluated. We compared four contemporary AI models with physician decision-making.
Methods
This single-center retrospective observational study incorporated a cross-sectional AI evaluation of consecutive patients treated at Sotiria General Hospital between July and November 2025, excluding acute coronary syndrome. ChatGPT, ChatGPT Plus, DeepSeek, and an ESC guideline-based retrieval-augmented generation (RAG) model independently assessed identical standardized cases. Recommendations were compared with physician decisions on diagnosis, treatment, guideline-directed medical therapy, timing, and prognosis using Cohen’s kappa, McNemar’s test, Cochran’s Q, Wilcoxon signed-rank testing, and multivariable logistic regression.
Results
Diagnostic performance differed significantly across systems (Cochran’s Q=8.477, P=0.037). In the balanced diagnostic cohort, ChatGPT Plus achieved the highest HF accuracy (accuracy 86.5%; sensitivity 87.0%, specificity 86.0%), followed by RAG (accuracy 77.5%), ChatGPT (accuracy 77.0%), and DeepSeek (accuracy 69.0%). For HF within the top three differential diagnoses, sensitivity was 90.0–97.0%; specificity was 70.0% for ChatGPT Plus, 68.0% for RAG, 19.0% for DeepSeek, and 15.0% for ChatGPT. Physician agreement was highest for guideline-directed therapies, with greater variability in prognosis and escalation of care. RAG more closely matched physician therapeutic timing but did not consistently outperform general-purpose models.
Conclusions
Contemporary AI models show promise in AHF decision support but substantial task-specific variability, supporting use as complements rather than replacements for physician judgment.