DOI: 10.47582/jompac.1986801 ISSN: 2717-7505

Comparative evaluation of large language models for preliminary diagnosis and initial treatment decision-making in intensive care unit emergencies

Agah Abdullah Kahramanlar, Engin İhsan Turan
Show PDF Cite
Aims: Timely and accurate decision-making is essential in intensive care unit (ICU) emergencies, where delays or errors in preliminary diagnosis and initial treatment may lead to adverse outcomes. Large language model–based Artificial Intelligence (AI) systems have shown promise in clinical reasoning tasks, yet their performance in high-acuity, time-sensitive ICU emergencies remains insufficiently studied. Methods: This simulation-based observational study evaluated the performance of three large language model–based AI systems (Gemini, DeepSeek, and ChatGPT) in preliminary diagnosis and initial treatment decision-making across 84 simulated ICU emergency scenarios. Scenarios reflected real-world ICU practice and covered bleeding, cardiac, electrolyte imbalance, neurologic, respiratory, and other critical conditions. Each scenario included two multiple-choice questions (diagnosis and treatment). Responses from AI models were compared with the expert-derived reference standards established by two boardcertified anesthesiology specialists, with adjudication by a third expert in cases of disagreement. Model performance was assessed using accuracy, macro-averaged sensitivity, specificity, F1 score, and Cohen’s kappa coefficient. Subgroup analyses were performed by emergency type. The study was designed and reported in accordance with the TRIPOD-LLM statement. Results: Overall diagnostic accuracy was 0.93 for Gemini, 0.92 for DeepSeek, and 0.65 for ChatGPT. Treatment decision accuracy was 0.98 for Gemini, 0.95 for DeepSeek, and 0.68 for ChatGPT. Agreement with the expert-derived reference standard was substantial for Gemini (κ=0.743 for diagnosis; κ=0.741 for treatment), substantial for DeepSeek diagnosis (κ=0.710) and moderate for DeepSeek treatment (κ=0.584), but low for ChatGPT (κ=0.144 and κ=0.085, respectively). Gemini and DeepSeek demonstrated consistently high subgroup accuracy across all emergency categories, including perfect performance in respiratory emergencies, whereas ChatGPT showed greater variability and lower accuracy across subgroups. Conclusion: In simulated ICU emergency scenarios, Gemini and DeepSeek demonstrated high diagnostic and treatment decision accuracy, although agreement with the expert-derived reference standard varied by model and task, whereas ChatGPT showed lower and more variable performance. These findings suggest that selected large language models may have potential as decision-support tools in critical care settings; however, performance varies markedly between models. Further validation using real-world clinical data is required before clinical implementation.

More from our Archive