Exploring the Effectiveness of Large Language Models in Generating Clinical Differential Diagnoses: A Case Study of ICU Patients

Recent advances in large language models (LLMs) have demonstrated their potential for a wide range of applications, including biomedical and clinical domains. Their ability to process and interpret large amounts of data, reason over clinical cases, and generate contextually relevant responses has created opportunities for their use in healthcare. In clinical settings, LLMs may assist clinicians in decision-making by generating and ranking potential differential diagnoses based on individual patient cases, thereby helping them consider a broader range of diagnostic possibilities. However, the rapid development of LLMs has resulted in an increasingly diverse range of models with different architectures, sizes, and capabilities, making it challenging to determine which models are most suitable for clinical diagnostic assistance tasks. Furthermore, clinical cases vary considerably in their characteristics and diagnostic complexity, potentially leading to differences in model performance across clinical conditions. This article evaluates 11 LLMs using 60 intensive care unit (ICU) patient cases covering a range of clinical conditions. Specifically, we assess the models’ ability to generate ranked differential diagnoses that align with reference diagnoses and compare their diagnostic performance across models and cases. The results demonstrate that LLMs, particularly recent-generation models, can generate differential diagnoses that align with the reference diagnoses under the evaluated benchmark conditions, while revealing notable differences in performance across models and cases. These findings highlight the importance of systematically evaluating and carefully selecting models when investigating their potential application in clinical diagnosis.