Evaluating the performance of large language models in haematopoietic stem cell transplantation decision‐making
Ivan Civettini, Arianna Zappaterra, Bianca Maria Granelli, Giovanni Rindone, Andrea Aroldi, Stefano Bonfanti, Federica Colombo, Marilena Fedele, Giovanni Grillo, Matteo Parma, Paola Perfetti, Elisabetta Terruzzi, Carlo Gambacorti‐Passerini, Daniele Ramazzotti, Fabrizio Cavalca- Hematology
Summary
In a first‐of‐its‐kind study, we assessed the capabilities of large language models (LLMs) in making complex decisions in haematopoietic stem cell transplantation. The evaluation was conducted not only for Generative Pre‐trained Transformer 4 (GPT‐4) but also conducted on other artificial intelligence models: PaLm 2 and Llama‐2. Using detailed haematological histories that include both clinical, molecular and donor data, we conducted a triple‐blind survey to compare LLMs to haematology residents. We found that residents significantly outperformed LLMs (p = 0.02), particularly in transplant eligibility assessment (p = 0.01). Our triple‐blind methodology aimed to mitigate potential biases in evaluating LLMs and revealed both their promise and limitations in deciphering complex haematological clinical scenarios.