ChatGPT (GPT-3.5) versus human reviewers for title-and-abstract screening in a scoping review: A retrospective comparative evaluation
Armaan Babaei, Georgi Georgievski, Benita Hosseini, Amy S. Li, Jordan PelcBackground
Conducting systematic literature reviews is a labour-intensive process requiring manual screening of thousands of citations. Artificial intelligence (AI) tools such as ChatGPT offer potential for reducing this burden, yet their acceptability for this purpose has not been widely established.
Objective
To evaluate the screening performance of ChatGPT (GPT-3.5) versus human reviewers for title-and-abstract screening in a scoping review, including its recovery of studies ultimately included after full-text review through the original human screening pathway.
Methods
We applied the Arksey and O’Malley scoping review framework to a set of 4,482 citations previously screened manually as part of a separate research project. Using the ChatGPT API integrated into MS Excel, we applied three predetermined yes/no inclusion questions to each citation. AI-generated answers were compared with the final human screening decisions available in the dataset to assess percent congruence. A prespecified operational success criterion was that ChatGPT would flag all nine studies ultimately included in the human-conducted scoping review.
Results
Overall congruence between ChatGPT and human screeners averaged 80.2% across the three inclusion questions (83.22%, 90.52%, and 66.87%, respectively). ChatGPT identified 61 citations for inclusion versus 50 identified by human reviewers, with only 12 overlapping (sensitivity of 24% against human inclusions). Specificity for excluded citations was 98.89%. All 9 studies ultimately included through the human screening pathway were also captured by ChatGPT, corresponding to 100% recovery of the known human-derived final set (9/9; 95% CI 66–100%).
Conclusions
In this single retrospective scoping review, ChatGPT disagreed substantially with human reviewers about which citations to advance but recovered all nine studies ultimately retained through the original human screening pathway. Further investigation of AI’s efficacy, efficiency, performance, and rigour in citation screening is needed before broad adoption can be recommended.