Development and Implementation of an AI System for Generating Clinical Urine Drug Test Sign-Outs
Nathan Laha, Michael Keebaugh, Hsuan-Chieh Liao, Bright Amankwaa, Olumuyiwa Adesoye, Abed Pablo, William S. Phipps, Andrew N. Hoofnagle, Geoffrey S. Baird, Patrick C. Mathias, Brody H. FoyImportance
Modern natural language processing tools have potential to improve clinical workflows, but few have been successfully deployed in practice.
Objective
To describe the development, deployment, and evaluation of an artificial intelligence (AI) language tool for generating preliminary sign-outs to support a urine drug testing service.
Design, Setting, and Participants
In this prognostic study, large language models (LLMs) were used to extract substance use patterns from clinical urine drug test interpretations at a single medical center between January 1, 2014, and February 29, 2024. An AI model using these data was trained to predict substance use from qualitative and quantitative urine testing results. Predicted substance use patterns were used to create preliminary clinical sign-out statements, which were then integrated into an existing clinical workflow.
Main Outcomes and Measures
Predeployment and postdeployment user studies were performed to evaluate model performance and user experience within the workflow. Statistical differences between event rates were calculated using χ
2
tests, and between means using
Results
A total of 83 553 urine tests from 26 459 patients (12 413 male [46.9%]; mean [SD] age, 47.5 [16.7] years) were analyzed. LLM-based extraction of substance-use patterns was 99.9% accurate (13 509 of 13 520 tests), outperforming human labeling. Substance use prediction was similarly accurate, with area under the receiver operating curve greater than 0.99 for 23 of 26 substances. Workflow integration of the AI tool reduced clinical sign-out times by 28.5 seconds per case (23% efficiency gain), and by 65 seconds per case (51% efficiency gain) when integrated alongside a second, non-AI workflow improvement.
Conclusions and Relevance
In this prognostic study, AI-based interpretation of urine drug testing was fast and accurate, providing notable efficiency gains to the clinical service. These findings suggest that natural language processing tool integration can provide substantial clinical benefit, without compromising quality of care.