Local performance and fairness testing of an AI Scribe in a paediatric developmental assessment clinic in South Australia: a silent trial protocol
Melissa D McCradden, Sheng Tng, Deepa Jeyaseelan, Cathy Leane, Marnie Campbell, Timothy Braund, Lana Earle-Bandaralage, Mary Ebrahimi, Ashish Sharma, Jonathan TangIntroduction
Any tool that can reduce the administrative burden on healthcare providers while preserving safe, accountable and high-quality medical documentation is of immense value both to healthcare institutions and consumers. The key question we need to answer is whether a prospective tool can reduce these burdens while maintaining (and, ideally elevating ) quality documentation standards. The goal of this study is to describe the local performance of a large language model-based documentation assistive tool to draft safe, high-quality documentation in the Child Development Unit at the Women’s and Children’s Hospital. By generating local evidence of performance, we can assess the suitability of the artificial intelligence (AI) Scribe and inform a larger interventional study protocol and establish evidence-based governance.
Methods and analysis
Using an algorithmic audit framework developed specific to our context, we will compare clinician-written clinical notes to AI-generated notes produced in parallel to the standard of care (ie, a ‘silent’ or translational trial paradigm). We will compare the time required to review clinical documentation per the standard of care compared with the AI-supported workflow with consideration to the accuracy of the final documentation. Finally, we will qualitatively describe AI-generated notes and compare them to the current standard to identify specific areas where clinical guidelines (eg, performance information, risk mitigation) would support appropriate clinical use.
Ethics and dissemination
Ethics approval has been obtained by the Women’s and Children’s Health Network Human Research Ethics Committee (HREC) (HRE00067) and the South Australian Aboriginal HREC (#04-25-1185). This protocol offers an accessible example for health institutions looking to apply an evidence-based approach to AI Scribe assessment that prioritises clinical documentation standards. We will publish our study results in an academic journal and include a publicly accessible summary for the general public on the Women’s and Children’s website.
Trial registration number
10.17605/OSF.IO/P6TM5.