Usage of and Satisfaction with Artificial Intelligence-Generated Draft Replies to Patient Portal Messages

Abstract

This study aimed to describe the usage and perceptions of physicians and nurses in primary and specialty care on the acceptability, usability, and quality of artificial intelligence (AI)-generated draft replies to patient portal messages.

We conducted a pilot study of 80 users across seven clinics within a large academic health system in Pennsylvania. Epic's Augmented Response Technology (ART) used a large language model to present clinicians with draft replies to patient messages. To measure usage, we calculated the frequency with which replies were initiated with ART-generated drafts, as well as the overlap between drafts and replies sent to patients. To measure acceptability, usability, and quality, we developed and disseminated a 15-item survey.

Voluntary pilot participants included 80 users (36 physicians, 25 nurses [registered nurses/licensed practice nurses], 10 nurse practitioners, 9 medical assistants). The mean rate of starting with an ART draft was 20.2%, and 40% of replies exhibited little to no editing of the draft. Of the 52 survey respondents, 66% (n = 33) agreed that AI-generated drafts were useful, and 46% (n = 23) agreed that ART improved the quality of replies to patient messages. Across survey questions, a higher percentage of nurses reported favorable opinions of the technology than did physicians. In free-response questions, the most common themes were that the message content was helpful (28.3%, n = 13) and that it reduced perceived effort or time (23.9%, n = 11), but that drafts occasionally contained incorrect or inappropriate content (34.9%, n = 15) or too often suggested an appointment (27.9%, n = 12).

In this voluntary pilot study, AI-generated draft replies to patient portal messages were generally rated as acceptable and useful, with nurses and primary care clinicians more often reporting acceptance of the technology than physicians or specialty care clinicians. Scaling efforts should continue to monitor acceptability and effects on cognitive burden, in addition to safety.