Multilingual evaluation of a large language model-based primary care chatbot
Pei-Lun Chen, Amogh Ananda Rao, Sydney Pugh, Yingying Lu, Kevin B JohnsonAbstract
Objective
Large language model (LLM) chatbots show promise for pre-visit planning, but their English-centric development raises equity concerns for multilingual deployment. We evaluated the multilingual capabilities of the Primary Care Pre-visit Chatbot (PCP-Bot), a GPT-4o-based pre-visit chatbot, across English, Mandarin Chinese, Spanish, and Hindi.
Materials and Methods
In this mixed-methods study, 31 bilingual participants (11 Mandarin, 10 Spanish, and 10 Hindi) role-played as patients across 5 synthetic primary care cases, each first in English, then in their second language. Participants completed validated instruments (Bot Usability Scale [BUS-11], NASA Task Load Index [NASA-TLX], Human-Computer Trust Scale [HCTS], and Technology Acceptance Model-Perceived Ease of Use [TAM-PEU]), an adapted summary quality rubric (fluency, adequacy, meaning, and severity), and provided open-ended feedback.
Results
English interactions scored higher than non-English interactions on usability (difference = 0.15), conversation quality (0.62), and summary quality (0.25; all P < .001). Usability and conversation quality gaps varied by language (both P < .01), driven by Spanish and Hindi; Mandarin Chinese showed neither (P = .087). Trust, workload, and ease-of-use remained consistent across languages. Participants described PCP-Bot as natural and accurate overall but reported repetition, transcription errors, and more frequent breakdowns in non-English interactions.
Discussion
Multilingual capability in pre-visit workflows appears feasible for LLMs under controlled conditions in the 4 high-resource languages evaluated, with high absolute performance in each. Disparities were language-dependent: conversational performance declined most for Spanish and Hindi, while summary quality showed a small, uniform English advantage, suggesting addressable language-specific weaknesses rather than a fixed limitation of English-developed systems.
Conclusion
Large language models can support multilingual pre-visit interactions, but performance varies by language. Per-language validation and further testing are needed prior to clinical rollout.