Development and Validation of the ATRAI Questionnaire to Assess Attitudes Toward Large Language Models in Clinical Setting (ATRAI-LLM)

Background: Large language models (LLMs) are increasingly integrated into real-world medical practice as chatbots for answering clinical queries. However, the perceptions of this technology among its end-users remain understudied. Existing research on physicians’ attitudes toward LLMs relies on non-validated questionnaires, raising concerns about the accuracy and reliability of the findings. The aim of this study is to develop and validate a questionnaire to assess physicians’ attitudes toward LLM-based chatbots used as a reference tool for answering queries. Methods: The instrument was based on the previously developed and validated ATRAI-14 questionnaire assessing radiologists’ attitudes toward artificial intelligence. Items for the new questionnaire were formulated and refined through focus group testing. Validation involved 562 physicians of various specialties working in medical institutions within the Moscow healthcare system. Some respondents had prior experience working with medical LLMs. We assessed face, content, construct, and criterion validity. Criterion validity was evaluated through correlation between respondents’ self-assessed attitudes toward LLMs measured by visual analogue scale (VAS), and construct validity through confirmatory factor analysis. Results: The resulting ATRAI-LLM questionnaire comprised 19 items (8 in the background part and 11 in the main part). The questionnaire demonstrated acceptable internal consistency (Cronbach’s α = 0.770, McDonald’s ωt = 0.830). It encompasses three domains: “Willingness to Use”, “Implementation Perspective”, and “Hopes and Fears.” Confirmatory factor analysis supported the three-factor structure, with satisfactory fit indices achieved (RMSEA = 0.05, CFI = 0.97, TLI = 0.96, SRMR = 0.03). Criterion validity was confirmed as acceptable with moderate correlation between the final score and VAS scores (Spearman’s rho 0.68, p < 0.001). Conclusions: ATRAI-LLM is a validated instrument for assessing physicians’ attitudes toward LLMs as a knowledge base.