Development and Validation of a ChatGPT-Based Assessment for Conversation-Related Clinical Skills in Occupational Therapy OSCEs
Yi-Ching Wang, An-Ju Chen, Mon-Jane How, Chih-Wei Yang, Ya-Chen Lee, Sheau-Ling Huang, Ching-Lin HsiehImportance
Conversation-related clinical skills are central to Objective Structured Clinical Examinations (OSCEs) in occupational therapy. However, real-time examiner scoring is resource-intensive and may overlook students’ specific skills. Transcript-based scoring with Chat Generative Pre-Trained Transformer (ChatGPT) may address these limitations.
Objective
To develop a ChatGPT-based assessment for chief-complaint interview OSCEs (G-ACE) and examine criterion validity against expert-consensus ratings, test–retest reliability, and inter-rater reliability.
Design
Secondary analysis of de-identified transcripts from 8-min chief-complaint interviews with a simulated patient with stroke during an OSCE.
Settings
A university in northern Taiwan.
Participants
Fifty-eight de-identified transcripts from undergraduate occupational therapy students.
Results
Criterion validity was acceptable for “Primary Symptoms and Related Feelings” and “Impact on Daily Life” items (weighted kappa ≥ 0.60), but lower for “Rehabilitation Expectations” and “Communication Skills” items (0.18–0.59). For domain and total scores, intraclass correlation coefficients (ICCs) were 0.17 to 0.78 and percentages of mean absolute errors (MAE%s) were 4.6% to 20.7%. Test–retest reliability was acceptable at the item level (weighted kappa = 0.68–1.00), and acceptable to good at the domain and total score levels (ICC = 0.65–0.85; MAE% = 2.3%–9.1%). Inter-rater reliability was acceptable for most domain and total scores (ICCs = 0.52–0.78; MAE%s = 1.3%–12.0%), except for the “Rehabilitation Expectations” domain (ICCs = 0.17–0.30; MAE%s = 1.3%–7.3%).
Conclusion and Relevance
The G-ACE demonstrated generally acceptable test–retest reliability and small absolute differences from expert ratings for domain and total scores. Because some items showed lower consistency with expert ratings, formative feedback should be interpreted cautiously based on transcript evidence.