An LLM-based motivational interviewing training system for Japanese clinical education

Background

Motivational interviewing (MI) is an evidence-based approach for promoting behavior change in mental health, primary care, and public health. Effective MI training requires repeated practice with expert feedback, yet such resources remain scarce, and few tools support Japanese. Large language models (LLMs) offer scalable training, but whether LLM-based assessment agrees with expert evaluation remains largely unverified, particularly for Japanese.

Methods

We developed an LLM-based MI training system for Japanese clinical education and are validating its evaluation module against expert judgment. From book-derived case seeds, we generated 81 synthetic Japanese MI dialogues (27 scenarios x 3 intended counselor skill levels). The LLM evaluator assigns MITI 4.2.1 global scores (primary) and utterance-level behavior codes (secondary) with written rationales. Validation comprises (a) benchmarking against the official MITI 4.2.1 manual coding examples and (b) blinded coding of a held-out set by an expert rater, with a second expert reviewing uncertain judgments; agreement is assessed by intraclass correlation and weighted kappa.

Results

On 106 coding examples from the official MITI 4.2.1 manual, the evaluator achieved 71.7% accuracy (Cohen’s kappa=0.68), within the range of inter-coder agreement reported among trained human raters. Misclassifications concentrated in categories that also challenge human coders, such as simple versus complex reflections. Blinded expert coding of a held-out set is ongoing; agreement results will be presented at the conference.

Conclusions

Anchoring automated assessment to the official MITI standard and blinded expert judgment can make scalable MI training trustworthy for Japanese clinical education. Future work includes a feasibility pilot with medical students, strengthening workforce capacity for behavior change counseling.

Key messages

• An LLM-based system can provide scalable MI training for Japanese clinical education, addressing limited access to expert feedback and practice opportunities.

• Anchoring LLM evaluation to official MITI benchmarks and blinded expert coding provides a transparent template for validating AI-assisted skills training before educational use.