Retrospective Cohort Comparison of Self, Peer, Near‐Peer, and Faculty Assessment of a Pharmacy Teaching Objective Structured Clinical Examination
Chelsea A. Ludwig, Jamie L. Woodyard, Chelsea M. BakerABSTRACT
Background
Objective Structured Clinical Examinations (OSCEs) are systematic assessments that evaluate various skills in health care professional education curricula. OSCEs and Teaching OSCEs (TOSCEs) are traditionally evaluated by faculty, and there is currently a paucity of literature comparing the use of self, peer, and near‐peer evaluation in formative teaching OSCEs. The purpose of this study was to compare asynchronous self, peer, near‐peer, and faculty assessments of a patient counseling TOSCE station.
Methods
Second‐year student pharmacists in a skills laboratory course participated in a newly created formative TOSCE in fall 2023. The patient counseling station was evaluated by four individuals (self, peer classmate, near‐peer, and faculty) using a standardized analytical checklist. Mean scores were analyzed for differences between groups using a repeated measures Analysis of Variance (ANOVA) test. A quality improvement survey was distributed to students, and responses from two questions related to perceived accuracy of TOSCE evaluations and usefulness of feedback from each evaluator were examined.
Results
TOSCE scores from self, peer, and near‐peer evaluations were statistically significantly higher when compared with faculty (0.57, 0.85, 0.81, respectively). The Intraclass Correlation Coefficient (ICC) used to measure inter‐rater reliability was 0.840 (95% confidence interval [CI] 0.792–0.880). Most students perceived that the self, peer, near‐peer, and faculty evaluators were accurate and provided useful feedback.
Conclusion
A statistically significant difference was detected between faculty and non‐faculty (self, peer, and near‐peer) scores on the patient counseling TOSCE simulation; however, the absolute difference was small, suggesting that the finding may have limited educational or practical significance. Student perceived accuracy and feedback usefulness were similar for all evaluator groups. Both findings reveal an opportunity for the use of non‐faculty evaluators for assessment of TOSCE simulations.