Automated Point-Of-Care Assessment Metrics for Surgical Residents Performing Consultations: A Pilot Study
Rachael C Acker, Charles T Bradford, Isaac J Perron, J Walker Rosenthal, Sarah I Landau, Charles C Horn, Kaley Piersanti, Gary E Weissman, Ari Friedman, Rachel R KelzBackground:
Surgical resident assessment remains largely dependent on faculty observation and subjective evaluation. Electronic health record (EHR) metadata may enable scalable, objective assessment of resident performance during time-sensitive surgical consultations.
Study Design:
We conducted a proof-of-concept study of postgraduate-year-2 general surgery residents performing consultations for adults across three hospitals from July 2020 through June 2025. EHR audit-log data were used to measure consultation duration and medication reconciliation. Performance was compared across residents and between early (#1–50) and late (#150–200) consultations. Multivariate regression adjusted for patient age, sex, multimorbidity, consultation reason, hospital, inpatient versus emergency department location, shift assignment, and consultation volume per 12-hour shift.
Results:
Fifty-eight residents performed 19,139 consultations (median 328 per resident; range 200–440). Median consultation duration was 3.4 hours (IQR 1.5 hours), and medication reconciliation was completed for 71% of consultations (IQR 11%; range 47%–86%). Both metrics varied across residents (p<0.001), and unadjusted and risk-adjusted resident rankings were similar. With increasing experience, residents completed consultations 18% faster (p=0.002), without a significant change in medication reconciliation completion (p=0.22). Within-resident variation in consultation duration decreased with experience (p<0.001).
Conclusions:
EHR metadata can generate automated, objective measures of surgical resident consultation performance across a large number of clinical encounters. Resident-level variation and increasing consistency with experience suggest that, after further validation and an extension of the metrics to additional domains, this novel platform will provide a complement to existing assessments. This approach may also provide an objectivity not currently available, and could empower residents with timely, real-world data to inform their practices.