Medical Physiology Faculty Overpredict Assessment Question Complexity and Alignment with National Board Expectations

Background: Medical licensing exams require students in allopathic and osteopathic programs to apply foundational science knowledge to board-style patient presentations. This study aimed to assess Medical Physiology course questions and their alignment with the National Board of Medical Examiners’ (NBME) Item-Writing Guide.

Methods: The research team obtained practice and exam questions written by five physiology faculty, totaling 316 practice and 187 exam questions. Questions were reviewed by the research team and assessed for word count then coded into three categories: basic science (lowest complexity), clinical scenario (moderate complexity), or NBME-style (highest complexity). A Step-1 practice exam was obtained from the NBME and questions assessed for word count then coded into the same categories.

Faculty members completed a survey which assessed NBME-style question familiarity and estimating the percentages of their practice and exam questions in each category.

Results: Faculty predicted their practice and exam questions included 14% and 23%, respectively, of NBME-style questions. However, the research team coded and found only 3% of practice and 7% of exam questions met NBME-style criteria. In contrast, the Step-1 practice exam contained 92% NBME-style questions. Significantly lower word counts were identified in course exams compared to practice Step-1 questions. Survey results indicate a high degree of concern over student success on Step-1 licensure.

Discussion: Faculty predictions highlight extensive overprediction of NBME-style questions and may contribute to gaps in student Step-1 preparedness. Training with the NBME’s Item-Writing Guide may be needed to help address this gap.