Validity Evidence for Scores From a Rubric Used for Holistic Admissions—A Single Institution Study
Kathryn A. Champion, Muath AlDosariABSTRACT
Purpose
Few evidence‐based tools exist to support holistic review in dental school admissions. The Holistic Admissions Assessment Rubric (HAAR) was developed to score applicant data from the Associated American Dental Schools Application Service (AADSAS) applications at one institution as part of an overall holistic review process.
Methods
Validity evidence was collected across four sources: content, response processes, internal structure, and consequences. Content evidence was gathered through a multi‐institutional nominal group process. Response‐process evidence was explored through rater training and calibration. Internal structure was examined using inter‐rater reliability via intraclass correlation coefficient (ICC) and internal consistency (Cronbach's alpha). Consequences evidence was evaluated using logistic regression to determine whether HAAR scores predicted admissions decisions.
Results
The nominal group process produced a 16‐item rubric across experiences, attributes, and metrics. Inter‐rater reliability for overall HAAR scores was excellent (ICC = 0.85). Item discrimination ranged from 0.14 to 0.67, and inter‐item correlations were low (0.05–0.08), indicating non‐redundancy. Logistic regression showed HAAR scores significantly predicted likelihood of admission (OR = 1.28, p < 0.001).
Conclusions
Scores from the HAAR demonstrated promising reliability and meaningful validity evidence across multiple sources. HAAR scores were strongly associated with admissions outcomes at one institution and appear to support more structured holistic review processes.