Optimizing Operational Cut Scores under Misclassification Costs and Retake Policies Using IRT‐Based Conditional Uncertainty

Abstract

Cut scores are typically applied to observed or estimated scores that contain measurement error. Even if the criterion cut score itself were error free, error in examinee scores leads to false‐negative and false‐positive classifications. The relative costs of these classification errors may be asymmetric and may be further affected by retake policies that increase the probability of passing across repeated administrations. We propose a decision‐theoretic procedure for selecting an operational cut score that minimizes a weighted expected misclassification loss while respecting the committee‐designated criterion cut score. The method is formulated for item response theory (IRT) scales and accommodates one‐, two‐, or three‐parameter models. Rather than relying on a single reliability‐based standard error, we represent measurement error through the conditional sampling distribution of the proficiency estimator given true proficiency, approximated via parametric bootstrap. Expected false‐negative and false‐positive rates are obtained by Monte Carlo integration over a specified target proficiency distribution, and the optimal operational cut is found by numerical minimization over a feasible cut‐score set. The key feature is that misclassification probabilities are computed from an approximation to the conditional sampling distribution of the proficiency estimator, rather than from a reliability‐based approximation. An empirical illustration using a three‐parameter logistic item response theory model fit to response data from a 36‐item examination shows that the optimal operational cut increases as false positives are weighted more heavily and as the number of permitted administrations increases. The proposed approach provides a practical, model‐based tool for aligning cut‐score policy with explicit misclassification costs and retake rules.