Item Response Modeling of Situational Judgment Tests With Items Nested Within Scenarios
Anna Brown, Emma Morley, Fiona Patterson
Situational judgment tests (SJTs) are widely used to evaluate decision-making and problem-solving skills. These tests present respondents with situations (scenarios), and possible actions in response to each situation, which they must rate based on effectiveness or appropriateness. Despite having high face and criterion-related validities, these tests involve complex cognitive processes, and their construct validity has been notoriously difficult to establish. This is particularly challenging when several questions relate to the same scenario, thus creating local dependencies in item responses after controlling for the overall ability. Finding a suitable measurement model for SJTs would address the operational challenges of test development, maintenance, scoring, and equating the scores across multiple test forms and/or testing years. We propose to use a variant of the continuous response model (CRM;