A Socio-Technical Systems Evaluation of Simulation-Supported Instruction in Transportation Engineering: Configuration Outcomes, Within-System Relationships, and Diagnostic Evidence for Redesign

Simulation platforms are often evaluated as isolated teaching tools, although classroom outcomes reflect coordinated technical and instructional elements. This study evaluated a research-team-developed CARLA–SUMO platform as part of a bounded socio-technical instructional system for teaching ramp-merging and weaving analysis. A class-level controlled quasi-experiment involved 174 undergraduates from four intact classes, with 87 students in each instructional configuration. The evaluation combined configuration-level comparisons of formal knowledge, scenario-based analytical performance, and learning-experience profiles; within-system relational analyses in the simulation-supported group; and scenario- and question-level diagnostics. The simulation-supported configuration showed higher post-instruction knowledge (d=0.425) and scenario-based analytical performance (d=0.783), together with descriptively more positive learning-experience profiles. Perceived learning support was positively associated with cognitive engagement and with learning satisfaction and behavioral intention, while cognitive engagement provided the clearest positive information about scenario performance. Question-level diagnostics identified interacting-behavior identification in Scenario 2 as the clearest candidate for targeted redesign because it combined relatively low absolute performance with a small between-configuration difference. These findings illustrate a three-level approach linking system outputs, within-system relationships, and assessment-informed redesign.