COMPASS: Benchmarking Safety‐Aware Vision‐Language Navigation in Intelligent Transport Environments

ABSTRACT

Vision‐language models (VLMs) are increasingly considered for high‐level decision support in intelligent transport settings, where navigation errors and violations of explicit safety constraints can create deployment risks for pedestrians, vehicles, and mobile robotic systems. This work presents COMPASS (Compliance‐Oriented Model Performance Assessment for Safety Standards), an open‐source benchmark that unifies spatial navigation, ISO‐grounded safety compliance, and instruction following within a single evaluation framework for VLMs. COMPASS poses multi‐step urban navigation tasks in environments constructed from real‐world Google Street View imagery while enforcing safety constraints derived from ISO 13482 and ISO/TR 4448. These constraints are operationalised as formal rules, enabling measurable, reproducible, and safety‐aware evaluation. The tasks are generated procedurally, providing extensible evaluation instances and supporting future reinforcement learning with verifiable rewards. An evaluation of six frontier VLMs from Anthropic, OpenAI, and Google shows that COMPASS differentiates performance across navigation, safety compliance, and instruction following. The best‐performing model reaches only 24.2 out of 100, showing a substantial capability gap. All models achieve perfect instruction‐following scores and near‐perfect compliance with safety rules verifiable from command properties alone. However, spatial navigation and safety compliance requiring spatial awareness remain the main bottlenecks. Analysis of available reasoning traces shows that models can often identify navigation errors but fail to recover from them, and may fabricate safety rules absent from the prompt. These findings indicate that current VLMs are not yet suitable for safety‐critical embodied AI tasks requiring robust spatial reasoning. For benchmark users, this means that model developers should first test spatial navigation and context‐dependent safety compliance before considering deployment in transport‐relevant embodied settings.