DRIVE‐SAFE: Data‐Driven Robustness and Informed Validation for Evolving Specifications via Formal Evaluation

We present a regulator‐driven framework for the temporal evaluation of learning‐based, black‐box autonomous robot policies under evolving safety requirements. Motivated by real‐world certification processes in which regulators often assess observable behavior in inaccessible black‐box policies, we simulate a scenario where deployed policies must maintain ongoing compliance as human‐defined safety rules change over time. Natural‐language safety requirements are translated into Signal Temporal Logic (STL) specifications, which are then used to quantitatively evaluate rollout traces from black‐box policies. We compute Total Robustness Value (TRV), Largest Robustness Value (LRV), and Average Violation Robustness Voting (AVRV), to summarize average performance, worst‐case violation, and average specification violation across a set of rollout trajectories. These metrics are combined with domain‐informed importance weights to provide targeted feedback to model designers, indicating where retraining or redesign is most needed. In a virtual driving scenario, iterative retraining increased satisfaction of a global speed limit specification by 72% and a slippage mitigation specification by 32%. When the speed‐limit requirement was tightened, the resulting retrained model increased satisfaction by 47% but revealed trade‐offs with competing safety objectives. Overall, our approach integrates temporal‐logic‐based robustness metrics with statistically meaningful evaluation, bridging regulator oversight and model improvement under evolving safety requirements.