A Review of Model Uncertainty, Cyber–Physical Threats and Assurance Frameworks for Safety‐Critical AI in Process Industries

ABSTRACT

Artificial intelligence (AI) is increasingly being applied in process industries to support leak detection, hazard identification, predictive maintenance, emergency shutdown support and digital‐twin‐enabled decision‐making. Although existing studies largely emphasise the safety benefits of machine learning (ML) and deep learning (DL), less attention has been given to how AI itself may introduce new safety‐critical vulnerabilities. This review synthesises literature published between 2015 and 2025. It develops a process‐safety‐oriented taxonomy of seven AI‐induced risk categories: model uncertainty, data risk, model risk, operational risk, human risk, cyber‐physical risk and governance risk. For each category, the review examines the dominant causal mechanisms, representative failure modes and plausible links to major accident hazard (MAH) scenarios, including fire, explosion, toxic release and escalation of abnormal events. Building on this taxonomy, the paper advances an integrated assurance perspective for AI‐enabled safety functions, emphasising verification, validation and uncertainty quantification; adversarial resilience; safety‐case evidence; and lifecycle governance. The review also discusses the concept of an AI Safety Integrity Level (SIL) and outlines future research priorities for benchmark datasets, incident learning, rare‐event testing and postdeployment surveillance.