AIS-Based Abnormal Ship Behavior Detection for Sustainable Maritime Traffic Management Using a Dual-Error Fusion LSTM–Transformer Framework

Abnormal ship behavior detection is important for maritime traffic surveillance, navigation safety, and risk prevention. However, existing methods often depend on handcrafted features or a single reconstruction or prediction signal, which limits their ability to detect both sustained trajectory abnormalities and abrupt vessel movement changes. This paper proposes a Dual-Error Fusion LSTM–Transformer framework, referred to as DEFLT, for AIS-based abnormal ship behavior detection. A motion-aware vessel representation was first constructed by combining the geographical position, speed over ground, course over ground, and their temporal variations. An LSTM autoencoder reconstructs historical trajectory windows, while a Transformer prediction module estimates subsequent vessel states. The standardized reconstruction and prediction errors are fused into a unified anomaly score to capture complementary evidence from historical trajectory inconsistency and unexpected future motion. Experiments were conducted using real-world AIS data collected during September 2019 from four representative Danish waters. The study considers four abnormal behaviors: speed anomalies, course anomalies, loitering, and route deviations. Compared with KNN, LOF, Isolation Forest, Random Forest, the LSTM-AE, and the Transformer, DEFLT achieves F1-scores of 0.96, 0.97, 0.88, and 0.93 across the four study areas. For type-specific detection, the Macro-F1 values range from 0.61 to 0.86, while Macro-Recall remains between 0.88 and 0.96. Friedman and post hoc Wilcoxon signed-rank tests further demonstrate that DEFLT provides a significant and consistent improvement over all baseline methods. These results verify the effectiveness of dual-error fusion for detecting heterogeneous abnormal ship behaviors from AIS trajectories. In operational settings, DEFLT can serve as an alert-prioritization tool for vessel traffic services and port authorities by directing attention to atypical trajectories that require timely review, thereby supporting safer and more resource-efficient maritime traffic coordination.