Hybrid LSTM–Transformer- and PPO–DQN-Based Intelligent Congestion-Aware Traffic Engineering in Software-Defined Networking

The rapid growth of cloud computing, multimedia services, Internet of Things applications, and data-intensive communication has increased the demand for adaptive and congestion-aware network management. This study developed a Software-Defined Networking (SDN) traffic-engineering framework integrating TShark-based packet capture, Hybrid LSTM–Transformer congestion prediction, and reinforcement-learning-based routing using DQN, PPO, and Hybrid PPO–DQN. Mininet and Open vSwitch were used to emulate controlled and external network topologies, while Iperf generated normal, moderate-congestion, and heavy-congestion traffic. Packet traces were captured as PCAP files and transformed into structured time-series data for congestion classification and routing evaluation. The Hybrid LSTM–Transformer achieved 99.99% accuracy under the controlled Mininet-generated test distribution. Repeated-run differences were small, and component ablation found a statistically insignificant performance degradation, so the controlled result does not establish inherent hybrid-model superiority. Under independently generated harder traffic, accuracy decreased to 46.52% with a macro F1-score of 43.91%, demonstrating a substantial performance degradation and sensitivity to traffic-distribution shift beyond the controlled traffic conditions. The frozen predictor achieved 100% accuracy on the Abilene-inspired dataset and 99.20% pooled accuracy on the exact AARNet dataset. To remove state-input confounding, DQN, PPO, and Hybrid PPO–DQN were independently retrained across five seeds with identical observed-state and frozen-predictor-assisted inputs. Under predictor assistance, DQN achieved 774.0476±0.1671 mean reward versus 760.6236±18.5135 for Hybrid PPO–DQN, without a statistically significant difference after Holm correction. Replacing the observed congestion state with the frozen predictor output did not significantly change the reward, throughput, delay, packet loss, PDR, or CRR for any agent. DQN required fewer parameters and lower local inference latency than Hybrid PPO–DQN. Five-seed PPO retraining showed seed-dependent action concentration rather than consistent selection of the high-throughput action, indicating limited policy diversity under the tested configuration. Overall, the findings support the feasibility of prediction-assisted traffic engineering while emphasizing architectural complexity versus practical utility rather than inherent hybrid superiority. Broader traffic validation and real-time SDN-controller integration remain necessary.