Hybrid Graph Neural Network–Reinforcement Learning Framework for Intelligent Programmable Network Automation

ABSTRACT

Software‐Defined Networking (SDN) and Network Function Virtualisation (NFV) offer the benefit of dynamic control and flexible resources that can be managed with ease. However, the growing complexity of network topologies and heterogeneous traffic patterns, combined with high‐quality‐of‐service (QoS) demands, requires innovative routing schemes and existing AI‐based optimisation procedures. Most existing methods model network structures using Graph Neural Networks (GNNs) or implement Reinforcement Learning (RL) to route traffic adaptively, but rarely combine the two in a single, more representative framework. This paper proposes a hybrid GNN‐RL system coupled with a cloud‐based automated experimentation system to optimise topology‐aware, multi‐objective, and scalable programs in the network. Graph representations were created based on network flows of two heterogeneous datasets, namely NetBench and SDNFLow and a synchronised hybrid model. A normalised, multi‐objective reward function that includes throughput, latency, packet loss, and a congestion penalty was proposed to address the reward imbalance, a frequent issue in RL routing. The RL component is trained and evaluated within the abstracted simulated environment and policy‐derived QoS metrics are generated by routing actions. The controlled environment enables reproducible evaluation of the hybrid‐GNN framework. Experimental analysis indicates that the optimised framework is more effective at QoS than standalone RL and GNN baselines. Latency was reduced by more than 70.7 ms in the original experiments to 17.7 ms and further to 9.6 ms with reward optimisation. Packet loss dropped by 6.3% to 1.49%, and throughput remained steady at 151–231 Mbps, improving over the baseline (85–88 Mbps). The coordinated training scheme enhanced convergence rate and routing stability in dynamic traffic environments. These results affirm that the combination of structural learning, adaptive decision‐making, and automated evaluation on a cloud platform offers a realisable, scalable route to intelligent, autonomous, programmable network management that can be used in next‐generation communication infrastructures.