Multimodal Deep Learning based Automatic Modulation Recognition: Fusion of Signal Modalities

Automatic modulation recognition (AMR) has been becoming an indispensable part in intelligent communications systems, especially for cognitive radio and radio regulation. With the fast development of machine learning in the recent years, deep learning (DL) has been applied to AMR. However, existing DL based AMR methods only rely on a single signal modality, which limits the full utilization of signal features and restricts recognition performance. Thus, this paper proposes to develop multimodal DL based AMR, which is established, in particular, by exploring a variety of modalities to represent the received signals (e.g., in-phase and quadrature sequences, constellation diagram), and then fusing two or more of the modalities at three different stages of the DL architecture. The three stages of the multimodal fusion include the early fusion, intermediate fusion and late fusion in which the multiple signal modalities are fused before, in between and after the DL model, respectively. The algorithms for the three fusion methods are proposed and implemented in experiments. Evaluation of the algorithms is made according to accuracy, complexity and flexibility. The results show that (1) the early fusion exhibits satisfactory classification accuracy, least complexity and not good flexibility; (2) the intermediate fusion gives best accuracy, high complexity and satisfactory classification flexibility; and (3) the late fusion gives low accuracy, high complexity and best flexibility. Moreover, the proposed multimodal DL based AMR algorithms consistently outperform single signal modality approaches under tested conditions, including different DL model structures, sample quantities, and channel models, demonstrating strong generality and universal superiority for automatic modulation recognition.