Hierarchical Modulation Classification with Channel-Type-Guided Blind Preprocessing for High-Order QAM in Multipath Fading Channels

Automatic modulation classification (AMC) becomes challenging in multipath fading channels, particularly for high-order quadrature amplitude modulation (QAM) signals whose constellation points are strongly distorted by inter-symbol interference, phase rotation, and fading. This paper presents a channel-type-guided hierarchical AMC framework that combines blind preprocessing with deep learning. In the first stage, the received in-phase and quadrature (IQ) signal is downsampled and preprocessed using blind signal processing techniques. Blind source separation (BSS) is used for additive white Gaussian noise (AWGN) and flat fading channels, whereas the constant modulus algorithm (CMA) followed by BSS is used for multipath fading channels. A convolutional neural network (CNN) then performs first-stage modulation classification and generates a QAM-family flag. If the first-stage output corresponds to a QAM-family signal, a second-stage refinement path is activated. In this path, a convolutional denoising autoencoder (CDAE) is applied to the original received signal to mitigate multipath-induced distortion, followed by BSS preprocessing and a dedicated CNN classifier for 16-QAM, 64-QAM, and 256-QAM. Simulation results over AWGN, flat fading, and multipath Rician fading channels show that the proposed hierarchy improves high-order QAM classification in the considered settings, especially for 64-QAM and 256-QAM under multipath fading with stronger time variation. Multi-frame Softmax averaging further improves decision stability. The results support the use of classical blind preprocessing and selective CDAE-based refinement as a practical, complementary front end for AMC in controlled multipath simulation scenarios, while real over-the-air validation and automatic channel-category detection remain future work.