Recognition of Tieguanyin Oolong Tea Fermentation Stages Based on RGB-HSI Bimodal Deep Fusion

Accurate recognition of successive Tieguanyin fermentation stages is essential for consistent processing control; however, adjacent stages often exhibit highly similar appearance characteristics, making fine-grained discrimination difficult. Although RGB images and hyperspectral imaging (HSI) data capture complementary appearance and spatial–spectral information, effectively integrating these heterogeneous representations remains challenging. To address this issue, an RGB–HSI bimodal dataset comprising 695 paired samples was constructed, and a bimodal deep fusion network, termed TeaMFNet, was developed for fine-grained fermentation stage recognition. For feature fusion, a Bidirectional Cross-modal Fusion (BCF) module enabled bidirectional information exchange between sample-level RGB appearance representations and HSI spatial–spectral representations. Building on this fusion, a Residual Feature Alignment (RFA) module refined the BCF-enhanced HSI features through lightweight residual correction, using the RGB representation as a discriminative reference to improve the alignment of heterogeneous features in a shared discriminative space. Furthermore, a Stage Prototype Module (SPM) introduced prototype-based constraints to enhance intra-stage compactness and inter-stage separability. During training, a Modality Confidence Modulation strategy was applied to balance the contributions of the two modalities. Experimental results showed that TeaMFNet achieved an accuracy of 93.28% and a Macro-F1 score of 93.38% on the random sample-level test split, which were numerically higher than those of DMD by 1.14 and 1.91 percentage points, respectively. However, leave-one-source-out evaluation yielded accuracies ranging from 74.10% to 91.45%, indicating reduced performance under source shifts and the need for broader cross-source validation. Ablation studies further confirmed the effectiveness of BCF, RFA, and SPM. Overall, these results demonstrate the potential of RGB–HSI bimodal fusion for distinguishing visually similar Tieguanyin fermentation stages, providing a basis for further investigation of objective fermentation stage assessment and digital monitoring of tea processing.