An attention-based multimodal hybrid deep learning framework for pneumonia classification using chest X-rays and radiologist-annotated severity labels

Adequate classification of pneumonia is essential for timely disease management and effective resource allocation in healthcare systems. As pneumonia presents with varying severity and its assessment utilizing chest radiographs is challenging, specifically detecting MILD and NORMAL-PCR+ cases, where chest radiographs may show slight fog and haze in the lungs or no visible abnormalities, respectively. To overcome this problem, we developed a robust hybrid framework that utilized both clinical and imaging data for automated assessment of pneumonia and then subdivided it into four categories: NORMAL-PCR+, MILD, MODERATE, and SEVERE. Our hybrid approach utilized two models: MobileNetV3, to extract spatial features from chest radiographs, and encoder-based transformer, Robustly Optimized BERT Pretraining Approach (RoBERTa), to encode severity label semantics which is a textual modality. Attention-based fusion is employed in our work to effectively integrate both modalities, enhancing the model’s ability to learn complex inter-modal dependencies. Effective preprocessing techniques such as Contrast Limited Adaptive Histogram Equalization (CLAHE) and data augmentation were implemented to improve the quality of images and overcome the class imbalance. Experimental results demonstrated that the proposed framework achieved state-of-the-art performance, reaching a final accuracy of 98.74%, precision of 97.96%, recall of 98.7%, F1-score of 98.3% and an Area Under the Receiver Operating Characteristic Curve (AUC-ROC) score of 0.9980. Furthermore, class-wise performance showed 99.3% accuracy for MILD, 97.4% for MODERATE, 99.03% for SEVERE and 99.13% for NORMAL-PCR+, confirming the model’s ability in distinguishing fine-grained severity levels. These findings demonstrate the future of multimodal approaches in advancing automatic and interpretable diagnosis of pneumonia.