Development of Hybrid Deep Learning Systems Combining Convolutional Neural Networks and Vision Transformers for Medical Ultrasound Image Classification

Background: Accurate classification of thyroid nodule ultrasound images is clinically significant. This study evaluated whether hybrid architectures combining the local feature extraction of convolutional neural networks (CNNs) with the global context modelling of vision transformers (ViTs) outperform each category in isolation. Methods: Two public datasets (6905 images; 3331 benign, 3574 malignant) were used. Eleven ImageNet-pretrained architectures were compared: CNN-based (ResNet-50, EfficientNet-B4, ConvNeXt), ViT-based (Swin Transformer, DeiT, PVTv2, XCiT) and hybrid (ConViT, CoaT, MobileViT, HuviT). Backbones were frozen and only the classification head was trained; HuviT’s new projection and encoder remained frozen at random initialisation. Wilson and Newcombe 95% confidence intervals (CIs) were computed; explainability used gradient-weighted class activation mapping (Grad-CAM). Results: ConViT achieved the highest accuracy (83.30%; 95% CI 80.91–85.45) and an area under the receiver operating characteristic curve (ROC-AUC) of 89.94% (95% CI 87.8–91.7). ConvNeXt ranked second (82.63%) but had the highest ROC-AUC (90.02%; 95% CI 87.9–91.8); the intervals overlapped. HuviT reached 76.35%. Grad-CAM for a representative case highlighted lesion areas. Conclusions: While ConViT achieved the highest performance on threshold-based metrics, these findings suggest the potential of specific hybrid designs rather than establishing general superiority of hybrid architectures. Given the retrospective design and the partial association between image source and class label, prospective external validation is warranted.