Attention-Enhanced YOLOv26 with Tree-Structured Parzen Estimator Optimization for Robust Dental Surgical Tool Detection

Object detection remains a fundamental challenge in computer vision and plays a pivotal role in safety-critical medical applications, including surgical instrument recognition and operating-room workflow automation. This study presents a comprehensive comparative evaluation of five attention mechanisms—Squeeze-and-Excitation (SE), Convolutional Block Attention Module (CBAM), Efficient Channel Attention (ECA), Simple Attention Module (SimAM), and an enhanced multi-kernel Spatial Pyramid Pooling Fast module (SPPF+)—integrated into the YOLOv26n backbone, together with two neck-level attention variants (ECA-Neck and CBAM-Neck). A total of 16 model configurations were systematically investigated on a 22-class dental surgical instrument detection dataset under both default training settings and hyperparameter configurations optimized using the Optuna Tree-structured Parzen Estimator (TPE), enabling a rigorous full-factorial ablation study. Experimental results demonstrate that TPE-based hyperparameter optimization consistently enhances detection performance across all architectures. Among the evaluated models, CBAM-Opt achieved the highest detection accuracy, attaining an mAP@50 of 0.959 and an F1-score of 0.913, although the margins among the top optimized configurations fall within run-to-run variability. In contrast, Base-Opt delivered the strongest strict-localization capability with an mAP@50–95 of 0.800, highlighting the competitive performance of the baseline architecture when appropriately optimized. Notably, the parameter-free SimAM module exhibited the largest improvement following optimization (ΔmAP@50 = +0.040), indicating a pronounced sensitivity to training configuration. Furthermore, neck-level attention integration achieved performance comparable to backbone-based attention, with ECA-Neck-Opt reaching an mAP@50 of 0.959, suggesting an effective alternative that preserves pretrained feature representations while maintaining high detection accuracy. Beyond performance evaluation, this work provides a unified benchmarking framework for attention mechanisms in medical object detection, accompanied by computational complexity analysis and practical architectural insights. The findings establish evidence-based guidelines for selecting attention modules in resource-aware surgical vision systems and contribute toward the development of more accurate and reliable computer-assisted clinical workflows.