CO-YOLO: An Improved YOLOv11 Detector for Camellia oleifera Fruit Detection Under Complex Orchard Conditions

Accurate detection of Camellia oleifera fruits in natural orchards is essential for automated harvesting and yield-related phenotyping but is challenged by small fruit size, dense distribution, foliage occlusion, and fruit–leaf similarity. To address these challenges, we developed CO-YOLO, an enhanced YOLOv11-based detector that integrates a spatial attention module (SAM) and coordinate attention (CA) to jointly enhance spatial discriminative feature extraction and positional encoding. Unlike approaches that employ attention mechanisms primarily to reweight features, CO-YOLO combines spatial attention with coordinate-aware feature representation to improve the identification and localization of partially occluded fruits while retaining the YOLOv11 detection framework. A field dataset of 674 raw RGB images was collected under different fruit developmental stages, weather conditions, illumination environments, and occlusion levels. Compared with YOLOv11, CO-YOLO improved precision, recall, mAP@0.50, mAP@0.75, mAP@0.50:0.95, and F1-score by 2.46, 3.17, 2.96, 3.64, 3.42, and 2.56 percentage points, respectively, with greater improvements under stringent localization criteria. Ablation experiments confirmed the complementary contributions of SAM and CA, while environmental evaluation showed that immature fruits, rainy conditions, and dense occlusion remained the main sources of performance degradation. Fruit-counting analysis further indicated improved consistency between automated and manual counts, although dense fruit clusters continued to introduce counting errors. Overall, CO-YOLO improves the detection of Camellia oleifera fruit under complex orchard conditions while preserving the efficiency of the YOLOv11 framework, and thus offers a practical visual perception approach for automated harvesting and yield-related phenotyping.