Automated Coral Detection and Instance Segmentation in Underwater Imagery: Evaluating YOLO Model Performance and Transferability to Mesophotic Coral Ecosystems

Automated coral image analysis remains challenging because methods must resolve individual colonies while preserving taxonomic information. We present a YOLO-based instance-segmentation framework trained on more than 9500 hand-annotated images of Caribbean scleractinian corals representing 57 species and 24 genera. Four architectures, YOLOv8x, YOLOv9e, YOLOv11x, and YOLOv12x, were evaluated under consistent training conditions at both species and genus levels. On the held-out test set, species-level mask mAP50 reached 0.789 and genus-level mask mAP50 reached 0.870 (both YOLOv9e), with inference times of 4.1–5.5 ms per image. YOLOv8x, YOLOv9e, and YOLOv11x performed similarly (mask mAP50 within 0.01 of one another), with YOLOv8x and YOLOv11x the fastest, while YOLOv12x showed no improvement despite added architectural complexity. Segmentation performance varied among taxa and growth forms, with compact or visually distinct colonies generally producing stronger masks than branching, encrusting, low-contrast, or substrate-integrated forms. External model transfer was evaluated on independent mesophotic coral ecosystem imagery using published results and expert review. Expert-confirmed precision was higher at genus level than species level and increased with confidence filtering, reaching 82.8% for genus-level and 71.1% for species-level predictions at confidence >0.90. These results indicate that YOLO-based instance segmentation can produce rapid, taxonomically resolved coral observations, but external applications require confidence-based filtering and expert review, particularly for species-level inference in visually complex reef imagery.