Multi-Reference Evaluation of Deep Learning Tumour Segmentation in Multi-Sequence Nasopharyngeal Carcinoma MRI

Background/Objectives: Deep learning segmentation of nasopharyngeal carcinoma is usually reported as one Dice similarity coefficient (DSC) against one reference mask, although multi-sequence MRI is often delineated separately on each sequence. We aimed to quantify how far the choice of reference changes reported performance, and to compare this with the disagreement between radiologists. Methods: We analysed a public dataset of 277 patient records with separately delineated T1-weighted, T2-weighted and contrast-enhanced T1-weighted MRI, 30 of whom were annotated by two radiologists. Tumour volumes were compared without registration. Spatial agreement was measured after rigid registration, with the loss from transferring masks between acquisition grids estimated separately. nnU-Net models were scored against every reference, including the second radiologist’s, in 102 held-out patient records. Results: Without registration, the sequence effect on tumour volume was 10.9%, against a reader effect of 5.7% (ratio 1.9, 95% CI 1.40 to 2.49). After rigid registration, per-sequence references agreed at a median DSC of 0.783, against 0.898 between radiologists; in the double-annotated patient records, the paired gap narrowed only from 0.129 to 0.083 (95% CI 0.063 to 0.111) when grid transfer was matched. One fixed set of predictions scored 0.726, 0.685 and 0.673, depending only on the reference (median per-patient reference spread 0.075), and switching the reference sequence moved scores 2.7 to 3.6 times as much as switching the radiologist. Conclusions: Part of the cross-sequence difference is a difference in what each sequence makes visible rather than annotation error, so a single-reference DSC on multi-sequence data depends on a rarely stated choice. Performance should be reported against every available reference and read against measured inter-reader agreement.