DOI: 10.64808/engineeringperspective.1965542 ISSN: 2757-9077

Reliability and Calibration Analysis of Deep Learning Models for Aircraft Recognition in Remote-Sensing Imagery

Nilgün Şengöz
Show PDF Cite
Aircraft recognition in remote-sensing imagery is usually reported through detection or classification accuracy. Accuracy alone says nothing about how confident a model is when it is wrong, or how much of an apparent architectural advantage comes from the training budget rather than from the architecture. This study reports a reliability and calibration analysis on the MAR20 benchmark under a complete two-factor design. YOLOv8n and YOLOv8s were trained for aircraft detection, and MobileNetV3-Large, EfficientNet-B0 and ResNet50 were each trained under two training protocols across five fixed seeds, giving thirty classification runs. Classification is evaluated on ground-truth crops and therefore describes an oracle-crop setting rather than an end-to-end pipeline. The effect of the training protocol proved to depend on the architecture and to be of the same order as the differences between the architectures themselves: the longer schedule improved ResNet50 and MobileNetV3-Large on every seed, while EfficientNet-B0 was not separated from its own baseline. Once the protocol was held fixed, EfficientNet-B0 and ResNet50 both reached approximately 0.98 top-1 accuracy and could not be separated by any paired test at either the sample level or the seed level, although both separated from MobileNetV3-Large. Model ranking also depended on the calibration estimator, because the highest equal-width bin holds over 90 percent of the test crops: after temperature scaling, equal-width expected calibration error favored EfficientNet-B0 and equal-mass expected calibration error favored ResNet50. Temperature scaling left accuracy unchanged in all thirty runs while reducing wrong predictions issued above 0.95 confidence by between a quarter and a half. Code, configurations and per-run outputs are archived.

More from our Archive