Software-Only Registration and Cross-Spectral Classification of Unsynchronized RGB–LWIR Video: A Multisensor Benchmark for Conveyor-Based Waste Sorting

Reliable multisensor perception is a key requirement for practical waste sorting, yet many low-cost sensor configurations cannot rely on hardware synchronization or carefully controlled acquisition. We present a pilot-scale multisensor waste-sorting testbed that combines an unsynchronized RGB camera with a long-wave infrared (LWIR) camera for object classification on a continuously moving conveyor, and introduce ThermalRGBTrash, a new paired RGB–LWIR video dataset for this task. To enable fusion under asynchronous acquisition, we develop a fully software-based registration pipeline that combines SuperPoint–SuperGlue matching with an adaptive sliding-window strategy designed to recover from long-wave infrared sensor artifacts, including non-uniformity correction events. Across 281,439 matched frame pairs from 19 paired videos, the registration pipeline achieves a mean spatial alignment error of 2.27 pixels and matches 99.98% of attempted frame pairs. We then detect and segment objects with Mask R-CNN, track them across the conveyor, and classify each tracklet using frozen DINOv2 self-supervised Vision Transformer (ViT-L/14) features with a lightweight multilayer perceptron head. RGB and LWIR representations are combined through late fusion. On 550 tracklets under video-disjoint 10-fold cross-validation, the fused pipeline reaches a macro F1 score of 0.924, outperforming RGB alone (0.886) and LWIR alone (0.856). On a mixed-class test set of 351 tracklets reserved exclusively for final evaluation, fusion reaches a macro F1 score of 0.947. The fusion advantage persists across multiple backbone and pretraining choices, while ablation studies support the chosen temporal sampling and pooling design. These results show that accurate RGB–LWIR object classification is achievable without synchronization hardware, and establish ThermalRGBTrash as a benchmark for future work on practical multisensor perception in conveyor-based waste sorting.