A Multi-Modal Deep Learning Framework for High-Resolution Alpine Land Use/Land Cover

Accurate Land Use and Land Cover (LULC) mapping in high-resolution alpine environments is challenging due to complex terrain, heterogeneous vegetation, seasonal snow and ice cover, and the limited spectral information provided by conventional aerial imagery. Although foundation models such as the Segment Anything Model (SAM) effectively capture structural features, their class-agnostic design, limits fine-grained semantic discrimination and typically requires large annotated datasets. This study proposes a multi-modal deep learning framework for alpine LULC mapping using sparse annotations, which would fall under the category of weakly supervised learning. The framework employs a dual-encoder architecture that integrates RGB imagery, six-band multispectral imagery, and custom adapters for spectral indices, and Digital Surface Models (DSMs). A SAM-based encoder extracts geometric and contextual features from RGB imagery, while a dedicated encoder learns complementary spectral representations from multispectral data. To address the boundary uncertainty introduced by sparse supervision, we propose post inference hybrid refinement strategy that combines a Canopy Height Model (CHM) derived from the DSM to improve tree crown delineation with edge-based refinement for low vegetation classes, such as shrubs, and mathematical methods to refine road and building edge delineation with DSMs. Experimental results across fourteen alpine classes highlight that the framework achieves a validation mIoU of 0.777, macro-averaged over the fourteen classes. For the present sensor choice and classification scheme, no comparable multi-modal baseline exists. Thus, results are reported in absolute terms. The present multi-modal data fusion sets the baseline for future scalable alpine LULC mapping applications.