Depth–Normals Fusion with Albedo‐ and Image‐Stack‐Conditioned Self‐Correction of Photometric Stereo Gradients

ABSTRACT

We address depth–normals fusion in cases where the input structured‐light (SL) depth map contains holes (regions with missing or unreliable depth information) and the photometric stereo (PS) normals are affected by distortions caused by imperfect modelling and calibration. Since the reconstruction inside depth holes is driven by gradients derived from the PS normals, PS errors directly degrade the recovered depth. Building on our previous joint fusion–correction approach [1], we introduce a correction of the PS‐derived gradients that utilizes not only the reliable SL depth, but also additional photometric attributes extracted from the raw PS image stack, such as albedo and intensity statistics. These attributes multiplicatively modulate the polynomial correction model, making the effective correction dependent on the local photometric properties. The resulting joint fusion–correction scheme alternates between a convex total generalized variation (TGV) fusion step and a least‐squares correction step fitted over reliable‐depth regions. We evaluate the proposed photometric‐conditioned joint TGV fusion–correction approach on both real‐world data from our own sensor system and data derived from the DiLiGenT‐MV dataset.