Integration of Variational Methods and Deep Learning for Image Processing: A Task-Oriented Review

Variational methods and deep learning represent two distinct paradigms that have each shaped the development of modern image processing. Variational formulations are grounded in carefully designed energy functionals and partial differential equations (PDEs) that encode prior knowledge about image structure; they offer interpretability, theoretical guarantees, and strong performance. Deep learning, by contrast, learns image priors implicitly from data through highly expressive parametric models and has achieved impressive improvements across a wide range of tasks. Over roughly the past decade, a growing body of work has sought to integrate the two paradigms, combining the mathematical rigor of variational models with the representational power of deep networks. This article provides a task-oriented review of this integration. We survey the principal mathematical frameworks—total variation (TV) and its generalisations, the Mumford–Shah functional, Euler’s elastica, level set methods, and nonlinear diffusion PDEs—and describe how each has been coupled with deep architectures. We then organise the literature by tasks, covering image denoising, segmentation, reconstruction, super-resolution, inpainting, and deblurring. For each task, we survey major learning paradigms and representative deep learning integration frameworks—centred on variational loss formulation, algorithm unrolling, PDE-driven architectures, plug-and-play priors, deep image prior, and score-based generative modelling—highlighting the synergy between mathematically rigorous variational principles and empirical model performance. We further establish theoretical links: residual networks to ordinary differential equations, and score-based diffusion models to stochastic PDEs. Standard benchmarks and metrics are summarised, followed by open challenges such as cross-domain generalisation, robustness, interpretability, hallucination control, and data-efficient learning. Finally, conclusions are drawn based on the available evidence, and the maturity of each research direction is explicitly discussed. This review differs from existing surveys in three respects. It classifies methods by the locus at which the variational and the learned component are coupled—the training objective, the network architecture, or the energy minimised at inference—rather than by task or by assumption on the forward operator. It applies that single classification uniformly across six tasks, including segmentation and inpainting, which carry no explicit forward operator and therefore fall outside the scope of existing surveys of learned inverse problems. Moreover, it uses the resulting coverage map, audited entry by entry against a formal membership criterion, to distinguish combinations that have been realised from those that remain unexplored, those the taxonomy excludes, and those connected only loosely.