KPI‐NeRF: Hyperspectral Neural Radiance Fields from a Single Kaleidoscopic Plenoptic Image

Abstract

Plenoptic imaging has excessive sampling requirements associated with the high dimensionality of the desired data. At the same time, this data is characterized by a high redundancy since images, in general, do not drastically change when changing the view point or when moving to neighbouring spectral channels. It is therefore desirable to interpolate data across viewpoints or spectral channels, exploiting this redundancy.

In practice, a convenient way to acquire plenoptic data is via snapshot hyperspectral light field imaging. Such data can be obtained by individually spectrally filtering particular slices of the light field, e.g. individual light field views in the case of Kaleidoscopic Plenoptic Imaging (KPI) [MRK*13], or individual lenslet images in traditional light field cameras. In this work, we focus on Kaleidoscopic Plenoptic Imaging because of the simple optical modification that enables hyperspectral imaging: attaching a macroscopic broadband spectral filter array close to the sensor to acquire differently spectrally filtered light field subviews.

The main computational challenge associated with this configuration is the registration of the spectrally filtered light field subviews to enable an interpolation of the data followed by spectral demultiplexing for every light field pixel. The underlying problem is the motion parallax between the light field views caused by the varying scene depth, an estimate of which is needed to register the subviews. However, classical intensity‐based registration fails in the case of spectrally pre‐filtered subviews.

We therefore resort to an implicit scene representation using a newly developed NeRF variant that enables quantitatively correct spectral light field interpolation. Technically, we introduce a subspace‐representation of the desired scene spectra and estimate their coefficients using a NeRF modification. We successfully decode up to 29 spectral channels for lab and outdoor scenes. Additionally, we present an application of our technique to time‐sequences of snapshot images to acquire full plenoptic data. The temporal consistency is reasonable even though no special attempts at enforcing temporal coherence are made.