RENI++: A Rotation-Equivariant, Scale-Invariant, Natural Illumination Prior

abstract

Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. This results in limitations for the inverse setting in terms of the expressivity of the illumination conditions, especially when taking specular reflections into account.

We propose a conditional neural field representation based on a variational auto-decoder and a transformer decoder. We extend Vector Neurons to build equivariance directly into our architecture, and leveraging insights from depth estimation through a scale-invariant loss function, we enable the accurate representation of High Dynamic Range (HDR) images. The result is a compact, rotation-equivariant HDR neural illumination model capable of capturing complex, high-frequency features in natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations.

overview

RENI++ is a prior over natural outdoor illumination. Each environment map is a latent code, decoded by a neural field on the sphere. Interpolating between fitted codes gives plausible environments, and rotating a code about the vertical axis rotates the decoded illumination by the same amount.

The figures here come from my PhD thesis. Most show RENI++ (thesis), a later version that predicts two exposure brackets under a fixed exposure gauge and is trained with latent resets. The model in the TPAMI paper, which predicts log radiance with a scale-invariant loss, is labelled published RENI++.

Environment maps decoded from interpolated latent codes, and a latent code rotated about the vertical axis — Interpolating between fitted latents (left) and rotating a latent about the vertical axis (right)

architecture

The query direction and the Vector Neuron latent code are split into parts along and across the gravity axis. The decoder only sees quantities that stay fixed when the direction and the latent rotate together about that axis, so an ordinary attention decoder still produces a rotation-equivariant field.

reconstruction

Fitting environments unseen in training, with the same number of parameters for every method. RENI++ keeps more high-frequency detail than spherical harmonics (SH) and spherical Gaussians (SG) and reproduces a wider HDR range. The heat maps show log luminance.

Against outdoor lighting models, RENI++ preserves cloud structure, atmospheric colour gradients and the sun's position. SOLD-Net reconstructions are both smooth and noisy, and the parametric Hosek-Wilkie sky cannot represent clouds at all.

Environment maps reconstructed by RENI++, SOLD-Net and the Hosek-Wilkie sky model — RENI++ against SOLD-Net and the Hosek-Wilkie sky model

completion

Given only a clipped LDR view of 90° by 60°, RENI++ completes the full HDR environment. Gaussian priors on SH and SG fill the gap with a blurred mean sky or spurious bright lobes, while RENI++ completes plausible scene structure with the sun near its true direction.

latent space

Walking between fitted latent codes passes only through plausible environments, which is what makes RENI++ useful as a prior. The bottom row interpolates between two held-out environments, with the decoded endpoints at either end.

Three rows of environment maps interpolated between latent codes — Interpolations between training latents (rows 1 and 2) and two held-out environments (row 3)

inverse rendering

Lighting is recovered by fitting only the latent code through a differentiable renderer, with geometry, camera and material held fixed. As the specular term k_s rises from left to right, RENI++ fits better than SH of the same size at every step. It keeps the sun position and sky structure where SH collapses to a single lobe.

Environments recovered by RENI++ and SH from renders of a teapot and a bunny with increasing specularity — Fit PSNR on each panel; both models use D_z = 300

A recovered environment can be used directly for rendering. This scene is lit by a ground-truth environment map and by its published RENI++ reconstruction.

Car rendered in Blender under the ground-truth environment — Ground truth

Car rendered in Blender under the RENI++ reconstruction — Published RENI++

ablation

The original RENI conditioned a SIREN decoder by concatenating the latent code with its input. RENI++ conditions an attention decoder instead. At matched model sizes the attention decoder recovers much more high-frequency detail.

SIREN and attention decoders at three model sizes against the ground truth — SIREN (RENI) and attention (RENI++) decoders at D_z = 27, 147 and 300

data & weights

RENI HDR contains 1,704 outdoor HDR environment maps, all released under CC0. They were curated by hand to remove images containing people's identifiable information, offensive content or mostly artificial light, and are split into 1,673 training, 10 validation and 21 test maps.

reni-hdrThe environment maps, with the fixed masks used for completion.
reni-modelsCheckpoints for published RENI++ and its baselines, the thesis model, and the prior used by NeuSky.

citation

@article{gardner2026reni,
  author  = {James A. D. Gardner and Bernhard Egger and William A. P. Smith},
  title   = {{RENI++}: A Rotation-Equivariant, Scale-Invariant, Natural Illumination Prior},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year    = {2026},
  volume  = {48},
  number  = {9},
  pages   = {11094--11110},
  doi     = {10.1109/tpami.2026.3691593}
}