The Sky's the Limit: Relightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibility

We address the problem of outdoor scene relighting given a set of multi-view images with varying illumination. In such settings, the illumination is dominated by the sun and the sky. We make two observations: first, the sky is visible in the scene, providing a direct cue to estimate illumination; second, the sun is distant, providing a near-parallel light source whose shadows can be modeled efficiently. We propose NeuSky, which uses a neural field to represent outdoor scenes, decomposing them into material properties (albedo), geometry (normals, depth), and illumination (an environment map per image). Our key insight is a sky-pixel loss that constrains estimated environment maps using the actual sky colors observed in the training images. For visibility, we propose an outside-in approach using a directional distance field that is queried from the surface point looking outward toward the light source. This is in contrast to conventional inside-out approaches that trace rays from the light toward the surface. We show that our approach enables photorealistic relighting of outdoor scenes captured under varying illumination conditions.

From a collection of outdoor photos taken under changing weather and time of day, NeuSky recovers scene geometry, albedo, per-image illumination and sky visibility. Illumination and visibility are both spherical neural fields, so the sky pixels in each photo constrain the lighting directly, and cast shadows are modelled instead of being baked into the albedo.

The figures here come from my PhD thesis, which extends NeuSky with a new synthetic benchmark.

The scene volume sits inside two spherical fields: a directional distance field at radius 1 that gives sky visibility, and a RENI++ illumination prior at infinity. Rays that land on sky pixels supervise the illumination directly. Visibility is sampled only at the surface and shared along each ray, which keeps it cheap.

Visibility comes from a directional distance field queried outside-in, from the surface point towards the light. Averaging it over directions gives ambient occlusion; evaluating it in a single direction gives a sharp cast shadow.

Decompositions of training views from two NeRF-OSR sites: the photo, NeuSky's reconstruction with its fitted environment inset, the albedo and the normals.

Albedo and normals against NeRF-OSR, FEGR and SOL-NeRF. NeRF-OSR is rendered from its released checkpoints. FEGR and SOL-NeRF released no code or models, so their panels are reproduced from their papers.

The same decomposition and relighting on the Trevi Fountain.

Illumination is a latent code, so a reconstructed scene can be relit under any environment the prior can represent.

Real photo collections have no ground truth for albedo, normals or illumination, so the thesis adds a synthetic benchmark. NeuSky Synthetic has five outdoor scenes rendered in Blender under 167 HDR environments: 1,500 views at 1920×1080, with ground-truth albedo, normals, depth and materials for every validation and test view. It is available on Hugging Face.

rgbalbedonormalsdepth

On each scene NeuSky is compared with NeRF-OSR and GS-IR on rendering, decomposition and illumination estimation. NeuSky's albedo and recovered sky stay close to the ground truth. NeRF-OSR's second-order spherical harmonics can only represent a smooth sky, and GS-IR bakes one environment for the whole scene, so it recovers neither a sun nor sky structure.

Visibility trains together with the scene, so shadow gradients can shape geometry that no camera saw. Rendering Site 1 from behind every training camera shows the effect: with a stop gradient that geometry is missing, and without one NeuSky builds geometry that explains the observed shadows.

Without the visibility network, shadows bake into the albedo. On NeRF-OSR Site 3, where many photos are shaded, NeuSky removes shadows from the ground, doors and facade.

• neusky-modelsCheckpoints for the three NeRF-OSR scenes and the five synthetic scenes.
• reni-modelsThe RENI++ illumination prior every NeuSky model uses (the neusky-prior group).
• neusky-syntheticThe synthetic benchmark: renders, ground-truth layers and camera and lighting metadata.
• neusky-nerfosr-overlaySegmentation masks, point clouds and environment-map rotations to add to an official NeRF-OSR download.