Light Source Estimation via Intrinsic Decomposition for Novel View Synthesis

Abstract

In this thesis, we propose a method for light source estimation via intrinsic decomposition within a Neural Radiance Field-like setting. We begin by exploring a learned method to recover light sources from intrinsic components - reflectance, shading, and surface normals. As demonstrated, the learned mapping is unable to properly learn the relationship between the different intrinsic components, and instead overfits some components while ignoring the rest. However, inference of local SH lighting, while somewhat spatially discontinuous, shows promise for future extensions. For these reasons, we later pivot our approach to directly optimize global spherical harmonics lighting, in which the in-scene radiance is represented by coefficients of a single global basis function. With this optimization-based approach, we successfully reconstruct a coarse estimate of the lighting of our scenes. To validate and support our method, we introduce a novel synthetic dataset, which consists of intrinsic components of several scenes. For each scene in our dataset, we provide geometry, shading, reflectance, and fully diffuse and rendered passes. We hope that extensions of our work could serve as modules of future NeRF models, enabling the joint optimization and refinement of lighting, geometry, and radiance.