Towards Practical Real-time Reflectance Estimation

Abstract

Virtual experiences are becoming increasingly popular, primarily due to tremendous progress in Virtual Reality (VR) and Augmented Reality (AR) technologies improving immersion and making related devices more affordable. These immersive experiences frequently rely on the detailed and accurate reconstruction of real-world objects or people. While some applications can utilize assets captured offline in highly calibrated environments, others depend on real-time online scene reconstruction. <br /> A complete reconstruction consists of information regarding geometry, illumination, and reflectance properties. Especially capturing the reflectance characteristics of real-world scenes is very challenging as it relies on the disentangling of intrinsic scene properties based on appearance samples. While a dense sampling and consecutive fitting of reflectance models may be feasible in an offline setting, this is not the case for applications depending on real-time reflectance estimation, as they usually impose additional constraints preventing a structured and controlled capturing process. <br /> To this end, we identify two main challenges for the field of reflectance estimation in this thesis, which must be overcome to build practical real-time reflectance estimation pipelines: strong time constraints and sparsity of appearance samples. As part of the thesis, we present three previously published projects to address these: First, we propose a complete real-time reflectance estimation pipeline efficiently implemented on the GPU and leveraging deep learning techniques. Afterward, we explore denoising to restore reflectance estimates corrupted by noise-like artifacts due to the mentioned challenges. Finally, a novel lightweight edge and boundary detection approach is proposed to improve scene understanding and provide additional helpful information to reflectance estimation pipelines.