Scalable 3D Reconstruction for Immersive Virtual Reality Applications

Abstract

The recent advances in Augmented Reality (AR) and Virtual Reality (VR) technology and their growing popularity in the past years has significantly influenced emerging trends towards more intuitive and user-centric applications that are accessible to a large community. Sharing immersive experiences is the central challenge here and requires the accurate presentation of virtual content to the user which not only depends on several technical aspects of the respective display devices but also on the visualized scene elements. However, many applications are particularly designed for immediate scenarios which cannot entirely rely on pre-generated content and, thereby, require on-the-fly acquisition of the unknown surrounding scene environment in an efficient and progressive way.<br> In this thesis, we developed and investigated techniques for real-time reconstruction of 3D scene geometry and appearance with a specific focus on practical methods and systems that could be used in AR and VR scenarios. To this end, we present several contributions which can be categorized into three major areas of these systems. First, we studied the <em>L</em>₁-based Locally Optimal Projection (LOP) operator in the context of data prefiltering and introduced a family of generalized operators in which each element corresponds to a localized <em>L</em>_<em>p</em> estimator. Furthermore, we revealed their close relation to the Mean Shift framework and derived various theoretical properties of the respective kernels and, in turn, the projection operator. We applied the gained insights to define an improved density weighting scheme, a more accurate kernel approximation for the continuous projection operator, as well as a set of robust loss functions which correspond to the kernels. Secondly, we developed a practical multi-client live telepresence system which enables streaming live-captured 3D scene data to remotely connected users who can independently explore and interact with the scene. We introduced a bandwidth-efficient volumetric data structure based on Marching Cubes indices as well as fast GPU hash data structures to efficiently maintain and progressively stream the reconstructed model with only moderate network requirements. In subsequent work, we further improved the overall performance and scalability of our telepresence system by introducing several algorithmic improvements to the reconstruction component. Thirdly, we investigated a segmentation-based approach for estimating appearance information in terms of the spatially-varying surface albedo from RGB-D and additional infrared (IR) input data. In addition to an improved formulation of the coupling between the color and infrared channels, we also incorporated temporal information from previous frames to accelerate the Total Variation-based optimization process.