Scalable 3D Reconstruction for Immersive Virtual Reality Applications
Scalable 3D Reconstruction for Immersive Virtual Reality Applications
View/Type of Scholarly Publication
DissertationDate of Exam
14.12.2022Date of Publication
19.12.2022Advisor
Klein, ReinhardCo-Referee
Hullin, Matthias B.Metadata
Show full item record
Citable Links 
Abstract
The recent advances in Augmented Reality (AR) and Virtual Reality (VR) technology and their growing popularity in the past years has significantly influenced emerging trends towards more intuitive and user-centric applications that are accessible to a large community. Sharing immersive experiences is the central challenge here and requires the accurate presentation of virtual content to the user which not only depends on several technical aspects of the respective display devices but also on the visualized scene elements. However, many applications are particularly designed for immediate scenarios which cannot entirely rely on pre-generated content and, thereby, require on-the-fly acquisition of the unknown surrounding scene environment in an efficient and progressive way.
In this thesis, we developed and investigated techniques for real-time reconstruction of 3D scene geometry and appearance with a specific focus on practical methods and systems that could be used in AR and VR scenarios. To this end, we present several contributions which can be categorized into three major areas of these systems. First, we studied the L1-based Locally Optimal Projection (LOP) operator in the context of data prefiltering and introduced a family of generalized operators in which each element corresponds to a localized Lp estimator. Furthermore, we revealed their close relation to the Mean Shift framework and derived various theoretical properties of the respective kernels and, in turn, the projection operator. We applied the gained insights to define an improved density weighting scheme, a more accurate kernel approximation for the continuous projection operator, as well as a set of robust loss functions which correspond to the kernels. Secondly, we developed a practical multi-client live telepresence system which enables streaming live-captured 3D scene data to remotely connected users who can independently explore and interact with the scene. We introduced a bandwidth-efficient volumetric data structure based on Marching Cubes indices as well as fast GPU hash data structures to efficiently maintain and progressively stream the reconstructed model with only moderate network requirements. In subsequent work, we further improved the overall performance and scalability of our telepresence system by introducing several algorithmic improvements to the reconstruction component. Thirdly, we investigated a segmentation-based approach for estimating appearance information in terms of the spatially-varying surface albedo from RGB-D and additional infrared (IR) input data. In addition to an improved formulation of the coupling between the color and infrared channels, we also incorporated temporal information from previous frames to accelerate the Total Variation-based optimization process.
In this thesis, we developed and investigated techniques for real-time reconstruction of 3D scene geometry and appearance with a specific focus on practical methods and systems that could be used in AR and VR scenarios. To this end, we present several contributions which can be categorized into three major areas of these systems. First, we studied the L1-based Locally Optimal Projection (LOP) operator in the context of data prefiltering and introduced a family of generalized operators in which each element corresponds to a localized Lp estimator. Furthermore, we revealed their close relation to the Mean Shift framework and derived various theoretical properties of the respective kernels and, in turn, the projection operator. We applied the gained insights to define an improved density weighting scheme, a more accurate kernel approximation for the continuous projection operator, as well as a set of robust loss functions which correspond to the kernels. Secondly, we developed a practical multi-client live telepresence system which enables streaming live-captured 3D scene data to remotely connected users who can independently explore and interact with the scene. We introduced a bandwidth-efficient volumetric data structure based on Marching Cubes indices as well as fast GPU hash data structures to efficiently maintain and progressively stream the reconstructed model with only moderate network requirements. In subsequent work, we further improved the overall performance and scalability of our telepresence system by introducing several algorithmic improvements to the reconstruction component. Thirdly, we investigated a segmentation-based approach for estimating appearance information in terms of the spatially-varying surface albedo from RGB-D and additional infrared (IR) input data. In addition to an improved formulation of the coupling between the color and infrared channels, we also incorporated temporal information from previous frames to accelerate the Total Variation-based optimization process.
Note
In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of University of Bonn’s products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink.Subjects
3D Reconstruction, Telepresence, Virtual Reality, Appearance Reconstruction, RGB-D, Computer Graphics, Computer Vision
Classification (DDC)
004 InformatikStotko, Patrick: Scalable 3D Reconstruction for Immersive Virtual Reality Applications. - Bonn, 2022. - Dissertation, Rheinische Friedrich-Wilhelms-Universität Bonn.
Online-Ausgabe in bonndoc: https://nbn-resolving.org/urn:nbn:de:hbz:5-69190
Online-Ausgabe in bonndoc: https://nbn-resolving.org/urn:nbn:de:hbz:5-69190
@phdthesis{handle:20.500.11811/10530,
urn: https://nbn-resolving.org/urn:nbn:de:hbz:5-69190,
author = {{Patrick Stotko}},
title = {Scalable 3D Reconstruction for Immersive Virtual Reality Applications},
school = {Rheinische Friedrich-Wilhelms-Universität Bonn},
year = 2022,
month = dec,
note = {The recent advances in Augmented Reality (AR) and Virtual Reality (VR) technology and their growing popularity in the past years has significantly influenced emerging trends towards more intuitive and user-centric applications that are accessible to a large community. Sharing immersive experiences is the central challenge here and requires the accurate presentation of virtual content to the user which not only depends on several technical aspects of the respective display devices but also on the visualized scene elements. However, many applications are particularly designed for immediate scenarios which cannot entirely rely on pre-generated content and, thereby, require on-the-fly acquisition of the unknown surrounding scene environment in an efficient and progressive way.
In this thesis, we developed and investigated techniques for real-time reconstruction of 3D scene geometry and appearance with a specific focus on practical methods and systems that could be used in AR and VR scenarios. To this end, we present several contributions which can be categorized into three major areas of these systems. First, we studied the L1-based Locally Optimal Projection (LOP) operator in the context of data prefiltering and introduced a family of generalized operators in which each element corresponds to a localized Lp estimator. Furthermore, we revealed their close relation to the Mean Shift framework and derived various theoretical properties of the respective kernels and, in turn, the projection operator. We applied the gained insights to define an improved density weighting scheme, a more accurate kernel approximation for the continuous projection operator, as well as a set of robust loss functions which correspond to the kernels. Secondly, we developed a practical multi-client live telepresence system which enables streaming live-captured 3D scene data to remotely connected users who can independently explore and interact with the scene. We introduced a bandwidth-efficient volumetric data structure based on Marching Cubes indices as well as fast GPU hash data structures to efficiently maintain and progressively stream the reconstructed model with only moderate network requirements. In subsequent work, we further improved the overall performance and scalability of our telepresence system by introducing several algorithmic improvements to the reconstruction component. Thirdly, we investigated a segmentation-based approach for estimating appearance information in terms of the spatially-varying surface albedo from RGB-D and additional infrared (IR) input data. In addition to an improved formulation of the coupling between the color and infrared channels, we also incorporated temporal information from previous frames to accelerate the Total Variation-based optimization process.},
url = {https://hdl.handle.net/20.500.11811/10530}
}
urn: https://nbn-resolving.org/urn:nbn:de:hbz:5-69190,
author = {{Patrick Stotko}},
title = {Scalable 3D Reconstruction for Immersive Virtual Reality Applications},
school = {Rheinische Friedrich-Wilhelms-Universität Bonn},
year = 2022,
month = dec,
note = {The recent advances in Augmented Reality (AR) and Virtual Reality (VR) technology and their growing popularity in the past years has significantly influenced emerging trends towards more intuitive and user-centric applications that are accessible to a large community. Sharing immersive experiences is the central challenge here and requires the accurate presentation of virtual content to the user which not only depends on several technical aspects of the respective display devices but also on the visualized scene elements. However, many applications are particularly designed for immediate scenarios which cannot entirely rely on pre-generated content and, thereby, require on-the-fly acquisition of the unknown surrounding scene environment in an efficient and progressive way.
In this thesis, we developed and investigated techniques for real-time reconstruction of 3D scene geometry and appearance with a specific focus on practical methods and systems that could be used in AR and VR scenarios. To this end, we present several contributions which can be categorized into three major areas of these systems. First, we studied the L1-based Locally Optimal Projection (LOP) operator in the context of data prefiltering and introduced a family of generalized operators in which each element corresponds to a localized Lp estimator. Furthermore, we revealed their close relation to the Mean Shift framework and derived various theoretical properties of the respective kernels and, in turn, the projection operator. We applied the gained insights to define an improved density weighting scheme, a more accurate kernel approximation for the continuous projection operator, as well as a set of robust loss functions which correspond to the kernels. Secondly, we developed a practical multi-client live telepresence system which enables streaming live-captured 3D scene data to remotely connected users who can independently explore and interact with the scene. We introduced a bandwidth-efficient volumetric data structure based on Marching Cubes indices as well as fast GPU hash data structures to efficiently maintain and progressively stream the reconstructed model with only moderate network requirements. In subsequent work, we further improved the overall performance and scalability of our telepresence system by introducing several algorithmic improvements to the reconstruction component. Thirdly, we investigated a segmentation-based approach for estimating appearance information in terms of the spatially-varying surface albedo from RGB-D and additional infrared (IR) input data. In addition to an improved formulation of the coupling between the color and infrared channels, we also incorporated temporal information from previous frames to accelerate the Total Variation-based optimization process.},
url = {https://hdl.handle.net/20.500.11811/10530}
}
The following license files are associated with this item:
