Towards LiDAR-based Spatio-temporal Scene Understanding for Autonomous Vehicles

Abstract

Self-driving cars are expected to reduce the number of casualties caused by traffic accidents, since a machine is always attentive, it can exploit various input modalities, and it always obeys the traffic rules. Liberating people from driving a vehicle will also enable them to do more pleasant activities while getting from one place to another. A fleet of self-driving cars could also lead to less parked cars in the city as cars could efficiently shared and be available on-demand. All these prospects of self-driving cars led to an increasing activity in this area of research and many large automotive companies invested substantially in the research and development of self-driving cars. <br/>

A central aspect of self-driving cars is perception to make sense of the different sensory inputs available. Most self-driving car prototypes rely on a combination of different sensors, such as cameras and 3D LiDAR sensors. In particular, 3D LiDAR sensors provide accurate and dense depth measurements of the environment. Since the advent of fast 3D LiDAR sensors that can produce millions of measurements of the 360° field-of-view, research on 3D LiDAR-based perception attracted increasing attention in the recent years. <br/>

In this habilitation thesis, we present our contributions in the area of 3D LiDAR-based perception. We cover our work on 3D LiDAR-based spatial perception to enable an autonomous system to localize itself in the environment. We present our approaches for Simultaneous Localization and Mapping (SLAM) for building maps on-the-fly, localization using existing maps, mapping to generate detailed maps, and map compression to efficiently transfer mapping data. <br/>

Furthermore, we cover our approaches for semantic interpretation of a single 3D LiDAR scan. All the presented work in semantic perception is based on our dataset, SemanticKITTI, that provides the data needed to train machine learning approaches for semantic interpretation. Furthermore, we present our work on semantic segmentation and panoptic segmentation. Additionally, we present our approach to reduce the need for labeled data. <br/>

Lastly, we cover also our work on unifying spatial and semantic interpretation in the area of spatio-temporal interpretation. In this part, we present our approach for moving object segmentation using a sequence of 3D LiDAR scans. We present our approach for semantic SLAM that use semantic information to improve pose estimation. Lastly, we present our work on panoptic segmentation on a sequence of 3D LiDAR scans that provides spatio-temporal interpretation.