Social-aware Robot Navigation based on Deep Reinforcement Learning

Abstract

With the increase of applications involving autonomous mobile robots, there is a growing need for them to navigate safely and effectively in environments shared with humans. In recent years, social-aware robot navigation has received a lot of attention as it enables robots to understand and follow human social norms, thereby avoiding potential conflicts and dangers. Although several methods have been proposed for environment modeling and motion planning to guide robot behavior, these methods often tend to ignore social rules and focus mainly on motion control and path planning. To address this issue and enhance robot navigation efficiency and safety in dense crowds, this thesis introduces social-aware robot navigation algorithms based on Deep Reinforcement Learning (DRL) to capture crowd interactions and group motion characteristics. <br /> To this end, we first propose a novel Foresighted Social-Aware Reinforcement Learning (FSRL) framework aimed at enabling mobile robots to achieve collision-free navigation. Due to the sparsity of traditional reward signals, it is difficult for robots to learn effective strategies from complex environments. Such sparse rewards may lead to extremely inefficient learning for the robot, requiring a significant amount of time and attempts to learn useful strategies. To address this problem, we employ reward shaping techniques to provide additional reward signals to guide the robot's learning process. Compared to previous learning-based methods, our approach, supported by rich reward mechanisms, is foresighted. It considers not only the current human-robot interactions to avoid immediate collisions but also estimates upcoming social interactions to maintain an appropriate distance. Additionally, our method introduces efficiency constraints, significantly reducing navigation time. Comparative experiments are conducted to validate the effectiveness and efficiency of our proposed method in more realistic and challenging simulated environments. <br /> To further enhance the generalization performance of the navigation method, we then present a novel deep graph learning architecture based on the attention mechanism. While previous works have demonstrated the effectiveness of using reinforcement learning frameworks to train efficient navigation strategies, their performance deteriorates when crowd configurations change (i.e., become larger or more complex). Therefore, it is crucial to fully understand the complex, dynamic interactions of the crowd in order to bring proactive and foresighted behaviors to robot navigation. Our method utilizes spatial-temporal graphs to augment robot navigation, using spatial graphs to capture current spatial interactions, and integrating with RNNs, temporal graphs employ past trajectory information to infer the future intentions of each agent. The reasoning capability of spatial-temporal graphs enables robots to better understand and interpret the relationships between agents over time and space, thereby making wiser decisions. Compared to previous state-of-the-art methods, our approach exhibits exceptional robustness in safety, efficiency, and generalization across various challenging scenarios. <br /> Lastly, this paper considers the challenge posed by the limited perception range of sensors for robot navigation, which leads to incomplete and uncertain information about the observed environment. To achieve collision avoidance in crowded and partially observable environments, we propose a novel deep reinforcement learning architecture. The architecture combines spatial graphs and attention reasoning to enhance the modeling of relationships among moving robots, static obstacles, and surrounding individuals. In this way, our method significantly outperforms state-of-the-art methods in crowded scenarios with limited robot sensor range, particularly in reducing collisions and improving navigation efficiency. Additionally, the adoption of parallel double deep Q-learning significantly reduces training time. <br /> In conclusion, by employing advanced deep learning techniques and effective model design, this thesis has made significant advancements in robot navigation. It provides valuable insights for real-time navigation and interaction of robots in complex crowd environments.