Learning Perception and Manipulation of Objects in Cluttered Environments

Abstract

Robots are increasingly deployed to operate autonomously in cluttered, everyday environments such as homes, warehouses, or retail spaces. In such settings, robust manipulation requires more than isolated grasping or motion planning, i.e., robots must perceive partial scenes, reason about uncertainty, interact purposefully, and respect physical constraints. To address these challenges, this thesis presents multiple strategies for manipulation using object pushing, semantic scene understanding, and physically grounded pick-and-place reasoning. In particular, we focus on four core research directions that structure the contributions of this thesis: (i) Learning object-centric manipulation strategies from raw interaction without object models. (ii) Building uncertainty-aware semantic representations that support purposeful exploration and action. (iii) Selecting manipulation and perception actions by reasoning over uncertainty in cluttered scenes. (iv) Evaluating and exploiting object stability and grasp feasibility directly from partial geometry without strong priors, to enable safe object placement. First, two learning-based non-prehensile manipulation frameworks are introduced that discover robust pushing strategies around clutter. Then, a manipulation-enhanced semantic mapping system is proposed that models occupancy and semantics probabilistically and actively uses manipulation actions to resolve occlusions. Building on this, an uncertainty-informed action selection framework is presented that uses reinforcement learning to balance exploration and minimally invasive manipulation, improving efficiency and map completeness. Finally, stable object placements are addressed through a generalized placeability metric that jointly evaluates stability, grasp feasibility, and clearance directly from partial point clouds, enabling model-free, physically grounded pick-and-place in real-world environments. The proposed methods are validated through extensive simulation and real-robot experiments using KUKA iiwa and UR5 robotic arms in cluttered scenes. The results show that the individual frameworks achieve robust non-prehensile manipulation, accurate semantic mapping, efficient uncertainty-aware action selection, and safe placement under physical constraints, consistently outperforming classical and recent learning-based baselines. In summary, this thesis presents principled methods that unify perception, action, and physical reasoning. It demonstrates that model-free, uncertainty-aware, and physically grounded manipulation is both feasible and effective, advancing autonomous robotic capabilities toward reliable operation in complex real-world environments.