This book approaches robotics from a deep learning perspective. Artificial intelligence (AI) has transformed many fields, including robotics. This book shows you how to reimagine decades-old robotics problems as AI problems and is a handbook for solving problems using modern techniques in an era of large foundation models.

The book begins with an introduction to general-purpose robotics, how robots are modeled, and how physical intelligence relates to the movement of building artificial general intelligence, while giving you an overview of the current state of the field, its challenges, and where we are headed. The first half of this book delves into defining what the problems in robotics are, how to frame them as AI problems, and the details of how to solve them using modern AI techniques. First, we look at robot perception and sensing to understand how robots perceive their environment, and discuss convolutional networks and vision transformers to solve robotics problems such as segmentation, classification, and detection in two and three dimensions. The book then details how to apply large language and multimodal models for robotics, and how to adapt them to solve reasoning and robot control. Simulation, localization, and mapping and navigation are framed as deep learning problems and discussed with recent research. Lastly, the first part of this book discusses reinforcement learning and control and how robots learn via trial and error and self-play.

The second part of this book is concerned with applications of robotics in specialized contexts. You will develop full stack knowledge by applying the techniques discussed in the first part to real-world use cases. Individual chapters discuss the details of building robots for self-driving, industrial manipulation, and humanoid robots. For each application, you will learn how to design these systems, the prevalent algorithms in research and industry, and how to assess trade-offs for performance and reliability. The book concludes with thoughts on operations, infrastructure, and safety for data-driven robotics, and outlooks for the future of robotics and machine learning.

In summary, this book offers insights into cutting-edge machine learning techniques applied in robotics, along with the challenges encountered during their implementation and practical strategies for overcoming them.

What You Will Learn

• Explore ML applications in robotics, covering perception, control, localization, planning, and end-to-end learning
• Delve into system design, and algorithmic and hardware considerations for building efficient ML-integrated robotics systems
• Discover robotics applications in self-driving, manufacturing, and humanoids and their practical implementations
• Understand how machine learning and robotics benefit current research and organizations

Alishba Imran is a machine learning and robotics developer focusing on robot learning for manipulation and perception. She is currently conducting research in reinforcement learning and unsupervised learning with Pieter Abbeel at the Berkeley AI Research Lab. Alishba has worked on perception at Cruise, developed simulation object manipulation methods at NVIDIA, and led efforts to reduce prosthetics costs at SJSU. At Hanson Robotics, she improved manipulation in Sophia the Robot, and at Kindred.AI, she contributed to robots that have picked up over 140 million units in production.

Keerthana Gopalakrishnan is a senior research scientist at Google Deepmind, working on robot manipulation and the Gemini project. She was educated at Carnegie Mellon University and Indian Institute of Technology. Her research concerns large language models for robotic planning, scaling visual language models for low level control, and cross-embodiment robot learning.