Intuitive and Effective Interaction with Multi-Robot Systems

Abstract

Many tasks in our everyday lives require us to interact with others. Interaction, or collaboration, allows us to combine our strengths and enables us to achieve more than what an individual could achieve on their own. Similarly, when trying to automate such tasks, it might be impossible or prohibitively expensive to just employ one single robot. But coordinating multiple robots is difficult: From the computational challenges of the large state-space of a multi-robot system to the question of how human input from one operator can be mapped to multiple robots, there are multiple levels to an interaction that might be subtle when interacting with humans, but need to be explicitly addressed when working with robots. The goal of this thesis is to tackle some of the most pertinent of these problems. We first take a look at some challenges related to direct teleoperation of single and multi-robot systems. Using manipulability optimization and a nullspace projection scheme, we improve the ability of multi-arm systems to avoid singularities and follow operator input more swiftly. We present a fast, local optimization scheme as well as a second-order optimization using Newton’s method and compare the performance of both and show how the method performs on a real ABB YuMi bimanual robot. We then extend our system to mobile robots. While mobile robots have many advantages over fixed-base systems, they also introduce additional complexities. We examine the driving dynamics for non-holonomic skid-steering robots and how the manipulator pose and payload influence the steering capabilities. Additionally, we introduce terms to avoid inter-agent collision, tip-over avoidance and the ability to move through highly constrained environments. We deploy our combined method on multiple Clearpath Husky platforms with UR5e manipulators and show how it improves the open-loop performance over traditional control schemes. Finally, we investigate in more detail how users can interact with multi-robot systems. We compare a tablet-based Augmented Reality solution to a HoloLens Mixed Reality headset. Presenting study participants with different touch interactions as well as hand-tracking enabled gestures, we compare their performance on multiple tasks with the goal of guiding a team of tiny, differential drive robots. We compare the objective and subjective performance across the different tasks and show that especially spatial interactions benefit from a 3-dimensional user interface. In conclusion, we explore and investigate the interplay of humans and robots for effective and intuitive interaction with multi-robot systems at multiple levels. We improve the ability of robots to follow user input, present a methodology for trajectory optimization of a highly complex, non-holonomic multi-robot system and investigate intuitive gesture-based interactions with mobile robots. And we deploy our methods on real robots, with the goal of hopefully bringing them to real building sites or warehouses in the future, to improve the workplace of tomorrow.