Augmented Autonomy: Improving human-robot team performance in Urban Search and Rescue Yashodhan Nevatia, Todor Stoyanov, Ravi Rathnam, Max Pfingsthorn, Stefan Markov, Rares Ambrus and Andreas Birk Abstract— Exploration of unknown environments remains one of the fundamental problems of mobile robotics. It is also a prime example for a task that can benefit significantly from multi-robot teams. We present an integrated system for semiautonomous cooperative exploration, augmented by an intuitive user interface for efficient human supervision and control. In this preliminary study we demonstrate the effectiveness of the system as a whole and the intuitive interface in particular. Congruent with previous findings, results confirm that having a human in the loop improves task performance, especially with larger numbers of robots. Specific to our interface, we find that even untrained operators can efficiently manage a decently sized team of robots.
I. I NTRODUCTION The task of exploring an unknown environment remains one of the fundamental problems of mobile robotics. While many advances have been made in mechatronics, and the development of remotely controlled and autonomous single robot systems, the development of teams containing multiple autonomous robots is still an open research question. Using cooperative robot teams for urban search and rescue (exploration and mapping) [22], space robotics [15], construction [8], or other tasks seems intuitively better than using uncoordinated teams or single robots. Using multiple robots has the obvious advantage of allowing more goals to be pursued simultaneously. Additionally, receiving data from multiple robots improves our confidence in the data, and allows the use of heterogeneous sensors (beyond the capacity of a single robot) for confirmation as well as heterogeneous manipulators. However, until advances in autonomous robots permit effective fully self-governed agents, the human operator cannot be cut out of the loop. Thus, a large number of robots in a team inevitably requires the operator to process more data and issue more commands. Recently, the general trend has shifted from a one-to-one operator robot ratio [9], [20], [19], [24] to one operator supervising and controlling multiple robots [17], [22], [18], [13], which is also our main focus. This paper presents a modular design for the control of a team of semi-autonomous mobile robots tasked with cooperatively exploring an unknown environment. We have made an effort to combine the state of the art in autonomous robotics with an intuitive centralized operator interface. The Robotics Lab, Dept. of EECS, Jacobs University Bremen, D-28725 Bremen, Germany. http://robotics.jacobs-university.de,
[email protected]
design presented here was implemented by the Jacobs University team that participated in the Rescue Virtual Robots Competition and took second place at RoboCup 2007 and first place at German Open 2008. The design was tested on simulated robots in the Urban Search And Rescue Simulation (USARSim) Environment. USARSim is a high fidelity robotics simulator [11], designed primarily to aid the development of controllers for individual robots as well as robot teams. Implemented on top of Unreal Tournament 2004, it uses the high end Unreal Engine 2 and Karma Physics Engine to accurately model the robots, the environment and interactions between the two. In addition to the simulated environment, the group also has experience with using cooperative real robot teams in USAR scenarios (figure 1)
Fig. 1. Left: A Jacobs land robot cooperating with an aerial robot at the European Land Robotics Trials (ELROB) 2007 in Monte Ceneri, Switzerland. In this technology evaluation event, the aerial robot has to search for hazard areas like sites of fire in a forest, which the land robot then has to reach. Right: Two Jacobs robots support first responders at ELROB 2007 in the situation assessment after a simulated terrorist attack with NBC substances at a large public event. The two robots are supervised by only a single operator. The robots can operate fully autonomously and perform a coordinated exploration of the area.
The user interface employed in the system was designed to facilitate continuous supervision of the team as a whole, while still retaining the ability to access specific information and sensor data from individual robots. This seems to be contrary to most older approaches in USAR (e.g. [24]), but has yielded very promising results recently (e.g. [22], [18]). Using a map, the location of each participating robot should be easily perceptible, allowing the interface to scale well with the number of robots as previously reported in [18]. Further intervention is possible, including a tele-operation mode. Experience and lessons learned from previous work on user interfaces [7] was factored into the design. The robots controlled by the user were implemented with the basic tasks of Urban Search and Rescue in mind. The team was to cooperatively explore and map an unknown area with occasional human guidance. Burgard et al [10] proposed
a promising approach for goal assignment in the team. In our system, goal points are frontier cells [23] or user selected points of interest. However, Burgard’s original approach relies on perfect communication between all robots, which is not a feasible assumption in real world scenarios. Thibodeau et al [21] present a straightforward hierarchical way to organize a group of robots without perfect communication, which we have integrated into our design, extending it to allow for dynamic reallocation of the hierarchy. In the rest of this paper we describe and evaluate the presented system. The next section details the design decisions. In section 3, we describe metrics for evaluating the performance of the system and the user interface, presenting the results from a preliminary study. Finally, we provide conclusions and directions for future research in the subsequent section. II. S YSTEM D ESIGN One of the main shortcomings of multi-robot exploration teams is that the operator has to constantly divide his attention between multiple robots. To counteract that, the system operates on two levels of control - a background coordinated autonomy and a foreground human operated system. Thus, if the operator fails to give goal points, the robots do not remain idle, but select goal point autonomously. We will first describe the design of our autonomous agents and then show how the user interface is built on top. A. Autonomy The autonomy of our team is designed with the idea to have it as self-sufficient as possible. To this end, a modular approach is used, shown in figure 2.
Fig. 2. Autonomous Agent Design: Dataflow between the modules of a single agent
Each module is easily replaceable, as long as it provides the same functionality, as described in the following subsections:
1) Interface: The Interface module deals with the lowlevel communication between the robot and the controller, allowing flow of sensor data and actuator commands. The Interface module uses a priority based structure for actuator commands, providing for a subsumption-based control architecture. 2) Obstacle Avoidance: This is a low level module which can assume control over the robot to prevent it from bumping into obstacles or other robots. Obstacle avoidance silently monitors the Laser Range Scanner and Sonars of the robot and only takes action when needed. In such cases it takes control of the actuator interface, drives away from the obstacle and releases control. 3) Motion Planning: The Motion Planning module is one of the crucial modules in our system. It provides an interface using which any other module can set a goal position and moves the robot to that position. For that purpose Harmonic Potential maps are computed. Following the gradient leads the robot to the goal point. The algorithm behind computing the path is based on the theory of Harmonic Potential Functions [12], which are solutions to Laplace’s equation: ∇2 φ =
n X ∂2φ i=1
∂x2i
=0
With a Taylor series approximation of this equation (from [12]), we obtain a discrete version of a harmonic function on a grid. Assuming φ(x, y) is a Harmonic Function, it has the following properties: • φ attains its global maxima on the boundary of its definition space • φ attains its global minima on the boundary of its definition space • The value of φ at any point inside the definition space is equal to the average of the neighboring points • φ is complete - whether or not there is a solution to our problem (i.e. a path from start to goal) will be known The second property ensures the fact that harmonic functions are free of spurious local minima: The function attains its minima only on the boundary of the configuration space, which is one major advantage over other potential fields approaches. The Harmonic Potential map is only computed in a bounding box that contains the robot and the goal. If a good path can not be found, the boundaries are dynamically grown. This map is updated every time the robot travels a certain distance, as the map itself is likely to change during that time. Priorities are also provided for the goal, thus enabling a few levels of control for the higher level modules and goal overriding. 4) Mapping: The Mapping module creates a common map for all connected robots. This is of crucial importance, as the Motion Planning module depends on the existence of a map. In order to reduce network traffic and still maintain a common map, the whole map is not sent over the network for every update. Instead, the latest range scan and the estimated robot pose at the time of the scan are sent. This
accomplishes both goals - it reduces network traffic and also allows for easy replacement of the mapping module. Upon receipt of a range scan, it is simply integrated into the common map. Individual robots use a particle filter based SLAM implementation [16] to perform localization, but do not render this map, thus decreasing computational load. 5) Communication: The next module provides for communication between robots. This module has two main tasks - it builds a spanning tree (”Leader Selection”) and then creates the ad-hoc wireless network (Routing). This module provides a straightforward interface to send, broadcast and receive messages. 6) Coordination: The coordination module is a high-level control module that has two major task - Goal Coordination and Leader Following. Both of these tasks are done in a completely distributed fashion, in order to ensure robustness and scalability. Goal Coordination is done on a ”first come first serve” basis - once a robot determines the best goal that has not been claimed a message to reserve it is broadcasted to all other robots. That ensures no duplication of effort, as multiple robots will not go after the same goal point. The selection of goal points is implemented through frontier exploration, in a similar fashion as the one in [10]. First, all cells that are on the border between unoccupied and unexplored space are marked as frontier cells. Then adjacent frontier cells are grouped and the center of the set is marked as the frontier point. For each frontier point a utility is computed, which is the area of unexplored space that is visible from that point. A cost is also computed using simple wave propagation from the current position to the frontier point. The frontier points are sorted according to their utility − cost value. Keeping in mind the specifics of the USAR domain, potential victim locations are also treated as goal points, superceding the importance of frontier points. The second task of the Coordination module is to provide the Leader Following behavior. On recognizing that the connection to the leader is on the verge of being broken, the goal in the Navigation module is set to the last known position of the leader and a re-computation of the potential map is forced. The priority of this goal is higher then that used to set frontiers, which ensures that there will be no race conditions between exploration and maintaining the leader relations. B. Augmented Autonomy On top of the extensive autonomy system, an operator interface with multiple levels of control was designed. The interaction between the operator and the autonomous features is shown in figure 3. Based on sensor information and the current map from the robot, the operator chooses if and how to interact with the autonomous functionality. This can be done through overriding the autonomously selected goal, enabling or disabling the obstacle avoidance and giving explicit actuator commands.
Fig. 3.
Control flow for Operator-Autonomy Interaction
C. Operator Interface The sensor readings of the robot team are integrated and presented in an organized fashion. A priori and domainspecific data are presented conveniently for review by the operator. Figure 4 presents a screenshot of the interface taken while two robots were exploring the Mapping Test arena from the USARSim RoboCup 2007 Package. The main area of the interface contains the current common map generated online by the team. The map is overlaid on top of color-coded a priori information, specific to the Virtual Robots competition, but easily extensible to many application domains. The a priori data lets the operator infer what the difficulty of an area is. By convention, yellow represents low difficulty, orange - medium, and red - hard. There can be three types of challenges in the a priori data - communication, mobility and victim identification challenges. The operator can select which type, if any, to display. If none is selected, the main widget will only display the current map, with white indicating free space, black occupied, and gray - unknown. Such a map-centric interface was found to facilitate the operator’s awareness of location and internal status of the controlled robots [14]. Other important features on the map are the robots (triangles) as well as the verified and potential victims (blue circles). Those are linked to the two lists on the right that provide various specific options. For example, we found it convenient to change the readings of most of the widgets, depending on what robot is selected in the robot list. Thus, in case of possible failures we can inspect a given robot closely and see if it is unusually pitched, or if the camera sees an object of interest. While the sensor fusion and efficient presentation of the vast amount of data from the robots were of high priority, our main goal was to provide effective means to control the team. The preferred method to achieve this is through manual entry of goal points. The operator can select a robot at any point in time and set a new goal or override an old one. The path generated by the Navigation module is then displayed on the map and refreshed at each re-computation. This frees the operator from the cumbersome task of selecting waypoints and leaves more time to concentrate on the next robot. Moreover, the background autonomy, namely the coordinated exploration, guarantees that even if the operator is busy inspecting data from one robot, none of the other robots
Fig. 4.
A snapshot of the Operator interface
will remain idle. Thus, when the operator has to concentrate on one robot, the rest of the team can continue pursuing their tasks, or even find new tasks, while staying comfortably within communication with the operator station. We provide two more levels of control for the cases when the operator has to manually maneuver a robot around a difficult obstacle or get it unstuck. Specifically, the Orientation and Speed Controllers and the override Obstacle Avoidance check-box. The first two let the operator give a robot a particular global orientation or speed. The check-box switches off the onboard obstacle avoidance and gives absolute control to the operator and is to be used only in extreme cases. This interface provides a very intuitive approach to controlling the mobile agents and aims to integrate most of the data before presenting it to the operator. It demonstrates many of the benefits of autonomy, without taking away the power of the operator. In fact, we could use this interface to perform both a completely autonomous exploration run by not interacting with the robots at all, or a completely manual run by taking control of all robots and overriding obstacle avoidance. Therefore, we can compare the efficiency of all these approaches without any reconfiguration of the system.
Robots league. The motivation behind both is similar - to develop more reliable platforms and systems to perform exploration of an unknown disaster environemnt and localization of any trapped survivors. The Virtual Robots Competition runs on USARSim, which permits for a cost effective way to develop teams of multiple robots and then migrate the algorithms to the real platforms. Maps from the RoboCup 2006 competition were used to perform sample 10 minute runs of our system. A screenshot of one of the simulated disaster zones, an indoor office environment with multiple rooms and long corridors, is presented in figure 5.
III. S YSTEM P ERFORMANCE A. Performance Metrics The Virtual Robots Competition, for which this system was developed, is closely related to the RoboCup Rescue
Fig. 5.
The RugBot virtual robot in the simulated disaster environment
To compare the performance of the robot team on different runs, similar measures to the ones employed at RoboCup were used. From the maps produced, we measured two major parameters - the explored area and the area cleared of victims. Explored area was calculated as the number of pixels that were seen by the laser range scanners of the robots, converted into square meters. The cleared area, on the other hand is the number of pixels seen by the victim sensor and we feel it gives a more realistic measure of the effectively explored area. Moreover, the cleared area seems to be a more appropriate measure than the number of victims discovered, as those were placed randomly in the environments and discovering them incorporates a great amount of chance. In order to demostrate the benefits of semi-autonomy, two trained operators(members of the Jacobs University Virtual RoboCup team) drove teams of 2, 3, 4 and 5 robots through modified versions of the RoboCup 2006 arenas (unknown to the operators). The same conditions were replicated for a group of operators without any prior experience with the interface. For each team size a fully autonomous and a fully manual(by an experienced operator) run were recorded as a benchmark.
Fig. 6.
Comparing area explored by different operators
B. Discussion The results we obtained from the performance tests are shown in figure 6 for the explored area and in figure 7 for the cleared area. As expected, augmented autonomy performs better then the other two methods. This is especially noticeable when the number of robots in the team increases. Manually controlling many robots requires the operator to split his or her attention and decreases the effectiveness of individual robots, on occasion even decreasing the performance of the team as a whole. The operator-augmented autonomous system avoids these pitfalls by allowing the operator to make high-level decisions, such as manually adding extra goal points, while the background autonomy takes care of the low level navigation without depriving the operator of the option to manually intervene. Figures 6 and 7 also demonstrate that unexperienced operators perform comparably to trained users. A noticable discrepancy occurs in the performance of all five-robot teams, in different quantities. This is largely due to the high computational load on the simulation environment when such a large number of robots and sensors are operated simultaneously. This degradation of the simulator performance presents a greater obstacle to unexperienced users than to trained operators. Nontheless, even first time users managed to produce excellent maps(fig 8) with significant amounts of explored and cleared areas. IV. C ONCLUSION
AND
Fig. 7.
Fig. 8. plan
Comparing area cleared of victims by different operators
Map produced by a 3-robot run overlaid on the ground-truth floor
F UTURE W ORK
In this paper we reviewed a semi-autonomous system, designed to perform the complex tasks of exploration and mapping in Urban Search and Rescue. The experiments performed showed that augmented autonomy multi-robot systems, like the one described, can perform equally and better than both manually controlled and autonomous ones. The increased efficiency of such a system is most readily
observed when the number of robots in the team is increased to four, five or more robots. We also stressed on another key point of designing semi-autonomous systems, namely the operator interface. Such interfaces have to be intuitive and easy to learn and use. Thus, we believe that point-andclick interfaces are the best option for controlling multi-robot
teams. Most likely, users adapt to such interfaces quickly because of the strong analogies to modern day graphical operating systems. For future competitions we plan to emphasize the map display even more, integrating more data and allowing for more immediate interaction directly with the map. Additional data displays, such as roll/pitch information, was only accessed in extreme cases, and does not need to be constantly displayed. However, a more detailed and faceted map display would further facilitate the presented ease of use. Another important research direction is the autonomy behind the interface. We plan to further investigate semiautonomous and also dynamically adjustable autonomous systems and integrate them tightly with a comprehensive user interface. Jacobs University Bremen has also been very active in the development of physical rescue robots [4], [6] which were used in the RoboCup Rescue Robots league for some years [5], [1], [2], [3]. The work described here will have a significant impact on the future development of the real rescue robots and their command interfaces as well, with the next logical step being the migration of the described system from simulation to reality. ACKNOWLEDGMENT Please note the name-change of our institution. The Swiss Jacobs Foundation invests 200 Million Euro in International University Bremen (IUB) over a five-year period starting from 2007. To date this is the largest donation ever given in Europe by a private foundation to a science institution. In appreciation of the benefactors and to further promote the university’s unique profile in higher education and research, the boards of IUB have decided to change the university’s name to Jacobs University Bremen. Hence the two different names and abbreviations for the same institution may be found in this article, especially in the references to previously published material. R EFERENCES [1] A. Birk. The IUB 2004 rescue robot team. In D. Nardi, M. Riedmiller, and C. Sammut, editors, RoboCup 2004: Robot Soccer World Cup VIII, volume 3276 of Lecture Notes in Artificial Intelligence (LNAI). Springer, 2005. [2] A. Birk, S. Carpin, and H. Kenn. The IUB 2003 rescue robot team. In D. Polani, B. Browning, A. Bonarini, and K. Yoshida, editors, RoboCup 2003: Robot Soccer World Cup VII, volume 3020 of Lecture Notes in Artificial Intelligence (LNAI). Springer, 2004. [3] A. Birk, H. Kenn, M. Rooker, A. Akhil, B. H. Vlad, B. Nina, B.S. Christoph, D. Vinod, E. Dumitru, H. Ioan, J. Aakash, J. Premvir, L. Benjamin, and L. Ge. The IUB 2002 rescue robot team. In G. Kaminka, P. U. Lima, and R. Rojas, editors, RoboCup-02: Robot Soccer World Cup VI, LNAI. Springer, 2002. [4] A. Birk, S. Markov, I. Delchev, and K. Pathak. Autonomous rescue operations on the iub rugbot. In IEEE International Workshop on Safety, Security, and Rescue Robotics (SSRR). IEEE Press, 2006. [5] A. Birk, K. Pathak, W. Chonnaparamutt, S. Schwertfeger, I. Delchev, S. Markov, and R. Rathnam. The IUB 2006 rescue robot team. In G. Lakemeyer, E. Sklar, D. Sorrenti, and T. Takahashi, editors, RoboCup 2006: Robot Soccer World Cup X, volume 4434 of Lecture Notes in Artificial Intelligence (LNAI). Springer, 2007. [6] A. Birk, K. Pathak, S. Schwertfeger, and W. Chonnaparamutt. The iub rugbot: an intelligent, rugged mobile robot for search and rescue operations. In IEEE International Workshop on Safety, Security, and Rescue Robotics (SSRR). IEEE Press, 2006.
[7] A. Birk and M. Pfingsthorn. A hmi supporting adjustable autonomy of rescue robots. In I. Noda, A. Jacoff, A. Bredenfeld, and Y. Takahashi, editors, RoboCup 2005: Robot Soccer World Cup IX, volume 4020 of Lecture Notes in Artificial Intelligence (LNAI), pages 255–266. Springer, 2006. [8] J. Brookshire, S. Singh, and R. Simmons. Preliminary results in sliding autonomy for coordinated teams. In Proceedings of The 2004 Spring Symposium Series, March 2004. [9] D. J. Bruemmer, J. L. Marble, D. D. Dudenhoeffer, M. O. Anderson, and M. D. McKay. Intelligent robots for use in hazardous doe environments. In E. Messina and A. Meystel, editors, Measuring the Performance and Intelligence of Systems: Proceedings of the 2002 PerMIS Workshop, NIST Special Publication 990, August 2002. [10] W. Burgard, M. Moors, C. Stachniss, and F. Schneider. Coordinated Multi-Robot Exploration. IEEE Trans. on Robotics, 21(3):376–386, June 2005. [11] S. Carpin, M. Lewis, J. Wang, S. Balakirsky, and C. Scrapper. Bridging the gap between simulation and reality in urban search and rescue. In RoboCup 2006: Robot Soccer World Cup X, LNAI, pages 1–12. Springer, 2006. [12] C. Connolly. Applications of harmonic functions to robotics. In Intelligent Control, Proceedings of the 1992 IEEE International Symposium on, pages 498 – 502, August 1992. [13] J. W. Crandall and M. L. Cummings. Developing performance metrics for the supervisory control of multiple robots. In HRI ’07: Proceeding of the ACM/IEEE international conference on Humanrobot interaction, pages 33–40, New York, NY, USA, 2007. ACM Press. [14] J. L. Drury, B. Keyes, and H. A. Yanco. Lassoing hri: analyzing situation awareness in map-centric and video-centric interfaces. In HRI ’07: Proceeding of the ACM/IEEE international conference on Human-robot interaction, pages 279–286, New York, NY, USA, 2007. ACM Press. [15] A. Elfes, J. Dolan, G. Podnar, S. Mau, and M. Bergerman. Safe and efficient robotic space exploration with tele-supervised autonomous robots. In Proceedings of the AAAI Spring Symposium, pages 104 – 113., March 2006. to appear. [16] G. Grisetti, C. Stachniss, and W. Burgard. Improving grid-based slam with rao-blackwellized particle filters by adaptive proposals and selective resampling. In Proceedings of the IEEE International Conference on Robotics and Automation, ICRA, 2005. [17] S. G. Hill and B. Bodt. A field experiment of autonomous mobility: operator workload for one and two robots. In HRI ’07: Proceeding of the ACM/IEEE international conference on Human-robot interaction, pages 169–176, New York, NY, USA, 2007. ACM Press. [18] C. M. Humphrey, C. Henk, G. Sewell, B. W. Williams, and J. A. Adams. Assessing the scalability of a multiple robot interface. In HRI ’07: Proceeding of the ACM/IEEE international conference on Human-robot interaction, pages 239–246, New York, NY, USA, 2007. ACM Press. [19] M. W. Kadous, R. K.-M. Sheh, and C. Sammut. Effective user interface design for rescue robotics. In HRI ’06: Proceeding of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, pages 250–257, New York, NY, USA, 2006. ACM Press. [20] B. A. Maxwell, N. Ward, and F. Heckel. A human-robot interface for urban search and rescue. In AAAI Mobile Robot Competition, pages 7–12, 2003. [21] B. Thibodeau, A. Fagg, and B. Levine. Signal Strength Coordination for Cooperative Mapping. Computer Science Technical Report 04-64, University of Massachusetts Amherst, July 2004. [22] J. Wang and M. Lewis. Human control for cooperating robot teams. In HRI ’07: Proceeding of the ACM/IEEE international conference on Human-robot interaction, pages 9–16, New York, NY, USA, 2007. ACM Press. [23] B. Yamauchi. Frontier-based exploration using multiple robots. In Second International Conference on Autonomous Agents(Agents ’98), pages 47–53, 1998. [24] H. A. Yanco and J. L. Drury. Rescuing interfaces: A multi-year study of human-robot interaction at the aaai robot rescue competition. Autonomous Robots, 22(4):333–352, 2007.
