Tools and Techniques for MOMU (Multiple Operator Multiple UAV) Environments; an Operational Perspective. Tal Oron-Gilad1, Talya Porat1, Lisa Fern2, Mark ...
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
86
Tools and Techniques for MOMU (Multiple Operator Multiple UAV) Environments; an Operational Perspective
4
Tal Oron-Gilad1, Talya Porat1, Lisa Fern2, Mark Draper3, R. Jay Shively4, Jacob Silbiger5 and Michal Rottem-Hovev6 1 Ben-Gurion University of the Negev, 2San Jose State University, 3US Air Force Research Laboratory
US Army Aeroflight Dynamics Directorate, AMRDEC, 5Synergy Integration Ltd. and 6Israel Air Force
Multiple operators controlling multiple unmanned aerial vehicles (MOMU) can be an efficient operational setup for reconnaissance and surveillance missions. However, it dictates switching and coordination among operators. Efficient switching is time-critical and cognitively demanding, thus vitally affecting mission accomplishment. As such, tools and techniques (T&Ts) to facilitate switching and coordination among operators are required. Furthermore, development of metrics and test-scenarios becomes essential to evaluate, refine, and adjust T&Ts to the specifics of the operational environment. To illustrate, tools that were designed and developed for MOMU operations as part of a US-Israel collaborative research project are described and associated research findings are summarized.
Copyright 2011 by Human Factors and Ergonomics Society, Inc. All rights reserved DOI 10.1177/1071181311551018
INTRODUCTION Multiple operators controlling multiple UAVs (MOMU) is a new operational setup for covering areas of interest, particularly in reconnaissance missions, and highly relevant to homeland security and surveillance operations. With the increase in UAVs' self-control tasks requiring less human execution, current UAV systems transit to one operator supervising a team of semiautonomous UAVs, as opposed to the converse (Brzezinski, Seybold, and Cummings, 2007). Eventually, when flight control becomes fully automated, operators will only have to manipulate the payloads rather than having to fly the vehicles (e.g., Cooper and Goodrich, 2008). Consequently, an increasing body of literature has examined the effectiveness of a single operator controlling multiple UAVs (e.g, Cummings, Nehme, and Crandall, 2006). However, this mode of operation often increases the cognitive burdens of its operators. Besides the challenge of preventing high operator workload and low situation awareness, caused by the need to attend to multiple sources of information at once, this mode also requires switching of information sources, i.e., tasks, missions, video feeds or camera manipulations, and coordination among operators. This switching is a timecritical, cognitively demanding task. In MOMU environments, where operators may handoff aircraft, payloads, targets, or missions to each other, switches may have a vital effect on mission accomplishment. Cognitive costs of switching may be loss of orientation and situation awareness (SA), increase in workload, and decrease in efficient verbal team communication.
Consequently, switching between sources can disrupt operator performance (Draper et al., 2008). As the autonomy of UAVs increase and interfaces improve, switch costs gradually become the bottleneck that limits the number of UAVs that a single operator can manage or be aware of (Hancock, Moulou, Gilson, Szalma, and Oron-Gilad, 2007). The 'RICH' (Rapid Immersion tools/techniques for Coordination and Hand-offs) research project is a USIsrael collaboration. The project aims to research, design and develop tools, techniques and procedures to aid operators in MOMU environments; to facilitate task switching and/or coordinate with other operators all for the benefit of improving overall mission performance. METRICS Development of metrics and modes of evaluation is an essential process when new T&Ts are being developed. Good metrics distinguish and afford comparison of efficiency and effectiveness of tools in various experimental scenarios. While in operational environments the first and upmost priority is to measure mission execution success (the key to any successful military operation), there are also a variety of interfacerelated and operator-related attributes which facilitate or impede overall mission performance and should be measured specifically. Additional measures can be related to the usability and adequacy of the interface, the stress and load imposed upon each individual operator, as well as to team performance communication qualities.
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
Table 1 provides a template of performance related measures used within this framework. Table 1. Summary table of measures. Mission effectiveness Operational Detection measures area coverage, productivity hits/misses/false alarms/correct rejections. Time-related measures, response time, task completion time Supervisory controlOptimal UAV selection, Time to set handoff/switching up UAV sensor, Time until operator performance begins to correctly manipulate a newly acquired sensor, Neglect time Efficiency Objective Interaction efficiency (number of) : Operator related sketch/revisit request, mouse clicks, button presses, joystick movements, zo Tool related Frequency of use, utility of use to missi effectiveness Payload related Efficiency of payload movements, e.g., time in auto-track, field-of-view distribution etc. Accumulated path while tracking an object in meters. Summation of all changes in the field-of-view. Subjective usability Questionnaires, heuristic evaluations, etc. Interface adequacy E.g., Modified Cooper Harper Scale
TOOLS & TECHNIQUES Within the framework of the current project, several display layouts and tools were designed, developed and examined. Here we briefly describe exemplars from the RICH developed tools. For each tool we specify its origins, descriptions and how beneficial it may be to the MOMU environment. Tools were tested and evaluated separately, and thus experimental findings associated with each tool are also mentioned. Tools can be divided into two categories: 1) tools that facilitate 'quick setup', i.e., aimed to ease the way of the operator into a new mission or area of operation; and 2) tools that facilitate on-going missions where acquiring new UAVs, delegating, or switching is necessary to complete the tasks at hand. The first two tools here belong to the first category while the others belong to the second one. “Get in the Zone” Camera Transition The “Get in the Zone” (GITZ; shown in Figure 1) camera transition concept was designed to rapidly build operator SA and minimize negative transfer effects when switching between multiple UAV missions and their associated camera views. Rather than discretely switching between camera views, GITZ utilized the
87
concept of visual momentum to support the rapid comprehension of relevant environmental information following the transition to a new display (Woods, 1984). When switching to a new UAV (i.e., new camera view), the camera imagery from the current UAV would first convert to a synthetic vision correlate of the real video image, then follow a “fly-out, fly-in” metaphor over a few seconds (similar to Google Earth transitions), finishing with a transition back from synthetic vision to real video at the new camera viewpoint. During the transition, mission-relevant areas were highlighted with overlaid, geo-registered computer-generated symbology.
Figure 1. GITZ Transition Concept
Two studies were conducted to evaluate the efficacy of the GITZ transition. An initial study (Draper et al., 2008), indicated improved SA and more accurate initial camera movement in trials employing GITZ. However, this transition did not influence performance on the key task – the mean time to locate or designate targets was only slightly faster when GITZ was used. Several enhancements to GITZ were identified, including changing the transition speed and providing the operator direct control over certain transition parameters. A second study (Calhoun, Warfield, Ruff, Spriggs and Wright, 2010) found significantly improved spatial awareness of the new environment (measured objectively) and a trend towards increased speed in moving the new camera to the intended landmark area. Airspace Transition Display The Airspace Transition Display was developed in response to operational interviews with U.S. Army and Air Force UAS operators that identified airspace and clearance information as a critical issue for UAS operations and handoffs. The display was designed to provide airspace information (i.e. clearance, point of contact, instructions) ‘at a glance’ that would be critical to an operator when taking over a particular mission, so that the operator could rapidly build airspace SA. Fern and Shively (2011) conducted a simulation experiment to compare the effects of various formats of
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
the airspace transition display on operator workload, performance and SA. Three airspace transition display formats, a text-based transition page resident in the Multi-Function Display (MFD), a graphics-based transition display resident in the MFD and a graphical map overlay on the tactical map display, were compared to an Internet Relay Chat (mIRC) room resident in the MFD that was designed based on current Shadow UAS airspace and clearance operations (shown in Figure 2). Results of the experiment indicated improved task performance and SA, as well as lower workload ratings with all of the airspace transition display formats compared to a baseline condition. In particular, the transition displays showed significantly quicker reaction times to determine an airspace status (e.g. approved clearances) and significantly higher airspace SA scores compared to baseline. Subjective ratings by operators on the usefulness and ease of use of the various display formats indicated overall preference to graphical (as opposed to text-based) formats.
88
UAV which has the best view of their target at any given moment. This tool received excellent feedback in the design evaluation (see Porat, Oron-Gilad, Silbiger and Rottem-Hovev, 2010). The castling rays can be displayed automatically or by a press of a button, according to users' preference definitions. The high utility of this tool has been demonstrated particularly when multiple UAVs share the same area of operation.
Figure 3. 'Castling Rays' tool. Left. In this example, the operator is monitoring a target with the red payload. The castling tool shows the green payload with a continuous line indicating that it has a good line of sight of 'my' target. The yellow payload is shown with a thinner dashed line which indicates that it has limited line of sight of 'my' target. Right. Similar implementation in the US Army research environment.
Dynamic Layout
Figure 2. Clockwise from the top left: Internet Relay Chat room resident in the MFD (baseline condition), text -based transition display resident in the MFD, Graphics-based transition display resident in the MFD, and the airspace transition display overlaid on map display.
Castling Rays The 'Castling Rays' tool (shown in Figure 3) is a payload-switching decision aid, enabling operators to visually view, on the video stream, which payload has the best view of 'their' target at any given moment. This reduces the need of the operators to view and analyze data from the command and control map. This tool has significant advantage in MOMU environments, where operators can view and select immediately the optimal
Whether user-controlled or automation-controlled, this technique enlarges the window of payload operation (Porat, Oron-Gilad, Silbiger and Rottem-Hovev, 2011). When the operator controls or needs to be aware of multiple video feeds then it is possible that these feeds should be presented to him/her in a way that conveys their importance/relevance to the mission at hand. Thus, the video feed window which is most in use (e.g., time on window, mouse clicks) grows on account of the other windows. Preliminary findings indicated that these techniques reduced the need to manually perform zoom operations compared to the non dynamic layout. According to this result, the following "Maintain Video Quality" tool was developed to examine the relation and interaction between the video feed window size, necessity for zoom manipulations and desired target size. Reducing the zoom manipulations by the operator may reduce the amount of workload while performing the mission. Maintain Video Quality The Maintain Video Quality tool enables operators to define a minimum desired video quality. Video quality is defined as window size (pixels) divided by footprint size (meters) (see Figure 4). The system will preserve this quality, as long as it can, by increasing the
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
available window size or/and changing the zoom. The tool contains two sliders: Zoom value (display only) and Video quality value – interactive slider. The user defines the minimum video quality she/he is willing to absorb by clicking on the desired value (a yellow mark will be displayed). This is important in surveillance tasks, when the target needs to maintain a certain level of detail and therefore within a certain size. This tool has a significant advantage when switching payloads in MOMU environments. The assumption is that the user would like to view the target from a different point of view, but maintain the defined video quality. Therefore, when switching payloads, the system will make the necessary adjustments (zoom and window size) to the switched payload video, to maintain the desired video quality (see Figure 5). Initial results (Oron-Gilad, Porat, Silbiger and Rottem-Hovev et al., 2011). showed no significant differences in performance, throughout the experiment, with or without the tool. However, a learning curve was found whereby participants improved their performance significantly while using the tool. Their performance in the third scenario was significantly better than the first scenario. This may imply that users need time to learn and get used to tool behavior before it can improve their performance (as they also indicated in their questionnaire responses).
89
another UAV. This is a very effective tool as it maximizes the use of UAVs that are currently not assigned to a specific mission of their own. Previous research demonstrated the efficiency of the coupling tool, particularly in dense urban areas where concealments are frequent and when the switched payload was relatively distant from the target. The ‘Advanced’ version of the tool seeks to optimize its behavior. In addition to coupling the 'point-of-view' among tied UAV's (as done previously), now coupled UAV's also share the same video quality (Oron-Gilad et al., 2011). This may increase the advantage when switching payloads in MOMU environments. Thus, not only will the two payloads look at the same target, they also share the same video quality constraint saving the operator any setup time for the coupled UAV's zoom settings. Figure 6 displays on the left the payloads before coupling. Top-left video is defined as quality 0.25 and bottom-left video is defined as quality 0.75. After coupling the two UAVs (right), the system automatically increases the top-left video to quality 0.75 (by zoomingin). This tool should be further examined.
Figure 6. Coupling optimization: Left. before coupling. Right. after coupling.
Figure 4. Maintain Video Quality tool. Left. video quality = 1/8. Right. video quality = 1/4.
Summary The development of T&Ts has been conducted among all three collaborating research teams, US Army, US Air Force and the Israeli research team. Up to now, the tools were adjusted and examined separately in the different simulated environments. Therefore, the next required step would be to evaluate the tools and techniques in an integrated MOMU experiment.
Figure 5. Maintain Video Quality tool. Left. In this example the operator switched between the top-left payload (green) to the bottomright payload (blue). Right. The video quality of the blue payload (zoom and window size) changed to suit the quality definitions (defined in the quality slider).
Advanced Coupling The 'Coupling' tool allows the operator to "tie" two UAVs, thus allowing one UAV to follow the path of
NEXT STEPS Results of the separate experimental evaluations of tools and techniques has highlighted specific performance benefits of each separate tool for the MOMU operator. Further research is required to examine how a suite of handoff/switching tools supports operators networked in a MOMU environment. Evaluating the RICH tools in an integrated MOMU environment requires defining and developing a
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
common work environment and operational scenarios that facilitate switching, along with utilization of measurement metrics. These serve as essential common ground enabling all three parties to utilize and translate the results to their own work environment and mission needs. The main interest is to examine whether RICH tools are indeed beneficial to the MOMU operators for a variety of mission tasks and when used as a tool set, i.e., there is a need to clarify whether operators will have sufficient resources to utilize the entire set of tools or whether they will adhere to one or two tools that they find helpful. Furthermore, it is possible that various operational environments will demand a different utilization of the tool-set. Finally, there may be a methodology to optimize the use of the tools for specific tasks (e.g., route recon, target detection, surveillance. etc.). As such, the examination of the RICH tools as a set is essential. An experiment will take place later in 2011 and will consist of several operators controlling three or five UAVs while performing multiple tasks (e.g., monitor buildings, route reconnaissance, and track targets) where multiple instances of platform handoffs and task switching must take place. CONCLUSION Tools and techniques are essential to improve MOMU operator's performance, however, not all T&Ts are alike and some may share more commonalities among work environments while others need to be modified and adjusted. Common metrics are essential to allow comparison of efficiency and effectiveness of tools in various experimental environments – those serve as common ground for comparison and identification of weaknesses and area that need to be improved. ACKNOWLEDGMENTS This work is part of the US/Israel MOA/FMF on Rotorcraft Aeromechanics and Man/Machine Integration under Task 24, Rapid Immersion tools/technique for Coordination and Hand-offs (RICH). REFERENCES Brzezinski, A. S., Seybold, A.L., and Cummings, M. L. (2007). Decision Support Visualizations for Schedule Management of Multiple Unmanned Aerial Vehicles. Presented at the AIAA InfoTech@Aerospace, Rohnert Park, CA, May, 2007. Calhoun, G.L., Warfield, L., Ruff, H.A., Spriggs, S.E., and Wright, N.F. (2010). Automated Aid Design for Transitioning Between Camera Views. In
90
Proceedings of the Human Factors and Ergonomics Society AnnualMeeting, pp. 413-417. Cooper, J., and Goodrich, M.A. (2008). Towards combining UAV and sensor operator roles in UAVenabled visual search. Presented at HRI 08', Amsterdam, The Netherlands, March, 2008. Cummings, M.L., Nehme, C.E., and Crandall, J. (2006). Predicting operator capacity of supervisory control of multiple UAVs. Humans and Automation Laboratory. MIT, Cambridge, MA. Draper, M., Calhoun, G., Ruff, H., Mullins, B., Lefebvre, A., Ayala, A., & Wright, N. (2008). Transition display aid for changing camera views in UAV operations. In Proceedings of the Humans Operating Unmanned Systems. Retrieved from http://conferences.telecom-bretagne.eu/. Draper, M.H., Calhoun, G., Nelson, J., Lefebvre, A., & Ruff, H. (2006). Synthetic Vision Overlay Concepts for Uninhabited Aerial Vehicle Operations: Evaluation of Update Rate on Four Operator Tasks, Proceedings of the NATO RTO Human Factors and Medicine Panel Symposium, HFM-135, held in Biarritz, FR, 9-11 October 2006. NATO RTO: Neuilly-sur-Siene, CEDEX. Fern, L., & Shively, R. J. (in review). Designing airspace displays to support rapid immersion for UAS handoffs. Proceedings of the 55th Annual Meeting of the Human Factors and Ergonomics Society, Las Vegas, NV. Hancock P.A., Mouloua, M., Gilson, R., Szalma, J., and Oron-Gilad, T. (2007). Is the UAV control ratio the right question? Ergonomics in Design, 15 (1), 7, pp. 30-31. Porat, T., Oron-Gilad, T., Silbiger, J., and RottemHovev, M. (2011). Switch and Deliver: display layouts for MOMV (Multiple Operators Multiple Video feeds) environments. IEEE CogSIMA 2011 Conference Proceedings, Miami, February, 2011. Porat, T., Oron-Gilad, T., Silbiger, J., and RottemHovev, M. (2010). 'Castling Rays' a Decision Support Tool for UAV-Switching Tasks. CHI 2010 Conference Proceedings, Atlanta, April, 2010. Oron-Gilad, T., Porat, T. Silbiger, J. and Rottem-Hovev, M. (2011). Decision Support Tools and Layouts for MOMU (multiple operator multiple UAV) Environments. ISAP Dayton OH, May 2-May 5, 2011. Woods, D. (1984). Visual momentum: a concept to improve the cognitive coupling of person and computer. International Journal of Man-Machine Studies, 21, 229-244.
