CogSIMA 2012, New Orleans, LA
Support of Collaborative Work in Battlespace Management: Shared (Loss) of Situation Awareness Sébastien Tremblay 1, Richard Breton 2, François Vachon 1, & Dave Allen 2 1
2
Université Laval
Defence R&D Canada _
Abstract— In this paper, we report results from a large-scale military experiment with Canadian Forces (CF) officers on the impact of collaborative work support systems—in the present experiment, a set of integrated tools such as operations planning systems, joint fire systems and other logistics tools—on individual and shared situation awareness (SA). In order to measure SA and the ability to share SA, we used the Quantitative Assessment of Situation Awareness (QUASA) technique: We first analyzed SA quality (sensitivity, response bias, accuracy) and metacognition (level of confidence, and calibration bias), and then computed the level of response concordance within and across groups (three different Operational Commands of the CF). The addition of the new support system led to a significant improvement in shared SA. However, this beneficial effect comes with a drop in both objective and perceived individual SA. From this pattern of results, we conclude that there might be a tradeoff between SA sharedness and quality. Supporting the process of sharing SA through enhanced means of information integration and exchange, communication and coordination can also lead to a considerable decrease in individual SA and meta-SA. Index Terms— Military C2, Shared Situation Awareness, Collaborative work support system, Experimentation.
I. INTRODUCTION HE importance of promoting situation awareness (SA) in Thuman operators and decision makers has proved to be a central issue for system design in military command and control (C2). One common simple definition of SA on which there is a relative consensus and that is used in a wide range of work domains refers to the notion of being aware of what is Manuscript received October 31, 2011. This work was supported by Defense Research and Development Canada (DRDC) Valcartier, the Centre of Operational Research and Analysis (CORA), Thales Canada, and Canadian Forces Warfare Centre (CFWC). The corresponding author is S. T., who is with the School of Psychology, Université Laval, Québec, Québec, G1V 0A6, Canada (phone: 418-656-2131, x4886; fax: 418-656-3646; e-mail:
[email protected]) R. B. is with Defence R&D Canada – Valcartier, Québec, Québec, G3J 1X5, Canada (e-mail:
[email protected]). F. V. is with the School of Psychology, Université Laval, Québec, Québec, G1V 0A6, Canada (e-mail:
[email protected]). D. A. is affiliated with Defence R&D Canada – CORA, Ontario, K1A 0K2, Canada (e-mail:
[email protected]).
978-1-4673-0345-3/12/$31.00 ©2012 Crown
going on in our environment and how the current situation is likely to evolve (e.g., [1], [2]). A key challenge is to provide officers involved in the operational-to-tactical battlespace management with support systems that can increase SA accuracy, facilitate the process of sharing their SA, and therefore enhance their so-called common operating picture (COP) and management capability [3]. In order to deal with the growing need for enhanced real-time SA in modern military C2, emerging network-centric technologies provide operators with access to unprecedented amounts of real-time operational space information, and offer enhanced coordination and communication capabilities through the use of sophisticated systems of systems. Such technologies are widely assumed to improve decision-making and mission effectiveness through shared SA [4] and their architecture is often based upon systematic analyses of requirements [5]. Modern military C2 requires rapid and optimal decisionmaking despite severe constraints to human cognitive capacity in coping with complex and dynamic information environments, often massively distributed collaboration, under stress and time pressure [6]. The process of sharing comes with cognitive costs due to the need for C2 staff to communicate [7, 8] and to process a large amount of information from COP systems [9]. Cognition is a crucial dimension on which support systems can fail, often with disastrous consequences. The development of tools to assist C2 staff has been largely technology-driven without a commensurate understanding of the cognitive fit with the human user. The benefit of these tools is difficult to evaluate and evidence suggests that they may even have a negative effect on C2 teamwork [10, 11]. Measures of SA— both individual and shared—provide one way of assessing the impact of a new set of tools upon C2 performance. In this paper, we report results from a large-scale military experiment with Canadian Forces (CF) officers on the impact of collaborative work support systems—in the present experiment, a set of integrated tools such as operations planning systems, joint fire systems and other logistics tools— on individual and shared SA. The experiment was conducted in the Joint Battle laboratory (see Figure 1) of the Canadian Forces Warfare Centre (CFWC).
214
designed to increase the ease of integrating information from different sources, and facilitate communication and coordination between officers within and across Operational Commands).
Fig. 1. Picture of the Joint battle Lab (setup used in the experiment).
II. METHOD A. Participants Thirty-nine military officers of CF participated in the experiment (additional officers filled experiment control roles). The participants manned specific positions within three different CF organizations focusing on the operational level. B. Experimental Setup The experiment—a simulation of battlespace management that involved the collaboration of three national Operational Commands and a theatre Operational Command—was conducted over a period of eight days at the CFWC Joint Battle Lab. Figure 2 displays the layout of the lab as set up for the experiment. The participants were located in three different rooms. C. Scenario The experiment scenario included three main Areas of Operations at the domestic and international levels. The incidents requiring the involvement of the participants included: a helicopter crash; an Unmanned Aerial Vehicle crash; a missing soldier; personnel replacement issues; a military convoy damaged by improvised explosive device (IED); the highjack of a ship transporting military equipment; air intrusion within Canadian territories; drug smuggling into Canada; a senior level visit to Kabul; and, other similar incidents. The main roles of the players were to monitor incoming information, manage that information, answer requests, prepare reports, and brief the officers for them to make adequate decisions. Responses to specific (scripted) incidents were required (e.g., high risk fire support requests, IED incidents, detainees handling issues, resource replacement issues). D. Design and Procedure The experiment compared two conditions that differed based on the systems available to the participants: A baseline system of systems (representative of the current set of tools used by the relevant military organizations) and a prototype set of systems (that involved modern systems of which tools were
Fig. 2. Schematic representation of the layout of the Joint Battle Lab. The three national commands (CANCOM, CEFCOM, CANOSCOM and the theatre command (Regional Joint Task Force Joint Operation Centre) were located in different buildings.
E. SA Measurement In order to measure SA and the ability to share SA, we used the Quantitative Assessment of Situation Awareness (QUASA) technique and computed the level of response concordance within and across groups (two national Operational Commands—CEFCOM and CANOSCOM—and the theatre Operational Command were the focal points of this analysis). QUASA, as developed by Edgar et al., [12], is one of the most frequently used methods of collecting and analyzing data on SA (e.g., [13, 14], for a review). The technique combines both objective SA—from accuracy of responses to queries (true/false probes) about the situation— and subjective SA—from self-ratings of confidence for each probe response. The QUASA had the same proportion of true and false items. After evaluating the statement, the participant was invited to rate his confidence on a 7-point Likert scale.
215
QUASA also includes calibration estimates that provide an index of over- and underconfidence. The QUASA probes that were administered twice daily corresponded to the events embedded in the scenarios: Participants were asked to identify if specific statements on the overall situation were True or False (see Table 1). A total of 10 statements were used at each session (20 statements per day). Of course, in battlespace management, not all Operational Commands are expected to be aware of the full spectrum of situational elements. This was taken into account in the analyses.
TABLE I SAMPLE OF THE QUASA QUESTIONNAIRE At Checkpoint 4-3, an individual was on a rooftop waiving a black flag. A local national indicated that this is a signal […] After rescuing a sinking ship, Her Majesty Canadian Ship (HMCS) Calgary has 6 Tamil indentured slaves looking for asylum in Canada. VIPs including the Chief of Defence Staff are on their way to visit the Kandahar Air Field and Kabul NATO headquarters. In vicinity of town A, a patrol troop received small arms fire but could not return fire due to large crowd at market. Locals from town A informed patrol that insurgents keep a regular presence in village and continually ask villagers to set up improvised explosive devices targeting Canadian Forces patrols.
Fig. 3. Calibration curve from the QUASA technique. The level of perceived accuracy (obtained from self-ratings of confidence in individual responses) is plotted against the actual accuracy level to SA probes for each of the three groups in the control and the new system conditions averaged across the four days of testing. The diagonal represents perfect calibration.
IV. DISCUSSION
III. RESULTS Metrics extracted from QUASA can be divided into two classes. The first class, at the individual level, is concerned with measures of SA quality (sensitivity, response bias, accuracy) and metacognition (level of confidence, and calibration bias; see Figure 3). A second class of measures evaluates the extent to which SA is shared among team members. Shared SA is reflected through the level of response concordance. The impact of the new support system on the levels of SA and shared SA of each group was analyzed separately for the first two and the last two days of testing in order to assess the effect of familiarization with the system. A summary of the effects of the new system on each SA metric is presented in Table 2 for all three groups. The ‘+’ sign indicates a positive impact of the new system while the ‘–’ sign points to a negative impact. The ‘=’ sign indicates that the new system had no reliable impact (less than 5% variation) on the variable.
The addition of the new system led to a drop in sensitivity (i.e. the ability to discriminate between true and false descriptions of the situation) for most days and groups. Although calibration improved eventually for two of the Operational Commands (mainly due to the diminution of the confidence level with the new system), yet individuals remained overconfident (i.e., perceiving accuracy as better than it actually was) with the new system. If C2 personnel become overconfident, their actions are more likely to rely on a general knowledge of the situation and on past experiences, rather than basing judgment upon the actual situational information presented to them [15]. If, for example, a problem develops suddenly (see, e.g., [16]), a full awareness of the current situation would be necessary rather than dangerously relying on more general knowledge and heuristics. Overall, the prototype C2 system had a rather detrimental impact on both objective and perceived individual SA. With regards to shared SA, according to the Kappa magnitude guidelines [17], each group showed a relative agreement between members. However, it seems that this agreement was as a result of shared “poor” SA, which is far from achieving the main objective of a sophisticated collaborative work support system to facilitate a clear and accurate common
TABLE II SA METRIC IN THE FIRST TWO AND LAST TWO DAYS OF TESTING FOR EACH GROUP
Sensitivity
Confidence
Calibration
Shared SA
Days
1-2
3-4
1-2
3-4
1-2
3-4
1-2
3-4
Group OC1
▬
+
▬
▬
=
+
=
+
Group OC2
▬
▬
▬
▬
▬
+
+
+
Group OC3
▬
▬
▬
▬
▬
▬
=
=
216
picture of the battlespace. The capacity to gain and maintain SA is dependent on many factors including the mission goals, the complexity of the operational situation, the commander’s workload, the level of stress and expertise. It has been assumed that technological support must necessarily increase SA of C2 personnel; however, advanced technology may in fact degrade performance. V. CONCLUSION From the pattern of results obtained in the present largescale experiment, we conclude that there might be a tradeoff between sharedness and accuracy of SA in terms of costs and benefits. Shared SA has become a buzzword over the last decade or so and is often the basis for providing emergency managers and C2 personnel with network-centered or K-wall systems [18]. However, shared SA does not seem necessarily beneficial. Supporting the process of sharing SA through enhanced means of information integration and exchange, communication and coordination can lead to a decrease in individual SA.
[11] F. Vachon, D. Lafond, B. R. Vallières, R. Rousseau, and S. Tremblay, “Supporting situation awareness: A tradeoff between benefits and overhead,” Proceedings of the IEEE Conference on Cognitive Methods in Situation Awareness and Decision Support, 2011. [12] G. K. Edgar, H. E. Edgar, and M. B. Curry, M.B. “Using signal detection theory to measure situation awareness in command and control,” Proceeding of the Human Factors and Ergonomics Society 47th Annual Meeting, 2003, pp. 2019–2023. [13] B. McGuinness, “Signal detection theory and the assessment of situation awareness”, in Decision making in complex environments, M. Cook, J. Noyes and Y. Masakowski, Eds., Aldershot, UK: Ashgate Publishing Ltd., 2007, 299–310. [14] A. Kirlik and R. Strauss, “Situation awareness as judgment I: Experimental demonstration,” International Journal of Industrial Ergonomics, vol. 36, pp. 475-484, 2006. [15] R. Rousseau, S. Tremblay, S. Banbury, R. Breton, and A. Guitouni, “The role of metacognition in the relationship between objective and subjective measures of situation awareness,” Theoretical Issues in Ergonomics Science, vol. 11, pp. 119–130, 2011. [16] B. M. Huey and C. D. Wickens, Workload transition, Washington, DC: National Academy Press, 1993. [17] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, pp. 159–174, 1977. [18] Hansen, D. “Effects of Buzzwords on Experiential Learning: The Swedish Case of Shared Situation Awareness.” Journal of Contingencies and Crisis Management, vol 17, pp. 169-179, 2009.
ACKNOWLEDGMENT We are thankful Jean-François Gagnon and Helen Hodgetts for critical readings of an earlier draft and to Frederick Lichacz for his assistance in the construction of the QUASA items. REFERENCES [1]
M. R. Endsley , “Theoretical underpinnings of situation awareness: a critical review,” In Situation awareness analysis and measurement, M. R. Endsley and D. J. Garland, D. J., Eds. LEA, 2000. [2] R. Rousseau, S. Tremblay, and R. Breton, “Defining and modeling Situation Awareness: A critical review,” in A Cognitive Approach to Situation Awareness: Theory and Application, S. Banbury and S. Tremblay, Eds. Aldershot, U.K.: Ashgate and Town, 2004, pp. 3–21. [3] P. M. Salmon, N. Stanton, G. Walker, and D. Green, “Situation awareness measurement: A review of applicability for C4i environments,” Applied Ergonomics, vol. 37, no. 2, pp. 225–238, 2006. [4] F. T. Durso and A. Sethumadhavan, “Situation awareness: Understanding dynamic environments,” Human Factors, vol. 50, pp. 442–448, 2008. [5] S. Banbury, S. Tremblay, R. Rousseau, K. Forbes, and R. Breton,, “Applying the Contextual Control Model to the identification of situation awareness requirements for tactical army commanders,” Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting, 2008. [6] D. S. Alberts and R.E. Hayes, “Understanding Command and Control,” The Future of Command and Control. CCRP, DoD, Washington, District of Columbia, 2006. [7] D. Lafond, M.-E. Jobidon, C. Aubé, and S. Tremblay, S., “Evidence of structure-specific teamwork requirements and implications for team design,” Small Group Research, vol. 42, pp. 507-535, 2011. [8] J. MacMillan, E. E. Entin, and D. Serfaty, „Communication overhead: The hidden cost of team cognition,” in Team cognition: Process and performance at the inter and intra-individual level, E. Salas & S. M. Fiore, Eds.,. Washington, DC: APA Press, 2004. [9] J. MacMillan, M. J. Paley, Y. N. Levchuk, and D. Serfaty, D. “Designing the information space and physical layout for a command center based on an optimized organizational structure,” Proceedings of the Command and Control Research and Technology Symposium, 1999. [10] F. J. Lerch and D. E. Harter, “Cognitive support for real-time dynamic decision making,” Information Systems Research, vol. 12, pp. 63-82, 2001.
217
