Cogn Tech Work DOI 10.1007/s10111-011-0208-1
ORIGINAL ARTICLE
Visualization framework of macrocognition functions Emily S. Patterson • Robert R. Hoffman
Received: 26 August 2011 / Accepted: 10 November 2011 Springer-Verlag London Limited 2012
Abstract Researchers and domain practitioners have identified macrocognition functions that characterize work in complex domains. These functions provide a broad conceptual lens to aid description and thematic analysis of overwhelming, messy, interrelated observational and interview data from cognitive field research. In this article, an integrated theoretical framework is proposed for five macrocognition functions: detecting problems, sensemaking, re-planning, deciding, and coordinating. An example of each function is provided, from observations during the 76th Space Shuttle (STS-76) mission. Keywords Macrocognition Decision making Coordination Planning Event detection
1 Introduction ‘‘There are powerful regularities to be described at a level of analysis that transcends the details of the specific domain, but the regularities are not about the domainspecific details, they are about the nature of human cognition and human activity’’ (Hutchins, cited by Woods and Hollnagel 2006, p. 3). Macrocognitive work systems (MWSs) are systems in which people use advanced technology to collaborate for E. S. Patterson (&) School of Health and Rehabilitation Sciences, College of Medicine, Ohio State University, Columbus, OH, USA e-mail:
[email protected] R. R. Hoffman Institute for Human and Machine Cognition, 40 South Alcaniz St., Pensacola, FL 32502-6008, USA e-mail:
[email protected]
the purpose of conducting work. Their primary activity is macrocognitive, which is defined as the adaptation of cognition to complexity (Klein et al. 2003). True to the definition of ‘‘system,’’ MWSs involve dynamics and emergent phenomena (Hoffman et al. 2009). The notion of a MWS is similar in spirit to the notion of ‘‘sociotechnical’’ systems, which emerged in Industrial Relations research in the 1950s. However, macrocognition places explicit emphasis on cognitive processes and functions (e.g., sensemaking, problem detection, and others) while at the same time including significant social processes and functions (e.g., coordinating, re-planning, and others). An additional attribution to MWSs that is common in the literature is that macrocognitive work is conducted in domains that are societally and economically important, such as air traffic control or military command. These are, not coincidentally, the topical areas of much of the work in the field of cognitive systems engineering. This attribution of ‘‘importance’’ emphasizes a number of things, such as the fact that macrocognitive work is carried out under constraints of organizational resources, requirements, and cultures. The ‘‘importance’’ attribution also emphasizes that the work is often stressful, high risk, and high stakes and that work success depends on domain practitioner expertise. All of these are features that typify the domains that have historically been the topic for Naturalistic Decision Making research, and so it is no coincidence that ‘‘macrocognitive work’’ is coming to be regarded as a better, if not more inclusive, designation for the intent of NDM as a paradigm. As the introductory quotation from Edward Hutchins implies, MWSs is also an appropriate designation for research that has been less formally called ‘‘cognition in the wild.’’ The macrocognitive approach distinguishes between primary functions and supporting processes. Primary functions are the most important functions that a work
123
Cogn Tech Work
system must conduct in order to achieve its primary goals. Primaries are the things that a leader will say are the most difficult and yet most important tasks that the work system as a whole must accomplish. Looking at research on domains and domain expertise, the primary functions are sensemaking, detecting problems, adapting, re-planning, coordinating, and deciding (Klein et al. 2010; Patterson et al. 2010). Each of these, to different degrees and in different circumstances, will rely more heavily on certain of the supporting processes. Again looking across domains of expertise, supporting processes are maintaining common ground, developing mental models, managing risk, and managing uncertainty. The primary functions are the goals of the work; the supporting processes are the ways in which cognition is applied to achieve the goals. Continued development of a theory or methodology for a science of MWSs will require the identification of common interactions or blends of functions and processes. In this article, we present a ‘‘wrapped package’’ in which a number of macrocognitive primary functions are illustrated in a single case: command and control work during the 76th Space Shuttle (STS-76) mission. This provides an integrated look at how the functions are parallel, continuous, and interacting. This case illustrates the primaries of detecting problems, sensemaking, re-planning, deciding, and coordinating.
hand default plans are not applicable to the situation, this can include creating a new strategy for building one or more goals or desired end states. This function includes adapting procedures, based on possibly incomplete guidance, to an evolving situation where multiple procedures need to be coordinated, procedures that have been started may not always be completed, or when steps in a procedure may occur out of sequence or interact with other actions. In describing MWSs, one refers always to ‘‘re-planning,’’ and not to ‘‘planning.’’ Executing a plan is not distinguished from re-planning, even when the individual or team that generates a plan is different from the individuals or teams who perform the actions to execute it (Klein 2007a, b). 1.4 Deciding
This is noticing that events may be taking an unexpected direction. Whether positive or negative with respect to goal accomplishment, change requires explanation and might signal a need or opportunity to reframe how a situation is conceptualized (sensemaking) and/or revise ongoing plans (re-planning) in progress (executing).
Deciding is a complex activity that should not be thought of simply as the act of committing to some course of action in order to reach certain fixed goals. Deciding can involve questioning the appropriateness of standard courses of action or default decisions. It can involve considering trade-offs in ongoing plan trajectories. It can involve sacrificing previous decisions or commitments. Deciding may be the activity of a single individual, but more often is the activity of a team. It can be a consensus activity involving the accommodation of different stances toward decisions. The deciding process involves a host of questions and issues, some being more and some being less important in different contexts. Deciding does not end when a button is pushed—most decisions (in complex contexts) are actually contingencies, it really does not make sense to think of decisions as ‘‘things that are made’’ (see Hoffman and Yates 2005). Thus, deciding is far more complex than classical discussions of decision-making sometimes make it out to be.
1.2 Sensemaking
1.5 Coordinating
This includes activities of collecting, corroborating, and integrating information and assessing how the information maps onto potential scenarios or explanations. It includes generating new potential hypotheses to consider and revisiting previously discarded hypotheses in the face of new evidence.
This is managing interdependencies of activity and communication across individuals acting in roles that have common, overlapping, or interacting (and possibly conflicting) goals.
1.3 Re-planning
Any of the five primary functions can be engaged either by events in the world, communication between people, or individuals’ reasoning. While a researcher might focus on a particular function (such as re-planning), other functions are ongoing, such as monitoring the environment (detecting problems), re-assessing the current situation (sensemaking), re-examining depth of commitment to a particular decision in order to increase or decrease flexibility in constraints on
1.1 Detecting problems
This is adaptively responding to changes in objectives, from any of a variety of sources including supervisors (MMACS) and peers, obstacles, opportunities, events, or changes in predicted future trajectories. When ready-to-hand default plans are applicable, there is still a need to adapt them into actions within a window of opportunity. When ready-to-
123
1.6 Visualization framework
Cogn Tech Work
the plan (deciding), and coordinating with other actors who need to sequence their activities and share resources in order to execute the plan effectively (coordinating). Coordinating might often be a primary function that continuously ‘‘wraps’’ the others. It would be valuable to have some means of representing the common dynamic overlaps and interactions of the functions. In Fig. 1, we propose a way of visualizing the macrocognition functions. While examining the Figure, the circles can be imagined as rings that are continuously rotating, each about its own centroid. Over time, a ring overlaps more or less with other functions, as the functions engage or interact with one another. One function might wrap some or all of the others. Across time is a vector of events that can entrain the functions. This might be visualized as changes in the speed of rotation of a circle. Events might also increase the emphasis that a particular work system places on the function, which might be visualized as increasing the radius of a circle and hence altering the overlaps. Although notional as a means for visualizing, we propose that such a ‘‘dance of rings’’ can serve as a means to represent actual scenarios, that is, a process tracing description of macrocognitive work across a given work episode. We now provide the examples from the anomalous STS-76 Space Shuttle mission.
2 Overview of 76th Space Shuttle (STS-76) mission In the following analysis of the Shuttle mission, we have provided ‘‘clean’’ examples of the macrocognition
Fig. 1 Visualization framework of macrocognition functions
functions by separately describing elements of the interrelated fabric to where only elements related to one function are included. In the actual audio communications (usually conducted over the voice loops system described in Patterson et al. 1999), all of the functions were interleaved and co-occurring. In addition, we have spelled out or eliminated acronyms and removed pauses and filler words (um, uh). As described in Patterson and Woods (2001), the 76th Space Shuttle mission (STS-76; March 1996) was flown primarily to place a United States astronaut on the Russian space station. Additional workload was anticipated for this mission, and therefore, a NASA Johnson Space Center mechanical systems engineer position (Mech) was staffed for the entire mission, rather than just the busy periods of ascent and entry. This allowed a human factors researcher (EP) to observe this position for the entire mission in an unobtrusive fashion, since the position was staffed in a socalled back room, physically removed from the ‘‘front room’’ mission control center. In addition, an additional human factors researcher (Jennifer Watts-Perotti) was called into observe the supervisor of all of the mechanical systems (MMACS) on the Space Shuttle in the front room after a significant anomaly was observed during the ascent period. A significant anomaly was detected by the Mech controller during ascent: a hydraulic leak on the third Auxiliary Power Unit. Most of the examples provided below deal with repercussions from this anomaly. 2.1 Macrocognition function #1: detecting problems Two unexpected events were detected by the Mech controller during ascent: (1) a freeze in the Water Spray Boiler (WSB) that cools the third Auxiliary Power Unit and (2) a hydraulic leak on the third Auxiliary Power Unit. The first event, the Water Spray Boiler freeze-up, was detected by Mech based on reviewing his data screens, and was reported by him to his MMACS. During a post-ascent interview, the Mech controller explained that this event was relatively frequent and usually the boiler would work nominally once the coolant warmed up, so no immediate action was required and no one else needed to be informed. The second event was relatively easy to detect, but it was harder to definitively diagnose (sensemaking) and triggered an escalation of re-planning, deciding, and coordinating for the remainder of the mission. Mech detected that the reservoir quantity of hydraulic fluid on the third Auxiliary Power Unit was steadily decreasing over time and that the trend was different than the two other units. The relevant portion of the communication between the mechanical systems controller (Mech), his supervisor,
123
Cogn Tech Work
and the overall MMACS for the mission controllers (Flight Director) is (to see a detailed transcription that includes all of the verbalizations in interleaved fashion, see Patterson and Woods 2001): MECH (to MMACS):
MMACS (to Mech):
Mech (to MMACS):
MMACS (to Mech): MMACS (to Flight Director):
Reservoir quantity in System Three has steadily been decreasing. Going to keep a close eye on that Could you get me a rate on that? That’s pretty serious It’s only showing up point three percent a minute. But it is continuing to go down We’re showing point six We’re looking at what may be a small hydraulic leak. A small hydraulic leak on System Three
2.2 Macrocognition function #2: sensemaking The sensemaking activities triggered by detecting the hydraulic leak anomaly are grouped by increasingly longer time horizons based on anticipated impacts to plans: (1) prior to completing ascent, (2) prior to docking with the space station, and (3) prior to the entry. The tight coupling between sensemaking and planning activities and their relationship to time horizons are represented in Fig. 1 by having the sensemaking and replanning circles bolded and overlapping and constrained by the timeline. 2.2.1 Time horizon 1: prior to completing ascent: assess flammability risk Sensemaking activities concluded in less than a minute via a series of short communications over the voice loops. Interleaved with the above communications were the following statements related to the sensemaking function: MECH (to MMACS): MMACS (to Flight Director):
Auxiliary Power Units are looking real good Leak rate of less than one percent
No action was warranted to reduce flammability risks prior to completing ascent. Although some difference between the two mechanical systems controllers regarding the estimate of the amount of the leak (0.3% vs. 0.6%) was noted, obtaining a better estimate was delayed since it was not judged important to deciding about what actions to
123
immediately take. This judgment was implied by Mech’s statement to MMACS that: ‘‘Auxiliary Power Units are looking real good.’’ Therefore, the conclusion of this first stage of sensemaking was the report from the MMACS controller to the Flight Director over the voice loop system (thus informing all of the NASA Johnson mission controllers and engineers): ‘‘Leak rate of less than one percent.’’ 2.2.2 Time horizon 2: prior to docking with the space station: assess whether leak was increasing Once the orbit phase had been reached, there was a desire to improve the accuracy of the initial assessment of the leak rate, for a number of reasons. An initial meeting between the mechanical systems controllers and engineering personnel responsible for the design and maintenance of the shuttle aircraft across missions was held to assess changes to the risk profile with the hydraulic leak. During this meeting, it was discovered that the mechanical systems controllers suspected that the leak rate could have been increasing since hydraulic systems are sensitive to thermal effects, whereas engineering personnel assumed a stable leak rate and no leaks in the low-pressure condition used during orbit. Both teams subsequently did relatively timeconsuming detailed analyses of the leak rate using different methods. The mechanical systems controllers plotted fuel quantity based on telemetry data that were displayed on their screens during ascent, with thermal effects removed, and the engineering personnel used a mathematical modeling tool to calculate the rate based upon a set of assumed parameters derived from telemetry data. Both analyses suggested a constant leak rate during ascent (More detailed descriptions of the particular analyses used and the content and outcome of a series of five meetings during the orbit phase of the mission between mechanical systems controllers and the engineering community are available in Watts-Perotti and Woods (2007, 2008). Another sensemaking activity involved determining whether hydraulic fluid was continuing to leak or whether the leak had already been effectively isolated. The engineering team assumed that there was no leak when the hydraulic system was in a low-pressure configuration (which it was during orbit), and, although the mechanical systems controllers agreed that this was the most likely hypothesis, they were unsure whether they could rule out the more dangerous hypothesis of continued leaking during orbit. Since there was no obvious way to obtain more information to make a definitive assessment and no large impact to re-planning for this mission based on this difference in assessment, there was no resolution during the period of observation.
Cogn Tech Work
2.2.3 Time horizon 3: prior to entry: determine location of leak
Sensemaking and re-planning were tightly coupled in this example, meaning that the sensemaking activities were conducted to inform re-planning activities and that replanning activities were based largely upon tentative conclusions derived from sensemaking activities. In addition, the same time horizons provide natural groupings of replanning activities: (1) prior to completing ascent, (2) prior to docking with the space station, and (3) prior to the entry.
Station did not need to be canceled to reduce the risk of contaminating the station with hydraulic fluid. In order to mitigate the risk, an early suggestion was made to close the vent doors immediately prior to docking. These doors are normally left open during orbit to allow oxygen to escape prior to entry, thereby reducing flammability risks during entry. This idea was generally dismissed as being not particularly helpful in reducing contamination by both the mechanical systems controllers and engineering. Therefore, the re-planning process ended with no changes to plans for docking with the space station. Interestingly, immediately prior to docking, a quick cycle of re-planning was conducted that resulted in overturning the decision not to close the vent doors. Parallel macrocognition activities conducted separately by the Russian Space Agency led them to a conclusion that the risk of contamination was sufficient to justify closing the vent doors prior to entry. Therefore, the mechanical systems controllers were taken by surprise by being asked to change the docking plans at a late time. Although there were no negative consequences from this last-minute replanning, it was judged to be avoidable and undesirable.
2.3.1 Time horizon 1: prior to completing ascent: no changes to plans
2.3.3 Time horizon 3: prior to entry: do not use compromised Auxiliary Power Unit during entry
In the relevant communications (presented below) there was no discussion between Mech and his MMACS about changes to plans prior to completing ascent, presumably because there was a common understanding that there would be none. When the Flight Director responded to the report of the leak rate being less than one percent, his response indicates the shift from sensemaking to re-planning activities. Specifically, his response simultaneously verified that there were no changes to plans prior to completing ascent, which is signaled by the Main Engine CutOff, as well as confirmed that attempting to isolate the leak was part of post-ascent planning, which would impact other NASA Johnson controllers plans.
In the final meeting between mission controllers and engineers to finalize changes to entry plans, the mechanical system controllers recommended using the compromised Auxiliary Power Unit during entry in case another Auxiliary Power Unit failed, since likely the compromised system could still provide some capability, and flammability risks were minimal. The engineering team pointed out that the flight rules on which this recommendation was based (which were generated prior to this mission) did not take into account risks due to pressure. They suggested that since the location of the leak was unknown, pressure could be released in a way which could damage nearby equipment during entry. The plan was changed not to turn on the compromised Auxiliary Power Unit prior to entry (unless another Auxiliary Power Unit became compromised).
In another meeting between mechanical systems controllers and engineers to finalize changes to entry plans, the engineering team pointed out that the location of the leak was unknown. The mechanical systems controllers agreed that the exact location was unknown, although they pointed out that the leak appeared to be isolated when a valve was closed so there was some information available about the location of the leak. 2.3 Macrocognition function #3: re-planning
MMACS (to Flight Director): Flight Director (to MMACS):
MMACS (to Flight Director):
Leak rate of less than one percent You’re looking at [isolating the thrust vector control] after Main Engine Cut-Off? Yes
2.3.2 Time horizon 2: prior to docking with the space station: no changes to plans Based upon the assumption that the leak rate was constant and that any leak during orbit would be small, there was quick agreement that docking with the Russian Space
2.4 Macrocognition function #4: deciding Deciding was sometimes conducted in relation to planning, but not always. In Fig. 1, deciding is represented as potentially occurring in relation to any of the other macrocognition functions. The visual representation of deciding is different from the other functions in that it is the only one that has a strong temporal component. While being careful to avoid the trap of seeing decisions as ‘‘final points,’’ one also notes that deciding is strongly influenced by its relationship to temporal landmarks—triggering
123
Cogn Tech Work
events—and also that there are commitments to actions, or contingent actions. In some domains, sensemaking, deciding, and re-planning have been described as ‘‘overtaken by events’’; there is even an acronym for it: OBE (White 1981). Some deciding activities were relatively easy for the mechanical systems controllers to conduct in response to detecting the hydraulic leak and in changing existing plans and were not reopened for examination again. For example, it was decided to minimize circulation pump operations that used the leaking Auxiliary Power Unit and to use a circulation pump instead of an Auxiliary Power Unit to check the flight control system. This gave slightly less diagnostic information but reduced the risk of losing another Auxiliary Power Unit system capability. In contrast, deciding about whether to close the vent doors was strongly influenced by the temporal landmark of docking with the Russian Space Station. Also, the level of uncertainty and commitment with relation to the future decision varied dynamically in relation to the temporal landmark. Eventually, one day prior to docking, the NASA Mission Control teams decided not to close the vent doors since it would not provide much protection to the space station and might introduce other risks such as being unable to open them again later. This decision appeared to be vouchsafed by the Russian Space Agency in that a representative stated that they were ‘‘90% go on docking’’ during a conference call the day prior to docking, and no mention was made of closing the vent doors. Nevertheless, 10 min prior to the docking, the Russian Space Agency requested closing the vent doors and was surprised at how long it took the mechanical system controllers to do so. During the shift change handover, the Mech controller described the incident as ‘‘In the unlikely event that we do it, I didn’t want to be stumbling around, and then all of a sudden we’re doing this…’’ Therefore, although deciding by the mechanical system controllers and engineering community readily resulted in a consensus about how to understand the situation and execute the mission, there was a reversal in the decision by other stakeholders in the larger organization, and this caused a delay in the docking timeline. Deciding about when and where to land was interesting in that a fairly definitive set of decisions were made that were overtaken by events during the actual entry. For example, it was decided to land a day early. Yet as the actual entry commenced, the mission was extended by a day, due to concerns about the weather patterns at the landing site. Therefore, the mission extension resulted in returning to the original mission time line for entry. When the entry was again attempted the next day, at the last minute, the landing site was changed, again due to uncertainty about weather conditions. The Shuttle landed at
123
Edwards Air Force Base (in California, where weather conditions were favorable). At no time during our observations about changes to plans did we observe the possibility of landing at Edwards Air Force Base discussed. Nevertheless, we learned that all mission controllers were aware that Edwards Air Force Base was always fully staffed during the entry phase in order to serve as a contingency landing site with more reliable weather patterns. Although this contingency plan was viewed as undesirable due to additional costs associated with ground transportation of the shuttle from California to Florida, this plan had been executed in the past. The additional risks with only using two Auxiliary Power Units during landing likely contributed to this real-time decision. 2.5 Macrocognition function #5: coordinating As depicted in Fig. 1, coordinating is a primary function that seems to ‘‘wrap’’ all of the other macrocognition functions in the sense that it is continuous across the entire episode and interacts with all the others. All of the above functions necessitated coordinating at some level including: •
•
•
•
Detecting Problems: The Mech controller and MMACS controller verified in real-time with two unexpected events during ascent. Sensemaking: The Mech controller and MMACS controller assessed the leak rate was too low to pose a flammability risk. The mechanical system controllers and engineering team jointly determined the leak rate during ascent did not increase and the exact location of the leak was unknown. Re-planning: The Mech controller and MMACS controller decided not to change ascent plans. The mechanical system controllers and engineering team jointly determined not to change the plans for docking with the space station and to avoid using the compromised Auxiliary Power Unit during entry. Deciding: The mechanical system controllers and engineering team jointly decided to use a circulation pump rather than an Auxiliary Power Unit to check the flight control system; the Russian space agency overturned the MMACS/engineering decision not to close the vent doors prior to docking.
In addition to the role that coordinating has in mediating the other macrocognition functions, coordination has its own purposes, and these also are illustrated in the Shuttle mission case. Members of teams engaged in coordination and communication activities with members of other teams in order to synchronize and sequence their activities, share resources, or request expert opinions. Team members need to remain aware of how interruptible other personnel would
Cogn Tech Work
be prior to interacting. They must utilize knowledge of common ground during communications, have a leader prioritize goals and activities when there are conflicts and resource mismatches, and engage in collaborative crosschecking activities to insure safety and progress toward mission success.
3 Conclusion This Space Shuttle mission case study shows how a set of macrocognition functions are continuous, occur in parallel, and also interact across an unfolding scenario. We have proposed a visualization scheme for showing how macrocognition functions are dynamically interacting and integrated across the course of unfolding scenarios. This may be a useful aid for researchers conducting the description and thematic analysis of messy, interrelated observational and interview data of ‘‘in situ’’ cognitive work. Acknowledgments We thank Drs. David Woods, Gary Klein, Emilie Roth, and Nancy Cooke for a number of stimulating discussions about macrocognition and macrocognitive functions, from which many of the ideas presented in this article originated. A colleague, Dr. Jennifer Watts-Perotti, conducted ethnographic observations and interviews during the STS-76 mission and analyzed many of the examples described in this paper. Numerous colleagues also contributed to the elicitation and understanding of other examples used in prior versions of this paper, including David Woods, Marta Render, Richard Cook, Patricia Ebright, Stoney Trent, Martin Voshell, and Jennifer Watts-Perotti. This research was partially supported by the Air Force Research Laboratory (S110000012) and the Office of Naval Research (N00014-08-1-0676). The views expressed in this article are those of the author and do not necessarily represent the view of the air force or the navy.
References Hoffman RR, Yates JF (July/August, 2005) Decision(?)-Making(?). IEEE Int Syst 22–29 Hoffman RR, Norman DO, Vagners J (2009, May/June) Complex sociotechnical joint cognitive work systems? IEEE Int Syst 82–89 Klein G (2007a) Flexecution as a paradigm for replanning, part 1. IEEE Int Syst 22:79–83 Klein G (2007b) Flexecution, part 2: understanding and supporting flexible execution. IEEE Int Syst 22:108–112 Klein B (2010) Macrocognitive measures for evaluating cognitive work. In: Patterson ES, Miller J (eds) Macrocognition metrics and scenarios: design and evaluation for real-world teams. Ashgate Publishing, Farnham, pp 47–64 Klein G, Ross KG, Moon BM, Klein DE, Hoffman RR, Hollnagel E (2003) Macrocognition. IEEE Int Syst May/June:81–85 Patterson ES, Woods DD (2001) Shift changes, updates, and the oncall model in space shuttle mission control. Computer supported cooperative work. J Collab Computing 10(3–4):317–346 Patterson ES, Watts-Perotti J, Woods DD (1999) Voice loops as coordination aids in space shuttle mission control. Computer supported cooperative work. J Collab Computing 8(4):353–371 Patterson ES, Miller JE, Roth EM, Woods DD (2010). Macrocognition: where do we stand? In: Patterson ES, Miller J (eds) Macrocognition metrics and scenarios: design and evaluation for real-world teams. Ashgate Publishing, Farnham. ISBN 978-07546-7578-5 Watts-Perotti J, Woods DD (2007) How anomaly response is distributed across functionally distinct teams in space shuttle mission control. J Cogn Eng Decis Mak 1(4):405–433 Watts-Perotti J, Woods DD (2008) Cooperative advocacy: an approach for integrating diverse perspectives in anomaly response. Comput Support Coop Work 18(2–3):175–198 White H (1981) The Narrativization of Real Events. Crit Inq 7(4):793–798 Woods DD, Hollnagel E (2006) Joint cognitive systems: patterns in cognitive systems engineering. Taylor & Francis, UK
123
