Ergonomics

This article was downloaded by:[University of Wollongong] [University of Wollongong] On: 1 June 2007 Access Details: [subscription number 773326362] Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Ergonomics

Publication details, including instructions for authors and subscription information: http://www.informaworld.com/smpp/title~content=t713701117

Situational awareness ability and cognitive skills training in a complex real-world task To cite this Article: O'Brien, K. S. and O'Hare, D. , 'Situational awareness ability and cognitive skills training in a complex real-world task', Ergonomics, 50:7, 1064 - 1091 To link to this article: DOI: 10.1080/00140130701276640 URL: http://dx.doi.org/10.1080/00140130701276640

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf This article maybe used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material. © Taylor and Francis 2007

Downloaded By: [University of Wollongong] At: 04:01 1 June 2007

Ergonomics Vol. 50, No. 7, July 2007, 1064–1091

Situational awareness ability and cognitive skills training in a complex real-world task K. S. O’BRIEN*{{ and D. O’HARE{ {Department of Psychology, University of Otago, P.O. Box 56, Dunedin, New Zealand {School of Health Sciences, University of Wollongong, NSW 2522, Australia

Successful performance in complex dynamic environments depends on domain-dependent factors, such as situational awareness (SA). Underlying SA in a domain are domain-independent cognitive abilities in perception, memory, attention and executive control. Individuals with lower underlying ability perform relatively poorly in complex dynamic real-world tasks. The first experiment examined whether cognitive skills training could overcome limitations in underlying SA ability that impact on complex dynamic task performance. Participants were taught a mix of cognitive management strategies (e.g. divided and focused attention and visual search) in a simulated air traffic control task. A second experiment investigated the link between underlying SA ability, TRACON and SAGAT, a widely used measure of domain-specific SA. In a third experiment, the focus was on encouraging participants to plan ahead and consider the interrelations of elements (aircraft) in the environment. Whilst both training methods ameliorated the negative impact that lower SA ability had on complex dynamic task performance, the results of the third study indicated that this may have been achieved through improved planning behaviour. Finally, participants with higher underlying SA ability performed well irrespective of training condition. Keywords: Situational awareness; Cognitive skills; Planning; Training

1. Introduction Driven by technological advances, modern systems increasingly challenge the operator’s perceptual and cognitive, rather than physical, abilities. This increased complexity and dynamism in modern systems, and associated demands on cognitive abilities, has seen the concept of situational (situation) awareness (SA) emerge as an important construct in human factors and applied cognitive research. Accordingly, SA research can be found in an increasing number of complex task domains, which include, but are not restricted to, *Corresponding author. Email: [email protected] Ergonomics ISSN 0014-0139 print/ISSN 1366-5847 online ª 2007 Taylor & Francis http://www.tandf.co.uk/journals DOI: 10.1080/00140130701276640


Situation awareness ability and training

1065

anaesthesiologists (e.g. Gaba et al. 1995), air traffic controllers (e.g. Endsley and Rodgers 1996, Durso et al. 1998), aircraft pilots (e.g. Endsley 1993, Endsley and Bolstad 1994, O’Hare 1997), military personnel (e.g. Federico 1995, Randel et al. 1996), automobile drivers (e.g. Gugerty 1997) and nuclear power plant controllers (e.g. Hogg et al. 1995). Lack of SA in complex environments has been shown to have tragic consequences. For example, Woodhouse and Woodhouse (1995), in an analysis of controlled flights into terrain between 1978 and 1992, which accounted for over 4000 deaths, found that lack of crew awareness, and not skill or procedural failures, was responsible for approximately three-quarters of the accidents reported. In spite of considerable interest in SA from researchers over the last decade and a half, there still remain divergent views on what SA is and how it should be defined, conceptualized and measured (see Dominguez 1994, Gilson 1995). Perhaps the most widely cited definition of SA is provided by Endsley (1995), who defines SA as: ‘The perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future’. Endsley (1995) accompanies her definition with a hierarchical model of SA based on the three major components identified within it: perception (level 1 SA); comprehension (level 2 SA); and projection (level 3 SA). Level 1 SA can be seen as the building blocks of SA. In order to achieve SA, the operator must first perceive the relevant elements from the environment. In effect this level of SA refers to the data detection and collection aspect of SA. Level 2 SA is the cognitive phase said to be responsible for cementing together the disparate elements (building blocks) acquired from the environment in order to comprehend the present situation. Endsley (1995) considers level 3 SA to be the highest level of SA and is defined by the ability to predict the forecast future system states and plan ahead based on the comprehension and understanding of elements in the present. Despite her theoretical definition, including cognitive processes, such as perception, comprehension and prediction, Endsley considers SA to be a picture, ‘product’ or mental model of the situation and not the cognitive processes underlying and supporting it per se. Companion et al. (1990) offer an alternative perspective on SA to that offered by Endsley (1995). Companion et al. (1990) suggest that SA is: ‘The ability to extract, integrate, assess and act upon task-relevant information is a skilled behaviour known as ‘‘Situational Awareness’’’ (cited in Dominguez 1994). Companion et al. (1990) infer that SA is the ‘process’ of making sense of the environment and stimuli within it, through the ability to coordinate various cognitive activities thought central to complex dynamic tasks (e.g. perception, attention, working memory). 1.1. Abilities underlying situation awareness Despite conceptual (process vs. product) and measurement differences, most researchers agree on a general grouping of cognitive processes or activities thought to underlie the attainment and maintenance of SA (Endsley 1995, Durso and Gronlund 1999). These cognitive processes include attention (divided and focused), memory (short- and long-term working memory), perception (pattern recognition/matching, visual search, processing speed), time-sharing, spatial ability and potentially executive control (see Goettl 1997). Since SA is dependent on, and indeed resultant from, several underlying cognitive processes, and importantly their amalgamation under complex dynamic conditions (Hardy and Parasuraman 1997), it seems logical to examine SA from this perspective. The WOMBAT-CS (complex operator’s) situational awareness and stress tolerance test was designed (Roscoe and Corl 1987, Roscoe 1993) to measure individual ability to


1066

K. S. O’Brien and D. O’Hare

‘search for, evaluate, and integrate information about all relevant events, conditions, and resources, quickly assess changes in situational priorities, and allocate attention accordingly’ (Roscoe et al. 2001). In effect, WOMBAT may be seen as a measure of SA ability. That is, a measure of the ability to combine underlying cognitive process thought to support and result in awareness of the situation and complex dynamic task performance. WOMBAT is considered to be a relatively culture- and domain-free measure that requires coordinated performance on four cognitively demanding tasks under varying priorities (Roscoe et al. 2001). The four component tasks of WOMBAT include a primary and ongoing psychomotor task (dual tracking), along with three ‘bonus’ tasks, including a spatial processing task (3-D rotation), skilled perception task (quadrant location) and a working memory task (two-back serial digit cancelling). WOMBAT is presented on a computer and monitor and is operated via a specifically designed control panel. While WOMBAT contains most of the cognitive activities thought to underlie SA, its true worth as a SA ability measure lies not just in the need to perform these tasks well individually, but in the need to manage these tasks in concert with one another under dynamically changing conditions and tight time constraints. WOMBAT has been shown to be predictive of performance in complex dynamic realworld tasks requiring high levels of SA (O’Hare 1997). Further, O’Hare (1997) showed that WOMBAT SA scores were able to distinguish between elite pilots (as determined by world championship gliding performance) and highly (if not equally) experienced pilots. O’Hare and O’Brien (2000) found that WOMBAT SA ability scores were also largely independent of age and previous computer or computer game experience. What is most striking about these studies is that overall WOMBAT SA scores were not well predicted by individual cognitive abilities thought to underlie SA, namely, visual search, pattern recognition, working memory and visual imagery (O’Hare 1997). These findings are consistent with reports that traditional test batteries, designed to assess basic cognitive abilities, regularly fail to account for more than 25% of the variance in performance for real-world tasks requiring high levels of SA (Roscoe et al. 2001). This suggests that it is not the ability to perform cognitive activities individually that best predicts performance, but the ability to coordinate and integrate these cognitive activities under temporal constraints and changing priorities. This suggestion is consistent with more recent theories of performance in complex dynamic environments (Klein et al. 1993). 1.2. Training cognitive skills Given that SA ability, as measured by WOMBAT, represents the ability to attain high levels of SA in complex dynamic real-world tasks (O’Hare 1997, O’Hare and O’Brien 2000, Roscoe et al. 2001), it then begs an obvious question: Can training of higher order cognitive skills overcome individual limitations in SA ability that impact on SA and performance? This question is not new. Gaba et al. (1995) suggest that: ‘Perhaps the key question for all fields is whether situation awareness abilities can be identified in a person and the degree to which these abilities can be taught’. Endsley (1989) and Endsley and Robertson (2000) suggest that training aimed at improving the cognitive skills thought to underlie SA (e.g. attention sharing, contingency planning) should result in an increased capability to attain and maintain SA. Endsley and Robertson’s (2000) acknowledgement of the role of planning in SA is timely. Several researchers have begun exploring the role of planning in complex dynamic



1067

tasks (e.g. air traffic control (ATC)) said to require high levels of SA (Taylor et al. 1997, Gronlund et al. 2001, 2002, Moertl et al. 2002). However, these approaches tend to focus on tools and system designs that aid planning and SA, but do not directly address the training of these cognitive skills and abilities. Despite a paucity of research on higher-order cognitive skills training for SA, particularly with regard to attention management and planning behaviour, there is a reasonable base of research to suggest the potential utility of attention management training. However, there have been only a handful of studies examining the strategic deployment or control of attention (e.g. Logan, 1985, Gopher et al. 1989, 1994, Gopher 1993, Vidulich et al. 1996). Work by Gopher and his colleagues provides the most encouraging evidence for the potential utility of attention management training. North and Gopher (1976) and Gopher et al. (1989) explored two core issues relating to management of attention: How effective are people at controlling their attention and can training improve attentional control? Gopher et al. (1989) developed the ‘emphasis change’ training technique in order to facilitate learning on a complex dynamic task. This technique involved directing participants to allocate attention to a specific and important component of the task, while still attending as much as possible to all the elements of the task as a whole (i.e. without detriment to the specified component). Building upon the emphasis change training technique, Gopher et al. (1994) explored the utility of the emphasis change training, and a variant technique, on a group of Israeli Air Force flight cadets in their early stages of flight training (10 h flying light aircraft and pre-jet trainer transition). Gopher et al. (1994) explored two skill-training techniques. The first technique was virtually identical to the emphasis training approach detailed in Gopher et al. (1989). The second was a hybrid technique incorporating facets from Gopher et al.’s (1989) emphasis change technique and Fredrickson and White’s (1989) hierarchical part-task approach. The hybrid emphasis change technique involved training of several individual component tasks during the initial period of each of the first six training sessions. For the remaining period of each of these sessions, participants were trained under the standard emphasis change procedures. Additionally, participants were verbally given tips or strategies for performing task segments (e.g. ‘Do not leave your path to chase mines; wait for them to approach you’; Gopher et al. 1994). Gopher et al. (1994) found that, following training, both training groups performed significantly better in several flight performance tests than a matched sample of pilots who had received no attention training. The present research explores whether variants of the training techniques developed by Gopher et al. (1989, 1994) and others (e.g. Fredrickson and White 1989) can ameliorate the negative impact that low SA ability has on complex dynamic task performance. The first experiment examines whether training of an assortment of higher order cognitive strategies, such as attention management, and goal-directed strategy development (i.e. planning) can ameliorate the negative impact that low SA ability has on task performance in an ATC simulation.

2. General methods 2.1. The WOMBAT system The WOMBAT system consists of a computer program and associated control panel that is adaptable to any modern PC. Participants interact with the WOMBAT task via a control panel consisting of two joysticks and a 13-button keypad. The two joysticks are


1068


used primarily for a dual-tracking task and each controls a separate component of the dual tracking task. Ten numbers (0–9), a ‘bonus’ button and a left and right arrow make up the keypad. The keypad is primarily employed during bonus tasks. The dual tracking task requires participants to track, with the left joystick, the vertical side of a randomly expanding and contracting hexagon, while simultaneously using the right hand joystick to track a roaming circle. The dual tracking task has two alternating modes of operation, velocity mode and acceleration mode. When in velocity mode, the joysticks control tracking movement in direct proportion to the amount of movement of the joysticks. However, in the acceleration mode, the tracking movement gains momentum and speed in proportion to the degree of movement of the joysticks. In velocity mode, centring the joysticks results in no tracking movement whereas in acceleration mode, centring results in the symbols continuing to move at a constant rate. When accurate tracking in the dual tracking task is achieved, a trigger button on the right joystick can be used to engage an autotrack function, which is similar to the autopilot in an aircraft. While in autotrack, participants have the option of completing one of three 60 s bonus tasks; however, participants must ensure that the autotrack for the dual tracking task is functioning optimally at all times. The autotrack function is prone to frequent failure, although there are two modes of failure. One mode of failure sees the tracking performance deteriorate rapidly and dramatically, whereas the other failure mode sees the tracking performance deteriorate slightly. This creates a ‘trade-off’ situation, in which the operator must decide whether to correct the auto track or pursue bonus tasks in order to counter the loss of tracking points. The autotrack can be monitored during bonus tasks via a small indicator window in the top left-hand corner of the bonus screen. Although controlling the dual tracking task is the primary responsibility, in order to achieve high performance scores, participants must frequently attempt to complete the bonus tasks while ensuring good tracking performance at all times. 2.1.1. Bonus tasks 2.1.1.1. 3-D figure rotation. Two 3-D block figures (derived from Shepard and Metzler 1971) are presented alongside each other on the screen. Each figure can be individually selected and manipulated by the use of both joysticks. The aim of the task is to view the figures, by rotating them on all three axes, in order to determine if the two figures are the same, different or mirror images. The quicker the task is completed correctly, the more points are earned. If the task is completed correctly within 60 s, further 3-D pairings are offered, thus allowing more points to be scored within the 60 s. If a wrong response is given, then no points are earned and no further 3-D pairings are offered within the 60 s bonus period. 2.1.1.2. Quadrant location. During this bonus task, the screen is divided into equal quadrants in which the numbers 1–32 are dispersed equally. The aim is to eliminate each number from the screen in ascending order by pushing the button on the keypad that corresponds to the quadrant in which the number is present. If correctly located and indicated, the number will disappear from the screen and the next number in ascending order must be located. At initial glance, the numbers appear to be randomly, yet equally, dispersed across the quadrants. However, a pattern in the dispersal across the quadrants is present and may be detected, in time, by participants, thus allowing them to predict the



1069

location of the next numbers in the order. If performance in locating the numbers is exceptional and is completed well within the 60 s bonus task time, then WOMBAT infers that the pattern of dispersal has been learnt and will present a new and more difficult pattern, which is worth increased points when played. It is not unreasonable to equate this bonus task to a combination of a visual search and a pattern recognition task under time pressure. 2.1.1.3. Digit cancelling. Perhaps better titled ‘two-back cancelling’, this task involves remembering and indicating on the keypad which number was presented two items before the most recently presented digit. Only numbers 1–8 are used and these are presented individually and randomly in the centre of the screen for approximately 1.5 s before disappearing. At the beginning of the task, a number is briefly presented for recall. Another number quickly follows, providing two numbers that are to be remembered. When, however, a third number appears briefly, a keyboard with the eight numbers displayed will also be presented permanently on screen and indicates to the participant that they are required to push the number on the keypad that corresponds to the number presented two numbers before the most recently presented. A new number will then appear briefly and once again the participant is required to indicate the number presented two before the most recent. This process repeats for the entire 60 s and will continue, regardless of whether the right or wrong number is entered. The worth of the task is decreased if wrong responses are made or time is wasted. 2.1.2. Autotrack failures. A complicating factor in the successful completion of bonus tasks is the frequent failure of the autotrack facility. When autotrack fails, performance on the dual-tracking task begins to deteriorate to the point where, in order to stem the loss of points, the participant must leave the bonus task to fix the autotrack, usually before the 60 s bonus time is complete. However, the 60 s timer runs down only when the bonus task is being played. Therefore, when the participant returns to the bonus task, whatever time was remaining before fixing the autotrack, will still be present when returning to the bonus task. The quadrant location task requires participants to either recall or re-establish the location of the next number to be located. For example, if they leave the quadrant location task after locating the number 16, when they return they must find where the number 17 is located. If they can hold the number 17 and location in working memory while fixing the autotrack, then they are at an obvious advantage when returning to the quadrant location task. Those who are unable to achieve this need to scan the numbers in each quadrant in order to establish what the next number in ascending order is and where it is located. Again, points are deducted for incorrect and slow responses. The digit-cancelling task is even more demanding on memory and task management, following an autotrack failure and correction. The digit-cancelling task requires participants to either recall the two numbers that were presented prior to the failure or to re-establish the sequence by making two guesses of the number sequence, but remember the new numbers presented, until they can then know with some certainty the correct sequence. 2.2. WOMBAT testing procedure The present research used two versions of WOMBAT; version 4.0 and 4.9. Version 4.0 differs slightly from version 4.9 and the description of WOMBAT above, in that the 3-D


1070


rotation task only allows one problem to be solved within the 60 s period. The 3-D task also differs in that after participants decide whether the 3-D pairings are the same, different or mirror image, they then indicate how ‘certain’ or ‘uncertain’ they were of their decision. If the participants indicated they are certain about their decision, they earned maximum points if correct. If participants indicated they were uncertain and are correct, they earned fewer points than if certain and correct. However, if participants indicate that they were uncertain and they were incorrect, then the participants were offered another chance to make the correct response. The top score that could be obtained in WOMBAT version 4.0 was 300; however, this was never reached. Version 4.9 has no ceiling imposed on scoring. Training and testing procedures for WOMBAT were standardized for all experiments within the present study. All experiments were carried out in a noise- and light-controlled environment. Participants were seated in front of the WOMBAT console and 15-inch VGA monitor, which were connected to a PC-General 486DX-50 providing a screen resolution of 8006600 with 256 colour graphics. WOMBAT training took approximately 35 min and was followed by a specifically developed 15-item questionnaire directly related to the critical aspects for successful operation of the task. The questionnaire covered all trained aspects of the task. If participants failed to provide a correct answer, further instruction related to the specific task question was provided and the participants then re-tested. A 40-min uninterrupted testing session was carried out immediately after the training and competency-testing phase had been completed. The WOMBATTM training and testing phase of the experiments was completed within 90 min. WOMBAT provides the experimenter with performance measures for each of the subtasks (dual tracking, 3-D figure rotation, quadrant location and digit cancelling) as well as an overall final performance score. In addition to a final overall performance score for WOMBAT, scores for each 10 min block of testing time are also provided. This allows an examination of learning and performance blocks of time, although performance appeared to be stable after 30 min of testing, with no significant difference in scores (p 4 0.05) between the 30 and 40 min blocks. Although WOMBAT provides data on the individual components of the task, they were not individually assessed within the present study because the score that can be obtained on one component is dependent on the performance of the other components or performance of the task as a whole. 2.3. TRACON Terminal Radar Approach Control simulation TRACON is a PC-based ATC simulation that embodies an extremely high degree of realism. Versions of TRACON (Wesson 1990) have been used extensively in both research (Ackerman 1992, Ackerman and Kanfer 1993, Ackerman et al. 1995) and training environments (customers include the National Aeronautics and Space Administration and the Federal Aviation Administration). Version 1.2, a commercial game version, was used in the present studies. TRACON allows researchers extensive control over the simulation content and demands and is therefore extremely useful within a laboratory setting. Although the game version yields less data in terms of the program’s output, TRACON performance is video-recorded to allow full analysis of task performance, including errors made, commands made and delays accrued. TRACON presents operators with a simulated colour radarscope displaying fix points, airports, boundary markers and the aircraft themselves. Pending and active flight stacks containing aircraft flight strips and related flight information are displayed alongside the



1071

radarscope. A communications box is also presented at the bottom right of the screen. This communications box presents a recent history of communications between the aircraft and operator in text form and allows the operator to confirm aircraft requests and commands given. The researcher has considerable control over simulation dimensions, such as weather conditions, pilot competency, frequency of emergencies, equipment failure and the number of aircraft that will be delivered to the operator over a set period of time. Operators interact with the simulation through the use of a mouse, keyboard and pop-up menus. Although voice commands cannot be given to aircraft in the simulation, audio requests, and feedback from aircraft confirming the commands given and action taken, is provided to the operator through speakers or headphones. The primary role of the air traffic controller is to guide aircraft to their intended destinations safely and efficiently, with ‘safely’ holding the greatest import. In order to meet this primary role, operators must acquire a considerable amount of declarative and procedural knowledge. The knowledge required includes aircraft acceptance, hand-off and landing procedures, the position of navigational fix points and airports, reading and understanding flight strips (intentions and destination), issuance of commands and aircraft separation rules. Optimal performance requires extensive scanning and monitoring strategies, as well as aircraft traffic management procedures (Ackerman 1992). 2.3.1. Initial TRACON training. Initial TRACON training was identical for participants in both experiments. Participants were first presented with a specially developed 22min TRACON ‘general task requirements’ video, which contained a demonstration simulation and verbal description of specific task elements. TRACON was run on a PC General AMD K-5 133 MHz computer with 15-inch VGA monitor providing a screen resolution of 8006600 with 256 colour graphics, along with two Genius speakers, keyboard and mouse. The video, and subsequent hands-on training, instructed participants on the basic task requirements in regard to the function of items presented on the TRACON screen and the function of commands given to aircraft (environmental and operational information). The video and subsequent training avoided presentation of any ATC management strategies. Participants were first trained to understand the task requirements regarding environmental information, such as how to locate and identify the names, position and function of various fix points and airports. Moreover, participants were taught to interpret information related to aircraft type, destination, intentions, call sign, altitude and speed, both from information displayed on flight strips in the pending and active stacks and aircraft on the radarscope. Participants were also taught the task requirements related to operating information, such as command function and use. Procedural requirements, such as final approach altitude and heading for successful hand-off of an airport arrival, were taught, but were also present during all TRACON sessions via a simple information sheet displayed immediately below the computer screen. Throughout training, participants were warned repeatedly to keep minimum aircraft separation of 5 miles or 1000 feet. Participants were also instructed to handle heavy aircraft first (where possible) as they had priority over smaller aircraft and would allow higher scoring. Participants were also told that in order to score well they would have to handle the aircraft in the most efficient manner, that is, with as few delays and commands as possible without compromising safety. TRACON training and testing sessions were carried out on consecutive days. Handson TRACON training was conducted over two sessions with final criterion task testing


1072


taking place immediately after the second training session. Initial hands-on training presented a three-aircraft simulation that required handling of three possible aircraft flight types: arrivals (landings); departures (sector hand-offs); and overflights (en route). This simulation enabled participants to acquire and practise the necessary task knowledge without becoming overloaded. Following the video and initial TRACON training, a 13-item questionnaire was administered to ensure that participants knew the basic procedural requirements for TRACON. Similar to the WOMBAT questionnaire, the questions covered all trained aspects of the task. If participants failed to provide a correct answer, further instruction related to the specific task question was provided and the participants were then re-tested. A five aircraft practice simulation was then provided for participants. Advanced TRACON training procedures are described in the respective experimental procedures to follow. 3. Experiment 1 Previous research (O’Hare and O’Brien 2000) has shown that lower SA ability individuals performed more poorly on the complex real-world task of TRACON than higher SA ability individuals. More specifically, lower SA ability participants, as measured by the WOMBAT test, showed what appeared to be poor attention allocation skill and strategy use, often allowing aircraft to wander into another sector without a hand-off command being issued. Further, lower SA ability participants failed to prevent separation conflicts between aircraft and were on the whole less efficient in their use of commands to control aircraft. It is unclear whether these errors were due to poor executive processing and attention allocation/management skill or simply a result of participants not employing effective strategies for handling aircraft (e.g. projecting ahead and planning). Schneider (1985) and Gopher (1993) have emphasized the importance of attention management skill in complex or multi-task environments, with attempts to improve effective attention allocation skill and associated task performance proving encouraging (Gopher et al. 1989, 1994). However, as noted previously, other higher-level cognitive strategies (e.g. planning) may also play a role in the observed SA ability differences. The present experiment employed a training methodology, which was termed ‘cognitive management’ (CMG), designed to overcome deficiencies in attention management and strategy use in TRACON that are related to underlying SA ability limitations as measured by WOMBAT. The CMG training method was developed following experimenter observations and participant reporting from an earlier TRACON study (O’Hare and O’Brien 2000). It was apparent from this study that attentional control was crucial to good TRACON performance and where absent led to errors. More specifically, low SA ability participants had trouble in switching between focused and divided attention and attending to the appropriate information at the appropriate time. It was noted that low SA ability participants failed to regularly examine all aircraft on the screen, instead tending to focus attention on one aircraft to the detriment of others. This failure to divide attention and scan effectively often led to separation conflicts and failure to hand-off aircraft to another controller when leaving the sector, both critical errors. It was hypothesized that lower SA ability individuals receiving CMG training would show better TRACON performance than lower SA ability individuals receiving standard procedural TRACON training. It was also hypothesized that higher SA ability individuals would do well regardless of which TRACON training condition they received.



1073

3.1. Method 3.1.1. Participants. A total of 28 students recruited via the University of Otago student job search scheme were paid NZ$30 to participate in the experiment. Ages ranged from 18–36 (mean 22.5, SD 5.1) years. All participants had normal or corrected to normal hearing and eyesight. There were 23 females and five males. None of the participants reported having ever encountered WOMBAT, TRACON or any other ATC simulation. A performance incentive of NZ$20 was promised to the four participants with the highest combined scores on WOMBAT and TRACON. 3.1.2. Apparatus. WOMBATTM version 4.0 was used in the present experiment. TRACON version 1.2 and associated hardware, as described in the general apparatus, was employed as the model of a complex real-world task. A PV-500 video splitter and Panasonic Superdrive NV-SDI video recorder (Panasonic, Osaka, Japan) were used to record participants’ TRACON performance for later analysis. 3.1.3. Procedure. The experiment was run over three sessions, with each session taking approximately 90 min. Upon arrival, participants signed an ethical consent form and filled out a human factors questionnaire designed to gather participant profile and demographic data across several independent variables, such as age, gender, academic background, occupation and computer experience. No differences were found between these independent variables on any of the dependent measures (all p 4 0.05). Participants first received TRACON training and testing in sessions 1 and 2 followed 1 week later by WOMBAT testing. The initial TRACON training procedure and task description was identical to that previously described. For advanced training, participants were randomly assigned to either the CMG or procedural training groups. 3.1.4. Cognitive management training. The eight aircraft simulation developed for advanced task training required handling of four departures and four sector entry aircraft. Two of the departures acted also as arrivals and returned for landing at their departure points, with the remaining two departures leaving the sector. The four sector entry aircraft provided two overflights and two arrivals for handling. The primary aim of the CMG training methodology was to provide some structure to participants’ attention management that would encourage participants to divide attention and establish priorities for managing aircraft within the sector. The need to divide attention frequently was made explicit to participants in the first of the CMG training statements (‘To do well in TRACON you do need to focus your attention on one aircraft at a time, but at the same time you also have to be aware of what the other aircraft are doing’). Establishment of priorities for the allocation of attention (selective attention) was encouraged through additional training instructions (e.g. ‘When the simulation resumes, I want you to tell me which aircraft is the most important to deal with next and why?’ ‘With all remaining aircraft, please indicate which aircraft, in order of importance, should be dealt with next?’) Finally, the training method encouraged participants to predict the future status of each aircraft (e.g. ‘Can you foresee any difficulties that may arise later in the simulation? If so, how might you deal with them?’). Five pauses (one every 4 min for the first 20 min of the simulation) were inserted in the CMG training simulation. For the first pause only, participants were given the first


1074


statement on the importance of dividing attention. Immediately after this first pause statement, and for all subsequent pauses, participants were given the additional questions designed to guide visual scanning, strategic division of attention based on priorities and exploration. After the fifth pause, and subsequent probe questions, participants continued the simulation uninterrupted until 30 min of hands-on simulation time had elapsed. Care was taken to ensure that participants received no additional training of procedural activities, such as issuing commands. 3.1.5. Procedural training. The procedural training group (control) practised issuing commands and manoeuvring aircraft around the sector using the same simulations as those used in the CMG training. Both groups received identical training time on the simulations. No other instructions or practice techniques were provided. 3.1.6. TRACON testing. Following advanced training, all participants completed (experimenter and instruction-free) an eight aircraft practice simulation. The eight aircraft simulation comprised four departures and four sector entry aircraft, with all aircraft arriving in the sector within a 12-min period. One of the aircraft departures also acted as an arrival (landing), with the remaining three leaving the sector. Three of the four sector entry aircraft acted as arrivals and one as an overflight. TRACON criterion testing was conducted following completion of this practice simulation. The TRACON criterion task contained eight aircraft entering the screen within a 10min period. The test simulation consisted of four overflights and four arrivals. The four overflights were all initiated as take-offs from airports within the sector. The four arrivals were all initiated from outside the sector. The four arrivals were all destined for the same airport, thus creating a context in which good SA was required in order to successfully handle the simulation and avoid separation conflicts. 3.2. Results 3.2.1. Description of performance. Individual WOMBAT and TRACON performance scores were collected and descriptives calculated. The overall mean WOMBAT score was 212.1 (SD 36.9). It is conceivable that the type of TRACON training received could have had an impact on WOMBAT SA ability scores. To address this possibility, an ANOVA was conducted to compare the WOMBAT scores as a function of training condition. There was no significant difference between training conditions for SA ability scores (F(1,26) ¼ 0.55, p ¼ 0.817). A maximum of 6500 points were available for handling the aircraft in the TRACON simulation. The mean TRACON criterion scores for CMG and procedural training conditions were 5503.6 (SD 617.5) and 5280.0 (SD 1145.2) respectively. The overall mean TRACON criterion test score was 5391.8 (SD 909.9). 3.2.2. Effects of ability and training. Training condition (CMG vs. procedural), SA ability and the interaction between SA ability by training were entered in a single step in a linear regression to predict TRACON criterion test scores. Training was a significant predictor of TRACON test scores, (t ¼ 72.07, p ¼ 0.05), but SA ability was not a significant predictor (t ¼ 71.20, p 4 0.24). However, the effect of training condition is qualified by a marginally significant (t ¼ 1.99, p 5 0.06) interaction. Together the three variables accounted for 31% of the variance in TRACON test scores.



1075

3.3. Discussion The primary aim of the experiment was to assess whether training of CMG strategies could aid the performance of lower SA ability participants in the complex and dynamic real-world task of TRACON. It was expected that lower SA ability participants would perform as well as high SA ability participants on TRACON after receiving CMG training, whilst lower SA ability participants who simply received procedural (control) training would show poorer TRACON performance than the other groups. The results showed that lower SA participants who received procedural training showed significantly poorer performance on the TRACON criterion task than either the higher SA performers or lower SA ability performers who received CMG training. Put another way, CMG training appeared to overcome limitations in SA ability that impact on TRACON criterion task performance. Participants with high SA ability, as a whole, appeared to perform well regardless of the training method they received. The present experiment also examined differences related to the efficiency and handling of aircraft. It was found that participants with low SA ability delayed aircraft for significantly longer than did high SA ability participants. This difference in delays corresponds with, and is perhaps explained by, the significant difference also observed between high and low SA ability participants for the number of commands given during the TRACON criterion task. Lower SA performers, on average, issued 12 more commands than high SA ability performers. What is interesting about this finding is the lack of further correspondence with errors committed. Despite the lower rate of errors committed by low SA ability participants receiving CMG training, compared to low SA ability procedural participants, there was no difference between low SA ability groups receiving either procedural or CMG training on the number of aircraft minutes delayed or commands issued. It appears that both low SA ability groups adopted an inefficient command-oriented approach to handling aircraft as evidenced by the number of commands issued and delays accrued. However, in the case of the low SA ability CMG group, their handling of the simulation, while not efficient, was effective in terms of avoiding conflicts. Taken together, the low SA ability groups’ TRACON performance appears to have differed via a means other than pure handling efficiency. One possible interpretation would be that low SA ability participants in the two training conditions differed in their awareness of the dynamic interrelations of aircraft and associated strategies for dealing with potential problems (planning ahead). While low SA ability participants receiving CMG training may have controlled the sector in a similar command-oriented manner to the procedural training group, they were perhaps more aware of the future status of aircraft in relation to one another. Randel et al. (1996) offered a similar interpretation when examining the difference between expert and novice electronic warfare technicians’ decision-making. Although they found no differences in the use of cues, knowledge and imagery in decision-making, they suggested that experts might instead integrate these cues in a manner that facilitated SA and subsequent performance. Further support comes from Lipshitz and Ben Shaul (1997), who found that expert gunboat commanders not only displayed better all-round information gathering skills from the combat environment, but also sought to understand the potential impact or ‘role’ of other boats in the situation to their own. Taken together, it is reasonable to suggest that the CMG training methodology went beyond encouraging the effective management of attention and evoked higher order


1076


cognitive processes, such as planning and future prediction. This explanation sits well with Endsley’s (1995) definition and conceptualization of SA levels. The CMG training method encouraged the participants to extend their cognitions beyond the mere perception of elements in the environment (level 1 SA), to consideration of aircraft interrelations and potential future states (levels 2 and 3 SA). It was of interest to consider how participants with high SA ability performed well regardless of the type of training they received. Either high SA ability participants possess sufficient cognitive resources to rapidly and continuously acquire information from the environment, which allows them to react quickly to identified problems, or they have sufficient cognitive resources available to take a more proactive approach, perhaps planning ahead in order to avoid errors. While CMG training appears to improve performance, if only modestly, it is unclear through what means this was achieved. The CMG training combined an eclectic mix of higherorder cognitive strategies including attention management and, possibly, planning behaviours. 4. Experiment 2 Experiment 1 showed the benefits of a training technique that encouraged participants to adopt more effective CMG strategies in the complex real-world task of TRACON. However, the effect of training on TRACON performance was qualified by an interaction between SA ability and training, with higher SA ability participants performing well regardless of training. It was not possible, however, to identify what cognitive aspects related to TRACON performance were most affected by the training protocol employed. It was suggested that the attention allocation/ CMG strategies taught helped participants to overcome limitations in cognitive ability (SA ability) that impact on TRACON performance. However, it is not clear if the CMG training facilitated TRACON performance by improving attention allocation skills such as divided/selective attention, visual scanning, etc. or by structuring information acquisition, which was supportive of higher order cognitive strategies such as planning. A combination of both is possible. A more detailed examination of the relationship between SA ability and TRACON performance is needed. The present study explores in more detail the relationship between SA ability, as measured by WOMBAT, and performance in the complex dynamic real-world task of TRACON. Specifically, the present experiment seeks to answer a number of questions. For example, what information is being attended to, what understanding arises from this information and is there clear involvement of higher order cognitive processes based on information acquisition and understanding? Answers to these questions will aid in developing more effective training procedures. To do this, a variation of Endsley’s (1995) SAGAT method is utilized, which mirrors that used by Endsley and Rodgers (1996), in order to identify individual differences in TRACON performance and provide insights into the cognitive processes and strategies that distinguish high and low SA ability participants. SAGAT has been extensively used and validated in a number of domains (Endsley and Garland 2000). It is important to assess the relationship between TRACON performance and WOMBAT, a SA ability measure, with SAGAT a direct SA performance measure, along with the implicit performance measure of SA provided by the embedded conflict situation.



1077

4.1. Method 4.1.1. Participants. A total of 20 students (11 female and nine male) from the University of Otago were recruited via the ‘student job search’ programme and offered NZ$30 for 4 h of experimental participation. Participants who agreed to take part in the experiment were also informed of a performance incentive (NZ$20) to be given to the top four participants as determined by aggregate performance on the three criterion measures. Ages ranged from 20 to 39 (mean age of 25.6 years). 4.1.2. Apparatus. Aside from the computer and video equipment described in the previous experiment, paper and pencil SAGAT questionnaires were used to present questions and record participant answers. The first page of the questionnaire presented participants with a black and white picture of the TRACON radarscope, which was void of all information other than fix points and airports. Participants used this radarscope picture to indicate the position of aircraft that were present on the screen when the simulation was stopped. For each aircraft indicated, participants were asked to answer the following questions as precisely as possible: 1. Please indicate as accurately as possible, the position of each aircraft that was present on the radarscope when the simulation was stopped; 2. What were the altitude, heading and speed of each aircraft indicated? 3. Were the aircraft climbing, descending or level? 4. What were the intentions of each aircraft you have indicated (were they landing, overflights or take-offs leaving the sector)? 5. What were the destinations of the aircraft indicated? 6. What current or future plans for action did you have for each aircraft you have indicated? 7. Were you aware of potential separation conflicts, for any of the aircraft in the near future? 4.1.3. Procedure. The experiment was conducted over four sessions of approximately 1 h each. During the first session, participants signed an ethical consent form and filled in a questionnaire designed to gather demographic data, computer game playing experience, gender and age. Participants were then taken through initial TRACON training identical to that described in the ‘general TRACON training procedure’. The advanced TRACON training was identical to the ‘procedural training’ described in experiment 1, employing the same eight aircraft training simulation as described in experiment 1. A nine aircraft advanced training simulation was also used and was identical in structure to the TRACON criterion task simulation (same number of arrivals, overflights and take-offs). However, the flights were initiated at, and released to, quite different points in the sector than the aircraft in the criterion task and the simulation presented the aircraft over a 12 min period rather than a 10 min period as in the criterion task. Participants completed advanced TRACON training in session 2 and immediately after were tested on the TRACON criterion task. SAGAT measures for TRACON were taken during the third session. Participants received WOMBAT training and testing in session 4. 4.1.4. TRACON performance testing. The test simulation presented nine aircraft within a 10 min period and was structurally more difficult than the eight aircraft training and testing simulations from experiment 1. The test simulation required participants to handle five flights requiring hand-offs to en route controllers. Four of these hand-offs were initiated as take-offs from airports within the sector and one that was an overflight. The four remaining flights to be handled were arrivals (tower hand-offs) with three arriving from outside the sector and one as a take-off from within the sector.


1078


Aside from the obvious increase in workload in receiving and handling nine aircraft within 10 min, the simulation was also constructed in such a way that if SA was poor, a separation conflict would result. More specifically, three of the nine aircraft entered the sector from different fix points at different times and at different airspeeds, but converged on the same airport (LAX) for landing, at the same time and altitude, thus creating a condition for a separation conflict if no action was taken. The rationale for this simulation construction was not arbitrary. Durso and Gronlund (1999) suggest that although general performance measures may not be useful direct measures of SA, implicit performance measures of SA may have some utility. To quote Durso and Gronlund (1999): ‘For example, an experimenter would embed operationally relevant information into a high-fidelity simulation (or that information may be naturally embedded). Discovery of that information would indicate good SA by virtue of the changes in behaviour that such a discovery would naturally involve’. In addition to the need to display implicit SA, the three aircraft convergence aspect of the simulation required participants to establish a plan for handling the aircraft, which met their goals of successfully and efficiently landing the aircraft. This required participants to consider command actions for sequencing, which included adjustments to aircraft speed, heading, altitude adjustments and descent rates in order to maintain separation. 4.1.5. SAGAT task. SAGAT training and testing was carried out in session 4. The present SAGAT method was virtually identical to that described by Endsley and Rodgers (1996); however, it differed in that the simulation was not restarted following the first simulation stoppage and SAGAT questionnaire. The intrusiveness of the freeze and restart technique typical to Endsley’s SAGAT procedures has previously been questioned (Sarter and Woods 1995). Accordingly, a slightly different approach was taken here. After administration of the first SAGAT simulation and questionnaire, a distractor task was given for 10 min, then a new SAGAT simulation was activated and questionnaire administered. Two SAGAT simulations and questionnaires were used to gather data. Prior to SAGAT testing, participants received two practice SAGAT tests that enabled them to become accustomed to the requirements of the procedure. The two SAGAT practice simulations contained nine aircraft, each containing an equivalent number of each flight type (arrivals, overflights and take-offs) and were designed to ensure that participants were handling nine aircraft when the screen freeze occurred. Following the SAGAT practice sessions, participants were removed from the experimental room and presented with a distractor task (map-reading task) aimed at reducing the chance of information from the practice sessions interfering with recall of information in the SAGAT test sessions. The two SAGAT test simulations presented nine aircraft for handling within a 5-min period. The two test simulations had similar workloads, with each containing four arrivals (landings) and five en route hand-offs (departures/overflights). The SAGAT test simulations were stopped when participants had been handling all nine aircraft for at least 1 min but before any aircraft had left the sector. This situation invariably occurred between the 7th and 8th min for the first SAGAT simulation and between the 9th and 10th min for the second SAGAT simulation. It was thought important that all participants had similar exposure and handling time on aircraft to ensure an equal opportunity to recall information during the SAGAT questionnaires. This is consistent with Endsley (1995), who advises to not introduce a query until at least 5 or so



1079

min into the simulation. This allows time for a picture or SA to be built before being assessed. Participants were not informed as to when the screen freezes would occur. Each SAGAT questionnaire took approximately 5 min to complete, with no participant taking longer than 6 min, which Endsley (2000) suggests is the outer time limit for query responses. SAGAT questions 1 through to 5 were aimed at assessing participants’ awareness of ‘operational information’, which is considered synonymous with Endsley’s (1995) level 1 SA. Questions 6 and 7 were designed to probe levels 2 and 3 SA integration and utilization of relevant information (i.e. the degree to which participants were able to plan and make predictions about future states of the simulation). The accuracy of each participant’s responses was assessed using individual print copies of the SAGAT simulation screen at the point at which the screen freezes occurred. Participant responses were coded as either correct or incorrect. When assessing participant responses to aircraft position, heading, altitude and speed, ‘operationally determined tolerance intervals’ (Endsley and Rodgers 1996) were used. The tolerances were: aircraft position þ/75 miles, heading þ/7158: altitude þ=300 feet; speed þ=10 knots: The SAGAT questions relating to planning and separation conflicts required considerable care in the assessment of their accuracy. Two judges individually coded each participant’s planning responses by implementing the suggested plan, for each aircraft, to assess if the plan was appropriate and would be successful/correct or unsuccessful/incorrect in achieving its goals. Altitude, heading, position and speed data, along with the assessment of planning responses, were used to assess the accuracy of potential separation conflicts that were reported by participants. Where a separation conflict was evident but not reported, it was coded as incorrect. The judges were in 100% agreement in judgements of accuracy of plans and possible separation conflicts. Missing responses to SAGAT questions were coded as incorrect. Question 7 was eventually excluded from analysis due to the virtual absence of responses to this question. However, it should be noted that participants did sometimes allude to potential conflicts when answering the question related to planning (question 6). That is, they sometimes advocated, in their plans, a certain course of action in order to avoid possible conflicts in the future. 4.2. Results 4.2.1. Description of performance. Two participants were excluded from the analysis due to equipment malfunctions during testing. Means were calculated for each of the measures from the remaining 18 participants. Computer game-playing experience, gender and age had no impact on either TRACON or WOMBAT scores (all p 4 0.05). The mean WOMBAT and TRACON criterion test scores were 231.5 (SD 73.1) and 2410.5 (SD 5135.2) respectively. The maximum possible score for each SAGAT question was 18, with a possible overall SAGAT score of 144 for the complete SAGAT questionnaire. Mean SAGAT test scores were collated for each of the test questions along with an overall SAGAT test score. However, prior to analysis the SAGAT scores for each question across participants were converted into proportions and an arcsine transform conducted on these values. This procedure is recommended by both the SAGAT originator (Endsley 2000) and by statistical experts (Zar 1974).


1080


4.2.2. SAGAT. Pearson’s correlation coefficients were calculated to identify the relationship between WOMBAT SA ability, SAGAT and TRACON criterion performance. While all three measures were significantly correlated (WOMBAT and SAGAT r(16) ¼ 0.561, p ¼ 0.015), WOMBAT had a slightly higher association with TRACON performance (r(16) ¼ 0.691, p ¼ 0.001), than did SAGAT (r(16) ¼ 0.519, p ¼ 0.031). As can be seen in table 1, participants were aware of, and recalled, more aircraft positional and heading information in the SAGAT procedure than other operational information. Knowledge of the aircraft intentions and destinations was clearly poorer than positional and heading awareness rates. Participants were largely unable to recall aircraft speed and altitude data. Interestingly, and despite poor rates of altitude recall, participants were aware, at least to some degree, of whether the aircraft were climbing or descending. The present experiment also found that, on average, participants were nearly as adept in establishing and recalling plans for aircraft (level 3 SA future projection/planning) as they were for being aware of the correct position of that aircraft on the radarscope. A welcome benefit of SAGAT lies in its ability to tap into different levels of SA. It was of interest here to examine level 1 SA (perception of environmental/operational information, as measured by questions 1–5), and level 2/3 SA (comprehension and future projection/planning, as measured by question 6), independently of each other, to establish each one’s relationship to TRACON criterion task performance. A similar approach to SAGAT queries has been used successfully elsewhere (Endsley et al. 2000). Pearson’s correlation coefficients were calculated for these two new categories to establish their relationship to TRACON criterion task performance. Level 1 SA (perception of environmental/operational information) was not significantly correlated with TRACON performance (r(16) ¼ 0.468, p 4 0.05), whereas, level 2/3 SA (comprehension and future projection/planning) was strongly related to TRACON performance (r(16) ¼ 0.629, p 5 0.01). WOMBAT SA ability was also strongly related to level 2/3 SA (r(16) ¼ 0.586, p 5 0.01). That is, the higher the SA ability of participants, the greater the number of correct plans established and recalled for individual aircraft. 4.2.3. TRACON errors. Counts of errors committed by participants during the TRACON criterion task were taken and means calculated for each error type (see table 2). There were three error types that could be committed: hand-off errors; missed approaches; and separation conflicts.

Table 1. Mean SAGAT score for each of the questions and overall mean SAGAT score*. Question type Aircraft Aircraft Aircraft Aircraft Aircraft Aircraft Aircraft Aircraft

position altitude heading speed climbing/descending intentions destination plan

Overall SAGAT score

Mean SAGAT score (SD) 9.56 2.72 8.17 0.89 4.94 7.33 7.39 8.56

(2.59) (1.96) (2.75) (1.28) (2.34) (2.57) (2.52) (3.29)

49.56 (16.43)

*These scores are also displayed as percentage correct for each SAGAT question.

Mean (%) 53.1 15.1 45.4 4.9 27.5 40.7 41.1 47.5 34.4


1081


Table 2. Mean number of errors committed during TRACON criterion task performance for each error category and aircraft minutes delayed. Error type Hand-off errors Vectored off screen Missed approaches Separation conflicts Total errors Aircraft minutes delayed

Mean (SD) 0.16 0.16 0.83 0.89 2.05 28.05

(0.51) (0.38) (1.85) (0.90) (2.75) (19.21)

As can be seen in table 2, on average, two errors were committed during the TRACON criterion task, with the most critical error, separation conflicts, the most common error committed. While total error had a strong negative relationship with TRACON criterion scores (r(16) ¼ 70.743, p 5 0.001), there was no significant relationship between aircraft minutes delayed and TRACON criterion score (r(16) ¼ 70.268, p ¼ 0.283). Total errors were strongly associated with the number of commands given (r(16) ¼ 0.602, p ¼ 0.008). The only significant relationship for SA ability and SAGAT for error type was with separation conflicts (SA ability, r(16) ¼ 70.693, p 5 0.001; SAGAT (r(16) ¼7 0.497, p 5 0.05). The separation conflicts, without exception, involved the three aircraft converging to LAX lending some general support for both WOMBAT SA ability and the inclusion of the embedded (implicit measure) SA-reliant task. No main effects were found for the other error types or aircraft minutes delayed. 4.3. Discussion The present experiment explored the relationship between the WOMBAT, SAGAT and TRACON measures. The data showed that the WOMBAT SA ability scores had a strong association with TRACON performance scores. SAGAT scores were also associated with TRACON performance, but not to the same extent as WOMBAT SA ability scores. The SAGAT data showed that, overall, participants displayed knowledge of one-third of the information available at the time of the simulation freeze. This is considerably less than that found for the participants in Endsley and Rodgers’ (1996) study of air traffic controllers. Despite the low overall SAGAT scores, when the SAGAT questions were analysed individually and categorized according to Endsley’s three levels of SA, a more interesting and insightful picture of the data emerged. SAGAT responses to level 1 SA (perception of environmental/operational information) questions failed to show a significant association with TRACON performance. However, SAGAT responses obtained from tapping into level 2/3 SA (comprehension and future projection/planning) were strongly associated with TRACON performance. Durso et al. (1998) similarly found that correct responses to queries about the future were a better predictor of performance, as assessed by subject matter experts, than queries about present information. Durso et al. (1998) further suggested that knowing where a piece of information can be obtained in the environment may be more indicative of good SA than being able to report the specific information directly from memory. When examining individual differences in SA ability, it was found that high SA ability was positively associated with SAGAT scores. Again, this finding supports the predictive validity of WOMBAT as an SA ability measure. The data from individual SAGAT


1082


questions showed that SA ability, as measured by WOMBAT, was positively associated with correct recall of aircraft position and successful establishment of plans for aircraft. Knowing the relative position and interrelations of aircraft, along with an understanding of their movement dynamics, is an inherently complex and cognitively demanding task, although critical to maintaining separation between up to nine aircraft at one time. The ability of participants to manage the cognitive demands of maintaining separation was well predicted by WOMBAT with SA ability negatively associated with separation conflicts. The findings from the SAGAT question probing comprehension and future prediction/ planning, along with the implicit performance measure, suggests that it may be what is done with the information, both in terms of integrating and acting upon it in a goaldirected manner (requiring future prediction/planning or level 2/3 SA), that is crucial to achieving high TRACON criterion performance rather than merely having the information available (level 1 SA). In their review of planning, Mumford et al. (2001) offer a similar conception, which suggests that following an assessment of the environment and identification of goal priorities, an initial or ‘prototype plan’ is constructed. This initial plan guides further information acquisition and prediction and leads to the development of a more formalized plan. This plan, when implemented, then guides further monitoring of environmental information and assessment of the evolving situation. However, they also suggest that the plan itself is also frequently evaluated and changes made where necessary. Of course, this idealistic information gathering and planning process first assumes that the situation has been correctly assessed, interpreted and priorities set according to taskappropriate goals and sub-goals. However, this may only be possible for participants who are initially able to process large amounts of information in an integrated manner, i.e. those with high SA ability, leading to the correct situation assessment on which to base their information gathering and planning. Because WOMBAT is a domain-free test of the ability to manage the various cognitive demands of a complex task and integrate them in a dynamic setting and under changing priorities, it is worth suggesting that when operating in a complex dynamic real-world task such as TRACON, those with higher SA ability possess the cognitive skill or resources necessary to first acquire, and then integrate, information in a coherent goaldirected way that allows better planning and decision-making. Alternatively, it is possible that participants with high SA ability possess sufficient cognitive resources to allow them to divert spare resources to the development of strategies for handling complex tasks. Certainly from experiment 1, it appears that lower SA ability participants find it difficult to free up sufficient cognitive resources to independently formulate more effective TRACON management strategies. However, when prompted to employ more effective CMG strategies, participants appeared to overcome the initially detrimental impact of SA ability and improved their TRACON performance. The aim of experiment 1 was to train participants with varying degrees of SA ability to adopt more efficient cognitive and task management strategies in order to overcome SA ability limitations and improve performance in the real-world task of TRACON. Specifically, the training method encouraged participants to scan the sector, divide their attention effectively between aircraft and avoid focusing attention on one aircraft to the detriment of others. Additionally, participants were instructed to identify and justify the priority of each aircraft in relation to one another. Finally, the participants were asked to make judgements about the potential future state of the simulation.



1083

Experiment 1 suggested that training can overcome limitations in SA ability. Experiment 2 examined the relationship of the WOMBAT SA ability test, a relatively new research tool, and SAGAT, a widely used and validated SA measure, with TRACON, a complex dynamic real-world task. In addition, SAGAT was used to identify the candidate cognitive processes and performance aspects that may underlie TRACON performance and SA ability differences. Results from the SAGAT procedure suggested that higher-order cognitive processes, such as future projections and planning, were more strongly associated with TRACON performance than mere perception and knowledge of elements within the environment. Moreover, SA ability was positively associated with constructing successful plans for aircraft. Accordingly, it can be hypothesized that the CMG training, employed in experiment 1, improved TRACON performance by enabling participants to develop plans and project the future status of the aircraft rather than by improved information scanning/acquisition or workload management. Experiment 3 was designed to explore the role of planning in relation to TRACON performance and SA ability more directly. 5. Experiment 3 Experiment 1 showed that training of general CMG skills could ameliorate the negative effects of low SA ability on TRACON performance. More specifically, the CMG training method provided training in information acquisition through guided selective attention, visual scanning and identification of aircraft priority. But also, both implicitly and explicitly, the CMG training encouraged participants to understand interrelations between aircraft, integrate information and plan ahead. Given the rather eclectic mix of strategies employed, the specific cognitive means by which the CMG training improved performance was somewhat unclear. Experiment 1, along with other research (Taylor et al. 1997), and experiment 2 suggested that integrating information and planning ahead may be the critical aspects responsible for overcoming SA ability limitations that impact negatively on performance in the complex dynamic real-world tasks such as TRACON. This suggestion, although unexplored empirically, is consistent with recent lines of research looking at training of, and technological aids for, planning in order to support SA (Endsley and Robertson 2000, Moertl et al. 2002). However, little direct research has been conducted into the role of planning in SA and complex dynamic task performance. Mumford et al.’s (2001) review of planning offers several relevant avenues of research that suggest how planning may benefit performance. One experiment examining submarine naval officers’ decision-making found that planning aids construction of strategies for more efficient information search (Kirschenbaum 1992). In a similar vein, planning appears to aid construction of mental models critical to problem solving by guiding selection of relevant information (O’Brien and Albrecht 1992). Work by Goschke and Kuhl (1993) suggests that representations of intentions, or plans, facilitate memory. This facilitation is thought to be due to the organizational properties that planning brings to the storage and recall of information. Finally, the combined results of work by Ward and Allport (1997) and Gobet (1998) suggest that planning reduces the demands on resources by organizing multiple problem solving steps into easily and quickly executable chunks. This would be consistent with Durso and Gronlund’s (1999) suggestion that chunking and reorganizing of the environment facilitates SA. Higher-level cognitive strategies such as planning are thought to help reduce demands on cognitive resources and improve performance (see Mumford et al. 2001). But the


1084


strategies themselves may, initially, be somewhat dependent on allocation and effective management of significant attentional resources (Tulga and Sheridan 1980). In support, Hart and Wickens (1990) found that proactive behaviours (planning ahead) were more common when participant resources adequately met the workload requirements of the task, but became reactive as workload increased. This requirement for considerable initial investment of cognitive resources to effective strategy use may be a limiting factor for already resource-limited individuals. Given the benefits of workload-reducing strategies, such as planning, participants with limited cognitive resources and poor attentional control strategies need to be encouraged to make this initial investment. In view of this, the role of planning in ameliorating the effect of lower levels of SA ability needs to be explored further and clarified. The next experiment adopted a similar format for training to that employed in experiment 1. However, the present training was constrained to encouraging planning and behaviours facilitative of skilled planning whilst avoiding training of other cognitive skills, such as selective attention or visual scanning and acquisition of operational information. Although there is little research aimed at encouraging planning behaviours, previous research with children offers some potential avenues for developing planning training. Trabasso et al. (1992) showed that interpretation and understanding of story events was improved when strategic questions aimed at encouraging identification and verbalization of hierarchical goals and plans were used. Although the task context is somewhat different here, it is worth exploring the utility of this approach for enhancing performance in a complex, dynamic real-world task. It was hypothesized that SA ability and training aimed at encouraging planning would factor strongly in TRACON performance. 5.1. Method 5.1.1. Participants. A total of 24 participants were recruited via the University of Otago student job search scheme and were paid NZ$30 for participation in the experiment. There were 11 females and 13 males, ranging in age from 18–33 years (mean 23.8, SD 4.6). All participants had normal or corrected to normal hearing and eyesight. None of the participants reported having ever encountered WOMBAT, TRACON or any other ATC simulation. A performance incentive of NZ$20 was promised to the four participants with the highest combined scores on WOMBAT and TRACON criterion. 5.1.2. Apparatus. WOMBAT version 4.9 and associated control panel. TRACON for Windows version 1.2 and associated hardware as described for experiment 1. 5.1.3. Procedure. The experiment was run over three sessions, with each session taking approximately 90 min. Upon arrival, participants signed an ethical consent form and filled out a human factors questionnaire designed to gather participant profile and demographic data for several independent variables, such as gender, age, academic background, occupation and computer experience. No differences were found between these independent variables on any of the dependent measures (all p 4 0.05). Participants received either WOMBAT testing first or TRACON training first. In order to avoid experimenter bias, but to still ensure even numbers of participants in each condition, an independent judge examined WOMBAT scores for those receiving WOMBAT testing first and assigned participants to either the planning training or procedural training based on a projected potential median split for high vs. low SA ability.



1085

TRACON training and then testing was conducted on consecutive days with WOMBAT testing conducted within 1 week of initial participant contact. The WOMBAT and initial TRACON training procedure and task description were identical to that previously described. Advanced training was conducted using the same eight aircraft simulation as described in experiment 1. However, for the criterion simulation nine aircraft were used instead of eight. The simulation presented all nine of the aircraft to the participants within a 12 min period. Participants completed advanced TRACON training in session 2 and immediately after were tested on the TRACON criterion task. 5.1.4. Planning training. The training method was similar in style to that employed in the first experiment but with an exclusive focus on planning. Participants received verbal prompts that were aimed at encouraging the development of planning, rather than the more general CMG strategies in experiment 1. The eight aircraft training simulation was paused every 4 min for the first 20 min and all flight strips and aircraft flight details removed from the screen to reduce visual scanning behaviour, selective attention and acquisition of operational information during this phase of training. The planning training used a strategic questioning technique that encouraged participants to consider the interrelations between aircraft and explore future possibilities while establishing a plan of action for aircraft. The technique parallels that used extensively used by King (1991) to teach problem solving and metacognitive skill, and was based on Gick’s (1986) model of problem solving. Participants receiving the strategic questioning technique were asked to provide a plan of action for solving an identified goal or problem. Within the formulation of the plan, participants were asked a number of probe questions that were constructed in such a way as to encourage higher level long-term planning and problem solving (e.g. ‘What would happen if. . .’ King 1991). The questions used in the technique are typically general in nature but tailored to the specific domain or problem of interest. In this experiment, the probe questions were tailored toward encouraging the initial establishment of a long-term plan that considered aircraft interrelations, contingencies for re-planning, and consideration of the intangible ‘what if’ or future projection aspects of the situation. Layton et al. (1994) acknowledge the important role ‘what if’ questions can play in relation to route planning by airline pilots. Accordingly, the participants were then given the following instructions, which were read to them from a standard cue card: ‘We are just going to pause the simulation for a moment. I want you to answer, to the best of your abilities, a series of questions concerning the simulation. Please describe your long-term plan for ensuring this aircraft achieves its goal, by arriving at its final intended destination safely and efficiently.’ The experimenter concurrently indicated on the radar screen which aircraft they were referring to and gave the participant a printed flight strip with all flight details for the specific aircraft in question. Participants were asked if they understood what a plan was and were given the following definition. A plan is a ‘series of steps to be carried out or goals to be accomplished’ (Princeton University 1997). Participants were then given the following instructions, during which they described their plan to the experimenter and elaborated when asked each of the probe questions: ‘When constructing your plan, I want you to consider: what you would do if another aircraft is going to the same destination at a similar point in time; what you would do if another aircraft appears to be crossing the path of the indicated aircraft at some point in time; what you would do if another aircraft is entering the sector or taking off from the


1086


destination of the proposed aircraft; what possible problems could arise in the future, that may not be obvious at present’. Participants verbally described their initial plan in full to the experimenter. Where participants failed to provide a response to a probe question they were prompted to respond by the experimenter, who re-read the specific probe question. The experimenter provided no additional feedback or advice other than that on the planning technique cue cards. Following the completion of the planning task, the simulation was unpaused and all operational information returned to the screen. Participants continued with the simulation for approximately another 4 min, whereupon the planning technique was repeated with a new aircraft. This process occurred five times in the first 20 min of the simulation. After the first 20 min, participants were left to complete the training simulation alone. 5.1.5. Procedural training. This was identical to experiment 1, with the exception that a nine aircraft practice simulation was used instead of the eight aircraft simulation used in experiment 1. 5.1.6. TRACON testing. The test simulation presented nine aircraft within a 10 min period and was structurally more difficult than the eight aircraft testing simulation from experiment 1. Five of the flights required a hand-off to en-route controllers. Four of these hand-offs were initiated as take-offs from airports within the sector and one that was an overflight. The four remaining flights were arrivals (tower hand-offs), with three initiated from another sector and one as a take-off from within the present sector. Aside from the obvious increase in workload in receiving and handling nine aircraft within 10 min, the simulation was also constructed with an implicit (embedded task) measure of SA. That is, the simulation incorporated a deliberately constructed situation where there was a convergence of three aircraft to the same airport, at the same time and altitude. The assumption of this implicit performance measure, or embedded task, being that a person with good SA should detect this potential separation conflict and take appropriate avoidance actions, whereas a separation conflict between aircraft would indicate poor participant SA through not taking appropriate avoidance action. The maximum possible score for the criterion task simulation was 8330. As before, extensive data were gathered on the observed behaviours and actions carried out by participants during TRACON criterion testing. All data were initially examined for correct entry and missing data values. 5.2. Results 5.2.1. Description of performance. Individual WOMBAT and TRACON performance scores were collected and descriptives calculated. The overall mean WOMBAT score was 264.5 (SD 61.3). It should be noted that the TRACON testing simulations were somewhat more difficult than those in experiment 1 and provided a greater scope for attaining a higher TRACON score (experiment 1 TRACON maximum possible score ¼ 6500; experiment 2 TRACON maximum possible score ¼ 8330). The mean TRACON criterion scores for the planning and procedural (control) training conditions were 7389.2 (SD 697.2) and 5766.5 (SD 2565.1) respectively. The overall mean TRACON criterion test score was 6577.8 (SD 2016.5). There was no significant difference between training conditions for SA ability scores (F(1,22) ¼ 0.66, p ¼ 0.800).



1087

5.2.2. Effects of situation awareness ability and training. Training condition (planning vs. procedural), SA ability and the interaction between SA ability and training were entered in a single step in a linear regression to predict TRACON criterion test scores. Training was a significant predictor of TRACON test scores (t ¼ 74.421, p 5 0.001) and SA ability was marginally significant (t ¼ 72.041, p ¼ 0.055). The main effects of training and SA ability were qualified by a significant interaction between these two variables (t ¼ 3.727, p 5 0.001). Together, the three variables accounted for 69% of the variance in TRACON test scores. These findings are somewhat unexpected as it was assumed that higher SA ability participants would perform equally well regardless of the training condition received. It is worth suggesting that the increased difficulty of the present criterion task compared to experiment 1, along with the greater scope for scoring, may have allowed previously unseen differences to emerge. 5.3. Discussion The third experiment explored whether a training method aimed exclusively at encouraging planning could enhance performance on the complex real-world task of TRACON. It was expected that lower SA ability participants who received planning training would show significantly better performance on TRACON than lower SA ability participants receiving the procedural training. Moreover, it was expected that the higher SA ability participants would show equally good performance on TRACON regardless of the training condition they received. Combined, these hypotheses predicted that lower SA ability participants receiving standard procedural training would show significantly poorer performance on TRACON than the other three groups. This prediction was supported, with lower SA ability participants in the procedural training condition obtaining lower TRACON scores than the other groups. This poorer performance was also reflected in the number of separation conflicts committed, with higher SA ability participants committing fewer separation conflicts than lower SA ability participants. Similarly, those who received planning training committed fewer separation conflicts in the criterion task than those receiving procedural training. It is noteworthy that participants with lower SA ability, who received planning training, avoided any separation conflict for the three converging aircraft. Main effects for SA ability and training condition also emerged. In contrast to experiment 1, higher SA ability participants performed better than lower SA ability participants regardless of training. Second, participants who received planning training performed better than those who received procedural training. Logically, higher SA ability participants given a superior training method should perform better than lower SA ability participants given the same training method, particularly in demanding circumstances requiring high levels of SA. If this argument is accepted, then the practical issue becomes one of whether training can improve the lower SA ability participants’ performance on a complex real-world task to an operationally acceptable level. The present planning training method is encouraging in this respect. Clear differences between training groups and between high and low SA ability participants were found in both the quantity and type of errors committed during the TRACON simulation. The planning training groups committed fewer overall errors than the procedural training groups. Similarly, higher SA ability participants made fewer errors overall than lower SA ability participants. It is also noteworthy that participants


1088


with lower SA ability who received procedural training were the only group to commit any separation conflicts. This is a critical error in the ATC task. The success of the guided questioning technique in teaching problem solving is not new (Schoenfeld 1985, King 1991, Woloshyn et al. 1994). However, the domain in which it has been achieved here is. Previous research using guided questioning has typically been within classroom settings and often involved rather static problem tasks, such as mathematical problem solving and story comprehension. The present results extend these findings to the complex and dynamic environment of ATC. The role of ‘what if’ questions in planning was as successful here as has been elsewhere (Layton et al. 1992, 1994). Layton et al. (1992, 1994) specifically developed and tested several different en-route flight planning systems that were designed to support airline pilots ‘what if’ questioning about potential plans and alternative options with some success. The use of the novice participants in the present research must also be considered. Research examining novice vs. expert problem solving suggests that novices, when planning, lack the same hierarchical structure to knowledge that enables experts to produce high levels of planning (Hoc 1988). In some respects, the present planning training method may have provided this structure for participants. Higher SA ability participants receiving procedural training performed as well as the planning training groups. This suggests that the planning training alone cannot account for all of the TRACON performance differences exhibited in this experiment. Redding et al.’s (1992) cognitive task analysis of ATC is of relevance in addressing this issue. They state that: ‘Two interrelated cognitive tasks are central to effective ATC: maintaining situation awareness and developing and revising sector control plans. Experts use their situation awareness to develop and revise a long-term control plan, and they do this with great facility. Their adeptness at pre-planning is shown in the fact that they develop highlevel strategic plans significantly more often than less experienced controllers, but revise their plans significantly less often. Experts prefer efficient long-term planning over reactive, short-term planning.’ (Redding et al. 1992). Redding et al.’s (1992) statement, while supporting the rationale for the present planning training method, also suggests an explanation for the impact of SA ability on TRACON performance. Although WOMBAT is not strictly a measure of SA but of the underlying cognitive abilities and processes that support attainment and maintenance SA, it is still worth suggesting that high SA ability participants regardless of the training received are better positioned, cognitively speaking, to plan and may do so regardless of training. Further research needs to address this issue in more depth before firmer conclusions can be drawn. Again, financial and participant time constraints prohibited a more in-depth analysis of the impact of training on participants from the high and low SA ability groups. Issues raised by Redding et al. (1992) related to the types of planning taking place and their relationship to SA (in this case SA ability) and, in turn, performance in complex tasks need to be explored further. 6. General discussion and conclusions Given appropriate training, participants were able to overcome performance decrements imposed by limited SA ability. However, with all things being equal (training and experience), it is suggested that higher SA ability participants will always outperform their lower SA ability counterparts in complex dynamic tasks. Durso and Gronlund



1089

(1999) have come to a similar conclusion: ‘Although experience plays a role, it seems only to distinguish the conscientious pilot with good hands from his or her peers. Distinguishing among pilots with the same amount of experience seems to depend only on cognitive factors’. Indeed, SA ability, as measured by WOMBAT, may be what distinguishes good performers from great (O’Hare 1997). This issue deserves more comprehensive investigation. Regardless of these issues, the effectiveness of the training methods used here is encouraging for SA researchers and operators interested in achieving optimum performance in complex and dynamic tasks. Specifically, the research conducted here addresses an area of SA research that has had only meagre attention paid to it. Endsley and Robertson (2000) highlighted the paucity of research in this area of SA and, as noted previously, called for research on training methods that improve SA. The present research goes some way in addressing this plea. Both the CMG training method and planning training method were shown to be effective and address many of the skills identified by Endsley and Robertson (2000) as being of benefit to SA. From a practical standpoint, the present research provides yet more support to that already gathered (O’Hare 1997, O’Hare and O’Brien 2000) for the utility of WOMBAT as a domain independent SA measure. Accordingly, it is worth suggesting that future approaches to SA would benefit from this and other measures that assess the ability to dynamically manage and control the diverse cognitive processes and associated information said to support SA. References ACKERMAN, P.L., 1992, Predicting individual differences in complex skill acquisition: Dynamics of ability determinants. Journal of Applied Psychology, 77, 598–614. ACKERMAN, P.L. and KANFER, R., 1993, Integrating laboratory and field study for improving selection: Development of a battery for predicting air traffic controller success. Journal of Applied Psychology, 78, 413–432. ACKERMAN, P.L., KANFER, R. and GOFF, M., 1995, Cognitive and noncognitive determinants and consequences of complex skill acquisition. Journal of Experimental Psychology: Applied, 1, 270–304. COMPANION, M.A., CORSO, G.M., KASS, S.J. and HERSCHLER, D.A., 1990, Situational Awareness: An Analysis and Preliminary Model of the Cognitive Process (IST-TR-89 – 5). (Orlando, FL: University of Central Florida, Institute for Simulation and Training). DOMINGUEZ, C., 1994, Can SA be defined? In Situation Awareness: Papers and Annotated Bibliography, M. Vidulich, C. Dominguez, E. Vogel, and G. McMillan (Eds.), pp. 5–15 (Wright-Patterson Air Force Base, OH: Air Force Material Command). DURSO, F.T. and GRONLUND, S.D., 1999, Situation awareness. In Handbook of Applied Cognition, F.T. Durso, R. Nickerson, R. Schvaneveldt, S. Dumais, M. Chi and S. Lindsay (Eds.), pp. 283–314 (New York: Wiley). DURSO, F.T., HACKWORTH, C.A., TRUITT, T.R., CRUTCHFIELD, J., NIKOLIC, D. and MANNING, C.A., 1998, Situation awareness as a predictor of performance for en route air traffic controllers. Air Traffic Control Quarterly, 6, 1–20. ENDSLEY, M.R., 1989, Pilot situational awareness. The challenge for the training community. In Proceedings of the Interservice/Industry Training Systems Conference (Ft Worth TX: American Defence Preparedness Association), pp. 111–117. ENDSLEY, M.R., 1993, A survey of situation awareness requirements in air-to-air combat fighters. International Journal of Aviation Psychology, 3, 157–168. ENDSLEY, M.R., 1995, Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64. ENDSLEY, M.R., 2000, Theoretical underpinnings of situation awareness: A critical review. In Situation Awareness Analysis and Measurement, M.R. Endsley and D.J. Garland (Eds.), pp. 3–32 (London: Erlbaum). ENDSLEY, M.R. and BOLSTAD, C.A., 1994, Individual differences in pilot situation awareness. International Journal of Aviation Psychology, 4, 241–264. ENDSLEY, M.R. and GARLAND, D.J., 2000, Situation Awareness Analysis and Measurement, pp. 3–32. (London: L. Erlbaum Associates).


1090


ENDSLEY, M.R. and ROBERTSON, M.M., 2000, Training for situation awareness in individuals and teams. In Situation Awareness Analysis and Measurement, M.R. Endsley and D.J. Garland (Eds.), pp. 3–32 (London: Erlbaum). ENDSLEY, M.R. and RODGERS, M.D., 1996, Situation Awareness Information Requirements Analysis for En Route Air Traffic Control (DOT/FAA/AM-94/27). (Washington, DC: Federal Aviation Administration, Office of Aviation Medicine). ENDSLEY, M.R., SOLLENBERGER, R., NAKATA, A. and STEIN, E., 2000, Situation Awareness in Air Traffic Control: Enhanced Displays for Advanced Operations (DOT/FAA/CT-TN00/01). (Atlantic City, NJ: Federal Aviation Administration William J. Hughes Technical Center). FEDERICO, P., 1995, Expert and novice recognition of similar situations. Human Factors, 37, 105–122. FREDERICSON, J.R. and WHITE, B.Y., 1989, An approach to training based upon principled task decomposition. Acta Psychologica, 71, 89–146. GABA, D.M., HOWARD, S.K. and SMALL, S.D., 1995, Situation awareness in anaesthesiology. Human Factors, 37, 20–31. GICK, M.L., 1986, Problem-solving strategies. Educational Psychologist, 21, 99–120. GILSON, R.D. (Ed.), 1995, Situation awareness. Human Factors 37, 3–4. GOBET, F., 1998, Expert memory: A comparison of four theories. Cognition, 66, 115–152. GOETTL, B.P., 1997, Situation awareness and executive control processes: Quot homines, tot sententiae. Proceedings of the Human Factors and Ergonomics Society, 41st Annual Meeting (Santa Monica, CA: Human Factors and Ergonomics Society), p. 61. GOPHER, D., 1993, The skill of attention control: Acquisition and execution of attention strategies. In Attention and Performance 14: Synergies in Experimental Psychology, Artificial Intelligence, and Cognitive Neuroscience, D.E. Meyer and S. Kornblum (Eds.), pp. 299–322. (Cambridge, MA: MIT Press). GOPHER, D., WEIL, M. and BAREKET, T., 1994, Transfer of skill from a computer game trainer to flight. Human Factors, 36, 387–405. GOPHER, D., WEIL, M. and SIEGEL, D., 1989, Practice under changing priorities: An interactionist perspective. Acta Psychologica, 71, 147–178. GOSCHKE, T. and KUHL, J., 1993, Representation of intentions: Persisting activation in memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 1211–1226. GRONLUND, S.D., CANNING, J.M., MOERTL, P.M., JOHANSSEN, J., DOUGHERTY, M.R.P. and MILLS, S.H., 2002, An Information Tool for Planning in Air Traffic Control (DOT/FAA/AM-02/1), (Washington, DC: FAA Office of Aviation Medicine). GRONLUND, S.D., DOUGHERTY, M.R.P., DURSO, F.T., CANNING, J.M. and MILLS, S.H., 2001, Planning in Air Traffic Control, DOT-FAA-AM-01 – 16 (Washington, DC: FAA Office of Aviation Medicine). GUGERTY, L.J., 1997, Situation awareness during driving: explicit and implicit knowledge in dynamic spatial memory. Journal of Experimental Psychology: Applied, 3, 42–66. HARDY, D.J. and PARASURAMAN, R., 1997, Cognition and flight performance in older pilots. Journal of Experimental Psychology: Applied, 3, 313–348. HART, S.G. and WICKENS, C.D., 1990, Workload assessment and prediction. In Manprint, an Approach to Systems Integration, H.R. Booher (Ed.), pp. 257–296 (New York: Van Nostrand Reinhold). HOC, J.M., 1988, Cognitive Psychology of Planning (London: Academic Press). HOGG, D.N., FOLLESO, K., STRAND-VOLDEN, F. and TORRALBA, B., 1995, Development of a situation awareness measure to evaluate advanced alarm systems in nuclear power plant control rooms. Ergonomics, 11, 2394– 2413. KING, A., 1991, Effects of training in strategic questioning on children’s problem-solving success. Journal of Educational Psychology, 83, 307–317. KIRSCHENBAUM, S.S., 1992, Influence of experience on information-gathering strategies. Journal of Applied Psychology, 77, 343–352. KLEIN, G. ORASANU, J. CALDERWOOD, R. AND ZSAMBOK, C.E. (Eds.), 1993, Decision Making in Action: Models and Methods, pp. 172–181 (Norwood, NJ: Ablex). LAYTON, C.F., SMITH, P.J. and MCCOY, C.E., 1994, Design of a cooperative problem-solving system for en route flight planning: An empirical evaluation. Human Factors, 36, 94–119. LAYTON, C.F., SMITH, P.J., MCCOY, C.E. and BIHARI, T.E., 1992, An empirical evaluation of tools to aid in enroute flight planning. In Proceedings of the Human Factors and Ergonomics Society 36th Annual Meeting, (Santa Monica, CA: The Human Factors and Ergonomics Society), pp. 97–99. LIPSHITZ, R. and BEN SHAUL, O., 1997, Schemata and mental models in recognition-primed decision making. In Naturalistic Decision Making. Expertise: Research and Applications, E. Zsambok and C.G. Klein (Eds.), pp. 293–303 (Mahwah, NJ: Erlbaum).



1091

LOGAN, G.D., 1985, Executive control of thought and action. Acta Psychologica, 60, 193–210. MOERTL, P.M., CANNING, J.M., DOUGHERTY, M.R.P., JOHANSSON, J., MILLS, S.H. and GRONLUND, S.D., 2002, Aiding planning in air traffic control: An experimental investigation of the effects of perceptual information integration. Human Factors, 44, 404–412. MUMFORD, M.D., SCHULTZ, R.A. and VAN DOORN, J.R., 2001, Performance in planning: Processes, requirements, and errors. Review of General Psychology, 5, 213–240. NORTH, R.A. and GOPHER, D., 1976, Measures of attention as predictors of flight performance. Human Factors, 18, 1–14. O’BRIEN, E.J. and ALBRECHT, J.E., 1992, Comprehension strategies in the development of a mental model. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 777–784. O’HARE, D., 1997, Cognitive ability determinants of elite pilot performance. Human Factors, 39, 540–552. O’HARE, D. and O’BRIEN, K.S., 2000, Individual differences in situational awareness and training for complex tasks. In Aviation Resource Management, A.R. Lowe and B.J. Hayward (Eds.), vol. 2, pp. 369–377 (Aldershot: Ashgate). PRINCETON UNIVERSITY, 1997, WORDNET 3.0. Available online at: http://www.cogsci.princeton.edu (accessed 2007). RANDEL, J.M., PUGH, H.L. and REED, S.K., 1996, Differences in expert and novice situation awareness in naturalistic decision making. International Journal of Human-Computer Studies, 45, 579–597. REDDING, R.E., CANNON, J.R. and SEAMSTER, T.L., 1992, Expertise in air traffic control (ATC): what is it, and how can we train for it? In Proceedings of the Human Factors Society 36th Annual Meeting (Santa Monica, CA: Human Factors Society), pp. 1326–1330. ROSCOE, S.N., 1993, Predicting and enhancing flight deck performance. In Aviation Training: Pilot Instructor and Organization, R. Telfer and P. Moore (Eds.), pp. 195–208 (Aldershot: Avebury). ROSCOE, S.N. and CORL, L., 1987, Wondrous original method for basic airmanship testing. In Proceedings of the Fourth International Symposium on Aviation Psychology, R.S. Jensen (Ed.), (Columbus: The Ohio State University), pp. 493–499. ROSCOE, S.N., CORL, L. and LAROCHE, J., 2001. Predicting Human Performance, 5th ed. (Pierrefonds, Quebec: Helio Press). SARTER, N.B. and WOODS, D.D., 1995, How in the world did I ever get into that mode? Mode error and awareness in supervisory control. Human Factors, 37, 5–19. SCHOENFELD, A.H., 1985, Making sense of ‘out loud’ problem-solving protocols. Journal of Mathematical Behavior, 4, 171–191. SCHNEIDER, W., 1985, Training high-performance skills: Fallacies and guidelines. Human Factors, 27, 285–300. SHEPARD, R.N. and METZLER, J., 1971, Mental rotation of three-dimensional objects. Science, 171, 701–703. TAYLOR, R.M., FINNIE, S. and HOY, C., 1997, Cognitive rigidity: The effects of mission planning and automation on cognitive control in dynamic situations. Paper presented at the 9th International Symposium on Aviation Psychology. Columbus, OH. TRABASSO, T., STEIN, N.L., RODKIN, P.C., MUNGER, M.P. and BAUGHN, D., 1992, Knowledge of goals and plans in the on-line narration of events. Cognitive Development, 7, 133–170. TULGA, M.K. and SHERIDAN, T.B., 1980, Dynamic decisions and work load in multitask supervisory control. IEEE Transactions on Systems, Man and Cybernetics, 10, 217–232. VIDULICH, M.A., MCCOY, A.L. and CRABTREE, M.S., 1996, Attentional control and situational awareness in a complex air combat simulation. In Situation Awareness: Limitations and Enhancement in the Aviation Environment AGARD Conference Proceedings, CP575 (Neuilly-sur-Seine, France: AGARD), pp. 18.1–18.5. WARD, G. and ALLPORT, A., 1997, Planning and problem-solving using five-disc tower of London task. Quarterly Journal of Experimental Psychology A, 50A, 49–78. WESSON, R., 1990, TRACONTM (Washington, DC: Wesson International). WOLOSHYN, V.E., WOOD, E. and WILLOUGHBY, T., 1994, Considering prior knowledge when using elaborative interrogation. Applied Cognitive Psychology, 8, 25–36. WOODHOUSE, R. and WOODHOUSE, R.A., 1995, Navigation errors in relation to controlled flight into terrain (CFIT) accidents. In Proceedings of the 8th International Symposium on Aviation Psychology, R. Jensen (Ed.), (Columbus, OH: Ohio State University), pp. 45–50. ZAR, J.H., 1974, Biostatistical Analysis (Englewood Cliffs, NJ: Prentice Hall).