Air Traffic Controllers' Performance in Advance Air

This article was downloaded by: [pratik jha] On: 10 July 2011, At: 07:25 Publisher: Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

The International Journal of Aviation Psychology Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/hiap20

Air Traffic Controllers' Performance in Advance Air Traffic Management System: Part I—Performance Results Pratik D. Jha

a b

, Ann M. Bisantz

c

Parasuraman & Colin G. Drury

a b

, Raja

a b

a

Department of Industrial and Systems Engineering, University at Buffalo, Buffalo, New York, USA b

State University of New York, Albany, New York, USA c

George Mason University, Fairfax, Virginia, USA

Available online: 05 Jul 2011

To cite this article: Pratik D. Jha, Ann M. Bisantz, Raja Parasuraman & Colin G. Drury (2011): Air Traffic Controllers' Performance in Advance Air Traffic Management System: Part I—Performance Results, The International Journal of Aviation Psychology, 21:3, 283-305 To link to this article: http://dx.doi.org/10.1080/10508414.2011.582456

PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.tandfonline.com/page/termsand-conditions This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan,

sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden.

Downloaded by [pratik jha] at 07:25 10 July 2011

The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

THE INTERNATIONAL JOURNAL OF AVIATION PSYCHOLOGY, 21(3), 283–305 Copyright © 2011 Taylor & Francis Group, LLC ISSN: 1050-8414 print / 1532-7108 online DOI: 10.1080/10508414.2011.582456


Air Traffic Controllers’ Performance in Advance Air Traffic Management System: Part I—Performance Results Pratik D. Jha,1 Ann M. Bisantz,1 Raja Parasuraman,2 and Colin G. Drury1 1

Department of Industrial and Systems Engineering, University at Buffalo, Buffalo, New York; and State University of New York, Albany, New York 2 George Mason University, Fairfax, Virginia

The current air traffic management (ATM) system will soon reach its capacity limits, so new concepts for the future are being developed. Many proposed schemes include shifting the task of maintaining safe distances between aircraft to ground-based automation and assigning air traffic controllers a supervisory role. This research investigated air traffic controller performance using advanced ATM concepts for the task of conflict resolution. Performance using a medium fidelity air traffic control simulator was investigated across 2 proportions of free-flying traffic and 3 types of decision support. Conflict resolution performance was significantly degraded when free-flying aircraft failed to negotiate a conflict and controllers had to intervene. Conflict resolution performance varied across the aiding condition and was superior when negotiation information was provided. However we did not find any differences in performance across proportions of free-flying traffic. Implications of this research are in further exploration of ATM concepts, design of decision aids, and training needs assessment.

There is general consensus that the current air traffic management (ATM) system cannot be maintained in the face of a predicted rapid rise in capacity demands. Among the future ATM concepts that have been put forward as part of the future architecture of the National Airspace System (NAS) are free flight (FF; Correspondence should be sent to Pratik D. Jha, Department of Industrial and Systems Engineering, University at Buffalo, The State University of New York, 342 Bell Hall, Amherst, NY 14260. E-mail: [email protected]


284

JHA ET AL.

Parasuraman, Hilburn, & Hoekstra, 2001; RTCA, 1995), the Joint Planning and Development Office (2007) NextGen Concept of Operation, and related proposals such as Distributed Air-Ground Traffic Management (DAG-TM; NASA, 1999), and the Automated Airspace Concept (Erzberger, 2001). Each proposed concept is intended to significantly alleviate problems with air traffic congestion through a combination of enhanced automation and redesigned control procedures. Any change is likely to favor joint responsibility for separation between pilots and air traffic controllers (ATCos) as in DAG-TM and free-flight concepts. Aircraft will have automated capability for conflict resolution and pilots will bear the responsibility of maintaining safe separation. The ATCos are likely to act as a second line of defense and will resolve conflicts in case the flight crew is not successful in conflict resolution. However, the process of conflict resolution between the pilots might not be straightforward, as there are economic considerations in resolving a conflict. Krozel and Peters (1997) reported that vertical maneuvers are most economical for conflict resolution, and Westrenen and Groeneweg (2003) stated that such maneuvers are superior in terms of passenger comfort and safety. Such economic considerations could hinder the capability of pilots to successfully negotiate a resolution in the allocated time frame. In such cases, ATCos would need to resolve the conflict. However, the ability of ATCos to intervene in the advanced air traffic concept has been a question of serious debate. Human factors research in this area has revealed that changes in ATM strategies will have the effect of changing the ATCo’s role from one of active control to passive monitoring (Metzger & Parasuraman, 2001), which might lead to a condition where controllers are not able to maintain an adequate level of situational awareness and are unable to intervene in time to ensure the safety of the system. As noted by Dekker and Woods (1999), determining when a controller should intervene is a challenging question in itself. Other researchers (Endsley, Mogford, Allendoerfer, Snyder, & Stein, 1997) have found that ATCos might have degraded situational awareness in monitoring autonomous system, and have suggested the need for conflict resolution aids. The situation is further complicated by the fact that some aircraft will be equipped for advanced ATM and some will not, due to the projected cost of equipping aircraft with the necessary systems. Thus, not all aircraft will be able to carry out the separation assurance responsibility on their own and will have to be actively managed by ATCos. Hence, there will be a mix of free-flying (unmanaged) and non-free-flying (managed) aircraft. The impact of mixed equipment on ATCo performance has raised some serious issues and questions (Corker, Fleming, & Lane, 1999; Metzger, Rovira, & Parasuraman, 2003). For example, how successful are controllers in maintaining separation between aircraft with mixed equipment? How well can they allocate their attention differentially to


ATC PERFORMANCE IN ADVANCED ATM CONCEPTS

285

the traffic mix? What is the effect of proportions of various types of aircraft in the sector? Metzger et al. (2003) studied the effect of traffic mix and automation on controller’s performance and workload in the mixed equipment environment. They reported a degradation in ATCos’ conflict detection performance in such an environment of both managed and unmanaged aircraft. Automated decision aids in the air as well as on the ground might hold the key to the future ATM concepts of operation. Woods and Roth (1988) noted that predictive information like that provided by a decision aid enhances a person’s ability to anticipate system behavior and can greatly improve human performance in controlling a complex system for example process control. Recently a conflict probe for controllers, called the User Request Evaluation Tool (URET), was fielded by the Federal Aviation Administration (Brudnicki & McFarland, 1997). This decision aid helps the controllers in assigning user preferred routing, so that they can provide better service to the airline customer. Problem Analysis, Resolution and Ranking (PARR) is currently being prototyped and is an extension of URET that generates and ranks conflict resolution advisories, when requested by controllers. Wickens, Mavor, Parasuraman, and McGee (1998) noted that although in theory it might be possible to build a decision aid that incorporates “rules of the road” and provides conflict resolution advisories, there exists substantial evidence (research and aircraft accidents) to show that such a system might have significant shortfalls. For example, although the Traffic Collision Avoidance System (TCAS) has prevented numerous aircraft collisions, in rare cases it has been found not to work consistently and effectively because of human error. Pritchett and Hansman (1997) suggested that automated advice in air traffic avoidance maneuvers will not always be followed. Additionally, research on the role of communication in free flight has revealed an increased need for air-to-air and air-to-ground communication between pilots and controllers (Mackintosh et al., 1998). Endsley et al. (1997) reported that air-to-ground communication (pilots sharing their intent with controllers) is essential for controllers to maintain situational awareness. Duley, Galster, Masalonis, Hilburn, and Parasuraman (1997) noted that information about delineation of separation of responsibility needs to be effectively communicated for safety reasons. This is especially true in cases where the resolution has to be generated by ATCos (Andrews & Hollister, 1997). In the near future, much air-to-ground voice communication will be replaced by data link communication. A data link is a messaging system that transfers text messages back and forth. However, it is not clear how much information sharing or communication is required between pilots and ATCos in the conflict resolution process. Studies conducted to understand the impact of using data link communication have reported an increase in collaboration between controllers and pilots in a data link environment (Farley, Hansman, Amonlirdviman, & Endsley,

286

JHA ET AL.

2000); however other data have shown that pilot-initiated communications decline with data link usage (Lozito, McGann, & Corker, 1991). Furthermore, the nature of communication changes from synchronous to asynchronous, which could complicate the conflict resolution process (Wickens et al., 1998).


RESEARCH GOALS To date, no study has looked into the ability of controllers to resolve conflicts using advanced trajectory negotiation concepts as proposed in DAG-TM and other concepts. This study examined how successful ATCos are in ensuring safe operations (specifically, appropriate aircraft separations) in an advanced ATM situation involving pilot-based trajectory negotiations. Various researchers have noted that pilots’ intent information is extremely important in cases where ATCos have to intervene. However, whether providing the negotiation information to the controllers yields any performance advantage has never been tested. Furthermore, benefits of decision aids that provide conflict resolution advisories and the impact of traffic mix (proportion of free-flying vs. non-free-flying traffic) on controller performance have yet to be explored. This research addresses this gap by explicitly comparing three types of decision aids (aids supporting detection alone, aids for detection as well as negotiation, and aids for detection, negotiation, and conflict resolution), across different proportions of free-flying aircraft. We hypothesized the following: 1. Degraded conflict resolution performance will exist in advanced air traffic concepts that involve trajectory negotiations. 2. Superior conflict resolution performance will occur when decision aids such as negotiation information and conflict resolution aid are available. 3. Degraded controller performance will be seen in the condition with a higher proportion of free-flying aircraft. METHOD Simulator A PC-based medium-fidelity air traffic control (ATC) simulator (Masalonis et al., 1997) was used to simulate a generic airspace. The simulation consists of two 21-in. monitors placed side by side to emulate a primary visual display (PVD) or radar display and a data link display. The PVD (see Figure 1) is a situation display and provides controller information on the sector traffic. The simulated generic airspace has a 50-mile radius and contains waypoints and jet routes. The sector boundaries are displayed as



FIGURE 1

287

Air traffic control simulator primary visual display (color figure available online).

a polygon and the controllers were responsible for the aircraft within the sector boundary. Each aircraft is represented as a data block that displays relevant aircraft information (e.g., call signs, flight level in feet, and ground speed). An automated capability of a conflict detection aid was made available. The conflict detection aid is based on the flight plan information and comes on as a red bubble around the aircraft pair 6 min before they would lose separation. A conflict button and a resolution button are provided on the display to record conflict detection and resolution time during the experiment. The type of resolution was verbally articulated by the ATCo and noted by the experimenter. The data link display (see Figure 2) is a communication display used to simulate electronic communication between ATCo and pilots similar to data link. The data link display has a flight list, electronic flight strip for each flight, a communication area for exchanging messages between aircraft and controllers, and other buttons to accept aircraft and hand off aircraft between sectors. When an aircraft was close to entering a sector, a message appeared in the communication area (incoming message) to enable the controller to accept the aircraft. Conflict information with countdown timer, negotiation information, and conflict resolution aid were displayed on a third display for some experimental


288

JHA ET AL.

FIGURE 2

Air traffic control simualtor data link display (color figure available online).

conditions (see Figure 3). The third display was a stand-alone display that ran on a Gateway Tablet PC and was kept on top of the PVD. Information about aircraft that were in conflict along with a countdown timer was displayed on the left side. The purpose of the countdown timer was to indicate to the controller when he or she could resolve the free-flying conflicts. The countdown timer was synchronized with the conflict detection aid and came up 6 min before the conflict in a red box. After 2 min had elapsed (thus, 4 min prior to the conflict) the red box turned green, giving controllers an indication that free-flying aircraft had failed to resolve the conflict and the controller must resolve it. In some conditions the negotiation information between the aircraft was displayed to keep controllers informed about how the two aircraft intend to resolve the conflict. In conditions when negotiation information was not available, a message was displayed indicating that information was not available. In one condition a conflict resolution aid was also provided to display possible resolution options and also an advisory. The resolution aid was designed to provide the most economic and safe maneuvers to the aircraft such that the cost of trajectory deviation is minimized. This aid was displayed when the timer turned green (4 min before the conflict). The timing of the resolution advisory



289

FIGURE 3 Negotiation information and conflict resolution display. The figure describes the interface used to display negotiation information and the conflict resolution aid. On the left, information is provided about the aircraft that are in conflict with a countdown timer. In the center, negotiation information between two aircraft in conflict is displayed. The conflict resolution aid describing optimal maneuver is displayed on the right (color figure available online).

is consistent with the Tactical Separation Assisted Flight Environment (TSAFE) concept proposed by Erzberger (2001). Experimental Design A 2 (Traffic Proportion: high proportion vs. low proportion of free-flying aircraft) × 3 (Decision Aid: detection aid, detection aid + negotiation information, detection aid + negotiation information + resolution aid) within-subjects design was used. The order of conditions was counterbalanced across participants. The high proportion of free-flying aircraft condition had 70% aircraft that were equipped for free flight and did not need active control by ATC. The other 30% were not equipped for free flight and were managed by ATC. This condition was reversed in the low traffic mix condition. In the conflict detection aid condition, ATCos were supported with only the conflict detection aid as previously described. In this case controllers were instructed to give pilots 2 min for conflict resolution for free-flying aircraft. In

290

JHA ET AL.


case pilots were not able to resolve the conflict within the 2-min time period from when the conflict detection aid came up, controllers were instructed to resolve the conflict. The 2-min time window was indicated to the controller with the help of a countdown timer on a secondary interface (in all aid conditions). In the detection aid + negotiation information condition, along with the previous detection support, the ATCos were provided with information about the ongoing negotiations (e.g., text messages) between the pilots to resolve the conflict. The negotiation protocol was simulated as follows on the controller display: 1. The pilot who detects the conflict first alerts the other aircraft to the situation by sending a message. In its message it includes aircraft information and conflict information. 2. The pilot of the other aircraft acknowledges the message. 3. The aircraft that identifies the conflict initiates the negotiation process. 4. The pilots go back and forth until they agree on a resolution. If they cannot, they send a message to ATC requesting intervention. In the detection aid + negotiation information + resolution aid condition a conflict resolution aid was available that provided resolution advisories to ATCos. The resolution aid interface (Figure 4) was designed such that information was presented to the controllers in a form that revealed the possible options and aided them in selecting the best option (the principle is consistent with the problem analysis resolution and ranking concept proposed by MITRE; Kirk, Heagy, & Yablonski, 2001). The options were rank ordered from most economic to least economic maneuver as suggested by Krozel and Peters (1997). The advisory given by the resolution aid was based on a simple heuristic: Altitude ascending is the best maneuver followed by heading changes. The ratio used to depict the most economic maneuver was somewhat arbitrary and showed altitude maneuvers were at least three times as preferable as heading changes. Speed maneuvers were not considered in the experiment because it is not efficient and often harder to create required spacing between aircraft using speed differential during the cruise phase with time horizons considered in this study. Participants Eleven ATCos (4 women and 7 men) from the Washington region (Washington ARTCC and Potomac TRACON) with en route and terminal experience volunteered for the study. Eight controllers were full performance level, 1 retired within the last year, and 2 were associate controllers. Their ages varied from 21 to 62 years (M = 38.09, SD = 11.81) and experience from 1 to 39 years (M = 15.04,


291

Resolution: Climb COA300 to 400 from 380

5 4 2


1 0 FIGURE 4

Altitude High

Altitude Low

DAL126

COA399

3

COA399

DAL126

6

DAL126 COA399

DAL126

7

COA399

8

COA399

9

DAL126

10

Heading Right Heading Left

Example interface used for the resolution aid (color figure available online).

SD = 10.6). Volunteers were compensated $30 per hour and travel costs for their participation. Scenario Development A generic en route airspace was used for this study. As has been suggested in many concepts of operations (Beers & Huisman, 2001) there were two types of airspace in the sector: free-flight airspace and non-free-flight airspace. In this experiment the free-flight airspace was high-altitude airspace extending upward of 36,000 ft. The airspace below 36,000 ft was non-free-flight airspace. The non-free-flying aircraft were restricted to non-free-flying airspace. Air traffic was at cruising altitude and aircraft did not climb or descend in the scenario except in the case of self-separation. A total of six scenarios were created. Three scenarios were created initially and were rotated to create the others. Rotating the scenarios changed the sector characteristics so that the scenarios did not look identical. Also, the names of the waypoints and aircraft call signs were changed between different scenarios. The methodology used for rotation of scenario is consistent with previous research conducted by Metzger and Parasuraman (2001) and Rovira (2006). Each scenario was of 30 min duration with a 7-min ramp-up time. During the ramp-up period the aircraft were entering the sector. Scenarios were designed for high traffic density (between 18 and 22 aircraft). One scenario was used for each of the six withinsubjects conditions. Each scenario had three conflicts between free-flying aircraft in which the pilots failed to negotiate a trajectory change within the 2-min time window and

292

JHA ET AL.


hence the aircraft entered the ATC zone of control. This situation required ATCo to intervene and resolve the conflict. Each scenario contained two successful self-separations between free-flying aircraft. A self-separation was defined as an evasive maneuver by a free-flying aircraft that is on a conflict course with the other aircraft. The evasive maneuver changed altitude for one of the aircraft. Lastly, each scenario also had one conflict between the non-free-flying aircraft. It has been found that conflict geometry (Castano & Parasuraman, 1999) has an impact on conflict detection. Both acute and obtuse types of conflicts were included in the scenarios. However climbing and descending conflicts were not considered in this study. The non-free-flying aircraft were distinguished by a colored yellow box around the call sign in the data block. There was no box for free-flying aircraft. Experimental Task ATCos performed the following two primary tasks: 1. Conflict detection and resolution for the free-flying aircraft. ATCos were required to monitor traffic in their sector and report potential conflicts between aircraft by selecting the aircraft and clicking a conflict button provided on the display. A potential conflict was defined as two aircraft that are on course that will eventually bring them within 5 nm horizontally or 1,000 ft vertically of each other. Potential conflicts could result in a self-separation or develop into an actual conflict if they were not resolved by the pilots. 2. Hand off an aircraft when it left the sector and accept an aircraft when it entered the sector. Every aircraft entering the sector was blinking to indicate that it needed to be accepted in the sector. Additionally, its data tag was orange in color. To accept a handoff the ATCo had to use the data link interface and press the accept button. To initiate a handoff the ATCo selected the aircraft from the flight list and pressed the auto hand button. The ATCos were instructed to accept the handoff and initiate a handoff in a timely manner. ATCos performed the following secondary task: 1. Update flight progress through the sector by marking flight progress strips. The ATCos were provided electronic flight progress strips through the data link display. The electronic flight progress strip had information about each waypoint the aircraft is supposed to cross in that sector. ATCos were required to check mark the waypoint as soon as the aircraft crossed a waypoint.


293

Dependent Variables


The following primary task performance aspects were measured: 1. Number of conflicts detected. 2. Advanced potential conflict notification time. The advanced potential conflict notification time was defined as the difference in time between when the ATCo reported a potential conflict and the time when aircraft would lose separation. A higher value of advanced potential conflict notification time indicates better conflict detection performance. 3. Number of conflicts detected before and after the conflict detection aid was displayed. 4. Number of conflicts resolved. 5. Advanced resolution time. This is defined as the time between when the ATCo resolved the conflict and the time the aircraft would lose separation. A higher value of advanced resolution time indicates better conflict resolution performance. 6. Type of resolution selected to deconflict the aircraft (to measure resolution effectiveness and performance relative to conflict resolution aid). 7. Timeliness of accepting and initiating handoffs. Timeliness was calculated by subtracting the time between when a message appeared in the data link display indicating that an aircraft was entering the sector and the time the ATCo accepted the aircraft by clicking the accept button. A shorter value of handoff acceptance time indicates better task performance. The timeliness of handoff initiation was measured by when the ATCo handed off the aircraft by clicking the auto hand button relative to when the aircraft passed the designated handoff circle. If the aircraft was handed off before the designated handoff circle, it was counted as an early handoff. Counts were also kept for the aircraft that were not handed off (missed handoffs). 8. Advanced conflict detection time relative to detection aid was calculated to measure performance with the conflict detection aid. The advanced conflict detection time relative to detection aid is defined as the time when the ATCo reported the potential conflict minus the time the detection aid came on. A positive time indicates that the ATCo detected the conflict before the aid came on. A negative time indicates that ATCo detected the conflict after the aid came on. The following are measures of secondary task performance: 1. Proportion of updated waypoints and elapsed time between when the aircraft crossed the waypoint and when it was checked by the controller on the electronic flight strip.

294

JHA ET AL.

Subjective workload, trust, and self-confidence measures were also collected but those results are reported elsewhere (Jha, Bisantz, Parasuraman, & Drury, 2009). Procedure After providing informed consent and demographic information, participants experienced approximately 1 hr of training using the simulator. Participants then completed all six conditions (in counterbalanced order) with breaks between conditions as desired.


RESULTS Within-subjects analyses of variance (ANOVAs) were performed for primary task measures, secondary task measures, and subjective ratings of workload, trust, and self-confidence. Chi-square tests were performed to analyze the differences between type of resolution selected in different conditions. A regression was performed to ascertain the relation between various dependent variables (e.g., conflict detection time, type of aircraft, proportion of aircraft, etc.) and conflict resolution time. Data from all 11 participants are included in the analysis. Each condition had two self-separations, three free-flight conflicts, and one non-free-flight conflict. The results presented are for free-flight conflicts unless specifically noted. Primary Task Performance Conflict Detection Performance Participants detected 100% of conflicts in all conditions. Certain aircraft pairs that were close calls but were not really in conflict were also detected as possible conflicts by ATCos. (The conflict detection aid did not appear for these situations). ATCos suggested that in real-life situations due to wind uncertainty and other factors, these close calls could develop into actual conflict and they would prefer to detect and resolve the aircraft in such cases. ANOVA results revealed no significant difference across conditions for advanced conflict detection time, as expected, as all three conditions provided the same conflict detection aid. Conflict Resolution Performance Operational errors due to failure to resolve conflict. Operational errors occurred when the separation standards were violated and aircraft were within 5 miles and 1,000 ft of each other. Across all the conditions, six operational errors were recorded out of 198 conflict situations. Fourteen conflicts were resolved only 60 sec or less to the time of conflict and are indicative of alarming situations.


295

TABLE 1 Resolution Performance Distribution (Operating Errors, Alarming, and Risky) by Scenario

Proportion Low

Aid

Scenario

CD CD+NI

Low 1 Low 2 (rotated Low 1) Low 3 (rotated high 3)


CD+NI+CR

High

CD CD+NI

CD+NI+CR Total

High 1 High 2 (rotated High 1) High 3

Operation Errors (< 0 sec)

Alarming (Resolved < 60 sec)

Risky (Resolved < 120 sec)

Total

5

5 3

4 4

14 7

1

5

6

3

5 1

8 2

2 14

12 26

14

1

6

Note. CD = conflict detection; NI = negotiation information; CR = conflict resolution.

Twenty-six conflicts were resolved between 60 sec and 120 sec to conflict (“risky” situations). Table 1 presents the distribution of operational errors, alarming, and risky situations across conditions. Conflict resolution time is the time prior to conflict that ATCo resolved the conflict. The interaction between proportion of free-flying aircraft and the aid condition was significant, F(2, 20) = 4.05, p < .05. Figure 5 presents the interaction plot for conflict resolution time. When only the conflict detection aid was available, the resolution performance was better at higher proportion of free-flying aircraft. However when all the aids were available (CD+NI+CR) performance was better at low proportion of free-flying aircraft compared to high proportion of free-flying aircraft. In terms of main effects, there was no significant difference in conflict resolution time between proportions of the free-flying aircraft. A significant difference was found for conflict resolution time for the aid condition, F(2, 20) = 6.28, p < .05. The result of paired comparison using Tukey’s honestly significant difference (HSD) revealed that the performance was best in the condition when the negotiation information aid (CD+NI) was available (M = 176.94 sec, SD = 8.28 sec), compared to the condition when only the conflict detection aid (CD) was available (M = 149.32 sec, SD = 14.31 sec). The resolution performance was worst in the condition when all three aids (CD+NI+CR) were present (M = 146.5 sec, SD = 7.24 sec) and was different from when the negotiation (CD+NI) aid was available.


296

JHA ET AL.

FIGURE 5

Conflict resolution performance.


Type of resolution. Differences between experimental conditions and type of resolution selected were analyzed using a two-way chi-square test. The result of the analysis suggests that ATCos’ choice of resolution differed across the six experimental conditions, χ 2 (10) = 18.51, p < .05. (These results include resolutions provided to non-free-flying aircraft.) Overall, assigning aircraft a higher altitude was preferred by ATCos compared to lower altitude followed by the heading changes. Higher altitude resolution was preferred in high free-flying condition (count = 93) compared to low free-flying condition (count = 81). When the proportion of free-flying aircraft was lower, ATCos preferred assigning a higher altitude when conflict resolution aid was provided compared to other conditions. However, this trend was not consistent when the proportion of free-flying aircraft was high. Table 2 shows the contingency table for resolution type across various conditions. Additionally, three one-way chi-square tests were performed for each resolution type to find differences between experimental conditions. Result of this analysis revealed no significant differences between experimental conditions for any of the resolution types. Regression analysis. A regression analysis to predict conflict resolution time was conducted to analyze the effect of type of aircraft (free-flying vs. nonfree-flying) in conflict, conflict detection time, proportion of free-flying aircraft (low or high), number of aircraft outside the sector, and overlapping conflict on conflict resolution time. The type of aircraft was coded 0 for non-free-flying and 1 for free-flying. The overlapping conflict was coded as 1 if there was another conflict in progress or 0 if there was no other conflict. The results of our analysis suggest a significant regression, R2 = 0.72, F(5, 247) = 125.45, p < .05. The regression equation is:


297

TABLE 2 Contingency Table for Type of Resolution Type of Resolution


Proportion

Aid

Higher Altitude

Low

CD CD+NI CD+NI+CR

22 27 34

High

CD CD+NI CD+NI+CR

33 31 29 29.3 176

Expected counts Total

Lower Altitude 17 14 10 10 10 8 11.5 69

Heading Change

Total

5 3 0

44 44 44

1 3 7 3.2 19

44 44 44


Conflict Resolution Time = 319 (Constant) − 196.54∗ Type of Aircraft (0 vs. 1) + 0.01∗ Conflict detection time − 2.115∗ Number of aircraft outside the sector − 5.62∗ Overlapping conflict + 8.857∗ Proportion of free-flying aircraft (number of non-free-flying/number of free-flying aircraft). Greater conflict resolution time corresponded to better performance. Coefficients for type of aircraft and conflict detection time were significant. The type of aircraft (non-free-flying vs. free-flying) had most impact on conflict resolution time: Resolution time was reduced for free-flying aircraft by 196.54 sec. This result was expected, as controllers were instructed to resolve free-flying aircraft only 240 sec before the conflict compared to non-free-flying aircraft, which could be resolved at any point based on ATCo discretion. Conflict detection time had a very small positive effect on conflict resolution time (mean conflict detection time across all condition was 381.62, sec, which resulted in 3.86 sec in early conflict resolution); that is, detecting the conflict early had small benefits in terms of early conflict resolution, indicating that ATCos “decoupled” these tasks in time. Handoff Performance Results include data for non-free-flying aircraft. There were no significant main effects or interactions for the handoff acceptance times or in the response time for initiating handoff across various conditions. Overall, 137 handoffs were missed out of 1,923 (7%), across all conditions, resulting in operational errors due to missed handoffs. The interaction between proportion of free-flying aircraft and the aid condition on missed handoff frequency was significant, F(2, 20) = 5.77, p < .05.


298

JHA ET AL.

FIGURE 6

Missed handoffs (numbers).


Figure 6 shows the interaction plot for missed handoffs. When only the conflict detection aid was available, the number of missed handoffs was similar at a higher proportion of free-flying aircraft (M = 2.00, SD = 1.04) and a lower proportion of free-flying aircraft (M = 2.09, SD = 0.96). When a conflict detection aid and negotiation information was available, the performance was better (fewer missed handoffs) at a higher proportion of free-flying aircraft (M = 0.45, SD = 0.27) compared to a lower proportion of free-flying aircraft (M = 2.27, SD = 0.81). However, when all the aids were available (CD+NI+CR), performance was better at a low proportion of free-flying aircraft (M = 2.54, SD = 0.76) compared to a high proportion of free-flying aircraft (M = 3.63, SD = 0.93). There was also a significant main effect of aid condition on missed handoffs, F(2, 20) = 4.011, p < .05. The result of paired comparison using Tukey’s HSD revealed that the performance was best in the condition when the negotiation information aid (CD+NI) was available (M = 1.36, SD = 0.44), compared to other conditions. ATCos tended to hand off the aircraft before it reached the circle (count = 1,083) compared to when the aircraft had crossed the circle (count = 703). ATCos were instructed to initiate handoffs when the aircraft had crossed the handoff circle and before it reached the sector boundary. An analysis of late handoffs showed a significant interaction between proportion of free-flying aircraft and the aid condition F(2, 20) = 9.85, p < .05 for late handoffs. Figure 7 shows the interaction plot for late handoffs. When only the conflict detection aid was available the number of late handoffs was fewer at a higher proportion of free-flying aircraft (M = 8.45, SD = 2.07) compared to a lower proportion of freeflying aircraft (M = 12, SD = 2.71). When a conflict detection aid and negotiation information was available the performance was better (fewer missed handoffs) at a



FIGURE 7

299

Late handoffs (numbers).

higher proportion of free-flying aircraft (M = 6.27, SD = 2.3) compared to a lower proportion of free-flying aircraft (M = 12.56, SD = 2.55). However when all the aids were available (CD+NI+CR) performance was better at a low proportion of free-flying aircraft (M = 9.36, SD = 2.35) compared to a high proportion of free-flying aircraft (M = 14.91, SD = 2.67). In terms of main effects, a significant difference was found for number of late handoffs for the proportion of free-flying aircraft F(1, 10) = 5.01, p < .05. The number of late handoffs was greater for a low proportion of free-flying aircraft (M = 11.27, SD = 2.53) compared to the high proportion condition. A significant difference was also found as a function of the aid condition, F(2, 20) = 8.70, p < .05. The result of paired comparison using Tukey’s HSD revealed that the performance was best in the condition when a negotiation information aid (CD + NI) was available (M = 9.36, SD = 2.27). Secondary Task Performance (Updating Flight Progress Strips) Analysis on secondary task performance for updating flight progress strips was not performed because overall performance on this task was extremely low and in most of the cases ATCos either did not perform this task at all or merely checked the flight progress strip at the time of initiating handoffs. Aid Performance Conflict detection aid. Overall, 186 conflicts were detected before the conflict detection aid (i.e., earlier) compared to 209 after the detection aid came up (the data include non-free-flying aircraft). The results suggest that the aid helped controllers in detecting conflict. A detailed analysis of how controllers used the

300

JHA ET AL.

aid and their belief regarding whether it improved their performance (based on subjective ratings) is presented in Jha et al. (2009).


Conflict resolution aid. A comparative analysis was performed between the resolution advisory generated by aid and the resolution selected by the controllers. Our result indicates ATCo-generated advisories had moderate conformance with the conflict resolution aid. ATCos selected the same resolution (higher altitude) as the aid nearly 80% of the time and resolved the same aircraft nearly 70% of the time. There were 25 instances where the ATCo resolution differed by either type of resolution or aircraft selected for resolution, and 7 instances where both differed. These differences could be due to differences in style of ATC as various degrees of freedom are usually available to separate aircraft. DISCUSSION Overall Impact of Advanced Air Traffic Concepts on Controller Performance The first research issue examined in this study was how successful ATCos are in ensuring safe operations in advanced ATM concepts that involve time-based tactical conflict resolution. In this study, ATCos were very successful in detecting conflicts across all conditions: 100% of the conflicts were detected with average look-ahead time of 350 sec. The results are consistent with earlier research findings (Galster, Duley, Masalonis, & Parasuraman, 2001; Metzger, Galster, & Parasuraman, 1999; Metzger & Parasuraman, 2001) on conflict detection performance when a conflict detection aid is available. The result is also consistent with Rovira’s (2006) findings on conflict detection performance with various proportions of free-flying aircraft. Conflict detection performance did not vary with presence of negotiation information or conflict resolution aid. This outcome was anticipated because these aids were not designed to provide assistance with the conflict detection task. In terms of conflict resolution (rather than detection), we hypothesized that conflict resolution performance would be degraded under air traffic concepts (such as those tested) involving trajectory negotiations: This hypothesis was supported. Conflict resolution performance (as measured in terms of operational errors) was significantly impaired when free-flying aircraft failed to negotiate a conflict and controllers had to intervene. The conflict resolution performance result is consistent with predictions of various human factors researchers of degraded performance in a management by exception paradigm (Endsley et al., 1997; Wickens et al., 1998). Overall, controllers made six operational errors and experienced 14 alarming situations in which the conflicts were resolved only 60 sec before the operational error. More operational errors were noted in the condition with



301

fewer free-flying aircraft and with only a conflict detection aid available. This effect is hard to explain and it could be interpreted as being due to the particular configuration of the scenario. However, as this scenario was rotated to create another scenario, we should have seen a similar number of operational errors in other scenario, which was not the case. Overall, controllers also failed to initiate handoff for 130 aircraft that also constitute operational errors. One possibility of this could be due to task overload. Another could be the experimental definition chosen for handoff success, which might not have been the same as that currently used by ATCos. We found that average time for resolving conflict for non-free-flying aircraft was around 250 sec before the conflict compared to 168 sec for free-flying aircraft. Controllers were instructed to allow free-flight pilots to separate themselves and to intervene only 4 min before the conflict. The result suggests that controllers were able to differentially treat each type of aircraft in terms of separation time or separation distance. The conflict detection time had a significant but a small positive effect on conflict resolution time based on results of regression analysis; that is, detecting the conflict early had little benefits in terms of earlier resolution. The operational errors due to failure to resolve conflict associated with freeflying aircraft could be because the trajectory negotiation concept breaks the conflict detection and resolution task for controllers into two independent tasks. ATCos are trained to proactively resolve the conflict and they tend to resolve the conflict soon after they have detected one. However, the trajectory negotiation concept requires that any resolution task be placed in a suspended mode until a later time (2 min). Controllers might have had difficulties in resuming the task due to other ongoing activities (accepting aircraft in the sector or initiating handoffs) that resulted in operational errors. This effect is also supported by Sheridan, Corker, and Nadler’s (2006) discussion on prospective memory (Winograd, 1989; ability to remember future tasks) and how task performance can be seriously degraded in a multitasking environment with high task load. Another reason for degraded performance could be that controllers were not able to prioritize tasks in the correct order. Although we instructed the controllers that conflict detection and resolution was their highest priority task, research indicates (Willems & Truitt, 1999) that under high task load, controllers have difficulties in prioritizing tasks. The trajectory negotiation concept required a shift in paradigm that is contradictory to their current training practices where controllers are taught to be proactive and always act when in doubt. The proposed change in approach has been pointed out to be difficult by controllers (Bolic & Hansen, 2005). Results of this study confirm this finding, as many controllers pointed out during the course of the study that it would be easier for them to resolve the conflict and move on rather than having to continuously monitor aircraft progress and wait for them to resolve it themselves. They further suggested


302

JHA ET AL.

that the concept explored during this study gave them a false sense of security that aircraft will resolve themselves and this would make them very nervous in the real situation. Furthermore, we noted that controllers handed off the aircraft well within the sector even when they were instructed not to handoff the aircraft before it reached the handoff circle close to the sector boundary. The fact that the controllers were handing off the aircraft much ahead of time could be a working strategy to reduce the workload. Once the aircraft was handed off the controllers were no longer were responsible for it and did not have to pay attention to those aircraft. However, this strategy could lead to greater workload for controllers in the downstream sector as they have to take control of aircraft well before it is anticipated to be in their sector. Also, the performance on the secondary task of checking the flight progress strip was not often performed or in certain cases it was observed it was done at the time of handing off aircraft, a clear indication of task shedding. This result is consistent with Sperandio’s (1978) findings on changes in controller working strategy with traffic as density as a workload management strategy (see also Parasuraman & Hancock, 2001). Impact of Decision Aids on Conflict Resolution Performance We hypothesized that superior conflict resolution would occur when both negotiation information and the conflict resolution aid were available, but this was not confirmed. Best performance (in terms of conflict resolution time) in both aircraft proportion conditions occurred for the aid with conflict detection and negotiation information. The addition of conflict resolution information had an advantage when there were fewer free-flying aircraft, but resulted in worse performance (worse even than only the conflict detection aid) with more free-flying aircraft. On average (across both aircraft type conditions) the resolution times with the conflict resolution aid were similar to when only conflict detection aid was available, and best performance was with the negotiation aid. The analysis suggested moderate conformance of the participants with the conflict resolution advisory. Overall resolution determined by ATCo (either type or aircraft selected for resolution) differed by less than 50% (25 times out of 66) than the advisory suggested by aid. Controllers consistently preferred to climb the aircraft to resolve the conflict then other maneuvers. Moderate conformance with conflict resolution aid also suggests that ATCos were actively engaged in the task and were not overrelying on the resolution aid. Our results are consistent with the findings of Corker (2004), who concluded that conflict resolution aids are rarely used for planning in high traffic density conditions. The reason for this could be attributed to the fact that ATCos are trained to always verify the advisory suggested by any automation before acting on it (this fact was brought to light by controllers during the course of the study). Bolic


303

and Hansen (2005) also reported that controllers are trained to believe in the radar screen more than the decision aid. This fact might have nullified any potential gains in term of early conflict resolution when the aid was provided.


Impact of Proportion of Free-Flying Aircraft Results of our analysis did not support our hypothesis of overall degraded controller performance with a higher proportion of free-flying aircraft. In fact, results of the regression analysis showed that conflict resolution time was better for a higher proportion of free-flying aircraft compared to a low proportion of freeflying aircraft. We did not find other significant main effects of traffic proportion on ATCo performance. However, there were some significant interactions of aid type and proportion of free-flying aircraft. The number of missed or late handoffs was greater for the condition with more free-flying aircraft in the condition with the most decision aiding, perhaps implying a workload effect due to the aid complexity. In the other aid conditions, the condition with more free-flying aircraft showed the same or better performance in terms of missed or late handoffs, as might be expected if there was less workload associated with fewer controlled flights. Also, when there were more free-flying aircraft, having access to conflict resolution as well as negotiation information and conflict detection resulted in slower resolution than when resolution information was not present. In contrast, with fewer free-flying aircraft, participants with both resolution and negotiation information improved performance when there were fewer free-flying aircraft. In both aircraft proportion conditions, as noted earlier, providing negotiation information resulted in the best performance.

CONCLUSIONS Understanding the implications of advanced ATM concepts on controller’s performance is very important. This knowledge can help guide the development of a NextGen system architecture, allocation of functions, and decision support technology. In this article we have simulated ATCo performance in advanced ATM concepts and decision support technology. The results highlight that although the new concepts might have potential to increase the air traffic system capacity, they could also lead to operational errors. Additionally, ATCos’ handoff strategy was impacted adversely. Our findings confirmed the advantage of providing a conflict detection aid with 6-min look-ahead capability and displaying negotiation information. However there were no performance advantages for providing a conflict resolution aid.

304

JHA ET AL.


REFERENCES Andrews, J. W., & Hollister, W. M. (1997). Impact of intent based maneuver constraints on alert rates. Air Traffic Control Quarterly, 5(3), 157–179. Beers, C., & Huisman, H. (2001, December 3–7). Transition between free flight and managed airspace: A controller’s perspective. Paper presented at the 4th International Air Traffic Management R&D seminar, FAA/Euriocontrol ATM 2001 Seminar, Santa Fe, NM. Bolic, T., & Hansen, M. (2005, July). User request evaluation tool (URET) adoption and adaptation, three center case study. Paper presented at the 6th ATM R&D Seminar, Baltimore, MD. Brudnicki, D., & McFarland, A. (1997). User Request Evaluation Tool (URET) conflict probe performance and benefits assessment (Tech. Rep.). McLean, VA: MITRE. Castano, D., & Parasuraman, R. (1999). Manipulation of pilot intent under free flight: A prelude to not-so-free flight. In Proceedings of the 10th International Symposium on Aviation Psychology (pp. 170–176). Columbus: The Ohio State University. Corker, K. (2004, June). User request evaluation tool: Implementation and fielding lessons learned. Presentation at FAA/NASA IAIPT J12 Human Factors TIM, Atlantic City, NJ. Corker, K., Fleming, K., & Lane, J. (1999). Measuring controller reactions to free flight in a complex transition sector. Journal of ATC, 4, 9–16. Dekker, S. W. A., & Woods, D. D. (1999). To intervene or not to intervene: The dilemma of management by exception. Cognition, Technology & Work, 1, 86–96. Duley, J., Galster, S., Masalonis, J., Hilburn, B., & Parasuraman, R. (1997, October). En route controller information requirements from current ATM to free flight. Paper presented at the 10th CEAS Conference on Free Flight, Amsterdam, the Netherlands. Endsley, M., Mogford, R., Allendoerfer, K., Snyder, M., & Stein, E. (1997). Effects of free flight condition on controller performance, workload and situation awareness (Tech. Rep. No. DOT/FAA/CT-TN97/12). Atlantic City, NJ: Federal Aviation Administration William J. Hughes Technical Center. Erzberger, H. (2001). The automated airspace concept. In Proceedings of 4th USA/Europe ATM R&D Seminar. Santa Fe, NM. Farley, T. C., Hansman, R. J., Amonlirdviman, K., & Endsley, M. (2000). Shared information between pilots and controllers in tactical air traffic control. Journal of Guidance, Control, and Dynamics, 23, 826–836. Galster, S., Duley, J., Masalonis, A., & Parasuraman, R. (2001). Air traffic controller performance and workload under mature free flight: Conflict detection and resolution of aircraft self-separation. International Journal of Aviation Psychology, 11, 71–93. Jha, P., Bisantz, A., Parasuraman, R., & Drury, C. (2009). Air traffic controllers performance in advanced air traffic management concepts: Part II. Workload and trust. Journal of Air Traffic Control, 51(2), 46–52. Joint Planning and Development Office. (2007). Concept of operations for Next Generation Air Transportation System, version 1.2. Retrieved from www.jpdo.aero Kirk, D. B., Heagy, W. S., & Yablonski, M. J. (2001). Problem resolution support for free flight operations. IEEE Transactions on Intelligent Transportation Systems, 2(2), 72–80. Krozel, J., & Peters, M. (1997). Conflict detection and resolution in free flight. Air Traffic Control Quarterly, 5(3), 181–212. Lozito, S., McGann, A., & Corker, K. (1991). Data link air traffic control and flight deck environments: Experiments in flight crew performance. In R. Jensen (Ed.), Proceedings of the 6th Symposium on Aviation Psychology (pp. 1009–1015). Columbus: The Ohio State University. Mackintosh, M., Dunbar, M., Lozito, S., Cashion, P., Mcgann, A., & Dulchinos, V. (1998). Self separation from air and ground perspective. In Proceedings of 2nd USA/Europe Air Traffic Management R& D Seminar, Orlando, FL.



305

Masalonis, A. J., Le, M. A., Klinge, J. C., Galster, S. M., Duley, J. A., Hancock, P. A., . . . Parasuraman, R. (1997). Air traffic control workstation mock-up for free flight experimentation: Lab development and capabilities. In Proceedings of the Human Factors and Ergonomics Society 41st annual meeting. Metzger, U., Galster, S., & Parasuraman, R. (1999). Effects of a conflict detection aid and traffic density on controller mental workload and performance in an ATC task simulating free flight. In Proceedings of 10th International Symposium of Aviation Psychology. Metzger, U., & Parasuraman, R. (2001). The role of the air traffic controller in future air traffic management: An empirical study of active vs. passive monitoring. Human Factors, 43, 519–528. Metzger, U., Rovira, E., & Parasuraman, R. (2003). Controller performance, workload and attention allocation in distributed air ground traffic management: Effects of mixed equipage and decision support. In Proceedings of 11th International Symposium of Aviation Psychology (pp. 541–547). Columbus, OH. NASA. (1999). Distributed air/ground traffic management (DAG-TM) concept definition, version 1. National Aeronautics and Space Administration. Parasuraman, R., & Hancock, P. A. (2001). Adaptive control of workload. In P. A. Hancock & P. E. Desmond (Eds.), Stress, workload, and fatigue (pp. 305–320). Mahwah, NJ: Erlbaum. Parasuraman, R., Hilburn, B., & Hoekstra, J. (2001). Free flight. In W. Karwowski (Ed.), International encyclopedia of ergonomics and human factors (pp. 1005–1008). New York, NY: Taylor & Francis. Pritchett, A., & Hansman, J. (1997). Pilot nonconformance to alerting system command. In Proceedings of the 9th International Symposium on Aviation Psychology. Columbus, OH. Rovira, E. (2006). Human performance and workload in future air traffic management: Decision support tools for controller. Unpublished doctoral dissertation,Catholic University of America, Washington, DC. RTCA. (1995). Report of the RTCA Board of Directors select committee on free flight. Washington, DC: Author. Sheridan, T., Corker, K., & Nadler, E. (2006). Final report and recommendations for research on human automation interaction in the next generation air transportation system (Department of Transportation Tech. Rep. No. DOT-VNTSC-NASA-06-05). Cambridge, MA: U.S. Department of Transportation, John A. Volpe National Transportation Systems Center. Sperandio, J. C. (1978). The regulations of working methods as a function of work load among air traffic controllers. Ergonomics, 21, 195–202. Westrenen, F., and Groeneweg, J. (2003). Cooperative conflict resolution: An experimental validation of aircraft self separation in a classroom environment. International Journal of Aviation Psychology, 13, 233–248. Wickens, C., Mavor, A., Parasuraman, R., & McGee, J. (1998). The future of air traffic control: Human operators and automation. Washington, DC: National Academy Press. Willems, B., & Truitt, T. (1999). Implications of reduced involvement in en route air traffic control (FAA Tech. Rep. No. DOT/FAA/CT-TN99/22). Atlantic City, NJ: Federal Aviation Administration. Winograd, T. (1989). Debate: On the cruelty of teaching computer science (a response to Dijkstra). Communications of the ACM, 32, 1412–1413. Woods, D., & Roth, E. (1988). Aiding human performance II: From cognitive analysis to support systems. Le Travail Human, 51, 139–172.

Manuscript first received: April 2010