Aviator 2030 - DLR

ISSN 1434-8454 ISRN DLR-FB-2009-28

H. Eißfeldt et al.

ISRN DLR-FB--2009-28

Forschungsbericht 2009-28

Aviator 2030 Ability Requirements in Future ATM Systems II: Simulations and Experiments Hinnerk Eißfeldt Dietrich Grasshoff Catrin Hasse Hans-Juergen Hoermann Dirk Schulze Kissing Claudia Stern* Jürgen Wenzel** Oliver Zierke Deutsches Zentrum für Luft- und Raumfahrt Institut für Luft- und Raumfahrtmedizin Abteilung für Luft- und Raumfahrtpsychologie, Hamburg *Abteilung für operationelle Medizin, Köln **Abteilung für Flugphysiologie, Köln

Aviator 2030 Ability Requirements in Future ATM Systems II: Simulations and Experiments

Deutsches Zentrum für Luft- und Raumfahrt e. V. Institut für Luft- und Raumfahrtmedizin, Luft- und Raumfahrtpsychologie Sportallee 54a, 22335 Hamburg Hamburg, 21. Dezember 2009

Institutsleiter: Prof. Dr. med. Rupert Gerzer Abteilungsleiter: Dr. phil. Peter Maschke

Verfasser: Dr. phil. Hinnerk Eißfeldt Dipl.-Psych. Dietrich Grasshoff Dipl.-Psych. Catrin Hasse Dr. phil. Hans-Juergen Hoermann Dr. rer. nat. Dirk Schulze Kissing Dr. med. Claudia Stern Prof. Dr. Jürgen Wenzel Dr. phil. Oliver Zierke

Zukunft der Luftfahrt, Personalauswahl, Fähigkeitsanforderungen Aviator 2030 - Ability Requirements in Future ATM Systems II: Simulations and Experiments Der Bericht gibt einen Überblick über das DLR-Projekt Aviator 2030. In Aviator 2030 wird ein Zukunftskonzept ATM aus Sicht der beruflichen Experten entwickelt und an gekoppelten Simulationsanlagen erprobt. Zukünftige Fähigkeitsanforderungen an operationelle Luftfahrtberufe werden so frühzeitig bestimmt und ermöglichen eine rechtzeitige Anpassung von Auswahlverfahren. Es werden die Ergebnisse von verschiedenen Workshops mit Fluglotsen und Piloten beschrieben. Anschließend werden zwei unterschiedliche Simulationen vorgestellt. AviaSim ist an der beruflichen Realität orientiert und wurde mit erfahrenen Fluglotsen und Piloten (N=20) eingesetzt. Die Self Separation Airspace Simulation (SSAS) hingegen stellt eine abstrakte Aufgabe aus der Verkehrsflusssteuerung und wurde mit (N=90) Berufsbewerbern durchgeführt. In beiden Versuchsdurchführungen wurden neben anderen Messungen auch Blickbewegungsmessungen vorgenommen. Mit SSAS konnte gezeigt werden, dass MonitoringParameter das Systemverständnis vorhersagen und signifikant mit Aspekten der Aufgabenleistung korrelieren. Die Ergebnisse der AviaSim zeigen u.a. eine Zunahme von Fähigkeitsanforderungen für Piloten bei im Wesentlichen gleichbleibenden Anforderungen für Fluglotsen. Die Übernahme der Staffelungsverantwortung ins Cockpit (airborne separation) geht also nach diesen Ergebnissen nicht einher mit einer Reduktion der Eignungsanforderungen am Boden. Future aviation, Personnel selection, Ability requirements Aviator 2030 - Ability Requirements in Future ATM Systems II: Simulations and Experiments This report provides a summary of the DLR project Aviator 2030. Based on domain experts’ points of view, Aviator 2030 develops future scenarios of ATM. Key aspects of these scenarios are tested with human operators in low-fidelity simulations which combine on-board and ATC systems. Potential changes in ability requirements for pilots and air traffic controllers are identified and allow for timely adjustment of selection profiles. Results of workshops with air traffic controllers and pilots are reported. Two different simulations are described: AviaSim is oriented towards the realistic job content and was applied with experienced pilots and air traffic controllers (N=20). The ‘selfseparation airspace simulation (SSAS)’ provides a more abstract flow control task and was conducted with 90 ab-initio air traffic controller and pilot candidates. In both experiments, amongst other measures eye movements were analyzed. With SSAS it could be shown that monitoring parameters have predictive power for system understanding and performance on the task. Amongst others results, the AviaSim experiments indicate increasing ability requirements for pilots whereas for air traffic controllers requirements overall remain constant. According to these results, taking responsibility for (airborne) separation into the cockpit does not reduce ability requirements on the ground.

Contents 1

Project Aviator 2030 ................................................................................................................ 5 1.1 1.2 1.3 1.3.1 1.3.2 1.4

Workshop design .............................................................................................. 6 Future Workshops findings ................................................................................ 7 Integrative workshop findings ........................................................................... 9 Workshop findings F-JAS Aviator 2030 .............................................................. 9 Workshop proposals for simulation scenarios................................................... 15 Deduction of experimental scenarios................................................................ 16

Part A: Transfer of control in Free Flight Airspace .......................................................................... 17 2

Current state of research ....................................................................................................... 17 2.1 2.2 2.3

Free flight and self-separation.......................................................................... 17 Transitions between Free Flight Airspace and Managed Airspace ..................... 19 Assistance systems........................................................................................... 21

3

Problem................................................................................................................................. 25

4

Experimental setup ................................................................................................................ 27 4.1 4.1.1 4.1.2 4.1.3 4.2 4.3 4.4

5

Method ................................................................................................................................. 37 5.1 5.2 5.2.1 5.2.2 5.2.3 5.3 5.4 5.5

6

Subjects........................................................................................................... 38 Objective data/measurements .......................................................................... 38 Performance data ............................................................................................ 38 Gaze tracking .................................................................................................. 38 Ophthalmology................................................................................................ 39 Ability requirements......................................................................................... 41 Questionnaires................................................................................................. 43 Debriefing ....................................................................................................... 44

Results................................................................................................................................... 45 6.1 6.2 6.3 6.4 6.5 6.6 6.7

7

Simulation platform ......................................................................................... 27 Flight simulator................................................................................................ 27 ATC simulator.................................................................................................. 29 Integrated simulator environment .................................................................... 29 Airspace structure............................................................................................ 31 Scenarios ......................................................................................................... 32 Experimental design and procedure ................................................................. 35

Objective data ................................................................................................. 46 Eye tracking data ............................................................................................. 46 Ophthalmic data.............................................................................................. 48 Ability requirements......................................................................................... 49 Questionnaires................................................................................................. 53 Correlation of measurements........................................................................... 56 Results of debriefings....................................................................................... 57

Discussion ............................................................................................................................. 58

Contents Part B: Identifying operators monitoring appropriately through the measurement of eye movements................................................................................................................................... 61 8

Introduction .......................................................................................................................... 61 8.1 8.2 8.3 8.4

9

Monitoring automated systems........................................................................ 61 Individual differences in monitoring ................................................................. 62 Monitoring performance in future personnel selection ..................................... 64 Devising a normative model of monitoring behavior (Assumptions).................. 65

Method ................................................................................................................................. 66 9.1 9.2 9.3 9.4 9.5 9.6 9.7

Simulation tool ................................................................................................ 66 Experimental paradigm .................................................................................... 69 Scenario construction ...................................................................................... 70 Measurements................................................................................................. 71 Experimental device ......................................................................................... 73 Test subjects .................................................................................................... 74 Procedure ........................................................................................................ 74

10 Results................................................................................................................................... 75 10.1 10.2

Link between eye movement parameters and manual performance ................. 75 Group comparisons ......................................................................................... 77

11 Discussion ............................................................................................................................. 82 Conclusion ................................................................................................................................... 85 12 Aviator 2030 - conclusion of the project ................................................................................ 85 12.1 12.2 12.3

Prospective human factors research ................................................................. 86 Adjustment of selection profiles....................................................................... 87 Outlook ........................................................................................................... 89

13 Appendix............................................................................................................................... 91 13.1 13.2 13.3 13.4 13.5 13.6 13.7 13.8

Additional F-JAS scales..................................................................................... 91 F-JAS profiles for ATCOs .................................................................................. 96 F-JAS profiles for pilots .................................................................................... 97 Detailed results for NASA-TLX.......................................................................... 98 Detailed results for SART................................................................................ 101 Reference list ................................................................................................. 103 List of figures................................................................................................. 109 List of tables .................................................................................................. 111

1 Project Aviator 2030 Improvements in air traffic management (ATM) and aircraft systems as well as organisational structures have become key challenges of aviation in the 21st century. These are especially important with regard to the considerable increase in air traffic. To allow maximum capacity and safety as well as minimum impact on environment and cost, Single European Sky (SES) will be implemented to coordinate air traffic in Europe. According to the SESAR’s Concept of Operation: “humans (with appropriate skills and competences and duly authorised) will constitute the core of the future European ATM System’s operations. However, to accommodate both the expected traffic increase and the reference performance framework, an advanced level of automation will be required. The nature of human roles and tasks within the future system will necessarily change. This will affect system design, current staff selection, training (especially for unusual situations and degraded modes of operations), competence requirements and relevant regulations” (SESAR, 2007a, p. 27). The key question of the project Aviator 2030 deals with changes that will concern pilots and air-traffic controllers introducing the SES. Which modifications of operators’ tasks, roles and responsibilities can be expected? Will ab-initio pilots or air traffic control trainees selected today ever work in the ATM system reflected in the current job analysis? If not, what ability requirements will change; what will remain?

Figure 1: Flowchart of the project Aviator 2030

Based on domain experts’ points of view, Aviator 2030 develops future scenarios of ATM. Key aspects of these scenarios are tested with human operators in lowfidelity simulators which combine on-board and ATC systems. Thus, potential changes in ability requirements for pilots and air-traffic controllers will be identified in advance and allow for timely adjustment of selection profiles (Figure 1).

6

1.1

Project Aviator 2030

Workshop design

Traditional system developments have shortcomings as they relate to a technicaloriented approach, which pays little attention to other important aspects of new technology, such as ergonomic, organizational, social, and political aspects. Therefore user participation, seen as the activities performed by end users during system development, should be part of the design process (Gulliksen, 2000; Kautz, 2009; Nielsen, 1993). To achieve this in Aviator 2030 a set of workshops with experienced job holders was planned with the support of the Deutsche Flugsicherung GmbH (DFS) and of Deutsche Lufthansa AG (DLH). Starting with so-called ‘Future Workshops’ (WS1, WS2) to develop ideas among the relevant professional groups, an integrative workshop (WS3) was organized about half a year later to establish a shared picture of future ATM systems and to direct the research towards relevant issues. In a final workshop (WS4) towards the end of the project, the scientific results would then be presented and discussed with the participants of the preceding workshops. The Future Workshop is an appropriate method to integrate domain experts in early phases of the development process. This technique, developed as ‘Zukunftswerkstatt’ by Jungk and Muellert in the 1970s (Dauscher, 2006; Jungk & Muellert, 1987), enables a group of people to generate new ideas or solutions to social, organizational and technological problems. It starts with the preparation phase (0): the introduction of the topic, the method and the schedule of the workshop. In the critique phase (1), the topic is analysed. Next, in a fantasy phase (2), participants work out a utopia to draw a picture of future possibilities. The workshop finishes with the implementation phase (3): ideas are checked and evaluated with regard to their practicability.

Figure 2: Phases of the Future Workshops involving pilots and air-traffic controllers separately

Future Workshops with experienced air-traffic controllers and pilots have been conducted separately to obtain job incumbents` expectations regarding their future tasks, roles and responsibilities. The first two-day Future Workshop was conducted with nine air-traffic controllers and the second involved ten pilots. Both workshops were designed correspondently. Figure 2 presents the main phases of the Future Workshops and the analysis of their results. In the first step participants were asked to anticipate how they might be working in twenty years time or how a system that they could not work with would look like. Based on their experience in the workshop, participants highlighted critical aspects of

7

Introduction

future developments, derived objectives for the future and developed visionary scenarios, which they evaluated with regard to their feasibility. About four months later the same pilots and air-traffic controllers met for an integrative workshop to exchange ideas and concepts (Figure 3). The participants discussed and integrated their points of view. Finally, the domain experts collected scenarios that describe conditions to simulate future aviation and developed standardized material to describe ability requirements for future aviators.

Figure 3: Integrative workshop involving both pilots and air-traffic controllers

According to Cresswell (1994), “qualitative study” is defined as an inquiry process of understanding a social or human problem, based on building a complex, holistic picture, formed with words, reporting detailed views of informants, and conducted in a natural setting. In short, the aim of qualitative research is to understand a phenomenon from the perspective of the participants. With this goal in mind, the workshops are studied. In the following, selected findings of the Future Workshops with pilots and controller are presented. More detailed information about the workshops is provided by Bruder et al. (2009).

1.2

Future Workshops findings

In an information session at the beginning of the Future Workshops, participants were informed about the idea of the project in general and more specifically on the goals of the ‘Vision 2020’ for European aeronautics (ACARE, 2004). As the relevant document had just recently been released, the Concept of Operations (CONOPS) for the Single European Sky (SESAR, 2007a) was introduced by making use of the Video ‘The SESAR target concept’ published by the SESAR Consortium. Pilots and controllers were asked for their criticisms about ‘Vision 2020’ and SESAR CONOPS. They mentioned their top five risks of future aviation, which are provided in Table 1. Both ATC and pilots emphasise the risk of single workplace replacing teamwork, a shift of competencies, an incapacitation, or an inappropriate system design.

8


Table 1: Top five risks of future aviation

Upon collecting risks about future aviation, participants were asked to list their ideas for future aviation. Table 2 presents the top five ideas of controller and pilots: Negotiation of trajectory, perpetuation of training standards, communication between the operators, and intuitive or indiscernible humanmachine interfaces. Table 2: Top five ranking of preferred ideas for the future pilots

rank of idea

air traffic controller

trajectory is flyable and negotiable between ATC and cockpit

1

high-quality training revised via continuous task analysis

flexible human resource planning

2

well-defined and accepted roles and task allocation

consideration of workload in line with age or current state

3

intuitive human-machine interface and ergonomic workplace

safety has the highest priority

4

appropriate communication and flexible use of channels

consistent systems or indiscernible interfaces between systems

5

free flight

On basis of the preferred ideas several visionary scenarios have been developed. These scenarios dealt with the process of negotiating 4D-trajectories, with tactical planning and operating flights. Also improvement of human resource planning, and a new approach to line and recurrent training were focused as well as a first draft of a virtual workspace.

Introduction

1.3

9

Integrative workshop findings

The integrative workshop was held in March 2008 with the same pilots and airtraffic controllers that had already participated in the Future Workshops described above. Out of the 19 potential participants, four could not attend the two-day workshop. The results of a standardized technique for the description of ability requirements will now be described, as well as results of the group work and round table-discussion. The integrative workshop started with the presentation of the results of the Future Workshops. Air-traffic controllers and pilots enjoyed sharing and discussing their scenarios for the future. Secondly, mixed groups consisting of controllers and pilots elaborated several ideas: a concept of trajectory negotiation, procedures for operating flights in the future, and an integrated training system for pilots and air-traffic controllers. In general, participants developed future scenarios including ATC’s and pilots’ perspectives. Bruder et al. (2009) describe the outcome of the integrative workshop in more detail. To obtain a first impression of potential changes in ability requirements in a more standardized way, participants of the integrative workshop were finally asked to rate the ability requirements for the future ATM system. The Fleishman Job Analysis Survey F-JAS (F-JAS; Fleishman, 1992; Fleishman, 1992) was used to depict ability requirements for the future ATM system. This survey requires participants to use a 1 to 7 scale to ”rate the task on the level of the ability required, not the difficulty, time spent or importance of the ability” (Fleishman & Reilly, 1992, p.7). 1.3.1 Workshop findings F-JAS Aviator 2030 The F-JAS Fleishman Job Analysis Survey (Fleishman, 1992) is a survey measuring human abilities, providing detailed definitions and anchored rating scales for 72 scales covering the domains of cognitive, psychomotor, physical and sensory abilities as well as interactive/social and knowledge/skills scales, the latter of which is still under research. It comes with a detailed ‘Administrators Guide’ (Fleishman & Reilly, 1992a) and the ‘Handbook of Human Abilities’ (Fleishman & Reilly, 1992b), providing some theoretical background and lists of validated tests measuring certain abilities including reference data of test providers. A typical example scale is shown in Figure 4. In 1995 the F-JAS was republished with 52 scales covering cognitive, psychomotor, physical and sensory/perceptual abilities. In 1996 the F-JAS Kit Part 2 was published covering 21 social/interpersonal abilities (MRI, 1996). The F-JAS has been used at DLR in a number of studies with high rates of success. With the Aviator 2030 project a special version of the F-JAS was developed, including not only the original scale material but also anchors representing the requirements of current pilots and air-traffic controllers. These mean ratings

10


reflect the results of prior studies conducted with air-traffic controllers of Deutsche Flugsicherung GmbH ((N = 88; Eißfeldt & Heintz, 2002) and pilots of Deutsche Lufthansa AG (N = 141; Goeters, Maschke, & Eißfeldt, 2004; Goeters et al., 2004). In this special F-JAS aviator version, the mean rating for air-traffic controllers of DFS is depicted in a blue box on the left, the mean rating for pilots of DLH in a yellow box on the right side of the central scale. To integrate these anchors graphically on the scale better allows the results to be interpreted as an increase or decrease in the requirements compared to today. Figure 4 shows an example scale, as used in the project Aviator 2030, with integrated anchors for air-traffic controllers and pilots.

1. Oral Comprehension

This is the ability to listen and understand spoken words and sentences.

How Oral Comprehension Is Different From Other Abilities Written Comprehension: Involves reading and understanding written words and sentences.

Oral Comprehension: Involves listening to and understanding words and sentences spoken by others.

vs.. Oral Expression and Written Expression: Involve speaking or writing words and sentences so others will understand.

7

Requires understanding complex or detailed information that is presented orally, contains unusual words and phrases, and involves fine distinctions in meaning among words.

6 Understand a lecture on metaphysics.

5,34

5

4

4,97

Understand instructions for a sport.

3

Requires understanding short or simple spoken information that contains common words and phrases.

2 1

Understand a television commercial

Figure 4: Example scale F-JAS Aviator: Oral comprehension with added anchor scales for air-traffic controllers and pilots. Adapted from Fleishman (1992), with permission.

In an earlier study, the comparison of current and future work conditions was achieved by using the F-JAS twice: First to obtain ratings for the everyday job

Introduction

11

experience as an air-traffic controller and second, after days of training and simulation in a new data link environment to collect the ratings for the new system (Eißfeldt, Deuchert, & Bierwagen, 1999). Due to time constraints this approach was not possible for the Aviator project; however using the aviation anchors described above the workshop participants could express their view using standardized scientific material. To do so participants teamed up in pairs with a participant from the other background to support the exchange of views. Each participant then gave a personal rating for his own professional role in the light of his understanding of the future ATM system. The F-JAS aviator version proved to be easy to work with and a total of 15 sets of ratings (8 pilots, 7 air-traffic controllers) were collected. Although this sample does not reach a size allowing generalization the combination with larger existing data sets (141 pilots, 88 air-traffic controllers) should enable interpretation of ratings obtained from workshop participants. However, it has to be considered that these results are preliminary. As Figure 5 shows, many of the scales in the cognitive domain were rated very similarly for the future ATM system as for the current job requirements. For airtraffic controllers, a strong increase was found with ‘problem sensitivity’ and ‘speed of closure’; a strong decrease was rated for ‘originality’, memorization’ and ‘spatial orientation’. For pilots, a strong increase was indicated for ‘deductive reasoning’ and a strong decrease was found in ‘number facility’. Given that “abilities with mean ratings of four or greater are generally considered to be important for the job” (Fleishman & Reilly, 1992, p.10) the impression is that the profile of cognitive ability requirements will not change significantly in future ATM concepts for both professions, with some minor adjustments being proposed.

12

Project Aviator 2030 21 Time Sharing 20 Selective Attention 19 Perceptual Speed 18 Visualization 17 Spatial Orientation 16 Flexibility of Closure 15 Speed of Closure 14 Category Flexibility 13 Information Ordering 12 Inductive Reasoning 11 Deductive Reasoning 10 Number Facility 09 Mathematical Reasoning 08 Problem Sensitivity 07 Memorization 06 Originality 05 Fluency of Ideas 04 Written Expression 03 Oral Expression 02 Written Comprehension 01 Oral Comprehension

1,00

2,00

3,00

4,00

5,00

6,00

7,00

Mean Rating Airline Pilots

Aviator Pilots

Aviator ATCs

ATC

Figure 5: F-JAS Aviator 2030 - Cognitive abilities rated by pilots and air-traffic controllers in the integrative workshop.

Secondly, we looked at the similarity of ratings for pilots and controllers: in the domain of cognitive abilities, most of the ratings are not much different for the two groups. Only two of the cognitive scales showed significant differences between pilots and air-traffic controllers: ‘spatial orientation’ and ‘visualization’. As air-traffic controllers need to have a mental ‘picture’ of future movements on the radar screen, ‘visualization’ is amongst those factors rated above six on the 7point scale, whereas for pilots this cognitive ability is rated slightly above five for the current job conditions. With the future ATM concepts there was a slight increase for ‘visualization’ in both groups, as was seen with a lot of the cognitive abilities. Also ‘oral comprehension’, ‘oral expression’, ‘problem sensitivity’, ‘deductive reasoning’, ‘inductive reasoning’, ‘category flexibility’, ‘speed of closure’, ‘perceptual speed’ and ‘time sharing’ all showed a slight increase with the future ATM concepts for both professional groups. With ‘spatial orientation’ it was different; here the rating for the current job condition was very high (>6) for pilots and a bit less for air-traffic controllers. With the future ATM concepts there was a slight increase in relevance for the pilots and a sharp decrease for the air-traffic controller group. A similar but only slight tendency was found in the ratings for ‘selective attention’ and ‘information ordering’.

13

Introduction

There was not a single cognitive ability showing an opposite pattern: decrease of relevance with pilots and increase with air-traffic controllers. In a third pattern of results the relevance of abilities decreased with the future ATM concepts for both professional groups. ‘Written comprehension’, ‘written expression’, ‘originality’, ‘memorization’, ‘problem sensitivity’, ‘mathematical reasoning’, and ‘number facility’ all showed decreasing relevance with future ATM concepts, as discussed in the Aviator 2030 workshops.

52 Speech Clarity 51 Speech Recognition 50 Sound Localization 49 Auditory Attention 48 Hearing Sensitivity

47 Glare Sensitivity 46 Depth Perception 45 Peripheral Vision 44 Night Vision 43 Visual Color Discrimination 42 Far Vision 41 Near Vision 1,00

2,00

3,00

4,00

5,00

6,00

7,00

Mean Rating

Airline Pilots

Aviator Pilots

Aviator ATCs

ATC

Figure 6: F-JAS Aviator 2030 - Sensory / perceptual abilities rated by pilots and airtraffic controllers in the integrative workshop

Figure 6 shows the sensory / perceptual abilities as rated by the workshop participants with the auditory scales shown on top of the table. From the perspective of future ATM concepts it is interesting to see ‘speech-related abilities’ rated with higher relevance for future ATM in both groups of workshop participants, although in future ATM a significant reduction in voice communication through data link is expected. Overall for most scales only small changes for pilots and controllers are observed. The only significant increase is found for Visual Color Discrimination, for both controllers and pilots, reflecting the high demand in decoding colour-coded information presented on the radar screen. This is true for the radar screen of the air-traffic controller as well as for the display of traffic information in the cockpit (CDTI).

14


81 Decision Making 80 Resistance to Premature Judgement 79 Oral Fact Finding 78 Situational Awareness 77 Behavior Flexibility 76 Stress Resistance 75 Self Awareness 74 Assertiveness 73 Motivation 72 Resilience 71 Persistence 70 Sales Interest 69 Leadership 68 Persuasion 67 Oral Defense 66 Communication 65 Cooperation 64 Social Sensitivity 1,00

2,00

3,00

4,00

5,00

6,00

7,00

Mean Rating Airline Pilots

Aviator Pilots

Aviator ATCs

ATC

Figure 7: F-JAS Aviator 2030 Social/interactive abilities rated by pilots and air-traffic controllers in the integrative workshop

Figure 7 lists social / interactive abilities as they have been rated by the participants of the integrative workshop. Most of these scales are from the original F-JAS set of abilities, although some were developed by DLR (Klamm, 1997) to better cover the content of Resource Management Trainings in Aviation (Eißfeldt, 1994). An initial finding for these scales is the increasing importance of social / interactive abilities for future ATM systems for both professions in general: in 11 out of these 18 ability scales, the required level increases slightly for pilots and controllers. For pilots there is an increase in all but two ability scales, and the only significant increase is for ‘resilience’. Results of the F-JAS presented in this section reflect the opinion of experienced aviation professionals after several days of detailed work on the future of their jobs. However, using F-JAS rating scales with special aviation anchors, workshop participants in general indicated neither relief nor much intensification of cognitive and sensory / physical ability requirements. What can be foreseen is pilot and air-traffic controller profiles assimilating with regard to cognitive abilities mostly linked to the new task of airborne separation. If there is an increase in requirement levels this can be stated for pilots rather than for controllers. The only strong change to be expected is the significant increase for Visual Color Discrimination with controllers and pilots calling for appropriate

15

Introduction

design of HMI’s as well as for stringent examination of the ophthalmologic criteria by which operators are selected. 1.3.2 Workshop proposals for simulation scenarios Finally, participants designed future scenarios which should, according to their background, be simulated and tested in the ongoing project. Table 3 consists of a schematic diagram of the simulation scenarios suggested by workshop participants. The number of stars in a certain field of the table equals the number of proposals reflecting this issue: e.g. two contributions mentioned vigilance aspects when monitoring. The suggestions addressed the planning and visualising of the 4D-trajectory, emphasizing operational aspects of future ATM. Other contributions dealt with specific aspects of flight operation, as tasks allocation, teamwork and monitoring. The following basic psychological topics were pointed out: vigilance, information reception, attention, detection of changes, and communication between operators. Table 3: Matrix of simulation scenarios suggested by workshop participants 4D-trajectory planning

visualising

flight operation tasks

teamwork

monitoring

vigilance information reception attention (divided, selective) detection of changes communication (between operators)

Air traffic controllers as well as pilots especially stressed the need for research into vigilance and attention while monitoring, and communication under changing task allocation. Furthermore, they anticipated a need for further research on teamwork if communication shifts from oral to electronic mode. In general the workshop concept was well regarded by the participants and proved to be very fruitful for the project. The Future Workshops were successful at getting people to anticipate future roles, responsibilities and tasks. In addition they provided an insight into potential disputes, which have to be solved in order

16


for the successful implementation of SES. The integrative workshop made it clear that task allocation, teamwork, and monitoring in a highly-automated workplace pose a challenge to future concepts of air traffic management.

1.4

Deduction of experimental scenarios

In following up the recommendations from the workshop, several simulation concepts have been discussed by the project team covering different experimental approaches. The experimental scenarios should meet the following criteria: They • should be highly relevant to the question of future ability requirements for pilots and controllers • should involve pilots as well as controllers • should be executable by using low fidelity simulation equipment. Additional aspects, such as availability of subjects, applicability of an eye movement system, and financial and personal resources, had to be taken into account. During the discussions two main issues were found interesting and were singled out for further work: the transfer of separation responsibility and the identification of good monitoring behavior. The two simulation concepts remaining were labeled ‘realistic’ and ‘abstract’, indicating one being close to the real job in terms of integrating and enhancing existing air traffic control and flight simulators; the other developing a new simulation platform of traffic flow control which better allows a task to be controlled in an experimental setting. Eye movement analysis, as specified in the project outline of Aviator 2030, was planned for both simulations. Both concepts had advantages and disadvantages: the ‘realistic’ setting allowed an integrated approach with pilots and controllers together in one simulation and thus followed the general approach of Aviator 2030; however, for the simulation trials trained job holders would be required to gain meaningful results. The ‘abstract’ setting could use ab-initio candidates (i.e., with no prior experience) or even students as participants as no prior knowledge was needed; however, the task would be somewhat artificial and could not cover the pilots position in future ATM. Following a detailed check of project resources and potential external support (provision of pilots and controllers, or applicants during the selection process by Deutsche Flugsicherung DFS and DLH Deutsche Lufthansa AG) it was decided to follow both simulation approaches parallel to one another. As both topics are elaborated independently by different team members of the Aviator 2030 project group, the particular projects will be documented separately in this report. The ‘realistic’ focus resulted in the subproject “Transfer of Control in FreeFlight Airspace” (part A). The ‘abstract’ approach resulted in ‘Identifying operators monitoring appropriately through the measurement of eye movements (part B).

Part A: Transfer of control in Free Flight Airspace

2

Current state of research

2.1

Free flight and self-separation

The European master plan for future ATM (SESAR, 2008) is committed to operational concepts of self-separation by using airborne separation assurance systems (ASAS). Free flight is defined as “a safe and efficient flight operating capability under instrument flight rules in which the operators have the freedom to select their path and speed in real time” (RTCA, 1995). According to RTCA (1995), free flight requires airborne self-separation and its monitoring on the ground. Functionally, ASAS is the main technical prerequisite for the implementation of free flight, for which CDTI (cockpit display of traffic information) is one example. The future growth in air traffic is an issue for which free flight is intended to offer solutions. A considerable number of studies have already been carried out which look at specific effects, e.g. workload and situation awareness (Endsley, Mogford, & Stein, 1997), conflicts (Hilburn, Bakker, & Pekela, 1997) or increased risk of collisions (Hoeckstra, Ruigrok, & van Gent, 2000). Schäfer and Modin (2009) used scenario-based exercises to test the impact of introducing a Free Flight Airspace (FFAS) concept and using ASAS on pilots’ and controllers’ workload, as well as on their situation awareness. They use the situation awareness rating technique (SART; Taylor, 1990) for measuring situation awareness and the NASA Task Load Index (NASA TLX; Hart & Staveland, 1988) for measuring workload. The main result is that the workload of controllers and pilots is more balanced under free route conditions. Comparing the free flight condition with a fixed route condition, there are no significant differences in situation awareness or in workload for either controllers or pilots. In a montecarlo simulation of a comparable model of an airborne self-separation concept (Blom, Klein Obbink, & Bakker, 2008), the distributions of three traffic scenarios (a two-aircraft head-on encounter, an eight-aircraft head-on encounter, and a dense random traffic scenario) are analyzed concerning the probability of safety related events occurring. The results generally show that the model under scrutiny works sufficiently safely for en-route airspace with low air traffic demand. For the dense random traffic scenario, a potential for the clogging of multiple conflicts is identified. Thus, the authors recommend some form of coordination in conflict resolution for busy en-route sectors.

18

Part A: Transfer of Control

Cásek and Keinrath (2008) report the concept of an advanced ASAS situated in an anticipated information sharing environment, as is envisaged by SESAR and NextGen. This envisioned future system, called iFly, will provide information about the current aircraft state (e.g. position and speed), as well as advanced information about expected behavior of other aircraft, based on their intended 4D trajectories, for conflict detection purposes. In this concept, an air-to-ground and air-to-air data link architecture is presumed. For pilots, the air-to-air data link range creates a short-term awareness zone. Additionally, the system provides information on a strategic level for planning purposes via the air-to-ground data link connection to the system-wide information management (SWIM). SWIM provides the data for pilots to form a mid-term, as well as a long-term awareness zone. The ASAS itself assists with conflict detection and resolution functions in a short-term and mid-term range and also communicates with the pilots for trajectory management purposes. DiMeo et al. (2002) tested the effect of shared-separation in free flight in a single, integrated experimental setup with ATCOs and pilots. This real-time human-in-the-loop experiment was conducted in high-fidelity simulator environment. Cockpit Display of Traffic Information (CDTI) as well as User Request Evaluation Tool (URET) prototypes were integral parts of the setup. ATCOs reported some safety concerns and higher workload under shared separation conditions due to increased monitoring and perceived lack of timely pilot intent knowledge. Performance data also indicated that ATCOs preferred to resolve conflicts earlier than pilots. The pilot participants, however, preferred shared separation conditions, particularly the condition in which they had the highest level of separation responsibility. Apparently, the perceived flexibility that shared separation provided for the pilots seemed to result in safety concerns and discomfort for the controllers. In a large-scale high fidelity simulation study NASA researchers examined potential capacity gains by mixed IFR and self-separating aircraft (Mogford & Kopardekar, 2004). Five controllers and twenty-two licensed pilots participated in this “Distributed Air-Ground Traffic Management” (DAGTM) simulation experiment. Similar as in the FAA study described above, pilots appreciated their free-manoeuvring ability even though it led to increased workload levels. However, ATCOs reported some safety concerns regarding the self-separating aircraft, because pilots did not always resolve conflicts in a timely manner and it was not always apparent to ATC whether self-separating aircraft would take action and what kind of action that would be. However, according to the simulation data increased system efficiency with maintained levels of safety seems rather likely. Nevertheless, the identified safety concerns need to be addressed in further studies as well as enabling air/ground technologies to support DAG-TM operations. As Hollnagel (2007) puts it, “A transition from managed flight to free flight will change the working conditions for air-traffic controllers as well as for pilots. Since the two groups can be considered both as individual (joint cognitive systems, JCS) and as part of a larger JCS, it is necessary to understand how the change to free


19

flight may change system boundaries as well as system interactions” (p. 415). The author picks up the point of criticism that first it must be ascertained whether the nature of work remains the same as before. “What we need to study is not different work under the same conditions, but rather different work under different conditions” (Hollnagel, 2007, p. 416). The main conclusion Hollnagel (2007) has drawn is that the two conditions of managed and free flight differ considerably regarding the demands of control and therefore regarding the tasks required. How can the new tasks be described, and is there a consequence for ability requirement testing in the recruitment of the future air traffic control and cockpit workforces?

2.2

Transitions between Free Flight Airspace and Managed Airspace

When introducing FFAS, a new problem arises. It seems unlikely that the whole air space will become FFAS. At least the terminal area will remain Managed Airspace (MAS). This will lead to the need for transition procedures between FFAS and MAS, i.e. the transition of the responsibility for the separation with other aircraft from the ATC to the cockpit crew in the moment of changing from MAS to FFAS. The following passage describes a concept for transition procedures elaborated by Beers and Huismann (2002) from the National Aerospace Laboratory (NLR) in Holland. There are two possible transitions: the vertical and the horizontal transition. The vertical transition will be used when FFAS is positioned above MAS. The horizontal transition will be used, when FFAS and MAS are adjacent. Since our experiment uses a horizontal transition procedure, it is worth considering this concept in detail. The MAS is still assumed to consist of more or less fixed airways (cf. Figure 8). Thus, MAS and FFAS are connected by fixed entry and exit points. The nature of these points can differ depending on the route they are connected with: there will be separated entry and exit points if these points are connected with a oneway route. There will be combined entry and exit points if these points are connected with a two-way route. A separation area of 5NM is defined as a border between MAS and FFAS. It is not allowed for any aircraft to enter this separation area. Around the MAS exit point, within FFAS, a circular transition zone with a 50NM radius is defined. This transition zone is introduced to prevent conflicts between aircrafts entering and exiting MAS in close proximity to one another. . To prevent conflicts, vertical separation is assured through flight level assignment by ATC for aircraft either entering or exiting MAS. Thus, in the transition zone the responsibility for separation is divided: ATC has vertical control and the cockpit crew has lateral control. By leaving the 50NM transition zone and entering the FFAS, the separation responsibility shifts completely to the cockpit crew. The operational transition procedure must be considered in two directions: from MAS to FFAS and from FFAS to MAS.

20


Figure 8: Entry and exit point for horizontal transition (from Beers & Huismann, 2002)

MAS to FFAS horizontal transition: The tactical controller (TC) clears the aircraft to pass the MAS exit point and instructs the cockpit crew to expect free flight operation about 10 minutes before the MAS exit point is reached. The cockpit crew then prepares to enter FFAS by switching on their Airborne Separation Assurance System. When passing the MAS exit point, the aircraft enters the transition zone. Within this transition zone no immediate conflict should occur because the traffic in the transition zone, coming from MAS, is still separated. Traffic coming from FFAS is separated from traffic coming from MAS by different altitude levels assigned for MAS exiting and entering traffic. While residing in the transition zone, the crew of the aircraft intending to enter FFAS has to ensure separation from traffic within the transition zone and traffic in FFAS. Only lateral maneuvers are allowed whilst in the transition zone since the levels above and below are reserved for MAS entering traffic. After entering FFAS, both lateral and vertical maneuvers are allowed. FFAS to MAS horizontal transition: Aircraft wanting to enter MAS will contact the planning controller (PC) of the MAS sector about 15-20 minutes before reaching the transition zone. The PC will assign a flight level to be reached before entering the transition zone and a required time of arrival (RTA). Although aircraft are


21

coming from FFAS, they need to negotiate an entry slot with the PC. Since the PC plans incoming traffic in such a way that it will be sequenced and separated when reaching the MAS entry point, the conflicts that could occur in the transition zone should be minimal. While flying in the transition zone, approaching the MAS entry point, the PC will hand over the aircraft to the tactical controller (TC) (closely based on Beers & Huismann, 2002).

2.3

Assistance systems

Controllers and cockpit crews use different assistance systems to support traffic separation tasks. While ATC traditionally control traffic movements with the help of flight plans and real-time radar images, the cockpit crews have only rudimentary information about the surrounding traffic. In the 1990s, the DFS introduced additional Short Term Conflict Alert (STCA) systems to the regional control centers, which can warn the controller of potential conflict situations between two aircraft in the near future. STCA predicts the flight path of each aircraft over a specific time period (usually up to two minutes) and then compares these predictions to see whether separation minima will be maintained. The predictions are based on information of current values of heading, ground speed and vertical speed. When a potential loss of separation is detected, STCA displays an alert to the controller, for example by a flashing red colored aircraft label for the involved aircraft. STCA does not provide controllers with advice on how to resolve a conflict - this decision is always made by the controller him- or herself (Figure 9). Eurocontrol has recently issued a set of minimum requirements for STCA (Eurocontrol, 2007a).

Figure 9: Visual warning of potential traffic conflicts by means of STCA

For more advanced predictions of up to 10 or 20 minutes, Medium Term Conflict Detection (MTCD) systems have been evaluated recently in several field trials by

22


Eurocontrol and other ANSPs (Costelloe, 2003; NATS, 2003). Whereas STCA uses a simple linear projection for single aircraft trajectories, MTCD uses more advanced models in order to create a profile for each flight, taking into consideration cleared flight levels and radar headings. These profiles are compared among all aircraft within the sector and a list of potential conflicts is generated. The controller can examine the results of the MTCD tool by graphical displays such as the Vertical Assistance Window (VAW, see Figure 10) or a graphical time-distance plot. Compared to the STCA, MTCD is not an alerting tool, it is a planning tool which enables controllers to avoid conflicts in an earlier stage.

Figure 10: Example of a MTCD Vertical Assistance Window

In order to permit airborne self-separation tasks by the cockpit crew, the surrounding traffic of an aircraft needs to be displayed. The first development of an Airborne Collision Avoidance System (ACAS) was initiated by the FAA in 1978 (FAA, 2000). If the cockpit crew is fully aware of surrounding traffic, near misses, midair collisions and abrupt avoidance maneuvers can be prevented. In today’s airline operations, this is supported by the Traffic Alert and Collision Avoidance System (TCAS), which outputs either to the Navigation Display (ND) or to the Instantaneous Vertical Speed Indicator (IVSI) on older non-glass cockpits. TCAS is a transponder-based system, which provides traffic warnings and proposes vertical collision avoidance maneuvers in case of dangerous approaches between two aircraft. The surrounding traffic is depicted graphically to the ND as shown in Figure 11.

23


Figure 11: TCAS traffic display on the Navigation Display (right half of the picture)

The surrounding aircraft are displayed as diamonds. An upwards or downwards facing arrow indicates the climb or descent of the respective aircraft. Its relative altitude to the pilot’s own aircraft (ownship) is depicted in feet/100. The ownship is presented as a triangle In case of a dangerous approach, the intruder’s diamond changes color from white to yellow. If a collision avoidance maneuver is necessary, the diamond becomes red and additional resolution advice is provided by visual guidance indicators on the Primary Flight Display (PFD) and by acoustic messages. The TCAS display was designed to supply the cockpit crew with information about potentially conflicting traffic in their proximity and to enhance their awareness. It is, however, not certified for free flight and self-separation scenarios. For this purpose, flight intention information from the surrounding traffic is required, such as heading and speed (Eurocontrol, 2005). Several studies were carried out to evaluate the interaction between TCAS and ATC (e.g., Eurocontrol, 2007b). Operational experience has shown that the vertical displacement resulting from an RA response is often much greater than 300 feet and that TCAS alerts can unexpectedly affect controllers’ behavior. Since there is no downlink of TCAS information to the ATC, the shared mental picture of the traffic situation between cockpit and ground is very limited. In some cases STCA and TCAS can lead to conflicting advice for aircrew and ground crew. This can cause adverse interference with a conflict resolution maneuver. Future broadcasting systems like ADS-B (Automatic Dependent Surveillance Broadcast) will offer better possibilities to assure shared traffic information between flight crews and ground crews. The concept is that aircraft determine their position themselves on the basis of satellite data. This information is distributed via ADS-B to ground stations and from there to ATC and other aircraft. In this way all participants share the same information about the traffic situation. On the ground, the ATC can convert these data streams into dynamic traffic displays that are more precise and up to date compared to traditional

24


radar images. In the cockpit, flight crews will have the same information about the local traffic situation relative to the ownship displayed on a Cockpit Display of Traffic Information (CDTI). Compared to the current TCAS display, the CDTI provides not only information about the current position of the surrounding traffic but also about aircraft flight intentions. This can be done by projections of current heading and speed information, which is usually depicted on the CDTI by a velocity vector attached to the aircraft symbol. Vector length correlates to the aircraft’s speed and vector direction to its heading. Flight crews can for example use the CDTI information to plan and execute tactical avoidance maneuvers (Figure 12).

Figure 12: Cockpit Display of Traffic Information (CDTI) developed by NASA

Research teams, for example at NASA and MITRE, have developed CDTI displays with different features. Some are two-dimensional, some three-dimensional. Some displays have additional weather and/or terrain information included and are sometimes more generally called Cockpit Situation Displays (CSD) (Johnson et al., 1997; Thomas & Wickens, 2006). Depending on how the three spatial dimensions are mapped, four CDTI designs can be distinguished: a. Expanded TCAS display: 2-dimensional display of x- and z-axis. The altitude (y-axis) is shown numerically for every single aircraft. This view is comparable to the radar screen of an air-traffic controller. b. Coplanar 2-D-mapping: Display like a) but with additional vertical display. The vertical display shows the traffic in an x- and y-axis view. c. 3-dimensional display: all 3 dimensions are integrated in a single view by providing a viewpoint that is set to a perspective angle. Two different perspectives can be chosen in order to reduce ambiguities.


25

d. 3-dimensional display: like c) but the perspectives can be chosen continuously. The four different CDTI-displays have been compared using criteria such as: conflict detection, conflict resolution, dimensionality, reaction time, workload, and efficiency of flight maneuvers (Alexander & Wickens, 2001; Thomas & Wickens, 2006; Dao, Battiste, & Granada-Vigil, 2006). In most of the experiments, the coplanar 2-D-display (b) performed best. An overall result of these studies was that pilots did not prefer the 3-dimensional display.

3 Problem Selecting candidates for pilot or controller training requires predicting how the chosen candidates will perform during job training and later career development. To make this prediction, the qualities of the candidates as quantified by psychological selection tests are compared to the relevant psychological requirements of typical job tasks. However, if job tasks change, these predictions may not be accurate. In order to anticipate future job requirements for pilots and controllers the simulation environment AviaSim was developed, which allows the flexible implementation of new modes of operation that may emerge in the future air-transport system. The scenarios used in the simulation AviaSim focus primarily on work processes at the interfaces between different human operators and between human operators and automated systems. These processes involve tasks such as monitoring, communication, task allocation and decision making of the actors in collaboration with members of a distributed team. Humans as well as automatic systems, both in the air and on the ground, are part of this team. It is assumed that during the different flight phases, members of the distributed team can delegate task responsibility to each other according to common operational procedures. This involves processes of negotiation as well as the handover of control. Factors of task complexity will require the use of suitable decision support systems. In the case of airborne self-separation, the controller will hand over responsibility for maintaining separation minima between different aircraft to the pilots. Pilots will need additional technical equipment to observe the traffic in their vicinity and to identify and avoid potential intruders in good time. During descent and arrival phases, traffic separation will be handed back from the cockpit to the ground (Figure 13).

26


Figure 13: Responsibility for separation tasks in distributed air-ground teams

It is assumed that job functions in the future will be more interchangeable than today. Consequently a convergence of pilots’ and air-traffic controllers’ expert ratings of job requirements for the free flight condition compared to the managed flight condition should be observable in the data. It is also assumed that the experimental setup is sensitive to the effects of free flight on pilots’ and controllers’ workload and situation awareness (e.g., Hilburn et al., 1997; Hoeckstra et al., 2000; Endsley et al., 1997). A replication of the effects can be interpreted as an initial indication that the simulation is also a usable platform for future operational concept validations. The main purpose of the experiments is to identify possible future task requirements as well as performance shaping factors. This information can then be used to complement the job profiles of aviators with the required competencies reflecting today’s as well as tomorrow’s work conditions. In the context of this project, it is useful to understand how the implementation of free flight structures will affect controllers’ monitoring behavior. For example, does the hand-over of separation control to pilots cause the controller to spend less time looking at the free-flight area, or more? In order to reconstruct the monitoring behavior the air-traffic controllers’ gazes are measured.

Experimental setup

27

4 Experimental setup Low fidelity flight simulations are widely used in psychological and ergonomic research. Examples are the research on transfer effects between learning in a simulation platform and its application in reality (Atkins, Lansdowne, Pfister, & Provost, 2002), the evaluation of Crew Resource Management Training (Prince & Jentsch, 2001), the comparison of 2-person crews and solo performers (Skitka, Mosier, Burdick, & Rosenblatt, 2000), the evaluation of head-up displays (Ververs & Wickens, 1998), or research on monitoring behavior in the cockpit (Weiss, 2000). Common to these approaches is that topics are addressed for which motion is not important. Typical are evaluations of displays. Using a LAN architecture for simulating an ATM-System is, as far as we are aware, new in this field. An additional application of low fidelity simulation is flight training (Callender, Dornan, Beckman, Craig, & Gosset, 2009).

4.1

Simulation platform

The simulation platform used in this study was designed to meet the requirements of: highest realism with lowest cost, high adaptability, and controllability for experimental purposes. With an open LAN architecture the simulation platform AviaSim (Hoermann, Schulze Kissing, Zierke, & Eißfeldt, 2009) was configured for one controller position and up to eight cockpit positions for pilots. However, one ATC and three cockpit-positions were used for the study. The workstations are PC-based and equipped with the necessary hardware periphery for task performance. The ATC environment, which is based on the off-the-shelf simulator London Control©, provides a Short-Term Conflict Alert (STCA) function, Mid-Term Conflict Detection (MTCD) as well as various flight-plan visualizations, interactive labels, and data link communication. The cockpit environment is basically the Microsoft Flight Simulator© with a B737800/900 layout in combination with a self-developed traffic visualization system (Cockpit Display of Traffic Information, CDTI). Also a transparent area is projected into the cockpit window on which the ATC instructions, transmitted via data link, can be displayed. All workstations are provided with headsets for voice communication. 4.1.1 Flight simulator In order to study the transfer of control in Free Flight Airspace it was important to ensure a highly realistic execution of the en-route flight phase. This means that the autopilot system must have a broad functionality and that the Flight Management System (FMS) must have realistic programming possibilities. For the purpose of self-separation in FFAS the TCAS must be functional in all A/C

28


participating in the experiment and it must be possible to integrate an additional CDTI into the cockpit instrument panel. The process of finding out which low fidelity simulator best meets these criteria is described in Bruder et al.(2009). The final setting consisted of a work station with the following parameters: • Dual core processor AMD Athlon with 2.3 GHz each, • memory 1.75 GB RAM, • hard drive with 80 GB • graphic card GeForce 8200 with 512 MB memory • two adjacent 17’’-monitors. The CDTI was displayed on a special 9’’screen and the CDTI program was operated on an additional Pentium 4 processor with 1.7 GHz and 256 MB working memory. The symbols used for the CDTI are based on the specifications created by Johnson et al. (1997) (for a more detailed description see Hoermann, Schulze Kissing, & Zierke, 2009). Peripherals included a keyboard, a mouse, and a joystick. The work stations were equipped with Windows XP as operating system and Microsoft Flight Simulator 2004 as flight simulation base program. Windows XP and the Flight Simulator 2004 were preferred to Windows Vista and Microsoft Flight Simulator X because they save a lot of working memory. For all experiments the add-on “Boeing 737-800/900” from PMDG was used. This addon has much more system depth than A/C models from the Flight Simulator Base package. First of all, it allows programming of the FMS via MCDU (multifunctional control and display unit) and preparing the A/C for the en-route flight. To get an impression of which level of realism was aimed for, the standardized setting of the simulated B737 is described now. Setting of the B737 simulation program • Enter call sign. • Enter fuel and cargo. The centre tank was set to be empty. • Flight Director and Auto Thrust “On”. This setting is usual for normal autopilot operation. • Activate TCAS at the radio unit in TA/RA mode and press the TFC button at the range knob. That means that TCAS will give traffic and resolution advisory. • Enter origin and destination airport into FMS and the complete route consisting of about 10 waypoints. These waypoints are artificial waypoints that were included in the B737 FMS. • Enter weights, reserves, cruising altitude, target speed, and cost index. • Activate the autopilot, activate LNAV (lateral navigation) and VNAV (vertical navigation). • The system time is distributed centrally.

Experimental setup

29

4.1.2 ATC simulator London Control© (DM Aviation Limited) is a purchasable air traffic control simulation which represents current air traffic operations. Its Add-On Germany Radar© (AviaScan) provides airspaces that represent airways, terminal areas and control zones in Germany, with flight plans created from real flight data. The airspace sectors and their operational procedures are the same as those used by the Deutsche Flugsicherung (DFS). The simulator includes assistance systems, of which some are currently in operation (e.g., Maastricht Center). The radar data processing simulates the real radar data feed to air traffic control centers and includes primary and secondary radar sources. The simulation system allows the user to input new flight plans. The system is fully adaptable, so that even generic sector structures can be generated. It is also possible to customize a vast array of options which alter the appearance and behavior of the system. 4.1.3 Integrated simulator environment The integrated simulation platform AviaSim allows for the investigation of processes of tactical decision making, task allocation, attention, monitoring, and information management of human actors working collaboratively in a distributed team environment. With open local area network architecture, AviaSim is currently configurable for up to nine aviator workplaces: one for an air-traffic controller and eight for pilots. Additional to the piloted aircraft, “experimental traffic” can be generated with pre-determined flight plans per experimental script files. As described in the previous two sections, each workplace has the standard equipment with additional automatic assistance functionality to support tactical decision making, continuous task monitoring and communication. Figure 14 displays a configuration with traffic information displays and collision warning functionality. Communication processes are facilitated through Voice over Internet Protocol (VOIP) and advanced by data link channels. This current configuration serves primarily the simulation of en-route scenarios. However, with different support systems such as airport moving maps or arrival/departure managers, traffic situations on ground or during departure and arrival can be simulated with AviaSim. The type of aircraft also permits the introduction of military traffic and uninhabited aerial systems (UAS).

30


Figure 14: AviaSim simulation platform with a networked configuration of eight flight simulators and one air traffic control simulator (Hoermann, Schulze-Kissing, Zierke, 2009).

AviaSim has the following features: • Freely configurable airspace • Traffic scenario defined via scripts • Automatic exchange of data about flight plans, flight data, weather • Air-ground voice and data link communication as well as air-to-air voice communication • ATC screen can display configurable extended aircraft labels which include the control status of each aircraft (“Ground”, “Air”, “Autonomous”) • Masterclock for data synchronization • Time- and event-based data logging • Decision support systems (CDTI, STCA, MTCD) Under development: Multi-sector and multi-pilot configuration as well as further decision support functionality.

Experimental setup

4.2

31

Airspace structure

With AviaSim an enroute sector is specified. The geometry of set waypoints results in a symmetric route structure (see Figure 15) across the x- and y-axis.

Figure 15: Airspace structure used in the future scenario (screenshot with colors reversed)

The airspace structures used for the baseline and the (free flight) future scenarios differ. The airspace structure used to run the future scenario shows a rectangular structure, signifying the free flight zone within the controlled sector. At each corner of the free flight zone a rhombus is cut off, representing the transition zones for inbound and outbound flights. Within the free flight zone, no waypoints and route-structures are defined. However, the baseline airspace has no additional zone within the sector boundaries (cf Figure 17). Instead, there is an additional waypoint in the sectors’ center where the two routes from the corners cross.


32

4.3

Scenarios

Traffic samples are constructed in a way that forces the operators into situations of mutual merging and spacing in the transition zone, as well as self-separation with crossing traffic in the middle of the sector. The traffic for the two experimental conditions (managed flight without CDTI and free flight with CDTI) is made comparable by changing call signs and the geometric relations without changing the spatiotemporal relations between aircraft.

Figure 16: Future scenario (screenshot at first point of measurement), five points of measurement in relation to piloted aircraft position (in sequence from left to right)

The following aircraft related details are specified in the script: call signs, type, current speed, flight level, starting waypoint, and planned route. 36 aircraft are specified for each scenario. Four aircraft are provided with network identifiers for connection with the flight simulators. Twelve (system generated) aircraft are crossing the sector via marginal routes without traversing the free flight zone. 24 aircraft enter the free flight zone in two waves. In each wave, three aircraft merge from each sector corner (three aircraft approaching from four directions in two waves = 24 aircraft). The merging waypoints of the aircraft triplets are situated in the four transition zones of the free flight sector. After merging, all triplets cross at the sector’s central waypoint and fan out afterwards. The piloted aircrafts are part of the first wave. The three of them that are controlled by the experimental subjects form one triplet. The piloted aircraft controlled by the

Experimental setup

33

experimental assistant approaches the central point from a different direction, instructed to produce crossing traffic.

Figure 17: Baseline scenario (screenshot at first point of measurement), five points of measurement in relation to piloted aircraft position (in sequence from right to left).

Figure 16 gives an example of the standardized traffic flows within the free flight condition; Figure 17 within the baseline condition. Purple crosses and arrows are used as markers to denote the position and trajectory of the three piloted aircraft (pilots) at the first system freeze (first point of measurement). The turquoise cross denotes the position of the aircraft piloted by an experimental assistant. The orange circles denote the situation (or region) of interest at the first, second, and third system freeze (from outer to inner). The first situation of interest is the controllers’ separation of one of the piloted aircraft from the crossing traffic on the marginal routes. The second is the point of merging of the three piloted aircraft (self-separated or controlled). The third is the (self-) separation of the piloted aircraft from the crossing flows of unpiloted (i.e., system generated) aircraft. At these moments, instant measurement of workload and situation awareness of pilots and controllers is taken.

34


Concept of horizontal transitions The general concept for the handling of transitions between Managed Airspace and Free Flight Airspace is described in Chapter 2.2. It is based on a concept of NLR (Beers & Huismann, 2002). In this chapter the most important features are outlined and differences to the NLR concept will be highlighted. The future scenario has four entry and exit points in the corners of the inner square of Figure 15: Pudlo, Quasi, Rambl, and Opaca. Their transition zones (TZ) have the form of a quarter circle instead of a half circle (cf. Figure 18). After passing the entry point, the TZ covers a radius of 30NM instead of 50NM (as in Figure 8). The lowest entry level into the transition zone is FL 270. In order to enter the TZ the aircraft (A/C) needs clearance. Flying within the TZ, the cockpit crew is responsible for the horizontal separation, while ATC is responsible for the vertical separation. That means that a clearance for the TZ always must include a cleared flight level. In contrast to the NLR concept, in this experimental scenario we did not differentiate between planner controller and tactical controller. Here, one controller is responsible for both tasks.

Figure 18: Abstract airspace structure

Operational concept In the following section the two scenarios, MAS to FFAS and FFAS to MAS, are described.

Experimental setup

35

Scenario 1: MAS to FFAS Pilot: contact ATC 5 min before TZ to request TZ entry; ATCO: clear A/C for entering TZ and instruct pilot to expect free flight operation at MAS exit point; Pilot: affirm with ATC and activate CDTI if not yet activated; ATCO: assign FL to be maintained in TZ; Pilot: affirm with ATC ATCO: hand over separation responsibility to pilot at MAS exit point; Pilot: accept separation responsibility and continue flight via the TZ into FFAS. Scenario 2: FFAS to MAS Pilot: contact ATC 5 min before TZ to request MAS entry; ATCO: assign FL and RTA at MAS entry point; Pilot: affirm FL and RTA Pilot: monitor traffic in TZ and solve conflicts with other inbound traffic as required; ATCO: take over separation responsibility from the pilot when passing the MAS entry point; Pilot: affirm change of responsibility. Rules for Free Flight Airspace In order to prevent ambiguous traffic situations the following rules were set (from Ruigrok, de Gelder, & Scholte, 2005): • Level A/C has priority over climbing and descending A/C. • Descending A/C has priority over climbing A/C. • If two A/C are level, A/C obeying the Flight Level Orientation Scheme (FLOS) has priority over A/C not obeying FLOS. • If two A/C are in the same flight phase, the overtaking A/C should give way. Overtaking is defined as an A/C on an aspect angle from 150 to 210 degrees of ownship. • If two A/C are in the same flight phase, the A/C approaching from the right has priority. From the right is defined as an A/C on an aspect angle from 0 to 150 degrees of ownship. • If none of the above applies, the A/C call sign will be used to determine priority. 4.4

Experimental design and procedure

A one-factorial complete repeated measurement design is used in the experiment. The independent variable is the control authority for the flights within the sector, with the two levels:

36


• Level A (Baseline): ‘MAS & no CDTI’: Traffic managed entirely by ATC (one run per trial) • Level B (Future): ‘FFAS & CDTI Active’: Transition of control authority and airborne self-separation (two runs per trial) Level A, with no airborne self-separation, is similar to conditions in the present ATM. This level therefore serves as the baseline scenario. Level B includes a number of “future elements” such as the CDTI, transfer of control authority, data link and airborne self-separation. Level B conditions are also referred to as “future scenario”. The simulation experiment was conducted at the DLR human factors laboratory in Hamburg. The subjects were examined in five groups, each consisting of one controller and three pilots. Each group was tested on a separate occasion. During the simulation runs participants were seated in separate rooms. One confederate experimenter controlled an additional aircraft in order to enrich the scenarios with a certain number of difficult situations. Each subject group participated for two days in the experiment. Advance information material about the aims of the experiment, the simulation system, and the task setting had been sent to brief the subjects prior to the experiment. Day 1 served as a familiarization day. It started with an hour long briefing and a comprehensive rehearsal of the advance information followed by some hands-on training on the simulators. All questionnaires and data gathering methods were explained. Finally, the group of subjects participated jointly in a one hour training run. Controllers were instructed to communicate via data link with the experimental traffic. The piloted aircraft could be reached by either voice or data link messages. Pilots were instructed to expect data link instructions from ATC, which they had to confirm using the voice channel. For air-to-air communications under the self-separation conditions as well as for all replies and requests, pilots were to use voice communication. During the debriefing session of day 1 further discussion of open questions and experiences took place. Day 2 started with an introduction of the concept of transition zones and the required procedures for transitions from MAS to FFAS and vice versa. Then three en-route scenarios of about 45 minutes in duration were exercised jointly. The sequence of experimental conditions (level A and level B) was rotated from group to group in order to reduce training effects. The two (free flight) future scenarios were identical. However, pilots swapped flight plans and call-signs between the runs in order to experience different airspace situations. The task in each of the scenarios was to manage the traffic and operate the aircraft safely and efficiently. Compliance with the specific rules and procedures provided for the transfer of control authority and for the separation of aircraft was a requirement. Conceptually, each scenario could be divided into five phases as shown in Table 4.

37

Experimental setup Table 4: Scenario phases

Level A Baseline

Level B Future

Phase 1

Entry to MAS

Entry to MAS

Phase 2

Merging traffic

Merging traffic transit into FFAS

Phase 3

Crossing traffic

Crossing traffic

Phase 4

Fanning out

Fanning out and transit into MAS

Phase 5

Exit

Exit

and

At the beginning of each scenario the piloted aircraft as well as the ‘synthetic’ aircraft were positioned airborne outside the sector boundaries. In the course of the scenario the sector filled up with 24 aircraft heading into different directions (northeast, northwest, southeast and southwest). After each phase the scenario was frozen in order to allow subjects to fill in two instantaneous self-ratings of workload and situation awareness (ISA, see below). Subsequent to each scenario, subjects completed the questionnaires as described in the following sections (NASA TLX, SART and F-JAS). At the end of day 2, an additional questionnaire was handed out to collect feedback on the simulation environment as well as scenarios. In a final one hour debriefing session further experiences, ideas and points of criticisms were collected in a workshop-like manner.

5 Method A mixed approach is used. Before rating the task requirements on a job analysis scale, experts of both domains (air traffic control and airline pilots) jointly worked on future scenarios (free flight and managed flight) that were presented on a simulation platform. It is assumed that expert ratings comprising the previous experience of standardized scenarios have higher reliability and validity compared to a mere questioning of aviation professionals without simulation exercises, where general attitudes towards certain concepts could have a stronger effect on the outcome.


38

5.1

Subjects

20 male operators participated in the study, five of whom are center controllers of the Deutsche Flugsicherung (DFS) with an average of 30 years work experience, and 15 licensed Lufthansa pilots with an average experience of 1394 flight hours. The mean age of the subjects was 31.9 years.

5.2

Objective data/measurements

Differences in work and working conditions between Level A (managed flights) and Level B (free flights) for controllers and pilots are assessed by using a number of different data sources including objective simulation data (5.2.1), Eye-Point-ofGaze Tracking (5.2.2), Ophthalmology Data (5.2.3), the Fleishman Job Analysis Survey (F-JAS; Fleishman, 1992) and some other questionnaires (5.4) measuring workload and situation awareness. Simulation log files are analyzed for the number and point in time of controllers’ actions for aircraft separation. System safety (or system performance) is measured by the total number of losses of separation, as well as by the total number of STCAs. 5.2.1 Performance data Data log files generally collect two types of data. The first type consists of the details of the air traffic scenario that is presented on each trial, including the type, timings, and durations of aircraft events. Participants’ actions are the second source of data. These actions include the ATCOs’ timing of interventions with aircraft trajectories and the timing of aircraft acceptances and handoffs. However, as the interaction with the pilots is essential in the current studies, data logged concerning these system generated aircraft are excluded from the analysis. System safety (or system performance) is measured by the first type of data. Specifically, the total number of losses of separation, separation regains, and STCA are analyzed. A 5-nm lateral and 1000ft vertical distance is used as the separation standard. The predictive horizon of the STCA function is set to two minutes. Simulation log files are analyzed for the second type of data, namely the number and point in time of the ATCOs’ instructions for traffic separation. The actions are subdivided into flight level clearances, direct routings, and heading instructions. 5.2.2 Gaze tracking The role of the human component in complex man-machine systems (MMS) has undergone some significant transformation in the last few decades: with the development of ever increasing computing power the operator has been more

Method

39

and more removed from the centre of action to become a mere observer whose interaction in the system processes becomes ever rarer. While it was possible to assess the quality of human-system interaction by measuring the physical processes connected to these actions, the mere process of observing does not leave any physical imprint to be assessed, with one exception: observing involves directing one's eyes to those elements of a process which are important for the dynamics and control of a system. This could mean looking at certain instruments or specific parts of a display or computer screen, or simply looking out of the window at the right moment. Using eye-tracking in the monitoring of human participation in system control, delivers some metrics for the quality with which system processes are followed. When human interaction will follow necessarily, its precision can then be correlated to preceding monitoring. Besides these immediately system correlated actions of the human vision system, eye observation can also give information on physiological processes, e.g. via blinks or pupil diameter. Eye-tracking technology has been enabled by the same factor that changed the human role in MMS, i.e. the development of (micro-)computers. In particular the rise in digital video technology for everyday use has created an environment in which the software for pattern recognition and image processing necessary for the automatic computation of eye-tracking parameters has become possible - for reasonable prices. In order to reconstruct the monitoring behavior, the air-traffic controllers’ gazes are measured with a Smart Eye Pro 5.0 system. Head positions and gaze directions are tracked in 3D. Reflections of IR flashes on the cornea are also used to determine gaze positions. The system’s cameras are placed at the bottom right and left side of the screen that displays the simulated radar-representation. Cameras are calibrated and a ‘world coordinate’ system is established. A personal profile for each air-traffic controller is created. In a final step before starting an experimental run, gazes are calibrated. A central time stamp is used to link gaze tracking data with the logfile-data of the simulation environment. A scene camera is used to generate a video image of what the air-traffic controller is seeing. A marker is superimposed on the video image at the position the subject is watching. This video is recorded for later analysis. The scene camera is run on a separate computer. 5.2.3 Ophthalmology Dynamic visual acuity is the ability of an observer to detect details of an object when either the object and/or the person is moving. It is an important topic in sports and transport medicine. Available data in transport medicine relate to vehicle driving and accidents.

40


Dynamic visual acuity is independent from static visual acuity. Dynamic vision serves mainly for the controlling of action. Besides perception, it also requires active exploration and the tracking of moving objects through eye, head and body movements. Measuring dynamic visual acuity is difficult because there is no standardized, efficient and flexible apparatus for its assessment (Smither & Kennedy, 2010). There is the possibility to measure the dynamic visual acuity by projecting moving targets from one direction to the other through improving the angular velocity with a fixed head (Ludvigh & Miller, 1953). Another possibility is to move a random dot Landolt ring on a computer screen (Schrauf, Wist, & Ehrenstein, 1999). The Landolt ring is briefly presented as a form-of-motion stimulus. Motion contrast between the ring and background is varied in terms of the percentage of dots moving coherently within the ring. The visual system is one of the most important senses. 80% of surroundings are perceived via the visual system. The European medical requirements for pilots and air-traffic controllers have changed during recent years. Despite an increased workload on the visual system, the medical requirements on vision have decreased. The question arises whether the medical requirements reflect the real demands on workplace cockpit and air traffic control. If gaze tracking is used to monitor gazes in simulations, it is helpful to have information about the function of the eyes. In addition to that it is important to know the visual acuity of persons involved in screen simulations, therefore the test subjects were examined by an ophthalmologist. The visual acuity, phoria and stereopsis were tested by a Titmus vision screener. The visual screen for the simulations is about 60 cm away from the test subject’s eyes. He therefore needs to accommodate in order to get a sharp image on the retina. To get information about the accommodation range and possible pathologies, the near point was identified. The direct and indirect pupil reaction to light was tested because both the lens and the pupil take part in accommodation. Accommodation is the process by which the eye changes its refractive power. The lens is soft and malleable. A ring of muscle around the lens, the ciliary body, can change the shape of the lens. This allows the eye to focus at different distances. With increasing age accommodation becomes more difficult and takes more time. The size and the reaction of the pupil can be influenced by light, medication, trauma or diseases. The sympathicus can activate the musculus dilatator pupillae with a subsequent mydriasis. In the case of mental overload the musculus dilatator pupillae can slack and the pupil narrows. The visual acuity was tested by Landolt rings (DIN 58220 and ISO 8596). The ring has a stroke width and a gap measuring 1/5 of the outer diameter and is shown in eight different directions (up, down, left, right, up right, up left, down right, down left). The visual acuity was tested in the far distance, at 60 centimeters (which was the intermediate distance to the screen) and at 30 centimeters. It was tested without correction or with the correction that was used during the

Method

41

simulation, if necessary. The vertical and horizontal phorias, as well as the stereopsis were also examined with correction for the intermediate distance where necessary. A phoria is a latent deviation. This means that the deviation is not apparent unless fusion is broken. The higher the phoria, the higher the probability of suffering from decompensation into a tropia which means strabism. The stereopsis test works after the polarizing system. The smallest detectable resolution in this test is 40 seconds of arc. In addition to that the dynamic visual acuity was checked. Dynamic visual acuity is concerned with moving targets, in contrast to static visual acuity, which is concerned with non-moving targets. For this test the subject sat in front of a computer screen at a distance of 60 cm. A Landolt ring was briefly presented as a form-of-motion stimulus as a moving random dot pattern. Motion contrast between the ring and background was varied in terms of the percentage of dots moving coherently within the ring. Through this method a kinetic figure arises comparable to an unmasking through motion. The task was to name the direction of the opening of the circle. There were four different levels with 100, 50, 30, 20% pixel density. In each level 20 circles were offered. The percentage of pixel density was reduced from level to level. The result was given by the computer program as the percentage of correct answers. This method is similar to the tasks in the simulation because the targets also move on the screen.

5.3

Ability requirements

In order to assess ability requirements for the baseline and future scenarios, the Fleishman Job Analysis Survey (F-JAS, Fleishman & Reilly, 1995) was chosen. The F-JAS is a questionnaire system for describing jobs and tasks in terms of the abilities, skills, and knowledge required. The questionnaire is used to identify characteristics of jobs and tasks that are related to the abilities people need to perform these jobs and tasks. Its usefulness has been demonstrated in various studies (Fleishman & Mumford, 1991). The F-JAS has been shown to have high reliability and internal and external validity for a number of applications in the occupational world. It has also proved its value for the job analysis of pilots and controllers (Goeters & Schwab, 1997; Deuchert & Eißfeldt, 1998; Maschke, Goeters, & Klamm, 1998). The ability requirements taxonomy is intended to reflect a comprehensive set of categories that describes performance in the widest variety of tasks. It includes 72 abilities grouped into the cognitive, psychomotor, physical, and sensoryperceptual domains. According to Fleishman and his colleagues, omitting scales that are not assumed to be relevant to a specific task is explicitly allowed. To allow repeated measurement the version used in this experiment was shortened to include only about half of the original scales. Psychomotor abilities as well as

42


some physical and social/interactive abilities were dropped. This led to a shortened version of 36 instead of 72 ability scales. In order to meet special requirements of the free flight phase, four new scales were added ( Appendix 13.1): Vigilance, Impulse Control, Trust in Humans, and Trust in Machines. This resulted in an experimental version including 40 ability scales (see Table 5). Vigilance addresses the ability to maintain alertness for a long period of time although acting is required only very rarely. Impulse Control means the ability to wait with the execution of an action until the optimal moment. Trust in Humans means the ability to rely on the decisions of other persons. Trust in Machines addresses the ability to rely on a machine or on an assistance system to generate adequate suggestions. Every single ability scale has to be rated by the subjects on a 1 to 7 scale. The number 7 at the top of the scale is the highest level of ability. The number 1 at the bottom of the scale indicates that the scenario requires only a very low or minimum level of ability. The rating reflects the degree to which the ability is required to perform the scenario successfully. For every scale examples for different degrees of affirmation are given. Examples for the description of the FJAS-scales are given in Appendix 13.1. Table 5: F-JAS ability scales used with AviaSim plus four additional scales

1 Oral Comprehension 2 Written Comprehension 3 Oral Expression 4 Written Expression 5 Fluency of Ideas 6 Originality 7 Memorization 8 Problem Sensitivity 9 Mathematical Reasoning 10 Number Facility 11 Deductive Reasoning 12 Inductive Reasoning 13 Information Ordering 14 Category Flexibility 15 Speed of Closure 16 Flexibility of Closure 17 Spatial Orientation 18 Visualization 19 Perceptual Speed 20 Selective Attention

21 Time Sharing 22 Response Orientation 23 Reaction time 24 Finger Dexterity 25 Wrist-Finger Speed 26 Visual Color Discrimination 27 Hearing Sensitivity 28 Auditory Attention 29 Speech Recognition 30 Speech Clarity 31 Resilience 32 Stress Resistance 33 Behavior Flexibility 34 Situation Awareness 35 Resistance to Premature Judgement 36 Decision Making 37 Vigilance 38 Impulse Control 39 Trust in Humans 40 Trust in Machines

Method

5.4

43

Questionnaires

Each scenario run was evaluated by the subjects with respect to workload (WL) and situation awareness (SA). The NASA Task Load Index (NASA TLX; Hart & Staveland, 1988) and the Situation Awareness Rating Technique SART (SART; Taylor, 1990) were administered after each run. Two Instantaneous Self Assessment scales (ISA) were used to assess current levels of workload and situation awareness when the scenario run was frozen after phases 1 to 5 (see Table 4). ISA uses three-point color coded rating scales. The ISA ratings for WL and SA were averaged into total scores for each scenario and also used as distinct scores to mark each of the five scenario phases. ISA scores can vary between 1 (lowest) and 3 (highest). NASA-TLX is a method to identify factors associated with variations in subjective workload. With respect to the tasks just accomplished, subjects assess their perceived levels of demand on six scales reflecting different sources of workload. In a second step, the relative importance of the six factors as contributors to individual workload in the task is evaluated by using 15 pair comparisons. • Mental Demand (MD): Extent to which a task requires mental and perceptual activity • Physical Demand (PD): Extent to which a task requires physical activity • Temporal Demand (TD): Time pressure due to rate or pace at which the task or task elements occurred • Performance (OP): Degree of perceived success in achieving the goals of the task • Effort (EF): How hard the subject has to work to accomplish his/her level of performance • Frustration (FR): Degree to which insecurity, discouragement, irritation, stress or annoyance is perceived during the task The overall workload score for the individual subject is calculated as the weighted sum of the products of value and importance for each of the six factors. The resulting overall score for the NASA-TLX is transformed so that values can vary between 5 (lowest) and 100 (highest). SART provides a validated and practical subjective rating tool for the measurement of SA, which is based on information gathered from experienced aircrew members (Taylor, 1990). Ten generic dimensions were identified via interviews and other knowledge elicitation methods with aircrew members. The original version of SART is based on subjective ratings on these dimensions. The structure of the SA dimensions has been interpreted as comprising three related domains, which form the principal scale of SART, namely:


44

• Demand for Attentional Resources (D): complexity, variability, instability • Supply of Attentional Resources (S): arousal, concentration, division of attention, spare mental capacity • Understanding of the Situation (U): information quality, information quantity, familiarity The SART version used here was taken from research done for a SA training project called ESSAI (Hoermann et al., 2003). In this version four further items were added, one for each of the three SART principal scales and one additional control item. Instead of numeric Likert-scales a graphical representation of the rating scales was applied, where the length of line from the left hand side of the scale to the participant’s mark (in millimeters) represents a respective rating score for one item. The possible range is between 0 (“lowest”) and 50 (“highest”). Items 1, 2, 3 and 4 are averaged to give a score for Attentional Demand Items 5, 6, 7, 8 and 9 are averaged to give a score for Attentional Supply Items 10, 11, 12 and 13 are averaged to give a score for Understanding An overall SA score is not part of the original SART but a method was proposed later by Crabtree, Marcelo, McCoy, and Vidulich (1993), which we have also adopted here. Overall situation awareness is calculated by the formula Overall SART = U - (D - S) The resulting values are transformed to a scale varying from 0 (lowest) to 100 (highest). Question 14 gives the participant’s confidence in their ratings of the above. This item has not been included in the analyses reported here.

5.5

Debriefing

The F-JAS presented in Chapter 5.3 and the questionnaires presented in Chapter 5.4 are highly structured methods where subjects have to express their thoughts and ratings in numbers only. In order to offer subjects the opportunity to express their thoughts and assessments more freely and interactively the experiments were completed by a debriefing. Participants were the air-traffic controller and the three pilots who participated in the experiment, and the three test conductors, one of whom participated in the experiment as a pilot as well. In order to launch a discussion six central questions were introduced by the moderators. These questions were:

Method

45

• Can you imagine that future free flight may look like the experimental scenarios 2 and 3? • (For pilots) Can you imagine working with the CDTI we just used? • (For controllers) Can you imagine working with the modified radar display we just used? • Which ability requirements will change for pilots and controllers, when the future concept of Free Flight Airspace comes true? What will become less important, what will stay just as important? • Do you assess response proactivity, attention and readiness to act (see the new ability scales we added to the F-JAS, Chapter 5.3) as relevant new factors for working in a FFAS? • In which situations did you experience more workload, in which less workload? After every question a free discussion was possible. In contrast to the questionnaires, the questions were directed to the group of participants rather than to each individual participant and an interactive discussion was expected. The moderator decided when to go on to the next question.

6 Results Data collection was completed in August 2009. Results which are reported here are primarily those of the main effects for the experimental variable „Control Authority by ATC“ (Level A) versus „Control Authority Transferred for Free Flight“ (Level B). The comparison of the two conditions reveals indications for differences in the requirements between the baseline and the future scenarios. Except for the analyses in section 6.1 and 6.6 the scores for future scenario one and future scenario two have been combined by averaging. This should make interpretation clearer and also reduce the effects of training from one scenario to the other. T-Tests for paired samples as well as Chi-square and Analyses of Variance (using SPSS General Linear Model procedure GLM) were used to compare the results. α = 10% with two-tailed testing was accepted as the level of statistical significance because generally the effect size can be considered as substantial with scores of mostly 0.40 and higher. First, results of objective simulation data are reported. This data source provides a view of how work for ATCOs and pilots may change by the introduction of self separation procedures.


46

6.1

Objective data

Controller Performance: As expected, there is a significant overall difference in the number of flight level clearances made by the air-traffic controller and the three pilots (χ2(2, N = 100) = 7.58; p< .10). In the first run of the future scenario, controllers give less instructions for traffic separation (n= 21) compared to the managed flight (status quo) condition (n= 43). However, in the second run the controllers increase the number of instructions for traffic separation (n= 36), so that when measurement is repeated the difference between the free flight and managed flight conditions in terms of the frequency of traffic separation instructions can no longer be observed. The first separation instructions after scenario start are given earlier in the second run of the future scenario (M= 11.86 min., SD= 6.49) than in the first run (M= 19.90 min., SD= 12.51). A test of within-subject contrasts revealed a marginal effect (F(1, 11) = 3.33, p< .10) of the repeated measure on the time controllers made their first separation instructions of small size (ε2 = .25). According to their own statements, the controllers were unaware that the same traffic sample had been used for the second as for the first future scenario. In total, 67 % of the separation instructions were flight level instructions, 30 % direct routings and just 3 % heading instructions. A Chisquare test of differences in the frequency of separation instructions per aircraft yielded no significance. Safety produced by the joint cognitive system (JCS): In the experimental trials a total number of N = 15 losses of separation, N = 15 separation regains, and N= 56 STCA are measured. System safety significantly differs between the three experimental conditions (χ2(2, N = 86) = 12.86; p< .10). This is indicated by a significant difference in the number of STCA (χ2(2, N = 56) = 6.46; p< .10), as well as by a trend to different frequencies of losses of separation (χ2(2, N = 15) = 4.80; p< .10). Compared to the managed flight condition there is a trend for more frequent losses of separation (n= 9) in the first future scenario (χ2(1, N = 12) = 3.00; p< .10), as well as a significantly lower number of STCA (n= 10) during the second run of the free flight condition (χ2(1, N = 31) = 3.90; p< .10). The number of losses of separation in the future scenario decreases from the first (n= 9) to the second run (n= 3) (χ2(1, N = 12) = 3.00; p< .10).

6.2

Eye tracking data

The video recordings of the ATCOs’ eye gazes were analyzed with a Videograph© multi-media player for video coding (see Figure 19). However, the eye-gaze measurements were insufficient to reliably test for the experimental hypotheses. In the context of our experimental setup, in which the superimposed gaze cursor has to be assigned to small targets in a high density traffic context on one radar screen (21’’), the eye-tracking system provided too low resolution and the errors in measurement were therefore too high.

Results

47

©

Figure 19: Screenshot of the Videograph interface with loaded traffic scene (left) and code system and timeline of codings (right)

The underlying reasons for measurement impreciseness during data collection remained unclear, even though SmartEye provided onsite support. The eyetracking system may prove its true power in follow-up experiments when the switching of gazes (i.e., visual attention) between different screens (e.g., radar, flight-strips, and assistance system) will presumably be the main requirement for hypothesis testing.


48 30

20

Frequency

10

Objects of fixation Aircraft Waypoint

0 30

60

90

120 150 180

Time windows (seconds)

Figure 20: Frequency of one ATCOs allocation of visual attention on fixed and dynamic objects reflecting initial planning activities during the first three minutes of a scenario.

To give an impression of what kind of data can be expected by executing this method of gaze-data analysis, the results of an exemplary coding of one ATCOs allocation of visual attention during the initial planning phase (low density traffic context) of one scenario is shown in Figure 20.

6.3

Ophthalmic data

18 out of the 20 test subjects had a valid aviation medical certificate needed for holding a licence. From the 15 pilots, two had a valid medical certificate for class 2. There were nine persons who wore glasses. 14 persons had an uncorrected or corrected visual acuity of 1.0 for both eyes at a distance of 60 cm. For six persons at least one eye had a reduced visual acuity at a distance of 60 cm. The visual acuity is specified in the following overview.

Results

49

Table 6: List of visual acuity measurements for the sample of n=6

Intermediate visual acuity right eye Intermediate visual acuity left eye 0,7 1,0 0,1 0,1 0,8 0,8 1,0 0,7 0,5 0,8 1,0 0,4 Apart from the first test subject in the table, all others were older than 38 years of age. In two out of the 20 males, convergence was not good due to strabism and pseudophacia. In four persons stereopsis could not be demonstrated; one person had only a reduced stereopsis. In the test of the dynamic visual acuity eight persons recognized 100% correctly in all four runs. Four test subjects did not recognize 100% in the third run (30% pixel density). They recognized 95, 95, 95 and 80%. In the last run (20% pixel density) they recognized only 55, 60, 80 and 55 %. All four persons had no demonstrated stereopsis.

6.4

Ability requirements

After completion of each scenario, ATCOs and pilots assessed the perceived ability requirements for the scenario tasks on the forty F-JAS Scales described in section 5.3. The mean ratings were compared via t-tests with paired samples for ATCOs and pilots separately. The significant results are summarized in Table 7, whilst Appendices 13.2 and 13.3 contain illustrations of the entire profiles for ATCOs and pilots. Comparing F-JAS scales showing significant within-group effects for baseline and future scenarios, the ATCOs perceived lower ability requirements for the future scenario with three scales Oral Comprehension, Originality, and Speed of Closure and a slight increase for Written Expression. For the pilots eight of the forty scales showed a significant increase in ability requirements. Some of these scales (e.g. Situation Awareness) have been identified as a job requirement for pilots in earlier studies (Maschke, Goeters & Klamm, 1998). However, generally the pilots seem to perceive increased requirements on cognitive ability scales, such as Information Ordering, Inductive Reasoning, Selective Attention, Situation Awareness, Flexibility of Closure, and Impulse Control under Level B (future scenario) conditions. Two further scales related to divergent thinking (Fluency of Ideas and Originality), also significantly increase in importance; however they do not yet reach the threshold of 4.0 and are thus not to be viewed as critical for the job.


50

Table 7: Mean ratings of F-JAS Scales for the baseline and future scenarios. Only those ratings with a significant within-group effect are listed F-JAS Scales

ATC

Pilots

Baseline

Future

p

Baseline

Future

p

4.8

4.1

0.08

3.0

3.8

0.04

3.2

3.9

0.10

4.6

4.0

0.07

2.5

3.8

0.00

3.2

4.0

0.01

4.0

4.6

0.01

6.2

5.1

0.10

Flexibility of Closure

3.5

4.4

0.01

Selective Attention

3.9

4.5

0.01

Situation Awareness

5.0

5.6

0.01

Impulse Control

3.4

4.1

0.03

Oral Comprehension Written Expression Fluency of Ideas Originality Inductive Reasoning Information Ordering Speed of Closure

In summary, working with the CDTI for self-separation tasks seems from the point of view of the pilot to require a higher number of abilities than work in the baseline scenario. As tasks are being shifted from ATC to cockpit crews, the respective requirements seem to shift from ground to air. In the between-groups analysis, we compared the similarity of the ability requirements of ATCOs and pilots for the baseline and future scenarios. This comparison was conducted by calculating the differences between ATCOs’ and pilots’ ratings on the same scale for the two experimental conditions. A positive difference in score means that the ATCOs perceived higher requirements than the pilots on the respective scale. A negative score means that the pilots perceived higher requirements. From Figure 21 we can draw two conclusions. First, in both scenarios the ATCOs perceived higher ability requirements to accomplish their tasks than the pilots did (most blue bars are longer than yellow bars). This changes only for the F-JAS scales Spatial Orientation and Visual Color Discrimination, where pilots score slightly higher in the future scenarios than the ATCOs (yellow bars point in negative direction). The second finding is that the similarity of the task profiles is obviously increasing. While the average difference between ATCOs’ and pilots’ ratings for the baseline scenario is 1.4, it is only 0.9 for the future scenario. The t-test between the two difference vectors is

Results

51

significant (α