validating a new task distribution between air traffic controllers and

VALIDATING A NEW TASK DISTRIBUTION BETWEEN AIR TRAFFIC CONTROLLERS AND FLIGHT CREWS Laurence Rognin*, Nathalie de-Beler♦, Isabelle Grimaud•, Eric Hoffman, Karim Zeghal Eurocontrol Experimental Centre, BP 15, 91222 Bretigny, France *Steria Transport Division, 78142 Vélizy, France. ♦ Finetis, 91370 Verrières le Buisson, France • CRNA Sud Est, F-13617 Aix en Provence e-mail: {firstname.lastname}@eurocontrol.int To address the validation of a new concept based on the redistribution of tasks between air traffic controllers and flight crews, a joint validation framework has been defined. Indicators and metrics were proposed to assess similarly the impact of the new task distribution on four dimensions: human shaping factors, human activity, human mission efficiency and safety. Examples are used to highlight how close and mutually dependent the controller and the flight crew performances are, at every level of the validation framework.

INTRODUCTION A new task distribution between air traffic controllers and flight crews is envisaged as a possible option to improve air traffic management. Starting with the analogy with existing practices (visual separation clearance), the proposed distribution relies on the use of a new instruction denoted ASAS spacing and tasking the flight deck to implement a solution defined by the controller (EUROCONTROL, 2001). The validation of a new concept aims at assessing first whether it addresses the problem of concern for which it was designed and second if it achieves its stated aims (MAEVA, 2001). As widely known, validation shall be approached in terms of process and not only as a final result (VALSUP, 2002). The validation strategy provides the framework for specifying and performing a set of validation activities, defining reference models, techniques, methods and rules to structure and standardise validation. Defined in the early stage of the design life cycle, the validation shall follow a series of iterative steps, consisting in defining the problem tackled, identifying general validation aims and deriving metrics. From these, appropriate validation techniques are selected and implemented in the context of validation exercises. Compared to the introduction of new individual tools, a new task distribution introduces specific issues. Beyond assessing if and how individual tasks are modified, it requires the whole system to be taken into account. Rather than considering successively and independently control and flight tasks, the approach followed consists in analysing the spacing task itself. The focus of the present paper is to describe a validation framework, consisting in assessing jointly controller and

pilot sides despite running separated experiments. The same technique (real time experiment) has been used to assess the benefits and limits of the new spacing instruction. Our validation framework applicable to both controllers and flight crews, presented in the method section is based on a preliminary literature review (Amalberti, 1996; Casso et al., 2001; Harwood, 1993). Then, indicators and metrics initially defined for controllers and then derived for flight crews are presented and illustrated in the results section. METHOD Bottom up approach Despite its fundamental relevance, the objective of defining a top-down approach to validation is far from achieved. Initially following a bottom-up process, the present study aims at taking advantage of initial results to propose and adopt a top down approach to validation. Four questions were identified as a guide to validate the concept: -

-

What is its impact on the main human shaping factors (such as workload, confidence, concept usability, skills)? What is its impact on human activity? As a result, how does the concept impact on human mission efficiency, addressed in terms of flow management efficiency (e.g. throughput, respect of schedule) and flight efficiency (e.g. optimised trajectory)? Last of all, given these previous assessment, is the existing level of safety maintained (i.e. assess the

absence of conventional errors, as well as the mitigation of possibly new induced errors)? The first two items are essential to assess human involvement: whereas the human shaping factors provide elements to understand motives for rejecting/accepting the concept and help explain the human activity, the impact on human practices could give information about the transition issue. The last two items can be associated to efficiency and safety ATM2000+ strategic objectives (EUROCONTROL, 2003). The overall validation aims at identifying if the concept is acceptable from stakeholders perspectives. Holistic approach to validation To validate the new concept, we replace the human actors (controllers and pilots) in their overall socio-technical system. The holistic approach considers jointly all the system components, human and technical, in the context of their mission. The objective of validation is double: first to assess locally to what extent the effort is acceptable (e.g. human workload) and second to assess globally the resulting system mission, in terms of efficiency and safety. Typically, controller and pilot missions correspond to the output of their tasks. These missions are aimed towards the system global performance expected in terms of safety (e.g. respect separation minima), flow management and flight efficiency. Note that even if from now on we focus on the human, the same approach needs to be applied to technical components. As stated earlier, the global objective of validation is to assess to what extent a component is able to perform its assigned mission at an acceptable cost. The validation exercise is applied for a given concept, in a given environment, itself part of a global socio-technical system. The validation aims at analysing the potential impact of the concept on the various elements: human shaping factors, human activity, human and system missions (Figure 1). Environmental constraints (e.g. traffic characteristics, airspace structure) and concept requirements can be seen as independent variables. Whereas they need to be clearly specified before the validation exercise, they can not be considered as an object of the validation itself. Typical results of a validation exercise could show the impact of different environmental constraints on benefits and limits of a concept. A set of human related factors denoted human shaping factors, influence the task performance and are also influenced by the task itself (HERA, 2002). Human shaping factors which are all mutually dependent need to

be analysed jointly. The human activity analysis is addressed through the description of the tasks performed. It aims at assessing the potential benefit of the concept for the concerned task (i.e. the task that the concept is expected to support/improve) and at checking that it does not impact negatively other tasks (e.g. co-ordination, detection and resolution of conflict). In the case of spacing instruction which involves controllers and flight crews, it is essential to identify if and how the control and the flight tasks are modified. In addition, the actions executed, the strategy used and the working methods need to be considered as a result of the human shaping factors, of the environment constraints (e.g. traffic load) and of the concept requirements (e.g. applicability conditions). The way the tasks are performed (observed activity) results in a service provision, in which efficiency and safety is more or less acceptable. This service provision is the goal of human missions. Concept validation aims at understanding how human missions contribute to overall system mission. Acceptance, defined as motives for accepting or rejecting the concept is informed by human shaping factors measures (e.g. perceived workload, confidence, skills required, usability) and by measures of the task outputs. Environmental constraints - Sectors - Flows (includes types of aircraft) - Existing tools

Service/Application - % equipped - system reliability

Human shaping factors Human factors Workload Confidence Skills Motivation Team work

Usability

Human activity (strategy, behaviour)

Human missions (effectiveness, safety)

ATM system missions Safety Efficiency Capacity

Stakeholders Acceptance (includes Users)

Figure 1. Proposed validation framework.

RESULTS AND DISCUSSION Similar framework, metrics and experimental set-up In the present study, the validation framework is applied to the main actors involved, that is controllers and flight crews. In addition to the description of the concerned environment (e.g. approach control, dense area, current working tools), the validation exercise provides the assessment of four dimensions (human shaping factors, activity, efficiency and safety) applicable to both controllers and pilots. The analysis of the spacing tasks from controllers' perspectives, helped refine hypotheses regarding benefits and limits of the concept and led to the identification of metrics for each dimension. Similar work was conducted for flight crews (Table 1). The next step consisted in identifying measures that could qualify the various metrics. Note that only some of the measures were collected during previous real time experiments. Successive iteration over the next few years, complemented with fast time simulation results, literature review and operational feedback will be necessary to address most of the identified measures. Dimensions Human shaping factors

Human activity

Efficiency

Safety

Controller metrics

Flight crew metrics

Human factors (e.g. workload, trust, skills) Usability Flow management

Flight management

Situation awareness

Situation awareness

Assume/transfer aircraft

Flight preparation

Ground coordination

Resource management

Quality of control

Quality of flying

Pseudo pilot perspective Pseudo controller perspective Flow efficiency

Aircraft (flight) efficiency

Control errors

Flight errors Delegation-related errors

Table 1. Controller and flight crew metrics.

The controller-in-the-loop and pilot-in-the-loop experiments, conducted on a yearly basis, use the same operational environment (Paris south-east arrival). Instructions, given by controllers during the ground experiment are used to build scenarios for the air experiment. Even though controller experiments provide a partial view of the flight deck issues (and reciprocally), questions raised during ground experiment (e.g. number of instructions per aircraft, transfer conditions) can also be investigated during air experiment.

Joint data analysis Following the definition of a similar validation framework, the data analysis aims at considering jointly both controllers' and pilots' perspectives. Rather than reviewing all metrics and their mutual dependencies from both perspectives, the approach will be illustrated here with one item per dimension. Note that detailed controller-in-the-loop results are presented in Rognin et al. (2002) and pilot-in-the-loop results in Hoffman et al. (2003). The objective of the concept is to increase controller availability through a better task allocation. It is expected that the reduced workload on the controller side can contribute to improved safety and efficiency. From pilots' perspectives, it is expected that despite additional tasks related to the execution of the instruction, communication and situation awareness related tasks are less demanding. The first objective of analysis is to assess the level of workload on both controller and pilot sides. The second objective is to assess if and how the new spacing instruction impacts controller and pilot activity. The last objective consists in assessing if and how the spacing instruction modifies the effectiveness of controller and pilots task, through analysing their outputs. Impact on human shaping factor Workload is one of the main human shaping factors influencing the activity and the user acceptance. On the ground side, the number of manoeuvring instructions is considered as one indicator of workload. Results from controller-in-the loop experiments show that using the spacing instruction enables an overall 35% reduction of the total number of instructions, and more specifically a 60% reduction of the speed instructions. To address pilot workload the number of actions performed in the cockpit can similarly be considered. A first insight can be extracted from controller-in-the-loop experiment. Analysing the repartition of manoeuvring instructions received per aircraft showed an overall reduction "equally" spread among every aircraft, and not detrimental to some of them. With the spacing instruction (Figure 2), it can be observed that more aircraft get fewer speed instructions. Note that more aircraft also get fewer heading instructions (not illustrated here). Although this is a positive indication, this does not reflect the actions, and typically the speed adjustments performed in the flight deck. Only pilot-in-the-loop experiment could provide us with initial insights on this issue. To assess the impact of spacing task on flight crew

50

Without spacing instruction

40

30

8 7

Maintaining

20

6 5

Building

10

4 3

0

2

0

20

40

60

1 0

50 0

10

20

30

40

With spacing, aircraft not instructed

160

180

200

180

200

With spacing instructions

50

Number of aircraft Without spacing instruction

80 100 120 140 Distance to IAF (nm)

Mean number of instructions

Number of instructions per aircaft

Distribution of speed instructions per aircraft

maintenance (Figure 3). Eye movement analysis also showed that with the spacing instruction, controllers focus their attention on the sequence building area. Note that according to initial results, aircraft "spacing instructed" are not forgotten by controllers, but monitored as frequently as other aircraft.

Mean number of instructions

activity, the number and the magnitude of speed adjustments performed in the cockpit were analysed. The spacing instruction induces an increase in the number of speed actions, up to 1.8 actions per minute. However, in most cases, actions correspond to small speed adjustments (within –10 and +10 knots) rather than large changes. Despite more numerous speed actions, the temporal and mental demands induced by the spacing task are perceived as still acceptable by the flight crews.

40

With spacing, aircraft instructed

Figure 2. Repartition of speed instructions per aircraft.

Impact on activity Controller and flight crew activity is more than maintaining the spacing. However, to illustrate our approach, we will focus on the specific task of spacing acquisition and maintenance. From a controller perspective, this task involves (1) the definition of a strategy (aircraft order and means to establish the selected order), (2) the use of manoeuvring instructions (usually heading and speed) to build and maintain the sequence, (3) the monitoring of the situation and (4) if needed some more adjustment actions. To assess the impact of the spacing instruction on the sequencing task, advantage was taken of the natural mapping between geographical sector and controller activity. Typically the geographical distribution of instructions was investigated. Results show that using the spacing instruction modifies flow management by reducing the number of instructions used, but does not modify controllers strategies. Heading (or equivalent spacing instructions) are used to build sequences, whereas speed adjustments are used to maintain them (even if they are no longer visible at controller level, but should be detected on the flight crew side). With the spacing instruction, controllers did anticipate the sequence building and were no longer in charge of their

30

Building

20

10

0

0

20

40 Speed

60

80 100 120 140 Distance to IAF (nm) Heading

160

Spacing

Figure 3. Geographical distribution of instructions. In conventional situation (top) and with spacing instructions (bottom).

On the flight deck side, to relate speed actions to flight phases, we looked at their temporal distribution (Figure 4). The distribution shows that speed actions are triggered by target events (descent followed by speed restriction). In addition, two phases are distinguished in the spacing task: acquire the spacing and maintain it. The acquisition phase is the least demanding in terms of number of actions: it usually requires 1 speed action, while maintaining the spacing in spite of target events, such as descent and speed reduction required many more (up to 7 in 2 minutes). Observations show that even with the spacing task, flight crews are able to perform their conventional tasks (e.g. briefing, checklist).

8

Target descent

7

Target speed reduction

6 5 4.1

4

3.7

3

2.6

2.6 2.1

2

1.6

1.4

1 0.2

0.0

[360;…[

[240;360[

[120;240[

[0;120[

]-120;0[

]-240;-120]

]-360;-240]

]-480;-360]

0 ]…;-480]

Number of speed adjustments

Time distribution of speed adjustments LFPG scenario Remain phase Acquisition phase

Time (sec.)

Figure 4. Temporal distribution of speed actions.

Impact on mission efficiency and safety) Effectiveness of controller mission is measured through analysing the output of all controller tasks. In the present paper we focus on the output of the sequencing task, which can be described in terms of transfer conditions. Note that the transfer conditions are simultaneously an indicator of effectiveness (in terms of accuracy and regularity of spacing) and of safety (in terms of catching up aircraft). Results showed that with the spacing instruction, aircraft transferred are more homogeneously and regularly spaced, in a more stable situation (Rognin et al., 2002). The improved transfer conditions reflect the results of a controller using the instruction, but in an idealistic air situation. Typically, in controller experiment the aircraft behaviour are optimistic: aircraft are always able to comply with their objectives. The spacing accuracy obtained by flight crew in a pilot-in-the-loop experiment could indicate how close to "reality" the simulated aircraft behaviour are. Results show that in both acquisition and maintaining phases the average spacing deviation is below 0.25NM. Flight crews seem to be in a position to maintain the required spacing within a tight tolerance margin, even smaller than the tolerance margin defined for controllerin-the-loop experiments. CONCLUSION To address the validation of a new concept, based on the redistribution of tasks between air traffic controllers and flight crews, a joint validation framework has been defined. Indicators and metrics were proposed to assess similarly the impact of the new task distribution on four related dimensions: the human shaping factors, the human activity, the human missions efficiency and safety. Examples were proposed to highlight how close and mutually dependent the controller and the flight crew performances are, at every level of the validation

framework. Typically, at the activity level, because the overall task is redistributed between the human actors, the analysis shall consider jointly the controller and the flight crew activity. The main point here was to jointly analyse both controller and flight crew perspectives at every level, keeping in mind that in the specific case of the spacing instruction, controller and flight crew activities are strongly coupled. The next step, in terms of validation is related to stabilising and refining the framework. Typically, at the first level, an explicit description of mutual links between human shaping factors and of their impact on activity is needed. In addition, definition of appropriate and usable data collection and analysis techniques addressing the human shaping factors assessment needs to be improved. In terms of measuring impact on activity, a joint task model, apprehending the spacing task in its collective dimension, considering jointly controllers and pilots in the various sub tasks allocation. AKNOWLEDGEMENTS This work is supported by the EUROCONTROL Experimental Centre, the EUROCONTROL AGC programme and the European Commission North European ADS-B Update Project (NUPII). The authors wish to acknowledge the work of the technical teams that made this work possible. REFERENCES Amalberti R. (1996). La conduite de systèmes à risque. PUF. Casso N.; Kopardekar P. (2001). Raytheon ATMSDI (Air Traffic management system development and integration). December 2001. Draft guidelines Subtask 4 Human factors metrics guidelines. EUROCONTROL (2003). ATM2000+ Strategy Volume 1, ed. 2003. EUROCONTROL/FAA (2001). Principles of operations for the use of airborne separation assurance systems, Cooperative R&D, Ed. 7.1. Harwood K. (1993). Defining Human-centered system issues for verifying and validating air traffic control systems. In J. Wise, V.D. Hopkin, and P. Stager (Eds.), Verification and validation of complex and integrated human machine systems. Berlin: Springer-Verlag. HERA Short report on Human Performance models and Taxonomies of Human Error in ATM. Version 1. EATMP EUROCONTROL report. April 2002 Hoffman, E., Pene, N., Rognin, L. & Zeghal, K. (2003). Introducing a new spacing instruction. Impact of spacing tolerance on flight crew activity. Human Factors and Ergonomics Society (HFES) 47th Annual Meeting. Denver, USA MAEVA (2001). Validation Guideline Handbook. D1.3. Rognin, L., Grimaud, I., Hoffman, E. & Zeghal, K. (2002). Investigating Delegation of Spacing Tasks from Air Traffic Controllers to Pilots. Impact on Controller Activity. Human Factors and Ergonomics Society (HFES) 46th Annual Meeting. Baltimore, USA. VALSUP- EEC, Validation guidelines. Version 1.2 - April 2002.