Standard Dokument-Vorlage (deutsch)

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/261686940

Aviator II - Simulator-based Research on Operational Monitoring and Decision Making for Human Operators in Future Aviation Technical Report · October 2013 CITATIONS

READS

4

125

13 authors, including: Carmen Bruder

Hinnerk Eißfeldt

German Aerospace Center (DLR)


54 PUBLICATIONS 190 CITATIONS


SEE PROFILE

SEE PROFILE

Dietrich Grasshoff

Hejar Gürlük





SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Pilot selection methods View project

Aviator II View project

All content following this page was uploaded by Dirk Schulze Kissing on 09 July 2015.

The user has requested enhancement of the downloaded file.

SEE PROFILE

Aviator II

Simulator-based Research on Operational Monitoring and Decision Making for Human Operators in Future Aviation

DLR FB 2013-04

Aviator II Simulator-based Research on Operational Monitoring and Decision Making for Human Operators in Future Aviation

Deutsches Zentrum für Luft- und Raumfahrt e. V. Institut für Luft- und Raumfahrtmedizin, Luft- und Raumfahrtpsychologie Sportallee 54a, 22335 Hamburg Hamburg, October, 11, 2013

Institutsleiter:

Verfasser:

Prof. Dr. med. Rupert Gerzer

Carmen Bruder Hinnerk Eißfeldt Dietrich Grasshoff Hejar Gürlük Maik Friedrich Catrin Hasse Hans-Jürgen Hörmann Alex Hoff Anne Papenfuß Dirk Schulze Kissing Maria Uebbing-Rumke Jürgen Wenzel Oliver Zierke

Abteilungsleiter: Dr. phil. Peter Maschke

Future Aviation, Ability Requirements, Eye Tracking, Operational Monitoring, Air Traffic Control Aviator II – Simulator-based Research on Operational Monitoring and Decision Making for Human Operators in Future Aviation This report summarises the DLR (German Aerospace Center) project Aviator II. Based on findings of the previous project Aviator 2030, Aviator II focused on researching operational monitoring and decision making in future aviation. In doing so, two simulations were developed: AviaSim and MonT (Monitoring Test). AviaSim is a lowfidelity simulation, which combines flight deck systems and ATC systems. With focus on the transition from the space-based to the time-based guidance of aircrafts, two AviaSim studies were conducted to empirically compare current scenarios and future scenarios with pilots and air traffic controllers. For this reason, the arrival manager ‘4DCARMA’ with specific levels of automation was integrated into AviaSim. Eye gaze, simulation and questionnaire data gave information about the requirements for future aviators. In contrast, MonT is a simplified and abstract simulation of the basic requirements for future flight operators. To identify operators monitoring appropriately, a number of eye tracking studies were conducted with both applicants and experts. Eye tracking, performance and questionnaire data were analysed. Results indicated that eye tracking is an appropriate method to analyse operators’ monitoring performance.

Zukunft der Luftfahrt, Fähigkeitsanforderungen, Blickerfassung, Operational Monitoring, Flugsicherung Simulationsbasierte Untersuchungen zukünftiger Formen der Überwachung und Entscheidungsfindung für Luftfahrtberufe In diesem Bericht sind die Ergebnisse des DLR (Deutsches Zentrum für Luft- und Raumfahrt e.V.)-Projektes Aviator II zusammengefasst. Aufbauend auf den Erkenntnissen des Vorgängerprojektes Aviator 2030, wurden in Aviator II zukünftige Anforderungen an die Überwachung und Entscheidungsfindung von Luftfahrtoperateuren untersucht. Dafür wurden die realistische Simulation AviaSim und das abstrakte Simulationsprogramm MonT (Monitoring Test) ausgebaut und weiterentwickelt. Mit der Bord-Boden-Simulation AviaSim wurde in einer Studie der Wechsel von räumlicher zu zeitbasierter Flugführung untersucht, in dem der DLReigene Arrival Manager 4D-CARMA mit seinen spezifischen Unterstützungsfunktionen in AviaSim integriert wurde. Die erhobenen Blick-, Simulations- und Fragebogendaten geben Aufschluss über Anforderungen der zeitbasierten Flugführung an zukünftige Piloten und Fluglotsen. Mit der abstrakten Simulation MonT wurde die Leistung in der Überwachung hochautomatischer Systeme von Bewerbern für Luftfahrtberufe gemessen. Dafür wurde in einer Reihe von Studien mit Bewerbern und Experten Blickdaten erhoben und mit Leistungsdaten der Person, wie z.B. der Güte der Entdeckung von Fehlern des automatischen Systems in Beziehung gesetzt. All diese Erkenntnisse führen zur Entwicklung eines Monitoring-Tests, der erstmals die Leistung in der Überwachung hoch-automatisierter Systeme anhand von Blickdaten misst.

Contents 1

Introduction ............................................................................................................. 6

2

A Brief Review of Aviator 2030 .............................................................................. 8

3

2.1

Main Results of Workshops ..................................................................................................9

2.2

AviaSim .................................................................................................................................... 10

2.3

SSAS ........................................................................................................................................ 11

2.4

Adjustment of Selection Profiles ...................................................................................... 11

2.5

Conclusions from Aviator 2030......................................................................................... 12

PART A: MonT – Measuring Monitoring Performance by Eye Movements ...... 14 3.1

Re-analysing SSAS Data with Regard to Applicants’ Complacency Potential and Ability Test Performance ......................................................................... 16

3.2

Teamwork within the Remote Tower Center ................................................................ 21

3.3

Description of the MonT Simulation ................................................................................ 31

3.4

Defining Adequate Monitoring Performance................................................................. 36

3.5

Analysing Eye Movements................................................................................................. 44

3.6

Study 1 - Eye Tracking Parameters as a Predictor of Human Performance in the Detection of Automation Failures................................ 49

3.7

Study 2 – Validating the Normative Model of OMA with experts............................ 56

3.8

Study 3: Comparison of Novices’ and Experts’ Monitoring Behaviour ................. 62

3.9

Study 4: Comparison of Monitoring Behaviour during a Passive and an Active Control Task ............................................................................... 67

3.10 Study 5 – The Effect of Teamwork on Monitoring Behaviour .................................. 72 3.11 Developing the ‘Monitoring Selection Test’ (MonSeT) .............................................. 96 3.12 Integrated Discussion of MonT Studies ....................................................................... 102

4

PART B: AviaSim ................................................................................................ 106 4.1

Introduction ........................................................................................................................... 106

4.2

Context of the Simulation Experiments ........................................................................ 106

4.3

Overview of Simulated Elements Referring to the SESAR ATM Master Plan . 106

4.4

Summary of Experimental Objectives and Hypotheses .......................................... 107

4.5

Simulation Scenarios ......................................................................................................... 108

4.6

Simulation Platform “AviaSim” ........................................................................................ 109

4.7

Choice of Methods and Techniques ............................................................................. 110

4.8

Experimental Exercises List and Dependencies ....................................................... 113

4.9

Summary of Experimental Results ................................................................................ 116

4.10 Conclusions and Recommendations ............................................................................ 117 4.11 Report on the Integration of a Gaze Tracking System into the Experimental Set-up of AviaSim .................................................................................... 119 4.12 Report on Simulation Experiment #1: Adaptive Support for Workload Attenuation ............................................................................................................................ 127 4.13 Report on Simulation Experiment #2a (Future Ability Requirements; ATCO Perspective): Selecting the Air Traffic Controller of the Future ............................ 138 4.14 Report on Simulation Experiment #2b (Future Ability Requirements; Cockpit Perspective): Selecting the Pilots of the Future ......................................... 181 4.15 Report on Experimental Exercise #3 Dynamic Anticipation Test Report........... 196

5

Summary of the Project’s Outcomes ................................................................ 202 5.1

Simulation Tools .................................................................................................................. 202

5.2

Analysing Eye Tracking Data .......................................................................................... 203

5.3

Psychological Requirements for Future Operators in Aviation ............................. 204

5.4

Psychological Tests ........................................................................................................... 205

6

References........................................................................................................... 207

7

List of Tables ....................................................................................................... 218

8

List of Figures ..................................................................................................... 220

9

List of Abbreviations .......................................................................................... 223

Introduction

1 Introduction Improvements in air traffic management (ATM) and aircraft systems as well as organisational structures have led to some of the key challenges facing aviation in the 21st century. According to the SESAR’s Concept of Operation: “humans (with appropriate skills and competences and duly authorised) will constitute the core of the future European ATM System’s operations. However, to accommodate both the expected traffic increase and the reference performance framework, an advanced level of automation will be required. The nature of human roles and tasks within the future system will necessarily change. This will affect system design, current staff selection, training (especially for unusual situations and degraded modes of operations), competence requirements and relevant regulations.” (SESAR, 2007a, p. 27). “As a result of the comprehensive transition process foreseen, the jobs, responsibilities and supporting technologies of approximately 200,000 people will significantly change. Approximately a further 330,000 operational ATM staff will be affected due to the requirement for training measures and a systematic management of social and legal implications” (SESAR, 2007b, p. 2). Based on findings of the previous project, Aviator 2030, Aviator II focused on researching operational monitoring and decision making in future aviation. The research carried out in the previous project resulted in important suggestions for the simulation of future scenarios. Based on simulation of future scenarios, this project aimed to answer following research questions: • How do improvements in air traffic management and aircraft systems affect monitoring and decision making in future aviation? • What impacts will the future changes have on working in teams? • How should future ability tests which complement current ability tests be designed? To put it in a nutshell, findings of Aviator 2030 pointed to a higher significance of operational monitoring in future ATM systems. The transfer from current operations (where the timely precise adherence to flight plans is not a high priority) to time-based guidance will be an important topic as well. Researchers from various DLR units with backgrounds in aviation psychology, operational medicine, flight physiology, and system ergonomics participated in the follow-up project Aviator II. Figure 1 illustrates the course of action of Aviator II. The project started with a knowledge exchange about future concepts of teamwork, operational monitoring and time-based guidance of approach control. Based on this, measurements and simulation tools were developed to conduct simulation studies. During the project, simulation studies of different levels of abstraction were carried out to 6

Introduction

investigate different aspects of monitoring and decision-making in future ATM. In doing so, experiments on standardized conditions were conducted. AviaSim is a linked low-fidelity ground-board simulation that is ideally suited for scenarios with pilots and air traffic controllers. MonT (Mont Test) is a rather abstract traffic flow simulation, which was used to test job applicants’ operational monitoring ability by measuring their eye movements. In addition, ophthalmic data was recorded and evaluated. The findings of these simulation studies were analysed to review future ability requirements and to derive criteria for future ability tests.

knowledge transfer between DLR projects

future ability requirements

simulating requirements & conducting studies

future simulation data

deriving criteria for testing future abilities

future ability tests

Figure 1: Aviator II – flow chart

This report starts with a brief overview about the previous project, Aviator 2030, and its finding (chapter 2). The third chapter presents the findings of MonT, which was used to measure operational monitoring under highly automated conditions by recording eye movements. The fourth chapter reports the AviaSim research activities focusing on the effect of future approach operations, i.e. time-based guidance. The summary of project’s outcomes of this report brings the findings of AviaSim and MonT together (chapter 5). Two future tests are presented which will be a worthwhile addition to existing ability tests.

7

A Brief Review of Aviator 2030

2 A Brief Review of Aviator 2030 The key question of the project Aviator 2030 was to describe ability requirements for pilots and air traffic controllers in future ATM systems. To identify potential changes in ability requirements in advance would allow for the timely adjustment of selection profiles. In order to tackle this question, researchers teamed up from various DLR units with backgrounds in aviation psychology, operational medicine, flight physiology and system ergonomics. At the start of the project, existing concepts were reviewed to gain an overview of new ATM developments. However, as no potential future system was described in detail at that time, the project had to find an approach to identify future operators’ tasks, roles and responsibilities. At least four innovative elements were combined to reach these goals. 1) Based on domain experts’ points of view, anticipated changes in the ATM system were described using a special workshop technique taken from sociological research. The ‘Future Workshop’ concept ('Zukunftswerkstatt', see Jungk & Muellert, 1987) was used for the first time in a high-tech environment such as aviation. A set of workshops with pilots and air traffic controllers described scenarios of future ATM, providing a valid basis for further research. 2) A low-fidelity integrated simulation platform (AviaSim) was developed following a bottom-up approach by combining two off-the-shelf simulators to meet the requirements of high realism, low cost, high adaptability, and full controllability for experimental purposes. Besides all normal functions, the ATC environment provides a short-term conflict alert (STCA), various flight plan visualisations for mid-term conflict detection, and interactive labels for data link communication. The cockpit environment was upgraded by a data link window and a traffic visualization system (Cockpit Display of Traffic Information, CDTI) to provide information about the proximate traffic situation and aircraft intent to the pilots. Using AviaSim in a linked simulation allowed for the examination of new tasks, such as the transfer of control between air and ground as well as airborne self-separation in Free Flight Airspace, as suggested by workshop subjects. 3) Using a top-down approach, the traffic flow simulation tool (SSAS) was developed with the special purpose of helping to identify future monitoring requirements for pilots and controllers. In combining task performance data and eye movement analysis, appropriate operator monitoring was the focus of research. Scenarios of different levels of difficulty were first monitored and then manually controlled by the subjects. Using SSAS the study – at applicant level – revealed the relevant parameters of appropriate monitoring behaviour. In the long run this might allow applicants whose monitoring skills match the patterns of successful behaviour identified by SSAS to be selected. Eye 8


movement analysis was planned to play an important role within the project. However, eye movement data could be only be analysed for SSAS. 4) A standard tool for job analysis (F-JAS, Fleishman, 1992) was tailored to aviation-related research by integrating aviation anchors for the current job conditions of air traffic controllers and pilots. In addition, new scales were developed in a similar style to measure requirements not covered in the original material. Applying the F-JAS Aviator 2030 with aviation anchors allowed for an interpretation of whether job incumbents anticipated an increase or a decrease in the ability requirements in future ATM systems.

2.1

Main Results of Workshops

Workshops with experienced air traffic controllers and pilots were conducted to obtain job incumbents’ expectations regarding their future tasks, roles and responsibilities. Each future workshop started with an information session: subjects were informed about the general idea of the project, the goals of the ‘Vision 2020’ for European aeronautics and the Concept of Operations for the Single European Sky (SESAR, 2007b). Controllers and pilots enjoyed sharing their future scenarios. Mixed groups consisting of controllers and pilots elaborated several ideas: a concept of trajectory negotiation, procedures for operating flights in the future and an integrated training system for pilots and air traffic controllers. In general, subjects developed future scenarios including ATCOs’ and pilots’ perspectives. Finally, subjects derived future scenarios which should, according to their background, be simulated and tested in the on-going project. Table 1 consists of a schematic diagram of the simulation scenarios suggested by workshop subjects. On the one hand suggestions address 4D-trajectory planning and visualising. On the other hand subjects specified different aspects of tactical operation, such as tasks allocation, teamwork and monitoring. The following psychological topics were pointed out: vigilance, information reception, attention, detection of changes, and communication between operators. In particular, controllers as well as pilots stressed research on vigilance and attention while monitoring, and communication under changing task allocation. Furthermore, they anticipated the need for further research on teamwork if communication shifts from oral to electronic. To tackle these issues from different perspectives, in Aviator 2030 two different lines of simulation were developed: a low-fidelity linked air ground simulation enabling studies on monitoring in a realistic setting to conduct studies, and an abstract simulation allowing the study of basic aspects of monitoring.

9


Table 1: Matrix of simulation scenarios suggested by workshop subjects (with numbers of times mentioned)

strategic operation planning visualising

tactical operation tasks teamwork monitoring ∗

vigilance information reception

∗

∗

∗

∗∗ ∗

attention (divided, selective)

∗

∗∗

detection of changes

∗

∗

communication (between operators)

∗∗

∗∗∗

2.2 AviaSim AviaSim is a low-cost simulation which provides the possibility to contribute empirical data to new procedure design and concept validation tasks. The AviaSim study not only looked into potential shifts in ability requirements for ATCOs and pilots with future ATM concepts. At the same time the study explored differences regarding workload and situation awareness by introducing new procedures for airborne separation. Compared to baseline findings, controllers experienced relief from their high workload and also a slight increase in situation awareness during the future scenario. Pilots reported very low workload during the baseline scenario, which increased to a moderate level in the future scenario. Situation awareness remains almost as high as before for the pilots, however the mental demands on attentional resources were considerably higher in the selfseparation condition. The shift of control authority itself produced just a small increase in workload for the pilots during entry and merging traffic phases. A different finding was of more interest concerning the transfer of authority. In the future scenarios the transfer of control was done in two steps: first, in the transfer zone, the horizontal separation was handed over to the pilots while the vertical separation was still with the ATCOs. A finding unanimously shared by pilots and air traffic controllers in the debriefings was to demand a full transfer of authority at a given moment in time rather than the stepwise transfer. Most, if not all, new automation aids display information relevant to the operator visually on a screen. A potential overload of the visual channel is a concern often noted by human factors experts. With AviaSim, a more complex eye gaze system failed to provide reliable data. Detailed ophthalmologic examination showed this was not due to any eyesight problems of AviaSim subjects. 10


2.3 SSAS The increase in automation requires operators monitoring appropriately (OMA). OMA are assumed to monitor in such a way as to enable them to detect system errors in time, and to take control if automation fails. According to models of adequate and efficient monitoring behaviour (e.g. Niessen & Eyferth, 2001; Whitfield & Jackson, 1982) as well as differences between experts and novices (Underwood, Chapman, Brocklehurst, Underwood, & Crundall, 2003), a normative model was devised which describes the monitoring behaviour of OMA. It can be concluded from a variety of psychophysiological and imaging studies that eye movements are appropriate measurements of an efficient and timely acquisition of visual information. Thus, an empirical study was undertaken to test the normative model of monitoring behaviour, i.e. its postulated monitoring phases and their relationships to manual control. A simulation tool was developed which assesses both monitoring performance and manual control performance. With SSAS, visual monitoring in an operational setting can be studied. The aim is to identify suitable eye tracking parameters which record the monitoring process and, at the same time, are related to manual control. Diverse eye movement parameters were applied to record monitoring performance in order to decide on their suitability for identifying OMA. The results support the assumption that suitable operators direct their attention to relevant areas during monitoring scenarios. Monitoring performance explains independent portions of variance. It could be verified that well performing operators direct their attention to relevant areas as predefined by the normative model. These findings show that testing individual monitoring behaviour based on eye movements is an appropriate method to identify suitable new recruits in future aviation. In conclusion, as part of the follow-up project, Aviator II, SSAS was planned to be a test called MonT (Monitoring Test) by including a measurement of the ability to detect automation failures. This test could later be used in the personnel selection of future aviation personnel, such as pilots and air traffic controllers.

2.4 Adjustment of Selection Profiles The F-JAS instrument was applied at different stages during the project. At the integrated workshop WS3 it was administered with aviation anchors reflecting the current jobs of pilots and air traffic controllers, measuring the job requirements in future ATM systems. With AviaSim a reduced set of scales was administered repeatedly with additional research scales to reflect the experience of subjects during the baseline and future simulation scenarios. AviaSim results reflect the findings of the WS3 well; pilot ratings are lower for most scales when compared to air traffic controller ratings, for the baseline as well as for the future scenario. 11


Concerning potential adjustments of selection profiles, results consistently show an increase in requirement levels for pilots rather than for controllers. The sharp increase anticipated for visual colour discrimination for both groups at WS3 was not found with the AviaSim future scenario; only for pilots was there a mild increase. The most important change in ability requirements for pilots is seen to be visualization, as in the future the task of conducting airborne separation in free flight airspace requires ‘having a picture’ of relevant elements of air traffic similar to that of air traffic controllers. This new requirement is not reflected in today’s selection profiles of pilots. Another interesting finding in this context is connected to the scale of originality. Under free flight, pilots experience a pronounced increase in requirements to come up with unusual or clever ideas about a given traffic situation. Their ratings and the controllers’ ratings are on a comparable level because pilots under free flight are doing parts of the work which the controller did before. It can be assumed that different ability levels concerning visualization and/or originality within the present pilot population exist, as these requirements are not directly tested in many ab-initio pilot selection systems. It will be interesting to see how effective pilot training for self-separation can compensate for these differences in the future. Other results need to be treated carefully. For instance the high requirement level for vigilance in pilots might be due to single pilot operation in AviaSim scenarios. However, findings from the workshop debriefings as well as from the SSAS suggested a new requirement to be crucial for humans operating in man-machine settings: ‘operational monitoring’. Operational monitoring means to follow up on meaningful information from various sources (e.g. an automated system) responsibly without direct need for action. It involves being prepared to fully take over the handling of a system at any given time. In view of the AviaSim findings, operational monitoring can be said to be composed of the following ability requirements: problem sensitivity, situation awareness, decision making and vigilance. They all merge in the requirement ‘operational monitoring’ when it comes to working with automation.

2.5 Conclusions from Aviator 2030 In February 2010 a final symposium was held to share the results with the scientific community. This was supported by high-level project reviews. The reviewers agreed that the simulation facilities developed in Aviator 2030 represent a very good platform for research into human-computer interactions. The combination of bottom-up and top-down approaches in the project was deemed particularly successful in providing initial insights into future ability requirements. A core aspect of future ability requirements for aviation professionals is marked by the need to switch immediately between monitoring and decision making. Future research into ability requirements for ATM systems should therefore focus 12


on operational monitoring. In this context the simulations developed in Aviator 2030 have been assessed as very helpful tools; however recommendations for further development have been made. Concerning AviaSim it was recommended to cooperate with the developers of future ATM systems more closely to assure valid simulation content. This could go as far as integrating a typical future tool, for instance an Arrival Management System (AMAN), into AviaSim. Describing use cases to define the future roles of pilots and controllers more closely in the scenario was another recommendation which, together with enhancing the training phases provided during simulation, should ensure that differences measured between future and current ATM system cannot be attributed to different levels of competence. Concerning the SSAS it was recommended to develop the concept of ‘Operators Monitoring Appropriately’ further and to link it to basic abilities and skills. A detailed analysis of correlations between SSAS performance and ability tests as well as with complacency was proposed. In addition, studying subject-matter experts’ performance in SSAS was suggested for confirming the normative model of appropriate monitoring performance for pilots and air traffic controllers. However, the main aspect of SSAS enhancement should be the development of the application to enable it to be used for teams of at least two subjects working in separate rooms. This would allow the study of shared monitoring and distributed decision making as well as other future core abilities. In the following chapters it is described, how Aviator II took up these recommendations and investigated future requirements for pilots and air traffic controllers.

13

PART A: MonT – Measuring Monitoring Performance by Eye Movements

3 PART A: MonT – Measuring Monitoring Performance by Eye Movements According to research on the future of aviation, such as the Single European Sky ATM Research Program (SESAR), operators will have to work with highly automated systems. Wickens, Mavor, Parasuraman, and McGee (1998) concluded that automation might affect system performance due to the new skills that may be required, and that human operators might not have been adequately selected and trained to prepare for these changes. In order to gather expectations about future tasks and roles, workshops were conducted with experienced pilots and air traffic controllers (Bruder, Jörn, & Eißfeldt, 2008). Findings from the workshop debriefings suggest that there is a crucial new requirement for humans operating in man-machine settings: ‘operational monitoring’, which “includes using one’s senses to follow meaningful information from various sources (e.g. an automated system) responsibly, even when there is no direct need for action. It involves being prepared to fully take over control of a system at any time, for example in the case of malfunction” (Eißfeldt et al., 2009, p. 88). Thus, the increase in automation requires operators monitoring appropriately (OMA). OMA are assumed to monitor in such a way as to enable them to detect system errors in time and to take control if automation fails. Findings of previous SSAS studies suggest that ‘adequate operational monitoring’ is associated with accurate manual control in case of system failure (Hasse, Grasshoff, & Bruder, 2012a; Hasse, Bruder, Grasshoff, & Eißfeldt, 2009b; c). Besides this substantial effect, the question arises of how to capture the construct of human monitoring in the context of other human abilities, especially those abilities required in aviation. In doing so, operational monitoring data of the previous SSAS study were related to other relevant parameters, such as applicants’ complacency potential, ability test performance and personality traits (chapter 3.1). To extend the scope from individual monitoring behaviour to operational monitoring in teams, workshops with the DLR project RAiCe (Remote Airport Traffic Control Centre) were conducted. This is in line with the reviewers’ recommendations from Aviator 2030 to extend the exchange of information with other projects. The DLR project RAiCe aims to investigate the technological and organizational problems of remote surveillance of several small airports. In doing so, simulation studies with air traffic controllers were conducted who monitored two simulated airports in teams. In chapter 3.2, an overview of findings concerning teamwork is given and recommendations for investigating monitoring in teams were concluded. The simulation SSAS was further developed to a monitoring test simulation, MonT. In contrast to SSAS, MonT consists of two separate systems which must 14


be monitored simultaneously. This was done to enable monitoring in teams of two people; each is responsible for one system. The MonT simulation consists of two working positions for test subjects and a supervisory working position for the test leader (chapter 3.3). All working positions are connected with each other. Operators monitoring appropriately (OMA) monitor in such a way as to enable them to detect system errors in time and to take control if automation fails. The analysis of human monitoring behaviour is based on a normative model of appropriate monitoring behaviour. Furthermore, this model is transferred from monitoring an automated system alone to monitoring an automated system in teams. In chapter 3.4, the normative model of adequate monitoring behaviour is described for both the single and team situations. In addition, the eye tracking software was modified to allow combined analyses of team monitoring behaviour. Beyond that, the recording and analysis of eye gaze data were simplified. By doing so, the eye gaze data of two subjects monitoring an automated system together could be analysed semi-automatically (chapter 3.5). Several empirical studies were conducted using MonT. Thereby, six research questions were addressed: Study 1: Testing scenario difficulty and eye tracking parameters This study dealt with finding eye tracking parameters and adjusting scenario difficulty to help identify OMA who are able to detect automation failures. An experiment was conducted with 33 applicants for the DFS (Deutsche Flugsicherung GmbH). MonT was modified to measure test subjects’ monitoring of an automatic process and their reporting of automation failures while eye movements were recorded. Results revealed suitable scenarios as well as eye tracking parameters that help differentiate between the subjects' performance level in detecting failures (chapter 3.6). Study 2: Validating the normative model of OMA with experts This study was undertaken to test the model of OMA with pilots and air traffic controllers. Initial analyses from the results of the 21 air traffic controllers and pilots reveal that experts’ eye tracking data resemble the model of OMA in some aspects. It is concluded that the monitoring behaviour of current pilots and air traffic controllers can in part be explained by the model of OMA designed to represent the monitoring performance of future aviation personnel (chapter 3.7). Study 3: Comparing experts’ and novices’ monitoring behaviour This eye tracking study focused on differences in monitoring behaviour between experts (experienced pilots and air traffic controllers) and novices (applicants for air traffic control training). Results from 21 experts and 33 applicants are reported (chapter 3.8). 15


Study 4: Comparing monitoring behaviour during active and passive control This study examines the effect of a monitoring task on eye movements within the monitoring process. A passive control task (reporting automation failures) was compared to an active control task (assuming control if automation fails). Subjects performed both tasks while their eye movements were recorded. Eye tracking data were categorized into monitoring phases and related to relevant AOIs (areas of interest). The data imply that eye movements adapt to the demands of the monitoring task and phase (chapter 3.9). Study 5: The effect of teamwork on monitoring behaviour A single control task (subject was responsible for both parts of simulation) was compared to a team task (two subjects monitored together, each team partner was responsible for one part of simulation). Subjects performed both tasks while their eye movements were recorded. Eye tracking data were analysed and were related to other relevant parameters, such as subjects’ failure detection performance. With regard to the team task, different aspects of teamwork such as mutual monitoring behaviour and backup behaviour were addressed (chapter 3.10). Study 6: Developing a monitoring selection test (MonSeT) This analysis was done to validate MonT as a psychological test for operational monitoring. In doing so reliability and validity of measuring the operational monitoring of applicants by their eye movements were analysed and discussed (chapter 3.11).

3.1

Re-analysing SSAS Data with Regard to Applicants’ Complacency Potential and Ability Test Performance

This chapter dealt with the question of how to capture the construct of human monitoring in the context of other human abilities, especially those abilities required in aviation. In doing so, operational monitoring data of the previous SSAS study were related to other relevant parameters, such as applicants’ complacency potential, ability test performance and personality traits. The results reported in this chapter were published by Bruder, Hasse and Grasshoff (2011). 3.1.1

Background

Recent research suggests that the optimal monitoring and handling of automated systems depends on certain operator characteristics such as personality traits and abilities. The importance of complacency in human use of automation has been clearly established (Singh, Molloy, & Parasuraman, 1993). Complacency 16


addresses the risk of an inappropriate level of trust placed in the automation by a human operator (Lee & See, 2004). Such excessive trust can lead to overreliance on the automated system, without recognising its limitations and the possibility of automation failure. Several studies have demonstrated that particularly highly and consistently reliable systems give rise to complacency effects (Singh et al., 1993; Prinzel, DeVries, Freeman &, Mikulka, 2001). We assumed these attitudes to also be connected with monitoring performance. From this we hypothesized that complacency potential improves the predictive power of monitoring behaviour. Ability tests are used to measure a broad range of knowledge a person brings to the job situation (Rathje, 2002). The pre-selection of the DLR’s Department of Aviation and Space Psychology consists of a battery of tests covering all important abilities such as memory capacity, spatial orientation, concentration, attention, numerical abilities and personality as well as some knowledge-based aspects like English language competency and mechanical comprehension (Eißfeldt & Deuchert, 2002). Current ability tests for airline pilots and air traffic controllers measure the ability requirements which are needed to be successful under current training and job situations (Lorenz, Pecena, & Eißfeldt, 1995). This raises the question of whether one’s monitoring performance could be predicted by these ability tests as well. We assume that the DLR’s tests concerning attention and concentration are related to monitoring behaviour. 3.1.2

Method

Simulation tool: The simulation tool represents a traffic flow simulation (see Figure 2). The traffic flow simulation can be controlled either automatically or manually by a human operator. During the automatic phase, the system works fully automatically and the reliability is perfect. During manual phase, a human operator controls the dynamic traffic manually. This allows performance data to be collected separately for both types of tasks. The task of both the automated and human operator control settings is to bring all actual values into agreement with target values (for further information, see Hasse, Bruder, Grasshoff, & Eißfeldt, 2009a). Four scenarios, each with a different degree of difficulty, were developed by varying the complexity and dynamics of the automatic system.

17


Figure 2: Air traffic flow display of the MonT simulation

Procedure: Subjects were first given the instructions for the experiment as well as a questionnaire measuring their complacency potential. They were informed they would work on four scenarios, each consisting of two phases: first an automated phase and then a manual phase. For the automated phase of each scenario, subjects were instructed to monitor the processes with the objective of understanding the rule-based dynamics of the given scenario. For the manual control phase, subjects were instructed to manually control the same system they had seen during the automated phase. Eye movement parameters were recorded by the Eye Gaze Analysis System. After a calibration phase (15 s), they were presented with the four scenarios, each with a duration of five minutes. The four scenarios were presented in a fixed order for every subject, beginning with the easiest (Scenario 1) and finishing with the most complex (Scenario 4). After each scenario, subjects evaluated their difficulty and also reported the strategies they identified during the automated phase. Test subjects: The experiment was conducted with a sample of 90 applicants for the DFS (Deutsche Flugsicherung GmbH) and DLH (Deutsche Lufthansa AG) ranging in age from 17 to 26 years. 82% were male. Experiments were conducted in conjunction with the regular selection process at the German Aerospace Center’s Department of Space and Aviation Psychology. Applicants received 20 € for participating and were assured that their performance in the experiment would not affect their selection results. 18


3.1.3

Results

First analyses indicate that scenario one and four did not differentiate sufficiently between subjects. This is due to ceiling and floor effects. In contrast, scenario two and three show an optimal degree of difficulty. Therefore, the results of these two scenarios are reported here. The effects of complacency potential were analysed. To get an overview, the thirty subjects with the best monitoring behaviour were compared to the thirty with the worst. The groups were identified according to their relative fixation counts on relevant AOIs during all operating phases. Contrasts were calculated between the extreme groups. No effect was found for technology-related complacency but the groups differed in the scales conscientiousness (t(84)= 3.39, p