Jun 3, 2003 ... EATMP ASA Programme. 1. EUROCONTROL a o. C R Conflict. Conflict.
Resolution. Resolution. Assistant. CORA. COnflict Resolution Assistant.
CoRa
Conflict Conflict Resolution Assistant
CORA COnflict Resolution Assistant Human Factors Lab experiments NEXTOR-FAA Conference June 3, 2003
EATMP ASA Programme 1
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Aims of the CORA HF Experiments
Evaluate different 4 User-driven 4 Automatic 4 Collaborative
‘design philosophies’ for CORA: (psychological - comparison) (technological - left-over) (cybernetic - complementary)
Evaluate different conflict detection (resolution) timelines: 4 5 minutes prior to conflict 4 10 minutes prior to conflict 4 15 minutes prior to conflict Determine whether resolutions should be presented in a: 4 Fixed order (by type) 4 Ranked order (by quality-index)
EATMP ASA Programme 2
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Conflict detection
User-driven philosophy Display indication of Conflict
Request resolutions ?
yes Calculates and ranks Resolutions
Display ranked resolutions
no Find own resolution
no
Accept one resolution ? yes
Implement resolution
EATMP ASA Programme 3
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Conflict detection
Automatic philosophy
Calculates and ranks resolutions
Display best-ranked resolution
View related conflict ?
no
yes Display conflict
Reject resolution ?
yes
Find own resolution
No (Automatic on implementation time reached)
Implement resolution EATMP ASA Programme 4
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Conflict detection
Calculates and ranks resolutions
Display indication of Conflict (+ first conflict to act on)
Display indication of resolution availability
View resolutions ?
Collaborative philosophy
yes
Display ranked resolutions
no
Find own resolution
no
Accept one resolution ? yes
Implement resolution EATMP ASA Programme 5
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
CORA system Calculation
Summary of the philosophies CORA philosophies User-driven
Automatic
Collaborative
On request
Automatic
Automatic
Resolution display
On request
Automatic
Number of resolutions
Several
One
(5 best-ranked by type)
(overall best-ranked resolution)
Conflict Indication
Automatic
No
Conflict data
On request
On request
EATMP ASA Programme 6
Automatic (indication of availability) +
On Request (resolutions)
Several (5 best-ranked by type)
Automatic + Indication of the first conflict to act on
On request
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
User-driven HMI
EATMP ASA Programme 7
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Automatic HMI
EATMP ASA Programme 8
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Collaborative HMI
EATMP ASA Programme 9
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Schedule
25 to 29 November 2002 10 Controllers (6 active), male, mean age 42.2, mean experience 18.7 yrs. 2 Austria, 1 Finland, 3 Great Britain, 1 Germany, 1 Hungary, 1 Italy, 1 Maastricht (Experience: All ACC and one Departure/Approach) 08.30 – 09.30
09.30 – 10.15 10.15 – 10.30 10.30 – 12:00 12.00 – 13.00 13.00 – 14.30 14.30 – 14.45 14.45 – 15.30 15.30 – 16.30 16.30 EATMP ASA Programme 10
Welcome and Overview Presentation of the simulation environment and the goals of the experiments Training Runs** Coffee Break Runs Lunch Runs Coffee Break Runs Questionnaire + Debriefing Departure
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Two Factorial Designs
Philosophy
User-driven
Collaborative
Time horizon
5 Minutes 10 Minutes 15 Minutes
5 Minutes 10 Minutes 15 Minutes
Order
Fixed order Ranked
Exercises
Fixed order Ranked
Fixed order Ranked
1
EATMP ASA Programme 11
2
3
Fixed order Ranked
4
5
6
Automatic
5 Minutes 10 Minutes 15 Minutes
Fixed order Ranked Fixed order Ranked
7
8
9
10
11
12
13
14
15
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Evaluation
Questionnaires 4 Post-exercise questionnaire (21 questions)
•Human-Automation Co-operation (Halden Co-operation scale) •Mental workload (NASA TXL revised) •Situation Awareness (SAHSA_Q revised) 4 End-of day questionnaire
•Design philosophy •Resolution timeline •Sorting •General questions
Debriefing: 4 End-of-day debriefing
•Based on observations and open questions •After the end-of-day questionnaire
EATMP ASA Programme 12
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
•
Experimental Design
Route Structure: – based on sectors from Reims FIR, France and Maastricht UAC.
•
Traffic: – based on 2 x 2.5 hour periods from 10 July 1998 (baseline). – Air traffic scenarios prearranged (Controllers unable to interact). – Each scenario presented one conflict.
•
Factorial Designs (two partly overlapping): – 3 x 2 x 2 (time/design/sorting) within subject. – 3 x 3 (time/design) within subject.
•
Incomplete counterbalancing: – Randomised - Controllers tested simultaneously experienced different scenarios and treatment levels.
EATMP ASA Programme 13
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
•
Experimental Design
Power analysis: – probability that the experimental design could produce statistically significant results (ok for large and typical effects).
•
Dependent variables: – checked to ensure no extreme violations about the assumptions about normal distribution and measurement level occurred.
•
Questionnaires: – Reliability and validity of subjective rating scales evaluated: • inter-item reliability testing (Cronbach’s Alpha) (All) • factor analysis (SA)
•
Hypotheses: – Non-directional – P-value approach for significance.
EATMP ASA Programme 14
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Timeline Human-Automation Co-operation Main effect of Treatment A (Timeline) F(2, 18)=8,2445, p=,00287 5,0
4,9
4,8 HA Co 4,7 op er ati 4,6 on 4,5
4,4
4,3 5 min
10 min
15 min
Timeline EATMP ASA Programme 15
EUROCONTROL
CoRa
Timeline Mental Workload
Conflict Conflict Resolution Assistant
Main effect of Treatment A (Timeline) F(2, 18)=2,9411, p=,07848 22 21 20 Me nta19 l W 18 or klo ad 17 16 15 14 5 min
10 min
15 min
Timeline EATMP ASA Programme 16
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Design Philosophy
No statistical evidence to suggest that the different design philosophies did not have an equal effect on Human-Automation Co-operation – Observational nature of the task – Controllers not able to tell the difference – Controllers’ preferences politically motivated – Experimenter bias
EATMP ASA Programme 17
EUROCONTROL
CoRa
Timeline & Design Philosophy Mental Workload
Conflict Conflict Resolution Assistant
Interaction between Treatment A and B Current effect: F(4, 36)=2,6085, p=,05167 28 26 24 Mental Workload
22 20 18 16 14 12 5 min
10 min
15 min
Automatic Collaborative User-driven
Timeline
EATMP ASA Programme 18
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Design Philosophy & Sorting Human-Automation Co-operation Interaction between Treatment B and C F(1, 9)=5,3315, p=,04631 5,00 4,95 4,90
H-A Cooperation
4,85 4,80 4,75 4,70 4,65 4,60 4,55 User-driven
Collaborative
Sorted by Type Sorted by Ranking
Design Philosophy
EATMP ASA Programme 19
EUROCONTROL
CoRa
Timeline & Sorting Principle Situation Awareness
Conflict Conflict Resolution Assistant
Interaction between Treatment A and C F(2, 18)=4,6003, p=,02434 5,3 5,2 5,1
Situation Awareness
5,0 4,9 4,8 4,7 4,6 4,5 5 min
10 min Timeline
EATMP ASA Programme 20
15 min
Sorting by Type Sorting by Ranking
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
EATMP ASA Programme 21
Timeline preference - forced & free choice -
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
EATMP ASA Programme 22
Timeline preference - by Design Philosophy -
EUROCONTROL
CoRa
Conflict Conflict Resolution Assistant
Results - End of Day Questionnaire / Debriefing
‘Design Philosophies’ for CORA: 4 User-driven 3/10 4 Collaborative 7/10 4 Automatic 0 (LOA/HMI - Controller bias - Researcher bias - Design) Situation Awareness - very difficult to measure
Timeline: 4 10 & 15 minutes - more support at Planning stage (7 - 20 minutes free choice)
Sorting: 4 By type for User-driven
4 By quality-index for Collaborative EATMP ASA Programme 23
EUROCONTROL
