Interaction

Technische Universität München – Institute of Ergonomics

Reading the Driver: Visual Workload Assessment in Highly Automated Driving Scenarios

Markus Zimmermann Iris M. Rothkirch Klaus J. Bengler

Technische Universität München – Institute of Ergonomics

Outline 1. Automation & Mental Workload 2. Visual Metrics

3. Experiment 4. Results

5. Next steps 21st July 2014

Reading the Driver

2

Technische Universität München – Institute of Ergonomics

Vision How to measure the user state in real time in order to facilitate a cooperation & adaptive interaction? Interface

Environment

System Automated

Interaction

Obstacles, Cars, …

Authority, Timing, …

Interface Workload

21st July 2014

Reading the Driver

User Demand Mode Awareness …

3

Technische Universität München – Institute of Ergonomics

Automation • Automation is known to take the

= + • Highly Automated (GASSER, 2012) • Longitudinal and Lateral control done by the Machine

• Task lane-change: Consent • No supervision necessary 21st July 2014

user out of the control loop:

passive monitoring role (NORMAN, 1990) – slower in detection

– loss of mode awareness – deskilling

• Effect on mental workload: – overload – underload

Gasser, T. M. (2012). Rechtsfolgen zunehmender Fahrzeugautomatisierung: Gemeinsamer Schlussbericht der Projektgruppe. Berichte der Bundesanstalt für Straßenwesen, (F 83). Norman, D. A. (1990). The 'Problem' with Automation: Inappropriate Feedback and Interaction, not 'Over-Automation'. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 327(1241), 585–593. doi:10.1098/rstb.1990.0101

4

Technische Universität München – Institute of Ergonomics

Example: Driver Assistance !!!

 21st July 2014



Reading the Driver

Page 5

Technische Universität München – Institute of Ergonomics

Mental Workload (MWL) 𝑀𝑒𝑛𝑡𝑎𝑙 𝑤𝑜𝑟𝑘𝑙𝑜𝑎𝑑 =

•

𝑛𝑒𝑒𝑑𝑒𝑑 𝑟𝑒𝑠𝑜𝑢𝑟𝑐𝑒𝑠 𝑎𝑣𝑎𝑖𝑙𝑎𝑏𝑙𝑒 𝑟𝑒𝑠𝑜𝑢𝑟𝑐𝑒𝑠

MWL can predict the drivers’ future performance

(PARASURAMAN ET AL., 2008) Multidimensional interaction of task and system demands

•

•

Internal and external factors

•

Relation MWL/performance can be transferred to the Yerkes Dodson law (YOUNG AND STANTON, 2002)

influence MWL (WAARD, 1996) •

Overload and underload are linked to

negative consequences  Optimization is crucial (STANTON AND YOUNG, 2000)

21st July 2014

Parasuraman, R., Sheridan, T. B., and Wickens, C. D. (2008). Situation Awareness , Mental Workload , and Trust in Automation : Viable , Empirically Supported Cognitive Engineering Constructs. Journal of Cognitive Engineering and Desicion Making, 2(2):140–160. Stanton, N. A., & Young, M. S. (2000). A proposed psychological model of driving automation. Theoretical Issues in Ergonomics Science, 1(4), 315–331. doi:10.1080/14639220052399131 Waard, D. D. (1996). The Measurement of Drivers Mental Workload. Technical report, University of Groningen. Young, M. S. and Stanton, N. A. (2002a). Attention and automation: New perspectives on mental underload and performance. Technical report, Brunel University.

6

Technische Universität München – Institute of Ergonomics (AHLSTROM & FRIEDMAN-BERG, 2006; STEIN, 1992)

(CALLAN, 1998; VAN ORDEN ET AL., 2001).

Long fixations frequency 

Fixations

Saccade frequency 

Saccades

(ANGUS ET AL, 2004; VAN ORDEN ET AL., 2001) (AHLSTROM & FRIEDMAN-BERG, 2006; STEIN, 1992)

Fixation rate 

Saccade duration 

(GREEF ET AL, 2009)

Fixation duration 

(WICKENS, 2000; VAN ORDEN ET AL., 2001; JESSEE 2010).

Saccade extent 

(JESSEE, 2010)

Fixation duration variability  (MAY ET AL., 1986) (CAIN, 2007; WIERWILLE & EGGEMEIER, 1993; ANGUS ET AL., 2004; VAN ORDEN, 2001; AHLSTROM, ET AL., 2006)

Saccade latency 

Mental workload 

Blink duration 

(ANGUS ET AL., 2004; GREEF ET AL., 2009; CAIN, 2007; AHLSTROM, ET AL., 2006)

(ANGUS ET AL., 2004; JESSEE, 2010; WIERWILLE & EGGEMEIER, 1993; VAN ORDEN, 2001)

Pupil diameter 

Blink rate 

(MARSHALL, 2002; MARSHALL 2005)

(CAIN, 2007; NEUMANN & LIPP, 2002)

Blink latency 

Glance frequency 

ICA

Pupil

(ROTHKIRCH, 2013)

Blinks 21st July 2014

Percent time on AOI  (IQBAL & BAILEY, 2004; JUST & CARPENTER, 1976).

Glances

Total glance duration  (ROTHKIRCH, 2013)

Literature available on http://www.d3cos.eu/dp/#STATE-WL-VIS

7

Technische Universität München – Institute of Ergonomics

Hypotheses

Driver out of the loop

Under load

?

Driver mentally challenged

Over load

High MWL

Prospective MWL Low MWL

Increase of MWL with increasing

U-shaped relation between mental

task demand through auditory

workload and performance/

prompt-verbal response n-back task

prospective mental workload

MWL optimisation is crucial to maintain effective task performance 21st July 2014

Reading the Driver

8

Technische Universität München – Institute of Ergonomics

Procedure •

3-7-5-9-4

Within subjects design

•

Taskload

…-…-3-7-5

Randomized conditions

•

n = 25

average age = 22,4

UI Design

Questionnaires

Activation

Cooperative Lanechange Task

21st July 2014

Reading the Driver

9

Technische Universität München – Institute of Ergonomics

Experimental design Evaluation of

Conditions

Right scenario

with CoS

without CoS

Timing short timing

with CoS

Efficiency of design

long timing

Left scenario

UI Quality

long timing

No nback

3-back

without CoS

Gaze patterns

with CoS

short timing

long timing

long timing

no WL

no WL

medium WL

high WL

no WL

medium WL

high WL

no WL

no WL

no WL

1

2

3

4

5

6

7

8

9

10

21st July 2014

Workload

1-back

Reading the Driver

10

Technische Universität München – Institute of Ergonomics

Equipment / Measures

Eyetracker FaceLAB 5 • Fixations Static driving simulator

• Saccades

• Total scenario duration

• Blinks

• Steering response latency

• Pupil size

21st July 2014

Reading the Driver

11

Technische Universität München – Institute of Ergonomics

Derived Measures: Glances

Glances on static areas of interest

Glances on dynamic areas of interest

•

Scenery

•

Look ahead

•

Side windows

•

Left motorway barrier

•

Left, center & right mirrors

•

Truck

•

Instruments

•

Cars left

21st July 2014

Reading the Driver

12

Technische Universität München – Institute of Ergonomics

Gaze Distributions

21st July 2014

Reading the Driver

13

Technische Universität München – Institute of Ergonomics

Workload: Complexity For an increasing n-back task, increasing workload is measurable using eye metrics like

*** ***

8

***

*** ***

Pupil diameter of right eye [mm]

blinks, fixations, saccades, glances, gaze, pupil

5,84

6

5,12

4,43 4

2

no n-back

1-back

3-back

Workload condition Error indicator: ±SD. ANOVA with p < 0.001 (***)



Significant for metrics pupil diameter and blink duration

21st July 2014

Reading the Driver

14

Technische Universität München – Institute of Ergonomics

Workload: Performance U-shaped relation between mental workload and performance/ prospective mental workload *

Amount of saccades

*

25 20 15 10

12,7 9,6

5

7,2

0 no workload

medium high workload workload Workload condition

Error indicator: ±SD. ANOVA with p < 0.05 (*).

 Shorter steering response latency (Significant quadratic trend; F(1,19,95%)=7.99; p < .05; η2𝑝 = .30) and total duration of scenario for medium workload scenario. 21st July 2014

 Significantly shorter/less glances to dynamic AOIs, smaller pupil diameter, less saccades, … for medium workload scenario

Reading the Driver

15

Technische Universität München – Institute of Ergonomics

conscious

Gaze extent ahead

left

(TOBII, 2013)

(TOBII, 2013)

< 17.19°

> 17.19°

No WL

Medium WL

High WL

No WL

Medium WL

High WL

SD for right eye [°]

7.80

7.44

7.61

8.11

8.84

9.02

SD for left eye [°]

7.72

7.62

7.71

8.17

9.12

9.21

(TOBII, 2013)

21st July 2014

subconscious

Reading the Driver

16

Technische Universität München – Institute of Ergonomics

Conclusion Eye Metrics

Realtime Processing

Realtime Adaptation

Proof of concept for

Situative & individual

Task optimization

cooperative lane-

differences of visual

(medium workload

change scenario

metrics

level) is necessary

•

•

•

Metrics for inferring workload set up,

•

Real time inference in

simulation

optimal assignment of control between a human and the system

Evidence for u-shaped relation between performance and

•

Real time analysis

mental workload 21st July 2014

Reading the Driver

17

Technische Universität München – Institute of Ergonomics

21st July 2014

Reading the Driver

18