Does Shifting Between Conditionally and Partially Automated Driving ...

Does Shifting Between Conditionally and Partially Automated Driving Lead to a Loss of Mode Awareness? Anna Feldhütter1, Christoph Segler 1, Klaus Bengler 1 1

Chair of Ergonomics, Technical University of Munich, Boltzmannstr. 15 85748 Garching, Germany {anna.feldhuetter, christoph.segler, bengler}@tum.de

Abstract. In near future, several complex automation modes – like SAE-level 2 and 3 – may be employed in one vehicle. In order to investigate the relevance of mode awareness and mode errors in the context of vehicle automation, a study with 49 participants was conducted. In the experiment, the participants experienced two stages: one stage with alternating partially and conditionally automated driving and another stage with only partially automated driving. Mode awareness and the occurrence of mode errors were compared in the two stages in order to examine the effect of shifting between the two modes. Additionally, the influence of a cognitive-auditive and a visual-motoric non-driving-related task as well as an adapted HMI was examined. Results showed that depending on the type of the non-driving-related task shifting between partially and conditionally automated driving leads to a loss of mode awareness and results in more mode errors compared to having only one automation mode. An enhancement of mode awareness by the suggested adapted HMI could not be found. Keywords: Conditional Automation, · Partial Automation, · Mode Awareness, · Mode Errors, · Human-Machine Interaction, · Human-Machine Interaction Concept

1

Introduction

The technical progress in vehicle automation promises many advantages, such as an enhancement of road safety and efficiency as well as an increase in drivers’ comfort by reducing their workload. However, the larger number of assistance systems and automation modes and the subsequent rise in the complexity of the technical system also entails risks. It is well known that the availability of multiple assistance and automation modes in one technical system may lead to a lack of mode awareness [1]. Especially if the driver is not aware of the current automation mode and its functional structure or assumes a different mode is currently activated, incorrect driver behavior can occur, which is referred to as mode error. This can in turn lead to dangerous situations or even accidents. In the aviation sector, a lot of research has been carried out, as several incidents, among them severe accidents, occurred with an increase in the number of modes. They were caused by a lack of mode awareness or by mode errors [2] (cited after Dorn-

heim [3]). In this context, Sarter and Woods [2, 4] have identified the following important factors that encourage insufficient mode awareness/mode confusion: insufficient/incorrect monitoring of the automation due to absent/poor feedback, lack of understanding of the functional structure of the automation, high level of automation and shifting between automation modes as well as a high degree of coupling between several automation modes. Especially in time-critical situations, insufficient mode awareness may result in mode errors, as the pilots recognize the error belatedly or not at all due to a high workload [5]. Several approaches were developed to either enhance mode awareness (e.g. adopting the interface concept dependent on the automation mode or implementing acoustic and haptic feedback for shifting between different modes [6, 7]) or to prevent the occurrence of mode errors [1]. In contrast to the aviation sector, almost no research has been carried out that investigates mode awareness and error in the automotive automation context. In particular, partially automated driving (PAD, level 2, according to SAE International [8]) and conditionally automated driving (CAD, level 3) are similar for the user regarding the obvious functionality, however, the driver’s role differs significantly: In partially automated vehicles the driver still needs to monitor the system, whereas in conditionally automated vehicles the drivers can engage themselves in non-driving-related tasks [8] (NDRT). It is, therefore, important in terms of controllability to investigate whether shifting between these two modes leads to a lack of mode awareness and to the occurrence of mode errors. For this purpose, a driving simulator experiment in a fix-based driving simulator was conducted. Further, an adapted human-machine interface (HMI) was evaluated, which aimed at enhancing mode awareness and thus preventing mode errors.

2

Methods and Apparatus

2.1

Simulation and Vehicle Automation

The experiment was conducted in a fix-based driving simulator. The simulator was equipped with an active steering wheel, pedalry, a projector for displaying the driving environment and a touchscreen for performing the NDRT. The driving simulation and the different automation modes were programmed using the Unity 3D 5.4.0 game engine (Unity Technologies) and the Visual Studio 2015 programming environment (Microsoft Corporation). In order to investigate mode awareness in two different automation modes, the Automated Lane Change Test (ALCT) of Spießl [9] was adapted. The ALCT was derived from the standardized Lane Change Test (LCT), which was developed for measuring drivers’ distraction by a NDRT in the context of manual driving [10]. The LCT consists of a simulated drive on a three-lane highway. At regular instances, road signs appear which specify a certain lane. The driver is instructed to steer the simulated vehicle onto this required lane (see Fig. 1). In the ALCT, the automated system takes over longitudinal and lateral control, scans the signs and changes lanes if necessary. In the experiment, the automation maintained a constant velocity of 60 km/h. The distances between the signs ranged between 100 m and 140 m. Even though the signs were visible very early on, the texture did not become clear before the defined distance of 60 m was reached in order to ensure that the participants had the same reaction time for each sign.

At the same time when the texture became clear, the automation initiated the according maneuver. In the CAD mode, the system carried out each maneuver as indicated by the signs. In the PAD mode, the system committed errors at a defined number of signs, meaning that the automation selected the wrong lane. If the participants recognized the automation error, they could switch off the automation by oversteering the system and correct the error. After reaching the correct lane, the participants were instructed to switch on the automation again. As a system reliability of less than 90% leads to a decrease in trust [11] (cited after [9]), the error rate in the partial automation mode was set to 10%. The position of the automation errors was randomized across all participants. The number of signs and the track length were the same for all participants.

Fig. 1. Automated Lane Change Test and head-up display in the PAD mode.

2.2

Non-driving-related tasks

It was assumed that an adaption of the drivers’ reaction to the two automation modes depends on the type of NDRT. Therefore, two different types of NDRT were tested in the two automation modes: a visual-motoric and an auditive-cognitive task. Both tasks were designed to be interruptible in order to give the driver the possibility to monitor the system if necessary. Further, a baseline for testing the two HMI concepts with no additional task was conducted. The visual-motoric task consisted of a destination-entering task (DET) on a simulated navigation system screen (see Fig. 2). The destination included a city and a street, which needed to be entered consecutively. Each destination had to be entered only once. The task was performed on a touch screen, which was located in the middle console. The application of this visual-motoric task aimed at examining the influence of this task type on mode awareness. In addition, the effect of a supporting adaptive human-machine interface implemented on the same screen as the task was investigated. The auditive-cognitive task consisted of the Twenty Questions Task (TQT), inspired by the study of Merat, Jameson, Lai and Carsten [12]. The participants were encouraged to guess an item from a predefined category by asking a maximum of twenty questions, the response to each question always being ‘no’ or ‘yes’. The question-answer style of this task is similar, for instance, to a complex telephone conversation, as it utilizes cognitive resources but can be properly replicated. In addition to the effect of this type of

NDRT on mode awareness, the influence of an adaptive HMI on mode awareness was also investigated.

Fig. 2. The destination-entering task in German language. The orange-labeled text displays the required city (Ort: Muenchen) and street (Strasse: Knorrstrasse)

2.3

Procedure

The procedure of this study is shown in Fig. 3. Before the experiment, the participants were informed about the ALCT and the capacity of both automation modes, PAD and CAD, as well as the corresponding responsibility of the driver according to SAE International [8]. Subsequently, the participants completed a questionnaire, which asked for demographics and experience with automated driving and assistance systems. To examine the effect of different HMIs (initial and adaptive), the participants were subdivided into two equally large groups (between-subject design). Before beginning the experimental drive, the participants familiarized themselves with the driving simulator, including the two automation modes (PAD and CAD), the ALCT and the two NDRTs. After completion of the familiarization drive, the experiment started. The experimental drive was subdivided into two stages: the two-mode stage and the one-mode stage (see Fig. 3 and Fig. 4). Each participant completed both stages, counterbalancing the order of the stages between the participants in order to avoid sequence effects. In the two-mode stage, the automation mode changed between CAD and PAD several times (see Fig. 4). After driving manually onto the three-lane highway and activating the conditional automation, the participants started with the first NDRT. After driving 1.5 km, the system initiated a mode change to PAD, which needed to be confirmed by the participants. Afterwards, the participants drove 2.0 km in partially automated mode, followed by a further mode change to CAD. After completing both mode changes, the next NDRT started. This procedure was repeated twice for each NDRT and once without any NDRT (baseline), permuting the order of both NRDTs and the baseline, which led to a total stage duration of 21 minutes. During the 15-minute one-mode stage no mode changes occurred. As in the two-mode stage, the participants completed each NDRT twice and the baseline once. In order to examine the effect of mode transitions, mode awareness was compared in the one-mode stage and the PAD periods of the twomode stage (within-subject design)

Fig. 3. Study procedure

Fig. 4. Procedure of the two-mode and one-mode stage. The two different non-driving-related tasks (destination-entering task and Twenty Question Task) were performed twice and the baseline without any task once per stage. The order of the tasks and the baseline was permuted and the same task was not performed consecutively.

After each stage, the participants rated their subjective mode awareness, the distraction by the NDRT and the performance of either monitoring the automation (in the PAD mode) or of the NDRT. The analysis of this rating is not part of this paper. 2.4

Human-Machine Interface

The HMI consisted of two parts: the head-up display (see Fig. 1), which was available to all participants during the entire experiment, and the adaptive HMI, which was

only available to the adaptive HMI group. Fig. 5 shows the symbols used for indicating the automation modes in the present study. In the one-mode stage, only two symbols were necessary, as only partial automation and manual driving (MD) was available. An activated partial automation mode was indicated by the green PAD symbol. In the case of a change to manual driving, the white MD symbol was displayed. In the two-mode stage, the blue symbol for CAD and the symbols for mode transition were added. If CAD was available, in addition to the green PAD symbol, the text Autopilot available was displayed and a one-time acoustic signal sounded. After confirmation with a specific button on the steering wheel, the CAD mode was activated (blue CAD symbol was displayed). In the case of a transition from CAD to PAD, the CAD symbol flashed red, and the acoustic signal gonged continuously until the participants confirmed the transition with the button on the steering wheel. All transitions were triggered by the system.

Fig. 5. Symbols used for indicating the automation modes

Fig. 6. Adaptive Human-Machine Interface (left) and initial Human-Machine Interface (right)

While the initial HMI only consisted of acoustic feedback and the above-mentioned information in the head-up display, the adaptive HMI additionally featured the same information on the same screen where the DET was presented (see Fig. 6). During PAD and MF, the afore-mentioned symbols were displayed in the left upper corner of the screen. Only in CAD, the blue CAD symbol was supplemented with a blue declining bar, which showed the remaining time in this mode. This implementation was based on the approach of Monk [6] and Sellen et al. [7], as different symbols were used depending on the automation mode. 2.5

Dependent and independent variables

In order to assess mode awareness, the monitoring performance during PAD was measured by the dependent variables (DV) reaction time to an automation error (time between the appearance of the sign texture and the intervention by the participant), the proportion of undetected automation errors and the proportion of false interventions. These DVs can be analyzed for all three conditions of NDRT (TQT, DET and no additional task). Further, mode awareness was evaluated by means of the task performance in entering the destination, which was assessed by the mean hitting rate for entering the city or street names (hits per second while entering one word), the mean pause between entering a city and a street and the mean pause between entering two destinations. These metrics indicate the attention directed toward the driving environment during PAD, as the NDRT needed to be interrupted for this purpose. In addition to measuring the task performance of the DET, eye-tracking was used for this NDRT to evaluate the distraction from the driving environment and, hence, to assess mode awareness. Therefore, the gaze-shifting proportion to either the driving surroundings or the NDRT, the mean gaze duration of one gaze towards either the driving environment or the NDRT and the mean gaze-shifting rate between the driving environment and the NDRT were measured. The effect of two independent variables (IV) on mode awareness was tested: the HMI concept and mode transition. In order to examine the effect of different HMI concepts, the above-mentioned DVs were compared for the two groups, where one group tested the initial HMI concept and the other group the adaptive one. For assessing the effect of mode transition on mode awareness, the DVs were compared for the one-mode stage (no mode transitions) and the PAD period of the two-mode stage (with mode transitions).

3

Results

A total of 50 volunteers participated in the experiment. One participant had to be excluded from data analysis due to technical issues during the experiment. Of the remaining 49 participants, 15 were female (31%), and the mean age of all participants was 34.65 years (SD = 9.92). Eleven participants (23%) had never experienced automated driving or driver assistance systems, whereas 22 participants (45%) use these systems occasionally and 16 (33%) regularly. To evaluate the effect of mode transition and HMI on mode awareness, a 2 (HMI, between-subject design) x 2 (stage, within-subject design) analysis of variance (ANOVA) was conducted for each NDRT condition (TQT, DET, none). In the case of a significantly differing variance between two groups, Welch’s T-Test was carried out. Throughout the entire analysis, a significance level of .05 was assumed. Due to different technical issues, further participants had to be excluded from data analysis. The number of participants varied for each NDRT condition. Results of the ANOVA showed that there was no significant effect of the HMI concept on mode awareness considering all NDRT conditions. Fig. 7 shows the reaction times to an automation error in the two stages (one-mode and two-mode), depending on the NDRT. Analyzing the no-task (N = 42) and the DET (N = 47) condition, the ANOVA revealed no significant effect of mode transitions, considering the reaction time to an automation error and the proportion of undetected automation errors. In the no-task condition there was no false intervention, so this metric could not be analyzed. In the DET condition, two participants intervened erroneously while the PAD mode was activated (one in the one-mode stage and one in the two-mode stage). Thus, this metric could also not be analyzed for the DET condition.

Fig. 7. Average reaction time to an automation error depending on the non-driving-related task and the stage

Considering the DET performance (hitting rate, pause between entering a city and a street and pause between entering two destinations), there was also no significant effect

of mode transitions. Processing the eye-tracking data resulted in a remaining sample size of 29 participants, where both the number of participants in the two HMI groups and the order of the two stages were still balanced. The ANOVA showed that regarding the gaze-shifting proportion to either the driving environment or the DET, no significant difference between the stages emerged. Welch’s Test (ANOVA: F(1, 27) = 34.83, p < .001, r = .60), Levene Test: p = .001) revealed a significant effect of the mode transition on the mean gaze duration towards the DET (t(39.22) = 5.62, p < .001, r = .67). Hence, the participants on average looked 3.89 sec longer at the screen of the DET in the two-mode stage than in the one-mode stage. Consistently, the ANOVA also showed that the mean gaze-shifting rate between the driving environment and the DET was significantly less in the two-mode stage (F(1, 27) = 9.39, p = .005; r = .17). Evaluating mode awareness for the TQT (N = 48) results of the ANOVA revealed that the reaction time to an automation error was significantly higher in the stage with two automation modes (t = 0.31 sec) than in the stage with only one mode (F(1, 46) = 7.07, p = .011, r = .18, see Fig. 7). The analysis of undetected automation errors did not show any differences between the stages. The number of false interventions could not be analyzed, as only one participant corrected the automation erroneously (in the one-mode stage). Throughout the entire analysis, no significant interaction effects were found.

Fig. 8. Mean gaze duration in partially automated driving towards the destination-entering task depending on the two different stages.

4

Discussion

4.1

Effect of mode transition on mode awareness

With regard to the effect of mode transition on mode awareness, the results differed according to the type of NDRT. With no additional task, based on the variables investigated, there were no differences in mode awareness between the stages, with only one or two alternating automation modes (CAD and PAD).

When the DET (visual-motoric task) was performed, a significant effect of mode transition on the visual monitoring performance was found: Changing the modes from CAD to PAD and vice versa led to significantly stronger distraction from the driving environment, as the participants averted their eyes from the road for a significantly longer period of time, compared to when only PAD was activated. Consequently, this effect could also be found in the gaze shifting rate, as in the two-mode stage the participants shifted their gaze back to the road significantly less often. It is assumed that the participants simply did not change their gaze behavior when shifting from CAD to PAD due to a lack of mode awareness. However, the different gaze behavior in the two stages did not lead to differences in reaction times in the case of an automation error. This is plausible, though, as looking away from the driving environment would probably be more likely to result in missing signs and therefore in an increase of undetected automation errors. This assumption cannot be definitely confirmed, however. A possible reason may be that the number of automation errors per participant was relatively low, with an error rate of 10 %. Additionally, even if the participants missed some signs, the probability that an automation error occurs at one of the missed signs was relatively low. Hence, the sample size available for the analysis was relatively small. Even though no change in reactions to automation errors could be found, the decrease in visual monitoring of the partly automated system and in checking the driving environment due to a lack of mode awareness represents a safety-relevant issue, as the driver would detect suddenly emerging automation errors late or not at all. When the TQT (auditive-cognitive task) was performed, a significant increase in reaction times to automation errors was detected during the stage with the two alternating automation modes. The number of undetected automation errors could not confirm this decrease in performance. However, here, too, the sample size was relatively small. There are already several studies that have found a deterioration of driving performance caused by auditive-cognitive secondary tasks such as talking on the phone [13, 14] or using the onboard voice assistant [15] due to limited mental resources. As the TQT always started during CAD, the participants were able to use the majority of their attentional resources for performing this task. With transition to PAD, the main task shifted to monitoring the automation. Obviously, in the two-mode stage the participants did not adapt their behavior by diverting enough attentional resources back to the monitoring task, as in this stage longer reaction times to automation errors occurred. It is assumed that the participants were not aware of the current automation mode PAD and did therefore not adequately adapt their behavior. However, it is also conceivable that the participants did not remember their responsibility to monitor the system in PAD. Further, it cannot be ruled out that, due to the safe driving simulator environment, the participants were more willing to take risks and, hence, concentrated more strongly on the NDRT. 4.2

Effect of different HMI concepts on mode awareness

The adaptive HMI did not lead to an improvement of mode awareness in comparison to the initial HMI. An interview after the experimental drive revealed that only nine participants recognized the adaptive HMI, which might be one reason why no differences were found. A further reason for the lack of improvement may be that the modi-

fication of the HMI was not conspicuous enough. An extended application of the automation symbol colors (green, blue and red) in the vehicle interior, using for instance LED concepts, would be conceivable. Apart from improving mode awareness, a reasonable further approach could be, according to Sarter and Woods [1], to prevent the occurrence of mode errors. Results of this study showed that it seems to be crucial for mode awareness which type of NDRT is performed by the driver. It might be expedient, as a further alternative approach, to interrupt these specific tasks, depending on the duration of engagement in the tasks or on the driver state regarding mode awareness. However, this could lead to low acceptance among users.

5

Summary

Results of this driving simulator study showed that shifting between the PAD and CAD automation modes affected mode awareness for two different types of NDRT. While performing a cognitive-auditive NDRT (Twenty Question Task), the alternating mode transitions led to longer reaction times when an automation error occurred. While performing a visual-motoric NDRT (destination-entering task), the mode alternating transitions resulted in less monitoring and less checking of the partially automated system, as the time spent looking at the driving environment and the automation decreased significantly. Without an additional task, no effect on mode awareness was found when shifting between the modes. It can therefore be stated that depending on the task type, shifting between the PAD and CAD automation modes leads to a loss of mode awareness and results in more mode errors compared to having only one automation mode. The suggested adaptive HMI concept, which was based on the proposals of Monk [6] and Sellen et al. [7] and which was compared to the initial concept, did not improve mode awareness, considering both the two task types and the control group without an additional NDRT. This study identified that the increase in the number of automation modes and in system complexity can lead to a loss of mode awareness. In particular, when the PAD and CAD modes are employed in one vehicle, the occurrence of mode errors is a safetycritical issue, as the responsibilities of the driver are significantly different. It is therefore important to develop a suitable HMI which supports the driver optimally in being aware of the current automation mode and to further examine under which circumstances mode errors may occur. Further studies should have a closer look on the reported effects.

References 1. Sarter, N.B., Woods, D.D.: How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control. hum factors 37, 5–19 (1995) 2. Sarter, N.B.: Investigating Mode Errors on Automated Flight Decks. Illustrating the ProblemDriven, Cumulative, and Interdisciplinary Nature of Human Factors Research. Human Factors: The Journal of the Human Factors and Ergonomics Society 50, 506–510 (2008) 3. Dornheim, M.A.: Dramatic incidents highlight mode problems in cockpits. Aviation Week and Space Technology,, 6–8 (1995)

4. Sarter, N.B., Woods, D.D.: Pilot Interaction With Cockpit Automation II. An Experimental Study of Pilots' Model and Awareness of the Flight Management System. The International Journal of Aviation Psychology 4, 1–28 (1994) 5. Sarter, N.B., Woods, D.D.: Mode Error in Supervisory Control of Automated Systems. Human Factors: The Journal of the Human Factors and Ergonomics Society 36, 26–29 (1992) 6. Monk, A.: Mode errors. A user-centred analysis and some preventative measures using keying-contingent sound. International Journal of Man-Machine Studies 24, 313–327 (1986) 7. Sellen, A., Kurtenbach, G., Buxton, W.: The Prevention of Mode Errors Through Sensory Feedback. Human-Comp. Interaction 7, 141–164 (1992) 8. SAE International: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems (2014) 9. Spießl, W.: Assessment and Support of Error Recognition in Automated Driving. München (2011) 10.ISO 26022: Road vehicles — Ergonomic aspects of transport information and control systems — Simulated lane change test to assess in-vehicle secondary task demand (2010) 11.Buld, S., Krüger, H.-P.: Wirkungen von Assistenz und Automation auf Fahrerzustand und Fahrsicherheit. Projekt: EMPHASIS, Effort-Management und Performance-Handling in sicherheitsrelevanten Situationen ; Abschlussbericht. Würzburg (2002) 12.Merat, N., Jamson, A.H., Lai, F.C.H., Carsten, O.: Highly Automated Driving, Secondary Task Performance, and Driver State. Human Factors: The Journal of the Human Factors and Ergonomics Society 54, 762–771 (2012) 13.Caird, J.K., Willness, C.R., Steel, P., Scialfa, C.: A meta-analysis of the effects of cell phones on driver performance. Accident Analysis & Prevention 40, 1282–1293 (2008) 14.Strayer, D.L., Johnston, W.A.: Driven to Distraction: Dual-Task Studies of Simulated Driving and Conversing on a Cellular Telephone. Psychological Science 12, 462–466 (2001) 15.Strayer, D.L., Cooper, J.M., Turrill, J., Coleman, J.R., Medeiros-Ward, N., Biondi, F.: Measuring Cognitive Distraction in the Automobile. AAA Foundation for Traffic Safety, 53ff (2013)