Dynamic distribution of control via grip force sensitive ...

R-En-02

R-En-02

Dynamic distribution of control via grip force sensitive devices in cooperative guidance and control Marcel C. A. Baltzer1,2, Daniel López2, Martin Kienle3,4 and Frank Flemisch1,2 Fraunhofer-Institute for Communication, Information Processing and Ergonomics (FKIE-SE) Research Area of Balanced Human Systems Integration (IAW-SE), RWTH Aachen 3 SENSODRIVE GmbH 4 Institute of Ergonomics (LfE), Technische Universität München 1

2

Keywords: cooperative guidance and control, assistance, automation, grip force sensing, dynamic control distribution

Abstract More and more innovations in the assistance and automation for vehicles are now ready for the market. With increasingly complex technical functions the questions arise if those technical systems can be operated intuitively, comfortably, safely, reliably and energy eicient. A possible approach to improve the interaction between human and technical system is H(orse)-Mode where a driver and a co-automation work together (like a rider and a horse) to fulil the driving tasks. To change the amount of control between driver and automation, luid transitions were designed, where control lows from one actor to another. This design uses haptic interaction, where the requested amount of control can be determined by the force with which the driver grasps the haptic interaction device. To validate the concept, grip-force sensitive prototypes in form of a sidestick and a steering wheel were designed and will be elaborated in this article.

Cooperative Guidance and Control of highly automated vehicles Since the initial appearance of driver assistance systems in the 1970s and 1980s, now highly and fully automated vehicles become possible and will hopefully improve safety, comfort, energy eiciency and other system qualities. With the progressively blurred distinction between assistance systems and automated systems a discrimination of assistance and automation degrees became necessary (Gasser et al., 2012; NHTSA, 2013; SAE, 2014), and highly automated driving as an alternative or complement of fully automated driving became available. A bridge between the extremes of manual, partially automated, highly automated and fully automated operation exists: cooperative guidance and control (CGC) of highly automated systems. Cooperativity (the qualiication for cooperation) in guiding a highly automated system includes several aspects: First, autonomous abilities of the automation and the human are necessary (Flemisch et al., 2012). Furthermore an intuitive interaction with suicient outer and inner compatibility is relevant (Flemisch et al., 2008, inspired by Bubb, 1993). Inner compatibility can be achieved by compatible mental models (Norman, 1983). Outer compatibility can be realised by itting interfaces and the respective representations (Bubb, 1993; Flemisch et al., 2008). A clear distribution of responsibility and control that might be changed dynamically and an arbitration of conlicts can improve the cooperativity (Kelsch et al., 2006; Flemisch et al., 2014). In Cooperative Guidance and Control of highly automated vehicles a cooperative human machine system is necessary that can be represented by two individual self-contained systems, like a 89

R-En-02

(cognitive) automation (or co-system) and a human, who work together. A cognitive automation implies a certain amount of intelligence in order to enable cooperation and control distribution between the human and the machine (as shared Control, e.g. Griiths & Gillespie, 2004 or Abbink et al., 2012; cooperative control, e.g. Sheridan, 1992; pilot and co-pilot, e.g. Flemisch et al., 2003 or Holzmann et al., 2006). Implementations of cooperative guidance and control of highly automated vehicles are the H(orse)-Mode (Flemisch et al., 2003) and Conduct-by-Wire (Winner & Hakuli, 2006). The H-Mode concept describes the haptic-multimodal interaction and the performance of the driving task by a human and a highly automated vehicle (Flemisch et al., 2014). The H-Mode is based on the H(orse)-Metaphor, a design metaphor for the development of cooperative guidance of movement (Flemisch et al., 2003). The H-Metaphor takes the relation between a rider and a horse as a blueprint for human machine cooperation, where the horse is capable to move by itself with all necessary sensors but can, depending on the reins, be guided with variable amounts of autonomy by the human rider. In H-Mode dynamic changes of control responsibility and authority between cognitive automation and the human are therefore possible. The initiation of such a mode transition can be enabled in a naturalistic way by haptic interaction as a so called luid transition (Schutte & Flemisch, 2003). These luid transitions are already hinted in the rein changes of the H-metaphor (see Figure 1).

Figure 1: Fluid transition with grip force sensitive steering wheel (Altendorf et al. 2015)

Interaction Mediation The interaction architecture of the human machine system in cooperative guidance and control of highly automated vehicles uses an Interaction Mediator (see Figure 2) to communicate system states between driver and automation, generate situation dependent interaction pattern to convey states the human and the co-automation aim at and resolves conlicting targets with the concept of arbitration: System states include the current distribution of responsibility and control as well as change states like transitions from one mode to another. These states are represented by visual, acoustic and haptic interfaces. Interaction pattern represent a certain combination of interaction elements and image schemas (Hurtienne, 2011) in terms of type, intensity and time (see e.g. “Collision Avoidance”, Figure 3). Image schemas are sensorimotor and subconscious forms of knowledge representation (Hurtienne, 2011), like e.g. RESTRAINT – REMOVAL (Talmy, 1988) as a force image schema and e.g. STRONG – WEAK (Baldauf, 1997) as an attribute image schema. A human initiating a strong force on the haptic interaction device that leads to a removal of a restraint or blockage to support the awareness of taking over a larger amount of control can therefore be seen as an interaction pattern to make the driver directly feel that he has more control than before.

90

R-En-02

Human – Machine arbitration is a structured negotiation between a human and a co-automation thats intention is to reach a common unambiguous course of action in due course of time (e.g. Kelsch et al., 2006). Implementation of the grip force sensitive haptic devices A main focus of this article is the implementation of an interface that allows the dynamic change of the control distribution between a human and a machine in the context of cooperative guidance and control of highly automated vehicles. A possible dynamic control change can be represented by a luid transition (see Figure 1).

Figure 2: Cooperative guidance and control with the H-Mode (Altendorf et al., 2015, inspired by Flemisch et al., 2014 and Baltzer et al., 2014)

In a irst step, the distribution of control between the human and the automation was considered. It was necessary to understand that authority, responsibility, ability and control are interrelated (Flemisch et al., 2012). The human should not be responsible in a situation he is not able to control or better: the human machine system’s abilities should be improved by moving responsibility to an actor (e.g. the co-automation) who is more capable of controlling the situation and improve the overall system’s stability. In terms of safety a co-automation should then again have the authority to take responsibility in a situation it might be more suited to control (as an adaptive automation, see e.g Parasuraman & Riley, 1997; Moray et al., 2000) or to improve the situation by moving the human’s focus of attention and thereby improving the human’s abilities (for teaming, see e.g. Christofersen & Woods, 2002). The next step focused on the interaction pattern, meaning how respective control shifts should be communicated between driver and machine, what modalities should be included and how 91

R-En-02

conlicting targets in terms of control distribution should be resolved. In order to create an eicient arbitration process the human and the automation should be multimodally coupled most of the time. Highly automated driving in high degrees of automation involves a reduced focus of the human on the driving task. Therefore means needed to be addressed that allow a safe and intuitive transition from a high assistance and automation mode to a lower assistance and automation mode. Also in certain situations a decoupling of the respective partner can be useful or even necessary, e.g. a decoupling of the human during a collision hazard that the automation detected (see Figure 3), or, respectively, a decoupling of the co-automation might become necessary during a malfunction of the automation the human has detected. These changes of control distribution were addressed as mode transitions in the Mode Selection and Arbitration Unit (see Figure 2) of the Interaction Mediator. Finally, in order to change control distribution luidly via a haptic multimodal interface, it was necessary to ind out about the required abilities that the technical subsystem needs in order to safely give control back to the human who initiates a mode transition request and vice versa. For an intuitive change of control the human should be able to control the human machine system depending on the amount of control he or she brings into the human machine interface. This human machine interface again should be able to detect the impact of a human’s control for example by measuring grip forces and steering torques. To enable haptic interaction, kinaesthetic and tactile sensors are necessary in order to haptically measure what pressure, forces or torques the human inserts in the haptic interface. On the one hand many vendors supply active devices with suitable force sensors to measure kinaesthetic forces. On the other hand tactile measurement is a challenge since force sensors need to be applicable to diferent surfaces and usages of the haptic device.

Figure 3: Interaction pattern „Collision Avoidance“ (Baltzer et al. 2014)

92

R-En-02

Especially in the aviation domain active inceptors are applied for quite some time (e.g. Whalley et al., 2000; Griith & Gillespie, 2005; Müllhäuser et al. 2007), but also in the vehicle domain active sidesticks have been investigated (e.g. Eckstein, 2001). Consequently grip sensitive haptic devices were improved over the past years, beginning with a sidestick with FSR sensors that were able to measure grip force after a certain grip threshold was exceeded (developed at LfE, TUM). This design was improved with a capacitive sensor in order to measure light touch as well (developed at IAW, RWTH, see Figure 4, left). This concept was inally applied to a steering wheel (developed at IAW, RWTH, see Figure 4, right). The mix of touch and force measurement with capacitive sensors and force sensing resistors (FSR) works as an artiicial skin for the technical system. The grip-force sensitive haptical device enables a co automation to haptically interact with the human, e.g. that the automation “feels” that the driver wants more control or, if the driver only touches the haptic device, that the driver can feel a haptic schema the automation inserts in the haptic device. This can improve the cooperativeness between both actors and enable the luid transitions from one mode to another by simply releasing the grasp or by irmly grasping the steering wheel or the sidestick (see Figure 1).

Figure 4: Grip force sensitive interceptors: Left: Sidestick. Right: Steering wheel

Summary and Outlook This article described relevant aspects to enable dynamic distribution of control in cooperative guidance and control and presented prototypes of grip force sensitive devices in form of a sidestick and a steering wheel. First tests with the grip force sensitive sidestick and steering wheel showed promising results and will now be tested with participants in terms of usability, comfort and mode awareness. Also the interaction pattern for a luid transition will be further tested in order to prevent unintended mode transitions and resulting mode confusion and control deicit or control surplus. Another subject area for grip force sensitive devices is controllability. We will further explore in how far the grip force can be used as an indicator of take-over ability of the driver.

93

R-En-02

Literature Abbink, D. A., Mulder, M., & Boer, E. R. (2012) Haptic shared control: smoothly shifting control authority?. Cognition, Technology & Work, 14 (1), 19-28 Altendorf, E., Baltzer, M., Kienle, M., Meier, S., Weißgerber, T., Heesen, M., & Flemisch, F. (2015). H-Mode 2D. In: H. Winner, S. Hakuli, F. Lotz, C. Singer: Handbook of Driver Assistance Systems. Springer Baldauf, C. (1997). Metapher und Kognition. Grundlagen einer neuen Theorie der Alltagsmetapher. Frankfurt am Main: Peter Lang Baltzer, M., Altendorf, E., Meier, S., & Flemisch (2014). Mediating the interaction between human and automation during the arbitration processes in cooperative guidance and control of highly automated vehicles: Basic concept and irst study. In: F. Stanton, N.; Landry, S.; Bucchianico, G. D. & Vallicelli, A. (Eds.) 8 Road and Rail - Highly Automated Driving - Aspects of Driver Vehicle Interaction I Advances in Human Aspects of Transportation Part I, AHFE Conference Bubb, H., & Schmidtke, H. (Eds.) (1993) Systemergonomische Gestaltung Ergonomie, München: Carl Hanser., 3 Christofersen, K., & Woods, D. D. (2002) How to make Automated Systems Team Players. Salas, E. (Ed.): Advances in Human Performance and Cognitive Engineering Research Automation, 2, 1-12, Amsterdam: JAI Donges, E. (1982) Aspekte der Aktiven Sicherheit bei der Führung von Personenkraftwagen Automobil-Industrie, 27, 183-190 Eckstein, L. (2001) Entwicklung und Überprüfung eines Bedienkonzepts und von Algorithmen zum Fahren eines Kraftfahrzeugs mit aktiven Sidesticks. Fortschrittsberichte VDI-Reihe 12, Nr. 471. Flemisch, F. O., Adams, C. A., Conway, S. R., Goodrich, K. H., Palmer, M. T., & Schutte, M. C. (2003). The H-Metaphor as a Guideline for Vehicle Automation and Interaction. Report No. NASA/TM-2003-212672. Hampton, NASA Research Center Flemisch, F. O., Kelsch, J., Löper, C., Schieben, A., & Schindler, J. (2008). Automation spectrum, inner / outer compatibility and other potentially useful human factors concepts for assistance and automation. In: D. de Waard, F.O. Flemisch, B. Lorenz, H. Oberheid, and K.A. Brookhuis (Eds.), Human Factors for assistance and automation (1-16). Maastricht, the Netherlands: Shaker Publishing Flemisch, F., Heesen, M., Hesse, T., Kelsch, J., Schieben, A., & Beller, J. (2012). Towards a dynamic balance between humans and automation: authority, ability, responsibility and control in shared and cooperative control situations. Cognition, Technology & Work 14(1), 3-18 Flemisch, F. O., Bengler, K., Bubb, H., Winner, H., & Bruder, R. (2014). Towards a Cooperative Guidance and Control of Highly Automated Vehicles: H-Mode and Conduct-by-wire. Ergonomics 57, 343-360 Gasser, T. M., Arzt, C., Ayoubi, M., Bartels, A., Bürkle, L., Eier, J., Flemisch, F., Häcker, D., Hesse, T., Huber, W., Lotz, C., Maurer, M., Ruth-Schumacher, S., Schwarz, J., & Vogt, W. (2012). Rechtsfolgen zunehmender Fahrzeugautomatisierung - Gemeinsamer Schlussbericht der Projektgruppe Bundesanstalt für Straßenwesen (bast) Griiths, P., & Gillespie, R. B. (2004) Shared control between human and machine: haptic display of automation during manual control of vehicle heading. Proceedings of the12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, IEEE Hoc, J.-M. (2001). Towards a cognitive approach to human-machine cooperation in dynamic situations. International Journal of Human-Computer Studies 54, 509-540 Holzmann, F., Flemisch, F., Siegwart, R., & Bubb, H. (2006) From Aviation down to Vehicles - Integration of a Motions-Envelope as Safety Technology. SAE 2006 Automotive Dynamics Stability and Controls Conference. Novi, Michigan Hurtienne, J. (2011). Image schemas and design for intuitive use. Exploring new guidance for user interface design. Doctoral Thesis. Technische Universität Berlin. Kelsch, J., Flemisch, F. O., Löper, C., Schieben, A., & Schindler (2006). Links oder rechts, schneller oder langsamer? Grundlegende Fragestellungen beim Cognitive Systems Engineering von hochautomatisierter Fahrzeugführung. In: Grandt, M; Bauch, A.: Cognitive Systems Engineering i.d. Fahrzeug- und Prozessführung; 48. FAS Anthropotechnik, Karlsruhe, Cognitive Systems Engineering in der Fahrzeug- und Prozessführung, Deutsche Gesellschaft für Luftund Raumfahrt, 227-240 Moray, N., Inagaki, T., & Itoh, M. (2000) Adaptive automation, trust, and self-conidence in fault management of 94

R-En-02

time-critical tasks. Journal of Experimental Psychology: Applied, 6(1), 44-58 Müllhäuser, M., Schieben, A., Flemisch, F. O., & von Grünhagen, W. (2007) Simulatorgestützte Studien zur aktiven Sidesticksteuerung von Luft- und Bodenfahrzeugen am DLR. Grandt, M. & Bauch, A. (Eds.): 49. Fachausschusssitzung Anthropotechnik: Simulationsgestützte Systemgestaltung, 27-45, Bonn: Deutsche Gesellschaft für Luft- und Raumfahrt e.V.National Highway Traic Safety Administration - NHTSA (2013) Preliminary Statement of Policy Concerning Automated Vehicles Pacaux-Lemoine, M.-P., & Debernard, S. (2007). Common Work Space or How to Support Cooperative Activities Between Human Operators: Application to Fighter Aircraft. In: Harris, D. (Ed.): 7th International Conference on Engineering Psychology and Cognitive Ergonomics, Springer, 796-805 Parasuraman, R., & Riley, V. (1997) Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39 (2): 230253, Schutte, P., Flemisch, F., & Goodrich, K. (2003). Personal communication Sheridan, T. B. (1992) Telerobotics, automation, and human supervisory control. Cambridge, MA: MIT Press Society of Automotive Engineers – SAE (2014) Taxonomy and Deinitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems. SAE J 3016 Talmy, L. (1988). Force dynamics in language and cognition. Cognitive Science, 12(1), 49-100 Whalley, M. S., Hindson, W. S., & Thiers, G. G. (2000) A Comparison of Active Sidestick and Conventional Inceptors for Helicopter Flight Envelope Tactile Cueing. AHS International, 56th Annual Forum, 181-204, Virginia Beach, VA: NASAWinner, H. & Hakuli, S. (2006) Conduct-by-Wire - following a new paradigm for driving into the future. FISITA 2006 World Automotive Congress. 22 - 27 October, Yokohama, Japan.

95