Design Patterns for Cooperation

Design Patterns for Cooperation Motivation

Design Patterns

The Catalogue

The research done in WP3.3 results in three different, but closely linked cross-domain solutions:

A design pattern is not a finished design. It is a template for solving special problems, which can be used in many different design projects.

Develop crossdomain reusable reference designs and design patterns for DCoS !

History of Design Patterns: • Solving problems in urban architecture (Alexander, 1977) • Software engineering system design (Beck et al., 1987) • Object oriented software design (Gamma et al., 1995) • Interface & interaction design (Borchers, 2000)

The overall design patterns catalogue has been established for cooperation, state inference and adaptation, and user interfaces as a hypertext database:

Agent Cooperation

Catalogue State Inference & Adaptation

Multimodal User Interface

Design Pattern (DP): proven solution for a recurring problem (method) Reference Design (RD): exemplary instantiation of one or more Design Patterns (techniques)

System • Initiation • Maintenance • Completion

State Adaptation

• Attention • Workload • Situation Awareness

• Arbitration • Transition • Resource Allocation

(2014) (2014)

Benefit (Gamma, 1995; Beck et al., 1996):  Common design vocabulary  Documentation and learning aid  Team communications medium  Capture essential parts  Adjunct to existing methods  Extracted from working design  Reusable best practice

This is available online:

D3CoS Approach to Design Patterns:

Vision: Patterns for DCoS design State Inference

• HMI design: • Cooperation design:

User Interfaces • Mode • Group • Direction • Modality

Design Pattern Catalogue

Scenario Catalogue Scenario n Scenario 2 Scenario 1

Design pattern n Design pattern 2

Scenarios relevant for D3CoS

selected design patterns with design requirements

is part of

Agent Cooperation

Safety Efficiency Comfort

influence

Design pattern relevant for D3CoS

...

The following patterns have been created for the design of cooperative multi-agent systems in the phases initiation, maintenance, and completion:

influence design pattern selection criteria

Unique Scenario

influence

updates

Dynamic system design / Functions

Interaction Problem

Composition

Interface Problem

Problem

DCoS

Bottom-up generalization: document

Top-down specialization: implement

Current dynamic Configuration

is part of

TARF’s

Reference dynamic Configuration

Analysis

influence structures

⟶Problem Awareness ⟶Role Allocation ⟶Explicit Addressing ⟶Allocation to Cooperative Population ⟶Awareness of Own Task and Goal ⟶Awareness of Partners’ Task and Goal ⟶Certainty of Tasks and Goals ⟶Rewarding

updates

Unique System Static system design / Framework

Current static/ framework configuration

Problem Categories Catalogue

Modification

Reference static/ framework configuration

Problem category n Problem category 2

Problem category relevant for D3CoS

Semantic and structural interplay between design patterns and system design in D3CoS

Contact Information Dipl.-Medieninf. Markus Zimmermann Institute of Ergonomics Technische Universität München Boltzmannstraße 15 85747 Garching Tel +49 89 289-15375 [email protected] www.ergonomie.tum.de

Methods, Techniques, Tools This is a …

X Method

Technique

Method

Design Patterns

Technique

-

Tool

Catalogue of Design Patterns

X Tool

Sections: • Title: Conveys the central idea • Rank: Confidence and alternatives • Picture: Application or scheme • Context: Larger scale patterns • Problem Statement: Addressed situation • Problem Description: Design problem • Solution: Central method for solution • Diagram: “If you can’t draw it, it’s not a DP” • References: Smaller scale patterns • Literature & Authors

Explicit Addressing Problem: • Diffusion of responsibility • Selection of non-optimal agent Solution: Determine optimal and unique cooperation partner I ACT!

References:

Other Agent/ Problem Environment/ Resource Trigger

Alexander, C., Ishikawa, S., & Silverstein, M. (1977). A pattern language: Towns, buildings, construction. New York: Oxford University Press. Beck, K., & Cunningham, W. (1987). Using pattern languages for object-oriented programs. Technical Report CR-87-43, Tektronix, Inc. Presented at the OOPSLA'87 workshop on Specication and Design for Object-Oriented Programming. Beck, K., Crocker, R., Meszaros, G., Vlissides, J., Coplien, J. O., Dominick, L., & Paulisch, F. (1996). Industrial experience with design patterns. In Proceedings of the 18th international conference on Software engineering (pp. 103–114). Berlin, Germany: IEEE Computer Society. Borchers, J. O. (2000). A pattern approach to interaction design. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 369–378). New York City, New York, United States: ACM. Gamma, E. (1995). Design patterns: Elements of reusable object-oriented software. Reading, Mass: Addison-Wesley.

Cooperative Agent 1 Acknowledgement

YOU ACT!

Cooperative Agent 2

Consortium

Acknowledgments This research has been performed with support from the EU ARTEMIS JU project D3CoS (http://www.d3cos.eu) SP-8, GA No.: 269336. Any contents herein are from the authors and do not necessarily reflect the views of ARTEMIS JU. France

Germany

Design Patterns for State Inference & Adaptation Motivation State inference and system adaptation mechanisms allow an optimized task allocation and shared authority among DCoS agents: How to measure the human or machine agents‘ state?

State Adaptation Task Allocation and Resource Functions

State Inference Metrics

State Inference & Adaptation

How to adapt the system‘s state?

The following design patterns have been elaborated for the human state inference and machine state adaptation aspect:

⟶State Inference ⟶Human Attentional State Inference ⟶Human Workload State Inference ⟶Workload Visual State Inference ⟶Human Situation Awareness Inference ⟶Task Allocation and Resource Functions ⟶Conflict Solving and Replanning ⟶Resource Allocation ⟶Distributed Allocation ⟶Central Allocation

State Inference

State Adaptation

Human operator state inference relies on behavioural, physiological and cognitive metrics to predict the degradation of human/system interaction as well as the operator’s state. Those metrics were further developed towards methods: How to predict those DCoS states?

State Adaptation includes mechanisms and decision rules how to do a dynamic DCoS transition like a shift of responsibilities or resources. Those methods define basic task allocation and resource functions (TARFs), describing how to reallocate tasks, agents, or resources.

Human Workload State Inference

Distributed Resource Allocation

Problem: Keep a regular workload Solution: Integrate objective and subjective metrics • Performance measures (Errors, …) • Self-assessed measures (NASA-TLX, …) • Psychophysiological measures (HRV, …)

Problem: Resource in conflict for ad-hoc cooperation partners Solution: Transfer of distributed algorithm by Ricart & Agrawala • Agent holding resource is resource manager • All agents must request resources • Any scheduling is possible

Application Story: • Application of reference design in UAV and MARITIME experiments in all project cycles • Bottom-up update of the design pattern Workload Visual State Inference Problem: Real-time-capable set of metrics Solution: Integrate visual metrics

Mining Story: 1. Requirement: Adaptation (UAV, MAV) 2. Collection of system design problems in DP-T:


3. Scenario of the design problem in the UAV domain: cooperative UAVs observe an area of interest and transmit regularly the acquired sensor data (e.g. high res images) to the ground control station (GCS). Specific problem: Bandwidth 4. Generic interaction problem derivation: How to share a resource 5. Cross-domain problem derivation: Resource in conflict 6. Discussion of the problem cross-domain: Assign, decide, criteria, manner, … 7. Top-down theory driven solution research

Mining Story: Validation in AUT experiments


X Method

Method

Design Patterns

Technique

-

Tool

-

Technique

Tool

Zimmermann, M., Rothkirch, I. M., & Bengler, K. (2014). Reading the Driver: Visual Workload Assessment in Highly Automated Driving Scenarios. Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics (AHFE 2014), 2014.

Consortium


Germany

Design Patterns for Cooperative User Interfaces Motivation Cooperative interaction via multimodal user interfaces targets the reduction of complexity and the communication of agents’ intentions: Cooperative Information

How to communicate cooperative aspects in dynamic systems

Information Modality

Cooperative UI

Cooperative Information

Information Modality

We transferred design patterns for cooperation to user interface design patterns for communicating cooperative information; such as Explicit Addressing to Directed Information or Allocation to Cooperative Population to Group Information.

We researched several design patterns concerning different information modalities in the demonstrator experiments: Multimodality Problem: Human operator has limited attentional resources; specific information modalities may not be available. Solution: Multimodal fusion • Distribute information over interfaces • Link modality to action and attention • (Re)capture attention multimodal • Escalate modalities • Relieve overused channels

Action Suggestion UI How to choose the appropriate information modality in cooperative situations?

The following design patterns have been elaborated for cooperative user interfaces:

⟶Cooperative User Interfaces ⟶Information Modality ⟶Ambient Information ⟶Augmented Reality ⟶Multimodality ⟶Mode Information ⟶Group Information ⟶Directed Information ⟶Mutual Control UI ⟶Action Suggestion UI

Problem: Information about state of traffic and available tasks and resources needed for performing a task. Solution: Suggest actions which can be accepted or declined (common course) • Suggest • Resource, e.g. lane • Task, e.g. lateral control • Interact: Arbitrate, execute, communicate • Conclude

Mining Story: Top-down research, issues tested in AUT. Ambient Information

Mining & Application Story: AUT experiments across different modalities for refinement of DP

Problem: Visual attention; focus overused. Solution: Use peripheral vision

Experience:

Augmented Reality Zimmermann, M., Bauer, S., Lütteken, N., Rothkirch, I. M., & Bengler, K. (2014). Acting Together by Shared Control: Evaluating a Multimodal Interaction Concept for Cooperative Driving. The 2014 International Conference on Collaboration Technologies and Systems (CTS 2014), 2014.

Problem: Gain attention during cooperative situation Solution: Use augmented reality

Mode Information Problem: Mode of cooperation and machine intention need to be communicated by the interaction without mode error. Solution: Communicate mode information • What is the current mode (actions, space) • Why is the system in that mode (explan.) • What will the system do next (time) • Who is the cooperation partner


X Method

Method

Design Patterns

Technique

-

Tool

-

Technique

Mining & Application Story: AUT

Group Information Problem: How to communicate group membership Solution: • Connect graphically • Link iconographically • Relate by shape / colour

Mining & Application Story:

Mining & Application Story: • Specialization of DP-COOP Allocation to Cooperative Population • Bottom-up AUT and UAV derivation


Mining & Application Story: MARITIME

Tool

Zimmermann, M., & Bengler, K. (2013). A Multimodal Interaction Concept for Cooperate Driving. In 2013 IEEE Intelligent Vehicles Symposium (IV) (pp. 1285–1290).

Consortium


Germany