Performance Augmentation through Cognitive Enhancement (PACE)

Performance Augmentation through Cognitive Enhancement (PACE) Nick Morizio, Michael Thomas and Patrice D. Tremoulet

Lockheed Martin Advanced Technology Laboratories 3 Executive Campus Cherry Hill, NJ 08002 {nmorizio, mthomas, ptremoul}@atl.lmco.com

Abstract Performance Augmentation through Cognitive Enhancement (PACE) is a domain and application-neutral framework for managing user tasks according to context, including an assessment of the user’s cognitive state. This assessment allows the use of presentation mechanisms that are beneficial during certain cognitive states but detrimental during others. For example, it is useful to break a complex task down to smaller pieces during high-stress periods, but this may introduce redundancy, which is undesirable during normal-stress situations. While the data used by PACE to determine user context has thus far come solely from physiological and neurological sensors, many other possibilities are supported. The PACE framework has also been designed to handle a wide variety of mitigation strategies, that is, context-based manipulations of the user interface designed to maximize operator effectiveness. To date, the strategies tested have involved either deferring the presentation of information and tasks or altering the modality in which they are presented during periods of high workload, high stress, or working memory (WM) overload and delivering them later, at more convenient points for the operator. The PACE architecture developed under the Improving Warfighter Information Intake Under Stress program provides a powerful, flexible framework that can help support the next generation in computing: making interfaces truly personal, by supporting not only customization through user preferences, but also adaptation based upon workload, cognitive state, and the user’s environment.

1

Introduction

As part of the DARPA program Improving Warfighter Information Intake Under Stress, Lockheed Martin Advanced Technology Laboratories (LM ATL) has developed a domain and application-neutral framework for managing user tasks according to context, including an assessment of the user’s cognitive state. This framework, called Performance Augmentation through Cognitive Enhancement (PACE), has been used with multiple domains and applications, including Aegis-based command and control operator tasks and Tactical Tomahawk missile monitoring and retargeting. Within these domains, it has been used to perform multiple mitigation strategies for managing undesirable cognitive states, e.g. verbal working memory (WM) overload. Cognitive states were measured by a set of physiological and neurological sensors worn by test subjects while they performed operationally relevant tasks. This data was fused by a Cognitive State Assessment component that produced conglomerate cognitive state gauges which triggered the activation and deactivation of the mitigation strategies used. While the data used by PACE to determine cognitive state has come solely from physiological and neurological sensors, many other possibilities are supported. A variety of mitigation strategies are also supported by the architecture.

2

Background

2.1

Augmented Cognition

PACE was developed under the auspices of the DARPA program Improving Warfighter Information Under Stress. The goal of this program, formerly known as Augmented Cognition, or AugCog, is to optimize performance of combat command and control operators by using neuro-physiological sensors to control the behavior of human computer interfaces, e.g. by tailoring information presentation and task assignments to best suit the currently available cognitive resources of operators. More specifically, if an operator is engaged in a task that has recruited nearly all visio-spatial reasoning resources and critical, task-relevant information arrives, the system may elect to present this information verbally, over an audio channel, to facilitate rapid assimilation. Augmented Cognition represents the cutting edge in adaptive interfaces, going a step beyond traditional advanced HCI techniques to use neuro-physiological sensors to enable adaptation based upon not only the environment and tasks at hand, but also a real-time assessment of operators’ current cognitive capacities.

2.2

Sensors and Cognitive States

As a part of the Augmented Cognition effort, LM ATL has experimented with sensors which provide a variety of neuro-physiological measures including pupil dilation, galvanic skin response (GSR), heart rate variability (HRV), body/head positioning and three different types of brain activation measures: continuous electroencephalography (EEG), event related potential electroencephalography (ERP), and blood oxygenation levels. One of the objectives of an early series of experiments was developing a sensor suite which could be used to produce reliable and accurate cognitive state ‘gauges’, based upon multiple neuro-physiological measures. In some cases multiple devices that produce the same sort of physiological data were tested. The selection of sensors included in an integrated suite was based upon a combination of three factors: correlation between sensor data and performance demands, ability to operate the sensing device simultaneously with other sensors with high correlations to performance demands, and the physical discomfort experienced by users due to wearing the sensing device. Other considerations relevant to the Improving Warfighter Information Under Stress program, but not yet used to downselect sensors include portability and robustness of sensing devices. Sensor down-selection was facilitated by the PACE architecture’s ability to support plug-and-play of sensing devices, making it possible to quickly and easily add and remove different physiological sensing devices. Currently, LM ATL’s sensor suite consists of a wireless continuous EEG sensor, EKG and GSR sensors, and an offhead binocular eyetracker that logs pupil diameter in addition to point of gaze. These sensors generate EEG, HRV, GSR and pupilometry data that are used to compute values for the following gauges: verbal WM, spatial WM, cognitive workload, arousal. To date, LM ATL has focused primarily upon using sensor data to estimate the level of utilization of verbal and spatial working memory relative to an individual’s capacity, as well as estimating cognitive workload; we have also experimented with individual-sensor based estimates of arousal, e.g. heart rate based arousal and EEG based arousal.

2.3

Mitigation Strategies

The types of adaptations, or behaviors which the system may use to increase the operator’s effectiveness include automating or offloading tasks as stress and workload levels approach critical values, changing of the modality in which existing and/or new information is presented, and reinforcing information by representing it in multiple modalities (aural, spatial, verbal). The overall goal of the mitigation strategies is to increase overall performance, while maintaining perceived information load at moderate levels and providing the latitude to increase task difficulty (c.f., Johnstone, 1980). LM ATL has implemented and pilot tested three mitigation strategies: pacing, intelligent sequencing, and modality switching. The first two have been formally tested through laboratory-based “concept validation experiments” (CVE’s). Modality switching will be further explored in future experiments.

2.3.1 Intelligent Sequencing This strategy ensures that secondary task activities are scheduled to support maximum operator performance by timing the presentation of verbal and spatial secondary tasks when gauges indicate that the operator is not overloaded verbally or spatially, respectively. So for example, as operators perform primary tasks which are a mix of both spatial and verbal tasks (as they are in most operational environments) and secondary tasks arrive, PACE classifies them as mostly verbal or mostly spatial. Meanwhile, spatial and verbal working memory gauges indicate when the subject’s verbal and/or spatial memory stores are taxed by the primary task. Based on secondary task classifications and the gauge readings, PACE schedules the secondary tasks so that they interrupt primary tasks on an optimum schedule. For example, if gauges indicated that an operator’s spatial working memory store is taxed, only verbal secondary tasks will be presented until memory taxation has stabilized.

2.3.2 Pacing This strategy involves directing tasks according to an operator’s workload and arousal levels, as determined by arousal and cognitive workload gauges. Research has indicated that the timing of an interruption relative to a user’s current task load can affect the user’s ability to cope with the interruption (Czerwinski et al., 2000; McFarlane, 2002; Monk et al., 2002). The Pacing mitigation builds upon the Intelligent Sequencing mitigation, a) by adding the identification of appropriate “cognitive breaks” to the total information determining when tasks should be presented, and b) by allowing primary tasks to be decomposed so tasks and subtasks can be optimally scheduled at a higher granularity than just primary vs. secondary tasks. For example, if arousal and workload gauges indicate that the operator’s task load is above maximum threshold, pending tasks are queued for delivery rather than presented immediately. The pending tasks are delivered when the gauges indicate a break in cognitive activity. If too many secondary tasks are competing for delivery, the primary task will be decomposed into subtasks so that cognitive breaks (during which secondary tasks can be delivered) occur more frequently. Overall throughput of memory-taxing tasks are optimized.

2.3.3 Modality-based Task Switching This strategy involves developing alternate display strategies to invoke specific sensory modalities. Based on output from sensory gauges (e.g., verbal and spatial WM), as well as an understanding of current system state (i.e., which verbal and spatial tasks are currently being performed and their relative loading, c.f., task intelligence), display strategies that invoke modalities with spare capacity and/or which are best suited for the information to be communicated will be employed. Table 1 summarizes the various sensory modality display options that will be investigated. For example, while spatial information (e.g., location of a threat) is generally best presented via visual imagery (e.g., target on radar screen), it could be presented as sound localization (e.g., auditory signal at a given location) or as tactile cues (e.g., vibration of a sensor in a tactile vest). Table 1: Modality-based task switching schemes. Task Type Modality Switching Options

Verbal

Spatial

Visual

Text Instructions or System Status Displayed on Screen (e.g., target ID, type, speed)

Graphics Displayed on Screen (e.g., threat on radar screen)

Auditory

Speech Instructions or Coded Auditory Cues (e.g., auditory warnings)

Sound Localization via Headphones (e.g., audio cue to left ear or to right ear depending on location of target threats)

N/A

Vibrations via Tactile Vest (e.g., vibractor tactor in sensory location depending on location of target threats)

Haptic

3

System

3.1

Architecture

A key goal of the design of PACE was to a produce a highly extensible, domain-neutral framework. The resulting architecture provides the flexibility necessary to interchange many different mitigation strategies in response to changes in user cognitive state (or other contextual or environmental factors which PACE is tracking) as well as to integrate with several different applications. To apply PACE to a new domain, it is only necessary to extend a few of the framework components of PACE, such as the Environment Director or Active Task Manager. To incorporate additional mitigation strategies, the management components such as the System Interface Director or Adaptive Workload Director can be extended.

External Application

Task Interactions

User

Cognitive State Assessor (CSA)

Presented Tasks

Environment Director (ED)

Sensor Data

Cognitive State

Configuration Files

User Actions

Task Attributes Active Task Manager (ATM)

User Performance

New Tasks

Proposed Tasks

System Interface Director (SID)

Proposed Tasks

Task Information Manager (TIM)

Adaptive Workload Director (AWD)

Delegated Tasks

Delegation Manager (DM)

Figure 1: PACE Architecture The PACE architecture is divided into seven components each responsible for different aspects of user task management: 1. Active Task Manager – Tracks context associated with a task actively being performed by the user 2. Adaptive Workload Director – Manages all pending tasks 3. Cognitive State Assessor – Analyzes the user’s cognitive state based on appropriate cognitive state gauges, computed from real-time sensor data 4. Delegation Manager – Responsible for reassigning tasks to other users or software entities 5. Environment Director – Selects appropriate interaction modalities for proposed tasks, interacts with the external user application 6. System Interface Director – Approves or rejects tasking based on current cognitive state 7. Task Information Manager – Manipulates tasks through decomposition and configuration of presentation options. Also tracks task performance.

3.2

Cognitive State Representation/Use

To this point, experiments with PACE have involved gauging the following cognitive states: verbal working memory, cognitive workload, and arousal. Spatial working memory, executive function, and attention gauges are planned and/or in development. The specific set of cognitive states included is highly configurable within PACE.

Only the Cognitive State Assessor needs to be aware of the specific sensors and their relationship to the cognitive gauges used for a particular experiment. Once an assessment is made, it is sent by the CSA to the rest of PACE which uses the assessment to trigger the appropriate mitigations.

3.3

Task Selection in PACE

Tasks in PACE represent goals that need to be achieved by the user. Typically, a task is a straight-forward sequence of actions that must be coordinated between the user and the system. PACE also represents more complex tasks, such as abstract tasks that may have many potential decompositions. In either case, the task itself must specify the complete set of ways that it can be executed. Task selection refers to the set of processing that determines which tasks to execute at which times and which particular plan to use when executing the task. Task selection is a multi-stage process within PACE. In the first stage tasks are inserted either by the application or by the task generator in the Task Information Manager. These tasks may be provided via messages from another system, or triggered via the completion of other tasks by the user. If new tasks are not directly executable, the second stage decomposes the task into a forest of subtasks where each tree represents one possible completion of the inserted task and edges in the trees represent temporal dependencies between the subtasks. Through decomposition, a task is able to represent alternative execution strategies. As a trivial example, consider a user that needs to click two buttons (say A and B) in any order. In this case the abstract parent task would decompose into two trees, one which has button A presented first and B second, and the other which has button B presented first followed by A. This sort of decomposition can play a key role in task mitigation, allowing PACE to select the decomposition trees which provide a better fit with the user’s and the application’s current capabilities. It also allows PACE to delay the execution of a task until an appropriate time. These tasks and dependency trees are managed by the Task Accomplishment Strategy Manager (TASM) which maintains a queue sorted by task priority. The priority for a task can be determined based upon the application, but defaults to a function based on insertion time and the urgency of the task as defined in the task description. Periodically, the TASM proposes new tasks which have both the highest priorities and resolved dependencies. This process is the third stage of task selection. If the task is rejected at any point after this phase, the task returns to the TASM queue and will be proposed again, up until the deadline specified for the task. Task rejection occurs when there is no way to execute the task, either because there is no empty slot to display the task in the application or because a mitigation strategy was activated. In the fourth stage, tasks are examined by the environment manager and compared to the latest cognitive workload assessment to make sure the task will not overload the user. The parameters available from the assessment which can be used to make this judgment correspond to the cognitive state gauges built out of neuro-physiological data. Individual parameters may or may not be enabled depending upon available sensors or depending upon the type of tasks. Each task comes with a set of thresholds for each parameter representing either the minimum or maximum acceptable values for that task. After checking to make sure a proposed task will not overload the user, the task moves on to the fifth stage, which is modality selection.

3.4

Modality Selection in PACE

Each PACE task is defined with preferences towards certain modalities. For example, an alert task may be best delivered as text in a window, but if that panel is not available or well-supported by the application, then the alert can be delivered as speech. In this case, we say that the task prefers the panel modality over the speech modality. In addition, an application interface in PACE specifies which modalities it is capable of, and what type of task quality can be expected for each of those modalities. The application also specifies the number of slots available in each modality. For example, an alert window may have space for only five alerts. During the last phase of task selection, the System Interface Director examines the available modalities and the proposed task. The task is rejected if no slots are available in any modality that can present the task. Otherwise the SID accepts the task and designates it for the modality with the greatest utility, defined as a combination of the

task’s preference, the application’s modality quality, and the user’s current available cognitive capacity with respect to the type of task. If the task passes this stage, it is handed to the Active Task Manager and will be presented to the user immediately.

3.5

Application Domains

The PACE architecture has already been successfully applied to several different domains and applications. These include a surrogate interface representing the Aegis Weapon System Identification Supervisor (IDS) console, a prototype Tactical Tomahawk Weapon Control System (TTWCS) interface for strike monitoring and retargeting, and a number of experimental research-driven applications intended for proof of concept of various mitigation strategies.

4

Future Work

4.1

Delegation

Delegation is a key capability supported by PACE which is not fully developed. Delegation within PACE can take two forms – human-peer delegation and autonomous execution. In human-peer delegation, the Delegation Manager within PACE will react to a user’s inability to accomplish a task (due to cognitive state or other limitations) by identifying another user who is capable of accomplishing the task and forwarding the task to that individual. In autonomous execution, software agents can be dispatched to take care of certain tasks for a user. While this is useful in overload situations, it is not desirable for agents to always handle these tasks. Humans may be able to perform the task more effectively than any form of automation, and automation may lead to a loss of operator situational awareness, increasing the risk of suboptimal decision-making after stress levels subside. Moreover, humans may develop strategies to defeat the triggering of automation if they believe that a loss of control makes them less effective.

4.2

Additional Task Selection Strategies

4.2.1 Chunking While chunking is generally the grouping of information based soley upon semantic content (Miller, 1956), such groupings may also be encouraged via modality differences. In a recent study, Nelson and Bolia (2003) demonstrated that spatial audio displays can enhance auditory-cue identification by approximately 50%, as well as speed reaction time. In this work they placed call signs at different auditory spatial locations (i.e., each call sign sender was assigned a specific spatial location). This is a form of spatialized auditory-verbal grouping that may be employed by PACE in future experiments. Chunking may also be done according to criticality. For instance, in a high stress and/or high workload situation, the method of delivering alerts to the user could be modified to help focus the user’s attention upon the most critical set(s) of information. That is, knowing a user is currently overloaded (based on cognitive workload assessment), the system could stop using ordinary methods (modality and information format) to inform the user of new alerts. Those alerts that concern things which are outside a predefined critical context can be queued, e.g., in a small alert window.1 Meanwhile, the system will immediately inform the user of any and all new high-priority alerts related to the critical task context through alternative interruption strategies that facilitate chunking, such as sound localization or vibrations from a tactile vest.

1 All unacknowledged alerts should be visible from this window, so that the user can elect to attend to a ‘non-critical’ alert if (s)he chooses. There should be some sort of visual distinction between high and low priority alerts, as well as between alerts related to tracks in the critical region and those related to tracks outside this region. The user should be able to sort the alerts according to differing criteria.

4.3

Other Sources of Cognitive State

While the data used by PACE to determine cognitive state come solely from physiological and neurological sensors, many other possibilities are supported. One alternate mechanism would be to have the application system help determine the user’s cognitive state, by monitoring the number of tasks being simultaneously performed, the amount of mouse movement the user is making, or any number of other metrics. These values can be sent to the Cognitive State Assessor and taken into account when computing cognitive state gauge values. The PACE architecture developed under the Improving Warfighter Information Intake Under Stress program provides a powerful, flexible framework that can help support the next generation in computing: making interfaces truly personal, by supporting not only customization through user preferences, but also adaptation based upon workload, cognitive state, and the user’s environment.

References Czerwinski, M., Cutrell, E., and Horvitz, E. Instant Messaging: Effects of Relevance and Time, In S. Turner, P. Turner (Eds), People and Computers XIV: Proceedings of HCI 2000, Vol. 2, September 2000, British Computer Society, p. 71-76. Johnstone, A H. (1980). Nyholm Lecture: Chemical education research: Facts, findings, and consequences. Chemical Society Reviews, 9(3), 363-380. Miller, G.A. (1956). The magical number seven plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97. McFarlane, D.C. (2002) "Comparison of Four Primary Methods for Coordinating the Interruption of People in Human-Computer Interaction," Human-Computer Interaction, 17(3), Laurence Erlbaum Associates, Mahwah, New Jersey. Monk, C., Boehm-Davis, D., & Trafton, J. G. (2002). The Attenional Costs of Interrupting Task Performance at Various Stages. In the Proceedings of the 2002 Human Factors and Ergonomics Society Annual Meeting. Santa Monica, CA: HFES. Nelson, T., & Bolia, R. (2003). Evaluating the effectiveness of spatial audio displays in a simulated airborne command and control task. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 202-206). Denver, CO, October 13-17. Proffitt, D. (2003). University of Virginia, results reported during the DARPA Augmented Cognition Phase 2 Kickoff and Technical Working Meeting, Redmond, WA. July 9-11, 2003. Sulzen, J. (2001). Modality Based Working Memory. School of Education, Stanford University. Retrieved, February 5 2003, from http://ldt.stanford.edu/~jsulzen/james-sulzen-portfolio/classes/PSY205/modality-project/paper/ modality-expt-paper.PDF