Proceedings of the SPIE Defense and Security Symposium, Biomonitoring for Physiological and Cognitive Performance during Military Operations. John A. Caldwell, Nancy Jo Wesensten, Eds. Vol. 5797: pgs. 90-99. 2005.
Evaluation of an EEG-workload model in the Aegis simulation environment Chris Berka∗1, Daniel J. Levendowski1, Caitlin K. Ramsey1, Gene Davis1, Michelle N. Lumicao1, Kay Stanney2, Leah Reeves2, Susan Harkness Regli3, Patrice D. Tremoulet3, Kathleen Stibler3 1 Advanced Brain Monitoring, Inc., 2850 Pio Pico Drive, Suite A, Carlsbad, CA USA 92008; 2 Department of Industrial Engineering and Management Systems, University of Central Florida, 4000 Central Florida Blvd., Orlando, FL USA 32816-2450; 3 Lockheed-Martin Advanced Technology Laboratories, 3 Executive Campus, 6th Floor, Cherry Hill, NJ USA 08002 ABSTRACT The integration of real-time electroencephalogram (EEG) workload indices into the man-machine interface could greatly enhance performance of complex tasks, transforming traditionally passive human-system interaction (HSI) into an active exchange where physiological indicators adjust the interaction to suit a user’s engagement level. The envisioned outcome is a closed-loop system that utilizes EEG and other physiological indices for dynamic regulation and optimization of HSI in real-time. As a first step towards a closed-loop system, five individuals performed as identification supervisors (IDSs) in an Aegis command and control (C2) simulated environment, a combat system with advanced, automatic detect-and-track, multi-function phased array radar. The Aegis task involved monitoring multiple data sources (i.e., missile-tracks, alerts, queries, resources), detecting required actions, responding appropriately, and ensuring system status remains within desired parameters. During task operation, a preliminary workload measure calculated in real-time for each second of EEG and was used to manipulate the Aegis task demands. In post-hoc analysis, the use of a five-level workload measure to detect cognitively challenging events was evaluated. Events in decreasing order of difficulty were track selection-identification, alert-responses, hooking-tracks, and queries. High/extreme EEG-workload occurred during high cognitive-load tasks with a detection efficiency approaching 100% for selection-identification and alert-responses, 77% for hooking-tracks and 70% for queries. Over 95% of high/extreme EEG-workload across participants occurred during high-difficulty events (false positive rate < 5%). The high/extreme workload occurred between 25-30% of time. These results suggest an intelligent closed-loop system incorporating EEGworkload measures could be designed to re-allocate tasks and aid in efficiently streamlining a user’s cognitive workload. Such an approach could ensure the operator remains uninterrupted during high/extreme workload periods, thereby resulting in increased productivity and reduced errors. Keywords: Human-computer interface, EEG, workload, task allocation, closed-loop system, cognitive overload, information delivery, mitigation strategies, command-and-control systems (C2).
1. INTRODUCTION Since the advent of graphical user interfaces (GUIs), a variety of users have sought to embrace computing technology for a wide range of applications. GUIs have eased interaction by providing tangible constructs (i.e., windows, icons, menus) via which users can access functionality primarily via a keyboard or pointing device. Yet, the information age has spawned vast data sources, which have pushed the limits of what GUIs can effectively convey. For example, C2 systems inundate commanders with an abundance of real-time data sources providing operational views of the environments (air, sea, ground) in which their forces are deployed. Through primarily visual GUIs, these systems seek to enable military commanders to determine the status of forces (i.e., friend versus foe) and which assets to allocate to meet mission ∗
[email protected]; phone 1 760 720-0099; fax 1 760 720-0094; www.b-alert.com
objectives. However, as the amount of data available has increased, the ability of GUIs to transform this data to decision-supportable information has been exceeded. Today, sources of data, in the military and many other fields, go unutilized as they are never rendered as information sources or, even if they are conveyed, their representation is such that it exceeds limits of the human information processor. HSI must be transformed such that information is presented when needed, in a modality that is readily perceivable, and in a form that is readily interpretable1. Current HSI loads information onto users without regard to their current workload. Whether or not a user is loaded at a given instance, if information is available it will be presented to a user if a computer routine is instructed to present it. The integration of real-time physiological indices of user workload into the HSI loop could assist in managing information flow, thereby enhancing human performance of complex tasks, such as those associated with C2 environments. With such a system, physiological indicators of user state would be used to adjust interaction to suit a user’s engagement level. For instance, when a user was overloaded, the indices would trigger information to be offloaded or withheld. When a user was inoperative or had additional capacity, the indices would trigger greater information dissemination or task re-allocation. The envisioned outcome is a closed-loop system that utilizes physiological indices for dynamic regulation and optimization of HSI in real-time with a goal of maintaining information load within the limits of the human information processor2. Although numerous physiological parameters, including cardiovascular indices, pupil diameter, eye movements and galvanic skin response, have been employed to detect cognitive state changes, the EEG is the only physiological signal that has been shown to reflect subtle shifts in alertness, attention and workload that can be identified and quantified on a second-by-second basis. EEG indices of cognitive state changes are sensitive and reliably correlated with performance39 , and several investigations have reported the ability to predict performance on a second-by-second basis using real-time EEG analysis10-12. EEG measures of workload and task difficulty have been reported in studies of air traffic controllers13, airline pilots14,15, drivers16, and participants performing cognitive tasks17,18. In a preliminary investigation of EEG correlates of workload conducted by the Advanced Brain Monitoring Research Team, an integrated hardware and software solution for acquisition and real-time analysis of the EEG was implemented to characterize EEG parameters associated with changes in mental workload17. A previously reported workload model used a set of EEG parameters to provide real-time identification of alertness and drowsiness and to predict vigilance decrements19-21. These EEG workload parameters were reliably associated with levels of workload during performance of simple tasks, including: the Warship Commander Task (WCT), a simulated navy C2 environment that allowed workload levels to be systematically manipulated22; a cognitive task with three levels of difficulty and consistent sensory inputs and motor outputs23, and a multi-session image learning and recognition memory test24,25. The present study was designed to utilize the previously reported EEG workload parameters as inputs to a closed-loop manipulation of a complex, cognitively challenging task. Specifically, an EEG indicator was used to assess workload while individuals performed the role of an identification supervisor (IDS) in an Aegis C2 simulated environment. The Aegis C2 is a combat system with an advanced, automatic detect and track, multi-function phased array radar. The Aegis task required an individual to monitor the current state of an abundance of entities and data sources (i.e., missile tracks, alerts, queries, resources), detect required actions, affect the appropriate response, and ensure system status remained within desired parameters (i.e., keeping air, surface, and sub-surface threats properly engaged). Because the IDS position entails the second highest workload of 32 operators in a typical Aegis Combat Information Center (CIC), it is a critical position to evaluate for potential human-system design issues and cognitive overload conditions. The Aegis C2 simulation environment provided a closer approximation to the cognitive challenges encountered in an operational military environment, allowing the EEG workload measures to be refined during post-hoc analysis. With the relationship between the EEG workload and Aegis task demands now established and implemented in real-time, future efforts will be directed toward the establishment of thresholds for identifying states of cognitive overload or states where operators can receive additional input. It is anticipated that these thresholds will serve as filters in a closed-loop system to control the delivery of information within military C2 environments such as Aegis.
2. METHODOLOGY 2.1 Acquisition hardware system
A battery-powered wireless EEG sensor headset was used to acquire six channels in a standard hardware montage that includes bipolar recordings from Fz-POz and Cz-POz, unipolar recordings from Fz, Cz and POz referenced to linked mastoids, and a bipolar configuration for horizontal and vertical EOG (see Berka, 2004 for a detailed description of the sensor headset). Data are sampled at 256 samples/second with a bandpass from 0.5 Hz and 65Hz (at 3dB attenuation) obtained digitally with Sigma-Delta A/D converters. The EEG sensor headset requires no scalp preparation and provides a comfortable and secure sensor-scalp interface for 12 to 24 hours of continuous use. The headset was designed with fixed sensor locations for three sizes (e.g., small, medium and large). Sensor placement was determined using a database of over 225 participants so that each sensor is no more than one centimeter from the International 10 – 20 system coordinates. The workload studies described in this paper required only bipolar recordings from Fz-POz and Cz-POz. The remaining channels are utilized to monitor other cognitive states including attention, learning and memory. Amplification, digitization and radio frequency (RF) transmission of the signals are accomplished with miniaturized electronics in a portable unit worn on the head. The combination of amplification and digitization of the EEG close to the sensors and wireless transmission of the data facilitates the acquisition of high quality signals even in high electromagnetic interference environments. 2.2 Additional sensors and cognitive state gauges In addition to the wireless EEG system, three alternative physiological sensor systems were employed to acquire data during the Aegis C2 task. Additional systems included Functional Near Infrared imaging (fNIR) from Drexel University, heart rate and galvanic skin response from AnthroTronix, Inc., and eye tracking from EyeTracking, Inc. This paper only reports on the data collected by the EEG system. 2.3 EEG artifact identification and decontamination, and signal processing Quantification of the EEG in real-time, referred to as the B-Alert® system, is achieved using signal analysis techniques to identify and decontaminate eye blinks, and identify and reject data points contaminated with electromyography (EMG), amplifier saturation, and/or excursions due to movement artifacts (see Berka, 2004 for a detailed description of the artifact decontamination procedures). Decontaminated EEG is then segmented into overlapping 256 data-point windows called overlays. An epoch consists of three consecutive overlays. Fast-Fourier transform is applied to each overlay of the decontaminated EEG signal multiplied by the Kaiser window (α = 6.0) to compute the power spectral densities (PSD). The PSD values are adjusted to take into account zero values inserted for artifact contaminated data points. The PSD between 70 and 128 Hz is used to detect EMG artifact. Overlays with excessive EMG artifact (“EMG”) or with fewer than 128 data points (“missing data”) are rejected. The remaining overlays are then averaged to derive PSD for each epoch with a 50% overlapping window. Epochs with two or more overlays with EMG or missing data are classified as invalid. For each channel, PSD values are derived for each one-Hz bin (“bin”) from 3 Hz to 40 Hz and the total PSD from 3 to 40 Hz (“band”). “Relative power” variables are also computed for each channel and bin using the formula (“total band power/total bin power”). This analysis creates 76 variables per channel that can be used to implement the model. The outputs of the PSD are used offline for the development of the model and in the real-time analysis program to provide the workload and alertness classifications. 2.4 Aegis C2 simulated environment Within an Aegis environment, an IDS is required to monitor the current state of multiple data sources (i.e., missile tracks, alerts, queries, resources), detect required actions, affect the appropriate response, and ensure system status remains within desired parameters (Figures 1 & 2).
Figure 1: Screen shot of Aegis simulated task environment.
Figure 2: Participant performing Aegis simulated task.
Within the Aegis simulated environment (Figure 1), two primary and one secondary task were evaluated. The two primary tasks consisted of 1) selecting the highest threat radar track from 10s to 100s of tracks based on a given ordered set of rules (Table 1), and 2) correctly identifying (ID) the selected track based on a non-ordered set of rules (Table 2). A user must select a track to ID by making an explicit action, such as hitting a “Select” button via the mouse after clicking on the selected track or saying “Select the hooked track” into the microphone. An example of a correctly selected most critical track would be: Suspect Hostile Air track headed towards Ownship with the shortest Time-toOwnship of any similarly critical tracks. No other tracks could be selected until the user entered an ID for the selected track (based on the ID non-ordered rules presented in Table 2). Users could ID a track by making an explicit action such as entering the ID designation in a text box via the keyboard or by speaking into the microphone the designation in a trained phrase format, such as “The ID of this track is Attack Hostile.” Information could be obtained about a track primarily from a list of designated fields on the right side of the display, which would appear once the track was selected. Some of the presented field values would appear missing, requiring the user to request information verbally, through the spoken language interface. Once a track was identified, it was removed from the display and overall scores were adjusted to reflect a correct ID or an incorrect ID. “Monitor” tracks were removed as well in order to simulate those tracks being delegated to another console station. Rules by Order of Priority 1. Bearing 2. Pre-ID 3. Type
4. Time-toOwnship/ Speed
Options by Priority
Criteria
Attack Hostile
Monitor
Ignore Friendly
Towards Ownship (inbound) Away from Ownship Suspect Hostile Unknown Assumed Friendly Air Surface Water
Platform Type
Armed
Armed or Unarmed
Unarmed
Respond to Query
No
Yes or No
Yes
Source of Radar
Shorter time Longer time Slower speed Faster speed
Multiple Source
Ownship radar or Ownship Manual Yes or No
Remote radar or Remote manual No
Ownship radar, Ownship manual, Remote radar or Remote manual Yes
Table 1: Ordered rules for selection of highest threat track.
Table 2: Non-ordered rules for correct identification of selected track.
A user’s performance was tracked by response time (i.e., how much time it took them to select a track) and accuracy (i.e., whether the track was correctly identified as Attack Hostile, Ignore Friendly, or Monitor). The secondary task was modeled after traditional n-back studies and involved the IDS responding to requests presented as alerts within the Aegis simulated environment (e.g., “What was the location of the 2nd-to-last track you identified?”). Users had to respond to alerts when presented—the system prevented users from completing other actions until an alert was answered. Alerts were categorized as either mostly verbal or mostly spatial information requests in order to allow flexibility in manipulating the types of cognitive loading induced in a user’s working memory. This allowed another potential factor to be tracked with EEG signals and subsequently accommodated for when necessary to streamline cognitive load via interface design changes. Response time and accuracy were also assessed for this task. The Aegis environment was simulated on an Intel Pentium 4- 2.4 GHz Dell desktop computer with 512 MB memory. The LCS (spoken language interface) software was run on an Intel Pentium 3- 1.2 GHz Dell laptop computer with 640
MB memory. The operating system of both machines was Red Hat/Fedora Linux 9.0 with 2.4 kernel. The visual interface was presented on a 17” screen. All user responses were with a standard keyboard, 2 button mouse, lapel microphone and foot pedal. 2.5 Task scoring Each of the three types of tasks was worth 10 points. Partial credit was given for selection according to the following scheme: if the second most critical track is selected instead of the most critical track, user earns 9 points, if the third most critical track is selected user earns 8 points, … if the 10th most critical track is selected user earns 1 point. No partial credit was given for identification--you either earn 10 points or earn none. For alerts, there was a penalty: correct responses earn 10 points and incorrect responses result in a loss of 2 points. The faster users selected critical tracks, correctly identified them, and correctly responded to alerts, the more points they earned. Thus, both speed and accuracy contributed to a user’s performance scores to deter random guessing. 2.6 Participants & Procedure Five participants completed the study (3 males, 2 females). Participants with a history of neurological disorders, head trauma, use of psychotropic or illicit drugs, or who were currently pregnant were excluded. All participants were college graduates. Participants read and signed an informed consent and then began studying and learning the rule-based hierarchies for determining optimal performance in the selection, identification, and alert tasks. The complexity of the Aegis simulation tasks required participants to complete a minimum of eight hours of supervised training before testing. Participants were randomly assigned to one of two ordered, counterbalanced task scenario conditions consisting of four 5 minute task scenarios with 30 seconds of rest between each scenario. Each subject completed a 15-minute baseline EEG session that included three 5-minute baseline conditions (eyes open (EO), eyes closed (EC), and 3-choice vigilance task (3C-VT)). All participants completed the baseline EEG testing sessions between the hours of 8am and 2pm. During the Aegis training and testing sessions, EEG data was acquired with the wireless EEG system. After each Aegis scenario session, users complete the NASA TLX subjective workload questionnaire and participated in a debriefing session. 2.7 Workload classification model A linear discriminant function analysis (DFA) model was previously reported17 that used 19 EEG variables empirically selected to provide EEG indices of alertness. The model classified each one-second EEG epoch, and was then classified into one of four states: “high vigilance”, “low vigilance”, “relaxed wakefulness”, and “sleep-onset.” These four states were derived using EEG acquired from participants in sleep deprivation studies. The high and low vigilance states were modeled by varying the level of task engagement. Relaxed wakefulness is the state induced when participants are instructed to relax with eyes closed and is generally characterized by predominance of EEG in the alpha frequency band (8 – 12 Hz.). Data for the sleep-onset class were obtained using EEG samples acquired just subsequent to sleep onset. Results showed that the percentage of high vigilance classifications increased as a function of mental workload as measured in the WCT C2 environment and other tests previously mentioned22,23,24,25. As a measure of workload, the model described above was limited because it did not provide a continuous measure sensitive to subtle shifts in required effort. In off-line analyses, the mean high vigilance probabilities generated by the DFA were found to be significantly correlated with workload levels as measured by performance in the WCT and the 3Level cognitive and learning and memory tests (referenced previously). The probabilities are generated as an intermediate result of the discriminant function model and range from 0.0 to 1.0, with the sum of the probabilities across the four classes equal to 1.0. The model previously reported17 used linear DFA. The classification model used in this study included a quadratic discriminant function with the EEG variables in Table 3. The strategy and methods used to develop this study’s model were similar to the previous models. The probabilities of high vigilance were generated in real-time for each second of EEG and submitted as inputs to the closed-loop Aegis manipulation during data acquisition. FzPOz CzPOz
32Hz 5Hz
34Hz 10Hz
36Hz 31Hz
38Hz 35Hz
Table 3: Selected EEG variables (R connotes relative power variables).
40Hz 37Hz
R6Hz R12Hz
R8Hz R14Hz
To refine the workload measure for use in the Aegis task, the first goal was to define multiple workload measures using thresholds that are applicable across participants and within-participants over time and that could be implemented in real-time. A second goal was to ensure the workload measures accommodated gradual changes in state of the individual, because users would be participating in the closed-loop application for several hours at a time. A means to qualify workload for a given one-second epoch was derived beginning with the computation of two-minute moving means and standard deviations of the high vigilance and relaxed wakefulness probabilities generated by the DFA. Through the use of data from three baseline conditions to fit the DFA model, the resulting probabilities included adjustments to accommodate individual differences in EEG pattern. The moving means for each epoch were then transformed into z-scored values. In order to apply workload thresholds to the z-score values that could be universally applied across participants, the standard deviations used for the z-score computations were based on the mean standard deviations across all epochs and participants. Z-score thresholds were then derived to quantify five levels of workload. Extreme, high, normal and low workload levels were based on probabilities of high vigilance. The distracted workload category was based on the probability of the relaxed wakefulness class. To assess the accuracy (sensitivity and specificity) of the high and extreme (“higher”) workload measure in the Aegis environment, time windows were established around the designated Aegis activities. The following Aegis events were selected as those most likely to elicit high workload: 1) Selection-Identification, including analysis of each one-second epoch between the selection and identification, 2) Alerts, including the period from the initiation of the alert to the alert response, 3) Hooking tracks, and 4) Querying the system via the LCS. To accommodate possible timing differences, events extracted from the Aegis Log and the EEG higher workload indices were aligned according to the following rules. LCS queries and Hooking events were matched with EEG higher workloads within a one-second window before or after the events. One-second before the Select, the interval between the Select and ID, and up to one second after the ID events was used to identify EEG workload correlates. For alerts, the interval between the initiation of the alert until one-second after the user responded was matched. 2.8 Experimental design A 2 (with or without pacing augmentation) x 2 (high or low workload) within-subjects design was conducted. The pacing augmentation consisted of delaying the presentation of alert tasks during high workload conditions and until workload decreased. This paper only reports on data from scenarios without pacing augmentation. Participants performed 4 scenarios per testing session: 2 with augmentation, one high workload (80 tracks) and one low workload (40 tracks); 2 without augmentation, one high and one low workload (Table 4). One participant (#9) only completed one of the two scenarios without augmentation. Response time and accuracy were tracked, and users were scored according to the aforementioned scoring procedures.
Augmentation Workload Level
No Low
4 Aegis Scenarios No Yes High Low
Yes High
Table 4: 4 Aegis Scenarios
3. RESULTS The relationships between the EEG higher workload levels and the Aegis activities are presented in Table 5 by and across participants. Each scenario provided approximately 600 seconds of EEG for analysis. The data revealed that greater than 96% of select-ID and alert events were correlated with EEG higher workload levels. Hooking and LCS events were associated with higher workload to a lesser extent (77% and 70% respectively). Over 95% of the total EEG higher workload across all participants was accounted for by Aegis events, resulting in an overall false positive rate of less than 5%. These results demonstrate the excellent sensitivity and specificity of the EEG high workload indices with
respect to the cognitive demands of the Aegis environment. Figure 3 provides a representative sample of the aligned patterns between the EEG workload indices and selected Aegis tasks.
User 8 8 9 10 10 11 11 12 12
Task Task # Difficulty 9 Easy 12 Medium 1 Medium 9 Easy 1 Medium 9 Easy 12 Medium 9 Easy 12 Medium Total
Total Select Hooks IDs Alerts LCS HW 116 50 12 49 206 166 38 12 42 248 165 26 12 30 165 87 25 12 43 214 111 22 12 27 210 110 26 12 21 197 105 22 12 15 193 118 35 12 30 178 136 27 12 29 193 1114 271 108 286 1804
Percentage Hooks 77.6% 83.7% 70.3% 73.6% 81.1% 74.5% 73.3% 77.1% 76.5% 76.6%
Select IDs 96.0% 97.4% 96.2% 96.0% 100.0% 100.0% 90.9% 100.0% 92.6% 96.7%
Alerts 91.7% 100.0% 91.7% 100.0% 100.0% 100.0% 100.0% 83.3% 100.0% 96.3%
LCS 77.6% 83.3% 66.7% 72.1% 59.3% 71.4% 60.0% 56.7% 65.5% 69.9%
Workload 94.2% 97.6% 93.9% 95.8% 95.7% 96.4% 90.7% 98.9% 95.9% 95.5%
Table 5: Relationship between EEG higher workload levels and Aegis tasks. HW = High Workload.
1
2
3
4a 4b
5
6 7
8
9 10
1
2
3
4a 4b
5
6 7
8
9 10
Alert
LCS
Correct Response Incorrect Response Unscored Activity
Extreme Workload High Workload Normal Workload Low Workload High Distraction
Workload 1. 2. 3. 4. 5. 6. 7. 8.
Hook
Select
ID
Subject is presented with an alert, and interprets the request. Subject hooks a track and responds to the alert. Subject realizes his error in hooking this track for the sixth time. a. Subject begins making decision between presumably final two “winning” tracks. b. Subject makes a correct selection. Subject initiates LCS Query, waits for response, interprets LCS response, and makes ID decision. Subject hooks track for the first time and encodes information. Subject makes decision to select the “winning” track that was not selected in 4. Subject makes non-optimal selection. Score = 0.6.
9. Subject interprets information provided by LCS Query and makes correct ID. 10. Subject hooks track for the first time and encodes information. Figure 3: Representative sample of the aligned patterns between the EEG workload indices and selected Aegis tasks.
4. CONCLUSIONS The primary goal of this study was to assess the feasibility of accurately detecting and quantifying an EEG indicator of mental workload during performance of a highly complex, cognitively challenging task. The Aegis simulation environment was selected because of its military relevance and similarities with cognitive challenges encountered in an operational environment. This study demonstrated that the wireless EEG system with real-time analysis software, previously shown to provide valid workload classifications during simple cognitive tasks, was also effective in acquiring high quality data, identifying and decontaminating artifacts and classifying EEG workload measures in a more complex and cognitively challenging C2 operational environment. The five-level EEG workload model provided good correlations with operator performance within the Aegis environment. When participants were engaged in the most cognitively challenging activities, the EEG workload levels were highest. Importantly, the calculation of workload levels was achieved in near real-time, suggesting the viability of creating a closed-loop system with EEG-workload as one of the inputs. The implication is that these indices could be used to intelligently control either the speed or mode of presentation of information to aid in ensuring an operator’s performance remains uninterrupted during the most cognitively challenging aspects of a task. Further, when an operator evidences continually high or extreme workload, the system could identify when tasks might need to be delegated to another operator. In the second phase of the present study (not reported in this paper), the EEG parameters in combination with functional near infrared imaging, pupillometry, heart rate and galvanic skin response served as inputs to control various aspects of the Aegis testbed. Although a full reporting of the results is beyond the scope of this paper, the control of the pacing of information presentation in the Aegis task showed significant promise as a workload mitigation strategy to improve performance during challenging work conditions. Other potentially promising mitigation strategies, such as modalitybased task switching and intelligent task sequencing, will be investigated in future studies. Although a promising approach, the design and implementation of investigations of EEG correlates of cognitive states during complex, challenging tasks is difficult. For example, the Aegis simulation required a minimum of eight hours of training for individuals to reach adequate levels of performance. Although this level of training would not be considered unusual in an operational setting, it is atypical for laboratory experiments investigating EEG and cognition4,5,8,10-12,26-29. Furthermore, additional investigations are needed to determine whether there is generalization to other complex tasks and operational environments. 4.1 Workload indices in Aegis The Advanced Brain Monitoring high and extreme EEG workload indices accurately tracked the most cognitively challenging events in the Aegis task with a detection efficiency approaching 100% for selection-identification and alert events. Hooking and LCS queries accounted for the remainder of the periods identified as high workload by the EEG indices. The high/extreme workload periods occupied between 25-30% of total task time, leaving more than 70% of the time available for additional allocation of tasks and sub-tasks. Taken together, these results suggest an intelligent closed-loop system incorporating EEG-workload measures could be designed to re-allocate tasks, control manipulations, and thus aid in efficiently streamlining a user’s cognitive workload. Such an approach could ensure the operator remains uninterrupted during high/extreme workload periods, thereby resulting in increased productivity and reduced errors. Another important finding was that the high and extreme workload EEG not associated with any of the target Aegis events (false positives) occurred less than 5% of the time across participants and scenarios. This guarantees that the workload gauge is not overly sensitive. In addition, the higher EEG workload periods occupied between 25-30% of the scenario total time, presumably leaving more than 70% of the sessions available for additional allocation of tasks and sub-tasks. Thus, in a closed-loop scenario, the system could re-allocate tasks from periods when the operator was
overloaded or introduce tasks from alternative operators within the system when periods of low workload were identified. Although this model proved accurate in the Aegis testbed, there is no guarantee that the model will generalize to other cognitively complex task environments. Additional investigation is required to determine the applicability of the model in alternative task domains. 4.3. Workload model development In order to compare workload levels across participants and to set system thresholds for triggering manipulations, the EEG outputs in the current model were standardized to account for individual differences. In the current study a twominute moving average was selected for the threshold mean and standard deviation because of its capability to adapt to the individual. The two-minute moving average worked well within the constraints of the current study, but the adaptive model does not provide information on within-user state changes that may be important to monitor in real-world settings. Because the entire experimental test session was completed in less than two hours, the second-by-second level of task engagement was likely higher overall than might be expected in a typical eight or ten-hour shift in an operational environment. It should be noted that although the computation of the adaptive mean achieved the desired output thresholds in the current paradigm, the real-time outputs of the model could also include the unadjusted probability values computed by the DFA for each one-second of EEG data. Changes in this parameter could also be monitored to identify within-user state changes over the course of the session. The integration of physiological monitoring into the man-machine interface holds great promise for real-time assessment of operator status and the possibility of intelligently allocating tasks between machines and humans based on determined operator cognitive states and performance levels. It is envisioned that once accurate real-time monitoring is achieved, intelligent feedback or “closed-loop” systems can facilitate active intervention by the operator or through a third party (man or machine) and thus simultaneously increase safety and productivity. For instance, potential performance improvement benefits of employing effectively designed augmentations (mitigation strategies) within C2 systems, such as an Aegis CIC, include: increasing the number of critical tracks correctly identified, increasing the number of alerts successfully handled, increasing the overall number of tracks identified correctly, improving the overall situational awareness of CIC operators and, ultimately, leading to reduced manning requirements. Before the closed-loop system can be implemented in real-world scenarios, additional research must be completed to ensure that the model will have the ability to account for differences within and across tasks and participants. Although these preliminary findings suggest that within the Aegis task environment one EEG-workload model may be suitable for driving closed-loop manipulations, it is not possible to generalize to other complex cognitive tasks and operational settings until more investigations are conducted. The present study’s findings, in combination with those previously reported by the investigators, do indicate, however, that this approach to model development will prove useful in a variety of tasks and settings.
ACKNOWLEDGEMENTS This work was supported by the DARPA program “Improving Warfighter Information Intake Under Stress”, in which Advanced Brain Monitoring is a sub-contractor to Lockheed-Martin Advanced Technology Laboratories.
REFERENCES 1. 2. 3. 4.
Stanney K, Samman S, Reeves L, et al., "A paradigm shift in interactive computing; Deriving multimodal design principles from behavioral and neurological foundations", International Journal for Human-Computer Interaction Special Issue, 17, 229-257, 2004. Schmorrow D, Stanney KM, Wilson GF, et al., "Augmented cognition in human-system interaction", Handbook of human factors and ergonomics, In press. Akerstedt T, Folkard S, "The three-process model of alertness and its extension to performance, sleep latency, and sleep length", Chronobiol Int, 14, 115-123, 1997. Gevins A, Smith ME, Le J, et al., "High resolution evoked potential imaging of the cortical dynamics of human working memory", Electroencephalogr Clin Neurophysiol, 98, 327-348., 1996.
5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28.
Gevins A, Smith ME, Leong H, et al., "Monitoring working memory load during computer-based tasks with EEG pattern recognition methods", Hum Factors, 40, 79-91., 1998. Gevins A, Smith ME, McEvoy L, et al., "High-resolution EEG mapping of cortical activation related to working memory: effects of task difficulty, type of processing, and practice", Cereb Cortex, 7, 374-385, 1997. Fabiani M, Gratton G, Coles MG, "Event-related Brain Potentials", Handbook of Psychophysiology, Caciooppo JT, Tassinary LG, Berntson GG, 53-84, Cambridge University Press, Cambridge, 2000. Wilson GF, Eggemeier FT, Physiological measures of workload in multi-task environments, 329-360, London, 1991. Davidson RJ, Jackson DC, Larson CL, "Human Electroencephalography", Handbook of Psychophysiology, Caciooppo JT, Tassinary LG, Berntson GG, 27-52, Cambridge University Press, Cambridge, 2000. Makeig S, Inlow M, "Lapses in alertness: coherence of fluctuations in performance and EEG spectrum", Electroencephalogr Clin Neurophysiol, 86, 23-35, 1993. Makeig S, Jung TP, "Changes in alertness are a principal component of variance in the EEG spectrum", Neuroreport, 7, 213-216, 1995. Makeig S, Jung TP, "Tonic, phasic, and transient EEG correlates of auditory awareness in drowsiness", Brain Res Cogn Brain Res, 4, 15-25, 1996. Brookings JB, Wilson GF, Swain CR, "Psychophysiological responses to changes in workload during simulated air traffic control", Biol Psychol, 42, 361-377., 1996. Sterman MB, Mann CA, "Concepts and applications of EEG analysis in aviation performance evaluation", Biol Psychol, 40, 115-130. Review., 1995. Sterman MB, Mann CA, Kaiser DA, "Quantitative EEG patterns of differential in-flight workload", Space Operations, Applications, and Research Proceedings, 3187, 466-473, NASA Conference Publication, Sepulveda VA Medical Center, 1992. Brookhuis KA, de Waard D, "The use of psychophysiology to assess driver status", Ergonomics, 36, 10991110., 1993. Berka C, Levendowski DJ, Olmstead RE, et al., "Real-time Analysis of EEG Indices of Alertness, Cognition and Memory with a Wireless EEG Headset", International Journal of Human-Computer Interaction, 17, 151170, 2004. Pleydell-Pearce CW, Whitecross SE, Dickson BT, "Multivariate analysis of EEG: Predicting cognition on the basis of frequency decomposition, inter-electrode correlation, coherence, cross phase, and cross power", Proceedings of the 36th Annual Hawaii International Conference on System Sciences, 131, Hawaii, 2003. Levendowski DJ, Berka C, Olmstead RE, et al., "Correlations between EEG Indices of Alertness Measures of Performance and Self-Reported States while Operating a Driving Simulator", 29th Annual Meeting, Society for Neuroscience, 25, Miami Beach, FL, 1999. Levendowski DJ, Berka C, Olmstead RE, et al., "Electroencephalographic indices predict future vulnerability to fatigue induced by sleep deprivation." Sleep, 24, A243-A244, 2001. Levendowski DJ, Olmstead RE, Konstantinovic ZR, et al., "Detection of Electroencephalographic Indices of Drowsiness in Real-time using a Multi-Level Discriminant Function Analysis", Sleep, 23, A243-A244, 2000. St. John M, Kobus DA, Morrison JG, et al., "Overview of the DARPA augmented cognition technical integration experiment", International Journal for Human-Computer Interaction Special Issue, 17, 131-149, 2004. Mathiak K, Hertrich I, Kincses WE, et al., "Involuntary crossmodal enhancement of preattentive auditory processing is selective in space", Brain Topogr, 16, 200, 2004. Levendowski DJ, Westbrook P, Berka C, et al., "Event-related potentials during a test of working memory differentiate sleep apnea patients from healthy subjects", Sleep, 25, A460-A461, 2002. Mitler MM, Westbrook P, Levendowski DJ, et al., "Validation of automated EEG quantification of alertness: methods for early identification of individuals most susceptible to sleep deprivation", Sleep, 25, A147-A148, 2002. Prinzel LJ, Freeman FG, Scerbo MW, et al., "A closed-loop system for examining psychophysiological measures for adaptive task allocation", Int J Aviat Psychol, 10, 393-410., 2000. Pope AT, Bogart EH, Bartolome DS, "Biocybernetic system evaluates indices of operator engagement in automated task", Biol Psychol, 40, 187-195., 1995. Gevins. Spatial measurement of EEG electrodes: Sam Technology, 1997
29.
Kramer HC, Trejo LJ, Humphrey DG, Psychophysiological measures of workload: Potential applications to adaptively automated systems, Lawrence Erlbaum Associates, New Jersey, 1996.
