Using Brain Activity to Predict Task Performance and Operator ...

Using Brain Activity to Predict Task Performance and Operator Efficiency Hasan Ayaz1,2, Scott Bunce2,3, Patricia Shewokis1,2,4, Kurtulus Izzetoglu1,2, Ben Willems5, and Banu Onaral1,2 1

School of Biomedical Engineering, Science & Health Systems, Cognitive Neuroengineering and Quantitative Experimental Research (CONQUER) Collaborative, Drexel University, Philadelphia, PA, 19104, USA 3 Penn State Hershey Medical Center and Penn State College of Medicine, Hershey, PA 17003 USA, 4 College of Nursing and Health Professions, Drexel University, Philadelphia PA 19102, USA 5 Federal Aviation Administration William J. Hughes Technical Center, Atlantic City International Airport, NJ, USA (ayaz,pas38,ki25,banu.onaral)@drexel.edu, [email protected], [email protected] 2

Abstract. The efficiency and safety of many complex human-machine systems are closely related to the cognitive workload and situational awareness of their human operators. In this study, we utilized functional near infrared (fNIR) spectroscopy to monitor anterior prefrontal cortex activation of experienced operators during a standard working memory and attention task, the n-back. Results indicated that task efficiency can be estimated using operator’s fNIR and behavioral measures together. Moreover, fNIR measures had more predictive power than behavioral measures for estimating operator’s future task performance in higher difficulty conditions. Keywords: fNIR, optical brain imaging, working memory, task efficiency, cognitive workload.

1

Introduction

To maximize efficiency and minimize error, an ideal human-machine system should be designed to maintain operator mental workload at an optimum level, which requires an accurate sensing and continuous update of operator workload. Such an informed mechanism is even more crucial when safety, efficiency and time-critical missions depend on the operation of the human-machine system. Importantly, behavioral measures (or task performance) alone may not be sufficiently sensitive to index overload, as operators can extend their work effort to maintain system performance, but this could come at a cost that may be reflected only in neural measures. H. Zhang et al. (Eds.): BICS 2012, LNAI 7366, pp. 147–155, 2012. © Springer-Verlag Berlin Heidelberg 2012

148

H. Ayaz et al.

The advent of new and improved technologies that allow the monitoring of brain activity in natural environments is expected to enable better identification of neurophysiological markers of human performance [1]. Functional magnetic resonance imaging (fMRI), positron emission tomography (PET) and Magnetoencephalography (MEG) have contributed a great deal to our current understanding of the neural basis of mental workload. Unfortunately, these techniques are not amenable to ecologically valid operational settings, as they variously are highly sensitive to motion artifact, require participants in confined positions, expose individuals to potentially harmful materials or loud noise, and are quite expensive. Other technologies with affordable and portable use potential that have been investigated for the purpose of workload assessment, such as electroencephalography (EEG) / eventrelated brain potentials (ERPs), can directly measure the summation of neural function with temporal resolution on the order of milliseconds. However, these technologies also have limited spatial resolution [2], and are susceptible to electromagnetic field artifacts and spatially related muscle movements. Functional Near-Infrared Spectroscopy (fNIR) is an emerging optical brain imaging technology that relies on optical techniques to detect changes of hemodynamic responses within the cortex in response to sensory, motor, or cognitive activation [2-9]. In its most common form factor, fNIR uses near infrared light absorption changes within the observed brain area to monitor cortical concentration changes of oxygenated hemoglobin (oxy-Hb) and deoxygenated hemoglobin (deoxyHb). For a review of fNIR please see [9-13]. fNIR technology allows the design of battery-operated, safe and ambulatory monitoring systems, qualities that position fNIR as an ideal candidate for monitoring cognition-related hemodynamic changes under working conditions as well as the laboratory. Recent studies at the Federal Aviation Administration (FAA) William J. Hughes Technical Center’s Research, Development, and Human Factors Laboratory have utilized fNIR to monitor certified air traffic controllers as they manage realistic scenarios under typical and emergent conditions. As part of these studies, certified controllers performed an n-back task, a standard attention and working memory (WM) task with 4 levels of difficulty. Earlier reports based on this dataset indicated that average oxygenation changes at Optode 2, located within left inferior frontal gyrus in the dorsalateral prefrontal cortex, correlated with the task difficulty and increased monotonically with increasing task difficulty [14]. At incremental levels of task difficulty (cognitive task or air traffic control scenarios), there were differences in the neural activation of cortical areas known to be associated with cognitive workload when a trained controller is operating. The purpose of this study was to assess the mental efficiency of operators utilizing both behavioral measures and a neurocognitive measure, i.e, an fNIR measure of hemodynamic response. We also evaluated the capacity to use the level of neural activation at a given level of task difficulty to predict performance at more difficult levels of task workload. In part, WM refers to the cognitive process of actively maintaining task related information in mind for brief periods of time. With consistent levels of increasing task difficulty, the n-back provided an opportunity to test the relationship and predictive power of activation in specified brain regions related to attention and working memory relative to operator performance. An efficiency graph can be a useful visual tool to assess the impact of learning on performance as defined

Using Brain Activity to Predict Task Performance and Operator Efficiency

149

by Clark et al. in [15]. The efficiency can be visualized by having the effort on the xaxis as input and task performance on the y-axis as output. Here, we used the construct of efficiency to determine if greater efficiency, measured at one level, would predict better operator performance at the next level of difficulty.

2

Method

2.1

Participants

Twenty-four certified professional controllers (CPC) between the ages of 24 and 55 volunteered. All participants were non-supervisory CPC with a current medical certificate and had actively controlled traffic in an Air Route Traffic Control Center between 3 to 30 years. Prior to the study, all participants signed informed consent forms. 2.2

Experiment Protocol

The n-back task used in this study included four incremental levels of difficulty [1618]. During the task, participants were asked to visually monitor single letters presented serially on a screen and to press a button with their dominant hand when a target stimulus was identified. Targets were variously defined across the four conditions so as to incrementally increase WM load from zero to three items. In the 0back condition, the target was defined as a single, pre-specified letter (e.g., “X”). In the 1-back condition, the target was defined as any letter identical to the one immediately preceding it (i.e., one trial back). In the 2-back and 3-back conditions, targets were defined as any letter that was identical to the one presented two or three trials back, respectively. The total test included four sessions of each of the four conditions (a total of 16 blocks) presented in a pseudo-random order. The task was implemented in E-prime (Psychology Software Tools). 2.3

Optical Brain Monitoring

During the experiment, the prefrontal cortex of each participant was monitored using a continuous wave fNIR system first described by Chance et al. [5], further developed at Drexel University (Philadelphia, PA), manufactured and supplied by fNIR Devices LLC (Potomac, MD; www.fnirdevices.com). The system was composed of three modules: a flexible sensor pad, which holds light sources and detectors to enable the rapid placement of 16 optodes (channels); a control box for hardware management; and a computer running data acquisition software (COBI Studio) [10] (Figure1). The system operated with a sampling frequency of 2Hz. The light emitting diodes (LED) were activated one light source at a time and the four surrounding photodetectors around the active source were sampled. One scan was completed once all four LEDs were activated and respective detectors were sampled sequentially. The positioning of the light sources and detectors on the sensor pad yielded a total of 16 active optodes (channels) and was designed to monitor dorsal and inferior frontal cortical areas underlying the forehead.

150

H. Ayaz et al.

Fig. 1. fNIR sensor pad that measures from 16 locations over the forehead (right, top), measurement locations (optodes) on brain surface image [19] (left, top), and fNIR sensor pad contains 4 light sources and 10 light detectors forming 16 optodes (bottom) [10]

3

Data Analysis

To attenuate high frequency noise, respiration and cardiac cycle effects, raw fNIR data (16 optodes x 2 wavelengths) were filtered with a low-pass finite impulse response, linear phase filter with order 20 and cut-off frequency of 0.1 Hz [20]. Saturated channels (if any), in which light intensity at the detector was higher than the analog-to-digital converter limit, were excluded. fNIR data epochs for task and rest periods were extracted from the continuous data using time synchronization markers. Blood oxygenation and volume changes within each of the 16 optodes were calculated using the modified Beer-Lambert Law for task periods with respect to rest periods at beginning of each task using fnirSoft software [21]. For efficiency analysis, oxygenation data and behavioral measures for each subject were normalized by calculating the respective z-scores. The main effect for task difficulty was tested using one-way repeated measures analysis of variance (ANOVA), with Subject and Task Difficulty designated as fixed effects. Geisser– Greenhouse (G–G) correction was used when violations of sphericity occurred in the omnibus tests. Tukey's post hoc tests were used to determine the locus of the main effects with a 0.05 significance criterion.

Using Brain Activity to Predict Task Performance and Operator Efficiency

4

151

Results

The behavioral data, as reported before, indicated a significant main effects of task difficulty (0-, 1-, 2- and 3-back conditions) for accuracy (correct click for targets) (F3,69=40.68, p< 0.001 , ηp2 = 0.639) and reaction time (F3,69=42.76, p< 0.001, ηp2 = 0.65). Moreover, task average fNIR data revealed that reliable changes in oxygenation as a function of n-back condition occurred only at optode #2 (F3,69= 4.37, p < 0.05 , ηp2 = 0.16). This site, close to AF7 in the International 10-20 System, is located within the left PFC (inferior frontal gyrus). In correspondence with previous results and our previous report, increasing task difficulty was accompanied by an increase in activation level during task performance [14]. An efficiency graph can be a useful visual tool to assess the impact of learning on performance as defined by Clark et al. in [15]. The efficiency can be visualized by having the effort on the x-axis as input and task performance on the y-axis as output. The efficiency graph in Figure 2 visualizes our overall results by plotting normalized oxygenation changes (that represent mental effort) against the normalized ‘hit’ ratio (that model behavioral performance). In this efficiency graph, the fourth quadrant (lower right) represents low efficiency, where minimum performance is achieved with maximum effort. The second quadrant (upper left) represents high efficiency where maximum performance is achieved with minimal effort. The diagonal y=x is the neutral axis, where efficiency is zero and effort and performance are equal.

Fig. 2. Efficiency graph of performance (behavioral) vs. effort (neural measures) indicates that as task difficulty increased, efficiency monotonically decreased

The linear regression analysis of performance and oxygenation changes for each condition also indicates a negative correlation for the last two conditions (see Figure 3). For 0-back and 1-back, there was a ceiling effect with the highest performance level which, most participants easily achieved. However, for 2-back and 3-back, a similar overall trend of negative correlation is observed with r=0.23 (rmse=0.25; slope=-23.4) for 2-back and and r = 0.44 (rmse=0.19; slope=-37.1) for 3-back.

152

H. Ayaz et al.

Fig. 3. First order linear regression for oxygenation vs. task performance for all n-back conditions

Next, we assessed the capacity to use the degree of neural activation required to perform at a given level of task difficulty to predict behavioral performance at a greater level of task difficulty. The 2-back condition is ideal for this purpose, as it provides a level of difficulty that most operators are able to manage behaviorally through sustained effort, whereas the next level, the 3-back, is significantly more difficult and represented a level of difficulty in which participants had the highest probability of disengaging. 0-back and 1-back had ceiling effects in task performance. Neural activation (oxygenation) in the 2-back condition was shown to be a better predictor of 3-back task performance (r = 0.57) than the performance score in the 2-back condition (r=0.33; See Figure 4).

Fig. 4. Relationship of 2-back neural measures (oxy) with 3-back task performance (left) and 2back and 3-back performance (right)

Using Brain Activity to Predict Task Performance and Operator Efficiency

5

153

Discussion and Conclusion

In this study, a standard visual WM task was used to assess, the capacity to utilize objective measures of brain activation to predict performance at greater levels of task difficulty. Operator efficiency was evaluated at four levels of task difficulty using an objective measure of effort (fNIR measures of oxygenation) versus behavioral measures of task performance. As expected, efficiency dropped as task difficulty increased, indicating that participants needed to exert more effort to achieve similar levels of performance at more difficult task levels, One important aspect of this finding is that the cognitive effort was assessed using a noninvasive, objective measure that did not require operator resources to monitor (as required by self-report measures). [14, 22, 23]. More important, however, was the finding that the degree of neural activation at one level of task difficulty predicted performance at the next level of task difficulty. fMRI has previously been used to demonstrate that neural activation during a visuospatial WM task can be used to predict performance in another cognitive domain, i.e., arithmetic [24]. In the current study, brain activation in the 2-back condition was a better predictor of performance in the 3-back condition than the 2back performance score, indicating that the neural measure was more sensitive than the behavioral measure. This finding suggests the utility of fNIR as an objective measure of cerebral hemodynamics and its potential role in human-machine interfaces. The current results suggest important, albeit preliminary, information about the relationship between fNIR measures of anterior prefrontal cortical hemodynamics and the performance of an attentional WM task. Although previous results suggest that it is possible to use brain activity in WM to predict another cognitive task performance [24], it remains to be determined if the relationship between task performance and brain activation is sensitive enough to be of practical use in a more complex, realworld task such as air traffic control. In summary, fNIR is a portable, safe and minimally intrusive optical brain monitoring technology that can be used to measure hemodynamic changes in the brain’s outer cortex. Changes in blood oxygenation in dorsolateral prefrontal cortex, as measured by fNIR, were shown to be associated with increasing cognitive workload [14, 23] and can be used to assess skill acquisition/learning [22, 25]. The current analysis suggested that fNIR can be used to predict future task performance for the optimization of learning/training regimens. Further work is needed to assess if these results can be generalized to other types of cognitive tasks, especially complex and realistic daily life scenarios. Acknowledgments. The authors would like to thank Dr. Sehchang Hah and Atul Deshmukh for data collection. This work was supported by the U.S. Federal Aviation Administration through BAE Systems Technology Solutions Services Inc. under Primary Contract, DTFA01-00-C-00068 and Subcontract Number, 31-5029862.

154

H. Ayaz et al.

References 1. Parasuraman, R., Wilson, G.: Putting the brain to work: Neuroergonomics past, present, and future. Human Factors 50, 468 (2008) 2. Strangman, G., Boas, D.A., Sutton, J.P.: Non-invasive neuroimaging using near-infrared light. Biological Psychiatry 52, 679–693 (2002) 3. Chance, B., Anday, E., Nioka, S., Zhou, S., Hong, L., Worden, K., Li, C., Murray, T., Ovetsky, Y., Pidikiti, D., Thomas, R.: A novel method for fast imaging of brain function, non-invasively, with light. Optics Express 2, 411–423 (1998) 4. Villringer, A., Planck, J., Hock, C., Schleinkofer, L., Dirnagl, U.: Near infrared spectroscopy (NIRS): a new tool to study hemodynamic changes during activation of brain function in human adults. Neuroscience Letters 154, 101–104 (1993) 5. Chance, B., Zhuang, Z., UnAh, C., Alter, C., Lipton, L.: Cognition-activated lowfrequency modulation of light absorption in human brain. Proceedings of the National Academy of Sciences of the United States of America 90, 3770–3774 (1993) 6. Coyle, S., Ward, T.E., Markham, C.M.: Brain-computer interface using a simplified functional near-infrared spectroscopy system. Journal of Neural Engineering 4, 219–226 (2007) 7. Obrig, H., Wenzel, R., Kohl, M., Horst, S., Wobst, P., Steinbrink, J., Thomas, F., Villringer, A.: Near-infrared spectroscopy: does it function in functional activation studies of the adult brain? International Journal of Psychophysiology 35, 125–142 (2000) 8. Irani, F., Platek, S.M., Bunce, S., Ruocco, A.C., Chute, D.: Functional near infrared spectroscopy (fNIRS): an emerging neuroimaging technology with important applications for the study of brain disorders. Clin. Neuropsychol. 21, 9–37 (2007) 9. Hoshi, Y.: Functional near-infrared spectroscopy: current status and future prospects. Journal of Biomedical Optics 12, 062106 (2007) 10. Ayaz, H., Shewokis, P.A., Curtin, A., Izzetoglu, M., Izzetoglu, K., Onaral, B.: Using MazeSuite and Functional Near Infrared Spectroscopy to Study Learning in Spatial Navigation. J. Vis. Exp. e3443 (2011) 11. Izzetoglu, M., Izzetoglu, K., Bunce, S., Ayaz, H., Devaraj, A., Onaral, B., Pourrezaei, K.: Functional near-infrared neuroimaging. IEEE Trans. Neural Syst. Rehabil. Eng. 13, 153– 159 (2005) 12. Orihuela-Espina, F., Leff, D., James, D., Darzi, A., Yang, G.: Quality control and assurance in functional near infrared spectroscopy (fNIRS) experimentation. Physics in Medicine and Biology 55, 3701 (2010) 13. Izzetoglu, K., Ayaz, H., Merzagora, A., Izzetoglu, M., Shewokis, P.A., Bunce, S.C., Pourrezaei, K., Rosen, A., Onaral, B.: The evolution of field deployable fNIR spectroscopy from bench to clinical settings. Journal of Innovative Optical Health Sciences 4, 1–12 (2011) 14. Ayaz, H., Shewokis, P.A., Bunce, S., Izzetoglu, K., Willems, B., Onaral, B.: Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59, 36–47 (2012) 15. Clark, R., Nguyen, F., Sweller, J.: Efficiency in learning: Evidence-based guidelines to manage cognitive load. Pfeiffer, An Imprint of Wiley, San Fransico (2006) 16. Owen, A.M., McMillan, K.M., Laird, A.R., Bullmore, E.: N-back working memory paradigm: a meta-analysis of normative functional neuroimaging studies. Human Brain Mapping 25, 46–59 (2005)

Using Brain Activity to Predict Task Performance and Operator Efficiency

155

17. D’Esposito, M., Aguirre, G., Zarahn, E., Ballard, D., Shin, R., Lease, J.: Functional MRI studies of spatial and nonspatial working memory. Cognitive Brain Research 7, 1–13 (1998) 18. Smith, E.E., Jonides, J.: Working Memory: A View from Neuroimaging. Cognitive Psychology 33, 5–42 (1997) 19. Ayaz, H., Izzetoglu, M., Platek, S.M., Bunce, S., Izzetoglu, K., Pourrezaei, K., Onaral, B.: Registering fNIR data to brain surface image using MRI templates. In: Conf. Proc. IEEE Eng. Med. Biol. Soc., pp. 2671–2674 (2006) 20. Ayaz, H., Izzetoglu, M., Shewokis, P.A., Onaral, B.: Sliding-window Motion Artifact Rejection for Functional Near-Infrared Spectroscopy. In: Conf. Proc. IEEE Eng. Med. Biol. Soc., pp. 6567–6570 (2010) 21. Ayaz, H.: Functional Near Infrared Spectroscopy based Brain Computer Interface. School of Biomedical Engineering Science & Health Systems, p. 214. Drexel University, Philadelphia (2010) 22. Ayaz, H., Cakir, M.P., Izzetoglu, K., Curtin, A., Shewokis, P.A., Bunce, S., Onaral, B.: Monitoring Expertise Development during Simulated UAV Piloting Tasks using Optical Brain Imaging. In: IEEE Aerospace Conference, BigSky, MN, USA, pp. 1–11 (2012) 23. Izzetoglu, K., Bunce, S., Onaral, B., Pourrezaei, K., Chance, B.: Functional Optical Brain Imaging Using Near-Infrared During Cognitive Tasks. International Journal of HumanComputer Interaction 17, 211–227 (2004) 24. Dumontheil, I., Klingberg, T.: Brain Activity during a Visuospatial Working Memory Task Predicts Arithmetical Performance 2 Years Later. Cerebral Cortex (2011) 25. Shewokis, P.A., Ayaz, H., Izzetoglu, M., Bunce, S., Gentili, R.J., Sela, I., Izzetoglu, K., Onaral, B.: Brain in the Loop: Assessing Learning Using fNIR in Cognitive and Motor Tasks. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) FAC 2011, HCII 2011. LNCS, vol. 6780, pp. 240–249. Springer, Heidelberg (2011)