School of Biomedical Engineering, Science, and Health Systems
Drexel E-Repository and Archive (iDEA) http://idea.library.drexel.edu/
Drexel University Libraries www.library.drexel.edu
The following item is made available as a courtesy to scholars by the author(s) and Drexel University Library and may contain materials and content, including computer code and tags, artwork, text, graphics, images, and illustrations (Material) which may be protected by copyright law. Unless otherwise noted, the Material is made available for non profit and educational purposes, such as research, teaching and private study. For these limited purposes, you may reproduce (print, download or make copies) the Material without prior permission. All copies must include any copyright notice originally included with the Material. You must seek permission from the authors or copyright owners for all uses that are not allowed by fair use and other provisions of the U.S. Copyright Law. The responsibility for making an independent legal assessment and securing any necessary permission rests with persons desiring to reproduce or use the Material.
Please direct questions to
[email protected]
Proceedings of the 3rd International IEEE EMBS Conference on Neural Engineering Kohala Coast, Hawaii, USA, May 2-5, 2007
FrD3.7
Detecting cognitive activity related hemodynamic signal for brain computer interface using functional near infrared spectroscopy Hasan Ayaz, Meltem Izzetoglu, Scott Bunce, Terry Heiman-Patterson, and Banu Onaral, Fellow, IEEE
Abstract— The ideal non-invasive brain computer interface (BCI) transforms signals originating from human brain into commands that can control devices and applications. Hence, BCI provides a way for brain output that does not involve neuromuscular system. This represents an advantage for those individuals suffering from neuromuscular impairments such as Amyotrophic Lateral Sclerosis (ALS) or various types of paralysis. In this study we propose to design a new noninvasive BCI that is based on optical means to measure brain activity by monitoring hemodynamic response. The proposed system uses functional near infrared (fNIR) spectroscopy to detect cognitive activity from prefrontal cortex elicited voluntarily by performing a mental task namely N-back test. Our findings indicate that fNIR signal correlates with cognitive tasks associated with working memory. These experimental outcomes compare favorably with previous functional magnetic resonance imaging (fMRI) and complement electroencephalogram (EEG) findings. Since fNIR can be implemented in the form of a wearable and minimally intrusive device, it also has the capacity to monitor brain activity under real life conditions in everyday environments leading the way to potential applications of fNIR in BCI development for communication and entertainment purposes.
I. INTRODUCTION An individual’s communication with the outside world can be ceased due to complete paralysis, locked-in syndrome, spinal cord injury or muscular dystrophy. Individuals suffering from such diseases, though conscious, may lose all voluntary muscle control and thus are often unable to relay even their most basic wishes. Unlike persistent vegetative state, in which the upper portions of the brain are damaged and the lower portions are spared, inability is caused by damage to specific portions of the lower brain and brainstem or muscles with no damage to the upper brain, which means that their cognitive abilities are intact. Manuscript received Feb 19, 2007. This work has been sponsored in part by funds from the Defense Advanced Research Projects Agency(DARPA) Augmented Cognition Program, the Office of Naval Research (ONR) and Department of Homeland Security (DHS), under agreement numbers N00014-02-1-0524, N00014-01-1-0986 and N00014-04-1-0119 B. Onaral, M. Izzetoglu M and H. Ayaz is with the Drexel University School of Biomedical Engineering Science and Health System, Philadelphia, 19104 USA (corresponding author mail:
[email protected]). T. Heiman-Patterson is with the Drexel College of Medicine. Neurology Dept., and S. Bunce is with the Department of Psychiatry, Drexel University College of Medicine, Philadelphia, PA 19102.
1-4244-0792-3/07/$20.00©2007 IEEE.
Brain Computer Interface (BCI) is defined as a system or approach that translates neurophysiological signals detected from brain to supply input to a computer or control a device. BCI is a relatively new research area that largely targets to eliminate the need for motor movement and develop mechanisms to relay information directly from brain to a computer or to the outside world. In addition to their use in neuroprosthetics, noninvasive BCI systems also have potential applications for healthy individuals especially for enhancing or accelerating the learning process, or in entertainment domain such as in computer games and multimedia applications as a neurofeedback mechanism. Development of alternative communication strategies are a recognized need for clinical applications. A technique that bypasses muscles and acquires signals directly from brain would be a notable help. Moreover, this technique should be minimally intrusive, non-invasive, accessible, and safe to be used continuously. The most commonly studied interface to monitor brain activity noninvasively has been Electroencephalogram (EEG), due to its fine temporal resolution, portability and cost of setup [1,2]. Various electrode placement schemes and advanced signal processing methods have been researched for its improved and practical use in BCI applications [2,3]. However, these EEG based systems still have certain drawbacks. For example, the end-user has to develop a new thinking mechanism to be able to interact with the EEG based BCI system which results in lengthy training time [4]. Furthermore, non-invasive EEG recordings are highly susceptible to noise and hence have much lower signal to noise ratio as compared to signals recorded from implanted electrodes [5]. In addition, EEG is cumbersome to use in practice due to the need for applying gel and restrictions on users’ movements. Therefore, existing BCI systems do not yet meet the desired characteristics that the optimal BCI should have. In fact, they are either invasive and hence not yet completely safe for continuous use or they are non-invasive but rely on a noisy signal and require mental adaptation mechanisms. In order to partially overcome the problems of existing BCI and provide an alternative communication mechanism for
342
individuals with locked-in syndromes, we propose to use continuous wave functional near infrared (fNIR) spectroscopy as a new functional neuroimaging modality for Brain Computer Interface. fNIR has emerged in the last decade as a non-invasive, safe, portable and accessible brain monitoring modality [6-13]. There are recent evidences showing that fNIR can be used for the assessment of attention [14] and cognitive task load [15,16]. In 2001, it is demonstrated by Hitachi Ltd. [17] that fNIR can be used as a communication tool for ALS (Lou Gehrig's disease) patients. In more recent studies, the suitability of optical methods for BCI has been investigated by Coyle et al (2004) [4], Ranganatha et al (2005) [5] and Sitaram et al (2007) [18]. Tough there are challenges for fNIR to be used as a basis for BCI, it also has the capability to provide an alternative signal source. Not only fNIR is non-invasive and safe, but also it would be possible to build an fNIR system that is battery-operated, miniaturized and communicate wirelessly to a base server, thus allowing for increased portability and usability, and therefore, allowing for an improvement in the life quality of patients with disabilities. All the described features highlight the potential of fNIR in BCI applications. Nevertheless, fNIR has been only marginally investigated for BCI applications. So far, only a few studies have been published [4, 5, 18]. Moreover, these studies have focused on motor tasks and imagery and they are based on offline analysis. The overall aim of our project is to build an fNIR based BCI system that will be operated by the voluntary activation of brain signals through the performance of selected cognitive tasks. This way, even the patients who only have their cognitive abilities intact can communicate with the outside world. As a first step to reach this broader aim, in this preliminary study, we investigate the potential of fNIR in the assessment of cognitive activity for various task loads. Our results based on a working memory task, namely N-back test suggest that increased cognitive task load results in an increase in oxygenation within the frontal lobe and this can be detected using the fNIR system. II.
FNIR SPECTROSCOPY
fNIR is a multi-wavelength optical spectroscopy technique introduced as a non-invasive brain activity monitoring modality [6-9]. fNIR can assess temporal progression of brain activity, through the measurement of hemodynamic changes within reasonable spatial resolution. Neuronal activity is determined with respect to the changes in oxygenation since variation in cerebral hemodynamics are related to functional brain activity through a mechanism which is known as neurovascular coupling [7]. fNIR is not only non-invasive, safe, affordable and portable [19, 20],
343
but also provides a balance between temporal and spatial resolution which makes fNIR a viable option for in-thefield neuroimaging. Typically, an optical apparatus consists of a light source and a light detector that receives light after it has interacted with the tissue. Photons that enter tissue undergo two different types of interaction: absorption and scattering [10,21]. Most biological tissues are relatively transparent to light in the near infrared range between 700 to 900 nm which is usually called the “optical window”. This is mainly due to the low absorbance of the main constituents allowing the light to penetrate the tissue (See Figure 1).
Near IR Optical Window
Figure 1. Absorption spectrum of main chromophores in the tissue The main absorbers (chromophores) in the tissue, oxyand deoxy-Hb are strongly linked to tissue oxygenation and metabolism [7]. Fortunately, in the optical window, the absorption spectra of oxy- and deoxy-Hb remain significantly different than each other allowing spectroscopic separation of these compounds to be possible using only a few sample wavelengths. Once the photons are introduced into the human head, they are either scattered by extra- and intracellular boundaries of different layers of the head (skin, skull, cerebrospinal fluid, brain, etc.) or absorbed mainly by oxy- and deoxy-Hb. A photodetector placed a certain distance away from the light source can collect the photons that are not absorbed and those that traveled along the “banana shaped path” between the source and detector due to scattering [6-13]. III. MATERIALS AND METHODS A. fNIR System There are three distinct types of fNIR implementations; time domain, frequency domain and continuous wave systems [10]. In this study we have used continuous wave (CW) system, in which light is applied to tissue at constant amplitude. Although, CW systems are limited to measuring only the amplitude attenuation of the incident light [22], they possess a number of advantageous properties that have
resulted in wide use among researchers interested in brain imaging relative to other near-infrared systems [22,23]. The cw-fNIR system used in this study had a flexible sensor consisted of 4 light sources having 3 built in LEDs with peak wavelengths at 730, 805, 850 nm and 10 detectors designed to image cortical areas underlying the forehead. With a fixed source-detector separation of 2.5 cm, this configuration generated a total of 16 signal channel measurements. The sampling rate of the system was 2Hz [13,16,25]. B. Subjects Eight healthy subjects of no neurological or psychiatric history (ages between 18 to 25) voluntarily participated in the study. All subjects gave written informed consent approved by the institutional review board of Drexel University for the experiment. C. Experiment Protocol fNIR signal has been collected from each subject through the sensor pad attached to the forehead while they are performing N-back test which is a commonly used task in working memory studies [24]. The stimuli are single consonants presented centrally, in pseudorandom sequences, on a computer monitor. Stimulus duration is 500 ms, with a 2500ms interstimulus interval. Four conditions were used to incrementally vary working memory load from zero to three items. In the 0-back condition, subjects respond to a single prespecified target letter (e.g., ‘‘X’’) with their dominant hand (pressing a button to identify the stimulus). In the 1-back condition, the target is defined as any letter identical to the one immediately preceding it (i.e., one trial back). In the 2-back and 3-back conditions, the targets were defined as any letter that was identical to the one presented two or three trials back, respectively. Subjects pressed one button for targets (approximately 33% of trials) and another for nontargets. The button pressing data for this preliminary experiment is for evaluating the subject performance and engagement level. Each n-back block contained 20 letters whether target or nontarget and lasted for 60s with 15s of rest periods between n-back blocks. Total test included 7 trials of each of the 4 n-back conditions (hence total of 28 n-back blocks) ordered in such a way that within one trial all 4 of the n-back conditions are presented however their order is changed randomly from trial to trial. D. Signal Analysis The raw optical intensity values in two wavelengths (730nm and 850nm) are recorded by the fNIR system on a total of 8 subjects. The physiologically irrelevant data (such as respiration and heart pulsation effects) and equipment noise, etc. is first eliminated from the raw fNIR measurements by using a low-pass filter with a cut-off frequency of 0.14Hz [13]. Then, for each n-back condition out of 7 trials, outliers are eliminated (the ones having a
344
mean that is 2.5 standard deviation away from the overall mean) and the resulting trials are averaged for each of the 16 voxels separately. Once the oxygenation data is obtained using modified Beer-Lambert law [21,23] on these averaged raw data, the overall mean for each n-back block is calculated and used as a feature for comparison purposes. IV. RESULTS AND DISCUSSION Statistically significant differences between the n-back conditions are obtained on the 4th voxel fNIR measurements. The location of 4th voxel measurement registered on the brain surface is as shown in figure 1. (Interested readers can find a more detailed explanation of our fNIR data registration and visualization scheme in Ayaz et al. (2006) [25]. This result is in agreement with the fMRI literature [26].
Figure 2 Location of fNIR Voxel 4 Statistical analysis revealed that the 0-back condition differed from 1-, and 2-back conditions; 1-back > 0-back, t=3.21, p=.012; 2-back > 0-back, t=2.58, p=.032. And, 1and 2-back differed from each other; 2-back > 1-back, t=2.77, p=0.0275. However, 3-back case is not significantly higher that 2-back condition since most subjects disengage due to difficulty of the 3-back condition.
Figure 3 Mean oxygenation of each subject for all tasks Mean oxygenation data for 8 subjects individually and averaged mean oxygenation data across 8 subjects for each workload condition are presented in figure 3 and 4, respectively. A positive relationship between increasing workload and the oxygenation is observed in dorsolateral prefrontal cortex, again in agreement with fMRI studies [26].
[9]
[10] [11]
Figure 4 Mean oxygenation of each task for all subjects
[12] [13]
V. FUTURE WORK This experiment suggests that N-back signal can trigger an increase of oxygenation within the frontal lobe and this can be detected using the fNIR system. Next step towards building of a BCI system is to utilize this finding within a binary selection scheme where user is up to select an answer of yes/no or select from two options such as left or right. In this scheme, user performs the N-back task to answer yes (or select left) and simply do nothing to answer no (or select right). When N-back is performed, oxygenation is expected to increase which will be detected by the fNIR system whereas for the other case, no significant change is expected to be present in the fNIR signal. One major question is whether we can collect repatable fNIR signals that can consistently generate similar changes for the same task over longer time period. Recent studies addressed this issue and showed that fNIR signals are in fact repeatable [27]. Another challenge is the speed. Since averaging is required for more robust results, an answer might be expected to form in the order of minutes. Our future work aims to investigate the effects of different task durations required for robust result generation and the use of pattern recognition techniques to shorten it.
[14]
[15] [16] [17]
[18]
[19] [20] [21]
VI. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8]
Wolpaw, J.R. & McFarland, D.J. (1994), Multichannel EEG-based brain-computer communication. Electroencephalography and Clinical Neurophysiology 90:444–449. Wolpaw J. R., Birbaumer N., McFarland D. J., Pfurtscheller G., and Vaughan T. M., (2002) "Brain-computer interfaces for communication and control. Clinical Neurophysiol, v. 113, p 767-91. Birbaumer N., Ghanayim N., Hinterberger T., Iversen I., Kotchoubey B., Kubler A., Perelmouter J., Taub E., and Flor H., (1999) "A spelling device for the paralysed," Nature, vol. 398, pp. 297-8. Coyle S., Ward T., Markham C., and McDarby G., (2004), On the suitability of near-infrared (NIR) systems for next-generation braincomputer interfaces, Physiol Meas, vol. 25, pp. 815-22. Ranganatha. S., Hoshi, Y., and C. Guan. “Near infrared spectroscopy based brain-computer interface,” in Proc. SPIE Exp. Mech, vol. 5852, Bellingham,WA, 2005, pp. 434-442. Chance, B., Zhuang Z., UnAh, Z., Alter C, Lipton, L., (1993) "Cognition-activated low-frequency modulation of light absorption in human brain",PNAS 90: 3770-3774 Villringer, A., Chance, B., (1997) "Non-invasive optical spectroscopy and imaging of human brain function", Trends Neuroscience ,20, 435–442. Hoshi Y., Onoe H., Watanabe Y., Andersson J., Bergstrom M., Lilja A., Langstom B., & Tamura M., (1994), Non-synchronous behavior
345
[22]
[23]
[24] [25]
[26] [27]
of neuronal activity, oxidative metabolism and blood supply during mental tasks in man, Neurosci. Lett. 172, 129-133. Villringer, A., Planck, J., Hock, C., Schleinkofer, L., & Dirnagl, U., (1993) "Near infrared spectroscopy (NIRS): A new tool to study hemodynamic changes during activation of brain function in human adults". Neuroscience Letters, , 154, 101–104 Strangman, G., Boas, D.A., Sutton, J.P., "Non-invasive neuroimaging using near-infrared light", Biological Psychiatry, 2002. 52(7), 679-93. Gratton E., Toronov V., Wolf U., Wolf, M., Webb A., (2005) Measurement of brain activity by near-infrared light, Journal of Biomedical Optics, 10(1), 011008 Hoshi, Y. (2003). Functional near-infrared optical imaging: Utility and limitations in human brain mapping. Psychophysiology, 40:511–520. Izzetoglu M, Izzetoglu K, Bunce S, Ayaz H., Deveraj A., Onaral B, Pourrezaei K, (2005). Functional Near-Infrared Neuroimaging. IEEE Trans. on Neural Systems and Rehabilitation Engineering, 13(2):153-159. Izzetoglu K, Yurtsever G, Bozkurt A, Yazici B, Bunce S, Pourrezaei K, Onaral B, (2003). NIR spectroscopy Measurements of Cognitive Load Elicited by GKT and Target Categorization. Proc. of Hawaii Int. Conf. on System Sciences, the Aug. Cog. & Human-Robot Inter. Izzetoglu, K.; Bunce, S.; Izzetoglu, M.; Onaral, B.; Pourrezaei, K. (2003); fNIR Spectroscopy as a Measure of Cognitive Task Load. Proc. of EMBS 2003 Izzetoglu K, Bunce S, Onaral B, Pourrezaei K, Chance B, (2004). Functional Optical Brain Imaging Using Near-Infrared During Cognitive Tasks. Int. J. of Human-Comp. Int., 17(2):211-227. Haida M., Shinohara Y., Ito Y., Yamamoto T., Kawaguchi F., and H. Koizumi, (2000) “Brain function of an ALS patient in complete locked-in state by using optical topography,” in The Frontier of Mind-Brain Science and Its Practical Applications II, H. Koizumi, ed. Hitachi, Ltd., Kokubunisi Tokyo, Japan, pp. 95–97. Sitaram, R., Zhang, H., Guan, C., Thulasidas, M., Hoshi, Y., Ishikawa, A., Shimizu, K., Birbaumer, N., (2007) Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface. NeuroImage, 34, p1416-27. Bozkurt A, Onaral B, (2004), Safety Assessment of Near Infrared Light Emitting Diodes for Diffuse Optical Measurements., Biomedical Engineering Online, 3(9). Bozkurt A, Rosen A, Rosen H, Onaral B, (2005). A Portable Near Infrared Spectroscopy System for Bedside Monitoring of Newborn Brain. Biomedical Engineering Online., 4(29). Cope, M. (1991). The Development of a Near-Infrared Spectroscopy System and Its Application for Noninvasive Monitoring of Cerebral Blood and Tissue Oxygenation in the Newborn Infant. Ph.D. Thesis. Univ.College London, London. Boas DA, Franceschini MA, Dunn AK, Strangman G. (2002). NonInvasive imaging of cerebral activation with diffuse optical tomography. In-Vivo Optical Imaging of Brain Function. Boca Raton: CRC Press. pp. 193–221 Chance B, Anday E, Nioka S, Zhou S, Hong L, Worden K, Li C, Murray T, Ovetsky Y, Pidikiti D, Thomas R. (1998). A novel method for fast imaging of brain function, non-invasively, with light. Optics Exp., 2(10). Smith, E. E., Jonides, J., Working Memory: (1997) A View from Neuroimaging, Cognitive Psychology 33, 5–42 Ayaz, H., Izzetoglu, M., Platek, S. M., Bunce, S., Izzetoglu, K., Pourrezaei K., and Onaral, B., (2006) “Registering fNIR Data to Brain Surface Image using MRI templates”, 28th IEEE EMBS Annual International Conference, New York, USA, p2671-4 Braver TS, Cohen JD, Nystrom LE, Jonides J, Smith EE, Noll DC. (1997) A parametric study of prefrontal cortex involvement in human working memory. Neuroimage. 5(1):49-62. Strangman G, Goldstein R, Rauch SL, Stein J. (2006) Near-infrared spectroscopy and imaging for investigating stroke rehabilitation: testretest reliability and review of the literature. Arch Phys Med Rehabil. 87(12 Suppl):12-9.
