ED: Susan Koshy Op:Vinoth Scan: Roopa WNR: lww_wnr_1930 NEUROREPORT
CLINICAL NEUROSCIENCE
Brain^computer interface using fMRI: spatial navigation by thoughts Seung-Schik Yoo,1,2,CA Ty Fairneny,3 Nan-Kuei Chen,1 Seh-Eun Choo,4 Lawrence P. Panych,1 HyunWook Park,5 Soo-Young Lee2 and Ferenc A. Jolesz1 1
Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, 75 Francis St, Boston, MA 02115, USA; 2Department of BioSystems; 5 Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Korea; 3Department of Biomedical Engineering; 4 College of Communication, Boston University, Boston, MA, USA CA,1
Corresponding Author and Address:
[email protected] Received 8 April 2004; accepted 28 April 2004 DOI: 10.1097/01.wnr.0000133296.39160.fe
A brain^ computer interface (BCI) is a way of conveying an individual’s thoughts to control computer or electromechanical hardware. Capitalizing on the ability to characterize brain activity in a reproducible manner, we explored the possibility of using real-time fMRI to interpret the spatial distribution of brain function as BCI commands. Using a high-¢eld (3 T) MRI scanner, brain activities
associated with four distinct covert functional tasks were detected and subsequently translated into predetermined computer commands for moving four directional cursors. The proposed fMRIBCI method allowed volunteer subjects to navigate through a simple 2D maze solely through their thought processes. NeuroReport c 2004 Lippincott Williams & Wilkins. 15:000^ 000
Key words: Biofeed; Cognition; fMRI; Imagery; Mental task; Rehabilitation
INTRODUCTION A brain–computer interface (BCI) is a way of using an individual’s thought processes to control computer or electromechanical hardware without using overt muscle activities [1–3]. This type of system has the potential to provide a new form of communication and control options for individuals paralyzed from high-level spinal cord injury, severe neuromuscular disorders, or amyotrophic lateral sclerosis (ALS). The methods and utility of a BCI are currently being investigated in the field of rehabilitation [3– 5] and neurorobotics [6–8]. Since a BCI is aimed at detecting signals generated from responsive or intentional brain activities in a real-time fashion, most BCI systems to date are based on the detection of electroencephalography (EEG) recordings, whereby the amplitude or temporal patterns of the brain activity associated with a task is interpreted as an analogue control signal. For example, several BCI techniques have been developed based on scalp-recorded EEG activity. These techniques include (1) the detection of the P300 component of an event-related evoked potential (ERP) [9], (2) EEG murhythm conditioning [1,4,10], and (3) the use of visual evoked potentials (VEPs) during visual target recognition [11]. Techniques for classifying, detecting and mapping EEG patterns were also described [12]. Implantable microelectrode arrays have also been used to associate neural activities with controlling a remotely located robotic arm in primates [13,14].
c Lippincott Williams & Wilkins 0959- 4965
fMRI provides a high-resolution map of the brain activities associated with neuronal activation. With spatial resolution in the millimeter range, the analysis of fMRI data can now be done in a real-time manner [15–17]. Taking advantage of this capability, our group previously reported the use of fMRI for the biofeedback of one’s brain function, whereby the level of activation in the hand motor areas were voluntarily changed [16]. More recently, new studies have included attempts to modulate amygdala activation during trials of self-induced sadness [18], activities in the rostralventral and dorsal part of the anterior cingulate cortex [17], and activation in somatomotor cortex during cognitive imagery tasks [19]. These studies suggest that regionally specific brain activities can be voluntarily adjusted, and that these changes can be detected and quantified using fMRI. This forms a strong framework whereby the spatial distribution of brain function can be detected and translated into discrete computer commands. We implemented an fMRI method to detect the brain activation patterns associated with four different functional tasks, and translated the activation associated with each task to one of the four directional cursor commands for navigation through a 2D maze presented during the BCI session. In addition, since we intended to make thought processes the means to control the computer, subjects were required to use only mental tasks that are not overtly expressed via physical activities.
Vol 15 No 10 & & 2004
1
NEUROREPORT
S.-S.YOO ETAL.
MATERIALS AND METHODS BCI overview: In order to demonstrate the feasibility of an fMRI-BCI, we limited the degrees of freedom for the computer control commands to accommodate only four directional cursors (up, down, right and left) necessary for navigating through a simple 2D maze. The overall schematic of our BCI implementation is illustrated in Fig. 1. The BCI procedure was divided into two distinct stages. The first stage was to inform and train the volunteers about the tasks (four distinct thought processes) to be performed during the BCI fMRI sessions. During this stage, template fMRI maps were created, whereby regions of interest (ROI) representative of each task were linked to specific cursor commands. In the second stage of the BCI procedure, subjects navigated a simple 2D maze using these mental tasks. Subjects viewed the 2D maze via MR-compatible goggles, and determined the appropriate cursor commands needed to complete the maze. In the subsequent fMRI sessions, the subject performed the mental task corresponding to each intended cursor movement. The data was then processed in a near real-time manner. The subject’s brain activation patterns within the representative ROIs were quantitatively compared to the templates using Dice’s similarity coefficient (DSC) [20]. The cursor moved through the maze in the direction that corresponded to the best matching (thus, the highest DSC) activation pattern. The process continued until the cursor reached the end of the maze. Subject preparation: Three healthy male volunteers (21–24 years in age) were recruited for the implementation and testing of the method. The participants were all righthanded according to the Edinburgh Handedness Inventory, and had no history of psychiatric or neurological disorder. Informed consent was obtained prior to the study according
Task instruction
Practice of tasks and real-time fMRI
Generation of reference activation template
Thoughts and corresponding cursor command Mental calculation
Activation pattern and ROI
fMRI parameters: The fMRI BCI was implemented using a high-field (3Tesla) MR scanner using VH3 software (GE Medical Systems, Waukesha, WI). T1-weighted anatomical images were acquired using a Spoiled Gradient Recalled (SPGR) sequence (TR/TE¼30 ms/minimum, flip angle¼301, 256 256 matrix, 240 mm field of view). Functional images were collected with a T2*-weighted gradient echo sequence (TR/TE¼1500/40 ms, flip angle¼801) with an in plane resolution of 3.75 mm (240 mm field of view, 64 64 inplane matrix). Twenty 5 mm axial slices (1 mm slice gap) were imaged covering most of the brain. Four initial dummy data acquisitions were included to allow for T1 signal equilibration. A block-based paradigm design was used to detect the functional areas for the BCI procedures. The task conditions, 15 s in duration, were interleaved by rest periods of the same duration, and repeated 3 times (Fig. 2). With a TR of 1.5 s, 10 sets of images were acquired for each period of task or rest condition. During the rest condition, the subject binaurally listened to a computer-generated 900 Hz monotone (sampled at 11 kHz) paced at 2 Hz using sound editing
1 min 51 s Dummy
15 s Fig. 2.
Rest
Task 15 s
fMRI task paradigm.
Start
Stage II
Choice of cursor command MR-goggle Thought generation
Real-time fMRI Mental speech generation
Finish
Command generation
Right hand motor imagery ROI application
Left hand motor imagery
DSC
Stage I
to the ethical guidelines set forth by the institutional review board of Brigham and Women’s Hospital and Harvard Medical School.
Reproducibility matching
Fig. 1. Overall schematics of the fMRI procedure for the BCI. The procedure is split up into two stages. The ¢rst stage is data calibration and subject preparation.The second stage is the BCI experiment.
2
Vol 15 No 10 & & 2004
NEUROREPORT
BRAIN^COMPUTER INTERFACE USING fMRI
four imagery tasks. Based on previous fMRI studies using similar mental tasks [21–23], we defined four distinct ROIs for further characterization and comparison (Table 1). The ROIs, which best represented each specific task, were manually segmented and stored as a bitmap using a graphical user interface (GUI) written in Matlab.
software (Goldwave 4.24, Goldwave, NF, Canada). This rest condition enabled subjects to remain at a constant level of attention throughout the experiment. The auditory stimulation was delivered using a MR-compatible headset (Avotec, Inc. Jensen Beach, FL). Near real-time processing of fMRI data: The raw fMRI data were temporarily stored in a MR control workstation. The data were then transferred to a computational workstation (1 GHz Intel Processor with 512 MBytes of memory) via FTP protocol (10 megabit/s transfer rate) where the image reconstruction and fMRI data processing were performed. MATLAB software (Mathworks Ver 6.0, Natick, MA) was used to reconstruct and process the fMRI data. The overall processing time was typically o15 s for the given fMRI imaging parameters. fMRI data were processed by calculating pixel-by-pixel t-test scores for rest and task periods. Linear detrending was applied to the time-course of the fMRI data. The data acquired from the dummy scans were excluded from the calculation. The pixels with significant activation were defined at the level of p o103.
Spatial navigation through a 2D maze using the BCIfMRI: After generating template fMRI maps with taskspecific ROIs, the subjects were shown the 2D maze and asked to become familiar with its structure. The difficulty of the maze was simple enough for that the subject to plan for the shortest solution at first glance. The sequence of movements needed to complete the maze, which corresponded to the four possible BCI controls, were randomized and balanced (each functional task needed to be performed 3 times to complete the maze). The contents of the maze (shown in the upper right corner of Fig. 1), was constructed from 10 10 component bins, and required a minimum of 12 external inputs to complete the maze. If an invalid input (such as wrong direction) were provided, the cursor would not move through the maze. Instead, a try again sign was shown to the subjects. The 2D maze and its controlling environment were programmed with interactive animation software (Director 6.0, Macromedia, San Francisco, CA). The subjects closed their eyes during the fMRI data acquisition. We employed the DSC to determine which of the four functional templates best matched a current brain activation jk map ðDSC ¼ 2Voverlap =ðVj þ Vk Þ; where Vj and Vk were the volume of activation from session j and template k jk respectively, and Voverlap is the volumes commonly activated from Vj and Vk). The highest DSC among the four different directional template maps was determined and used to
Calibration of fMRI data & ROI generation: Each subject was first informed of the four types of fMRI tasks and their corresponding computer control (Table 1). These tasks were, (1) mental calculation: sequential subtraction of 3 from the number 50 (up), (2) mental speech generation of the lyrics of a classic nursery rhyme or the Pledge of Allegiance (down), (3) motor imagery of sequential finger opposition of the right hand (right), and (4) the same motor imagery task on the left hand movement (left). After a brief practice, the subjects underwent real-time fMRI sessions to confirm that spatially distinct areas were reliably detected for each of the
Table 1. The fMRI tasks with their representative functional anatomy and corresponding cursor commands. Tasks
Di¡erential functional anatomy
Mental calculation Right hand motor imagery Mental speech generation Left hand motor imagery
Cursor command
Bilateral medial superior frontal gyrus & anterior cingulate gyrus Left primary and secondary somatosensory areas in pre- post- central gyrus Posterior aspect of superior temporal gyrus Right primary and secondary somatosensory areas in pre- post- central gyrus
Table 2. The tabulated results of Dice’s similarity coe⁄cients (DSC) with respect to four di¡erent template tasks. The maximum DSC values for each session are marked in bold character. It is notable that subject A missed one desired cursor command (‘Left’) in the 11th session (indicated in asterisk), and corrected the command at the 12th session. A ROI
B ROI
C ROI
Session
Up
Down
Right
Left
Session
Up
Down
Right
Left
Session
Up
Down
Right
Left
1 2 3 4 5 6 7 8 9 10 11 12 13
0.2 0.2 0.3 0.5 0.3 0.5 0.3 0.2 0.1 0.5 0.3 0.2 0.0
0.1 0.4 0.1 0.0 0.1 0.2 0.1 0.6 0.2 0.2 0.1 0.1 0.4
0.5 0.0 0.6 0.1 0.3 0.1 0.6 0.0 0.2 0.2 0.8* 0.1 0.0
0.0 0.0 0.2 0.0 0.9 0.0 0.1 0.0 0.5 0.0 0.1 0.8 0.0
1 2 3 4 5 6 7 8 9 10 11 12
0.4 0.4 0.3 0.3 0.4 0.8 0.4 0.3 0.4 0.5 0.3 0.2
0.1 0.7 0.3 0.1 0.1 0.1 0.2 0.4 0.2 0.1 0.1 0.3
0.8 0.3 0.7 0.0 0.4 0.1 0.6 0.2 0.4 0.1 0.2 0.1
0.3 0.2 0.6 0.0 0.7 0.1 0.3 0.1 0.5 0.0 0.4 0.0
1 2 3 4 5 6 7 8 9 10 11 12
0.3 0.4 0.4 0.7 0.3 0.7 0.0 0.4 0.2 0.7 0.3 0.5
0.0 0.6 0.0 0.1 0.0 0.0 0.0 0.6 0.0 0.0 0.0 0.6
0.4 0.0 0.5 0.1 0.4 0.1 0.8 0.0 0.1 0.1 0.3 0.0
0.0 0.0 0.2 0.1 0.7 0.0 0.5 0.0 0.6 0.0 0.5 0.0
Vol 15 No 10 & & 2004
3
NEUROREPORT
S.-S.YOO ETAL.
move through the maze. Upon completing a task, the subject was instructed to open their eyes to confirm the results of their attempted BCI navigation. If the actual cursor movement was different from the desired movement, it was recorded and the fMRI session was repeated.
z y x
4
Vol 15 No 10 & & 2004
L R
Is RO
D U R
D
R
U
L
BCI
L
D
R
U
U
L*
D
L
ns ssio
se
(b) DSC
1 0.5 L R
RO
D U R
D
R
U
L
U
D
R BCI
L
U
L
D
ns ssio
se
(c) 1 0.5 L R
Is
A method, implemented in a high-field MRI system, was designed to detect the spatial patterns of brain activities generated from four different functional tasks. The unique cortical activation pattern of each task was interpreted as the predetermined computer commands, used for moving a cursor for spatial navigation through a maze. Based on our preliminary data, we conclude that fMRI can be used as a working BCI prototype. fMRI is economically unfavorable compared to the EEGbased BCI methods. However, we have shown that fMRI opens a new dimension to the current BCI prototypes by using spatially-selective neural activities to generate discrete cursor controls. As the methodology matures, we believe that this fMRI-BCI can be integrated with the existing BCI modalities to enhance and complement their functions. For example, it may be possible to correlate a certain type of EEG activity with the brain activation pattern from fMRI sessions using simultaneous EEG and fMRI data acquisition [24]. In this case, fMRI could be used as the initial guide, fine-tuning the BCI protocols for classical EEG-based BCI procedures. The current fMRI-BCI method has a marginal temporal resolution of 42 mins for command generation. In order to increase the speed of the fMRI-BCI protocol, a single-trial design with duration of 60 s, similar to the paradigm used in Posse et al.’s implementation [18], can be employed in place of the current block-based design. With the optimization of
0.5
D
RO
DISCUSSION
1
Is
As shown in Table 1, four distinct patterns of global activation were found corresponding to each of the four tasks. These are (1) bilateral activation in the medial superior frontal gyrus, representative of mental calculation, (2) left Broca’s area (inferior frontal gyrus) and the auditory association areas for the internal speech task, (3) the left somatomotor areas for the right hand imagery task, and (4) right somatomotor areas for the left hand imagery task. The time needed to complete the generation of the four template functional map was o15 min (excluding subject preparation and initial training). The results of the fMRI-BCI sessions, obtained from three subjects, are shown in Table 2 and Fig. 3. As shown by the DSC values from Table 2, the four different tasks often shared common activation areas (i.e. DSC being greater than zero for the other tasks). For example, left hand motor imagery also involves the activation in the ipsilateral part of the somatomotor areas, which are also engaged during right hand motor imagery [23]. Due to this overlap in the neural substrates, the method misidentified one of the commands from Subject A during the 11th fMRI session, resulting in a performance accuracy of 92.3%. Subjects B and C, on the other hand, completed the maze navigation without error in 12 steps. The time needed to generate each movement command was B2 min 15 s (including 1 min 51 s fMRI scan time).
DSC
RESULTS
DSC
(a)
U R
D
R
U
L
U
D
R
L
U
L
D
ons
essi
s BCI
Fig. 3. Results of Dice’s similarity coe⁄cient (DSC) measurements for the three subjects (from A-C) tested with the fMRI-BCI procedure. The x-axis represents the sequence of BCI sessions and executed cursor commands.TheY-axis represents the four possible directional commands that can be executed by the computer.The Z-axis indicates the DSC.The maximum DSC in each sessions is marked with white circles.The pink circle on the 11th session from subject A indicates a false movement (‘R’ instead of ‘L’) and was corrected in the following attempt.
scan parameters and task timing, the temporal resolution of the method can probably be increased significantly. We also found that the accuracy of our fMRI-BCI method was not perfect although two subjects were able to complete the navigation with no error. Because the four tasks engaged overlapping areas of activation, we implemented an ROIbased approach to measure the degree of similarity between a current activation pattern and the template activation maps. Since only small regions of interest were analyzed, the method could be improved by using a more advanced pattern recognition algorithm such as a statistical neural network classifier [25].
BRAIN^COMPUTER INTERFACE USING fMRI
We utilized the spatial distribution of brain activation as the mediator for the BCI. However, the magnitude of the blood oxygenation level-dependent (BOLD) signal itself can also be utilized as the BCI signal. For example, earlier investigation showed that the BOLD signal in the anterior cingulate can be adjusted through the help of real-time fMRI neurofeedback [17]. The ability to control the BOLD signal magnitude in a discrete fashion (such as small, intermediate, and large MR signal increase during the task trial) could be used to increase the degrees of freedom for the BCI. The biofeedabck of one’s own brain function, i.e., neurofeedback, would be beneficial during pre-BCI traning sessions, where subjects learn the best strategy for modulating the BOLD signal responses for a given task.
CONCLUSION We have shown that fMRI can be used to detect the spatial pattern of activation and translate these patterns into distinct BCI commands. By exploring different types of mental tasks such as mental imagery and covert cognitive tasks, together with improving the speed and accuracy of the detection method, we believe that this fMRI-BCI can be used to control a virtual keyboard presented to a subject via MR compatible goggles. This keyboard, for instance, can be controlled by using both the different types of thought processes (different spatial distribution of activation) and its degree of exertion (different mganitude of BOLD signal contrast).
REFERENCES 1. Wolpaw JR, McFarland DJ, Neat GW and Forneris CA. An EEG-based brain–computer interface for cursor control. Electroencephalogr Clin Neurophysiol 1991; 78:252–259. 2. Nicolelis MA, Dimitrov D, Carmena JM, Crist R, Lehew G et al. Chronic, multisite, multielectrode recordings in macaque monkeys. Proc Natl Acad Sci USA 2003; 100:11041–11046. 3. Wolpaw JR, McFarland DJ, Vaughan TM and Schalk G. The Wadsworth Center brain–computer interface (BCI) research and development program. IEEE Trans Neural Syst Rehabil Eng 2003; 11:204–207. 4. Pfurtscheller G, Flotzinger D and Kalcher J. Brain–Computer Interface – a new communication device for handicapped persons. J Microcomputer Appl 1993; 16:293–299. 5. Pfurtscheller G, Muller GR, Pfurtscheller J, Gerner HJ and Rupp R. ‘Thought’–control of functional electrical stimulation to restore hand grasp in a patient with tetraplegia. Neurosci Lett 2003; 351:33–36.
NEUROREPORT 6. Kositsky M, Karniel A, Alford S, Fleming KM and Mussa-Ivaldi FA. Dynamical dimension of a hybrid neurorobotic system. IEEE Trans Neural Syst Rehabil Eng 2003;11:155–159. 7. Moore MM. Real-world applications for brain–computer interface technology. IEEE Trans Neural Syst Rehabil Eng 2003; 11:162–165. 8. Taylor DM, Tillery SI and Schwartz AB. Information conveyed through brain-control: cursor vs robot. IEEE Trans Neural Syst Rehabil Eng 2003; 11:195–199. 9. Farwell LA and Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol 1988; 70:510–523. 10. McFarland DJ, Neat GW, Read RF and Wolpaw JR. An EEG-based method for graded cursor control. Psychobiology 1993; 21:77–81. 11. Sutter EE. The brain response interface: communication through visuallyinduced electrical brain response. J Microcomputer Appl 1992; 15:31–45. 12. Keirn ZA and Aunon JI. A new mode of communication between man and his surroundings. IEEE Trans Biomed Eng 1990; 37:1209–1214. 13. Chapin JK, Moxon KA, Markowitz RS and Nicolelis MA. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neurosci 1999; 2:664–670. 14. Nicolelis MA. Brain-machine interfaces to restore motor function and probe neural circuits. Nature Rev Neurosci 2003; 4:417–422. 15. Yoo SS, Guttmann CR, Zhao L and Panych LP. Real-time adaptive functional MRI. Neuroimage 1999; 10:596–606. 16. Yoo SS and Jolesz FA. Functional MRI for neurofeedback: feasibility study on a hand motor task. Neuroreport 2002; 13:1377–1381. 17. Weiskopf N, Veit R, Erb M, Mathiak K, Grodd W et al. Physiological selfregulation of regional brain activity using real-time functional magnetic resonance imaging (fMRI): methodology and exemplary data. Neuroimage 2003; 19:577–586. 18. Posse S, Fitzgerald D, Gao K, Habel U, Rosenberg D et al. Real-time fMRI of temporolimbic regions detects amygdala activation during single-trial self-induced sadness. Neuroimage 2003; 18:760–768. 19. deCharms RC, Christoff K, Glover GH, Pauly JM, Whitfield S et al. Learned regulation of spatially localized brain activation using real-time fMRI. Neuroimage 2004; 21:436–443. 20. Machielsen WC, Rombouts SA, Barkhof F, Scheltens P and Witter MP. fMRI of visual encoding: reproducibility of activation. Hum Brain Mapp 2000; 9:156–164. 21. McGuire PK, Silbersweig DA, Wright I, Murray RM, Frackowiak RS et al. The neural correlates of inner speech and auditory verbal imagery in schizophrenia: relationship to auditory verbal hallucinations. Br J Psychiatry 1996; 169:148–159. 22. Burbaud P, Camus O, Guehl D, Bioulac B, Caille JM et al. A functional magnetic resonance imaging study of mental subtraction in human subjects. Neurosci Lett 1999; 273:195–199. 23. Lotze M, Montoya P, Erb M, Hulsmann E, Flor H et al. Activation of cortical and cerebellar motor areas during executed and imagined hand movements: an fMRI study. J Cogn Neurosci 1999;11:491–501. 24. Garreffa G, Carni M, Gualniera G, Ricci GB, Bozzao L et al. Real-time MR artifacts filtering during continuous EEG/fMRI acquisition. Magn Reson Imaging 2003; 21:1175–1189. 25. Feraud R and Clerot F. A methodology to explain neural network classification. Neural Netw 2002; 15:237–246.
Acknowledgements: This study was partially supported by the Korean Ministry of Science and Technology (grant No. M1- 0107- 070001).
Vol 15 No 10 & & 2004
5
AUTHOR QUERY FORM
LIPPINCOTT WILLIAMS AND WILKINS JOURNAL NAME ARTICLE NO:
WNR 1930
5/26/04
QUERIES AND / OR REMARKS Query No
AQ1
Details Required
Authors Response
Style for refs is 6 authors + et al. Please provide 6th author where needed.
1
