A direct brain interface based on event-related potentials - IEEE Xplore

180

IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 8, NO. 2, JUNE 2000

A Direct Brain Interface Based on Event-Related Potentials Simon P. Levine, Jane E. Huggins, Spencer L. BeMent, Senior Member, IEEE, Ramesh K. Kushwaha, Lori A. Schuh, Mitchell M. Rohde, Member, IEEE, Erasmo A. Passaro, Donald A. Ross, Kost V. Elisevich, and Brien J. Smith

Abstract—Cross-correlation between a trigger-averaged event-related potential (ERP) template and continuous electrocorticogram was used to detect movement-related ERP’s. The accuracy of ERP detection for the five best subjects (of 17 studied), had hit percentages 90% and false positive percentages 10%. These cases were considered appropriate for operation of a direct brain interface. Index Terms—Assistive technology, brain–computer interface (BCI), cross-correlation, detection, direct brain interface, electrocorticogram (ECoG), event-related potential (ERP).

included data from six subjects (out of a total of 15) with reasonably good electrode coverage over motor areas of the cortex. In this paper, data from two additional subjects with broad electrode coverage over cortical motor areas have been included, increasing the number of ECoG recordings from frontal and motor areas (frontal lobe, pre and post central gyri, supplementary motor area, premotor cortex) by about 50%, from 1631 to 2410. II. METHODS

I. INTRODUCTION A direct brain interface accepts voluntary commands directly from the human brain without requiring physical movement and can be used to operate a computer or other technologies. The direct brain interface approach described here uses cross-correlation of trigger-averaged electrocorticogram (ECoG) segments (“ERP templates”) with the continuous ECoG to detect ERP’s that correspond to specific movements. The choice of this methodological approach was made with the short-term goal of developing a single-switch direct brain interface and the longer term goal of developing a multichannel direct brain interface. Cross-correlation was selected as a detection method because it has been successfully used to detect human sensory evoked potentials in electroencephalogram (EEG) [1], [2] and ECoG [2] and it is a relatively simple, well-understood process that can be easily implemented with real-time computation. The research methods for this direct brain interface approach have been reported in detail [3]. The previously reported results Manuscript received August 8, 1999; revised March 27, 2000. This work was supported in part by grants from the National Institute on Disability and Rehabilitation Research (Field Initiated Research Project #H133G70120-97); the National Science Foundation (Graduate Student Fellowship), and the Whitaker Foundation (Graduate Student Fellowship). S. P. Levine and J. E. Huggins are with the Rehabilitation Engineering Program, Department of Physical Medicine and Rehabilitation and the Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109 USA. S. L. BeMent is with the Department of Biomedical Engineering and the Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109 USA. M. M. Rohde is with the Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI. R. K. Kushwaha and E. A. Passaro are with the Department of Neurology, University of Michigan, Ann Arbor, MI 48109 USA. L. A. Schuh and B. J. Smith are with the Department of Neurology, Henry Ford Hospital, Detroit, MI 48202 USA. D. A. Ross is with the Section of Neurosurgery, Department of Surgery, University of Michigan, Ann Arbor, MI 48109 USA. K. V. Elisevich is with the Department of Neurosurgery, Henry Ford Hospital, Detroit, MI 48202 USA. Publisher Item Identifier S 1063-6528(00)04108-2.

A. Data Collection and Template Calculation Seventeen patients from two epilepsy surgery programs participated in the research described here. As part of their presurgical evaluation they had 16–126 subdural electrodes implanted on the surface of their cerebral cortex to record seizure activity and/or map cortical function. The cortical locations of these electrodes were based solely on clinical considerations relating to epilepsy surgery (as opposed to research needs). The electrodes were 4-mm diameter platinum or stainless-steel electrodes arranged in either strips or grids with a center to center distance of 1 cm. Subjects participated in this research while in bed in their hospital room. Each subject performed one to four repetition sets (containing 50 repetitions each separated by at least 3 s) for each of one to six actions. Subjects were instructed in each action just prior to a repetition set. Actions included different movements of the face, tongue, hand, and foot. Verbalization of a sound or word was also used. The subject performed the action either in response to a prompt (first five subjects) or in a self-paced manner (remaining 12 subjects). An appropriate transducer (EMG electrodes, microphone, or switch) was used to document the time of each repetition on a “trigger channel.” The ECoG’s and the trigger channel were recorded at a sampling rate of 200 Hz using a Cadwell Spectrum 32 (Cadwell Laboratories, Inc., Kennewick, WA USA) at the University of Michigan or a Nicolet BMSI 5000 (Nicolet Biomedical, Inc., Madison, WI USA) at the Henry Ford Hospital. The trigger channel was processed [5] to identify “trigger points,” the times at which a voluntary action occurred. The recorded ECoG’s were digitally filtered to a passband from 0.05 to 100 Hz. The 17 subjects performed a total of 130 repetition sets that contained a total of 6498 ECoG recordings. Triggered averaging of the first half of each ECoG recording was used to produce an averaged ECoG segment for that electrode recording channel over a six second interval of ECoG, centered around the trigger point [3] (see Fig. 1). The second half of each ECoG recording

1063–6528/00$10.00 © 2000 IEEE

LEVINE et al.: DIRECT BRAIN INTERFACE BASED ON EVENT-RELATED POTENTIALS

181

Fig. 1. Triggered averaging process. Six second segments of ECoG, centered at the trigger point, are averaged to create the ERP template. The first five of the 23 (subject DV) and 25 (subject KR) segments used to create the average are shown.

was reserved as test data to evaluate the accuracy of the ERP detection method. Detailed descriptions of the data collection and template calculation methods for this research are described in Levine et al. [3]. B. Cross-Correlation Based Detection For each ECoG recording channel, normalized cross-correlation was performed between the triggered average created from the first half of the recording and the continuous test data from the second half of the recording. Cross-correlation, the moving summation of the point by point multiplication of the averaged template and continuous ECoG signal, provided a measure of the fit of the template and the corresponding interval of the ECoG for each successive alignment between the template and ECoG (see Fig. 2). The set of points at which the cross-correlation value exceeded an experimentally determined threshold [4] were defined as detection points and were compared to the trigger points to determine “hits” or “false positives.” Any detection that occurred between one second before and a quarter second after a trigger point was defined as a hit. These criteria for a hit were chosen to emphasize the importance of detecting the action before the movement-trigger. A more detailed description of the cross-correlation detection methods for this research is described in Huggins et al. [4].

The performance of the detection method for each ECoG recording was described by two statistics: the hit percentage, the percentage of the triggers in the test data that were correctly detected, and the false positive percentage, the percentage of the detections that were not hits. In order to facilitate comparison between the performance of the detection method on different ECoG recordings, a combined statistic, the difference between the hit percentage and the false positive percentage (the HF-difference) was determined. The HF-difference was set to zero when the false positive percentage exceeded the hit percentage. Thus, the HF-difference was a value between 0–100, with an HF-difference of 100 indicating perfect detection. As a means to evaluate the relationship between electrode channel performance and electrode location, every electrode location was mapped into one of 10 cortical areas [4] (shown in Fig. 3) and the percentage of ECoG recordings in each area that yielded an HF-difference greater than or equal to 50 was calculated. III. RESULTS The largest HF-difference for each subject is shown in Table I along with the action and the electrode location for the ECoG recording that produced it. HF-differences greater than 90 were found for 5 of 17 subjects, HF-differences greater than 75 were found for 10 of 17 subjects, and HF-differences greater than or

182

IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 8, NO. 2, JUNE 2000

Fig. 2. Cross-correlation of the average template with the continuous ECoG. The sum of a point-by-point multiplication (the cross-correlation value) between the template and the ECoG is calculated for successive time delays as the averaged template is shifted to the right along the time axis. The normalized cross-correlation value is plotted as a function of delay time to create the cross-correlogram. The asterisks mark the trigger points for each repetition and the shaded area shows the interval in which a detection is counted as a hit.

Fig. 3.

Cortical mapping (interhemispheric areas not shown).

equal to 50 were found for 15 of 17 subjects. HF-differences greater than 75 occurred on a total of 19 ECoG recordings, while HF-differences greater than or equal to 50 occurred on a total of 119 ECoG recordings. Of the subjects with the lowest HF-differences (Table I), GB performed only a single action while the electrode locations for VC, DT, DB, and MD were not very well suited for recording movement-related ERP’s. HF-differences greater than 75 were found for multiple actions with a number of subjects, usually from different electrode locations. The best subject, CB, had HF-differences of 96, 86, and 83 for middle finger extension, lip movement, and tongue protrusion, respectively. The general cortical distributions of ECoG recordings with HF-differences greater than or equal to 50 are shown in Table II. The cortical areas yielding the highest percentage of ECoG’s with an HF-difference greater than or equal to 50 were the supplementary motor area, the postcentral gyrus and the precentral gyrus. The three largest HF-differences (96 for subject TS, tongue protrusion, 96 for CB, middle finger extension, and 93 for DV, tongue protrusion) were recorded from the middle/inferior temporal gyri, parietal lobe, and postcentral gyrus areas respectively. Accurate determination of the ERP onset times was difficult because of low signal-to-noise ratios. Manual classification of

LEVINE et al.: DIRECT BRAIN INTERFACE BASED ON EVENT-RELATED POTENTIALS

183

TABLE I MAXIMUM HF-DIFFERENCE FOR EACH SUBJECT

TABLE II CORTICAL DISTRIBUTION OF ECoG RECORDING SITES WITH HF-DIFFERENCES GREATER THAN OR EQUAL TO 50

the onset time for the ECoG templates that produced HF-differences greater than or equal to 50 resulted in 72 pretrigger, 16 at-trigger, 15 posttrigger, and 16 ambiguous ERP onsets. IV. DISCUSSION Five subjects had HF-differences greater than 90, including two with no false positives. These results indicate that our crosscorrelation based method can detect ERP’s related to movement

with sufficient accuracy to form the basis of a functional direct brain interface. Further, movements related to different actions (often recorded from different parts of the cortex) can be accurately detected, indicating the strong possibility that multiple control channels for a direct brain interface can be obtained with this approach. Most of these ERP’s had an onset prior to the trigger point, implying that they include brain activity associated with the planning and initiation of the action.

184

IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 8, NO. 2, JUNE 2000

The probability of an ECoG recording producing an HF-difference of 50 or above was shown to be the greatest in sensorimotor areas (supplementary motor area, postcentral gyrus, and precentral gyrus). However, ECoG recordings with HF-differences greater than or equal to 50 also occurred in other cortical areas a smaller percentage of the time. This is consistent with previous research showing that motor responses are frequently obtained from stimulation sites distributed well outside the classical narrow motor strip [6]. The accurate detection of ERP’s related to tongue protrusion in the middle/inferior temporal gyri might possibly be attributed to cortical connections for tongue movements during speech production. The three subjects with the lowest HF-differences (Table I) had strips of electrodes placed over their temporal lobes, an area that would be considered to have a relatively low probability of containing ERP’s related to movement (both from a traditional understanding of cortical organization and from our experimental results). However, the highest detection accuracy (from subject TS) was recorded with a similar electrode placement. This may imply that the areas from which such detection accuracy can be identified are restricted. Anecdotal support for this contention is provided by the case of TS where the electrode adjacent (1 cm distant) to the one with an HF-difference of 96 had an HF-difference of only 19. This anecdotal result suggests that accurate detection may sometimes require very precise electrode placement and that utilization of smaller, more closely spaced electrodes may increase the probability of obtaining ECoG’s for which accurate detection is possible. However, in other instances adjacent electrodes can provide good detection accuracies, e.g., three adjacent electrodes (from motor areas) for subject KR had HF-differences of 92, 68, and 63. Our ongoing methodological work includes: 1) optimization of the detection methods, and 2) adaptation of ERP detection methods for use with people who cannot perform physical movements. We are also investigating the ability of subjects to control or modify their ERP quality, given appropriate feedback. In the near future we hope to demonstrate the real time use of a direct brain interface for communication or other functional tasks. While these studies will initially be performed with epilepsy surgery patients, we are also considering studies that involve short-term implantation of electrodes in subjects from potential user groups. The testing under these conditions will explore issues surrounding imagined movements, accuracy, reliability, and learning over time. Clinical trials of our direct brain interface will follow such studies. These trials will be designed to demonstrate significantly improved communication options and increased opportunities for people with severe disabilities such as locked-in syndrome, end-stage amyotrophic lateral sclerosis, and others.

REFERENCES [1] P. J. Cilliers and A. J. W. Van Der Kouwe, “A VEP-based computer interface for C2-quadriplegics,” Eng. Med. Biol., pt. 3, vol. 15, pp. 1263–1264, 1993. [2] E. E. Sutter, “The brain response interface: Communication through visually-induced electrical brain responses,” J. Microcomputer Appl., vol. 15, pp. 31–45, 1992.

[3] S. P. Levine, J. E. Huggins, S. L. BeMent, R. K. Kushwaha, L. A. Schuh, E. A. Passaro, M. M. Rohde, and D. A. Ross, “Identification of electrocorticogram patterns as the basis for a direct brain interface,” J. Clin. Neurophysiol., vol. 16, no. 5, pp. 439–447, 1999. [4] J. E. Huggins, S. P. Levine, S. L. BeMent, R. K. Kushwaha, L. A. Schuh, E. A. Passaro, M. M. Rohde, D. A. Ross, K. V. Elisevich, and B. J. Smith, “Detection of event-related potentials for development of a direct brain interface,” J Clin. Neurophysiol., vol. 16, no. 5, pp. 448–455, 1999. [5] J. E. Huggins, “Event-related potentials from the human cortex as the basis of a direct brain interface for the operation of assistive technology,” Thesis, The University of Michigan, Ann Arbor, 1997. [6] S. Uematsu, R. Lesser, R. S. Fisher, B. Gordon, K. Hara, G. L. Krauss, E. P. Vining, and R. W. Webber, “Motor and sensory cortex in humans: Topography studied with chronic subdural stimulation,” Neurosurgery, vol. 31, pp. 59–71, 1992.

Simon P. Levine received the B.A. degree in mathematics from the University of California at Los Angeles (UCLA) in 1973. He also received the M.A. degree in mathematics, the M.S. degree in bioengineering, and the Ph.D. degree in bioengineering from the University of Michigan, Ann Arbor, in 1975, 1976, and 1983 respectively. He currently holds the positions of Associate Professor and Director of Rehabilitation Engineering in the Department of Physical Medicine and Rehabilitation and the Department of Biomedical Engineering at the University of Michigan and is active in clinical service, research, and teaching. His research interests are focused on control of and performance with computerized and robotic assistive technology systems. He has over 100 publications in the area of rehabilitation engineering and technology has been awarded three U.S. patents. Dr. Levine is a member of the Rehabilitation Engineering and Assistive Technology Society of North America (receiving the RESNA Distinguished Service Award in 1992 and 1998) and the IEEE Engineering in Medicine and Biology Society.

Jane E. Huggins received the B.S. degree in computer engineering with a Biomedical Option and an Art minor from Carnegie Mellon, Pittsburgh, PA, in 1987 and the M.S. degree in bioengineering, the M.S.E. degree in computer science and engineering, and the Ph.D. degree in biomedical engineering from the University of Michigan, Ann Arbor, in 1993, 1994, and 1997 respectively. She currently holds the position of Assistant Research Scientist in the Department of Biomedical Engineering at the University of Michigan and is active in research and teaching. Her research interests are focused on the development of a direct brain interface for people with severe motor impairments. Dr. Huggins is a member of the Rehabilitation Engineering and Assistive Technology Society of North America and the IEEE Engineering in Medicine and Biology Society.

Spencer L. BeMent (SM’92) received the B.S.E. and M.S.E. degrees in electrical engineering and the Ph.D. degree in bioengineering from the University of Michigan, Ann Arbor, in 1960, 1962, and 1967 respectively. He progressed through the academic ranks to full Professor of Electrical Engineering and Computer Science. He has contributed strongly to the Bioengineering Program/Biomedical Engineering Department since its inception in 1963. He has contributed to the research areas of neurophysiology and electrophysiology, mobile robotics, bioinstrumentation and transducers, microprobe development, plastic surgery, and control and signal processing associated with computerized and robotic assistive technology systems. He has contributed several education papers in bioinstrumentation and computer aided instruction as part of his more than ninety publications in technical journals.

LEVINE et al.: DIRECT BRAIN INTERFACE BASED ON EVENT-RELATED POTENTIALS

Ramesh K. Kushwaha received the B.Eng. degree in electrical engineering from the M.L.N.R. Engineering College, Allahabad University in 1979 and the M. Eng. degree in biomedical engineering from the University of Saskatchewan, Sask., Canada, in 1983, and the M.S. and Ph.D. degrees in bioengineering from the University of Michigan, Ann Arbor, in 1986 and 1990, respctively. He currently holds Medical Engineer and research investigator positions in the Neurology department at the University of Michigan, Ann Arbor. His research interests include biosignal processing and bioinstrumentation.

Lori A. Schuh received the B.S.degree in chemistry and history from Juniata College, Huntingdon, PA, in 1984 and the M.D. degree from UMDNJ-Robert Wood Johnson Medical School, NJ, in 1988. She completed a Neurology residency at the University of Virginia, Charlottesville, in 1992 and fellowships in EEG and Epilepsy at the University of Michigan, Ann Arbor, in 1994. She is a Senior Staff Neurologist in the Comprehensive Epilepsy Program at Henry Ford Hospital and is active in clinical service, research, and teaching. Her research interests are focused on outcomes following resective epilepsy surgery, intracranial ictal and interictal patterns and cortical potentials. She has over 50 publications in these areas. Dr. Schuh is a member of the American Academy of Neurology, American Epilepsy Society, and the medical honor society Alpha Omega Alpha.

Mitchell M. Rohde (M’97) received B.S. and M.S. degrees in electrical engineering in 1994 and 1996, respectively, and the M.S. and Ph.D. degrees in biomedical engineering in 1997 and 2000, respectively, all from the University of Michigan, Ann Arbor. He has worked in the areas of neurological signal processing and medical instrumentation, and holds a U.S. patent on a novel instrument design. He was a Whitaker Fellow from 1994 to 1999. He currently works at Quantum Signal LLC, a signal processing solutions company he cofounded Dr. Rohde is a member of Sigma Xi, Eta Kappa Nu, Tau Beta Pi, and AAMI.

Erasmo A. Passaro received the B.S. degree in biology at the University of Chicago, IL, in 1984 and the M.D. degree from the Robert Wood Johnson Medical School, NJ, in 1988. He completed an internship in internal medicine (1988),his neurology residency (1993), and his fellowship training in epilepsy/clinical neurophysiology (1995) all at the University of California at Los Angeles (UCLA). Currently, he is an Assistant Professor in the Department of Neurology and dDirector of the Adult Epilepsy Laboratory at the University of Michigan, Ann Arbor. His research interests include the surgical treatment of medically refractory epilepsy; functional neuro-imaging of the epilepsies, and comparing motor localization by event related potentials with cortical stimulation. He has published several abstracts and papers on clinical epilepsy. Dr. Passaro is a member of the American Epilepsy Society and the American Clinical Neurophysiology Society. He is a past recipient of the Epilepsy Foundation of America Research Award, serves as an Editor for E-Medicine Neurology.

185

Donald A. Ross graduated first in his class from the Indiana University School of Medicine, IN. He completed a residency in neurological surgery at the University of California, San Francisco, and a Fellowship in cranial base surgery at the University of Pittsburgh, PA. He is Board Certified in Neurological Surgery. He joined the faculty of the University of Michigan Medical Center, Ann Arbor, as an Assistant Professor of Surgery. His practice at the University of Michigan included cranial base surgery, epilepsy surgery, brain tumor surgery, and stereotactic radiosurgery. He developed a national and international reputation and was promoted to Associate Professor with tenure. As a result of his close collaboration with the head and neck surgeons on difficult cranial base cases, he was given a joint appointment in the Otolaryngology Department. During his ten years on the University of Michigan faculty, he published over 80 scientific articles and book chapters and holds a patent on a new surgical device. He is currently part of the Medford Neurological and Spine Clinic, Medford, OR.

Kost V. Elisevich received the M.D. degree from the University of Western Ontario (UWO), London, Canada, in 1978 and completed his neurosurgical training at the Montreal Neurological Institute and the affiliated hospitals of McGill University, Montreal, P.Q., Canada, in 1986. He also received the Ph.D. degree in anatomy from UWO in 1986. He is a Fellow of the Royal College of Physicians and Surgeons of Canada and of the American College of Surgeons and holds a position of Associate Professor in the Department of Neurological Surgery at Case Western Reserve University. He is a Senior Staff Member and Surgical Director of the Comprehensive Epilepsy Program at the Henry Ford Hospital, Detroit, MI. His basic research interests are in mechanisms underlying epileptogenesis, intercellular electrical coupling and neuronoglial interactions. His clinical research interests center upon electrical pacing or perturbation of epileptic foci as a means of overcoming epileptogenicity. Dr. Elisevich is a member of the Society for Neuroscience, American Epilepsy Society and Congress of Neurological Surgeons.

Brien J. Smith received the undergraduate degree in applied biology from Ferris State College, Big Rapids, MI, in 1983 and the medical degree from Wayne State University, Detroit, MI, in 1987. He completed an internship at Oakwood Hospital, Dearborn, MI, in 1988, his neurology residency in 1991, and his fellowship training in EEG/Epilepsy at Indiana University in 1992. Currently, he is Directory of the Epilepsy Monitoring Unit in the Comprehensive Epilepsy Program at the Henry Ford Hospital in Detroit, Michigan and an Assistant Professor in the Department of Neurology at Case Western Reserve University, Cleveland, OH. His clinical and research interests include the surgical treatment of medically refractory epilepsy, prolonged video/EEG monitoring, SPECT, electrocorticograhy, intracarotid amobarbital testing, cortical mapping, magnetoencephalography, electronic (gap) junctions in epilepsy, effects of ionizing radiation on focal epilepsy and magnetic resonance spectroscopy. He has published several abstracts and papers on clinical epilepsy. He is a co-coordinator of the epilepsy surgery boards. Dr. Smith is a member of the American Medical Association, the American Epilepsy Society, for which he serves on the membership committee, the American Academy of Neurology, the American Clinical Neurophysiology Society, the Michigan Neurological Association, the Society for Neuroscience, the National Association of Epilepsy Centers, the Michigan State Medical Society, and the Wayne County Medical Association.