BRAIN-COMPUTER INTERFACING Rajesh P. N. Rao and Reinhold Scherer
Recently, CNN reported on the future of brain-computer interfaces (BCIs) [1]. Brain-computer interfaces are devices that process a user’s brain signals to allow direct communication and interaction with the environment. BCIs bypass the normal neuromuscular output pathways and rely on digital signal processing and machine learning to translate brain signals to action (Figure 1). Historically, BCIs were developed with biomedical applications in mind, such as restoring communication in completely paralyzed individuals and replacing lost motor function. More recent applications have targeted non-disabled individuals by exploring the use of BCIs as a novel input device for entertainment and gaming. The task of the BCI is to identify and predict behaviorally induced changes or “cognitive states” in a user’s brain signals. Brain signals are recorded either non-invasively from electrodes placed on the scalp (electroencephalogram, EEG) or invasively from electrodes placed on the surface of or inside the brain. BCIs based on these recording techniques have allowed healthy and disabled individuals to control a variety of devices [2]. In this article, we will describe different challenges and proposed solutions for non-invasive brain-computer interfacing. CHALLENGES IN NON-INVASIVE BRAIN COMPUTER INTERFACING EEG has emerged as the single most important non-invasive source of brain signals for brain-computer interfacing in humans. Two major problems confronting BCI developers using EEG are its non-stationarity and its inherent variability. Data from the same experimental paradigm but recorded on different days or even different times on the same day are likely to exhibit differences due to, for instance, shifts in electrode positions between sessions or changes in electromechanical properties of the electrodes (e.g., changing impedances). Additionally, the noisy, non-linear superposition of the electrical activity of large populations of neurons as measured on the scalp can mask the underlying neural patterns and hamper their detection. The user's current mental state (e.g. excessive workload or stress) may impact the ability to focus and generate specific mental events. Due to these factors, statistical signal processing and machine learning techniques play a crucial role in recognizing EEG patterns and translating them into control signals. BRAIN-COMPUTER CO-ADAPTATION An interesting problem confronting BCI developers is that the brain itself is a highly adaptive device, raising the question of how much of the learning should be relegated to the machine and how much should be left to the brain. At one extreme are approaches that fix a mapping a priori between the brain and a controlled device, and rely entirely on the brain to adapt to this fixed mapping.
Studies using invasive recording of single neurons or neural populations show that the motor system can rapidly learn to generate appropriate patterns for a fixed mapping (e.g., [3]). However, with EEG in humans, the same approach may take months to achieve an adequate level of performance [4]. Current approaches therefore typically rely on both user adaptation and machine learning. EEG activity patterns are recorded from the user prior to BCI use and this data is utilized to train a pattern recognition algorithm for classification or regression. Data collected from subsequent sessions are then used to further update the classifier or regresser to the user’s most recent brain patterns. Simultaneous on-line adaptation by the user and BCI remains a topic of active research. SIGNAL TYPES USED IN NON-INVASIVE BCIs The two major types of EEG signals used in BCIs are evoked potentials (EPs) and oscillatory activity patterns. EPs are electrical potential shifts which are phase-locked to external perceptual events such as a rare visual stimulus. EPs are typically analyzed by averaging EEG data over time beginning at the start of the perceptual event for a duration of up to 1 second. Oscillatory activity patterns, on the other hand, can be voluntarily induced by the user, for example, through the imagination of kinaesthetic body movements. Such imagery typically causes a decrease or increase in power in particular frequency bands. This decrease or increase in power is usually referred to as event-related desynchronization (ERD) or event-related synchronization (ERS) respectively. Since EPs are stereotypical brain responses that are stable over time, very little adaptation may be required on the part of the user. Oscillatory patterns of a user, however, typically change over time as a result of feedback during BCI use, making parameters that were learned off-line sub-optimal. Co-adaptation is therefore required: brain signals recorded during feedback are analyzed to track changes in oscillatory patterns and the BCI is updated whenever required. The two examples below illustrate the use of EPs and oscillatory patterns for achieving brain-computer interaction in physical and virtual environments. BRAIN COMPUTER INTERFACING USING EVOKED POTENTIALS One type of EP that has been used successfully in BCIs is the P300. The P300 is so named because it is characterized by a positive potential shift about 300 ms after the presentation of a perceptually significant event embedded within a series of routine stimuli. Figure 2A illustrates an experimental paradigm that uses the P300 to allow a user to select from a menu of choices. The choices are presented in a grid format on a computer screen. The choices in this experiment correspond to segmented images of objects from the current field of vision of a humanoid helper robot [5]. The P300 (Figure 2A, panel 1) is used to infer which object the BCI user would like the robot to pick up for transport to a different location (Figure 2A, panels 2-4). Once the object has been picked up, the menu switches to images of possible destination locations and the P300 is again used to infer the user’s choice of a destination for the robot.
To make a selection using the P300, the user focuses his or her attention on the image of choice while the borders of the images are flashed one at a time in a random order. Each image is flashed multiple times in this random order. Flashes on the attended image generate P300 responses while the other flashes do not (see Figure 2A, panel 1). A linear support vector machine (SVM) with slack variables [6] was trained to discriminate between P300 and non-P300 responses. Labeled training data for this purpose was obtained in a 10 minute data collection protocol at the beginning of the experiment. The input to the SVM was a low-dimensional feature vector obtained by applying a small set of spatial filters to 32 channels of EEG data recorded from electrodes placed over the entire scalp. These filters are “spatial” because they are applied not to samples over a time period but to the 32 samples spatially distributed over the scalp. The output of a filter is a linear weighted combination of the 32 EEG channels at each time step. Each channel was first band pass filtered in the 0.5-30 Hz range to exclude noise typically present at higher frequencies. The spatial filters were learned from the labeled data as follows. Let E denote the 32 x N matrix of EEG data, where N is the number of time points (in this case, representing a duration of 500 ms from the onset of each flash). Applying a 32 x 1 spatial filter f to the 32-channel EEG data results in the following time series of filtered data: x f TE . To aid classification, we would like a filter f that maximizes the squared distance between the means of the filtered data for the two classes (P300 and non-P300 responses) while minimizing the within-class variance. This is equivalent to maximizing the criterion:
J( f )
tr ( Sb ) tr ( S w )
where tr denotes trace of a matrix, and Sb and Sw are the between-class and within-class scatter matrix respectively of the filtered data x. Maximizing J can be shown to be equivalent to a generalized eigenvalue problem whose solution is a set of orthonormal eigenvectors (filters) f ordered by their eigenvalues: the larger the eigenvalue, the more discriminative the filter (see [5] for details). The three filters with the three largest eigenvalues were found to capture most of the discriminative information for the training data. These filters were applied to the 32-channel EEG data to yield three filtered outputs at each time step. This lowdimensional filtered time series data was used to train the classifier. During the operation of the BCI, the image with the highest number of P300 classifications after the completion of all flashes was selected as the user’s choice. An average classification accuracy of 95% across nine users was achieved for discriminating between four choices, using five flashes per choice. With the implemented rate of 4 flashes per second, the selection of one out of four options takes 5 seconds, yielding an information transfer rate of 24 bits/min.
BRAIN COMPUTER INTERFACING USING OSCILLATORY ACTIVITY In the P300-based BCI described above, command generation was synchronized with an externally generated stimulus or cue. Such BCIs are called cue-guided. In contrast, BCIs that allow the user to voluntarily modulate brain activity wheneve the user wishes to issue a command are called self-paced. Selfpaced BCIs are typically based on detecting changes in oscillatory activity. For example, imagining movements can cause changes in oscillatory EEG activity in the 8-30 Hz frequency range over sensorimotor areas (Figure 2B, panel 1). Furthermore, different types of imagined movements can result in different oscillatory patterns which can be classified using machine learning. As an example, consider navigating in a virtual environment: one could use left hand, right hand and foot motor imagery to move left, right, and forward respectively [7]. The subject’s task in the experiment was to navigate and find coins that are scattered randomly at different locations in the environment (Figure 2B, panels 2-3). A committee of Fisher’s linear discriminant analysis (LDA) classifiers [6] with majority voting was trained to discriminate between the three types of motor imagery. An additional LDA classifier was trained to detect whether the subject was engaged in motor imagery or not; only when motor imagery was detected was the committee of classifiers used to predict the type of movement. Features for classification were estimated from 1 second segments by band pass filtering the EEG signal for several frequency bands, and squaring and calculating the mean over the squared values for each band in each segment. To decrease variability, features used in classification were based on the logarithm of the band power estimates. The most discriminative frequency bands were identified for each subject independently. To allow real-time interaction, classification was performed every 40 ms. Given the focus on motor imagery, data was recorded from six EEG sensors placed over appropriate sensorimotor areas. Techniques for on-line muscle artifact detection and eye movement reduction were also used to reduce contamination of the EEG signal (see [7] for details). After a total of about 5 hours of co-adaptive training over several days, the average 3-class accuracy of the LDA committee classifier reached approximately 80%, with a false positive rate for motor imagery detection (by the additional LDA classifier) of about 17%. Subjects were able to successfully use the BCI to navigate and locate the coins in the environment (Figure 2B, panel 3). NON-INVASIVE BCIs: THE FUTURE The noisy nature of EEG and the fact that brain activity patterns are typically subject-specific means that signal processing and subject-specific optimization are essential for successful brain-computer interaction. The nonstationarity and inherent variability of the EEG, along with limited sample size and limited knowledge about the underlying signal, makes BCIs a challenging domain for signal processing. Much of past BCI research has focused on cue-based BCIs, where the
mental states are more or less well defined. An important challenge for the future is the design and implementation of self-paced BCIs, where a number of distinct patterns have to be reliably detected in ongoing brain activity. Although there have been several prototype systems (e.g., the navigation system discussed above), there is room for improvement. Another important issue is usability. Current electrode caps and wet electrodes are not practical for everyday use in non-laboratory settings. Future recording devices will need to be less time consuming to set up, more comfortable to wear, and less expensive to purchase and maintain. The first generation of wireless neuro-signal recording devices with dry electrodes have started appearing on the market (e.g., Emotiv Systems, San Francisco, CA, USA). Whether and to what extent these new technologies prove to be useful for BCI applications remains to be seen. Finally, the problem of co-adaptation of brain and machine in BCIs presents an interesting challenge for pattern recognition and machine learning algorithms. Although some promising preliminary results have been obtained, an overarching theory of co-adaptation remains to be developed. Such a theory would entail finding statistical methods that can predict changes in brain activity, allowing the BCI to adapt in sync with the human user for achieving the common goal of direct brain-computer interaction with the external world.
AUTHORS Rajesh P. N. Rao (
[email protected]) is an Associate Professor in the Department of Computer Science and Engineering at the University of Washington, Seattle, Washington, USA. Reinhold Scherer (
[email protected]) is a Posdoctoral Research Fellow in the Department of Computer Science and Engineering at the University of Washington, Seattle, W ashington, USA. REFERENCES [1] A. Hammock, “The Future of Brain-Controlled Devices”, CNN Online, January 4, 2010, Online. [Available] http://www.cnn.com/2009/TECH/12/30/brain.controlled.computers/index.html [2] G. Dornhege, J. d. R. Millan, T. Hinterberger, D. McFarland, & K. R. Müller (Ed.). (2007). Towards Brain-Computer Interfacing. Cambridge, MA: The MIT Press. [3] T. Blakely, K. J. Miller, S. P. Zanos, R. P. N. Rao, J. G. Ojemann. Robust, long-term control of an electrocorticographic brain-computer interface with fixed parameters. Neurosurg. Focus. 2009 Jul;27(1):E13. http://thejns.org/doi/full/10.3171/2009.4.FOCUS0977
[4] A. Kübler, B. Kotchoubey, T. Hinterberger, N. Ghanayim, J. Perelmouter, M. Schauer, C. Fritsch, E. Taub, and N. Birbaumer, "The thought translation device: a neurophysiological approach to communication in total motor paralysis," Experimental Brain Research, vol. 124, no. 2, pp. 223-232, January 1999. [Online]. Available: http://www.metapress.com/content/FGUE9H81NLPF2JBB [5] C. J. Bell, P. Shenoy, R. Chalodhorn, and R. P. N. Rao, "Control of a humanoid robot by a noninvasive brain–computer interface in humans," J. Neural Eng., vol. 5, no. 2, pp. 214+, June 2008. [Online]. Available: http://dx.doi.org/10.1088/1741-2560/5/2/012 [6] K. R. Müller, C. W. Anderson, and G. E. Birch, "Linear and nonlinear methods for brain-computer interfaces," Neural Systems and Rehabilitation Engineering, IEEE Transactions on, vol. 11, no. 2, pp. 165-169, July 2003. [Online]. Available: http://dx.doi.org/10.1109/TNSRE.2003.814484 [7] R. Scherer, F. Lee, A. Schlgl, R. Leeb, H. Bischof, and G. Pfurtscheller, "Toward self-paced brain-computer communication: Navigation through virtual worlds," Biomedical Engineering, IEEE Transactions on, vol. 55, no. 2, pp. 675682, 2008. [Online]. Available: http://dx.doi.org/10.1109/TBME.2007.903709
Figure 1: Basic Components of a Brain-Computer Interface. Brain activity is translated into a control signal for an external device using a sequence of processing stages. The user receives feedback from the device, thereby closing the loop.
Figure 2. Two Examples of Non-Invasive BCIs. (A) BCI based on Evoked Potentials. 1) Averaged response over 10 trials for attended (P300, solid line) and unattended images (dashed line). 2) Humanoid robot in front of two objects waiting for input from BCI user. 3) User attends to image of desired object while borders are randomly flashed (red square). 4) Humanoid robot picks up the object selected by the user. (B) BCI based on oscillatory activity. 1) Four trials of event related synchronization in the 10-13 Hz frequency band induced by foot motor imagery initiated at time 0s. 2) BCI user navigating through a virtual environment (VE). 3) Map of the VE showing the trajectory of the BCI user. Yellow markers indicate locations of coins the user was instructed to collect.
