P300-based Brain Computer Interface Mouse with ... - CiteSeerX

P300-based Brain Computer Interface Mouse with Genetically-optimised Analogue Control Luca Citi Department of Computer Science University of Essex, UK and Scuola Sant’Anna Pisa, Italy Caterina Cinel Department of Psychology University of Essex, UK

Riccardo Poli Department of Computer Science University of Essex, UK

Francisco Sepulveda Department of Computer Science University of Essex, UK

Technical Report CSM-451 Department of Computer Science University of Essex ISSN: 1744-8050 May 2006 Submitted to IEEE Transactions on Neural Systems and Rehabilitation Engineering

1

Abstract In this paper we propose a BCI mouse based on the manipulation of P300 waves in EEG signals. The system is analogue in that at no point a binary decision is made as to whether or not a P300 was actually produced in response to the stimuli. Instead, the 2–D motion of the pointer on the screen, using a novel BCI paradigm, was controlled by directly combining the amplitudes of the output produced by a filter in the presence of different stimuli. This filter and the features to be combined within it are optimised by an evolutionary algorithm. In the paper we illustrate our approach, describe our system, and report the results of the testing and validation of the system with 6 participants. The results are very promising, with all 6 participants being able to control the mouse with excellent accuracy only minutes after wearing the electrode cap despite their having no previous training with this system.

1

Introduction

Over the past few years an increasing number of studies have evaluated the possibility of converting signals generated from the brain (especially EEG signals) into appropriate signals for applications in various disciplines, from virtual reality to hands-free control of augmentative communication technologies for individuals with disabilities. Brain-Computer Interface (BCI) systems can measure specific signals of brain activity intentionally and unintentionally induced by the participant and translate them into device control signals. The development of BCI is particularly important as it could provide, for example, new technology that does not require muscle control (e.g., [15, 45, 31, 3, 38, 21, 17, 42, 10]; see [46] for a comprehensive review). BCI studies have showed that non-muscular communication and control is possible and might serve useful purposes for those who cannot use conventional technologies. To people who are “locked-in” or lack any useful muscle control, a BCI system could give the ability to answer simple questions, express wishes to their caregivers, control the environment, perform slow word processing, or even operate a prosthesis (see [4, 47, 26, 30, 40]). Naturally, BCI systems can also be used for able-bodied human-machine interfaces, for example for control applications in robotics and intelligent environments (be they real or virtual). However, in these cases they tend to complement other forms of control (based on musculoskeletal activity) rather than being the only source of user input. Many factors determine the performance of a BCI system. These factors include the brain signals measured, the signal processing methods that extract signal features, the algorithms that translate these features into device commands, the output devices that execute these commands, the feedback provided to the user, and the characteristics of the user.

2

The signals used in BCI studies to date include P300 waves [15] and other event related potentials (ERPs), µ or β rhythms [45], evoked potentials (EPs) [41, 28, 21], ERD/ERS [31], slow cortical potentials [3] recorded from the scalp, cortical neuron activity recorded by implanted electrodes (see [38] for a review), neuromagnetic signals recorded through MEG [17], BOLD responses recorded through fMRI [42] and activity-related, localised brain oxygenation recorded through near infrared systems (NIRS) [10]. ERPs are relatively well defined shape-wise variations to the ongoing EEG elicited by a stimulus and temporally linked to it. As opposed to EPs (which are a direct, reflex-like response to a stimulus), however, ERPs include an exogenous response, due to the primary processing of the stimulus, as well as an endogenous response, which is a reflection of higher cognitive processing induced by the stimulus [11]. The P300 wave is a late appearing component of ERPs with a latency of about 300 ms which is elicited by rare and/or significant stimuli (visual, auditory, or somatosensory). Effectively P300 potentials are ERP components that vary according to the degree of attention the user dedicates to a stimulus. This is what makes it possible to use them in BCI systems to determine user intentions. Given the point-and-click nature of most modern user interfaces for computers and other devices, one important application of BCI is controlling 2–D pointer movements. Over the years, there have been some attempts to develop BCI systems for this purpose, the most successful of which, to date, being those based on frequency analysis, in particular the detection of µ or β rhythms [43, 23, 44], and those using invasive cortical interfaces (e.g., [7, 13]). The former, however, require lengthy training periods before users can control them, while the latter are not very practical requiring surgery, presenting risks of infections, etc. These problems could be overcome by non-invasive systems based on the use of P300s. To date, however, only very limited studies with this approach have been performed [35, 2] and these have reported promising but limited successes. (A fuller review and comparison of all these approaches will be provided in Section 5.2.) In this paper, a P300-based system for the two-dimensional control of a cursor on a computer screen is presented. Four unobtrusive squares are superimposed on the screen, near its edges, and are used to represent the four directions up, right, down and left. These flash in random order. The user moves the mouse pointer in a particular direction by simply devoting his/her attention to the flashes of the square corresponding to the desired direction. By selectively focusing his/her attention, the user influences the endogenous components following each stimulus. So, by analysing the recorded EEG, the system can infer the desired direction and moves the cursor accordingly. Some of the techniques used in the present work are well-established technology. However, unlike previous approaches, we offer several novel features, which allow our system to deliver significantly better performance than other 2–D pointer control systems based on P300s. 3

Firstly, most P300-based BCI systems require a binary classification of the response to each of a set of stimuli as target or non-target. Instead, in our system the stimulus response for each of the four directions directly affects the movement of the pointer: the vertical component of the movement is determined by the difference between a filter’s output resulting from the stimuli up and down, while the horizontal component is similarly determined by the responses to the stimuli right and left. That is, logically our BCI mouse is an analogue (as opposed to digital) device (although, of course, physically all its signal processing is digital). Secondly, the reliable detection of P300s in a single trial is still very difficult. In most systems an action is performed only when the system is reasonably certain as to the intentions of the user. However, this requires averaging over multiple presentations of the stimuli, avoiding high stimulus-presentation rates or both, which in turn reduces the bit rate of the system. In this work, instead, we take a single trial approach and we use a high presentation rate. That is, our BCI mouse performs an action at the end of every cycle of stimulus presentation and this happens approximately once per second. This is possible also thanks to the self-correcting nature of the task (more on this later). Finally, we use these novel characteristics in conjunction with evolutionary algorithms, which effectively adapt the design of the system to each user and each session, thereby maximising performance. Evolutionary algorithms are search and optimisation algorithms inspired by Darwinian evolution, which have been applied very successfully to a large number of difficult problems with human-competitive results (e.g., see [25] for a comprehensive list). Many variants of evolutionary algorithms exist [19, 24, 27, 37, 39]. In this work we use Genetic Algorithms (GAs) [19]. In GAs a population of random tentative solutions to a problem (e.g., BCI classifiers) is created and evaluated to assess the degree to which they solve the problem at hand (e.g., the fraction of P300 events correctly classified). New generations of individuals are created by recombining the characteristics of individuals in the previous generation, giving better performing parent solutions a higher chance of reproduction. In this way, like in nature, generation after generation, better and better solutions emerge. However, unlike in nature where thousands of years are necessary to evolve highly fit individuals, in our BCI mouse the training phase performed by a GA requires only a few minutes. Because of their effectiveness, GAs are now frequently applied in the area of image and signal processing (e.g., [34, 33, 5, 18, 36] and see [6] for a recent review of the state of the art in this sector). In our BCI mouse, the analysis of the P300 components is preceded by a preprocessing phase in which the Continuous Wavelet Transform (CWT) of each EEG channel is performed. CWT is computed at several tens of scales and times after the presentation of the stimuli. So, the ERP response to each stimulus is a large array of features the first dimension of which represents the EEG channel used, the second dimension represents the CWT scale, and the third dimension represents the time elapsed after the presentation of 4

the stimulus. An enormous number of features is, therefore, available to the system. This is done so as to avoid losing important information for the subsequent stages of P300 processing. However, it is well-known that, in the presence of large numbers of features, adaptive classifiers tend to have a very hard time, and it is, usually, necessary to perform some form of selection to identify the most informative features. In this work we take a wrapper approach to feature selection [22] where the selection of features and the training of an adaptive system using them are performed jointly by the evolutionary algorithm. Preliminary results with an evolutionary approach to P300 processing using one of the Donchin datasets made available for the 2003 BCI competition [1] were reported in [9]. Encouraged by these results with the approach we decided to extend the work and build the complete BCI mouse system with realtime processing and classification described in this paper. The system makes it possible for a person having undergone no previous training and within minutes of wearing the electrode cap, to move a pointer to any location of a computer screen. The paper is organised as follows. In Section 2 we provide details on the participants, the stimuli, and the protocol followed during the training and use of the BCI mouse. We then describe (Section 3) the various phases through which EEG signals are translated into control commands in our BCI mouse. We report on the experimentation of the system with 6 volunteers in Section 4 and discuss our findings in Section 5. Finally, we draw some conclusions in Section 6.

2 2.1

Experimental Methods Participants

After several preliminary experiments performed during the development phase of the system, we performed extensive testing of the final design and approach. For this phase we used 6 participants. Table 1 reports the gender and age of each participant. Participants A–E were able bodied, while participant F has a neuromuscular disability (Dejerine-Sottas’ hypertrophic neuropathy).

2.2

Stimuli and Protocol

In building our BCI Mouse we drew inspiration from Donchin Speller [15, 12], where P300s were used to determine on which stimulus, out of a large set of stimuli concurrently presented, an observer’s attention was focused. In our BCI mouse, the following paradigm was used:

5

Table 1: Participants involved in the experiments Participant Gender Age A M 25 B M 28 C F 35 D M 44 E M 23 F M 40

Figure 1: Our BCI mouse when the stimulus “up” is presented. • Four gray rectangles are constantly superimposed on whatever is shown on a computer screen. They are unobtrusive being relatively small and aligned with the upper, lower, left and right borders of the screen (see Figure 1). • Each rectangle corresponds to a possible direction of movement for the mouse cursor. • Two additional rectangles, one on the lower left corner of the screen, the other on the lower right corner of the screen, can also be present. When they are enabled, they perform the function of the left and right buttons of an ordinary mouse. When this function is not required, the two “button” rectangles are still logically considered to be present, but are not physically shown on the screen to avoid distracting the user. At 180 ms intervals, this static display is altered by temporarily changing the colour of one of the rectangles from gray to bright red (for the rectangles controlling the direction of motion) or blue (for the rectangles representing mouse buttons). The stimulus remains 6

brightly coloured for 100 ms, after which it becomes gray again. As a result, the rectangle appears to flash. Which particular rectangle is selected for flashing is determined randomly. However, flashing of stimuli is done via sampling without replacement. In other words, after flashing, a rectangle is not allowed to flash again until all other rectangles have flashed once (i.e., for each flashing rectangle there is a refractory period of 900 ms). With these choices, after each flash, the display returns to its standard gray colouring for 80 ms. However, when the buttons’ function is disabled, as in all the experiments reported in this paper, the rectangles corresponding to the mouse buttons are not drawn and, so, their flashing has not effect except creating a 180 ms pause in the sequence of flashes. Naturally, when the two “button” rectangles happen to be “flashed” in sequence, a 360 ms pause is produced instead, although this event happens infrequently. As we stressed in previous work [8], repetition blindness [20] and other perceptual errors often observed in Rapid Serial Visual Presentation experiments can reduce the accuracy of BCI systems. To limit the risks of this type of perceptual errors occurring in our BCI mouse we ensured that the last rectangle to flash in each series of 6 was never allowed to be the first to flash in the following series. Participants were comfortably sitting on an armchair with their neck sustained by a Cshaped inflatable travel pillow to reduce muscular artifacts. The eyes were approximately 1 meter from a 21 inch CRT screen. Participants were instructed that in order to move the mouse pointer in a particular direction they needed to focus their attention on the corresponding rectangle on the screen. To facilitate this, they were asked to count how many times that particular rectangle flashed during a sequence of mouse movements in the same direction. Participants were asked to ignore whatever else was happening on the screen (particularly the other flashing stimuli). The system has three modes of operation: training, validation and normal use. During the phases of acquisition of a training set for our control system, the experimenter selected one of the rectangles on the screen as a target, and participants were asked to focus their attention only on the target stimulus. During training, the screen had a light gray background with no stimuli other than the rectangles mentioned above. During validation phases, participants were asked to perform the same task with the same stimuli and an homogeneous background, except that, at this stage, the system had already been optimised and we could immediately show participant the trajectory of the mouse pointer produced by their efforts. During normal use, of course, all sorts of windows and icons were present on the screen in addition to the rectangles necessary to control the BCI mouse. Because the rectangles flash in random order, the sequence of flashes of any particular one of them appears to be very random with onset-to-onset intervals ranging from 360 ms (for example, when the rectangle of interest is the last to flash in a series of 6, and the second in the following series) to 1980 ms (when the rectangle of interest is the first to flash in a series of 6, and the last in the following series). However, there was some regularity 7

Table 2: Impedance for each of the 6 participants used in our tests. Participant Typical impedance (max) Typical ear impedance A 7 kΩ (10 kΩ) 5 kΩ B 5-7 kΩ (20 kΩ) 7-10 kΩ C 5-7 kΩ (10 kΩ) 7 kΩ D 2-7 kΩ (10 kΩ) 5-7 kΩ E 7 kΩ (12 kΩ) 7 kΩ F 2-7 kΩ (12 kΩ) 12 kΩ in this process, and, on average every stimulus was flashed once in 1080 ms. During testing, we used the sequences of stimuli described above. Each run of our experiment involved presenting a full series of 4 flashing rectangles for 30 times (or 6 rectangles, if buttons were enabled). The participant was requested to count the flashes of one of the rectangles representing direction of motion. The process was repeated for each of the four possible directions, multiple times for each direction. Every few runs, participants were given a few minutes to rest and stretch. For participant A, 12 runs were recorded (3 for each of the four directions) while the others performed 16 runs (4 for each direction).

2.3

EEG apparatus

We used a 19 channel setup in a Mindset24 System to acquire and digitise EEG signals. This device allows the simultaneous acquisition of up to 24 channels, both in differential mode and in common reference mode. Each channel has 90 dB gain and a signal-to-noise ratio of 64 dB. Within the device the signal is first filtered with two order-3 analogue bandpass filters with 3 dB band between 1.5 and 34 Hz. The device has a typical Common Mode Rejection Ratio of 87 dB in the pass band. Once conditioned, the signal is sampled with a user-settable frequency of between 64 and 512 samples per second, and the samples are digitised with a 16 bit resolution. The samples are buffered and buffers are transferred to a PC via a SCSI interface. To simplify and ensure consistency in the positioning of electrodes we used an electrode cap (Electrocap International) with electrodes already pre-arranged according to the 10-20 international standard. Recent studies have shown that EEG electrode-skin impedances can be as high as 40 kΩ without significantly compromising the information content in the signal [16]. Nonetheless, efforts were made to obtain impedances below 7 kΩ in all experiments. The actual values obtained were as shown in Table 2, which reports, for each of our 6 participants, the impedances measured for the scalp electrodes and for the earlobes electrodes. Different reference configurations are possible [32]. Physically linked earlobes should be 8

avoided and re-referencing to the average of earlobes via software should be preferred [29]. However, in our case we noticed that the impedance for ear electrodes, besides being higher than for other electrodes, was very unstable and could significantly change just by moving the electrode or the head. By using the linked earlobes as reference, the unreliability of the reference could affect all the channels during the experiment causing not only changes in the amplitude of the signal but also shifts in the position of the virtual reference for the channels. Therefore, we decided to use a single earlobe reference and let the genetic algorithm deal with possible asymmetries.

3 3.1

Preprocessing and Translation Methods Preprocessing

We used 19 channels corresponding to the 10-20 international system to acquire EEG. Each channel is lowpass filtered using a FIR filter of order N = 30. We used FIR filters since they have a linear phase. The coefficients of the FIR filter were obtained via a least mean squares fit between the frequency response of a target ideal filter and the FIR filter’s frequency response. In this method one can specify the relative importance of errors in the pass band and errors in the stop band ranges via two weights, Wpass and Wstop , respectively. In our application we decided to set Wpass = Wstop = 1. The sampling frequency was 256 Hz, the frequency below which the signal was to be left unaltered was fpass = 34 Hz, while the frequency above which all components of the signal were required to be totally suppressed was fstop = 47 Hz. After low pass filtering, the signal is decimated by leaving every other sample out. This does not imply any loss of information or aliasing since the sampling rate of the decimated signal (128 Hz) is still well above twice the maximum frequency in the lowpass filtered signal.

3.2

Feature extraction

After filtering and decimation, we compute the features to be made available to the genetic algorithm which will be responsible for feature selection and classification. Our features are obtained by performing CWT on the 19 channels. We used the rbio3.3 wavelet family because its mother wavelet resembles the shape of typical P300s (once they are filtered by the bandpass filters present in our EEG acquisition hardware). The features are extracted for each epoch of the signal. An epoch is a window of 1 second starting when a stimulus is presented on the screen. In each epoch the system needs to process the EEG signals and appropriately emphasise and utilise a P300 if this is present in the epoch. Note, however, that stimuli are presented at a very fast rate, and, 9

so, epochs overlap. This makes the task much harder since the late responses to a stimulus may interfere with P300s generated by another. In our system, pointer control is determined by a filter (more on this later). The filter is applied to each epoch. That is, no averaging takes place. In principle, the filter can use all of the coefficients of the CWT of an epoch. We use 30 different scales between 2 and 40. The scales, however, are not equally spaced. This is because with a uniform distribution, the pseudo-frequencies corresponding to smaller scales are too far apart, while those corresponding to bigger scales are unnecessarily close. To circumvent this problem, we used the following quadratic distribution instead: a=

1 2 2 s + s+2 45 3

(1)

for s = 0, 1, · · · , 29. Since P300s occur within a well known time window after a stimulus, we compute CWT only for a range of relevant samples within each epoch. Only the following translations are considered ¯b = t + 30 (2) for t = 0, 1, · · · , 39. At a sampling frequency of 128 Hz, these correspond to a temporal window from 235 ms and 540 ms after the beginning of each epoch. To sum up: • for each epoch we process 19 channels, • for each channel we compute CWT at 30 different scales, and • at each scale we compute 40 consecutive samples. This gives us a 3–D array V(c, s, t) of features, where c indexes the channel, s the scale (see Equation 1) and t the time (see Equation 2) corresponding to a feature. In total we have 19 × 30 × 40 = 22, 800 components. As mentioned in Section 1, such a large number of features requires some form of feature selection. One can do this by either using a filter approach or a wrapper approach [22]. In a filter approach one selects the features (e.g., on the basis of some measure of information content) before training a classifier. This has the advantage of making each task – feature selection and training – independent and, therefore, more manageable. It presents the disadvantage, however, that the combination of features chosen may not be optimal for a particular classifier. The alternative – the wrapper approach – involves searching for the best features to use for classification and for the best parameters for a classifier concurrently. This implies a much bigger search space to explore, but, it also has the advantage of potentially giving the best possible combination of features for a particular task and classifier, resulting is a higher classification performance. 10

Given the high noise and limited information content in EEG signals and the difficulty of the task of recognising P300s, it appeared immediately obvious that in our BCI mouse we had to follow this second approach, although, as we mentioned before, in our case we are not training a classifier, but a filter that directly controls the motion of the mouse pointer. It was also obvious that the size of the search space and its discontinuities required a robust search algorithm such as a GA. In our system the GA needs to choose the best features and classifier parameters to control the 2–D motion of the pointer. As described in the next section at the hearth of our system is a polynomial classifier.

3.3

Polynomial Filter

Our general classifier is a polynomial P that combines, possibly non-linearly, a subset of elements of the feature matrix V: P (V) = a0 +

N X h=1

ah

M Y

V(ch,k , sh,k , th,k )eh,k

(3)

k=1

where N is the number of terms in the filter, M is the maximum number of features that can be combined in each term, eh,k is the power to which the k-th feature in term h is raised (note: this is not necessarily a positive number), the coefficients ch,k , sh,k , th,k identify which component of V is used as the k-th feature in the h-th term, and finally the values ai are coefficients weighing the relative effect of each term. The value P (V) is then passed through a squashing function φ to produce a classifier output, O(V) = φ(P (V)), in some interval [−Φ, Φ], where Φ is a positive constant. We could interpret a positive value of O(V) as an indication of the degree to which an epoch contains a target and, vice versa, for a negative O(V). In our experiments, targets are the stimuli on which a participant is focusing his/her attention for the purpose of selecting a particular direction of movement for the mouse cursor. Polynomial classifiers of this type are very powerful since they can perform a variety of useful operations on the features, including: linear combinations, ratios between features, squares (which have the important property of rectifying a feature), etc. Furthermore, polynomials have the important theoretical property of being general approximators (e.g., via the Taylor expansion), and, so, in principle, given enough terms, P (V) can approximate to the desired degree of accuracy any function in the space of all possible ways of combining the features in V. Non-linear polynomial filters similar to Equation 3 have been used very successfully in the treatment of other biomedical signals [34, 5]. A large set of preliminary experiments with evolving polynomial classifiers, however, revealed that linear terms play a predominant role in P300 detection, non-linear terms appearing rarely in the evolved classifiers and, when present, typically contributing marginally to the output. Therefore, to simplify the task of the GA and, correspondingly, increase the reliability of the optimisation of the 11

features, we decided to reduce the size of the search space by disallowing non-linear terms in Equation 3. That is, our polynomial classifier reduces to the linear combination: P (V) = a0 +

N X

aj · V(cj , sj , tj ).

(4)

j=1

3.4

Motion control

Every time a stimulus flashes, an epoch (an observation period of 1 second) starts. Epochs begin as soon as a rectangle changes colour (i.e., at the onset of the stimulus). Because, stimuli occur every 180 ms, the epochs corresponding to different stimuli overlap. For each epoch the system records the position of the corresponding stimulus on the screen and acquires and processes a 1 second segment of EEG signal. Epochs acquired during the period of training are annotated also with the ground truth (i.e., the direction the participant was asked to focus on). In epochs corresponding to target stimuli (i.e., where the rectangle that flashed was actually the one attended), we expect to find a P300 wave, while in epochs where a nontarget stimulus flashed this should not be present. The job of the GA is to use the information contained in the training set to evolve a polynomial filter that can discriminate between these two cases. Computer mice are fundamentally analogue devices (the more you move the mouse the more the pointer on the screen moves). So, it seemed inappropriate to turn analogue brain activity recorded in the EEG into binary form (traditionally, in P300-based BCI, by thresholding the output of classifiers) to later turn the signal in analogue form again (which is necessary to move the pointer on the screen). Given the limited information available in EEG, we felt that an analogic BCI approach would limit any further losses introduced by the detection process and would offer the potential to use P300 amplitude information. To obtain this, the motion of the pointer is directly determined by the squashed output of the polynomial filter. More precisely, the vertical motion of the pointer is proportional to the difference between the output produced by the filter when processing an epoch where the “up” rectangle was flashed and the output produced by the filter when processing an epoch where the “down” rectangle was flashed. Similarly, the horizontal motion of the pointer is determined by the difference between the outputs produced in response to the flashing of the “right” and “left” rectangles. Therefore, the task of the GA is not just selecting features and designing detectors to best discriminate between P300 and non-P300 responses, but also to do so in such a way that the responses to pairs of stimuli provide the fastest and most precise way of moving the pointer in the desired direction. This, in principle, allows the full exploitation of any analogue variations present in P300s. 12

Naturally, mouse button activation must instead be based on the more traditional thresholding of filter outputs. In order to turn P300s into mouse pointer motion, we divide the stream of epochs into groups of four. Each group contains, albeit in random order, epochs corresponding to the flashing of all four possible stimuli (one for each possible direction of motion for the cursor). As soon as a full group is acquired, the features and the output of the function P (V) are computed for each of the four epochs. As a squashing function we used φ = arctan, thereby limiting the output O(V) to the range [− π2 , + π2 ]. As a result of these operations we obtain a tuple of output values (Ou , Od , Ol , Or ), where subscripts refer to the up, down, left and right stimuli, respectively. These are used to compute the motion vector v = (vx , vy ) for the mouse cursor on the screen, as follows ( vx = O r − O l (5) vy = O u − O d

3.5

Evolving classifiers

The GA has the task of identifying a high-quality set of parameters for the classifier in Equation 4, these include: • N + 1 real-valued coefficients aj , • N feature channels cj , which are integers in the set {0, · · · , 18}, • N integer feature scales sj in the set {0, · · · , 29}, and • N integer feature samples tj in the set {0, · · · , 39}. The operation of feature selection is performed by the GA choosing N tuples (cj , sj , tj ), while the classifier training is performed by optimising the corresponding coefficients aj . These operations are performed jointly by the GA, which, therefore, provides all the benefits associated with a wrapper approach. All we need to do to apply a GA to this problem is to specify a representation, a set of genetic operators and a fitness function. The representation used here is simply a concatenation of the floating-point and integer parameters of the linear classifier in Equation 4. The most natural representation is a hybrid between real-coded and integer-coded, which, in principle, would require the design of specialised genetic operators. To avoid this, we decided to encode also integer parameters as real numbers, taking care, of course, of rounding cj ’s, sj ’s and sj ’s to the nearest integer before using them in Equation 4 (e.g., during fitness evaluation). With this trick, we were then able to use any operators available for real-coded GAs. In particular, we decided to 13

use blend crossover [14], also known as BLX-α, where the offspring (as1 , . . . , asi , . . . , asn ) is obtained, component by component, as follows asi = a1i + ci (a2i − a1i ) where, for each i, a different value of ci is drawn uniformly at random from the interval [−α, 1 + α], α being a suitable non-negative constant. In this work we used α = 0.1. As a selection operator we chose tournament selection with tournament size 3. To maximally speedup evolution we used a steady state GA, where, as opposed to a generational GA, each individual created is immediately inserted in the population, without waiting for a full new generation to be ready. Since in GAs the population size is kept constant, this requires defining a procedure to decide which individual gets replaced by the newcomer. In this work we used negative tournaments (where the worst individual in the tournament is selected) for this purpose. As a mutation operator we used headless chicken crossover, i.e., the recombination (via BLX) between the individual to be mutated and a randomly generated individual. Given the complexity of the task we used very large populations including 50, 000 individuals.

3.6

Fitness function

While we were able to use a standard representation and standard genetic operators, the design of the fitness function involved much more work and required numerous successive refinements. The fitness function we eventually settled for is described below. The natural objective function for a mouse is, of course, the extent to which the pointer was moved in the desired direction. So, this is clearly a necessary component in our fitness function. However, this is not enough. For example, it is possible that the pointer moved a great deal in the desired direction, while at the same time drifting significantly in a direction orthogonal to the desired one. A fitness function based on the natural objective would reward this behaviour, effectively leading to very sensitive but not very precise controllers. So, clearly the problem is multi-objective, that is we want to obtain both maximum motion in the desired direction and minimal motion in the orthogonal direction. As is routinely done in these cases, to deal with this problem we adopted a penalty method to combine the multiple objectives into a single fitness measure. With this method the fitness function f is a linear combination of the objectives ωi of the problem: X f= λi ω i (6) i

where λi are appropriate coefficients (with λ1 conventionally being set to 1).

14

Naturally, the quality of a controller cannot be evaluated on a single trial. This would not only be unrealistic (ten or more cursor movements may be required to move the mouse to the desired spot in a large screen) but it would also be very unreliable given the noise in EEG signals and complexity of the task at hand. So, the performance of the controller was evaluated on the basis of its behaviour over groups of 30 repetitions of a command (up, down, left or right), which we will term runs. Furthermore, in order to ensure that the controller performed well in all directions, these runs were acquired for all possible directions (and, in fact, multiple times for each direction to limit risks of over-fitting). All the resulting trials formed the controller’s training set. In each of the examples in the training set the controller produced a velocity vector. Let us call v i,t = (vdi,t , voi,t ) the velocity vector produced at repetition i in the t-th run. This vector is expressed in a reference system where the component vdi,t represents the motion in the target direction, while voi,t represents the motion produced in the direction orthogonal to the desired direction. In the mouse control problem we used three different objectives. The first objective, ω1 , assesses the extension of the motion in the target direction, the other two, ω2 and ω3 , evaluate the motion in orthogonal direction. In particular, ω2 assesses the average extension of the motion in such a direction at the end of runs. It, therefore, ignores any errors that have been later cancelled by errors with opposite sign (i.e., in the opposite direction). Instead, ω3 evaluates the average absolute error deriving from motion orthogonal to the target direction. So, it assesses the extent to which the trajectory towards the target is convoluted. These objectives are computed as follows: ω1 = ν ω2 = ν ω3 = ν

30 Nr X X

vdi,t

(7)

voi,t

(8)

|voi,t |

(9)

t=1 i=1 Nr X 30 X

t=1 i=1 30 N r X X t=1 i=1

1 where Nr is the number of runs and ν = 30×N is a normalisation factor. Since we want to r maximise ω1 , but minimise |ω2 | and ω3 , the penalty coefficients were set as follows: λ1 = 1, λ2 = −0.2 and λ3 = −0.2. So, the resulting fitness function is

f = ω1 − 0.2 × |ω2 | − 0.2 × ω3 . Figure 2 shows a sample fitness calculation.

15

(10)

Nr = 1 v 1,1 = (+a, −f ) v 2,1 = (−b, +g) v 3,1 = (+c, −h) v 4,1 = (+d, +i) 4 e 1 X i,1 vd = ω1 = 4 4 ω2 = ω3 =

1 4

i=1 4 X i=1

voi,1 =

j 4

4 1 X i,1 f + g + h + i vo = 4 4 i=1

fitness =

e j f +g+h+i − 0.2 × − 0.2 × 4 4 4

Figure 2: Example of fitness calculation, for a single run of four trials all aiming at achieving upward motion.

16

4

Results

4.1

Evaluation of performance

A 4-fold cross-validation (3-fold for participant A) has been applied to train the system and test its performance and generalisation ability. For each participant the total of 16 (or 12 for participant A) runs has been split in groups of four each containing one run for each direction (1–4, 5–8, etc.). Therefore there were 4 groups for each participant (3 for participant A). Then two of these groups have been used as training set while a third one as validation set. For each of the 12 combinations (for participants B–F) a population has been evolved while for each of the 3 combinations for participant A 4 populations have been evolved. So, for each participant a total of 12 different classifiers have been evolved for 40 generations. The three different objectives (Section 3.6) have been evaluated for the training-set as well as for the validation-set. The outcomes in the validation-set have in no way influenced the evolution phase nor its termination and therefore can be considered a reliable index of the generalisation ability of the classifier when applied to new data. Table 3 shows the values of ω1 , ω2 and ω3 for classifiers trained with the n-fold cross validation method explained above. For each participant the average across the 12 evolutions, together with the standard deviation in brackets, is reported. The same results (for the validation set) are also depicted in Figure 3. It is apparent from the analysis of the results that all participants were able to move the pointer in the target directions (as quantified by ω1 ) with very limited lateral error at the end of a sequence of commands (quantified by ω2 ). In fact, if we consider ω1 as the amplitude of the signal and ω2 as the standard error of the noise in our system, then ω1 /|ω2 | gives us an idea of the signal to noise ratio (SNR) during the movement of the pointer. Averaged over our 6 participants, ω1 /|ω2 | is 28.1, which gives us a good SNR = 29.0 dB.1 This is achieved thanks to the self-correcting nature of the task. That is, a mouse in an integral controller, where the final position of the pointer is the result of summing many contributions (velocity vectors) over a period of time. For this reasons, if the noise (in our case, the motion in a direction orthogonal to the target) has a zero mean, the noise contributions will tend to cancel. This is what happens in our BCI mouse, as one can easily see by comparing ω3 , which gives us an idea of the noise on each single velocity vector, with ω2 which, instead, gives us an indication of the residual noise at the end of a sequence of commands. 1

The notion of SNR seems particularly appropriate for a analogue pointer control system. This is because in these systems one can easily achieve any desired speed of motion by multiplying the pointer velocity vectors by an appropriate constant. This multiplication effectively acts as a perfect, (almost) noise-free, amplifier for the original control signals. Like in any amplifier, the SNR of the original signal will still have an effect on the output.

17

18

Participant A B C D E F

ω1 1.0943 (0.1349) 1.5610 (0.0836) 1.3335 (0.1457) 1.0893 (0.0489) 1.4921 (0.0572) 1.0335 (0.0922)

Table 3: Testing and validation accuracy scores Training set Validation set ω2 ω3 ω1 ω2 ω3 0.0262 (0.0236) 1.2868 (0.0968) 0.7126 (0.1161) 0.0415 (0.0728) 1.3516 (0.0880) -0.0035 (0.0282) 1.0283 (0.1432) 1.2713 (0.1857) -0.0563 (0.1145) 1.0794 (0.1877) 0.0208 (0.0626) 1.1798 (0.1307) 1.0831 (0.2377) 0.0508 (0.0634) 1.2161 (0.1966) 0.0158 (0.0434) 1.3240 (0.1677) 0.5253 (0.2325) 0.0112 (0.1684) 1.3676 (0.1268) 0.0215 (0.0333) 1.1918 (0.1114) 1.1101 (0.1628) 0.0267 (0.1032) 1.2477 (0.1695) -0.0068 (0.0328) 1.2790 (0.0827) 0.5101 (0.1299) 0.0269 (0.1449) 1.3617 (0.1290)

Orthogonal Motion

0.15 0.1 0.05 0 -0.05 -0.1 -0.15

Orthogonal Motion

0

Orthogonal Motion

0.6 0.8 Motion in Target Direction

1

0.6 0.8 Motion in Target Direction

1

0.6 0.8 Motion in Target Direction

1

0.6 0.8 Motion in Target Direction

1

1.2

1.4

Participant A

0.2

0.4

1.2

1.4

Participant B

0.15 0.1 0.05 0 -0.05 -0.1 -0.15 0

Orthogonal Motion

0.4

0.15 0.1 0.05 0 -0.05 -0.1 -0.15 0

0.2

0.4

1.2

1.4

Participant C

0.15 0.1 0.05 0 -0.05 -0.1 -0.15 0

Orthogonal Motion

0.2

0.2

0.4

1.2

1.4

Participant D

0.15 0.1 0.05 0 -0.05 -0.1 -0.15 0

0.2

0.4

0.6

0.8

1

1.2

1.4

Participant E

Orthogonal Motion

Motion in Target Direction 0.15 0.1 0.05 0 -0.05 -0.1 -0.15 0

0.2

0.4

0.6

0.8

Motion in Target Direction

1

1.2

1.4

Participant F

Figure 3: Average accuracy of pointer movement on the validation sets across all evolved classifiers and for all six participants. Error bars represent standard deviation of performance across classifiers. 19

These results have been obtained by processing offline data acquired with the BCI mouse in training mode. In this mode stimuli are presented to a participant but no feedback is provided as to the extend and direction of motion of the pointer. This is done to ensure that the cleanest possible dataset is gathered. As soon as the acquisition of an appropriate training set is completed the system performs all the necessary preparatory steps, including filtering and feature detection, for the application of the GA. GA runs are typically successful within 30 generations. The data preparation and the training process require between 5 to 10 minutes. The system can then be used. Figure 4 illustrates the BCI mouse behaviour at the end of a typical training run (for participant A using runs 1–4 and 9–12 as training set and 5–8 as validation set). The plot on the left represents the positions the mouse pointer took over a sequence of stimuli in the training set. On the right we show the behaviour of the mouse when examples in the validation set are used instead. In both cases the same number of examples has been drawn. As it can easily be seen, in both cases the motion of the pointer is reasonably straight and mostly in the target direction (upward), the only difference between the training data and the validation data being the slight reduction in performance (the extension of the motion towards the goal) when validation data are used. This is naturally to be expected, since a small amount of over-fitting cannot be avoided. Also, note how the steps performed by our mouse are not all of the same size. This is a result of our analogue approach. The small lateral deviations observable in both plots could be easily compensated by a participant by focusing attention on stimuli orthogonal to the desired direction of motion for a one or two flashes. This was not allowed, however, during the acquisition of the data shown in Figure 4 nor in any other phase of training and validation.

4.2

Strengths of the Analogue Approach

These results strongly suggest that our analogue approach to controlling the pointer movement is viable. One might wonder, however, exactly how the system operates. To explain this we will use a representative example extracted from our validation set. In this example the system had evolved the following filter P (V) = −2.1328 + 0.1909 · V(6, 23, 17) + 0.1741 · V(8, 20, 22)

(11)

+ 0.4210 · V(9, 18, 8) − 0.5247 · V(13, 18, 9). The output of the filter was then squashed through the arc-tangent function, as explained in Section 3.4, before being used to control the mouse. So, the system found that channels 20

Figure 4: Sample behaviour of the BCI mouse with a fragment of data from the training set (left trace) and the validation set (right trace). Every dot in the plots corresponds to a motion vector issued by the system (about once per second).

21

6 (F8), 8 (C3), 9 (Cz) and 13 (P3) provided the best information for the task, with Cz and P3 having a predominant role. Figure 5 shows the EEG signal recorded on these channels in one particular group of stimuli. In this example, the target was the rectangle on the right of the screen. The signal corresponding to the epoch where such a rectangle flashed is plotted in a thick line. The remaining three signals represent neighbouring epochs where non-target rectangles flashed. Figure 6(top) shows the output produced by the filter in Equation 11 when combining the CWTs of these four channels when the up (“up” plot) and down (“down” plot) rectangles flashed. The figure also shows the result of squashing these outputs (“atan(up)” and “atan(down)” plots, respectively). Finally, the figure shows a plot representing the difference between these last two curves. Note that we shifted the origin for these plots by approximately 300ms (30+8 samples) since this is the time at which the GA decided the “atan(up)-atan(down)” signal should sampled when deciding the direction and amplitude of the motion of the mouse pointer. So, the vertical component of the velocity vector of the pointer is given by the tiny arrow at the origin in Figure 6(top). Since this is not the desired direction of motion, this is exactly what we want to see. Figure 6(bottom) reports the same results but for the directions left and right. In this case, the “atan(right)-atan(left)” curve reaches its peak near the origin. Since this is the time at which the horizontal component of the pointer’s velocity vector is computed, we obtain a large movement towards the target direction (the right). In order to better understand how this result is obtained, we can use an approximation: P (V) ≈ −2.1328 + 0.4210 · V(9, 18, 8) − 0.5247 · V(13, 18, 8)

(12)

which is obtained by neglecting the terms involving channels 6 and 8 in Equation 11 and ignoring the one-sample shift between channel 9 and channel 13 in the original formula (i.e., we replace V(13, 18, 9) with V(13, 18, 8)). Here we can see that both remaining terms are the result of convolving the corresponding signal with wavelet 18 (which corresponds to the mother wavelet at scale a = 21.2 as one can calculate from Equation 1). Because CWT is linear, we could calculate the r.h.s. of Equation 12 by first computing the a linear combination of the raw signals from channels 9 and 13, and then convolving the result with wavelet 18. To give an idea of what this linear combination might look like for the example discussed above we provide it in Figure 7 for the epochs corresponding to the flashing of our four stimuli. Interestingly, all signals plotted appear to be significantly less noisy than the original channels. In addition, the P300 component present when the target stimulus was flashed (the right rectangle) appears to be better delineated. As shown in Figure 8, after convolution with wavelet 18, only the signal corresponding to the flashing of the right rectangle presents an ample positive wave in the P300 area. All other signals present a negative wave in the same region. In addition, the “up” and 22

Channel 6 (F8) 20

up right down left

15 10 5 0 -5 -10 -15 -20

0

100

200

300

400

500

600

700

800

900

1000

Channel 8 (C3) 20

up right down left

15 10 5 0 -5 -10 -15 -20

0

100

200

300

400

500

600

700

800

900

1000

Channel 9 (Cz) 20

up right down left

15 10 5 0 -5 -10 -15 -20

0

100

200

300

400

500

600

700

800

900

1000

Channel 13 (P3) 20

up right down left

15 10 5 0 -5 -10 -15 -20

0

100

200

300

400

500

600

700

800

900

1000

Figure 5: Raw data recorder from channels 6 (F8), 8 (C3), 9 (Cz) and 13 (P3) after the flashing of the up, right, left and down rectangles. The right rectangle was the target stimulus.

23

Signal sampled here

20

up atan(up) down atan(down) atan(up)-atan(down)

15 10

vy=-0.25

5 0 -5 -10 -15 -20

-200

-100

0

100

200

300

400

500

Signal sampled here

20

right atan(right) left atan(left) atan(right)-atan(left)

15 vx=13

10 5 0 -5 -10 -15 -20

-200

-100

0

100

200

300

400

500

Figure 6: Filter output (before and after squashing) produced after the flashing of different stimuli, and corresponding pointer motion in a direction orthogonal to the target (top) and in the direction of the target (bottom). For easier visualisation we plot 5 × atan(x) instead of atan(x).

24

20

up down

15 10 5 0 -5 -10 -15 -20

0

200

400

600

800

1000

20

right left

P300

15 10 5 0 -5 -10 -15 -20

0

200

400

600

800

1000

Figure 7: Linear combination of the input channels shown in Figure 5 based on Equation 12. “down” signals remain almost identical well until after the P300 region. After squashing and pairwise subtraction, the signals in this approximate model are almost identical to those provided by the full equation (Equation 11) shown in Figure 6, particularly in the region around the origin (where the signal is sampled to move the mouse pointer). This analysis indicates that effectively the crucial component in the filter evolved by the GA is the weighted difference 0.4210 × ch9 − 0.5247 × ch13, which appear to provide better information on the user’s intentions that any of the channels separately.2

5

Discussion

The purpose of this study was to explore the advantages and limits of analogue control and of the use of evolutionary optimisation in a P300-based brain-computer interface. This required to use standardised conditions across multiple users. In these tests, the performance of our BCI mouse were very encouraging. Control in validation was accurate and all participants were able to use the system within minutes of wearing the electrode cap. In this section we discuss possible applications for the system, we compare it with previous work and we indicate problems that still require addressing. 2

We believe that the other channels are used to minimise exogenous errors. Particularly, channel 6 appears to be used to inhibit pointer motion when participants performed ocular movements or blinked.

25

Signal sampled here

20

up atan(up) down atan(down) atan(up)-atan(down)

15 10

vy=0

5 0 -5 -10 -15 -20

-200

-100

0

100

200

300

400

500

Signal sampled here

20

right atan(right) left atan(left) atan(right)-atan(left)

15 vx=13

10 5 0 -5 -10 -15 -20

-200

-100

0

100

200

300

400

500

Figure 8: Result of convolving the signals in Figure 7 with wavelet 18 (“up”, “down”, “left” and “right” plots). The signals are then squashed (“atan(up)”, “atan(down)”, “atan(left)” and “atan(right)” plots) and subtracted pairwise to obtain an approximation of the actual motion of the pointer.

26

5.1

Test applications

Having established that the approach works and appears to be accurate, we decided start exploring more realistic uses of the mouse with participant D – the oldest and worst performing of our subjects. So, the systems we describe below and the results obtained with this participant can only be considered as first, conservative indications of what can be achieved. We will need to perform much deeper investigations of these systems in future research. The first application we considered was the use of the mouse to control a standard computer system (in particular a personal computer running the Windows XP operating system). In this application, the screen included our four flashing rectangles and a fixation cross as in the tests of the BCI mouse described above. However, naturally, the background was not uniformly gray: instead it included the standard windows and icons normally available in a Windows machine (see Figure 9). In order to make the system as user friendly as possible for people with limited or no saccadic control, instead of moving the mouse pointer on a fixed screen, we decided to scroll the screen, thereby ensuring that the entities of interest for the user were always near the fixation cross. In addition, we used a zoom factor of 2 to ensure maximum readability. We did not retrain the system with this richer set of stimuli. Nonetheless, the user was immediately able to move the pointer to the desired locations on the screen, albeit slowly at first, despite the large number of moving elements on the screen being a clear distracting factor. After a few minutes the participant had clearly improved his ability to control the system, as shown in the “film strip” in Figure 9, where the participant was able to scroll from the top left corner to the lower left corner of the main window in around 13 seconds, which we find encouraging. We expect that much better performance could be obtained by retraining the control system for this specific application and by giving participants some time to adjust to the system. Also, as we mentioned before, participant D was the worst performing in our study. So, we should expect to see better average performance when looking at a larger set of participants. Naturally, the availability of 2–D pointer motion makes it possible input data of any other type. Numerous systems exist, for example, that can turn mouse movements into text. So, as our second test application we designed a simple prototype speller system. We wanted to test the viability of spelling by BCI mouse and determine whether the accuracy of the control is sufficient for the task. In our BCI speller system, in the centre of the screen a circle with 8 sectors is drawn. Each sector represents one of the 8 most used characters in English (Fig. 10). The idea is for the system to use a T9 cell-phone-type predictor which makes it possible to input complete words with very few key presses. However, we made not use of this feature during our tests. The system starts with the pointer at the centre of the screen. In order to enter a 27

Figure 9: Snapshots (ordered from left to right and from top to bottom) taken every 20 frames (800ms) from a video recording of our BCI mouse in action. The user wanted to scroll down.

28

Figure 10: Our BCI-mouse based speller after the user was been able to correctly enter the text “SHE HAS”. character, the user moves the cursor in the desired sector of the screen. To give time to the user to move the pointer into the desired sector, character input is prevented for a certain amount of time (15 s in our tests). The character is acquired as soon as the cursor reaches the perimeter of the circle or a maximum time has elapsed. The entered character is appended to the current string shown at the upper right in the screen and the pointer is repositioned at the centre. The preliminary testing of this system with participant D has been promising, despite, again, our not retraining the system for this application, and the tests being hampered by technical problems. The objective of the tests was to input the sentence “she has a hat”. The user repeated the task three times. The first time the user made 2 errors out of 7 characters inputting characters at a rate of one every 35 seconds, then the test was stopped by a system error. In the second attempt the user was able to spell “she has” without any errors but at a much slower rate (one character every 80 seconds). Unfortunately, again an error stopped the system, before the sentence could be completed. After getting rid of these problems, the participant was able to do the test again spelling again without any errors “she has” plus an additional space, before feeling drained and unable to continue. In this last test the user inputted characters at a rate of one every 54 seconds. So, despite us running these tests with a user who was tired after the many other tests he had run during his session, after rapidly adjusting to the new application, the user was able to control the device and input data without errors, although this was achieved with a considerable concentration effort and at a very slow rate of 3.2 bits/minute. Again, we should expect much better performance with well-rested and better performing par29

ticipants. Also, training the control system specifically for this application and giving participants some time to adjust to the system would certainly provide significant performance improvements. We plan to further develop this system in future research.

5.2

Comparison with other systems

Our BCI mouse is similar to the one proposed in [2]. This is also based on P300s and on four stimuli (arrows) placed at the margins of the screen. There is, however, a very substantial difference. In [2] the algorithm performs a binary classification of the stimulus response for each of the four directions as target or non-target and then moves in the direction of the target one. Instead, in our system the stimulus response for each of the four directions directly affects the movement of the pointer. So, as we mentioned in Section 1 our BCI mouse is an analogue device. Other differences include the fact that the system in [2] used rather long inter-stimulus intervals (2.5 seconds), which led to the pointer moving at the rate of one movement every 10 seconds, while this happens approximately once per second in our system. Also, in [2] motion was quantised (being based on digital commands) unlike in our system where step size varies with the shape and size of the elicited P300s. Another approach for 2–D cursor movement based on P300 detection had previously been proposed in [35]. There, too, four stimuli positioned to the north, east, west and south of a fixation cross were used. However, in this case the stimuli (four crosses) were only about 1 cm away from the middle of the screen. At intervals of 1 second one random stimulus was replaced by an asterisk for 250 ms. When all stimuli had flashed, the amplitude of each signal was averaged over the intervals 300 ms–600 ms typical of P300s. A decision as to which stimulus the subject was attending would be then made on the basis of which of the four stimuli had produced the highest average. So, also this system, like [2], is nonanalogue. The best speed was one cursor movement every 4 seconds, which is four times slower than our system. The best accuracy achieved in the system in detecting P300s was about 50%, which is very low as suggested by the authors themselves: “With 50% accuracy in the P300 detection (and assuming that errors in P300 detection are uniformly distributed over the three unattended targets), the subject would be expected to need 30 steps to reach the goal of ten steps in the attended target direction. These 30 steps, at four seconds per step, would require about 2 minutes of actual time on task.” [35, pp. 180] In order to increase accuracy, the system could be set up so as to perform multiple repetitions of each trial before making a decision. However, since this significantly reduces the number of commands per unit of time the system can issue, this feature was shown not to produce any benefits w.r.t. a single trial approach.

30

After encouraging initial successes [43, 23], much better performance can now be obtained in systems based on frequency analysis, and in particular the detection of µ or β rhythms. For example, [44] reports recent results where four users where able to move a 2–D pointer to one of 8 target locations at the margins of the screen and occupying 4.9% of the screen area with accuracies ranging from 70% to 92% and with average movement times of between 1.9 and 3.9 s. Although we have not explicitly performed tests of this type with our BCI mouse, we are sure that much more work will be needed to achieve performance as good as this. The disadvantage of systems based on µ or β rhythms is that their use requires an extensive training period. For example, in [44], users underwent training periods of between 22 and 68 sessions at a rate of 2 to 4 sessions per week, which does not compare favourably with our system where totally inexperienced users were able to control the BCI mouse within minutes. Another disadvantage of systems based on µ or β rhythms is that around 20% of users are unable to develop enough control of their rhythms to use the system. In the present study, instead, although there were significant individual difference in performance, all users showed a good level of control. Whether over a period of time this would improve with practice or, in fact, worsen due to habituation is something we will need to determine in future studies. Finally, it is important to mention that there is intense research on invasive brain computer interfaces for 2–D pointer control (e.g., see [7, 13]). In these, electrodes implanted in the brain record action potentials of one or more cortical neurons, which in principle can provide excellent control signals. As discussed in [44] the performance of these systems is good, being similar to that achieved by non-invasive systems based on µ or β rhythms, the former achieving movement times of between 1.5 and 2.2 s when hitting targets of size between 1.3 and 7.7 % of the screen area. However, of course, the invasiveness of the approach makes it impractical for humans (and, in fact, studies have been limited to monkeys).

5.3

Open problems

All the tests conducted so far with our mouse have been very encouraging. There are, however, still problems that will need ironing before an experimental system such as this can be used outside the lab. We briefly discuss them below. The system is accurate in moving the mouse pointer in the standard four directions. However, when using computer mice, it is very common to need to move the pointer in diagonal directions. This was achieved in our tests with participant D, by switching attention from one target to another periodically, so as to obtain a sort of zig-zagging movement. However, it would be desirable to have some more effective way to move diagonally and reach any target location on a screen much more quickly and naturally. An obvious way to achieve diagonal motion would be to add four additional flashing rectangles 31

at the corners of the screen. However, there might be another, more natural solution. Our mouse is analogue and, so, in principle, it would not be impossible to achieve oblique movement if a user was able to concentrate his or her attention on two squares (up and right, right and down, down and left or left and up) at the same time, thereby producing P300 components both for horizontal motion and vertical motion. We have done a limited number of informal tests with this approach, which seem to indicate that there might be some mileage in it, but much more research will be needed to confirm this and/or find alternative methods for diagonal motion. After training and validation the BCI mouse can be used in trace mode. In this mode of operation, signal processing, feature extraction and classification are performed in realtime and the position of the pointer is constantly updated on the screen. In these conditions the motion of the pointer, which always starts from the middle of the screen, can distract the user, who will often focus on the stimuli for a few epochs and then perform saccadics to check whether or not this has resulted in the desired motion. These saccadic movements immediately generate myoelectric artifacts in the signals being acquired which result in some, otherwise valid commands, to be misclassified. Also, once distracted users may find it difficult to immediately refocus their attention on the stimuli, thereby reducing their control on the behaviour of the pointer. In informal testing we have noticed that these effects resulted in a slight worsening of the ability to control the pointer. The situation may be even worse when the screen contains numerous elements of interest for the user, like for example, in the BCI speller or when the system is used to with a user interface including icons, windows, etc. as in Figure 9. These effects reduce performance and so it is important to explore methods to mitigate them. For example we have tested updating the position of the pointer only every n epochs, which appears to improve the system. A full evaluation of this and other methods to improve feedback quality will need to be explored in future research. The concentration level required to operate the system is considerable. Our tests were all carried out over a period of about 2 hours including preparation time, but participants had plenty of breaks between experiments. The more extensive tests with participant D suggest that there is a considerable cognitive load involved in using our BCI mouse. We believe this is no different from the effort required to operate other BCI systems, but in future research we will need to evaluate the usability of the BCI mouse with purposely designed experiments.

6

Conclusions

In this paper we have proposed a BCI mouse based on the use of P300 waves. The system is analogue, in that at no point a binary decision is made as to whether or not a P300 was

32

actually produced in response to a particular stimulus. Instead, the motion of the pointer on the screen is controlled by directly combining the amplitudes of the output produced by a filter in the presence of different stimuli. Beyond providing carefully designed stimuli, a rich set of features (wavelet coefficients) and a very flexible combination mechanism (a polynomial filter) through which we thought a solution to the problem of controlling a pointer via EEG could be found, we actually did not do any other design. The biggest part of the design in this system (i.e. the feature selection, the selection of the type, order and parameters of the controller) was entirely left to a genetic algorithm. If fact, if we had not analysed the coefficients of the evolved filters, the features chosen by the algorithm and the behaviour of the system, we might have never known that this was a P300 based device! The performance of our system has been very encouraging. All participants have been able to use the system very quickly. The GA was very effective and efficient at finding good designs for the system. Indeed, it succeed in every run, suggesting that we had chosen the infrastructure for the system and the feature set reasonably well. In validation, the trajectories of the pointer have achieved high accuracy with a SNR = 29 dB. The system issues control commands at a much faster rate (approximately once per second) than other P300-based computer mice previously described in the literature. These encouraging results indicate that there may be a lot more information about user intentions in EEG signals, and that perhaps, traditional design techniques may be a limiting factor. We would have found, for example, extremely hard to design an analogue BCI mouse system, without an enormous amount of trial end error. Evolution, on the other hand, not being biased by prior knowledge and being entirely guided by objective measures of success (the fitness function), was able to achieve this almost effortlessly. The systems evolved were also rather robust. For example, it was possible to control the mouse even in situations very different from the ones originally considered in training, such as in tests with control in a real Windows environment and in our BCI speller, without retraining the system. So, we suspect that other single-trial BCI applications might benefit from the use of properly guided evolutionary algorithms. We plan to explore this in future research.

References [1] Documentation 2nd Wadsworth BCI Dataset. ida.first.fraunhofer.de/ projects/bci/ competition/ albany desc/ albany desc ii.pdf. [2] Fabrizio Beverina, Giorgio Palmas, Stefano Silvoni, Francesco Piccione, and Silvio Giove. User adaptive BCIs: SSVEP and P300 based interfaces. PsychNology Journal, 1(4):331–354, 2003.

33

[3] N. Birbaumer, N. Ghanayim, T. Hinterberger, I. Iversen, B. Kotchoubey, A. Kbler, J. Perelmouter, E. Taub, and H. Flor. A spelling device for the paralysed. Nature, 398(6725):297–298, Mar 1999. [4] Niels Birbaumer, Thilo Hinterberger, Andrea Kbler, and Nicola Neumann. The thought-translation device (TTD): neurobehavioral mechanisms and clinical outcome. IEEE Transactions on Neural System and Rehabilitation Engineering, 11(2):120–123, Jun 2003. [5] S. Cagnoni, A. B. Dobrzeniecki, R. Poli, and J. C. Yanch. Genetic algorithm-based interactive segmentation of 3D medical images. Image and Vision Computing, 17:881– 895, 1999. [6] S. Cagnoni and R. Poli, editors. Special issue of the EURASIP Journal of Applied Signal Processing on Genetic and Evolutionary Computation for Signal Processing and Image Analysis, number 8, July 2003. [7] J.M. Carmena, M.A. Lebedev, R.E. Crist, J.E. O’Doherty, D.M. Santucci, D.F. Dimitrov, P.G. Patil, C.S. Henriquez, and M.A.L. Nicolelis. Learning to control a brainmachine interface for reaching and grasping by primates. PLoS Biol, 1(2):193–208, 2003. [8] Caterina Cinel, Riccardo Poli, and Luca Citi. Possible sources of perceptual errors in P300-based speller paradigm. Biomedizinische Technik, 49:39–40, 2004. Proceedings of 2nd International BCI workshop and Training Course. [9] Luca Citi, Riccardo Poli, and Francisco Sepulveda. An evolutionary approach to feature selection and classification in P300-based BCI. Biomedizinische Technik, 49:41– 42, 2004. Proceedings of 2nd International BCI workshop and Training Course. [10] S Coyle, T Ward, C Markham, and G McDarby. On the suitability of near-infrared (NIR) systems for next-generation Brain-Computer Interfaces. Physiological Measurement, 25(4):815–22, 2004. [11] E. Donchin and M. G. H. Coles. Is the P300 a manifestation of context updating? Behavioral and Brain Sciences, 11:355–372, 1988. [12] Emanuel Donchin, Kevin M. Spencer, and Ranjith Wijesinghe. The mental prosthesis: assessing the speed of a P300-based Brain-Computer Interface. IEEE Transactions on Rehabilitation Engineering, 8(2):174–179, Jun 2000. [13] J.P. Donoghue. Connecting cortex to machines: recent advances in brain interfaces. Nature Neuroscience, 5:1085–1088, 2002. 34

[14] Larry J. Eshelman and J. David Schaffer. Real-Coded Genetic Algorithms and Interval Schemata. In L. Darrell Whitley, editor, Foundations of Genetic Algorithms 2, San Mateo, CA, 1993. Morgan Kaufmann Publishers. [15] L. A. Farwell and E. Donchin. Talking off the top of your head: A mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology, 70:510–523, 1988. [16] T.C. Ferree, P. Luu, G.S. Russell, and D.M. Tucker. Scalp electrode impedance, infection risk, and eeg data quality. Clinical Neurophysiology, 112(3):536–544, 2001. [17] Apostolos Georgopoulos, Frederick Langheim, Arthur Leuthold, and Alexander Merkle. Magnetoencephalographic signals predict movement trajectory in space. Experimental Brain Research, pages 1–4, Jul 2005. [18] Neal R. Harvey, James Theiler, Steven P. Brumby, Simon Perkins, John J. Szymanski, Jeffrey J. Bloch, Reid B. Porter, Mark Galassi, and A. Cody Young. Comparison of GENIE and conventional supervised classifiers for multispectral image feature extraction. IEEE Transactions on Geoscience and Remote Sensing, 40(2):393–404, February 2002. [19] J. H. Holland. Adaptation in Natural and Artificial Systems. The University of Michigan Press, Ann Arbor, MI, 1975. [20] N. G. Kanwisher. Repetition blindness: type recognition without token individuation. Cognition, 27(2):117–143, Nov 1987. [21] S.P. Kelly, E.C. Lalor, R.B. Reilly, and J.J. Foxe. Visual spatial attention tracking using high-density ssvep data for independent brain-computer communication. IEEE Transactions on Neural System and Rehabilitation Engineering, 13(2):172–178, 2005. [22] Ron Kohavi and George H. John. Wrappers for feature subset selection. Artificial Intelligence, 97(1-2):273–324, 1997. [23] Aleksander Kostov and Mark Polak. Parallel man-machine training in development of eeg-based cursor control. IEEE Transactions on Rehabilitation Engineering, 8(2):203– 205, June 2000. [24] J. R. Koza. Genetic Programming: On the Programming of Computers by Natural Selection. MIT Press, Cambridge, MA, USA, 1992. [25] John R. Koza and Riccardo Poli. Genetic Programming. In Edmund K. Burke and Graham Kendall, editors, Search Methodologies: Introductory Tutorials in Optimization and Decision Support Techniques, chapter 5. Springer, 2005. 35

[26] A. K¨ ubler, F. Nijboer, J. Mellinger, T. M. Vaughan, H. Pawelzik, G. Schalk, D. J. McFarland, N. Birbaumer, and J. R. Wolpaw. Patients with ALS can use sensorimotor rhythms to operate a Brain-Computer Interface. Neurology, 64(10):1775–1777, May 2005. [27] W. B. Langdon and Riccardo Poli. Foundations of Genetic Programming. SpringerVerlag, 2002. [28] M Middendorf, G McMillan, G Calhoun, and K S Jones. Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Transactions on Rehabilitation Engineering, 8(2):211–4, 2000. [29] G.A. Miller, W. Lutzenberger, and T Elbert. The linked-reference issue in eeg and erp recording. Journal of Psychophysiology, 5:273–276, 1991. [30] G. R. M¨ uller-Putz, R. Scherer, G. Pfurtscheller, and R. Rupp. EEG-based neuroprosthesis control: a step towards clinical practice. Neuroscience Letters, 382(1-2):169–174, 2005. [31] Gert Pfurtscheller, Doris Flotzinger, and Joachim Kalcher. Brain-computer interface: a new communication device for handicapped persons. Journal of Microcomputer Applications, 16(3):293–299, 1993. [32] T. W. Picton, S. Bentin, P. Berg, E. Donchin, S. A. Hillyard, R. Johnson, G. A. Miller, W. Ritter, D. S. Ruchkin, M. D. Rugg, and M. J. Taylor. Guidelines for using human event-related potentials to study cognition: recording standards and publication criteria. Psychophysiology, 37(2):127–152, Mar 2000. [33] R. Poli. Genetic programming for image analysis. In John R. Koza, David E. Goldberg, David B. Fogel, and Rick L. Riolo, editors, Genetic Programming 1996: Proceedings of the First Annual Conference, pages 363–368, Stanford University, CA, USA, 28–31 July 1996. MIT Press. [34] Riccardo Poli, Stefano Cagnoni, and Guido Valli. Genetic design of optimum linear and non-linear QRS detectors. IEEE Transactions on Biomedical Engineering, 42(11):1137–1141, November 1995. [35] J. B. Polikoff, H. T. Bunnell, and W. J. Borkowski Jr. Toward a p300-based computer interface. In Proc. Rehab. Eng. and Assistive Technology Society of North America (RESNA’95), pages 178–180, Arlington, Va, 1995. RESNAPRESS. [36] Marcos I. Quintana, Riccardo Poli, and Ela Claridge. Morphological algorithm design for binary images using genetic programming. Genetic Programming and Evolvable Machines, 2006. forthcoming. 36

[37] Ingo Rechenberg. Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Frommann–Holzboog, Stuttgart, 1973. [38] Andrew B Schwartz. Cortical neural prosthetics. Annual Review of Neuroscience, 27:487–507, 2004. [39] H.-P. Schwefel. Numerical Optimization of Computer Models. Wiley, Chichester, 1981. [40] Eric W Sellers and Emanuel Donchin. A P300-based brain-computer interface: Initial tests by ALS patients. Clinical Neurophysiology, 117(3):538–548, Mar 2006. [41] Erich E. Sutter. The brain response interface: communication through visuallyinduced electrical brain responses. Journal of Microcomputer Applications, 15(1):31– 45, 1992. [42] Nikolaus Weiskopf, Klaus Mathiak, Simon W Bock, Frank Scharnowski, Ralf Veit, Wolfgang Grodd, Rainer Goebel, and Niels Birbaumer. Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI). IEEE Transactions on Biomedical Engineering, 51(6):966–970, Jun 2004. [43] J. R. Wolpaw and D. J. McFarland. Multichannel eeg-based brain-computer communication. Electroencephalography and Clinical Neurophysiology, 90:444–449, 1994. [44] J. R. Wolpaw and D. J. McFarland. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proceedings of the National Academy of Sciences, 101(51):17849–17854, 2004. [45] J. R. Wolpaw, D. J. McFarland, G. W. Neat, and C. A. Forneris. An EEG-based brain-computer interface for cursor control. Electroencephalography and Clinical Neurophysiology, 78(3):252–259, Mar 1991. [46] Jonathan R Wolpaw, Niels Birbaumer, Dennis J McFarland, Gert Pfurtscheller, and Theresa M Vaughan. Brain-computer interfaces for communication and control. Clinical Neurophysiology, 113(6):767–791, Jun 2002. [47] Jonathan R Wolpaw and Dennis J McFarland. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proceedings of the National Academy of Sciences of the U.S.A., 101(51):17849–17854, Dec 2004.

37