A Real-time EEG-based BCI System for Attention Recognition in Ubiquitous Environment Yongchang Li School of Information Science and Engineering,Lanzhou University Lanzhou,China
[email protected]
Xiaowei Li School of Information Science and Engineering,Lanzhou University Lanzhou,China
[email protected]
Martyn Ratcliffe School of Computing, Telecommunications and Networks, Birmingham City University Birmingham, UK
[email protected]
Li Liu School of Information Science and Engineering,Lanzhou University Lanzhou,China
[email protected]
Yanbing Qi School of Information Science and Engineering,Lanzhou University Lanzhou,China
[email protected]
Quanying Liu School of Information Science and Engineering,Lanzhou University Lanzhou,China
[email protected]
ABSTRACT
Author Keywords
Several types of biological signal, such as Electroencephalogram (EEG), electrooculogram(EOG), electrocardiogram(ECG), electromyogram (EMG), skin temperature variation and electrodermal activity, may be used to measure a human subject’s attention level. Generally electroencephalogram (EEG) is considered the most effective and objective indicator of attention level. However, few systems based on EEG have actually been developed to measure attention levels. In this paper we describe a pervasive system, based on an electroencephalogram (EEG) Brain-Computer Interface, which measures attention level. After demonstrating the effectiveness of our system we then go on to compare our approach with traditional approaches. In our study, three attention levels were classified by a KNN classifier based on the Self-Assessment Manikin (SAM) model. In our experiment, subjects were given several mental tasks to undertake and asked to report on their attention level during the tasks using a set of attention classifications. The average accuracy rate is shown to reach 57.03% after seven sessions’ EEG training. Moreover, our system works in real-time while maintaining this accuracy. This is demonstrated by our time performance evaluation results which show that the time latency is short enough for our system to recognize attention in real-time.
EEG, BCI, Attention, Distance learning. ACM Classification Keywords
H5.m. Information Miscellaneous.
interfaces
and
presentation:
General Terms
Design, Experimentation, Human Factors, Measurement, Performance. INTRODUCTION
Lack of student attention in class is not unusual for many reasons: poor delivery styles, student motivation etc. However, with respect to distance learning these issues become more acute as it is difficult for the teacher to detect the attention state of the student remotely. Hence remote measurement of attention would be a very useful tool, providing feedback to the teacher in real-time. Measuring the learner’s attention to model motivation during computer-based instruction is a topic in the area of Artificial Intelligence in Education (AIED). This area has considered diverse mechanisms to recognize and react to human motivation with the aim of improving or sustaining optimal levels of motivation. The approach to attention recognition presented in this paper consists of modeling a learner’s attention by considering measurements taken from a brain computer interface (BCI) in combination with usergenerated data taken during subject testing. Here the BCI provides a reading of electric signals generated by neural activity in the brain.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. UAAII’11, September 18, 2011, Beijing, China. Copyright 2011 ACM 978-1-4503-0932-5/11/09...$10.00.
Our work is focused on the design of a real-time EEGbased BCI System for attention recognition in a ubiquitous
33
Figure 1. Architecture of the ubiquitous EEG-based BCI system. environment. Our BCI system meets three requirements: 1) the attention recognition accuracy is high enough; 2) the time performance is good and the delay of each component is short enough for the system to recognize attention in realtime; 3) the system can be used in ubiquitous environments and the operation is simple.
Bayes classifier in [7] the authors describe a system to continuously monitor the user's voice and facial motions for recognizing emotional expressions. However it was found that these systems may not work if the subjects do not change their facial or vocal expressions for a long time, moreover, a subject may fake expressions.
The structure of this paper is as follows: In section 1, we introduced the background and related works of our BCI system for attention recognition. In section 2, we introduce our system methodology and the main system components. In section 3, we introduce our experiment details and the experimental results including attention recognition accuracy statistics and system time performance evaluation. In section 4, a discussion summarizes our work and future work is proposed.
However, more indirect methods such as, electroencephalogram (EEG), electrooculogram(EOG), electrocardiogram(ECG), electromyogram (EMG), skin temperature variation and electrodermal activity are considered more effective and objective ways to recognize emotion with a recognition accuracy of between 78.4% and 61.8%[8]. Physiological indexes are useful to evaluate emotions since they can be measured objectively. For recognizing emotions especially attention, many researchers also focus on EEG and functional Magnetic Resonance Imaging (fMRI). In [1], the authors applied principal component analysis (PCA) to full EEG log spectrum and used sub-space features to estimate a local error rate in a sustained attention task. In an experiment to prevent car accidents, EEG power spectrum, correlation analysis, principal component analysis and linear regression models were combined to indirectly estimate a driver’s drowsiness and attention level with the system alarming if a given level of attention is not maintained [2].
RELATED WORK
Previous related researches[1][2][3] has showed evidence that human biologic signals contain considerable information about emotion and attention, indicating the possibility of recognizing a learners’ attention level by studying these biologic signals. As early as 1979, Bassili found that in a video recording, moving displays of a actor’s happiness, sadness, fear, surprise, anger, and disgust were recognized more accurately than moving displays of white spots [4]. This lead researchers to believe that computers could recognize emotions by analyzing behavior scenes and researchers begin to study human emotion based on facial or voice expression [5][6][7]. For example, [5] built a system for recognizing emotions through facial expressions displayed in live video streams and video sequences. This system classifies emotions as: neutral, happy, surprised, angry, disgusted, afraid and sad with a Tree-Augmented-Naïve
The EEG is a record of the oscillations of brain electric potential recorded from electrodes on the human scalp [3]. As early as 1985, WJ RAY and HW Cole found an correlation between overall parietal alpha band and mental tasks in two experiments which were designed to examine the relation of mental effort and EEG during cognitive and tasks of an emotional nature [9]. Also, after observing induced alpha power changes derived from EEG activity,
34
W. Klimesch suggested that only slower alpha frequencies reflect attention characteristics such as alertness and expectancy [10]. Ramesh Srinivasan has shown that EEG spectral features can be used to predict the degree of attention [11]. But most of these researches focus on offline experimental based EEG, in which the attention recognition accuracy may be very different with that of online experimentation.
FIR filtering is: given a FIR filter which has N taps, the delay is: (N - 1) / (2 * Fs), where the Fs is the sampling frequency. For example a 512 tap linear-phase FIR filter operating at a 256 Hz has a delay 1 ms. EEG feature extraction is used to extract appropriate and relative features with respect to a user’s attention. EEG feature selection is a very important step in a BCI system. Early researchers found that the sensitivity of EEG changes with respect to mental effort. For example, Hans Berger [18] and others [19] report a decrease in the amplitude of the alpha (8-13 Hz) rhythm during mental arithmetic tasks Other researchers have shown that higher memory loads cause increases in theta (4-8 Hz) and low-beta (13-15 Hz) power in the frontal midline regions of the scalp [20], gamma (>30 Hz) oscillations [21], as well as inter-electrode correlation, coherence, cross phase, and cross power [22]. David Grimes showed a similar experiment measuring workload with EEG [23]. Prinzel et al. [24] built a system based on a task load metric and their result shows that alpha decreases and theta increases with higher task load. In our BCI system, 6 EEG features are extracted by our BCI. These 6 EEG features are the max power of Alpha band, the mean square of Beta band, the peak-to-peak value of Theta band, the variance of Theta band, the max power of Theta band, and the sum power of Theta band.
A UBIQUITOUS EEG-BASED BCE SYSTEM
Figure 1 shows the in the architecture of our ubiquitous EEG-based BCI system. In this system, we design two main modules: the training offline module and the attention recognition online module. The training offline module aims to create a user-characteristic EEG model for each subject. The attention recognition online module classifies the users’ EEG in real-time with the EEG model and feedback (i.e. the current attention level) to the users. The Users’ EEG Model is created in a training phase, while the online attention recognition is carried out in the attention recognition phase. EEG Recording.
The EEG recording component aims to collect the EEG signals through an EEG collection device, Nexus-4. Besides recording EEG, it validates the EEG signals and communicates with the Nexus-4. The EEG signals, sampled at 256 Hz, are transmitted to A/D converting and then sent to a computer by bluetooth. It should be noted that, with Bluetooth communication, our BCI system can be used in a pervasive environment. EEG signals received by our system from the cluster communication port are stored in a memory buffer and then segmented into 4-second epochs with 50% overlap [12] in order to strengthen the real-time response and smooth the result.
In a series of tests, the EEG signals are processed within 4second epochs. First, EEG signals are stored in a memory buffer and then segmented into 4-second epochs with 50% overlap. Next, we remove the first 4-second and last 4second recordings because these two segments are often contaminated with many artifacts. Subsequently, a preprocessing linear-phase FIR filter is used on the data epochs to remove most of the artifacts such as EOG, ECG and eye blinks. Finally, six correlative features are extracted.
With the aim of simplifying the users’ operations, fewer electrodes (Fpz, A1 and A2 illustrated in Fig.1 are selected in our system) are involved in our BCI. The A2 (located on the right earlobe) is the ground site, while A1 (located on the left earlobe) and Fpz (located in the center of forehead) make up the EEG signal channel. The Forehead, one of the most active areas for theta, alpha and beta band, is often the electrode site to study attention or drowsiness-related brain activities [13]. However, Fpz is the most common electrode site for emotion recognition and cognitive processing [14] [15] [16], so we recorded EEG signals from Fpz. EEG Processing
Two steps are conducted in EEG processing, EEG denoising and EEG feature extraction. EEG de-noising aims to remove the artifacts mixed in the EEG raw datas, such as EOG, ECG, and movement, failing electrode, sweat, breathing and pulse. In order to improve the efficiency and reduce the time delay, we use FIR filtering, as one of the most important tool in digital signal processing [17], to remove most of the artifacts. The formula of linear-phase
Figure 2. Self-Assessment Manikin. The Self-Assessment Manikin (SAM)
The SAM model [25] (Figure 2) is a standardized system to assess emotions on the valence and arousal dimensions (and
35
dominance, if wanted), which has been shown to be a effective tool to assess the attention level in our previous research [26] Both the valence and arousal dimensions have five grades, and the higher the grade, the stronger emotion. In practice, in order to determine a users’ emotional state, users will be asked to rate their emotions on the SAM model by selecting certain classes on each dimension.
comprehend it; a mental arithmetic task for which the subjects are asked to give a correct answer to an arithmetic question; a question answering task in which subjects a required to answer questions listed on a screen within a time limit. The base-line task records users’ EEG in a common state, while the other three mental tasks focus on mental the concentration state. Only by this way, can our system can get an optimal set of EEG signals corresponding to the attention level. Each kind of mental task will only take the subject about 1 or 2 minutes.
Classification
When the subject is learning, the system classifies his/her EEG signals to several attention levels which are consistent with the results of the Self-Assessment Manikin model. Apart from selecting some highly related EEG features, selecting a good classification algorithm is also very important to a BCI system. KNN is a very effective and simple classification algorithm and it works efficiently with low dimensional features, which meets our real-time and high-performance requirements. With enough training data sets, KNN can approximate any function which enables it to produce nonlinear decision boundaries [27]. In [14], the authors used KNN to detect the discrete emotions (happy, surprise, fear, disgust, and neutral) of human based subjects with EEG signals. EXPERIMENTAL RESULTS Subjects
A total of 8 student volunteers (ages from 20 to 25, 3 male and 5 female) participated in the Attention Recognition experiment. All the participants are right handed and healthy.
Figure 3. Student using our BCI system. Experimental Procedure
In the experiment, the subjects are required not to drink wine or coffee and not to take any medicines during the days of the experiment. Each subject takes seven sessions in the training phase and each session contains mental tasks, taking the subjects about one hour to complete. In each session, the subjects are asked some questions before the session is beginning, such as whether they have drunk wine or coffee; have taken medicines within the last 24 hours, have had poor sleep the previous night. If the subjects indicate that they have experienced one of the above,, the session is postponed for one day.
The subjects are seated in a soft armchair in a quiet room, and he/she is asked to perform a mental task after several minutes for relaxing. In each mental task, the computer screen shows a question or a paragraph or a mental arithmetic task, and subjects need to select an answer. As soon as the start button is clicked, the system begins to collect the EEG signals from the subject and stops at the time of an answer being selected. When the subject clicks the stop button, they will be asked to record their mental state based on the Manikin (SAM) model. Each session contains 15 mental tasks, and there is one minute relaxation between each task.
Mental tasks
Experiments Result
In our system, four kinds of mental tasks are involved in a experiment session: a base-line task in which the subjects are asked to relax as much as possible with eyes closed; a reading silently and comprehension task during which the subjects are asked to read a paragraph silently and
After all the subjects have finished seven sessions, we get a statistics result about the attention recognition accuracy of the KNN classifier in each session. The result is shown in Figure 4.
Accuracy Performance Evaluation
36
EEG Data Set A
recording latency 3 ms
artifact removing 15ms
features extracting 94ms
