CogniMeter: EEG-based Brain States Monitoring

CogniMeter: EEG-based Brain States Monitoring Xiyuan Hou1 , Yisi Liu1 , Olga Sourina1 , Wolfgang Mueller-Wittig1 , Wei Lun Lim2 , Zirui Lan2 , and Lipo Wang2 2

1 Fraunhofer IDM@NTU, Nanyang Technological University, Singapore School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore {houxy,liuys,eosourina,askwmwittig}@ntu.edu.sg {wllim2,lanz0001}@e.ntu.edu.sg, {elpwang}@ntu.edu.sg

Abstract. Electroencephalogram (EEG) techniques are traditionally used in the medical field. Recent research work focuses on applying these techniques to daily life with wireless and relatively low-price EEG devices available in the market. As a result, applications such as neurofeedback training, neuromarketing, emotion, stress, mental workload recognition, etc using EEG techniques on healthy adults have been developed. Since the EEG measures and records electrical activity in the brain, it is possible for it to reflect a person’s brain states. In this paper, we describe a novel brain computer interface called CogniMeter integrated with proposed real-time emotion, mental workload, and stress recognition algorithms. With this system, we can assess human emotions, mental workload, and stress in real time. This work can be applied as a human study tool in many fields. For example, the wellbeing of users within a system or workers in industry can be monitored to improve their protection from overly stressful workload conditions. In research, brain state monitoring can be applied in simulation scenarios during human factor study experiments. In marketing, a person’s emotional response toward products or advertisements can be studied using EEG-based brain states monitoring. Keywords: Visual Interface, EEG, Stress, Mental Workload, Emotions

1

Introduction

With an increasing number of wearable devices (Apple Watch, Fitbit, and Jowbone) becoming available in the commercial market, it becomes popular to use such wearable devices and mobile phones to monitor our daily physiological states based on number of walking steps, heart rate, calories consumed, and sleep pattern. All signals read by these wearable devices come either from the muscle activities or movement of the user. This however, limits the ability of these devices to monitor people’s mental states. A brain computer interface is more suitable for such analysis since Electroencephalogram (EEG) signals are directly captured from brain activity. A number of EEG-based methods and

2

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

corresponding applications have been designed and implemented in order to recognize the user’s brain states [1–3]. However, most of the contemporary works neither process EEG data in real-time nor provide intuitive visualization tools for data analysis and decision making. The EEG-based visualization of brain states can provide additional insights for human factor study experiments. In experimental design and analysis, the visualization tool can enable researchers to directly observe the relationship between task variables and subject’s states. In [40], an EEG-based brain states monitoring system is proposed to monitor emotion, mental workload and stress in real-time simultaneously. The stress level is inferred from emotion and workload recognition. In this paper, we describe a new version of the brain states monitoring system CogniMeter integrated with updated mental workload and stress recognition algorithms. This paper is structured as follows: Section 2 introduces related work on EEG visualizations tools and EEG-based emotion, mental workload and stress recognition algorithms. Section 3 presents the methodology used to implement these algorithms. Section 4 introduces the structure of the proposed visualization system and details of the visualization tools used to monitor brain states from EEG signals. Section 5 gives the conclusion.

2 2.1

Related Work Visualization Tools

There are a number of tools for analysis and visualization of EEG data such as EEGLab [4], Brainstorm [5], and ELAN [6]. These systems provide graphic user interfaces which enable users to interactively process high-density EEG data. For intuitive observation of brain activity, the amplitudes of EEG signals can be mapped to a 2-D or 3-D model of the scalp according to the EEG channel positions. These visualization tools are able to process EEG data epochs using spectral analysis or independent component analysis [4]. In [7], a blobby model is implemented for EEG signal visualization onto a 3D head model. Because EEG signals are non-stationary and high-dimensional, machine learning methods can be used for efficient real-time analysis of brain activity with specific tasks [8]. In this way, visualization tools can provide further interpretation of mental states from EEG data instead of showing only amplitudes of different channels. In [9], an EEG-based workload gauge is implemented and the workload level is monitored when subjects are doing cognitive and operational Air Traffic Controller’s (ATC) tasks. It is critical to reliably measure the brain states and performance of the controllers/pilots when the mode of automation is changed or new tasks appear. During simulation of the ATC task, the workload gauge is updated with the recognized level of workload (three workload levels). Other gauges depict dynamic changes in EEG signals and show spectrum powers in the form of brain maps simultaneously. In [8], an EEG-based mental text entry system Hex-o-Spell is proposed. By imagining left/right hand movement,

CogniMeter: EEG-based Brain States Monitoring

3

the user can generate different brain patterns. These patterns are used to control visual text input. The visualization tools include a text dial, an arrow, and a bar. The arrow’s orientaion and the graphic bar’s length are controlled by the brain states. 2.2

Emotion

Emotions can be defined from a dimensional perspective where arousal, valence, and dominance dimensions are considered [10]. In the dimensional model, the arousal dimension ranges from not aroused to excited, the valence dimension ranges from negative to positive, and the dominance dimension ranges from being controlled to be in control. The dimensional model is preferable for emotion recognition because it allows the location of discrete emotions in the dimensional space. Even feelings that cannot be described in words can be located using the dimensional model [11]. Emotions can be induced by different kinds of stimuli such as audio, visual and combined ones [12]. Different algorithms have been proposed for EEG-based emotion recognition. [3] extracted power features from EEG data and used Support Vector Machine (SVM) as a classifier. 82.37% accuracy for distinguishing four emotions was achieved with 32 channels. In [13], Short Time Fourier Transform (STFT) as a feature extraction method and SVM as a classifier were applied and a mean accuracy of 62.07% was obtained with 16 channels. But, all these accuracies are achieved for off-line emotion recognition. Additionally, as the EEG signal is nonlinear and chaotic, traditional features may hard to capture the nonlinear property of EEG. The fractal dimension (FD) can reflect changes of the EEG signal during different mental tasks in real-time [15]. In [16], FD, statistical and Higher Order Crossings (HOC) features were used for real-time EEG-based emotion recognition. 2.3

Mental Workload

Mental workload is described as a noticeable relationship between the human cognitive capacity and the effort required to process a particular function [17]. There are mainly three broad categories for workload definition: physiological, subjective, and cognitive workloads [18]. In this research, we are interested in cognitive workload which indicates the capability of a person to complete a task with some amount of mental effort. Performance in the task ascertains the cognitive workload [19]. In [8], mental workload is evaluated in online EEG monitoring during the security surveillance task. By comparing the mental workload index with the error rate for the subjects, the correlation coefficient is approximately 0.7, which indicates that when the workload increases, people have a tendency to make more errors. The significant positive correlation between workload and theta band power has been proven in [20–22]. In experiment [20], it is shown that the EEG theta band power increases when the workload is induced with a mental arithmetic task. In [21], the driver’s mental workload is significantly correlated with theta band power and alpha band power. In different driving

4

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

tasks, the frontal theta activity shows significant increase when working memory load increases. In another experiment studying the workload and fatigue in aircraft pilots [22], increased EEG theta band power and decreased alpha band power was observed during high mental workload as compared with low mental workload conditions. Additionally, in [22] it is shown that when the pilots have high mental workload and mental fatigue, their EEG theta band power, as well as delta and alpha band power increases. 2.4

Stress

Stress is a human state which is caused by a number of reasons, including high mental workload, emotions, or environmental influences. The stress can be measured and assessed from physiological variables including EEG [1, 2, 23, 24], blood pressure [25], heart rate variability [26], and skin conductance level [27]. EEG can be used to detect human stress levels. In work [25], the experiment shows that the stress is positively correlated with beta EEG power at the anterior temporal lobe. In [2], higher order spectra features are used for stress recognition. The SVM with RBF kernel is chosen as a classifier and the accuracy calculated with 5fold cross validation for recognition of two stress states is 79.2%. In [23], features such as Gaussian mixtures of EEG spectrogram, fractal dimension and magnitude square coherence estimation are used in the stress recognition algorithm. The classification of two levels of mental stress is done by k-Nearest Neighbor (k-NN) and SVM classifiers, and the best accuracy is 90%. However, neither [2] nor [23] used a standard stressor to induce stress in the experiments. In [1], a Stroop color-word test is used to induce stress. The discrete cosine transform is applied to reduce the data size and extract features from the frequency domain. Classification is implemented with an artificial neural network, linear discriminant analysis and k-NN. The best classification result for two stress states is 72% with k-NN. In [24], the band power of theta, alpha, and beta are used as features in logistic regression and are fed into the k-NN classifier. The results show a median accuracy of 73.96% for the recognition of relaxed and stressed states.

3 3.1

Methodology EEG Device

The Emotiv Headset [28] is used to capture the users’ EEG signals wirelessly with the USB receiver. It has 14 channels located at AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4 as shown in Fig. 1. 3.2

Feature Extraction

In our real-time EEG-based brain states monitoring system CogniMeter , emotion, mental workload and stress are recognized using machine learning methods. The statistic and FD features are used in real-time EEG-based brain states recognition.

CogniMeter: EEG-based Brain States Monitoring

5

Fig. 1. Location of 14 electrodes of Emotiv EEG device.

Fractal Dimension Feature FD measures the complexity and irregularity of time series [29]. It can be used as an index for characterizing the complexities of EEG signals. For a regular signal, the fractal dimension value is low. If the signal becomes irregular, the fractal dimension value increases accordingly. Wang et al. [30] proposed to use Higuchi fractal dimension to recognize different arithmetic mental tasks from EEG. It is also used in EEG-based serious games to identify attention level [31]. In this paper, the Higuchi algorithm is used to calculate FD feature from EEG data. The idea of Higuchi algorithm is as follows. Let X (1) , X (2) , . . . , X (N ) be a finite set of time series samples. Then, the newly constructed time series is     N −m Xtm : X (m) , X (m + t) , . . . , X m + ·t . (1) t where m = 1, 2, ..., t is the initial time and is the interval time [29]. t sets of Lm (t) are calculated by    N −m   ] t   [X   N −1   Lm (t) =  |X (m + it) − X (m + (i − 1) · t)| N −m /t. (2)  · t  i=1  t hL (t)i denotes the average value of Lm (t) , and one relationship exists hL (t)i ∝ t−dimH .

(3)

Then the Fractal Dimension dimH can be obtained by logarithmic plotting between different t (ranging from 1 to tmax ) and its associated hL(t)i [35] dimH =

ln hL(t)i . − ln t

(4)

6

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

Thus, the composed FD feature vector is F VF D = [dimH ].

(5)

Statistical Feature Statistical features are simple and widely used in EEGbased brain states recognition. For example, statistical features were used in EEG-based emotion recognition algorithms [32, 33]. Six statistical features including mean µX , standard deviation σX , mean of absolute values of the first differences δX , mean of absolute values of the first differences of normalized signals δX , mean of absolute values of the second differences γX , and mean of the second differences of the normalized signals γX are extracted from EEG for emotion recognition. The equations are listed as follows: µX =

σX

v u N u1 X (X(n) − µX )2 , =t N n=1

(6)

(7)

N −1 1 X |X(n + 1) − X(n)| , N − 1 n=1

(8)

N −1 1 X δX , X(n + 1) − X(n) = N − 1 n=1 σX

(9)

δX =

δX =

N 1 X X(n), N n=1

γX =

N −2 1 X |X(n + 2) − X(n)| . N − 2 n=1

(10)

The composed statistical feature is F Vstatistical = [µX , σX , δX , δX , γX , γX ]. 3.3

(11)

Data Processing

In the current version of CogniMeter, we use subject-dependent algorithms. The subject-dependent algorithms consist of two parts: calibration algorithm and real-time brain state recognition algorithm. The overall diagram of calibration and real-time emotion, mental workload and stress recognition algorithms is shown in Fig. 2. The emotion, mental workload and stress calibration algorithms follow the same pipeline but with different features. Different stimuli are used to induce brain states as well. First, the EEG data are labeled with classes such as levels of valence, mental workload or stress for emotion, workload or stress recognition correspondingly. Then, the EEG data are filtered, the corresponding features are extracted and the classifier is trained. After that, for real-time

CogniMeter: EEG-based Brain States Monitoring

7

recognition of valence, mental workload or stress levels, raw EEG data are filtered and the corresponding features are extracted using a 4 seconds sliding window with 3 seconds of overlap. Then, the data is input into the classifier with the pre-trained model acquired during calibration. After that, the classifier assigns the recognized valence, mental workload or stress level to each 4 second of EEG data.

Fig. 2. The overall diagram of calibration and real-time brain states recognition algorithms.

Emotion Recognition In this paper, the valence recognition algorithm proposed in work [39] is integrated into the CogniMeter system. The emotion can be evoked by audio or visual stimuli such as IADS [34] and IAPS [35].The algorithm calibration procedure is described in Section 4.1. The algorithm is tested on 10 subjects from DEAP database which is a publicly available affective database. Fractal dimension is used as the feature and the thresholds are employed to classify the brain states for up to four levels of valence, ranging from very unpleasant to very pleasant. 12 channels (AF3, F3, F7, T7, P3, P7, AF4, F4, F8, T8, P4, P8) are used and the weighted average voting strategy is applied to make the final decision about the current valence level. As shown in Table 1, the best accuracy is 79.31% and the mean accuracy is 49.4% for four levels valence recognition. In the calibration phase, two EEG datasets labeled with very pleasant and very unpleasant emotion are selected based on the questionnaire. Then, raw EEG data from the selected 12 channels are filtered by a 2-42 Hz bandpass filter, and features are extracted by subtracting the FD values of 6 right-hemispheric

8

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

channels from the 6 left-hemispheric channels: ∆F D = F Dlef t − F Dright , ∀lef t ∈ LH, ∀right ∈ RH

(12)

where a sliding window of size 512 and step size 1 (move forward 1 sample point at each step) is used to calculate the FD values. In the next step, the simple moving average (SMA) function with nonoverlapping window of size of 128 is applied to ∆F D and the obtained average ∆F D is used as features. To decide the threshold, the maximal and minimal average ∆F D values are obtained and the range between max ∆F D and min ∆F D is equally divided. As 12 channels are used in the proposed algorithm, 36 channel pairs are obtained (∆F DAF 3−AF 4 , ∆F DAF 3−F 4 , ..., ∆F DF P 1−F P 2 , ∆F DP 7−P 8 ) which leads to 36 sets of thresholds. In the real-time recognition phase, the average ∆F D is calculated from each channel pair and compared with the corresponding set of thresholds. The final valence level is determined by the weighted average vote strategy based on 36 valence results from each channel pair as follows. Define C(x) as the function that counts the votes for each valence level, where argument x ∈ {1, 2, 3, 4} is the valence level, 1 denotes the most unpleasant, 2 denotes unpleasant, 3 denotes pleasant, and P 4 denotes the most pleasant. 4 C(x) gives the vote counts of valence level x and x=1 C(x) = 36 . The final valence level L is determined as: P4 xC(x) (13) L = Px=1 4 x=1 C(x) where rounded L value can be used. Table 1. Classification Accuracy of Valence Level Recognition using Weighted Average Vote method Subject ID Weighted Average Vote 1 47.41% 5 48.28% 7 41.38% 10 74.14% 13 46.55% 14 39.66% 16 79.31% 19 47.42% 20 29.75% 22 40.09% Average 49.40%

Mental Workload Recognition Mental workload recognition algorithm proposed in [38] is integrated into the CogniMeter system. Since the mental workload is closely related to multitasking, SIMKAP simultaneous capacity test is

CogniMeter: EEG-based Brain States Monitoring

9

used to induce 4 levels of mental workload to calibrate the algorithm. The algorithm calibration procedure is described in Section 4.1. The algorithm was tested on the EEG database of 12 subjects with different feature combinations and classifiers used. In Table 2, it is shown, that for all feature combinations, the average accuracy of SVM classifier is 9.56% higher than k-NN classifier based on mental workload EEG data. By combining statistical and FD features of 14 channels and using SVM classifier, the best accuracy is 90.39% for 2 levels mental workload recognition and 80.09% for 4 levels mental workload recognition. Thus, in CogniMeter System, for mental workload recognition algorithm, we use statistical and FD features and SVM classifier. Parameters used for the SVM classifier are: polynomial kernel with penalty parameter C = 1, degree d = 5, gamma g = 1, coefficient r = 1. The feature vector defined for mental workload recognition is described as follows w F V w = [F V1 w , F V2w , ..., F V14 ],

(14)

w w F Vi w = [F Vi(F D) , F Vi(statstical) ],

(15)

where numbers 1 to 14 represent EEG channels AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4 according to the 10-20 international system. Table 2. Average Classification Accuracy of Different Feature Combinations for 2 and 4 levels mental workload recognition using SVM and k-NN Classifiers Classifier 2 Class SVM 2 Class k-NN 4 Class SVM 4 Class k-NN

Power 84.16% (6.39) 79.32% (5.24) 69.49% (8.03) 60.14% (7.60)

Stat. 89.79% (5.22) 84.67% (6.45) 78.56% (9.95) 69.65% (10.17)

FD Power+Stat. Power+FD Stat.+FD 82.92% 88.05% 85.79% 90.39% (6.48) (4.97) (5.80) (4.63) 80.27% 81.69% 80.04% 84.97% (6.08) (4.95) (4.70) (6.39) 65.26% 75.90% 71.78% 80.09% (10.06) (8.18) (8.14) (8.59) 60.63% 64.15% 61.47% 70.28% (9.05) (8.42) (6.48) (9.97)

Stress Recognition Stress recognition algorithm proposed in [37] is integrated in the CogniMeter System. For calibration, the Stroop Color Word Test [36] is adapted to elicit different levels of stress. The calibration procedure is described in Section 4.1. The algorithm was tested on the EEG database of 9 subjects where different levels of stress were induced using a Stroop color-word test. The comparison results shown in Table 3 also shows that the feature combination and classifier setting of stress recognition is very similar to mental workload recognition. By combining fractal dimension and statistical features with SVM as the classifier, four levels of stress can be recognized with an average accuracy of 67.06%, three levels of stress can be recognized with an ac- curacy of 75.22%, and two levels of stress can be recognized with an accuracy of 85.71%.

10

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

For SVM classifier, the polynomial kernel is chosen with penalty parameter C = 10 , degree d = 3 , gamma g = 1 , and coefficient r = 1 . By using EEG signal from all 14 channels, the feature vector defined for stress recognition is as follows s F V s = [F V1 s , F V2s , ..., F V14 ],

(16)

s s F Vi s = [F Vi(F D) , F Vi(statstical) ].

(17)

Table 3. Average Classification Accuracy of Different Feature Combinations for 2, 3, and 4 levels stress recognition using SVM and k-NN Classifiers Classifier 2 Class SVM 2 Class k-NN 3 Class SVM 3 Class k-NN 4 Class SVM 4 Class k-NN

4

Power 73.18% (10.20) 66.36% (8.96) 58.39% (11.32) 50.98% (10.99) 49.66% (12.20) 41.35% (9.64)

FD+Stat. Power+FD+Stat. 85.71% 80.96% (9.39) (8.86) 76.72% 69.93% (10.67) (9.44) 75.22% 69.82% (12.53) (11.88) 63.24% 54.44% (13.26) (10.63) 67.06% 60.71% (13.20) (11.50) 54.31% 44.97% (10.74) (11.42)

Visual Monitoring

Based on the emotion, workload and stress recognition algorithms described, we propose a real-time EEG-based brain state visual monitoring system named CogniMeter which is implemented to visualize the recognized brain states like emotion, workload, and stress levels. The overall pipeline of CogniMeter is illustrated in Fig. 3. It includes two parts: recognition and visualization. In the recognition part, the raw EEG data stream is obtained from the EEG device as input. Then, the EEG data is passed through the bandpass filter (2- 42 Hz). After that, different features are extracted from the filtered EEG data. For emotion recognition, the FD feature is used. For workload and stress recognition, statistical and FD features extracted from EEG data are used. These extracted features are compared with thresholds (emotion recognition) or fed into the pre-trained SVM classifier models (stress and workload recognition). Finally, emotion, workload and stress are recognized in real-time. In the visualization part, a Node.js server is created to receive the recognized emotion, workload and stress states. These recognized brain states are sent by each recognition program through TCP sockets. Then, a web browser connects

CogniMeter: EEG-based Brain States Monitoring

Fig. 3. The overall diagram of CogniMeter system.

11

12

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

to the server and renders three meters showing emotion, stress and workload in real time. The two advantages of this framework are the independence of recognition pro- grams and flexibility for use in various applications. When a new algorithm is proposed or new EEG device is used, they can be easily integrated to the visual monitoring system with a TCP connection. If the monitoring system is deployed in different applications like drivers model study, air traffic control or maritime human factor study, the visualization tools can be easily changed according to simulation scenarios. The users can monitor a subject’s brain states even from their tablet PC or smart phones. 4.1

Calibration Interface

As the proposed real-time EEG-based brain state recognition algorithms are subject dependent, calibration is required before real-time recognition. Each algorithm has its own calibration interface and calibration procedure. The emotion calibration interface is shown in Fig. 4. The stimuli to evoke certain emotions can be audio, visual, or both. In our calibration, sound clips from IADS database [34] targeted at evoking different valence levels are chosen and played one after another to the subjects. EEG data is recorded at the same time as when the subjects are listening to the sound clips. After each sound clip which lasts 60 seconds, the subject completes a prompted questionnaire (shown in Fig. 4a) to evaluate and describe his/her current emotion and feelings in words. After the recording is complete, the thresholds are trained based on the recorded EEG data and given emotional labels and are saved for future use. Fig. 4b, shows the training of positive (happy) and negative (sad) emotions. When the “Go to Emotion Recognition” button is clicked, the program starts real-time emotion recognition based on the trained thresholds and sends recognized emotional states to the server. The interface for mental workload calibration is shown in Fig. 5. SIMKAP simultaneous capacity test is used to invoke different workload levels. In low mental workload condition, subjects are required to complete item matching for different types of items including numbers, letters and shapes. For high mental workload condition, subjects are required perform multitasking as shown in Fig. 5a. It combines item matching and auditory questions such as telephone book search, schedule checking and answering arithmetic questions at the required time. After completing each workload level which lasts 60 seconds, the subject fills a prompted questionnaire to evaluate his/her mental workload level on the scale from 1 to 9 and also attempts to describe their feelings in words, as shown in Fig. 5b. Fig. 5c, shows labels of the recorded EEG data trained as three levels of workload : no task condition (Relaxed), item matching (Engrossed), and multitasking (Nervous). By clicking the “Start Recognition” button, the recognized mental workload states are sent to the server in real-time. The interface for stress calibration is shown in Fig. 6. Stroop color-word test, often used as a reliable psychological stressor, is applied to induce different levels of stress. The test has been shown to be one of the most effective methods for

CogniMeter: EEG-based Brain States Monitoring

13

Fig. 4. Screenshots of the emotion calibration interface: (a) questionnaire to label EEG data; (b) labels of the recorded EEG data.

14

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

Fig. 5. Screenshots of the mental workload calibration interface.

Fig. 6. Screenshots of the stress calibration interface.

CogniMeter: EEG-based Brain States Monitoring

15

re- searching human psychophysiological reactions in a stressful environment. In the Stroop color-word test, the sub- jects are given a list of color words in matching or non-matching colors as shown in Fig. 6a. In the congruent task, the words font color matches with the word’s meaning, e.g the word ”yellow” in a ”yellow” font, and subjects are tasked to identify the word’s font color. For the incongruent task, the word’s font color and the word’s meaning are different, e.g. the word ”yellow” but in a ”blue” font, and subjects are required to correctly identify the words font color. Due to the mismatch in font color and word meaning, the incongruent session is more stressful than congruent section. For the more stressful task, the subject can be required to make the response within a limited time (1.5 seconds). In such a situation, even higher stress levels can be elicited. After completing each Stroop color-word test, the subject is required to fill a prompted questionnaire to evaluate his/her stress level on the scale from 1 to 9 and describe their feeling in words, as shown in Fig. 6b. Fig. 6c shows the labels of the recorded EEG data as five stress levels : no task condition (Relaxed), congruent task (Normal), incongruent task (Engrossed), congruent task with time limit (Stressful), and incongruent task with time limit (Anxious). By clicking “Start Recognition”, the recognized stress level is sent to the server in real-time. 4.2

Visual Meters and Monitoring Report

Fig. 7. Screenshot of the CogniMeter for real-time brain stats monitoring system . Meters listed from left to are current level of workload, recognized emotion and current level of stress.

In the CogniMeter system, the recognized emotion, workload and stress levels are visualized as meters shown in Fig. 7. These visualization meters are

16

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

Fig. 8. EEG-based brain states monitoring report of workload, emotion and stress for three minutes monitoring.

CogniMeter: EEG-based Brain States Monitoring

17

developed based on JustGage [41] using JavaScript. These three meters display the current emotion, workload or stress level with changing positional angular bars and gradient colors (from green to red) in real time. A green colored bar on the workload/stress meter indicates a low workload/stress level, while a red colored bar represents a high workload/stress level. For the emotion meter, if the subject’s state is positive, the color of the bar will be green. Otherwise, it will be red. Besides color representation, there is a word in the center of each meter to describe current workload, emotion, and stress states. The words are updated at the bottom of the meter and correspond to the meter colors. For workload/stress, “High” corresponds to red color, “low” corresponds to green color and “mid” corresponds to yellow color. For emotion, “Pos” corresponds to positive emotions (green) while “Neg” corresponds to negative emotions (red). When the brain states monitoring is complete, a report appears on the screen. It is developed using HTML5 JavaScript charting library CanvasJS [42] and is generated to summarize the distribution of emotion, workload and stress states monitored. For example, Fig. 8 shows the distribution of mental workload in three levels (Low, Medium, and High), distribution of two emotions (positive and negative), and distribution of stress in five levels (Low, Medium low, Medium, Medium High, and High). The report can help researchers perform better analysis of the changes in a subject’s brain state during an experiment.

5

Conclusion

In this paper, a novel CogniMerter system is proposed and implemented. It allows monitoring of users real-time emotion, mental workload and stress using a head mounted EEG device. Visual meters and report generation provide the possibility to directly observe the relationship between the users emotions, mental workload, stress, and his/her task performance. This can help researchers to propose new hypotheses and refine their experimental procedures. We extracted different feature combinations and used SVM classifier for real-time emotion, mental workload and stress recognition. To monitor brain states in real-time, dynamic meters are proposed and implemented. CogniMeter system is used as a novel user study method in different simulation scenarios such as air traffic control and maritime training. Acknowledgments. The work is supported by Fraunhofer IDM@NTU, which is funded by the National Research Foundation (NRF) and managed through the multi-agency Interactive & Digital Media Programme Office (IDMPO).

References 1. Lim, C.-K.A., Chia, W.C.: Analysis of Single-Electrode EEG Rhythms Using MATLAB to Elicit Correlation with Cognitive Stress. International Journal of Computer Theory and Engineering 7, 149-155 (2015)

18

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

2. Hosseini, S.A., Khalilzadeh, M.A., Naghibi-Sistani, M.B., Niazmand, V.: Higher Order Spectra Analysis of EEG Signals in Emotional Stress States. In: 2010 Second International Conference on Information Technology and Computer Science (ITCS), pp. 60-63. (2010) 3. Yuan-Pin, L., Chi-Hong, W., Tien-Lin, W., Shyh-Kang, J., Jyh-Horng, C.: EEGbased emotion recognition in music listening: A comparison of schemes for multiclass support vector machine. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 489-492. (2009) 4. Delorme, A., Makeig, S.: EEGLAB: an open source toolbox for analysis of singletrial EEG dynamics including independent component analysis. Journal of Neuroscience Methods 134, 9-21 (2004) 5. Tadel, F., Baillet, S., Mosher, J.C., Pantazis, D., Leahy, R.M.: Brainstorm: A User-Friendly Application for MEG/EEG Analysis. Computational Intelligence and Neuroscience 2011, 13 (2011) 6. Aguera, P.-E., Jerbi, K., Caclin, A., Bertrand, O.: ELAN: A Software Package for Analysis and Visualization of MEG, EEG, and LFP Signals. Computational Intelligence and Neuroscience 2011, 5 (2011) 7. Sourina, O, Wang, Q., Liu Y., Nguyen M.: Fractal-Based Brain State Recognition from EEG in Human Computer Interaction. Biomedical Engineering Systems and Technologies 273, 258-272 (2013) 8. Muller, K.-R., Tangermann, M., Dornhege, G., Krauledat, M., Curio, G., Blankertz, B.: Machine learning for real-time single-trial EEG-analysis: From braincomputer interfacing to mental state monitoring. Journal of Neuroscience Methods 167, 82-90 (2008) 9. Monica Z. Weiland, Daniel M. Roberts, Michael S. Fine, Matthew S. Caywood: Real Time Research Methods: Monitoring Air Traffic Controller Workload During Simulation Studies Using Electroencephalography (EEG). Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 1615-1619 (2013) 10. Mehrabian, A.: Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology, 14(4), 261-292 (1996) 11. Mauss, I.B., Robinson, M.D.: Measures of emotion: A review. Cognition & emotion 23, 209-237 (2009) 12. Takahashi, K.: Remarks on emotion recognition from multi-modal bio-potential signals. IEEE International Conference on Industrial Technology (IEEE ICIT ’04), vol. 3, pp. 1138-1143 Vol. 1133 (2004) 13. Schaaff, K.: EEG-based emotion recognition. Diplomarbeit am Institut fur Algorithmen und Kognitive Systeme, Universitat Karlsruhe (TH) (2008) 14. Pradhan, N., Dutt, D.N.: Use of running fractal dimension for the analysis of changing patterns in electroencephalograms. Computers in Biology and Medicine 23, 381-388 (1993) 15. Lutzenberger, W., Elbert, T., Birbaumer, N., Ray, W.J., Schupp, H.: The scalp distribution of the fractal dimension of the EEG and its variation with mental tasks. Brain Topography 5, 27-34 (1992) 16. Liu, Y., Sourina, O.: Real-Time Subject-Dependent EEG-Based Emotion Recognition Algorithm. Transactions on Computational Science XXIII, pp. 199-223. Springer (2014) 17. Hart, S., Staveland, L.: Development of a multi-dimensional workload rating scale: Results of empirical and theoretical research. In, PA Hancock and N. Meshkati. Human Mental Workload (1988)

CogniMeter: EEG-based Brain States Monitoring

19

18. De Waard, D., Studiecentrum, V.: The measurement of drivers’ mental workload. Groningen University, Traffic Research Center (1996) 19. Sweller, J.: Cognitive load during problem solving: Effects on learning. Cognitive science 12, 257-285 (1988) 20. Sammer, G., Blecker, C., Gebhardt, H., Bischoff, M., Stark, R., Morgen, K., Vaitl, D.: Relationship between regional hemodynamic activity and simultaneously recorded EEG-theta associated with mental arithmetic-induced workload. Human Brain Mapping 28, 793-803 (2007) 21. Lei, S., Roetting, M.: Influence of Task Combination on EEG Spectrum Modulation for Driver Workload Estimation. Human Factors: The Journal of the Human Factors and Ergonomics Society 53, 168-179 (2011) 22. Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., Babiloni, F.: Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience & Biobehavioral Reviews 44, 58-75 (2014) 23. Khosrowabadi, R., Chai, Q., Kai Keng, A., Sau Wai, T., Heijnen, M.: A BrainComputer Interface for classifying EEG correlates of chronic mental stress. In: Neural Networks (IJCNN), The 2011 International Joint Conference on, pp. 757762. (2011) 24. Calibo, T.K., Blanco, J.A., Firebaugh, S.L.: Cognitive stress recognition. In: 2013 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), pp. 1471-1475. (2013) 25. Hassellund, S.S., Flaa, A., Sandvik, L., Kjeldsen, S.E., Rostrup, M.: Long-term stability of cardiovascular and catecholamine responses to stress tests: An 18-year follow-up study. Hypertension 55, 131-136 (2010) 26. Ssanghee, S., Yeongjun, G., Jungtae, L.: The Relation between Affective Style of Stressor on EEG Asymmetry and Stress Scale during Multimodal Task. In: Third International Conference on Convergence and Hybrid Information Technology, ICCIT ’08. pp. 461-466. (2008) 27. Jacobs, S.C., Friedman, R., Parker, J.D., Tofler, G.H., Jimenez, A.H., Muller, J.E., Benson, H., Stone, P.H.: Use of skin conductance changes during mental stress testing as an index of autonomic arousal in cardiovascular research. American Heart Journal 128, 1170-1177 (1994) 28. The Emotiv EEG device, http://www.emotiv.com/eeg/features.php 29. Higuchi, T.: Approach to an irregular time series on the basis of the fractal theory. Physica D: Nonlinear Phenomena 31, 277-283 (1988) 30. Qiang, W., Sourina, O.: Real-Time Mental Arithmetic Task Recognition From EEG Signals. Neural Systems and Rehabilitation Engineering, IEEE Transactions on 21, 225-232 (2013) 31. Wang, Q., Sourina, O., Nguyen, M.: Fractal dimension based neurofeedback in serious games.The Visual Computer 27, 299-309 (2011) 32. Liu, Y., Sourina, O.: EEG Databases for Emotion Recognition. In: 2013 International Conference on Cyberworlds (CW2013), 302-309. (2013) 33. Lan, Z., Sourina, O., Lipo, W., Yisi, L.: Real-time EEG-based Emotion Monitoring using Stable Features. The Visual Computer, 1-12 (2015) 34. Bradley, M.M., Lang, P.J.: The International Affective Digitized Sounds ( IADS2): Affective ratings of sounds and instruction manual. University of Florida, Gainesville, FL, Tech. Rep. B-3 (2007) 35. Lang, P., Bradley, M.M.: The International Affective Picture System (IAPS) in the study of emotion and attention. Handbook of emotion elicitation and assessment 29, (2007)

20

Xiyuan Hou, Yisi Liu, Olga Sourina et. al

36. Tulen, J., Moleman, P., Van Steenis, H., Boomsma, F.: Characterization of stress reactions to the Stroop Color Word Test. Pharmacology Biochemistry and Behavior 32, 9-15 (1989) 37. Hou, X., Liu, Y., Sourina, O.,Tan, E., Wang, L., Mueller-Wittig, W.: EEG based Stress Monitoring. In: IEEE International Conference on in Systems, Man and Cybernetics (SMC2015), 3110-3115, 2015. 38. Lim, W. L., Sourina, O., Wang, L., Liu, Y.: EEG-based Mental Workload Recognition Related to Multitasking. In: 10th International Conference on Information, Communications and Signal Processing (ICICS), in press, 2015 39. Lan, Z., Liu, Y., Sourina, O., Wang, L.: Real-time EEG-based User’s Valence Monitoring, In: Internation Conference on Information, Communications and Signal Processing (ICICS), 2-4 Dec, 1-5, 2015. 40. Hou, X., Liu, Y., Sourina, Mueller-Wittig, W.: CogniMeter: EEG-based Emotion, Mental Workload and Stress Visual Monitoring. In: 2015 International Conference on Cyberworlds (CW2015), 153-160, 2015. 41. JustGage, http://justgage.com 42. CanvasJS, http://canvasjs.com/