Affective computation on EEG correlates of emotion from musical and vocal stimuli Reza Khosrowabadi, Abdul Wahab, Kai Keng Ang and Mohammad.H Baniasad
Abstract—Affective interface that acquires and detects the emotion of the user can potentially enhance the humancomputer interface experience. In this paper, an affective brain-computer interface (ABCI) is proposed to perform affective computation on electroencephalogram (EEG) correlates of emotion. The proposed ABCI extracts EEG features from subjects while exposed to 6 emotionally-related musical and vocal stimuli using kernel smoothing density estimation (KSDE) and Gaussian mixture model probability estimation (GMM). A classification algorithm is subsequently used to learn and classify the extracted EEG features. An intersubject validation study is performed on healthy subjects to assess the performance of ABCI using a selection of classification algorithms. The results show that ABCI that employed the Bayesian network and the One-Rule classifier yielded a promising inter-subject validation accuracy of 90%.
I. INTRODUCTION
I
nter-personal human communication includes verbal as well as non-verbal cues such as hand gestures, facial expressions and verbal tones that express emotions. The autonomous recognition of emotions can potentially enhance human computer interface. The growing interest in this field has led to the development of affective computation, which is a branch of artificial intelligence that designs computer systems that recognizes, interprets and processes human emotions [1]. User experience can be potentially enhanced by the introduction of affective dimensions in human computer interface. Therefore, affection computation of human emotion is important for research in behavioral science, computer games and medicine [2]. Emotion recognition mainly involves emotional expression and emotional perception of different stimulus that is related to cognitive and affective neuroscience [3-6] . For example, when a person is happy, the perception of stimulus from the person is biased towards happy, and likewise for negative emotions [7]. Traditional emotion studies are based on physical factors such as speech, facial This work was supported by the Nanyang Technological University and the Science and Engineering Research Council of A*STAR (Agency for Science, Technology and Research), Singapore. Reza Khosrowabadi and Abdul Wahab bin Abdul Rahman are with Centre for Computational Intelligence, School of Computer Engineering, Nanyang Technological University, Nanyang Avenue, Singapore 639798. (e-mail: {reza0004, asabdul} @ntu.edu.sg). Kai Keng Ang is with Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR), 1 Fusionopolis Way, #21-01 Connexis, Singapore 138632. (email:
[email protected]). Mohammad Hasan Baniasad is with Department Department of Psychiatry, Lavasani Psychiatric Hospital, Tehran medical branch of Azad University, Iran. (email:
[email protected]).
expressions or combination of speech and facial expressions [8],[9]. However, speech and facial expressions vary across cultures and nations. For example, the gesture nodding and shaking of the head respectively indicate confirmation and negation in some cultures whereas they are completely opposite in other cultures. Nevertheless, human biosignals are relatively more consistent across cultures and nations. Hence, recent emotion studies are based on human biosignals, such as Blood Volume Pressure (BVP), Skin Conductance Response (SCR), Respiration (RESP), Electrocardiogram (ECG), Electromyogram (EMG), Electrocorticogram (ECoG), Electroencephalogram (EEG), Heart Rate (HR), Oxygen Saturation (SaO2) and Surface Temperature (ST) [10-12]. Emotion studies generally employ classified affective responses of users using data labeling from questionnaires [1],[13],[14]. However, the affective responses are not easily mapped into distinctive emotion responses. Nevertheless, there are researches that mapped them to six basic emotions: happy, sad, fear, anger, surprise and disgust [13],[15]. However, there is no coherent notion on what are the basic emotions [16]. Furthermore, there are also views that some complex emotions are a combination of some basic emotions [15]. Hence, five of the six basic emotions, which exclude disgust, are studied in this work because they can be easily mapped from the affective responses [12],[17]. The following types of stimuli are generally used in the emotion studies to elicit the basic emotion responses: 1) Sound clips – auditory stimulus such as music, voices or environment noises affects human emotion. For example, there are relaxing as well as annoying music [18]. 2) Static pictures – visual stimulus also affects human emotion. For example, the International Affective Picture System (IAPS) is commonly used as visual stimulus for studies of emotion and attention [19]. 3) Movies – is a type of dynamic visual and auditory stimulus that has an increased affect on human emotion [20]. 4) Others – such as Somatosensory, olfactory and gustatory stimuli are less studied in the literature, although touching, smelling and tasting respectively are also known to affect human emotion [14]. This paper proposes an affective brain-computer interface (ABCI) to perform affective computation on EEG correlates of emotion. The study is performed on healthy subjects using emotionally-related musical and vocal stimuli because these stimuli have relatively constant affect on people across cultures and nations. In addition, we postulate that music
mixed with voice has a stronger affect in human emotion than instrumental music alone, and as such will enhance the degree of emotion detection. The proposed ABCI employs the features extracted from EEG measurements at specific locations on the left and right hemispheres. These features are then processed using Kernel smoothing density estimation (KSDE) and Gaussian mixture model probability estimation (GMM). A leave-one-out inter-subject validation study is then performed to assess the performance of ABCI using a selection of classification algorithms. The remainder of this paper is organized as follows: Section II describes the proposed Affective Brain-Computer Interface. Section III describes the experiment and presents the results. Finally, section IV concludes this paper. II. AFFECTIVE BRAIN-COMPUTER INTERFACE A. Neurophysiological background Literature studies have revealed that listening to music involves higher order brain functions [18],[21]. In addition, there exists a pattern of interdependency between different regions of brain of subjects while listening to music [22]. These asymmetric interdependencies between different brain regions are shown to be detectable by means of EEG measurement [22]. A study on epileptic patients and healthy subjects while exposed to emotionally-related musical stimuli also showed that the right and left anteromedial temporal lobes participate in music emotion perception processing [23]. Hence, the neurophysiological foundation of the proposed ABCI is based on the fact that subjects exhibit distinct EEG correlates of emotion while exposed to emotionally-related musical and vocal stimuli. B. Related works Krause, Viemerö and Rosenqvist et al studied the EEG measurements of 18 healthy subjects while exposed to angry, sad and neutral emotionally-related movie clips in 2000 [24]. Their study showed that the EEG frequencies in the theta and alpha bands of subjects respond differently when shown different types of movie clips. Aftanas and Golocheikine studied the EEG measurements of 27 healthy subjects while performing blissful meditation in 2001 [25]. Their study showed that the theta and alpha band in the anterior and frontal areas correlates with positive emotional experience. Takahashi proposed an emotion recognition system using multi-modal biosignals in 2004 [26]. He studied the EEG, SaO2 and SCR measurements of 12 healthy subjects while exposed to joy, anger, sadness, fear, and relax emotionallyrelated auditory and visual stimuli. He employed support vector machines and reported an averaged accuracy of 41.7% in classifying leave-one-out biosignals of the subjects. Lin, Chiu, and Hsu studied the EEG measurements of 6 healthy subjects while exposed to metal, sonata and subjectspecific favorite musical stimuli in 2005 [27]. They used
frequency distribution analysis and independent component analysis to extract features from the EEG measurements. Their study showed a correlation between the EEG features extracted and the type of music stimuli. Baumgartner, Esslen, and Jäncke studied the EEG, HR, SCR, and ST measurements of 24 healthy subjects while exposed to visual and musical stimuli in 2006 [4]. The stimuli comprised pictures from IAPS and classical musical excerpts that are emotionally related to happiness, sadness and fear. They used Analysis of Variance (ANOVA) on the biosignal measurements and showed that music can enhance the emotional experience evoked by affective pictures. Lin, Wang, and Wu et al proposed an emotion recognition system using multilayer perceptron classifier [28]. They studied the EEG measurements from 5 healthy subjects while exposed to joy, angry, sadness, and pleasure related musical stimuli. They extracted EEG features using hemispheric asymmetry alpha power indices and reported an averaged accuracy of 69.69% in classifying intra-subject EEG measurements for the 4 classes of emotion. Murugappan, Rizon, and Nagarajan et al proposed an emotion recognition system from EEG measurements in 2008 [29]. They studied the EEG measurements from 6 healthy subjects while exposed to movie stimuli. Affective responses of the subjects were solicited through the use of questionnaires and four emotions: happy, disgust, surprise and fear were studied. EEG features are extracted using energy, recoursing energy efficiency and root mean square, but they did not the accuracy of their emotion recognition system. Therefore, existing literature [4],[23-29] showed evidence that an emotion recognition system from EEG measurement is feasible. However, to the best of the authors’ knowledge, the best accuracy of existing emotion recognition systems is 69.69% for intra-subject classification of 4 emotions [28], and no inter-subject classification accuracies were reported. C. Architecture of the proposed ABCI The proposed Affective Brain-Computer Interface (ABCI) is illustrated in Fig. 1. It comprises three stages of EEG measurements processing: 4-13Hz bandpass filtering using elliptic filter, feature extraction and classification. The first stage employs a bandpass elliptic filter to filter the EEG measurements. This stage filters away artifacts such as low frequency ocular artifacts, high frequency EMG due to muscle movements and the 50 Hz power line interference. Hence, a 4-13 Hz bandpass filtering is employed in this stage to retain only the theta and alpha bands that have been shown to correlate to emotional experience [24],[25]. The second stage extracts features from the estimated statistical models of the filtered data. The two methods studied are in this paper are: Kernel Density Estimation (KDE) and Gaussian Mixture Model (GMM).
Fig. 1. Architecture of the proposed Affective Brain-Computer Interface (ABCI) Kernel Density Estimation (KDE) or Parzen Window [3032] is a non-parametric technique of estimating the underlying probability density f at data point X using X − Xi 1 n ) pˆ ( X ; h) = K( (1) ∑ nh i =1 h where K is a kernel function centered at the data points Xi, i=1,…,n; h is the window width. A popular choice of K is the Gaussian kernel 1 − y2 K ( y) = exp( ) (2) 2 2π Assuming that the underlying data is normal, a choice of the window function h [33] is given by h = 1.06σ n−1/ 5 (3) where σ is the sample interquartile range of the variable X. Gaussian mixture model (GMM) [34] is a mixture of several Gaussian distributions given by C
pˆ ( X ; μ , ∑) = ∑ ac N ( X ; μ , ∑)
(4)
c =1
where C is the number of distributions, ac is the mixture C
proportion 0
