Valence, arousal and dominance in the EEG ... - Semantic Scholar

Int. J. Autonomous and Adaptive Communications Systems, Vol. 6, No. 1, 2013

Valence, arousal and dominance in the EEG during game play Boris Reuderink*, Christian Mühl and Mannes Poel Human Media Interaction, Faculty of EEMCS, University of Twente, P.O. Box 217, Enschede 7500 AE, The Netherlands E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] ∗ Corresponding author Abstract: In this paper, we describe our investigation of traces of naturally occurring emotions in electrical brain signals, that can be used to build interfaces that respond to our emotional state. This study confirms a number of known affective correlates in a realistic, uncontrolled environment for the emotions of valence (or pleasure), arousal and dominance: (1) a significant decrease in frontal power in the theta range is found for increasingly positive valence, (2) a significant frontal increase in power in the alpha range is associated with increasing emotional arousal, (3) a significant right posterior power increase in the delta range correlates with increasing arousal and (4) asymmetry in power in the lower alpha bands correlates with self-reported valence. Furthermore, asymmetry in the higher alpha bands correlates with self-reported dominance. These last two effects provide a simple measure for subjective feelings of pleasure and feelings of control. Keywords: affective computing; affective signal processing; emotion recognition; EEG; electroencephalogram; BCI; brain–computer interfaces; alpha asymmetry; frustration; valence; arousal; dominance; adaptive communication systems. Reference to this paper should be made as follows: Reuderink, B., Mühl, C. and Poel, M. (2013) ‘Valence, arousal and dominance in the EEG during game play’, Int. J. Autonomous and Adaptive Communications Systems, Vol. 6, No. 1, pp.45–62. Biographical notes: Boris Reuderink received his Master’s degree in 2007, after spending time on different machine learning problems, including recognition of hand-written text on envelopes and the detection of laughter in audio–visual data. Brains and intelligence – his lifelong interest – can be combined his PhD position at the University of Twente, in which he focuses on making brain–computer interfaces function in real-world settings for healthy users. Christian Mühl received his Master’s degree in Cognitive Sciences in 2007 at the University of Osnabrück, Germany. His education was focused on neuroscientific methods, specifically electroencephalography. Since then he is working as a research assistant in the Human Media Interaction Group at the University of Copyright © 2013 Inderscience Enterprises Ltd.

45

46

B. Reuderink, C. Mühl and M. Poel Twente, The Netherlands. In his PhD Thesis, he searches for neurophysiological correlates of affective states in the context of brain–computer interaction. Mannes Poel is an Assistant Professor in the Human Media Interaction Group at the University of Twente. His main research involves applied machine learning for vision-based detection and interpretation of human behaviour and the analysis and classification of EEG-based brain signals.

1

Introduction

A brain–computer interface (BCI) offers a direct link between the signals produced by the user’s brain and a computer, and is aimed traditionally at patients suffering from amytrophic lateral sclerosis (ALS). But for healthy users, brain signals can also provide information that can be used to augment traditional human–computer interaction (HCI). One source of information that a BCI may provide is an estimate of subjective emotions while they are being experienced. The enrichment of HCI by incorporating information about the emotional context of the interaction or the user’s behaviour is studied in the field of affective computing (Picard, 1997). Affective information enables the application to respond more adequately to the user, e.g. by offering help in situations of extreme frustration, stress or misunderstanding. Several modalities are able to convey information about the user state without directly intruding into the user’s consciousness or task at hand, as e.g. the observation of overt user behaviour (e.g. facial expressions or prosody) (Zeng et al., 2007). Another source of affective information that is less dependent on overt behaviour is the group of physiological responses, such as cardiovascular responses (Frazier et al., 2004; Lang et al., 1993), electrodermal activity (Codispoti et al., 2001; Lang et al., 1993) and muscle activity (Cacioppo et al., 1986; Magnée et al., 2007). A specific case of physiological activity is produced by neurons in the neocortex. Similar to the affective responses recorded from the other parts of the body, the brain activity has been strongly associated with perceptive, experiential and expressive emotional processes (Demaree et al., 2005). More specifically, cognitive theories of emotions claim the brain to be the primary source of affective responses, as it is processing the very stimuli, memories or thoughts that lead to an affective reaction in brain, body and behaviour (Sander et al., 2005). According to those theories, the use of brain signals would enable a faster and more direct recognition of emotional states, and be sensitive to more subtle emotions that do not cross the threshold of bodily expression. As first discriminative neuronal responses to emotional stimulation can already be observed within tens of milliseconds (Aftanas et al., 2002; Keil et al., 2001), brain activity responds faster to emotional stimulation than any other physiological measure. Thus, sensors measuring neuronal activity from the brain offer a relatively non-intrusive method of fast and continuous observation of the user’s affective state i.e. not dependent on overt behaviour. Several studies have shown the principal capability to yield estimates of user states with acceptable precision via physiological (Benovoy et al., 2008; Chanel et al., 2006, 2008; Kim and André, 2008; Kreibig et al., 2007; Liu et al., 2009; Mandryk et al., 2006; Picard et al., 2001; Rainville et al., 2006) and neurophysiological signals (Chanel et al., 2006,

Valence, arousal and dominance in the EEG during game play

47

2009; Khosrowabadi et al., 2009; Ko et al., 2009; Lin et al., 2009; Murugappan et al., 2008; Rizon et al., 2008; Takahashi, 2004). Correlates of affect have been studied via a multitude of exogenous and endogenous affect elicitation paradigms. Affect induction protocols have made use, among others, of pictures (Aftanas et al., 2001; Huster et al., 2009; Keil et al., 2001; Müller et al., 1999), sentences (Marosi et al., 2002), music (Sammler et al., 2007; Schmidt and Trainor, 2001), videos (Aftanas et al., 2006; Krause et al., 2000) and recall of emotional events (Chanel et al., 2009). These neural correlates can be used in the design of an automatic classifier of affect. But to ensure the generalisation of the findings from the controlled context of the laboratory to a real-world HCI context (ecological validity), the affective state is best elicited in a way resembling the context of application. The goal of the present study is to find affective neurophysiological correlates of frustration, valence, arousal and dominance in an ecologically valid context, in this case, game play. To evoke emotions, we introduced frustration by ignoring keyboard input periodically. Using a non-responsive controller is a well-known method in the fields of affective computing and HCI (Diener and Oertel, 2006; Klein et al., 2002; Scheirer et al., 2002). Prior studies have shown that it is in principle possible to identify neurophysiological correlates of affective states elicited by gaming events (Salminen and Ravaja, 2007, 2008) and with HCI in general (Spiridon and Fairclough, 2009). Section 2 will introduce the reader to the relevant types of brain activity, results from the affective neurosciences and consequently, specific hypotheses derived from these prior studies.

2

Emotions and the brain

Emotions can be studied within the framework of the circumplex model (Russell, 1980). This model defines a two-dimensional emotional space, spanned by a continuous valence (pleasure) and an arousal dimension. In this space, all categorical emotional labels can be defined in terms of valence and arousal. For example, fear is associated with a state of high arousal and negative valence, whereas excitement is characterised by a high arousal and a positive valence. A third dimension, named dominance, can be added to this space to signify the subjective feeling of control (dominant vs. submissive). Dominance can be understood as a social or cognitive interpretation of an affective event (Russell, 1980), indexing the “interactive relationship that exists between the perceiver and the perceived [object or situation]’’ (Bradley and Lang, 1994). By manipulating the affect of users, differences in this emotional space can be mapped to specific regions of the brain. For example, processes related to emotion recognition are associated with the right hemisphere (right hemisphere theory), whereas positive and negative emotional states have been attributed to regions in the left and right frontal cortices, respectively (the valence theory) (Silberman and Weingartner, 1986; Tucker, 1981). An alternative to this valence theory of emotion is the approach/withdrawal theory (Davidson, 1992), in which the different hemispheres are differently activated depending on the motivational direction of the emotional state: left hemispheric frontal activity is associated with approach, whereas right hemispheric activity is associated with withdrawal. As most approach-related emotions are associated with a positive feeling and most withdrawalrelated emotions with a negative feeling, there is a considerable overlap of both theories. The defining difference lies in the hemispherical activation caused by the emotion of anger; while it is a negative emotion, it is associated with an approach behaviour.

48

B. Reuderink, C. Mühl and M. Poel

Interestingly, the dominance dimension enables a differentiation between approach and withdrawal emotions that cannot be differentiated on the basis of valence and arousal alone. For example, anger is accompanied by high dominance (sense of control) ratings, whereas fear scores low on dominance (Demaree et al., 2005). This is especially important in the current study, as the induced frustration (low dominance) is close or might lead to anger (high dominance), which are both characterised by a negative valence and high arousal. While for the valence and arousal dimension, responses in the electroencephalogram (EEG) to affective stimulation have been reported, for dominance, we could only find Heraz and Frasson (2007). Heraz and Frasson used broad-band power features in the delta, theta, alpha and beta bands, and correlated continuous measurements with self-reported valence, arousal and dominance. Weak correlations with dominance were found, but it is unclear whether the reported correlations are significant, as not all observations were independent. In the following, we will shortly review literature on the association of emotional processes and oscillatory brain activity. The previously introduced hemispheric valence theory holds that positive emotions are processed in the left frontal cortex, whereas negative emotions are processed in the right frontal cortex. As cortical activation is held to be inversely related to alpha (8–12 Hz) activity (Pfurtscheller, 1999), the processing of positive emotional information is reflected in a decrease of alpha power of the left frontal hemisphere, the processing of negative emotional information is reflected by the decrease of alpha power over the right frontal hemisphere. This pattern was shown in studies using a wide variety of emotion induction protocols (Allen et al., 2004; Davidson, 1992; Huster et al., 2009; Schmidt and Trainor, 2001). The frontal alpha asymmetry is the most frequently found correlate of valence. Many other correlates for valence are observed in a less robust manner. For example, the frontal asymmetry was also found in the theta range (4–7 Hz) in response to positive and negative picture content (Aftanas et al., 2001) and sentences (Marosi et al., 2002). Müller et al. (1999), also presenting pictures, did not find hemispheric interactions with valence in the alpha frequency range, but in the gamma range at the temporal lobes. Similarly, Keil et al. (2001) found responses in the gamma range for aversive pictures. Medial and frontomedial effects of emotion were found in several studies. An increase in fronto-medial theta power was observed for positive music stimuli (Sammler et al., 2007) and during meditation (Aftanas and Golocheikine, 2001), which was interpreted as emotional processing closely associated with attentional processes. Krause et al. (2000) found a power increase in the low-theta band over midline electrodes in response to anger inducing movies, supporting the notion of an attention-related effect. The same region was shown to be more active in terms of delta (0–3 Hz) oscillations when aversive stimuli were presented (Klados et al., 2009). Arousal activates neural structures in general, and therefore should be associated with a global decrease of power in the alpha band (Barry et al., 2007, 2009). More localised effects in the alpha band are found in Schmidt and Trainor (2001) and Aftanas and Golocheikine (2001), but the effects seem to contradict each other; Schmidt and Trainor found a correlation between the activation of frontal cortical regions and the perceived arousal of musical excerpts, while Aftanas et al. found a deactivation (measured as an increase in high-alpha power) in the frontal regions associated with increasing arousal. Other EEG phenomena observed at posterior sites with increasing arousal include the increase of power in lowfrequency bands, as in the delta (Klados et al., 2009) and theta band (Aftanas et al., 2002), the decrease of alpha power, and an increase in upper gamma power (Aftanas et al., 2004).


3

49

Research questions

Based on the literature described in Section 2, we formulated hypotheses on the interaction of affective states and the EEG to be confirmed in an ecologically valid context for an adaptive interface. First, we describe the hypotheses for the valence dimension. In the delta band, we have the hypothesis Hvδ : fronto-medial delta power increases with increasing valence. For the theta band, we define three distinct hypotheses: Hvθ 1 : with increasing valence, left hemispherical theta power increases, Hvθ 2 : with increasing valence, right hemispherical theta power decreases and Hvθ3 : with increasing valence, posterior theta power increases. In the alpha band, we define the following hypotheses: Hvα1 : with increasing valence, left hemispherical alpha power decreases, Hvα2 : with increasing valence, right hemispherical alpha power increases. Finally, for the gamma band: Hvγ 1 : with increasing valence, left temporal gamma power decreases and Hvγ 2 : with increasing valence, right temporal gamma power increases. For arousal, we formulate the hypotheses Haδ : posterior delta power increases with increasing arousal, Haθ 1 : posterior theta power increases with increasing arousal, Haα1 : global alpha activity decreases with increasing arousal, Haα2 : frontal alpha power increases with increasing arousal, Haβ1 : posterior beta power is positively correlated with arousal and Haγ 1 : high arousal correlates with a power increase in the high-gamma band. These hypotheses are summarised in Table 1. Table 1 Dimension

Hypotheses for valence, arousal and dominance correlations in the frequency domain Delta

Theta

Alpha

Beta

Gamma

Valence ↑

Hvδ : fron.-med. ↑ Hvθ1 : l-hemi. ↑ Hvα1 : l-hemi. ↓ – Hvγ 1 : l-temp. ↓ Hvθ2 : r-hemi. ↓ Hvα2 : r-hemi. ↑ Hvγ 2 : r-temp. ↑ Hvθ3 : fron.-med. ↑ Arousal ↑ Haδ : posterior ↑ Haθ : posterior ↑ Haα2 : global ↓ Haβ : parietal ↑ Haγ : gamma ↑ Haα1 : frontal ↑ Dominance ↑ – – – – –

4

Methods

To test the above formulated hypotheses, the following methodology was used:

4.1

Data collection

To obtain EEG data containing different emotional states, we used a game designed to induce frustration for short periods by ignoring 15% of the keyboard input (Reuderink et al., 2009). In the blocks where frustration is induced, the screen periodically lags as well to simulate an underpowered computer. The experiment runs through a sequence built from random permutations of two normal, and one frustration block of 2 min each. After each block, the user is asked to rate his current mental state on the emotional dimension valence, arousal and dominance by pressing a numeric key displayed below the Self-Assessment Manikin (SAM) (Bradley and Lang, 1994), see Figure 1.

50


Figure 1

The game, and the self-assessment screen for valence, arousal and dominance (see online version for colours)

Note: In our analysis, we inverted the scale of valence and arousal, so that high valence corresponded to positiveness, and high arousal to a more aroused state.

4.1.1

Experimental procedure

Users were asked to read and sign a form of consent, and were subsequently wired with the EEG and physiological sensors. The experimenter briefly explained the game and the self-assessment procedure. The participant was allowed to practise the controls for 2 min, and then the experiment was started. When users mentioned that the game was unresponsive during the experiment, the experimenter asked them to continue playing and promised to find the source later. After 30 min, the experimenter stopped the experiment and the users were debriefed.

4.2

Subjects

Twelve healthy users (age 27 ± 3.9) participated in the experiment. All participants had normal, or corrected to normal vision, and reported no use of medication. Only three of our subjects were female, and all subjects were right-handed. Most participants had some experience with video games.

4.2.1

Sensors and recording

A BioSemi ActiveTwo EEG system was used to record the EEG and physiological signals at a sample rate of 512 Hz. For EEG, 32 Ag/AgCl active electrodes were used, placed at the


51

locations of the Extended International 10–20 system. To measure the influence of ocular and muscular artefacts, we recorded the EOG (horizontal and vertical pairs) and two pairs of EMG signals for the finger movement used to control the game. Additional physiological sensors, such as temperature, respiration, the galvanic skin response and the blood volume pulse were recorded as well, but not used in the present study.

4.3

Preprocessing

Raw EEG data can contain noise from the environment, eye movements or muscle tension. We applied the following preprocessing procedure to reduce the influence of these signals. First, the recording was decimated to 128 Hz. After decimating, the channels were highpass filtered using a fourth order Butterworth filter to remove frequencies below 0.2 Hz, and notch-filtered using a fourth order Butterworth filter from 49 to 51 Hz to remove power line noise. The EEG was subsequently corrected for eye movements using the (linear) regression analysis subtraction method of Schlögl et al. (2007) based on the covariance between EEG and EOG channels. Only after the removal of the EOG influence, the data was re-referenced to the common average reference (CAR), that subtracts the average signal from each sensor at each time point.

4.4

EEG features

Alpha asymmetries were calculated for lateral sensor pairs, using the procedure outlined in Allen et al. (2004): for each experimental block, Welch’s method was used to estimate power in different frequencies for each EEG-channel with 1 Hz resolution. Within each block, we summed the (natural) log-power in the alpha band (8–12 Hz) for each electrode, and subtracted the band power of electrodes on the left hemisphere from corresponding electrodes on the right hemisphere. This results in an alpha-asymmetry index for each sensor pair for each experimental block, which should correlate positively with valence. The same spectral estimates for each experimental block were used to calculate the correlations between log-power of narrow-band EEG oscillations and SAM ratings. Note that we have deliberately chosen for an analysis using spectral bins of 1 Hz, as the emotional response can differ for small differences in frequency bands (Krause et al., 2000). The drawback of this choice is that we have to perform more statistical tests, and that therefore the effects have to be quite strong to pass the multiple-test correction.

4.5

Statistical tests

For the alpha asymmetry, we will simply report correlations of the alpha asymmetry index of different sensor pairs, with the self-reported ratings for valence, arousal and dominance. Statistical significance is determined with a Wilcoxon signed-rank test over the correlations of a sensor pairs with emotional dimensions. We have chosen not to limit our significance tests to only correlations with valence, but to include the frustration condition, arousal and dominance as well. These extra tests are meant to be illustrative; therefore we do not perform multiple test correction on the alpha-asymmetry correlations. For the narrow-band oscillations, we do perform multiple test correction using a combination of Bonferroni correction and Fisher’s method. Bonferroni correction conservatively corrects for performing multiple tests by dividing the critical value α by

52


the number of tests performed. But when a large number of tests is used, its conservative formulation results in critical α values that cannot be passed by individual statistical tests. This problem is well-known in neuroscience, and there are various statistical methods to overcome this problem, e.g. by combining evidence over subjects (Lazar et al., 2002). One method to combine evidence is to use the distribution of the p-values associated with multiple statistical tests instead of using the distribution of the correlation coefficients ρ. Fisher’s method is suggested by Loughin for testing distributions of p-values for significance: X 2 = −2

k

loge pi

(1)

i=1

where pi is the p-value for subject i. When all the null hypotheses are true, and all the p-values are independent, X 2 follows a χ 2 distribution with 2k degrees of freedom. In our experiment, we have k = 12 independent subjects, therefore we compare the X 2 of the 12 p-values to a χ 2 distribution with 24 degrees of freedom. Note that we now combine the different p-values from different subjects into a combined null hypothesis H0 , that states that each of the individual null-hypothesis is true. The combined alternative, HA , is that at least one is not true (Loughin, 2004). While it is tempting to use Fisher’s methods not only over subjects, but also within the frequency band, we cannot because the observations within a frequency band are not independent. Bonferroni does not require independent p-values, and is therefore used to correct for the number of tests within a specific frequency band. To summarise: for the 12 subjects, the correlation coefficient ρ and it’s associated p-value of the band power at a specific sensor with a emotion dimension was calculated. Within a frequency band (delta: 0–3 Hz, theta: 4–7 Hz, alpha: 8–11 Hz, beta: 12–29 Hz and gamma: 30+ Hz), the one-sided p-values were combined over the 12 subjects using Fisher’s method, and these combined p-values were subsequently corrected for the number of tests within the band (α = 0.05/(2 × 32 × b), where b is the number of 1 Hz bands in the frequency band).

5 5.1

Results Self-assessments

For each user, we have approximately 15 different measures of subjective valence, arousal and dominance. These self-assessments were rescaled to unit intervals for easy interpretation. Most subjects rated the frustration inducing condition less positive than the normal condition, over subjects this difference is significant (T = 3, p < 0.01). While we expected to find a trend towards more arousal in the frustration condition, there appears to be no difference (T = 23, p = 0.26). The dominance scale indicates that people seem to be significantly (T = 4, p < 0.01) more in control in the normal condition. Individual ratings for the valence and dominance dimensions show spread values with generally higher ratings for both valence and dominance in the frustration condition (Figure 2). Users enjoyed the game; therefore the negative emotions are slightly underrepresented.

Valence, arousal and dominance in the EEG during game play Figure 2

53

Individual valence and dominance ratings for the normal (blue circles) and frustration (red squares) condition (see online version for colours)

Note: For most subjects, the ratings are nicely spread with higher dominance and valence for the ratings for the blocks with frustration induction. Table 2

The correlation among experimental blocks (time), condition (frustration induction), valence, arousal and dominance of the self-assessments Time

Time Condition Valence Arousal Dominance

1.00 0.01 −0.04 0.07 0.01

Cond. 0.01 1.00 −0.32 0.08 −0.32

Val.

Ar.

Dom.

−0.04 −0.32 1.00 0.10 0.43

0.07 0.08 0.10 1.00 −0.15

0.01 −0.32 0.43 −0.15 1.00

Note: There are no strong correlations with time, but valence, dominance and the experimental condition seem to be correlated.

The correlation coefficients in Table 2 confirm the observed relation among ratings of valence, dominance and the frustration condition. While dominance and valence are supposed to be orthogonal emotional dimensions, they do not seem to be uncorrelated. Correlations with time are included as well to show trends of the self-assessments over time. As expected, the experimental condition barely correlates with time, and the emotional ratings do not drift overtime. Interestingly, the dominance ratings correlate more strongly with the valence ratings (ρ = 0.43) than with the experimental condition (ρ = −0.32). This shows that the dominance ratings do not simply capture the physical lack of control of the user, but the subjective experience of dominance.

54


The non-orthogonality of the ratings complicate the interpretation of the results. In addition to the original ratings, we also analysed correlations with the scores of a principal component analysis (PCA) of these ratings. The loadings for the new dimensions are displayed in Table 3. More than half of the variance is explained by a component in the direction of normal keyboard control, and positive valence (happiness), which is not surprising as this is the main effect in our experiment. Orthogonal to this direction, we find a second principal component (PC) that models variance in the direction of the normal condition combined with negative valence (sadness). Arousal and dominance do not play a big role in PC0 and PC1, but are modelled in PC2 (high dominance with low arousal, or superiority) and PC3 (high dominance with high arousal, or anger) instead. These new dimensions can be used to disentangle effects found for both valence and dominance. Table 3

The PCs for the space spanned by the frustration condition, valence, arousal and dominance

Description Var. explained Condition Valence Arousal Dominance

5.2

PC0 Mainly positive emotions and controllable game (happiness) 52% −0.72 0.66 0.06 0.22

PC1 Mainly negative emotions and controllable game (sadness) 32% −0.70 −0.69 −0.11 −0.17

PC2 Relaxed and dominant mental state (superiority) 11% 0.05 −0.13 −0.67 0.73

PC3 Mainly aroused and dominant mental state (anger) 5% 0.00 −0.27 0.73 0.62

Alpha asymmetries

For each user individually, we calculated the alpha asymmetry index as outlined in Section 4 for each experimental block. Table 4 lists the Pearson correlation coefficients ρ between alpha asymmetries and the emotional ratings, averaged over subjects. We used a Wilcoxon signed-rank test to find ρ’s that deviate significantly from zero over subjects. While the correlations are quite low, we find significant correlations for the unmodified valence and dominance scale. A notable observation is that dominance correlates better with the alpha asymmetry than valence over the fronto-central cortex. In the PCA space, alpha asymmetry in the fronto-central is correlated more strongly with PC2, the PC that is associated with ‘superior’ feelings (relaxed and dominant). For PC0 (happiness), we do find a correlation with alpha asymmetry for frontal alpha power (Fp1–Fp2). For frontal alpha power, we also found a significant negative correlation with the condition variable, and a positive correlation with valence. PC0 is oriented in exactly this direction, so it is not surprising that we do find a significant positive correlation of alpha asymmetry in Fp1–Fp2 with PC0, but it remains unclear whether this asymmetry is caused by valence, or mainly by the experimental condition. Our results revealed a correlation over subjects of alpha asymmetry with valence, arousal and dominance. Traditionally, alpha asymmetry is associated with valence, but our results suggest that there is a stronger link between alpha asymmetry in fronto-central sensor pairs with dominance than with valence.

Valence, arousal and dominance in the EEG during game play Table 4

55

The alpha asymmetry for different sensor pairs, correlated with different labels for each subject individually, and averaged over subjects Fp1–Fp2

AF3–AF4

F3–F4

FC1–FC2

C3–C4

F7–F8

P3–P4

Condition Valence Arousal Dominance

−0.15* 0.19 0.00 0.07

−0.04 0.00 −0.13 0.01

−0.07 0.12 −0.12 0.18*

−0.00 0.16* −0.16* 0.18**

−0.09 0.11 0.08 −0.06

−0.05 −0.05 −0.04 0.03

0.05 0.06 0.06 −0.01

PC0 PC1 PC2 PC3

0.19* 0.03 −0.01 −0.05

0.04 0.04 0.13 −0.10

0.08 0.03 0.17* −0.08

0.05 −0.07 0.19** −0.14

0.11 0.03 −0.13 −0.04

0.02 0.10 0.05 −0.00

−0.01 −0.12 −0.04 0.02

∗

p < 0.05. p < 0.01. Note: Correlations were tested for significance over subjects using a Wilcoxon signed-rank test.

∗∗

Table 5

Significant narrow-band correlations for the self-assessment and experimental condition Delta

Theta

Alpha

Beta

Gamma

Frus. cond.

–

0.3 CP1**

–

–

Valence Arousal Dominance PC0

– – – –

−0.2 FC6* – – −0.3 CP1*

– – – –

– – – –

– –0.3 P8** –

– – –

0.3 F3** 0.2 FC1*** 0.3 CP1** 0.2 Fz** – 0.2 F8* – −0.3 F3** −0.3 FC1*** −0.3 Fz** – – –

– – –

– −0.2 Fp2** –

PC1 PC2 PC3 ∗

p < 0.001. p < 10−4 . ∗∗∗ p < 10−5 . Note: For these correlations, the combined p-values pass the Bonferroni corrected α threshold, but that the p-values are not adapted themselves to reflect the multiple test correction. ∗∗

5.3

Narrow-band oscillations

Afterwards, we looked at significant correlations that we found for the frustration condition, valence and arousal, and relate these to the hypotheses derived from literature (Table 5). For valence, we only find significant negative correlations in the theta range (at 5 Hz) on the right fronto-central region (FC6), which confirms hypothesis Hvθ 2 , that predicts a right frontal theta decrease for increasing valence. Due to lack of other significant correlations for valence, we can neither confirm nor reject the other hypotheses. However, we can relate the trends in Figure 3 to the hypotheses

56


for valence: the fronto-medial delta increase for positive stimuli (Hvδ ) is not visible for valence. The arousal dimension, however, does show an increase in delta activity, as does the frustration condition. Hvδ is based on Klados et al. (2009), where aversive stimuli were presented; perhaps these aversive stimuli were arousing as well. In the theta range, we find no evidence of Hvθ 1 that states a positive correlation between frontal theta and positive valence. Also for a posterior theta increase (Hvθ 3 ), no evidence was found. When we look at the alpha-range, the well-known alpha asymmetry is clearly visible at 11 Hz in Figure 3, mirroring the asymmetry in the same frequency band for the frustration condition. This can be related to the effect described in Hvα1 , and while this effect is not significant, the trend corresponds with the hemispherical valence theory. The related Hvα2 for right brain alpha increase with increasing valence could not be found, perhaps because negative emotions were underrepresented in this study. At the higher gamma ranges (>51 Hz), there is a trend towards decreasing gamma activity for increasing valence, but this is most likely caused by increased muscle tension in the musculus temporalis, and not by brain oscillations. So, a trend was observed for Hvγ 1 , but for Hvγ 2 no evidence was found. Figure 3

Mean subject correlations of oscillations in different frequency bands with the frustration condition (first row), valence (second row), arousal (third row) and dominance (last row) (see online version for colours)

Note: Due to space constraints, we have plotted only every second frequency bin of the lower frequency ranges. The alpha asymmetry for the frustration condition and valence is visible at 11 Hz, and also the central parietal correlation of frustration with theta (5 Hz) and the right fronto-central theta correlation with valence are visible. Arousal shows focused activation for the delta band, but the correlation of alpha in the right frontotemporal region with arousal is drowned in other effects. Dominance does not show correlations, except for the high-alpha range (13 Hz), where it shows an alpha asymmetry that corresponds with the effect found for the frustration condition.


57

For arousal, we find a significant (positive) narrow-band correlation for the right frontal region in the alpha band, but this correlation is quite weak, and barely visible in the 11 Hz plot of Figure 3. This positive correlation can be matched with Haα2 that states that arousal correlates with anterior/frontal alpha activity. This hypothesis is based on Aftanas et al. (2001), where an increase in anterior alpha in the range of 10–12 Hz was found. Unfortunately, Aftanas et al. do not mention whether there were hemispherical differences for this increase, and therefore we cannot fully confirm Haα2 . The delta range shows a clear increase over the posterior region for increasing arousal, which matches the effect described in Haδ , but seems to be located slightly more centrally. In the theta frequency range, Haθ 1 predicts an increase in posterior theta activity with increasing arousal. If any trend is visible, this is in the opposite direction, i.e. a theta decrease for higher arousal. Haθ1 cannot be rejected or confirmed. Global alpha decreases for higher arousal were also not found, so we have no evidence to support Haα1 . This could be due to the CAR we used, which cancels out global, phaselocked oscillations, or it could be the result of weakly induced arousal. Also for Haβ1 and Haγ 1 , we did not find any trend. Unfortunately, for dominance, we did not find any significant correlation with narrowband oscillations. However, we can denote some trends, that could serve as a hypothesis for future research. We found a correlation of midline (Fz, Cz) beta with high dominance (not shown in Figure 3), and in the alpha range, the left fronto-central regions showed increased power with higher dominance. This high-alpha asymmetry (around 13 Hz) shows a strong trend, but after Bonferroni correction for the 256 tests we performed in the alpha range, the effect is not significant. Interestingly, the alpha asymmetries found in the experimental condition seem to be composed of a low-alpha asymmetry related to valence, and a high-alpha asymmetry related to dominance. The spatial distributions of these asymmetries support both the observed link between dominance and the frontal-central alpha-asymmetry index, and the link between valence and the frontal asymmetry index presented (Table 4). For the PCs of the affective dimensions, we performed the same narrow-band analysis. We found a significant negative correlation between PC2 (superiority) and the right parietal power in the delta-band around 3 Hz. This correlation matches Haδ , that predicts an increase in posterior delta power for higher arousal. The delta increase is only visible in the 3 Hz band, and spreads over the full right hemisphere. While both its temporal and spatial distribution match the 3 Hz arousal plot, it is only significant when measured in relation to PC2, i.e. related to feelings of ‘superiority’. But indirectly, this supports hypothesis Haδ , as PC2 has high loadings for arousal. In the gamma band, we found a negative correlation of PC2 with right frontal power. However, these gamma correlations are spread over a large range of frequencies, and are therefore most likely the result of a muscle tension.

6

Discussion

The main aim of this work is to validate known neural correlates of emotion in an ecologically valid context, and to generate new hypotheses for correlates of emotional states. For many of the hypotheses that we derived from literature, no effects were found. One reason for not finding certain correlates could be that some correlates might be associated only with perceived or induced emotions and not with experienced emotions. Another reason is that the effects can be too weak to be reliably identified with relatively few subjects.

58


In this experiment, we confirmed Hvθ 2 , that associates right frontal theta activity with negative valence, Haα2 , that associates increasing arousal with frontal alpha power and Haδ , associating increasing arousal with posterior delta power. We found both a significant correlation of frontal alpha power with both valence and with dominance. The correlation of valence with alpha asymmetry supports the hypotheses Hvα1 and Hvα2 , but does not directly confirm either one. Perhaps there is a more complex relation among alpha asymmetry, valence and dominance: asymmetry in the lower alpha bands (8–10 Hz) correlates with valence, and asymmetry in the higher alpha bands (10–12 Hz) correlates with dominance. Without ratings for dominance, we would have concluded that alpha asymmetry is correlated with valence without noticing the interaction of alpha asymmetry with dominance. As most literature on emotional correlates isolates one specific emotional dimension, and attempts control for the others, we recommend the validation of the results on the controlled dimensions as well, as there appear to be dependences between ratings for valence, arousal and dominance. For the dominance dimension, we do not find significant narrow-band correlations due to the required multiple test correction. Some trends are visible, such as an increased medial beta-increase with high dominance, and the previously-mentioned alpha asymmetry in the high-alpha range over the left fronto-central region. These trends serve as a hypothesis for future research.

7

Conclusions

In the described study, we investigated the effect of naturally occurring emotions in the recorded EEG of players who were playing the Affective Pacman game. Our results indicate that frustration induction had a significant influence on the users’ reported emotions. Using these self-reported emotions, we confirmed correlates for both valence and arousal, in the theta, delta and alpha bands, respectively, during a natural activity like gaming. Interestingly, the different affective dimensions do not seem to be orthogonal. Specifically, the valence and dominance ratings are highly correlated, which can result in effects found in the EEGs that are related to one affective dimension, and attributed to the other. The alpha asymmetry is often used as a measure for valence. Our results indicate that a more careful interpretation is perhaps needed, but the value of the alpha asymmetry remains. In addition, right fronto-central theta power seems to be a good indicator of valence, and right frontal alpha power and the absence of right parietal delta power are indicators of arousal. These effects, and specifically, the stronger narrow-band effects, can be used to construct an automatic recogniser for affect. For future work, we suggest that a controlled study should be performed to find relations between the EEG and subjective feelings of dominance (also suggested by Demaree et al. (2005)), based on the hypotheses of high-alpha asymmetry, and medial beta increase. Furthermore, only the band power of narrow-band oscillations have been analysed. Properties of the phase (phase delay, phase coupling) of these oscillations could reveal different relations between the experienced emotions and the EEG signals.

Acknowledgements The authors thank Egon van den Broek, Dirk Heylen and Lynn Packwood for their valuable feedback, and gratefully acknowledge the support of the BrainGain Smart Mix Programme


59

of the Netherlands Ministry of Economic Affairs and the Netherlands Ministry of Education, Culture and Science.

References Aftanas, L.I. and Golocheikine, S.A. (2001) ‘Human anterior and frontal midline theta and lower alpha reflect emotionally positive state and internalized attention: high-resolution EEG investigation of meditation’, Neuroscience Letters, Vol. 310, No. 1, pp.57–60. Aftanas, L.I., Reva, N.V., Savotina, L. and Makhnev, V.P. (2006) ‘Neurophysiological correlates of induced discrete emotions in humans: an individually oriented analysis’, Neuroscience and Behavioral Physiology, Vol. 36, No. 2, pp.119–130. Aftanas, L.I., Reva, N.V., Varlamov, A.A., Pavlov, S.V. and Makhnev, V.P. (2004) ‘Analysis of evoked EEG synchronization and desynchronization in conditions of emotional activation in humans: temporal and topographic characteristics’, Neuroscience and Behavioral Physiology, Vol. 34, No. 8, pp.859–867. Aftanas, L.I., Varlamov, A.A., Pavlov, S.V., Makhnev, V.P. and Reva, N.V. (2001) ‘Affective picture processing: event-related synchronization within individually defined human theta band is modulated by valence dimension’, Neuroscience Letters, Vol. 303, No. 2, pp.115–118. Aftanas, L.I., Varlamov, A.A., Pavlov, S.V., Makhnev, V.P. and Reva, N.V. (2002) ‘Timedependent cortical asymmetries induced by emotional arousal: EEG analysis of event-related synchronization and desynchronization in individually defined frequency bands’, Int. J. Psychophysiology, Vol. 44, No. 1, pp.67–82. Allen, J., Coan, J.A. and Nazarian, M. (2004) ‘Issues and assumptions on the road from raw signals to metrics of frontal EEG asymmetry in emotion’, Biological Psychology, Vol. 67, Nos.1–2, pp.183–218. Barry, R.J., Clarke, A.R., Johnstone, S.J. and Brown, C.R. (2009) ‘EEG differences in children between eyes-closed and eyes-open resting conditions’, Clinical Neurophysiology, Vol. 120, No. 10, pp.1806–1811. Barry, R.J., Clarke, A.R., Johnstone, S.J., Magee, C.A. and Rushby, J.A. (2007) ‘EEG differences between eyes-closed and eyes-open resting conditions’, Clinical Neurophysiology, Vol. 118, No. 12, pp.2765–2773. Benovoy, M., Cooperstock, J.R. and Deitcher, J. (2008) ‘Biosignals analysis and its application in a performance setting’, Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, pp.253–258. Bradley, M.M. and Lang, P.J. (1994) ‘Measuring emotion: The self-assessment manikin and the semantic differential’, Journal of Behavior Therapy and Experimental Psychiatry, Vol. 25, No. 1, pp.49–59. Cacioppo, J.T., Petty, R.E., Losch, M.E. and Kim, H.S. (1986) ‘Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions’, Journal of Personality and Social Psychology, Vol. 50, No. 2, pp.260–268. Chanel, G., Kierkels, J.J.M., Soleymani, M. and Pun, T. (2009) ‘Short-term emotion assessment in a recall paradigm’, Int. J. Human–Computer Studies, Vol. 67, No. 8, pp.607–627. Chanel, G., Kronegg, J., Grandjean, D. and Pun, T. (2006) ‘Emotion assessment: arousal evaluation using EEG’s and peripheral physiological signals’, Multimedia Content Representation, Classification and Security, Vol. 4105 of Lecture Notes in Computer Science, Berlin/Heidelberg: Springer, pp.530–537.

60


Chanel, G., Rebetez, C., Bétrancourt, M. and Pun, T. (2008) ‘Boredom, engagement and anxiety as indicators for adaptation to difficulty in games’, MindTrek: Proceedings of the 12th International Conference on Entertainment and Media in the Ubiquitous Era, ACM, New York, NY, USA, pp.13–17. Codispoti, M., Bradley, M.M. and Lang, P.J. (2001) ‘Affective reactions to briefly presented pictures’, Psychophysiology, Vol. 38, No. 3, pp.474–478. Davidson, R. J. (1992) ‘Anterior cerebral asymmetry and the nature of emotion’, Brain and Cognition, Vol. 20, No. 1, pp.125–151. Demaree, H.A., Everhart, E.D., Youngstrom, E.A. and Harrison, D.W. (2005) ‘Brain lateralization of emotional processing: historical roots and a future incorporating “dominance’’’, Behavioral and Cognitive Neuroscience Reviews, Vol. 4, No. 1, pp.3–20. Diener, H. and Oertel, K. (2006) Experimental approach to affective interaction in games, Edutainment, Vol. 3942, pp.507–518. Frazier, T.W., Strauss, M.E. and Steinhauer, S.R. (2004) ‘Respiratory sinus arrhythmia as an index of emotional response in young adults’, Psychophysiology, Vol. 41, No. 1, pp.75–83. Heraz, A. and Frasson, C. (2007) ‘Predicting the three major dimensions of the learner’s emotions from brainwaves’, World Academy of Science, Engineering and Technology (WASET), Vol. 31, pp.323–329. Huster, R.J., Stevens, S., Gerlach, A.L. and Rist, F. (2009) ‘A spectralanalytic approach to emotional responses evoked through picture presentation’, Int. J. Psychophysiology, Vol. 72, No. 2, pp.212–216. Keil, A., Müller, M.M., Gruber, T., Wienbruch, C., Stolarova, M. and Elbert, T. (2001) ‘Effects of emotional arousal in the cerebral hemispheres: a study of oscillatory brain activity and event-related potentials’, Clinical Neurophysiology, Vol. 112, No. 11, pp.2057–2068. Khosrowabadi, R., Wahab, A., Ang, K.K. and Baniasad, M.H. (2009) ‘Affective computation on EEG correlates of emotion from musical and vocal stimuli’, IJCNN’09: Proceedings of the 2009 International Joint Conference on Neural Networks, pp.1590–1594. Kim, J. and André, E. (2008) ‘Emotion recognition based on physiological changes in music listening’, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 30, No. 12, pp.2067–2083. Klados, M.A., Frantzidis, C., Vivas, A.B., Papadelis, C., Lithari, C., Pappas, C. and Bamidis, P.D. (2009) ‘A framework combining delta event-related oscillations (EROs) and synchronisation effects (ERD/ERS) to study emotional processing’, Computational Intelligence and Neuroscience, Vol. 2009, Article ID 549419. Klein, J., Moon, Y. and Picard, R.W. (2002) ‘This computer responds to user frustration: theory, design, and results’, Interacting with Computers, Vol. 14, pp.119–140. Ko, K., Yang, H. and Sim, K. (2009) ‘Emotion recognition using EEG signals with relative power values and bayesian network’, Int. J. Control, Automation and Systems, Vol. 7, No. 5, pp.865–870. Krause, C., Viemeröc, V., Rosenqvista, A., Sillanmäkia, L. and Aström, T. (2000) ‘Relative electroencephalographic desynchronization and synchronization in humans to emotional film content: an analysis of the 4–6, 6–8, 8–10 and 10–12 Hz frequency bands’, Neuroscience Letters, Vol. 286, No. 1, pp.9–12. Kreibig, S.D., Wilhelm, F.H., Roth, W.T. and Gross, J.J. (2007) ‘Cardiovascular, electrodermal, and respiratory response patterns to fear- and sadness-inducing films’, Psychophysiology, Vol. 44, No. 5, pp.787–806. Lang, P.J., Greenwald, M.K., Bradley, M.M. and Hamm, A.O. (1993) ‘Looking at pictures: affective, facial, visceral, and behavioral reactions’, Psychophysiology, Vol. 30, No. 3, pp.261–273.


61

Lazar, N.A., Luna, B., Sweeney, J.A. and Eddy, W.F. (2002) ‘Combining brains: a survey of methods for statistical pooling of information’, NeuroImage, Vol. 16, pp.538–550. Lin, Y.P., Wang, C.H., Wu, T.L., Jeng, S.K. and Chen, J.H. (2009) ‘EEG-based emotion recognition in music listening: a comparison of schemes for multiclass support vector machine’, Proceedings of the IEEE International Conference on Acoustics Speech and Signal Processing, pp.489–492. Liu, C., Agrawal, P., Sarkar, N. and Chen, S. (2009) ‘Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback’, Int. J. Human-Computer Interaction, Vol. 25, No. 6, pp.506–529. Loughin, T.M. (2004) ‘A systematic comparison of methods for combining p-values from independent tests’, Computational Statistics and Data Analysis, Vol. 47, pp.467–485. Magnée, M.J., Stekelenburg, J.J., Kemner, C. and de Gelder, B. (2007) ‘Similar facial electromyographic responses to faces, voices, and body expressions’, Neuroreport, Vol. 18, No. 4, pp.369–372. Mandryk, R.L., Inkpen, K.M. and Calvert, T.W. (2006) ‘Using psychophysiological techniques to measure user experience with entertainment technologies’, Behaviour and Information Technology, Vol. 25, No. 2, pp.141–158. Marosi, E., Bazán, O., Yañez, G., Bernal, J., Fernández, T., Rodríguez, M., Silva, J. and Reyes, A. (2002) ‘Narrow-band spectral measurements of EEG during emotional tasks’, Int. J. Neuroscience, Vol. 112, No. 7, pp.871–891. Müller, M.M., Keil, A., Gruber, T. and Elbert, T. (1999) ‘Processing of affective pictures modulates right-hemispheric gamma band EEG activity’, Clinical Neurophysiology, Vol. 110, No. 11, pp.1913–1920. Murugappan, M., Rizon, M., Nagarajan, R., Yaacob, S., Hazry, D. and Zunaidi, I. (2008) ‘Timefrequency analysis of EEG signals for human emotion detection’, 4th Kuala Lumpur International Conference on Biomedical Engineering 2008, Vol. 21, pp.262–265. Pfurtscheller, G. (1999) ‘Event-related EEG/MEG synchronization and desynchronization: basic principles’, Clinical Neurophysiology, Vol. 110, No. 11, pp.1842–1857. Picard, R.W. (1997) Affective Computing. Cambridge, Massachusetts/London, England: The MIT Press. Picard, R.W., Vyzas, E. and Healey, J. (2001) ‘Toward machine emotional intelligence: analysis of affective physiological state’, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 10, pp.1175–1191. Rainville, P., Bechara, A., Naqvi, N. and Damasio, A.R. (2006) ‘Basic emotions are associated with distinct patterns of cardiorespiratory activity’, Int. J. Psychophysiology, Vol. 61, No. 1, pp.5–18. Reuderink, B., Nijholt, A. and Poel, M. (2009) ‘Affective Pacman: a frustrating game for braincomputer interface experiments’, Proceedings of the 3rd International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN 2009), Vol. 9 of Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp.221–227. Rizon, M., Murugappan, M., Nagarajan, R. and Yaacob, S. (2008) ‘Asymmetric ratio and FCM based salient channel selection for human emotion detection using EEG’, WSEAS Transactions on Signal Processing, Vol. 4, No. 10, pp.596–603. Russell, J.A. (1980) ‘A circumplex model of affect’, Journal of Personality and Social Psychology, Vol. 39, No. 6, pp.1161–1178. Salminen, M. and Ravaja, N. (2007) ‘Oscillatory brain responses evoked by video game events: the case of super monkey ball 2’, CyberPsychology and Behavior, Vol. 10, No. 3, pp.330–338. Salminen, M. and Ravaja, N. (2008) ‘Increased oscillatory theta activation evoked by violent digital game events’, Neuroscience Letters, Vol. 435, No. 1, pp.69–72.

62


Sammler, D., Grigutsch, M., Fritz, T. and Koelsch, S. (2007) ‘Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music’, Psychophysiology, Vol. 44, No. 2, pp.293–304. Sander, D., Grandjean, D. and Scherer, K.R. (2005) ‘A systems approach to appraisal mechanisms in emotion’, Neural Networks, Vol. 18, No. 4, pp.317–352. Scheirer, J., Fernandez, R., Klein, J. and Picard, R.W. (2002) ‘Frustrating the user on purpose: a step toward building an affective computer’, Interacting with Computers, Vol. 14, pp.93–118. Schlögl, A., Keinrath, C., Zimmermann, D., Scherer, R., Leeb, R. and Pfurtscheller, G. (2007) ‘A fully automated correction method of EOG artifacts in EEG recordings’, Clinical Neurophysiology, Vol. 118, pp.98–104. Schmidt, L.A. and Trainor, L.J. (2001) ‘Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions’, Cognition and Emotion, pp.487–500. Silberman, E. and Weingartner, H. (1986) ‘Hemispheric lateralization of functions related to emotion’, Brain and Cognition, Vol. 5, No. 3, pp.322–353. Spiridon, E. and Fairclough, S.H. (2009) ‘Detection of anger with and without control for affective computing systems’, Proceedings of the Conference of Affective Computing and Intelligent Interaction, pp.90–95. Takahashi, K. (2004) ‘Remarks on SVM-based emotion recognition from multi-modal biopotential signals’, Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication, RO-MAN 2004, pp.95–100. Tucker, D. M. (1981) ‘Lateral brain function, emotion, and conceptualization’, Psychological bulletin, Vol. 89, No. 1, pp.19–46. Zeng, Z., Pantic, M., Roisman, G.I. and Huang, T.S. (2007) ‘A survey of affect recognition methods: audio, visual and spontaneous expressions’, ICME’07: Proceedings of the 9th International Conference on Multimodal Interfaces, pp.126–133.