An Empirical Study of Machine Learning Techniques for Classifying ...

Master’s Thesis Computer Science September 2012

An Empirical Study of Machine Learning Techniques for Classifying Emotional States from EEG Data Ahmad Tauseef Sohaib & Shahnawaz Qureshi

School of Computing Blekinge Institute of Technology SE – 371 79 Karlskrona Sweden

This thesis is submitted to the School of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Computer Science. The thesis is equivalent to 2 x 20 weeks of full time studies.

Contact Information: Authors: Ahmad Tauseef Sohaib E-mail: [email protected] Shahnawaz Qureshi E-mail: [email protected]

University Advisors: Prof. Craig Lindley E-mail: [email protected]

Assist. Prof. Johan Hagelbäck E-mail: [email protected]

Game Systems and Interaction Research Laboratory (GSIL) School of Computing, Blekinge Institute of Technology Karlskrona, Sweden

School of Computing Blekinge Institute of Technology SE – 371 79 Karlskrona Sweden

Internet Phone Fax

: www.bth.se/com : +46 455 38 50 00 : +46 455 38 50 57 ii

ABSTRACT

With the great advancement in robot technology, smart human-robot interaction is considered to be the most wanted success by the researchers these days. If a robot can identify emotions and intentions of a human interacting with it, that would make robots more useful. Electroencephalography (EEG) is considered one effective way of recording emotions and motivations of a human using brain. Various machine learning techniques are used successfully to classify EEG data accurately. K-Nearest Neighbor, Bayesian Network, Artificial Neural Networks and Support Vector Machine are among the suitable machine learning techniques to classify EEG data. The aim of this thesis is to evaluate different machine learning techniques to classify EEG data associated with specific affective/emotional states. Different methods based on different signal processing techniques are studied to find a suitable method to process the EEG data. Various number of EEG data features are used to identify those which give best results for different classification techniques. Different methods are designed to format the dataset for EEG data. Formatted datasets are then evaluated on various machine learning techniques to find out which technique can accurately classify EEG data according to associated affective/emotional states. Research method includes conducting an experiment. The aim of the experiment was to find the various emotional states in subjects as they look on different pictures and record the EEG data. The obtained EEG data is processed, formatted and evaluated on various machine learning techniques to find out which technique can accurately classify EEG data according to associated affective/emotional states. The experiment confirms the choice of a technique for improving the accuracy of results. According to the results, Support Vector Machine is the first and Regression Tree is the second best to classify EEG data associated with specific affective/emotional states with accuracies up to 70.00% and 60.00% respectively. SVM is better in performance than RT. However, RT is famous for providing better accuracies for diverse EEG data.

Keywords: Human Robot Interaction (HRI), EEG Data Classification, Machine Learning Techniques

ACKNOWLEDGEMENT

At first, we wish to thank Almighty ALLAH who is the most beneficent and merciful for His blessings to let us complete our thesis work. Then we like to thank everyone who helped and advised us for our thesis work. We like to express our special thanks to our supervisors Prof. Dr. Craig Lindley and Assist. Prof. Dr. Johan Hagelbäck for their continuous assistance and support through our thesis work. Their kind comments and detailed feedback let us complete this study in an improved manner. We wish to praise the efforts of the members for Game Systems and Interaction Research Laboratory (GSIL) for their unconditional help in carrying out this study at each stage. We also like to appreciate the contribution of the participants who took their time out for the experiment and let us collect the data for our thesis work. We would not be able to complete our thesis work without it. At end, we like to thank our family and friends who have been supporting us morally to keep going with this study and inspired us to come up with expected results.

Sohaib & Shahnawaz

ii

CONTENTS

AN EMPIRICAL STUDY OF MACHINE LEARNING TECHNIQUES FOR CLASSIFYING EEG DATA IN HUMAN-ROBOT INTERACTION .............................. I   ABSTRACT ............................................................................................................................. I   ACKNOWLEDGEMENT .................................................................................................... II   CONTENTS .......................................................................................................................... III   LIST OF FIGURES ............................................................................................................. VI   LIST OF TABLES ............................................................................................................. VII   LIST OF GRAPHS ........................................................................................................... VIII   LIST OF ABBREVIATIONS .............................................................................................. IX   1  

INTRODUCTION .......................................................................................................... 1   1.1   BRAIN ....................................................................................................................... 2   1.1.1   Human Brain ........................................................................................................ 2   1.1.2   Structure of Brain ................................................................................................. 2   1.2   ELECTROENCEPHALOGRAPHY (EEG) ...................................................................... 3   1.2.1   Types of Signals .................................................................................................... 3   1.2.2   Delta Waves (δ) .................................................................................................... 4   1.2.3   Theta Waves (θ) .................................................................................................... 4   1.2.4   Alpha Waves (α) ................................................................................................... 4   1.2.5   Beta Waves (β) ..................................................................................................... 5   1.2.6   Gamma Waves (γ) ................................................................................................ 5   1.3   ACQUIREMENT OF EEG SIGNALS ............................................................................. 5   1.3.1   Artifacts ................................................................................................................ 6   1.3.2   10-20 System of Electrodes Placement ................................................................ 7   1.4   CLASSIFICATION OF EMOTIONS ................................................................................ 9   1.4.1   Emotions Recognition Algorithms ........................................................................ 9   1.5   MACHINE LEARNING TECHNIQUES TO CLASSIFY EEG DATA ............................... 10   1.5.1   K-Nearest Neighbor (KNN) ................................................................................ 10   1.5.2   Regression Tree (RT) ......................................................................................... 11   1.5.3   Bayesian Network (BNT) .................................................................................... 11   1.5.4   Support Vector Machine (SVM) ......................................................................... 12   1.5.5   Artificial Neural Networks (ANN) ...................................................................... 12   1.6   PSYCHOPHYSIOLOGICAL INTERACTION AND EMPATHIC COGNITION FOR HUMANROBOT COOPERATIVE WORK (PSYINTEC) ........................................................................ 13  

2  

BACKGROUND AND PROBLEM DEFINITION .................................................. 15   2.1   2.2   2.3  

PROBLEM FOCUSED ................................................................................................ 16   AIMS AND OBJECTIVES ........................................................................................... 16   RESEARCH QUESTIONS ........................................................................................... 16  

iii

2.4   3  

EXPECTED OUTCOMES ........................................................................................... 17  

RESEARCH METHODOLOGY ............................................................................... 18   3.1   CONSTRUCTIVE RESEARCH .................................................................................... 18   3.2   QUANTITATIVE RESEARCH..................................................................................... 19   3.3   APPLICATION OF METHODS.................................................................................... 19   3.3.1   Research Question 1........................................................................................... 19   3.3.2   Research Question 2........................................................................................... 19   3.3.3   Research Question 3........................................................................................... 19  

4  

THEORETICAL WORK ............................................................................................ 21   4.1   LITERATURE STUDY ............................................................................................... 21   4.1.1   BioSemi ActiveTwo System................................................................................. 21   4.1.2   EDF Browser...................................................................................................... 27   4.1.3   EEGLAB Toolbox for MATLAB ......................................................................... 28   4.1.4   Waikato Environment for Knowledge Analysis .................................................. 29  

5  

EMPIRICAL EVALUATION .................................................................................... 31   5.1   EXPERIMENT CONTEXT AND OPERATIONS ............................................................ 31   5.1.1   Experiment Planning .......................................................................................... 31   5.1.2   Subjects Demographics ...................................................................................... 32   5.1.3   Experiment Preparation ..................................................................................... 32   5.1.4   Experiment Design and Execution ..................................................................... 32   5.1.5   Experiment Limitation ........................................................................................ 33   5.2   THREATS TO VALIDITY........................................................................................... 33   5.2.1   Conclusion Validity ............................................................................................ 34   5.2.2   Internal Validity ................................................................................................. 34   5.2.3   Construct Validity .............................................................................................. 34   5.2.4   External Validity ................................................................................................ 34  

6  

DATA ANALYSIS AND INTERPRETATION ........................................................ 35   6.1   6.2   6.3   6.4   6.5  

7  

EEG DATA COLLECTION ........................................................................................ 35   EEG DATA SCREENING .......................................................................................... 35   EEG DATA PROCESSING ........................................................................................ 35   EEG DATA FORMATTING ....................................................................................... 36   EEG DATA CLASSIFICATION .................................................................................. 38  

DISCUSSION ............................................................................................................... 39   7.1   DISCUSSION FOR RESEARCH QUESTIONS ............................................................... 39   7.1.1   Research Question 1........................................................................................... 39   7.1.2   Research Question 2........................................................................................... 41   7.1.3   Research Question 3........................................................................................... 42  

8  

THE TOWER OF HANOI - PILOT STUDY ............................................................ 48   8.1   8.2   8.3   8.4   8.5   8.6  

9  

INTRODUCTION ....................................................................................................... 48   AIMS AND OBJECTIVES ........................................................................................... 49   EXPECTED OUTCOMES ........................................................................................... 49   RESEARCH OPERATIONS......................................................................................... 49   DISCUSSION ............................................................................................................ 50   SUMMARY AND CONCLUSION ................................................................................ 52  

SUMMARY AND CONCLUSION ............................................................................. 54   9.1   ANSWERS TO RESEARCH QUESTIONS..................................................................... 54   9.1.1   Research Question 1........................................................................................... 54   9.1.2   Research Question 2........................................................................................... 55  

iv

9.1.3   Research Question 3........................................................................................... 55   9.2   FUTURE WORK ....................................................................................................... 55   10   REFERENCES ............................................................................................................. 57   APPENDIX A: SELF-ASSESSMENT MANIKIN (SAM) AND INTERNATIONAL AFFECTIVE PICTURE SYSTEM (IAPS) ....................................................................... 62   APPENDIX B: QUESTIONNAIRE ................................................................................... 63  

v

LIST OF FIGURES

Figure 1.1 – Different Parts of Human Brain ........................................................................... 2   Figure 1.2 – An EEG Cap being used on a Participant to Record Brain Activity ................... 3   Figure 1.3 – EEG Signals ......................................................................................................... 3   Figure 1.4 – Delta Wave .......................................................................................................... 4   Figure 1.5 – Theta Wave .......................................................................................................... 4   Figure 1.6 – Alpha Wave ......................................................................................................... 5   Figure 1.7 – Beta Waves .......................................................................................................... 5   Figure 1.8 – Gamma Waves ..................................................................................................... 5   Figure 1.9 – Acquirement of EEG Signals using Headcap and Active-Electrodes ................. 6   Figure 1.10 – Schematic of the EEG Recording System ......................................................... 6   Figure 1.11 – Removal of Artifacts from EEG Signals ........................................................... 7   Figure 1.12 – 10-20 System of Electrodes Placement ............................................................. 8   Figure 1.13 – Location and Nomenclature of the Intermediate 10% Electrodes ..................... 8   Figure 1.14 – Emotion Labels in Arousal-Valence Dimension ............................................... 9   Figure 1.15 – Classification using KNN ................................................................................ 10   Figure 1.16 – Example of Regression Tree ............................................................................ 11   Figure 1.17 – Structure of BNT ............................................................................................. 12   Figure 1.18 – Maximum-Margin Hyperplane and Margins for an SVM ............................... 12   Figure 1.19 – An Artificial Neural Network .......................................................................... 13   Figure 1.19 – Functional Architecture of the PsyIntEC System ............................................ 13   Figure 3.1 – Constructive Research Diagram ........................................................................ 18   Figure 4.1 – BioSemi ActiveTwo System .............................................................................. 21   Figure 4.2 – Flat-Type Active-Electrodes .............................................................................. 22   Figure 4.3 – Pin-Type Active-Electrodes ............................................................................... 22   Figure 4.4 – Pin-Type Active-Electrodes ............................................................................... 23   Figure 4.5 – BioSemi Headcap with Electrode Holders and Active-Electrodes .................... 23   Figure 4.6 – Filling of Electrode Gel into Electrode Holders by Syringe .............................. 24   Figure 4.7 – Front of USB2 Receiver .................................................................................... 24   Figure 4.8 – Back of USB2 Receiver ..................................................................................... 25   Figure 4.9 – Analog Input Box (AIB) .................................................................................... 25   Figure 4.10 – 8 Touchproofs for EOG, EMG, ECG .............................................................. 25   Figure 4.11 – ActiveTwo AD-box with Battery .................................................................... 26   Figure 4.12 – ActiView BioSemi Acquisition Software ........................................................ 26   Figure 4.13 – EEG Data Acquisition using ActiView ........................................................... 27   Figure 4.14 – EDF Browser ................................................................................................... 27   Figure 4.15 – Manipulating EEG Signals and their Annotations ........................................... 28   Figure 4.16 – EEGLAB Toolbox ........................................................................................... 28   Figure 4.17 – Waikato GUI Chooser ..................................................................................... 29   Figure 4.18 – Weka Explorer ................................................................................................. 29   Figure 5.1 – Experiment Stages ............................................................................................. 31   Figure 6.1 – EEG Data Collection ......................................................................................... 35   Figure 6.2 – EEG Data Screening .......................................................................................... 35   Figure 6.3 – EEG Data Processing ......................................................................................... 36   Figure 6.4 – EEG Data Formatting ........................................................................................ 38   Figure 6.5 – EEG Data Classification .................................................................................... 38   Figure 8.1 – The Tower of Hanoi Puzzle ............................................................................... 48   Figure 8.2 – Research Operations .......................................................................................... 50   vi

LIST OF TABLES

Table 1.1 – Frequency and Mental States of Waves ................................................................ 4   Table 6.1 – Formatting Dataset for Different Emotions in Subject 1 using Model A ........... 37   Table 6.2 – Formatting Dataset for Different Emotions in Subject 1 using Model B ........... 37   Table 7.1 – Contrast of Selected Machine Learning Techniques .......................................... 41   Table 7.2 – Datasets and their Description while Passing to WEKA .................................... 42   Table 7.3 – Classification Accuracies by Selected Techniques for Dataset A ...................... 42   Table 7.4 – Classification Accuracies by Selected Techniques for Dataset B ....................... 43   Table 7.5 – Comparison of Model A and Model B Based on Findings ................................. 45   Table 7.6 – Division of Dataset B .......................................................................................... 45   Table 7.7 – Datasets and their Description while Passing to WEKA .................................... 45   Table 7.8 – Classification Accuracies by Selected Techniques for Dataset 1B, 2B and 3B.. 45   Table 7.9 – Passing Datasets to WEKA for Each Subject Individually ................................. 46   Table 7.10 – Classification Accuracies by Selected Techniques for Subject 1, 2 and 3........ 46   Table 8.1 – Dataset and its Description while Passing to WEKA ......................................... 50   Table 8.2 – Classification Accuracies by Selected Techniques for Dataset TB .................... 50   Table 8.3 – Comparison of Classification Accuracies by Selected Techniques .................... 51   Table 9.1 – Classification Accuracies by Selected Techniques ............................................. 55  

vii

LIST OF GRAPHS

Graph 7.1 – Contrast of Selected Machine Learning Techniques.......................................... 40   Graph 7.2 – Classification Accuracies by Selected Techniques for Dataset A...................... 43   Graph 7.3 – Classification Accuracies by Selected Techniques for Dataset B ...................... 43   Graph 7.4 – Comparison of Results Obtained Due to Dataset Formatting ............................ 44   Graph 7.5 – Classification Accuracies by Selected Techniques for Dataset 1B, 2B and 3B . 46   Graph 7.6 – Classification Accuracies by Selected Techniques for Subject 1, 2 and 3 ......... 47   Graph 8.1 – Classification Accuracies by Selected Techniques for Dataset TB ................... 51   Graph 8.2 – Comparison of Classification Accuracies by Selected Techniques ................... 52  

viii

LIST OF ABBREVIATIONS

AIB ANN ARFF BNT BDF EKG EEG EMG ECHORD GSIL HRI HCI ICA IAPS KNN LabVIEW PsyIntEC RT SAM SVM ToH WEKA

Analog Input Box Artificial Neural Networks Attribute-Relation File Format Bayesian Network BioSemi Data Format Electrocardiography Electroencephalography Electromyography European Clearing House for Open Robotics Development Game Systems and Interaction Research Laboratory Human-Robot Interaction Human-Computer interaction Independent Component Analysis International Affective Picture System K-Nearest Neighbor Laboratory Virtual Instrument Engineering Workbench Psychophysiological Interaction and Empathic Cognition for Human-Robot Cooperative Work Regression Tree Self-Assessment Manikin Support Vector Machine The Tower of Hanoi Waikato Environment for Knowledge Analysis

ix

1

INTRODUCTION

Human-computer interaction (HCI) is the study of interaction among people and computers. HCI aims to produce a design that should produce a good fit among the user, the machine and the required services in order to achieve a certain performance in terms of both quality and optimality of the services [1]. HCI is one of the most active research areas in computer science presently [2]. Modern types of human-driven and human-centric interaction with digital media have opened new possibilities to modernize the different areas of human life such as learning, working etc. Human-computer interface applications have increased the significance of emotion recognition as emotions are vital in daily life of humans [3] [4]. Human-robot interaction (HRI), a subfield of HCI, is of increasing importance as people and robots perform various works together [5]. It is now becoming feasible to integrate this technology into real-world, real-time systems to enhance human-machine interaction across a wide range of application domains including clinical, industrial, military and gaming applications [6] [7] [8] [9]. HRI is one of the significant fields under robots community. To achieve robust interaction of robots with human, robots must have proficient components of HRI [2]. It is vital for a service robot to deduce the co-worker for his/her needs in order to achieve efficient results. Electroencephalography (EEG) is one effective way of recording and analyzing the brain activity due to its ease of use, affordable cost and fine resolution. Due to brain activity neurons get fired producing higher electrical potential which is recorded by the electrodes attached to the scalp of a human being. The measures differ due to varying levels of cognitive stimuli [10]. EEG is beneficial due to high temporal resolution which helps to record variations in cognitive activity based on millisecond gauge. Hence, EEG measurements are insight of the cognitive situations of the participant [11]. However, EEG is sensitive to noise from electrical equipment, mostly useable in controlled lab environments and not so effective in real-world situations. In recent years, there has been great advancement in robot technology which has introduced different fields of applications such as space, war field and assistance in work etc. However, current robots are not completely dependable or autonomous to perform a certain tasks especially when it comes to a robot working with a human co-worker. Hence there is a need of robotic systems that are capable of understanding human emotions, desires and intentions which will ultimately enhance the performance of certain tasks. This research has been motivated by these needs and is an attempt to take a step in this direction. Psychophysiological Interaction and Empathic Cognition for Human-Robot Cooperative Work (PsyIntEC) is “a feasibility demonstration project targeting advances that address safe ergonomic and empathetic adaptation by a robotic system to the needs and characteristics of a human co-worker during collaborative work in a joint human-robot work cell” [12]. The human co-worker is the source of psychophysiological or biometric data that is input to a robot control system to provide the basis for affective and cognitive modeling of the human by the robot as a basis for behavioral adaption [12]. Different techniques are needed for the classification of psychophysiological data, i.e., taking signals from a human, including EEG and creating interpretations of these in terms of emotional and attention states. This involves acquiring EEG data and processing it, formatting the dataset and evaluating it using different technical approaches of machine learning. The results of the evaluation are presented which can be used to develop the

1

classifier system for the PsyIntEC project. The developed classifier will be part of the robot software.

1.1

Brain

To help understanding the further discussion by the readers, basic structure and function of human brain are explained under this section.

1.1.1

Human Brain

Brain is the most complex organ of human. All kind of physical tasks are led by it. It contains a neural network of 10 billion nerve cells, neurons. It handles feelings, hunger, thirst, body movements and sleep functions of human. It controls almost all the core activities required for the survival of a human. It communicates with body parts environment by sending and receiving signals. It is the core of central nervous system having brainstem, spinal cord and large brain as described in the Figure 1.1. Spinal cord and large brain are connected through brainstem. It is divided into three different parts based on its anatomy and functionality.

Figure 1.1 – Different Parts of Human Brain [13]

1.1.2

Structure of Brain

Based on anatomy, brain is divided into three parts as hind brain, mid brain and forebrain. At first part, myelencephalon exists in hind brain whereas cerebellum and fourth ventricle are located above it with spinal cord. At second part, mesencephalon exits in mid brain and contains tectum, tegmentum and cerebral aqueduct. At third part, diencephalon and telecephalon exits in fore brain [13]. Based on functionality, brain is divided into three parts. First part is called as forebrain; also known as cerebrum or large brain and handles high level mental tasks such as computational thinking etc. Second part is called as brainstem and controls the visual functions. Third part is called as cerebellum and is responsible for body movements.

2

1.2

Electroencephalography (EEG)

Electroencephalography (EEG) is a signal representation of brain activity. The signal waves hold the valuable information about the state of brain. It is one of the non-invasive techniques for brain imaging which provides electrical potential recording for the surface of the scalp due to the electrical activity of the large collections of neurons in the brain [14]. Non-invasive is a technique in which the body is not invaded or cut open as during surgical investigations or therapeutic surgery. Invasive technique is opposite to non-invasive.

Figure 1.2 – An EEG Cap being used on a Participant to Record Brain Activity [15]

1.2.1

Types of Signals

EEG signals are defined in term of rhythmic and transient and are complex signals as shown in Figure 1.3. The rhythmic activity is distributed into different frequency bands. Different people of different ages may have different amplitude and frequency of EEG signals while they are recorded in different states such as performing a task or relaxing.

Figure 1.3 – EEG Signals [16] Based on frequency ranges, five types of waves can be identified. They are alpha (α), theta (θ), beta, (β), delta (δ) and gamma (γ) from low to high frequency respectively. A specific wave is mostly available in specific lobe of cerebral cortex however this is not always true. Different mental states are associated with different waves which is helpful to define one’s situation at a specific time as described in the Table 1: Wave Delta (δ)

Frequency (Hz) 0–4

Mental State Deep Sleep

3

Theta (θ) Alpha (α) Beta (β) Gamma (γ)

4–8 8 – 13 13 – 30 > 30

Drifting Thoughts, Dreams, Creativity Calmness, Relaxation, Abstract Thinking Highly Focused, Highly Alertness Simultaneous Process, Multi-Tasking

Table 1.1 – Frequency and Mental States of Waves

1.2.2

Delta Waves (δ)

Delta waves are under the frequency range of 0 – 4 Hz. Mental states associated with these waves are deep sleep, coma or hypnosis and sometimes awake. In awake state, it is always considered to be pathological phenomenon. The higher is the amplitude, higher serious is the problem considered. These waves are decreased by the age and are normally present in healthy people in their awake state.

Figure 1.4 – Delta Wave [16]

1.2.3

Theta Waves (θ)

Theta waves are under the frequency range of 4 – 8 Hz. Mental states associated with these waves are drifting thoughts, creative thinking and unconscious materials. These waves appear in central, temporal and parietal parts of head. These waves are normally present in healthy people while they are in deep sleep.

Figure 1.5 – Theta Wave [16]

1.2.4

Alpha Waves (α)

Alpha waves are under the frequency range of 8 – 13 Hz. Mental states associated with these waves are relaxed and calm states. These waves appear on back side of head and occipital area of brain. These waves are of high amplitude as compared to others. This can be observed while subject is awake and clam. Sometimes, these waves interfere with µ-rhythm. These waves are normally present in people while they are calm and relax being in awake state.

4

Figure 1.6 – Alpha Wave [16]

1.2.5

Beta Waves (β)

Beta waves are under the frequency range of 13 – 30 Hz. Mental states associated with these waves are highly focused and alertness, such as during deep thinking and concentration. Beta waves are having large band of frequency as compared to others. These waves appear at front side of head and central area of brain.

Figure 1.7 – Beta Waves [16]

1.2.6

Gamma Waves (γ)

Gamma waves are under the frequency range of 30 Hz. Mental states associated with these are simultaneous work and multi-tasking. These waves are hard to notice due to their very low amplitude. These waves appear in each part of brain.

Figure 1.8 – Gamma Waves [16]

1.3

Acquirement of EEG Signals

EEG signals are acquired from sculp. Signals are measured using electrodes stick to head. Calculation of potential difference between two electrodes is the basic principle of EEG. One or more electrodes are attached either to mastoids or ear lobes. They are called reference electrodes. These electrodes help to find the background electric field of skull. The location of reference electrodes is very important. They should not be located neither very close to brain nor at any other part of body as signals can possibly be affected by the electrical activity of muscles or heart [17].

5

Figure 1.9 – Acquirement of EEG Signals using Headcap and Active-Electrodes [15] Different EEG devices are available with certain number of electrodes and filters for them. These devices help to acquire analog EEG signals which are then altered to digital data and sampling frequency. Filters help to remove the artifacts. Unwanted frequencies are ignored using low pass and high pass filters by removing signals such as Electromyography (EMG) and Electrocardiography (EKG) [18]. The data resolution is also important during data processing. Hence, sampling frequency, sampling rate and number of electrodes are important factors while recording EEG signals.

Figure 1.10 – Schematic of the EEG Recording System [19] Different electrodes acquire EEG data using different methods. Following electrodes are normally used to record EEG data: • • • • •

1.3.1

Disposable Electrodes (Gelled/Pre-Gelled) Reusable Electrodes Electrodes Caps (Headbands) Saline-Based Electrodes Needle Electrodes

Artifacts

Artifacts are unwanted signals due to noise from; for example electric circuits. These are not due to brain activity rather affecting the signal measurement making it difficult for analysis.

6

There are different types of artifacts. One of the main artifacts is due to impedance of system and another is sampling frequency artifact which is 50 Hz and 60 Hz caused due to ground loop. The importance of artifacts and their removal can be explained better by the Figure 1.11 as:

Figure 1.11 – Removal of Artifacts from EEG Signals [19] (A) The raw EEG signal having large artifacts. (B) The averaged imaging artifact. (C) The result of subtracting the averaged imaging artifact in B from the EEG in A, followed by down-sampling and showing Pulse artifact. (D) The averaged pulse artifact from trace C (not to scale). (E) Result of subtracting the averaged pulse artifact in D from the EEG in C. (F) The EEG from the same subject, recorded outside the scanner, i.e., free of imaging and pulse artifact. The character of this EEG appears to match closely the artifact corrected trace in E. However, some of the artifacts are useful. Biological signals such as EMG and EKG can help to predict different mental states. Such as, EMG artifact which is due to eye blinking can provide information about sleep or awake states.

1.3.2

10-20 System of Electrodes Placement

One of the commonly used methods of electrodes placement is 10-20 System for recording EEG signals which is standardized by the American Electroencephalographic Society. Using this system, a total of 21 electrodes are placed on scalp as shown in A of Figure 1.9. Location of placement for electrodes is as: Nasion: Electrodes placed at level of eyes and top of nose are Reference Points, Nasion. Inion: Electrodes placed on midline at backside of head and base of skull are Inion. Parameters of skull are measured from above points. Locations for electrodes are determined by division of parameters to intervals of 10% and 20%. Three electrodes are placed on middle of adjacent points as shown in B of Figure 1.9 [20]. Relationship between the location of an electrode and cerebral cortex is the basis of 10-20 system. Electrode letters are used to determine the placements such as: A – Ear Lobe

7

C – Central Lobe F – Frontal Lobe Fp = Frontal Polar O – Occipital Lobe P – Parietal Lobe Pg = Nasopharyngeal T – Temporal Lobe A combination of letter(s) and an integer number is used to determine the placement.

Figure 1.12 – 10-20 System of Electrodes Placement [20] Seen from (A) Left and (B) Above the Head

Figure 1.13 – Location and Nomenclature of the Intermediate 10% Electrodes [20] In addition to 10-20 system, there are many other systems available to record EEG signals. The Queen Square system of electrode placement was proposed as standard for recording electric potentials on the scalp [20]. EEG measurement can also be carried out using bipolar

8

or unipolar electrodes. In case of bipolar, potential different among pair of electrodes are measured whereas in case of unipolar, average of all electrodes is compared with potential of every electrode [20].

1.4

Classification of Emotions

Various systems for classification of emotions exist. This classification can be observed in two aspects as dimensional and discrete [21]. According to Plutchik, there are eight basic states of emotion as acceptance, anger, anticipation, disgust, fear, joy, sadness and surprise. Rest of the emotion states can be model using the basic states such as sadness and surprise make disappointment [22]. Considering dimensional aspects, the commonly used classification system is bipolar model, proposed by Russells [23] which considers arousal and valence dimensions. In this case, valance dimensions are from negative to positive whereas arousal dimensions are from not aroused to excited. The dimensional model is advantageous for emotion recognition because it can determine discrete emotions in its space even if no certain label can be used to determine a specific feeling [21] [24]. The dimensional model is the most commonly used model for classification of emotions [23].

Figure 1.14 – Emotion Labels in Arousal-Valence Dimension [25] (Circumplex Model of Russell)

1.4.1

Emotions Recognition Algorithms

There are many algorithms available for emotions recognition. There is much ongoing research on EEG-based emotion recognition algorithms. Different methods are employed to extract features and classify data into different emotion modes such as joy, sadness, anger and pleasure etc. However, none of the algorithm resulted in good accuracy [26].

9

Emotion recognition is an immature field without standards availability for different EEG signals for emotions. Limited types of emotions are identified until yet and emotion algorithms lack to classify them accurately [26].

1.5

Machine Learning Techniques to Classify EEG Data

Machine Learning is a branch of artificial intelligence and aims to identify unknown samples by learning from known sample. There are different machine learning techniques available with their own advantages and disadvantages for use. Currently, Artificial Neural Network, Genetic Algorithm and Support Vector Machine are the most commonly used to classify EEG data [27]. Various machine learning techniques are used to classify EEG data and some of them are successful but lacks in accuracy. However, the following machine learning techniques are considered in most of empirical studies: • • • • •

K-Nearest Neighbor (KNN) Regression Tree (RT) Bayesian Network (BNT) Support Vector Machine (SVM) Artificial Neural Networks (ANN)

The first four techniques are the most popular techniques [5] and are most widely used.

1.5.1

K-Nearest Neighbor (KNN)

K-Nearest Neighbor (KNN) is one of the most basic and simple classification techniques [28]. KNN is usually considered when there is no or very little knowledge available for the distribution of data. It is a potential non-parametric classification technique which completely bypasses the problem of probability densities [29]. While classifying using KNN, X is assigned with a label of most frequently represented between K-nearest samples. This determines that labels on KNN are examined; voting has been carried out and a decision is made. KNN classification technique was developed for the need to carry out discriminant analysis while reliable parametric estimates of probability densities are either unknown or difficult to find out [28].

Figure 1.15 – Classification using KNN [30]

10

Figure 1.15 shows classification performed using KNN; ‘.’ denotes the prototypes of the same class as y, and ‘x’ denotes the prototypes of the class different from y [30] .

1.5.2

Regression Tree (RT)

Regression Tree (RT) takes an object or situation as input which is categorized a number of properties and provides a decision as an output [31]. One input feature is represented by every node and possible test result values are represented by the branches of node. The positive test values can either be positive or negative. If that node is reached, leaf nodes, also called as terminal nodes; represent the decision value.

Figure 1.16 – Example of Regression Tree [32] Regression tree is widely used in medical field for classification such as speech recognition, heart attack and cancer diagnosis [33] [34].

1.5.3

Bayesian Network (BNT)

Bayesian Network (BNT) is the most effective classifier in terms of predictive performance and state-of-the-art classifiers [35]. BNT is a graph that contains a network structure N which put a set of conditional independence relations between a set of variables C = {A1, A2 … An} and a set of Tables T of local probability distributions associated with every variable. The joint probability distribution of A is determined by N and T. Network nodes have one-

11

to-one contact with A variables. Therefore in N, single nodes are denoted by Ai whereas parent nodes are indicated by Pi.

Figure 1.17 – Structure of BNT [35]

1.5.4

Support Vector Machine (SVM)

Support Vector Machine (SVM) is a linear machine that operates in k-dimensional space created due to n-dimensional input data X into k-dimensional space using non-linear mapping ȹ(X). This helps to isolate data normally by geometry and linear algebra. To find a linear classifier for any data points with known class labels, it can be done by identifying a separating hyper plane.

Figure 1.18 – Maximum-Margin Hyperplane and Margins for an SVM Trained with Samples from Two Classes [36]. Samples on the margin are called the Support Vectors. SVM was proposed by Vapnik [36] and can deal with over fitting by having misclassified instances on training data. Due to this, SVM is more appropriate for affect recognition because physiology data is noisy [5].

1.5.5

Artificial Neural Networks (ANN)

Artificial Neural Networks (ANN) has the capability of finding a nonlinear transformation of the pattern in order to classify more accurately [37].

12

Figure 1.19 – An Artificial Neural Network [37] Artificial neurons are interconnected in a neural network and process information with connected neurons. ANN molds itself and changes its structure based on the internal and external information of network while learning.

1.6

Psychophysiological Interaction and Empathic Cognition for Human-Robot Cooperative Work (PsyIntEC)

Psychophysiological Interaction and Empathic Cognition for Human-Robot Cooperative Work (PsyIntEC) is “a feasibility demonstration project targeting advances that address safe ergonomic and empathetic adaptation by a robotic system to the needs and characteristics of a human co-worker during collaborative work in a joint human-robot work cell” [12]. The human co-worker is the source of psychophysiological or biometric data that is input to a robot control system to provide the basis for affective and cognitive modeling of the human by the robot as a basis for behavioral adaption [12].

Figure 1.19 – Functional Architecture of the PsyIntEC System [12]

13

PsyIntEC extends the European Clearing House for Open Robotics Development (ECHORD) state of human-robot co-worker and hyper-flexible cells with focus on robot hands and complex management in cooperative human-robot tasks [12]. The industry targeted is a small medium enterprise involved in prototyping of novel devices. PsyIntEC is aimed to focus on demonstration of robots feasibility to guide, support and facilitate the production of human-robot co-worker prototype. It also focuses on human emotions to assess its intentions in order to proceed the required task accordingly. This is helpful to maintain optimum level of attention among robot and human co-worker by using biometric data to find human emotional states [12]. PsyIntEC is an ongoing EU funded research project at Blekinge Institute of Technology. It was required to develop a classifier system for PsyIntEC. The developed classifier will be part of the robot software. This study has evaluated different technical approaches of machine learning and presented the results of evaluation to help the selection of most suitable technique for the classifier system for PsyIntEC.

14

2

BACKGROUND AND PROBLEM DEFINITION

In last few years, robots have been developed to work together with humans in various environments. For example, entertainment is provided by the humanoid robots while patient assistance is provided by the haptic robots. However, it is hard to evaluate and interpret the interaction among human and robot [38]. Human interaction with computer applications is part of everyday life. Similarly, emotions are vital in everyday life of humans. Hence there is a growing need of emotion recognition to help human interacting with computer interfaces quickly and easily. Researchers have made successful emotion recognition using text, speech and facial expressions or gesture [26]. Emotions accompany everyone in the daily life, playing a key role in non-verbal communication, and they are essential to the understanding of human behavior. Emotion recognition could be done from the text, speech, facial expression or gesture. However, most of the researches have been carried out on emotion recognition from EEG [3] [39] [40] [41] [42] [4]. Human machine interaction on the base of physiological signals has been investigated by much past and recent research. Of particular interest are systems that can make interpretations about psychological states based upon physiological data. Linear classifiers [43][44][45] are considered to be the most appropriate classification technology due to their simplicity, speed and interpretability. However, non-linear classifiers are considered to be the most appropriate when it comes to signal features and cognitive state according to [46] [47]. Sequential Floating Forward Search and Fisher Projection methods are used by Picard and colleagues to classify eight emotions with 81% accuracy [48]. Lisetti and Noasoz used Marquardt Back propagation, Discriminant Function Analysis and K-Nearest Neighbor to distinguish between six emotions and acquired the correct classification in 83%, 74% and 71% [49]. According to [50], probabilistic models can be developed using a methodology provided which uses various body expressions of the user, personality of user and context of the interaction. Mental workload has been evaluated using Artificial Neural Networks providing mean classification accuracies of 85%, 82% and 86% for the baseline, low task difficulty and high task difficulty states respectively [51]. In [43] an emotion-recognizer based on Support Vector Machines has been analyzed which provided accuracies of 78.4% and 61.8%, 41.7% for recognition of three, four and five emotions categories respectively. According to [5], if the same physiological data is used then Support Vector Machines with a classification accuracy of 85.81% perform the best, closely followed by the Regression Tree at 83.5%, K-Nearest Neighbor at 75.16% and Bayesian Network at 74.03%. Performance of K-Nearest Neighbor and Bayesian Network algorithms can be improved using informative features [5]. According to [5], Support Vector Machine shows 33.3% and 25% accuracy for three and four emotion categories respectively when it comes to physiological signal databases acquired from ten to hundreds of users. Due to different experiment environment, data and pre-processing techniques used in different studies, it is not easy to compare results concluded by each study. However, studies show that various factors such as pre-processing and classification techniques can strongly affect the results and improve their accuracy. This means, a certain tradeoff can lead to desired results however a strong decision is required to make the best one. However several methods are used successfully to develop affect recognizers from physiological indices; it is still required to select an appropriate method for the classification 15

of EEG data to attain uniformity in various aspects of emotion selection, data collection, data processing, feature extraction, base lining, and data formatting procedures. Looking at the current research, it has been found that, various methods are successfully applied to develop different classifiers for the correct classification of physiological data. However, the classifiers are generally not available and there is always both room for improvement and the need to develop classifiers suitable for the needs of specific applications. In this case, for the PsyIntEC project, we needed to choose an appropriate method to classify EEG data. This involved taking EEG data, processing it to extract features and formatting the dataset to evaluate the dataset using different technical approaches of machine learning. The results of the evaluation can be used to develop the classifier system for PsyIntEC project. The developed classifier will be part of the robot software.

2.1

Problem Focused

The study focuses to find a suitable technique for classifying EEG data. To deal with the problem focused, standard methods are used for different steps involved such as recording EEG data, processing it to extract features and formatting dataset. Different parameters have been modified in order to achieve better results. The formatted dataset is then evaluated for different machine learning techniques to achieve the desired results.

2.2

Aims and Objectives

The main aim of this research work was to evaluate different machine learning techniques to classify EEG data associated with affective/emotional states along with the followings: • • • •

2.3

Find a suitable method to process the EEG data based on different signal processing techniques. Identify key features that can represent emotional states better than raw EEG data. Find a method to shape the dataset for EEG data to classify it. Identify and evaluate most commonly practiced techniques used to classify EEG data associated with specific affective/emotional states.

Research Questions

In contribution to this research work, we have developed the following research questions based on the aims and objectives: RQ1: Which techniques are available in current literature for classifying EEG data associated with specific affective/emotional states? RQ2: What EEG data features give the best results for different classification techniques? RQ3: How accurately can EEG data be classified affective/emotional states using the selected techniques?

according

to

associated

16

2.4

Expected Outcomes

Expected outcome of this research work was identification of the most appropriate machine learning technique to classify EEG data according to associated affective/emotional states. This included: • • • •

List of most commonly practices techniques used to classify EEG data. List of EEG data features which helps to classify EEG data more accurately. Analysis of challenges and issues based on software and hardware used. Identification of the most appropriate technique for the correct and most accurate classification of EEG data.

Discussion and conclusion about how the proposed technique can be used to classify EEG data associated with specific affective/emotional states.

17

3

RESEARCH METHODOLOGY

3.1

Constructive Research

Constructive Research is a common research methodology of computer science. As compared to other types of research, this one is not required to be validated quite as empirically as others [52]. Using this methodology, a system is developed and then evaluated. The aim is to construct artifacts and knowledge for them using practical potential value [53]. According to Crnkovic [54], constructive research method aims to construct an artifact (practical, theoretical or both) which solves a domain specific problem to build knowledge about how the problem can be solved (understood or explained) and provides with results in relevant to practical and theoretical. “Construct” as a term means a new constructed contribution. It can be a new algorithm, technique, framework or theory. From the case study by Lukka [55], constructive research can be considered as a form of conducting case research parallel to ethnography, grounded theory, theory illustration, theory testing and action research.

Figure 3.1 – Constructive Research Diagram [52] In constructive research, information is collected from various sources such as articles, literature, tutorials etc. Such sources help to acquire theoretical knowledge. Using this theoretical knowledge, solution to a problem is derived. This derived solution derives new knowledge as explained in the Figure 3.1. Our research work involved recording EEG signals using EEG headcap and BioSemi ActiveTwo recording device. The artifacts/innovative constructs developed and used in this setup are electrodes and EEG interface. Target knowledge of setup is human emotions as positive/negative arousal/valence. Hardware and software components used to acquire the assistance in completion of required tasks from the setup. Hardware and software

18

components have their complete documentation available and that is summarized in coming chapter.

3.2

Quantitative Research

In order to evaluate the validity of the developed construct, we have conducted an experiment. The theory obtained by constructive research was verified by quantitative methods. A standard experiment design was used to conduct experiment and collect the data from it. Collected data was then processed to extract features; format the dataset then evaluated using different machine learning techniques in order to test the theory. During the whole process, different factors affecting the process were identified and improvements are determined. This has motivated for the use of quantitative research [56].

3.3

Application of Methods

This section clarifies the research operations and how the application of selected methods helped to answer all the research questions.

3.3.1

Research Question 1

Which techniques are available in current literature for classifying EEG data associated with specific affective/emotional states? Literature has been studied in order to find the answer to this question. Research on classification of EEG data is explored. The literature collected was filtered in order to have research which work with classification of data associated with specific affective/emotional states. The filtered literature has been used to extract techniques used in them. While extracting the techniques, researches using quantitative methods were preferred keeping the validity constraints in consideration.

3.3.2

Research Question 2

What EEG data features give the best results for different classification techniques? Literature has been studied to acquire the answer to this question. Research based on EEG data are being searched specially those which involved recording of EEG data and processing it later. The collected literature was filtered to look into the research which worked with either 4 or 6 emotions. This has been considered in order to narrow down the problem domain and achieve better results. The features reporting best results for the selected techniques were extracted from the filtered literature. Different parameters used for those features and classification techniques applied on them were also extracted.

3.3.3

Research Question 3

How accurately can EEG data be classified according to associated affective/emotional states using the selected techniques? An experiment has been conducted in order to get the answer to this question. The experiment confirms the choice of a technique and selection of EEG data features for improving the accuracy of results. Hence, experiment actually validates the answer of both

19

RQ1 and RQ2; and answers RQ3. The experiment involved steps for planning, preparation, execution and reporting of results. Data collection was performed during the experiment using the device, BioSemi ActiveTwo System [57] and software, ActiView [57]. Data analysis was performed using the software, EDF Browser [58] and EEGLAB Toolbox for MATLAB [59]. EEG data classification is performed using the software, Waikato Environment for Knowledge Analysis (WEKA). Software components used during the experiment and analysis are described in the next chapter.

20

4

THEORETICAL WORK

4.1

Literature Study

4.1.1

BioSemi ActiveTwo System

BioSemi ActiveTwo System is a widely used system to achieve advanced EEG data and uses an active-electrode technology with open architecture [60]. The system is an improved version of ActiveOne and purely designed for research purposes. It features multi-channel and supports high resolution measurements for bio-potentials. With the help of activeelectrode technology, requirements like impedance measurement, skin preparation and faraday cage are not required to be used [57].

Figure 4.1 – BioSemi ActiveTwo System [61] Pin-Type Active Electrodes and Headcap on Head of Subject ActiveTwo AD-box with Battery on Back of Subject and Connected to Computer The engineers Robert Honsbeek, Ton Kuiper and physicist Coen Metting van Rijn have introduced BioSemi system for researches in 1998 at Medical Physics department of the University of Amsterdam. Different projects such as AGN1667, AGN3416 and AGN4098, funded by Technology Foundation STW [62]; are using BioSemi system as well as various researches are published using it. Foundation Vision Research Amsterdam [63] has tested various small prototypes using this system and made results available to more than 40 scientific and clinical workers in 9 different countries. BioSemi ActiveTwo System used in the experiment involved the following components: • • •

BioSemi Active-Electrodes BioSemi Headcaps ActiveTwo AD-box

21

• • 4.1.1.1

USB2 Receiver ActiView BioSemi Active-Electrodes

BioSemi active-electrode is a low output impedance sensor that discards any coupling of sources or interferences and artifacts caused by any medium. It helps to lower the level of noise to thermal noise level due to electrode material based on sintered Ag-AgCl [57].

Figure 4.2 – Flat-Type Active-Electrodes [57] It is alcohol and water resistant. It has an input protection circuit that shields electronic amplifier from static discharge and defibrillator pulses. It can fulfill the needs for EEG, Electrocardiography (ECG) and Electromyography (EMG). BioSemi active-electrodes are either flat-type active-electrodes or pin-type active-electrodes [57].

Figure 4.3 – Pin-Type Active-Electrodes [57] BioSemi pin-type active-electrode is specialized to be used with BioSemi headcap. It is capable of fitting into headcap having BioSemi electrode holders. It has sintered Ag-AgCl electrode on tip that provides low noise and offset voltages along with stable DC performance [57]. It is alcohol and water resistant. These electrodes are labeled with water resistant numbers to identify each channel. Standard set of these have 32 electrodes and 140 centimeter of cable length with a common connector. These electrodes hold fast application time [57].

22

Figure 4.4 – Pin-Type Active-Electrodes [57] 4.1.1.2

BioSemi Headcaps

BioSemi headcap is a plain elastic cap with plastic electrode holders and without any electrodes or wires attached. The electrode holders allow easy and fast placement of electrodes into them; making it 30 minutes for a 128 channel EEG headcap to be ready for EEG measurement. No skin preparation is required to use this headcap because high impedances do not influence the signal quality due to active-electrode principal [57]. Standard BioSemi headcaps are available in different sizes and have ear-slits for easy adjustment of headcaps over the ears area.

Figure 4.5 – BioSemi Headcap with Electrode Holders and Active-Electrodes into them [61]

23

While setting up the headcap for EEG measurement, the headcap is sited at the head of subject; electrode gel is filled into electrode holders using a syringe and active-electrodes are inserted into the electrode holders.

Figure 4.6 – Filling of Electrode Gel into Electrode Holders by Syringe [61] Dr. Peter Praamstra has developed the BioSemi headcap at Behavioral Brain Science Center, University of Birmingham, United Kingdom [57]. 4.1.1.3

USB2 Receiver

USB2 receiver takes the optical data from the AD-box and converts it to USB2 output. It has a trigger port with 16 independent triggers input and outputs. This keeps the subject separate from the stimulation setup. The triggered output signals are manipulated using BioSemi acquisition software, ActiView.

Figure 4.7 – Front of USB2 Receiver [61]

24

USB2 receiver is a plug and play device with complete reliable data throughput. A 32 channels analog input box with 24 bits ActiveTwo system having 256 channels at 4096 kHz has a total data throughput of 3.54 megabyte per second [57]. Using USB2 receiver, ActiveTwo system can be used with either desktops or notebooks. LED indicator with USB2 receiver helps to identify the incoming data.

Figure 4.8 – Back of USB2 Receiver [61]

Figure 4.9 – Analog Input Box (AIB) [61] 4.1.1.4

ActiveTwo AD-Box

ActiveTwo AD-box is the front-end of BioSemi system. It is ultra-compact, low power and can digitize 256 sensor-signals at 24 bits resolution. These sensors can be of various types such as active electrodes or bufferboxes with normal passive electrodes etc.

Figure 4.10 – 8 Touchproofs for EOG, EMG, ECG [57] 25

Every AD-box channel contains a low noise DC coupled post amplifier, first order antialiasing filter, steep fifth order sinc response and output of 24 bits high resolution. Digital output from all AD converters are digitally multiplexed and sent uncompressed and unreduced data is sent to computer through a single optical fiber [57].

Figure 4.11 – ActiveTwo AD-box with Battery [57] 4.1.1.5

ActiView

ActiView is a free open source program written in Laboratory Virtual Instrument Engineering Workbench (LabVIEW) [64]. It is BioSemi acquisition software that shows all ActiveTwo channels over the screen and allows saving data over the disk in BioSemi Data File (BDF) format. EEG/ECG/EMG signals can be acquired using ActiView. Data from extra sensors such as additional sensors connected to AD-box, AnalogInputBox (AIB) and digital triggers through USB2 receiver can also be acquired using ActiView [57].

Figure 4.12 – ActiView BioSemi Acquisition Software [61] 26

Layout of ActiView is designed to provide user with easy and quick check of data quality. ActiView have various options to select, filter and reference data for down sampling. It allows completely stable and reliable acquisition of data using single buffered method. Being an open source program, it allows to molds it according to any requirements making it a versatile program [57].

Figure 4.13 – EEG Data Acquisition using ActiView [61]

4.1.2

EDF Browser

EDF browser is an open source, free viewer and toolbox for EEG data files [58]. EDF is an abbreviation of European Data Format which is a file format for storage of multichannel biological and physical signals. It was developed by a few medical engineers which later become de-facto standard for EEG and PSG recording in commercial equipment and multicenter research projects [65].

Figure 4.14 – EDF Browser [58]

27

EDF browser allows analyzing and manipulating BDF files containing EEG data. It has various options such as converting one file format to another, annotation/events manipulation, header editing, heart rate detection, data reduction and cropping, down sampling signals, combining multiple BDF files to one, averaging using triggers, events or annotations etc.

Figure 4.15 – Manipulating EEG Signals and their Annotations [58]

4.1.3

EEGLAB Toolbox for MATLAB

EEGLAB is a toolbox and graphical user interface that runs under MATLAB (The Mathworks, Inc.) [66]. It can process collection of EEG data for any number of channels. It support various functions of EEG data processing such as importing channel and event information, visualization of data (plus multi-trial ERP-image plots, scalp map and dipole model plotting, scrolling), data pre-processing (including filtering, epoch selection, averaging and artifact rejection), independent component analysis (ICA) and time/frequency decompositions including channel and component cross-coherence supported by bootstrap statistical methods based on data resampling [59].

Figure 4.16 – EEGLAB Toolbox [67]

28

Three layers are designed to divide the functionalities of EEGLAB. First layer lets the user interact with data using graphical interface without using any syntax of MATLAB. User can also mold the operation of MATLAB according to memory available. Data processing, command history and interactive pop functions can be performed under middle layer. EEGLAB data structures and individual signal processing function can be used to write custom scripts by experienced users of MATLAB. Plug-in feature of EEGLAB allows easy integration of any modules to its main menu. EEGLAB is open source; available under GNU public license for noncommercial use and development, and can be downloaded for free from web1.

4.1.4

Waikato Environment for Knowledge Analysis

Waikato Environment for Knowledge Analysis (WEKA) [68] is used to apply classification algorithms and is most suited for developing new machine learning schemes. WEKA is a wide collection of machine learning algorithms and methods for data pre-processing with graphical user interface for manipulating data and comparison of various machine learning techniques for a problem [69].

Figure 4.17 – Waikato GUI Chooser

Figure 4.18 – Weka Explorer 1

http://www.sccn.ucsd.edu/eeglab/

29

As shown in the figure 4.17, WEKA has four graphical user interfaces as Explorer, Experimenter, KnowledgeFlow and Simple CLI. Explorer is the main interface of WEKA which allows inputting data for pre-processing and their results. However, the rest of interfaces are used for making new methods and combinations of methods for classification or clustering. The basic purpose of WEKA is to help user to extract useful information from data and determine an appropriate algorithm for producing accurate prediction model using it. Weka is written in Java and developed by University of Waikato, New Zealand [68]. WEKA is a free software and available under the GNU General Public License.

30

Booking Equipment and

Setting up Experiment Guide Preparation for Subjects and Equipment for Experiment

Planning Place, Arranging Subjects for Experiment

Arranging Equipment Using Design Execution Standards and Design Instructions

Conducting Experiment following Instructions and Recording Data

5

EMPIRICAL EVALUATION

5.1

Experiment Context and Operations

The aim of the experiment was to find the various emotional states in subjects as they look on different pictures that are inducing strong emotions in most people. International Affective Picture System (IAPS) [70] are used for this purpose which is a general picture database and especially designed for experiments conducted for researches. During the experiment, EEG data was recorded in order to process, format and evaluate against different machine learning techniques for the classification of EEG data. The experiment helped to confirm the selection of right features and technique, i.e., that providing the most accurate results. The experiment involved planning, preparation, execution and reporting the results as explained in Figure 5.1:

Figure 5.1 – Experiment Stages

5.1.1

Experiment Planning

Planning was the initial stage of experiment. In this stage, we arranged the equipment for the experiment and the subjects who helped us to perform this experiment. The equipment for experiment was arranged through the Game Systems and Interaction Research Laboratory (GSIL) [61] available at Blekinge Institute of Technology. The subjects were students. To reduce the risk factor of resource unavailability, we had a backup plan for the experiment by having more than one pre-booking of the equipment and more number of subjects than required. Hence, 30 subjects were invited, 20 out of them participated in the experiment and EEG data for them was recorded. The following equipment was involved in the experiment and booked under GSIL: •

• • •

BioSemi ActiveTwo System o BioSemi Pin-Type Active-Electrodes o BioSemi Headcaps o USB2 Receiver o ActiveTwo AD-box o Two Batteries for AD-box Electrode Gel Syringe Lab Computer (High Specification)

31

5.1.2

Subjects Demographics

A total of 20 subjects (15 men and 5 women) participated in the experiment. All the subjects were students of Blekinge Institute of Technology and aged from 21 to 35 years. All the subjects were having different cultural background, nationalities and field of studies. None of subjects were professionals.

5.1.3

Experiment Preparation

At this stage, we prepared the equipment for the experiment and invited the subjects. A chart of observations was prepared that we used to note down issues during the experiment. This let us identify the limitations and possible improvements in the experiment. An experiment guide for the subjects was also prepared to help them understanding the experiment and performing it with ease. Equipment was prepared for the experiment following the steps as below: • • • • • •

Lab computer was powered on with ActiView running on it. Lab computer was connected to USB2 receiver though a USB cable. USB2 receiver was connected to AD-box using optical fiber. Pin-type active-electrodes were connected through their connector to AD-box. Battery was connected to AD-box AD-box was powered on.

Each subject was prepared for the experiment following the steps as below: • • • •

5.1.4

Subject was asked to sit over the chair calmly in front of lab computer. Suitable BioSemi headcap was chosen to place on the head of subject and adjusted well to fit over the head. Electrode holders and pin-type active-electrodes were chosen according to the experiment design. Electrode gel was filled into electrode holders using syringe and pin-type activeelectrodes were inserted into the electrode holders.

Experiment Design and Execution

The asymmetry among left and right brain hemispheres are the major area where the emotion signals can be captured [71]. According to a model developed by Davidson et al. [72], two core dimensions i.e., arousal and valence are related to asymmetric behavior of emotions. A judgment about a state as positive or negative lies under valence whereas area among calmness, excitement, expressing the level of excitation lies under arousal. Davidson et al [10] captured the EEG signals from left and right frontal, central, anterior temporal and parietal regions (F3, F4, C3, C4, T3, T4, P3, P4 positions according to the 10-20 system [73] and referenced to Cz) in order to distinguish the happy and hatred emotions. Based on these findings, the experiment was executed with the instructions as [72] [74]: • • •

An appropriate interface was applied for the automated projection of the IAPS [70], emotion-related pictures. To compensate opening/closing of eyes, 30 seconds gap was maintained before starting the experiment. IAPS, 30 pictures (6 pictures for each emotion cluster as neutral, positive arousing/calm, negative arousing/calm) were displayed randomly for the duration of

32

•

•

•

5 seconds with a gap of a black screen between 5-12 seconds (the purpose of the black screen duration was to reset the emotional state of subjects offering them the time to relax having no emotional content) and a cross shape projection was displayed for 3 seconds to attract the sight of subject. This process was repeated for each picture. A subject may feel an emotion which differs from the one expected. To know this, the subject was asked to rate his/her emotion on a Self-Assessment Manikin (SAM) [70]. Each subject rated their level of emotion on a 2D arousal and valance scale. Two recording sessions for 25 to 35 trials having 5 pictures, displaying each picture for 2.5 seconds were completed. During the whole process, subjects were directed to stay quiet and still (to realize and observe the emotion instead of mimic the facial expression) without eye blink as much as possible to get rid of other artifacts (e.g., facial muscles) Fp1, Fp2, C3, C4, F3, and F4 positions were used to attain the EEG signals according to 10-20 system and all of the electrodes were referenced to Cz.

The IAPS (description of 30 pictures used) and SAM (required to be filled by the subjects) used in the experiment are available in the Appendix A section. During the experiment, EEG data for each subject was recorded and stored separately for processing, formatting and evaluating. The experiment last for 4 days having 5 subjects each day. Each subject took almost 20 minutes individually to complete the experiment.

5.1.5

Experiment Limitation

One of the tasks of experiment was to fill the SAM by each subject. It was very important to fill it exactly with what subject had felt while looking at each picture. A subject may either forget to write exact or change it which may lead to invalid data selection because SAM is used to filter the subjects and their EEG data in order to have valuable data and reject the rest. A subject reporting false information to SAM can be possibly due to bad memory of subject or afraid to report the right information. As some pictures from IAPS include nudity and different subjects from different cultures think it is not good to talk or report about this. Hence, even if a subject felt aroused after looking at a nude picture, he or she may not report it thinking this may not be considered good due to his or her cultural background. Similarly, a few subjects were afraid to see some pictures having blood however they may not report it thinking this may not be considered good as he or she may be considered as weak. To cope with these limitations, we already selected sharp subjects from different cultures, keeping the data obtained from them as anonymous and also taken their views after the experiment in order to know if there was any possibility of this limitation so that we may filter out those subjects completely.

5.2

Threats to Validity

When data for research is obtained and analyzed to report results, there is always a vital question about their validity. Research results must be valid and generalized to the population from which the sample is taken [75]. Threats to validity must be taken in consideration in order to extend the validity of research. For this research, the following threats to validity are considered.

33

5.2.1

Conclusion Validity

The making of correct conclusion among the results and operations of research belongs to this validity. To avoid poor experiment design, the proven standards are followed at different stages of experiment such as 10-20 system, IAPS, SAM and instructions of experiment etc. Instructions of experiment were reviewed several times by the colleagues keeping the reliability measure in consideration. Same dataset and attributes are used for all algorithms in order to avoid fishing bias which occurs looking for a certain outcome. The subjects participated in the experiment were equally heterogeneous as they were students from different fields of study and experience. Taking in consideration the random irrelevance, the experiment was performed in a controlled environment and we have screened the subjects using SAM and chosen the data for those who fulfilled the aim of experiment.

5.2.2

Internal Validity

The observation that either change found are derived from the operation or from other causes belongs to this validity [37]. During the experiment, subjects keep looking at the IAPS pictures displaying one after another and the duration of displaying each picture was fair to avoid boredom. After the experiment, SAM was required to be filled by the each subject. To keep this interesting, there were pictures to represent the rating and IAPS pictures were also shown again for the ease of subjects. Later, using SAM subjects were screened keeping the appropriate sample representation of the whole population.

5.2.3

Construct Validity

The relation among the research results and idea behind the research belongs to this validity [37]. Threats related to experiment design and social factors are taken in consideration for this. In the experiment design, the subjects chosen represent the both male and female representation to prevent the biasness of one sex. To avoid evaluation apprehension, each subject was given the complete instruction about the whole experiment and given the assurance to keep their data anonymous. The possible hypothesis guessing threat is also taken in consideration while planning the experiment design carefully.

5.2.4

External Validity

The level of research to be applied generally belongs to this validity [37]. The subjects were screened based on SAM to choose the valuable data for which we aimed. However this screening reduced the number of subjects but we still got the fairly enough data from each subject due to the duration of experiment. Furthermore, as all the subjects were from different fields and cultures, we had diverse representation of the population we wanted to generalize. As the experiment was executed in a controlled experiment, the affects and motion were controlled for each subject through the instructions given to them.

34

BioSemi ActiveTwo System

BioSemi ActiveTwo System

EEG Data EEG Data Recording and Recording and Storing Storing

EEG Data Screening using SAM

6

DATA ANALYSIS AND INTERPRETATION

6.1

EEG Data Collection

During the experiment, the EEG data was recorded using BioSemi ActiveTwo System with sampling rate 2048Hz and stored to lab computer in BioSemi Data Format (BDF) [76] using ActiView BioSemi acquisition software. High sampling rate was chosen to cover the maximum amount of data for each signal to reduce the risk of losing any valuable data. Data for each subject was stored separately in order to analysis every subject individually as well. This process can be summarized as shown in the Figure 6.1 below:

Figure 6.1 – EEG Data Collection

6.2

EEG Data Screening

The subjects were screened to select EEG data for analysis and processing. The screening was based on SAM; subjects with low valance and arousal rating were rejected. The reason for screening is to select the most valuable data and remove the rest to get reliable results from the obtained data. The screening left 15 subjects out of 20. Screening was further applied to EEG data of 15 subjects to select the signal duration which fulfill the aimed emotion based on SAM. The idea behind this was to screen out and separate the data for each emotion. Such as, signal for positive arousal were screened from the rest of emotions and so on. EDF Browser was used to reduce the signals individually for the required duration. While reducing the signals, the first and last seconds from the total duration of five seconds were eliminated in order to narrow down to exactly required data. The reason for this step was to focus on valuable data and filtering out extra because when a picture is displayed, it takes some time for the brain to react to new stimuli and therefore the first seconds are usually removed; similarly, after looking at the picture for a while, the brain goes into relax state, does not react in the same stimuli as before and therefore the second seconds are also removed. This process was completed for pictures with positive, negative and neutral arousal as well as for positive, negative and neutral valance. This process can be summarized as shown in the Figure 6.2 below:

Figure 6.2 – EEG Data Screening

6.3

EEG Data Processing

The screened data was pre-processed using EEGLAB Toolbox for MATLAB. Epoch and Event info were extracted from the data. Data was pruned and baseline was removed from it. 35

BioSemi ActiveTwo System

EEG Data Recording and Storing

EEGFeature Data ICA, Screening Selectionusing and SAM Extraction

Finally, Independent Component Analysis (ICA) [77] was performed on the data. Preprocessing data with these various techniques helps to remove the artifacts such as eye blinking etc. This also make easier to extract features from the signals. The features represent the state of the subject by which they went while looking at the pictures during the experiment. We have focused the features based on emotions as positive or negative situation, calmness to excitement, excitation and happiness. The pre-processed data was further processed to get the real values for the signals using EEGLAB Toolbox for MATLAB. Having real value for signals make it easier to analyze and classify them. The following features were extracted from the real values of each signal based on findings from [74] in order to process the data further: • • • •

Minimum Value Maximum Value Mean Value Standard Deviation

Values for the above features were obtained using the formulas of MATLAB. This process can be summarized as shown in the Figure 6.3 below:

Figure 6.3 – EEG Data Processing

6.4

EEG Data Formatting

The values obtained were formatted in Attribute-Relation File Format (ARFF) [68], which is an acceptable file format for WEKA. During this formatting, the attributes were declared for all the features of each signal. For example, first signal (Fp1) has its all features mentioned as attributes in the ARFF file. The values obtained make the data instances for the ARFF file with class values as negative and positive arousal/valance. Two models were designed to format the instances as below: Model A: S1 F1 F2 F3 F4 ClassValue … … … S6 F1 F2 F3 F4 ClassValue Where S1 is the signal one and F1, F2, F3, F4 are the features extracted for this signal and so on.

36

Formatting instances using model A resulted in more number of instances which is usually considered good for the classification as the system can learn more diversity and provide better accuracy. However, system may be slow at the same time and can take more time in classification. Model B: S1F1 S1F2 S1F3 S1F4 . . . S6F1 S6F2 S6F3 S6F4 ClassValue Where S1 is the signal one and F1, F2, F3, F4 are the features extracted for this signal and so on. Formatting instances using model B resulted in less number of instances which is usually not considered good for the classification as the system can learn less diversity and at risk to provide better accuracy. However, system is faster at the same time and can take less time in classification. For every emotion, two datasets were formatted in two ARFF files using each model for each subject. Hence two datasets were obtained for each emotion such as negative and positive arousal/valance for each subject. This can be explained better by the Table 6.1 and 6.2 below: Emotion Emotion 1 Emotion 2 Emotion 3 Emotion 4

Model Model A Model A Model A Model A

Subject Subject 1 Subject 1 Subject 1 Subject 1

Dataset Dataset 1A1 Dataset 2A1 Dataset 3A1 Dataset 4A1

Table 6.1 – Formatting Dataset for Different Emotions in Subject 1 using Model A Emotion Emotion 1 Emotion 2 Emotion 3 Emotion 4

Model Model B Model B Model B Model B

Subject Subject 1 Subject 1 Subject 1 Subject 1

Dataset Dataset 1B1 Dataset 2B1 Dataset 3B1 Dataset 4B1

Table 6.2 – Formatting Dataset for Different Emotions in Subject 1 using Model B Tables 6.1 and 6.2 shows the dataset formatting for different emotions in subject 1 using model A and B respectively. The same procedure was applied to other subjects in order to format the datasets for different emotions using model A and B. In this way, datasets for each subject were obtained for every individual emotion. Datasets with model A for subject 1 were combined to have all emotions in one dataset as: Dataset 1A1 + Dataset 2A1 + Dataset 3A1 + Dataset 4A1 = Dataset A1 Similarly, datasets with model B for subject 1 were combined to have all emotions in one dataset as: Dataset 1B1 + Dataset 2B1 + Dataset 3B1 + Dataset 4B1 = Dataset B1

37

BioSemi Dataset Analysis ActiveTwo and System Classification

EEG Data Dataset Recording and Modeling and Storing Formulation

EEGFeature Data ICA, Screening Selectionusing and SAM Extraction

The same procedure was repeated for all subjects to format their datasets. All of the datasets obtained were combined into one in order to have one dataset for all of the subjects for each model separately as: Dataset A1 + Dataset A2 + . . . + Dataset A14 + Dataset A15 = Dataset A Dataset A1 + Dataset A2 + . . . + Dataset A14 + Dataset A15 = Dataset B The basic aim of having more number of datasets in this way was to analyze the EEG data for each subject individually and combined for every model separately in order to find the possible differences among them. This process can be summarized as shown in the Figure 6.4 below:

Figure 6.4 – EEG Data Formatting

6.5

EEG Data Classification

Each dataset having EEG data in ARFF file format was classified using machine learning techniques available in WEKA. During the classification, the classifier is trained to classify negative or positive arousal/valence values as correctly classified whereas neutral values as incorrectly classified. The techniques used had all the default parameter values as implemented in WEKA except for cross validation. This process can be summarized as shown in the Figure 6.5 below:

Figure 6.5 – EEG Data Classification

38

7

DISCUSSION

The era for most advancement in human-robot interaction is considered to be from year 2000 to 2011. According to survey, World Robotics 2004 [78], the robots will soon become part of everyday life. At the end of 2003, there were around 60,000 domestic robots in use and about 4 million expected in 2007 [78]. The literature from 2000 to 2011 is studied to find the techniques available to classify EEG data. Various machine learning techniques are applied to classify EEG data by the researchers. However different techniques have different challenges with their pros and cons. The results of literature are compiled to identify and select the techniques that are used in empirical studies. The challenges for using selected techniques are also determined. The selected techniques are evaluated through the experiment to find their validity and accuracy they provide to classify EEG data.

7.1

Discussion for Research Questions

This section holds the discussion for all the research questions based on the findings of theoretical and empirical studies.

7.1.1

Research Question 1

Which techniques are available in current literature for classifying EEG data associated with specific affective/emotional states? In previous related work, various machine learning techniques are available to classify EEG data but the following machine learning techniques are considered in most of empirical studies for EEG data associated with specific affective/emotional states: • • • • •

K-Nearest Neighbor (KNN) Regression Tree (RT) Bayesian Network (BNT) Support Vector Machine (SVM) Artificial Neural Networks (ANN)

The authors of [5] found the KNN as one of the most widely used technique for classifying EEG data associated with specific affective/emotional states. According to [28], KNN for its nature of dealing with distribution of data with very little knowledge or no knowledge works better with EEG data. KNN ability to deal with discriminant analysis with parametric estimates of unknown or difficult probability densities are another reason of considering KNN better for EEG data [28]. As mentioned by [29], KNN being a potential non-parametric classification technique which completely bypasses the problem of probability densities can be suitable for classifying EEG data. According to experiment results available in [79], KNN helps to reach higher accuracy for classifying EEG data. RT is considered in most of the studies for classifying EEG data, claimed in [5]. According to [33], RT is largely used in medical fields including for classifying EEG data. Authors of [34] also mention the wide use of RT for classifying EEG data. Discussion in [31] shows that large scale use of RT is because of its nature of working which is suitable for medical related data such as EEG data.

39

BNT is considered to be the most effective technique for classifying EEG data as explained under [35]. The wide use of BNT for classifying EEG data is also mentioned in [5]. According to authors [80], BNT is more suitable for classifying EEG data for various emotional states. SVM is used for classifying EEG data in most of studies available and considered to be the most suitable. In [5], authors strongly support SVM and recommend it for classifying EEG data with better accuracy. Similarly, authors of [27] claim that SVM is a commonly used technique for classifying EEG data and more appropriate. It has been found that most of the authors think that SVM works better to distinguish the feeling of joy, sadness, anger and pleasure [26]. According to experiment results available in [79][81], SVM provides effective and promising results for classifying EEG data. Study described in [82] claims that SVM can show a high level of agreement on EEG data classification. ANN is recommended for classifying EEG data by [37]. Authors of [27] also claim that ANN is most commonly used techniques to classify EEG data due to its nature to deal with such data more efficiently. Study in [37] explains that if small number of electrodes are used then ANN provide higher accuracy results. Apart from the above discussed machine learning techniques, there are various other techniques available. However, not all of them works well when it comes to EEG data. The basic reason to this is because there are certain challenges faced while working with EEG data for its classification which cannot be copped by all of the techniques. Some of the other techniques, such as ZeroR used by authors [74] with 10-folds cross validation and default parameters values of WEKA provided the accuracy of only 10%. Lisetti and Nasoz used the Discriminant Function Analysis and Marquardt Backpropagation algorithms for six emotions and achieved the correct classification accuracies as 74% and 83% [49]. However, most of the authors have not preferred using these algorithms; because of the data pre-processing and extraction of certain features required for these. Selection of certain features from EEG data reflects different accuracies. Hence it is important to use those algorithms which can work better with the selected features. As compared to other machine learning techniques, the selected five techniques are found to be used in most of empirical studies and considered to be the most suitable for the classification of EEG data associated with specific affective/emotional states based on the accuracies achieved using them. A contrast of these techniques (when used to classify EEG data associated with specific affective/emotional states) found through the literature study is summarized in the Graph 7.1 with their accuracies and explained in Table 7.1:

Graph 7.1 – Contrast of Selected Machine Learning Techniques Techniques K-Nearest Neighbor (KNN)

No. of Emotions

Authors

6

F. Nasoz [49]

40

Regression Tree (RT)

4

P. Rani [5]

Bayesian Network (BNT)

4

M. Macas [80]

Support Vector Machine (SVM)

4

Yi-Hsin Yu [79]

Artificial Neural Networks (ANN)

3

C.Conati [50]

Table 7.1 – Contrast of Selected Machine Learning Techniques From the findings of literature study and accuracies reported by various authors, it is clear that the selected algorithms are used in the most of empirical studies and considered to be the most suitable for the classification of EEG data associated with specific affective/emotional states.

7.1.2

Research Question 2

What EEG data features give the best results for different classification techniques? Features selection and their extraction from raw EEG data are vital because the signals are in bulk space and it is not possible to analyze them in that way. During the features extraction process, the most important question is which one to be chosen to extract and leave? It has been noticed that the number of features and their type reflects on the results affectively [74]. Feature selection is one of the key challenges in affective computing due to phenomena of person stereotype [5]. This is because different individuals express same emotion with different characteristic response patterns for a same situation. Each subject involved in the experiment was having diverse physiological indices that showed high correlation with each affective state. The same finding has been observed in [27] and explained in [83]. From the obtained EEG data, it was observed that physiological features were highly correlated with the state of arousal among two subjects. According to [83], a feature can be considered significant and selected as an input to classifier if absolute correlation is greater for physiological features among subjects. Based on these findings, it was observed that accuracy was improved among different techniques when highly correlated features were used. However, the accuracy was degraded for some techniques. Such as KNN, BNT and ANN were improved in accuracy however RT and SVM were degraded. Selection of highly correlated features helps to exclude the less important features for affective state and hence improve the results [27]. Considering the above outcomes and statistical features proposed by [84], the following features were selected to represent the EEG signals: • • • •

Minimum Value Maximum Value Mean Value Standard Deviation

The raw EEG data is then processed to extract the selected features and real value for them. Different signal processing techniques are available for this purpose such as Fourier transform, wavelet transform, thresholding, and peak detection [5]. There are some tools available for the same purpose as well such as EEGLAB Toolbox for MATLAB. The obtained real values are then formatted into an ARFF file which is acceptable by the classification software such as WEKA.

41

Selecting a certain number of features is important for a complete EEG signal representation. However, the selected features should be highly correlated in order to achieve better accuracy results [5] [27] [83]. Similarly, while extracting the selected features from the raw EEG data, it is important to fetch the valuable data while rejecting the artifacts. Finally, formatting the extracted features can be tricky. A selected technique may classify the EEG data with better accuracy but may not be efficient at the same time and vice versa. A moderate approach to get rid of this is to choose a better model for data formatting in order to have a technique which is efficient in both time and space [5]. Identifying the EEG data feature in order to achieve better results for classifying EEG data using different machine learning techniques also depends on the number of emotions taken in consideration. A clear diversity in results was observed when the number of emotions is different even if the same technique is applied for classification of EEG data. Not only this, different dataset for each subjects shows a diverse accuracy results which means the diversity is just not limited to the number of emotions. Considering above findings, a wise decision must be made in a way that the extracted features should be the whole representation of raw EEG data, highly correlated and not too much in number in order to achieve a better accuracy.

7.1.3

Research Question 3

How accurately can EEG data be classified according to associated affective/emotional states using the selected techniques? All datasets compiled for all subjects were passed to classification software WEKA one after another for their offline analysis. The selected techniques KNN, RT, BNT, SVM and ANN were used for classification in WEKA. Each dataset was trained using each selected technique to classify negative or positive arousal/valence values as correctly classified and neutral values as incorrectly classified. The techniques used had all the default parameter values as implemented in WEKA and 10-fold cross validation (standard value for cross validation). Initially, the datasets having combination of datasets from all subjects were passed to WEKA. The sequence and description by which datasets were passed to WEKA are available in Table 7.2: Dataset Dataset A Dataset B

Description KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation

Table 7.2 – Datasets and their Description while Passing to WEKA As shown in Table 7.2, two datasets A and B were passed to WEKA. Both datasets were formatted using different models (as explained in previous chapter) in order to analyze the EEG data in various ways and to find the possible differences among them. The classification accuracies by selected techniques for Dataset A are shown in the Table 7.3 and Graph 7.2: Techniques Accuracy K-Nearest Neighbor KNN) 39.88% Regression Tree (RT) 50.00% Bayesian Network (BNT) 32.74% Support Vector Machine (SVM) 39.28% Artificial Neural Networks (ANN) 38.69% Table 7.3 – Classification Accuracies by Selected Techniques for Dataset A 42

Graph 7.2 – Classification Accuracies by Selected Techniques for Dataset A From Table 7.3 and Graph 7.2, it is cleared that RT provides the best accuracy among all others. However, KNN lies second, ANN third, SVM fourth and BNT last to RT. Model A can be considered a good model for formatting EEG data for classification due to the fact that more number of instances in a dataset are considered good for classification because the classifier can learn better due to diverse cases made available to it. The classification accuracies by selected techniques for Dataset B are shown in the Table 7.4 and Graph 7.3: Techniques K-Nearest Neighbor Regression Tree Bayesian Network Support Vector Machine Artificial Neural Networks

Accuracy 52.44% 52.44% 52.44% 56.10% 48.78%

Table 7.4 – Classification Accuracies by Selected Techniques for Dataset B

Graph 7.3 – Classification Accuracies by Selected Techniques for Dataset B

43

From Table 7.4 and Graph 7.3, it is cleared that SVM provides the best accuracy among all others. An interesting observation is that KNN, RT and BNT lies at the same level. However, ANN lies last to SVM. Model B can be considered a better model for formatting EEG data for classification due to the fact that it helps to achieve better accuracies. However the number of instances formulated using this model are comparatively less but classifier is even faster at the same time because it requires less time for training and classification of EEG data. For future work, more number of subjects can be used to increase the size of Dataset B for further testing which can hopefully show even better results. Due to the difference in accuracies obtained for using different models for dataset formulation, a comparison of both models have been carried out in order to identify the reasons and to select a better model to continue the research operations. A comparison of accuracies obtained by using both models is shown in Graph 7.4:

Graph 7.4 – Comparison of Results Obtained Due to Dataset Formatting using Model A and Model B Focusing on the findings, it has been observed that model A is better for formatting EEG data and formulating instances for it. Having less number of instances helps the classifier learn more information from one instance and even faster as compared to model B but as a rule of thumb, number of instances should be around 10 times the number of attributes or more as: 10 attributes –> 100+ instances Moreover, less number of instances only helps the classifier if there are no contradictions in the data, which is rarely the case for real-world problems. A comparison of both model derived from the findings is listed in the Table 7.5 below: Model A More Number of Instances Less Information Per Instance More Space Consumed by Dataset Slow Training of Classifier More Diversity

Model B Less Number of Instances More Information Per Instance Less Space Consumed by Dataset Fast Training of Classifier Less Diversity

44

Slightly Low Classification Accuracy

Better Classification Accuracy

Table 7.5 – Comparison of Model A and Model B Based on Findings Based on the comparison of accuracies achieved by formatting dataset using model A in Graph 7.4 and outcome of model A and B comparison in Table 7.5, rest of the research operations were continued using model B. Considering the findings based on Dataset B which is formatted using model B (preferred/selected model to continue research operation), SVM provides the best accuracy among all other selected techniques and can be considered as most suitable to accurately classify EEG data according to associated affective/emotional states. In order to derive further findings, the Dataset B is further divided into three dataset as explained in Table 7.6: Dataset Dataset 1B Dataset 2B Dataset 3B

Subjects 5 5 5

Table 7.6 – Division of Dataset B The basic aim of dividing the dataset in this way was to analyze the EEG data for different number of subjects in order to find the possible differences among them. As shown in the Table 7.6, each datasets contains EEG data for equal number of subjects. The obtained datasets were passed to WEKA one by one. The selected techniques KNN, RT, BNT, SVM and ANN were used for classification in WEKA. Each dataset was trained using each selected technique to classify negative or positive arousal/valence values as correctly classified and neutral values as incorrectly classified. The techniques used had all the default parameter values as implemented in WEKA and 10-fold cross validation (standard value for cross validation). The sequence and description by which datasets were passed to WEKA are available in Table 7.7: Dataset Dataset 1B Dataset 2B Dataset 3B

Description KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation

Table 7.7 – Datasets and their Description while Passing to WEKA The classification accuracies by selected techniques for Dataset 1B, 2B and 3B are shown in the Table 7.8 and Graph 7.5: Technique Dataset 1B Dataset 2B Dataset 3B K-Nearest Neighbor 70.37% 66.67% 51.35% Regression Tree 62.96% 44.44% 45.95% Bayesian Network 59.26% 55.44% 48.65% Support Vector Machine 77.78% 70.27% 51.35% Artificial Neural Networks 70.37% 61.11% 43.24% Table 7.8 – Classification Accuracies by Selected Techniques for Dataset 1B, 2B and 3B

45

Graph 7.5 – Classification Accuracies by Selected Techniques for Dataset 1B, 2B and 3B According to Table 7.8 and Graph 7.5, there are considerable differences among the accuracies provided by each technique for each dataset. The same considerable differences were observed when the selected techniques where applied to datasets of each subjects individually. The basic reasons to these differences are the diversity of data and number of instances in each datasets. Table 7.9 shows the datasets with their description which were passed to WEKA for each subject individually: Subject 1 2 3 … … 14 15

Dataset Dataset B1 Dataset B2 Dataset B3 … … Dataset B14 Dataset B15

Description KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation … … KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation

Table 7.9 – Passing Datasets to WEKA for Each Subject Individually The classification accuracies by selected techniques for Subject 1, 2 and 3 are shown in the Table 7.10 and Graph 7.6: Classifier K-Nearest Neighbor Regression Tree Bayesian Network Support Vector Machine Artificial Neural Network

Subject 1 54.54% 36.36% 36.36% 45.45% 45.45%

Subject 2 72.72% 54.54% 72.72% 45.45% 45.45%

Subject 3 83.33% 50.00% 66.66% 50.00% 50.00%

Table 7.10 – Classification Accuracies by Selected Techniques for Subject 1, 2 and 3

46

Graph 7.6 – Classification Accuracies by Selected Techniques for Subject 1, 2 and 3 Table 7.10 and Graph 7.6 shows considerable difference among accuracies obtained for each subjects using the selected techniques. It is the same difference which was observed in Table 7.8 and Graph 7.5. According to the results obtained for classifying EEG data for different datasets using different machine learning techniques, there are considerable differences among results with some as low accuracies and others as high. However, the accuracies found through literature study are comparatively high. There are various reasons for such results such as: • • • • •

Use of different EEG data collection methods Use of different EEG data screening methods Selection of different EEG data features Use of different EEG data formatting methods Use of different EEG data classification methods and their parameters

However, it is common for EEG data classification accuracies to relatively be low. This is because, bulk of noise and artifacts are present in the EEG signals which are not easy to avoid and hence make their analysis difficult. There are certain techniques available to remove the artifacts from the EEG data as discussed before however this may risks losing some valuable data and affect the classification accuracy results.

47

8

THE TOWER OF HANOI - PILOT STUDY

8.1

Introduction

The Tower of Hanoi (ToH) puzzle is one of the significant tools in cognitive psychology in order to comprehend planning with respect to information processing [85] or cognitive neuroscience [86]. The starting and end configuration for ToH puzzle is shown in Figure 8.1. It has three vertical pegs and discs of growing size (A, B, C, D and E). All of the discs are at the most left peg at the starting configuration while the largest disc on bottom and smallest at the top. In order to solve the puzzle, all the discs are required to move from the most left peg to the most right peg keeping the same order as on the most left peg during the starting configuration. During this, only one disc at a time can be moved and a larger disc cannot be at the top of smaller.

Figure 8.1 – The Tower of Hanoi Puzzle [87] ToH was taken as a minor puzzle and used by the mathematicians until 1970s [88]; such as [89] [90] [91]. However, ToH become far most interesting with respect to informationprocessing for the psychologists such as [85] [92] [93]. An experiment for ToH was conducted in order to test out the psychophysiological equipment on an interactive, physical task. To say something potentially meaningful about the task itself, two versions were tested; one with four discs (4-discs) and one with five discs (5-discs). The EEG data for the experiment was recorded, screened, processed, formatted and evaluated using different technical approaches of machine learning. The results of the evaluation are compared to the results obtained from the experiment conducted in the main study. The basic purpose of this comparison was to see the differences among the results obtained from the experiment conducted in the main study (visual task performed by

48

subjects) and experiment conducted in this pilot study (physical task performed by subjects) in order to relate the results with the tasks performed by a robot.

8.2

Aims and Objectives

The main aim of this pilot study was to evaluate different machine learning techniques to classify EEG data associated with affective/emotional states while a physical task is performed along with the followings: • • • •

8.3

Finding how hard the versions of ToH are and what is plausible length for HumanRobot experiments later on. Map out the learning of the task: is the second version always easier than the first? Is the 5-disc always harder than the 4-disc? How are the psychophysiological data correlated to performance of the subjects (this would have to be averages for now, total syncing is implausible). Grouping based on earlier tasks; do they reveal anything about performance in ToH? E.g. the subjects with lowest arousal during IAPS, are they different from the ones with highest arousal, with regards to performance in ToH?

Expected Outcomes

Expected outcome of this pilot study was identification of the most appropriate machine learning technique to classify EEG data (obtained from physical task performed by subjects) according to associated affective/emotional states. This included: • • • •

Indication harder version of ToH and plausible length for Human-Robot experiments. Identification of psychophysiological data correlation to performance of the subjects. Listing the differences among classification accuracies obtained from the experiment conducted for visual and physical tasks. Identification of the most appropriate technique for the correct and most accurate classification of EEG data.

Discussion and conclusion about how the selected technique can be used to classify EEG data (obtained from physical task performed by subjects) associated with specific affective/emotional states and differences among accuracies obtained from the experiment conducted for visual and physical tasks.

8.4

Research Operations

This pilot study has completed using quantitative research methodology. The experiment planning, preparation, design and execution were completed keeping the experiment limitation in consideration as explained in Chapter 5. Hardware and software used in the experiment are explained in Chapter 4. However, the experiment also included ToH puzzles apart from those mentioned in Chapter 4. Threats to validity of experiment as explained in Chapter 5 were taken in consideration. The experiment was conducted using the same subjects’ demographics as explained in Chapter 5. Each subject was instructed on how to solve the puzzle. Half of the subjects played the 4-discs first, and then the 5-discs. The other half of the subjects played the 5-discs first, and then the 4-discs. The purpose of this arrangement was to avoid any possible experiment limitation. During the experiment, EEG

49

BioSemi Dataset Analysis ActiveTwo and System Classification

EEG Data Dataset Recording and Modeling and Storing Formulation

EEGFeature Data ICA, Screening Selectionusing and SAM Extraction

data was collected, screened, processed, formatted and classified in the same manner as explained in Chapter 4. The questionnaire was filled by each subject and used as SelfAssessment Manikin (SAM) during the experiment. This SAM was used during the screening process. The questionnaire is available in Appendix B. The whole process is summarized in the Figure 8.2:

Figure 8.2 – Research Operations

8.5

Discussion

Dataset TB, formatted using EEG data obtained for this pilot study considering the model B (as explained in Chapter 6, Section 6.4); compiled for all subjects was passed to classification software WEKA for its offline analysis. The selected techniques KNN, RT, BNT, SVM and ANN were used for classification in WEKA. The dataset was trained using each selected technique to classify negative or positive arousal/valence values as correctly classified and neutral values as incorrectly classified. The techniques used had all the default parameter values as implemented in WEKA and 10-fold cross validation (standard value for cross validation). The description for datasets passed is available in Table 8.1: Dataset Dataset TB

Description KNN, RT, BNT, SVM and ANN with 10-fold Cross Validation

Table 8.1 – Dataset and its Description while Passing to WEKA The classification accuracies by selected techniques for Dataset TB are shown in the Table 8.2 and Graph 8.1:

Techniques K-Nearest Neighbor (KNN) Regression Tree (RT) Bayesian Network (BNT) Support Vector Machine (SVM) Artificial Neural Networks (ANN)

Accuracy 55.00% 60.00% 45.00% 70.00% 45.00%

Table 8.2 – Classification Accuracies by Selected Techniques for Dataset TB

50

Graph 8.1 - Classification Accuracies by Selected Techniques for Dataset TB According to Table 8.2 and Graph 8.1, SVM provides the best accuracy among all other techniques. RT is the second best and KNN is third; however, BNT and ANN are at the same level. A comparison of these results (based on physical task performed by subjects) has been performed with the results obtained from the experiment conducted for the main study (based on visual task performed by subjects) in order to see the differences among both. The classification accuracies by selected techniques for Dataset TB formatted for each study (main and pilot studies) individually are shown in the Table 8.3 and Graph 8.2:

Techniques K-Nearest Neighbor (KNN) Regression Tree (RT) Bayesian Network (BNT) Support Vector Machine (SVM) Artificial Neural Networks (ANN)

Main Study 52.44% 52.44% 52.44% 56.10% 48.78%

Pilot Study 55.00% 60.00% 45.00% 70.00% 45.00%

Table 8.3 – Comparison of Classification Accuracies by Selected Techniques for Dataset TB As shown in Table 8.3 and Graph 8.2, there are clear differences among results obtained from both studies. The SVM showed a big difference and the rest with considerable differences. One of the possible reasons to these differences is the difference in tasks performed by the subjects in each study. Performing a task such as ToH is considered to be more suitable for cognitive psychology in order to understand planning with respect to information processing [85] or cognitive neuroscience [86]. Psychologists in [85] [92] [93] have come to the same conclusion.

51

Graph 8.2 – Comparison of Classification Accuracies by Selected Techniques for Dataset TB While subjects were performing the experiment, they made a sequence of moves, visualized the movement of discs over the pegs in their mind, memorized the consequence of each move and evaluated their planned moves, all of this defined as a computational arena in which all task-relevant information converges and is manipulated in service of a future goal [94]. All of these focused activities in order to achieve the future goal lead to generate related brain activity. Hence the EEG data obtained holds highly correlated features which help to achieve better accuracies in this study as compared to the other.

8.6

Summary and Conclusion

ToH puzzle is famous for understanding different aspects of cognitive psychology and is widely used in various researches. It is a tool which is always preferred by the psychologists for information-processing especially. There are possible differences in brain activities when human perform a visual task and physical task. To find out these differences, an experiment for ToH was conducted in order to test out the psychophysiological equipment on an interactive, physical task. The main purpose of this research study was to evaluate different machine learning techniques to classify EEG data obtained while a task is performed. To achieve this, five machine learning techniques selected based on literature study, their use in empirical studies and accuracy results reported by different authors. The selected techniques such as K-Nearest Neighbor (KNN), Regression Tree (RT), Bayesian Network (BNT), Support Vector Machine (SVM) and Artificial Neural Networks (ANN) were evaluated. For validation, the selected techniques were analyzed using EEG data collected from 20 subjects through an experiment in a controlled environment. The data obtained was processed to remove the artifacts and extract features. The data was formatted using a model in a form that is acceptable by the classification software to analyze the classification accuracies of the selected techniques.

52

According to the results, Support Vector Machine (SVM) is the best to classify EEG data associated with specific affective/emotional states with accuracy as 70.00%. Regression Tree (RT) was second best with accuracy as 60.00%. A noticeable observation is; focused brain activities for a single goal can lead to better accuracy results as compared to diverse brain activities for diverse goals. While looking at the findings from the questionnaires, difference in opinions and data were found among each subjects. Table 9.4 summarizes the findings derived from questionnaire: Questions 4-Disc Problem was Hard? 5-Disc Problem was Hard?

Strongly Agree 30% 80%

Strongly Disagree 70% 20%

Table 8.4 – Findings from Questionnaire The difference among opinions of various subjects reported in questionnaire is one of the reasons of diverse data and different number of instances in dataset which lead to different accuracies obtained using selected machine learning techniques.

53

9

SUMMARY AND CONCLUSION

Different methods are successfully applied to develop different classifiers for the correct classification of EEG data but no general classifier is available for this purpose. However, there is always both room for improvement and the need to develop classifiers suitable for the requirements of certain applications. In this case, for the PsyIntEC project, we needed to choose an appropriate method for the classification of EEG data. The main purpose of this research study was to evaluate different machine learning techniques to classify EEG data. To achieve this, five machine learning techniques selected based on literature study, their use in empirical studies and accuracy results reported by different authors. The selected techniques such as K-Nearest Neighbor (KNN), Regression Tree (RT), Bayesian Network (BNT), Support Vector Machine (SVM) and Artificial Neural Networks (ANN) were evaluated. For validation, the selected techniques were analyzed using EEG data collected from 20 subjects through an experiment in a controlled environment. The data obtained was processed to remove the artifacts and extract features. The data was formatted using two different models in a form that is acceptable by WEKA to analyze the classification accuracy of the selected techniques. According to the results, Support Vector Machine (SVM) is the best to classify EEG data associated with specific affective/emotional states with accuracies 56.10% and 70.00% for visual and physical tasks respectively. Regression Tree (RT) was second best with accuracies 52.44% and 60.00% for visual and physical tasks respectively. SVM is better in performance than RT. However, RT is famous for providing better accuracies for diverse EEG data. Hence, there is trade-off required while considering one as the best to classify EEG data accurately. A remarkable observation is that when it comes to classify EEG data for each subject individually, the results are completely diverse. This means that there are significant differences between individual subjects. Similarly, another noticeable observation is, focused brain activities for a single goal can lead to more better accuracy results as compared to diverse brain activities for diverse goals.

9.1

Answers to Research Questions

This section maps the results to the relevant research questions.

9.1.1

Research Question 1

Which techniques are available in current literature for classifying EEG data associated with specific affective/emotional states? By summarizing the findings of literature study and accuracies reported by different studies, various machine learning techniques are available to classify EEG data but the following machine learning techniques are considered to be most suitable and widely used in most of empirical studies for classifying EEG data associated with specific affective/emotional states: •

K-Nearest Neighbor (KNN) 54

• • • •

9.1.2

Regression Tree (RT) Bayesian Network (BNT) Support Vector Machine (SVM) Artificial Neural Networks (ANN)

Research Question 2

What EEG data features give the best results for different classification techniques? According to literature study and findings of our experiments, the following statistical features give the best results when selected classification techniques are used to classify EEG data associated with specific affective/emotional states: • • • •

9.1.3

Minimum Value Maximum Value Mean Value Standard Deviation

Research Question 3

How accurately can EEG data be classified according to associated affective/emotional states using the selected techniques? The EEG data for one subject greatly differs from another. This is because different individuals express same emotion with different characteristic response patterns for a same situation. Due to this fact, there are considerable differences among accuracies provided by different machine learning techniques for different individuals and datasets. The EEG data for all subjects combined in a dataset was classified according to associated affective/emotional states using the selected techniques and following accuracies were obtained: Techniques K-Nearest Neighbor (KNN) Regression Tree (RT) Bayesian Network (BNT) Support Vector Machine (SVM) Artificial Neural Networks (ANN)

Visual Task 52.44% 52.44% 52.44% 56.10% 48.78%

Physical Task 55.00% 60.00% 45.00% 70.00% 45.00%

Table 9.1 – Classification Accuracies by Selected Techniques for Visual and Physical Task As shown in the Tables 9.1, there are noticeable differences among accuracies obtained for the EEG data based on visual task and physical tasks because of focused brain activities of subjects during the experiment conducted for pilot study in contrast to diverse activities of subjects during the experiment conducted for main study.

9.2

Future Work

A future work can be done to seek improvement from the followings:

55

•

• •

•

•

•

•

EEG data acquisition using a different experiment design. o A different system such as Emotive EPOC device can be used to seek improvement in EEG data acquisition. EEG data recording from more than six number of signals. o This may improve the representation of selected emotions. Increasing the number of subjects for EEG data acquisition. o More number of subjects will increase the dataset size and can result in better accuracy. EEG data processing using a different technique. o Other techniques such as Fourier transform, wavelet transform, thresholding, and peak detection etc. can be used to seek improvement in data processing. Different number of features selection and extraction. o More or various number of features can help improving the dataset formatting. EEG data formatting using a different model. o A better model can be designed using a different approach for efficient and valuable classification of dataset. EEG data classification using different parameters for selected machine learning techniques. o Other machine learning techniques such as Random Forest, ZeroR, Naïve Bayes etc. can be used to seek improvement in classification accuracy.

56

10

REFERENCES

[1] P. Zhang and D. F. Galletta, Human-Computer Interaction and Management Information Systems: Foundations. M.E. Sharpe, 2006. [2] A. C. Ramirez-Hernandez, J. A. Rivera-Bautista, A. Marin-Hernandez, and V. A. Garcia-Vega, “Detection and Interpretation of Human Walking Gestures for HumanRobot Interaction,” in Eighth Mexican International Conference on Artificial Intelligence, 2009. MICAI 2009, 2009, pp. 41 –46. [3] Y.-P. Lin, C.-H. Wang, T.-L. Wu, S.-K. Jeng, and J.-H. Chen, “EEG-based emotion recognition in music listening: A comparison of schemes for multiclass support vector machine,” in Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE International Conference on, 2009, pp. 489 –492. [4] M. Li, Q. Chai, T. Kaixiang, A. Wahab, and H. Abut, “EEG Emotion Recognition System,” in In-Vehicle Corpus and Signal Processing for Driver Behavior, K. Takeda, H. Erdogan, J. H. L. Hansen, and H. Abut, Eds. Springer US, 2009, pp. 125–135. [5] P. Rani, C. Liu, N. Sarkar, and E. Vanman, “An empirical study of machine learning techniques for affect recognition in human–robot interaction,” pp. 58–69, 2006. [6] T. O. Zander, C. Kothe, S. Jatzev, and M. Gaertner, “Enhancing Human-Computer Interaction with Input from Active and Passive Brain-Computer Interfaces,” in BrainComputer Interfaces, D. S. Tan and A. Nijholt, Eds. Springer London, 2010, pp. 181– 199. [7] C.-T. Lin, R.-C. Wu, S.-F. Liang, W.-H. Chao, Y.-J. Chen, and T.-P. Jung, “EEG-based drowsiness estimation for safety driving using independent component analysis,” Circuits and Systems I: Regular Papers, IEEE Transactions on, vol. 52, no. 12, pp. 2726 – 2738, Dec. 2005. [8] B. Blankertz, M. Tangermann, C. Vidaurre, S. Fazli, C. Sannelli, S. Haufe, C. Maeder, L. Ramsey, I. Sturm, G. Curio, and K.-R. Müller, “The Berlin Brain–Computer Interface: Non-Medical Uses of BCI Technology,” Frontiers in Neuroscience, vol. 4, 2010. [9] A. Lecuyer, F. Lotte, R. B. Reilly, R. Leeb, M. Hirose, and M. Slater, “Brain-Computer Interfaces, Virtual Reality, and Videogames,” Computer, vol. 41, no. 10, pp. 66 –72, Oct. 2008. [10] Anderson, C., Sijercic, Z, “Classification of EEG Signals from Four Subjects During Five Mental Tasks.Asymmetry During Positive and Negative Affect,Psychophysiology,” vol. 16, pp. 202–203, 1979. [11] C. Berka, D. J. Levendowski, M. M. Cvetinovic, M. M. Petrovic, G. Davis, M. N. Lumicao, V. T. Zivkovic, M. V. Popovic, and R. Olmstead, “Real-Time Analysis of EEG Indexes of Alertness, Cognition, and Memory Acquired With a Wireless EEG Headset,” International Journal of Human-Computer Interaction, vol. 17, no. 2, pp. 151–170, 2004. [12] “Psychophysiological Interaction and Empathic Cognition for Human-Robot Cooperative Work (PsyIntEC),” 10-Jul-2012. [Online]. Available: http://www.bth.se/com/gsil_cognitive_neural_eng.nsf/pages/psyintec. [13] T. Nykopp, “Statistical modelling issues for the adaptive brain interface,” Helsinki University of, 2001. [14] N. Kannathal, U. R. Acharya, C. M. Lim, and P. K. Sadasivan, “Characterization of EEG—A comparative study,” Computer Methods and Programs in Biomedicine, vol. 80, no. 1, pp. 17–23, Oct. 2005. [15] “Experiment for Psychophysiological Interaction and Empathic Cognition for HumanRobot Cooperative Work (PsyIntEC).” [Online]. Available: http://www.bth.se/com/gsil_cognitive_neural_eng.nsf/pages/psyintec. [Accessed: 06May-2012].

57

[16] K. S. Sri and J. C. Rajapakse, “Extracting EEG rhythms using ICA-R,” in IEEE International Joint Conference on Neural Networks, 2008. IJCNN 2008. (IEEE World Congress on Computational Intelligence), 2008, pp. 2133 –2138. [17] D. S. Sanei and J. A. Chambers, EEG Signal Processing. John Wiley & Sons, 2007. [18] W. V. Drongelen, Signal Processing for Neuroscientists: Introduction to the Analysis of Physiological Signals. Academic Press, 2007. [19] P. J. Allen, O. Josephs, and R. Turner, “A Method for Removing Imaging Artifact from Continuous EEG Recorded during Functional MRI,” NeuroImage, vol. 12, no. 2, pp. 230–239, Aug. 2000. [20] J. Malmivuo and R. Plonsey, Bioelectromagnetism: Principles and Applications of Bioelectric and Biomagnetic Fields. Oxford University Press, 1995. [21] I. B. Mauss and M. D. Robinson, “Measures of emotion: A review,” Cognition & Emotion, vol. 23, no. 2, pp. 209–237, 2009. [22] R. Plutchik, Emotions and life: Perspectives from psychology, biology, and evolution, vol. xix. Washington, DC, US: American Psychological Association, 2003. [23] J. A. Russell, “Affective space is bipolar,” Journal of Personality and Social Psychology, vol. 37, no. 3, pp. 345–356, 1979. [24] G. Chanel and others, “Emotion assessment for affective-computing based on brain and peripheral signals,” Computer Science Departement, University of Geneva, Geneva, 2009. [25] J. A. Russell, “A circumplex model of affect,” Journal of Personality and Social Psychology, vol. 39, no. 6, pp. 1161–1178, 1980. [26] Y. Liu, O. Sourina, and M. K. Nguyen, “Real-Time EEG-Based Human Emotion Recognition and Visualization,” in Cyberworlds (CW), 2010 International Conference on, 2010, pp. 262 –269. [27] G. Chen and R. Hou, “A New Machine Double-Layer Learning Method and Its Application in Non-Linear Time Series Forecasting,” in International Conference on Mechatronics and Automation, 2007. ICMA 2007, 2007, pp. 795 –799. [28] H. Parvin, H. Alizadeh, and B. Minaei-Bidgoli, “MKNN: Modified k-nearest neighbor,” in Proc. World Congress on Engineering and Computer Science (WCECS) San Francisco, USA, 2008. [29] B. V. Dasarathy, “Nearest neighbor (NN) norms: NN pattern classification techniques,” Los Alamitos: IEEE Computer Society Press, 1990, vol. -1, 1990. [30] Y. Wu, K. Ianakiev, and V. Govindaraju, “Improved k-nearest neighbor classification,” Pattern Recognition, vol. 35, no. 10, pp. 2311–2318, Oct. 2002. [31] L. Breiman, Classification and regression trees. Boca Raton, Fla.: Chapman & Hall, 1984. [32] N. Speybroeck, “Classification and regression trees,” International Journal of Public Health, vol. 57, no. 1, pp. 243–246, 2012. [33] S. Downey and M. J, Russell, “A Decision Tree Approach to Task Independent Speech Recognition in Proceedings of Inst Acoustics Autumn Conf on Speech and Hearing,” vol. 14, no .6, pp. 181–188, 1992. [34] L. E. Brown, I. Tsamardinos, and C. F. Aliferis, “A novel algorithm for scalable and accurate bayesian network learning,” in 11th World Congress on Medical Informatics (MEDINFO), 2004. [35] N. Friedman, D. Geiger, and M. Goldszmidt, “Bayesian Network Classifiers,” Machine Learning, vol. 29, no. 2, pp. 131–163, 1997. [36] V. N. Vapnik, “An overview of statistical learning theory,” Neural Networks, IEEE Transactions on, vol. 10, no. 5, pp. 988 –999, Sep. 1999. [37] C. K. ho and M. Sasaki, “EEG data classification with several mental tasks,” in Systems, Man and Cybernetics, 2002 IEEE International Conference on, 2002, vol. 6, p. 4 pp. vol.6. [38] C. L. Bethel, K. Salomon, R. R. Murphy, and J. L. Burke, “Survey of Psychophysiology Measurements Applied to Human-Robot Interaction,” in Robot and Human interactive

58

Communication, 2007. RO-MAN 2007. The 16th IEEE International Symposium on, 2007, pp. 732 –737. [39] Danny Oude Bos, “eeg based emotion recognition EEG-based Emotion Recognition The Influence of Visual and Auditory Stimuli,” 2006, pp. 1–17. [40] R. Horlings, D. Datcu, and L. J. M. Rothkrantz, “Emotion recognition using brain activity,” in Proceedings of the 9th International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing, New York, NY, USA, 2008, pp. 6:II.1–6:1. [41] M. Murugappan, M. Rizon, R. Nagarajan, S. Yaacob, I. Zunaidi, and D. Hazry, “Lifting scheme for human emotion recognition using EEG,” in International Symposium on Information Technology, 2008. ITSim 2008, 2008, vol. 2, pp. 1 –7. [42] Kristina Schaaff, “‘EEG-based emotion recognition,’ Diplomarbeit am Institut fur Algorithmen und Kognitive Systeme, Universitat Karlsruhe (TH), 2008.” [43] R. A. Fisher, “THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS,” Annals of Human Genetics, vol. 7, no. 2, pp. 179–188, Apr. 2011. [44] B. Efron, “Least angle regression,” Ann. Statist., vol. 32, no. 2, pp. 407–499, 200404. [45] T. Jaakkola and M. Jordan, “A variational approach to Bayesian logistic regression models and their extensions,” in Proceedings of the sixth international workshop on artificial intelligence and statistics, 1997. [46] G. F. Wilson, C. A. Russell, J. W. Monnin, J. R. Estepp, and J. C. Christensen, “How Does Day-to-Day Variability in Psychophysiological Data Affect Classifier Accuracy?,” Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 54, no. 3, pp. 264–268, Sep. 2010. [47] J. R. Millan, F. Renkens, J. Mourino, and W. Gerstner, “Noninvasive brain-actuated control of a mobile robot by human EEG,” Biomedical Engineering, IEEE Transactions on, vol. 51, no. 6, pp. 1026 –1033, Jun. 2004. [48] R. W. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: analysis of affective physiological state,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 23, no. 10, pp. 1175 –1191, Oct. 2001. [49] F. Nasoz, K. Alvarez, C. L. Lisetti, and N. Finkelstein, “Emotion recognition from physiological signals for presence technologies,” International Journal of Cognition, Technology, and Work-Special Issue on Presence, vol. 6, no. 1, 2003. [50] C. Conati, “Probabilistic assessment of user’s emotions in educational games,” Applied Artificial Intelligence, vol. 16, no. 7–8, pp. 555–575, 2002. [51] G. F. Wilson and C. A. Russell, “Real-Time Assessment of Mental Workload Using Psychophysiological Measures and Artificial Neural Networks,” Human Factors: The Journal of the Human Factors and Ergonomics Society, vol. 45, no. 4, pp. 635–644, Dec. 2003. [52] A. Oyegoke, “The constructive research approach in project management research,” International Journal of Managing Projects in Business, vol. 4, no. 4, pp. 573–595, Sep. 2011. [53] P. Isokoski, “Manual text input: experiments, models, and systems,” University of Tampere, Department of Computer and Information Sciences, 2004. [54] G. Crnkovic, “Constructive Research and Info-computational Knowledge Generation,” in Model-Based Reasoning in Science and Technology, vol. 314, L. Magnani, W. Carnielli, and C. Pizzi, Eds. Springer Berlin / Heidelberg, 2010, pp. 359–380. [55] K. Lukka, “‘The constructive research approach,’ Case study research in logistics.,” Publications of the Turku School of Economics and Business Administration, Series B,, vol. 1, no. 2003, pp. 83–101. [56] J. W. Creswell, Research design  : qualitative, quantitative, and mixed methods approaches. Thousand Oaks, Calif.: Sage, 2009. [57] “Biosemi EEG ECG EMG BSPM NEURO amplifier electrodes.” [Online]. Available: http://www.biosemi.com/products.htm. [Accessed: 28-Jul-2012]. [58] “EDFbrowser.” [Online]. Available: http://www.teuniz.net/edfbrowser/. [Accessed: 02Aug-2012].

59

[59] A. Delorme and S. Makeig, “EEGLAB: an open source toolbox for analysis of singletrial EEG dynamics including independent component analysis,” Journal of Neuroscience Methods, vol. 134, no. 1, pp. 9–21, Mar. 2004. [60] “Starlab - Living Science.” [Online]. Available: http://starlab.es/products/biosemi. [Accessed: 03-Aug-2012]. [61] “About GSIL - Blekinge Institute of Technology - in real life.” [Online]. Available: http://www.bth.se/com/gsil. [Accessed: 05-Aug-2012]. [62] “Technology Foundation (STW) - SIINN Safe Implementation of Innovative Nanoscience and Nanotechnology.” [Online]. Available: http://www.siinn.eu/en/partners/technology-foundation-stw/,30. [Accessed: 05-Aug2012]. [63] “FOUNDATION UNIVERSITY.” [Online]. Available: http://www.foundationuniversity.com/FuA/Welcome.html. [Accessed: 05-Aug-2012]. [64] K. N. WHITLEY and A. F. BLACKWELL, “Visual Programming in the Wild: A Survey of LabVIEW Programmers,” Journal of Visual Languages & Computing, vol. 12, no. 4, pp. 435–472, Aug. 2001. [65] “European Data Format (EDF).” [Online]. Available: http://www.edfplus.info/. [Accessed: 07-Aug-2012]. [66] “MathWorks - MATLAB and Simulink for Technical Computing.” [Online]. Available: https://www.mathworks.com/. [Accessed: 07-Aug-2012]. [67] “EEGLAB - Open Source Matlab Toolbox for Electrophysiological Research.” [Online]. Available: http://sccn.ucsd.edu/eeglab/. [Accessed: 08-Aug-2012]. [68] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The WEKA data mining software: an update,” SIGKDD Explor. Newsl., vol. 11, no. 1, pp. 10–18, Nov. 2009. [69] E. Frank, M. Hall, L. Trigg, G. Holmes, and I. H. Witten, “Data mining in bioinformatics using Weka,” Bioinformatics, vol. 20, no. 15, pp. 2479–2481, Oct. 2004. [70] P. Lang, M. Bradley, and B. Cuthbert, “International affective picture system (IAPS): Affective ratings of pictures and instruction manual,” 2008. [71] P. C. Petrantonakis and L. J. Hadjileontiadis, “Emotion Recognition from Brain Signals Using Hybrid Adaptive Filtering and Higher Order Crossings Analysis,” Affective Computing, IEEE Transactions on, vol. 1, no. 2, pp. 81 –97, Dec. 2010. [72] R. J. Davidson, P. Ekman, C. D. Saron, J. A. Senulis, and W. V. Friesen, “Approachwithdrawal and cerebral asymmetry: Emotional expression and brain physiology: I,” Journal of Personality and Social Psychology, vol. 58, no. 2, pp. 330–341, 1990. [73] R. W. Homan, J. Herman, and P. Purdy, “Cerebral location of international 10–20 system electrode placement,” Electroencephalography and Clinical Neurophysiology, vol. 66, no. 4, pp. 376–382, Apr. 1987. [74] O. AlZoubi, R. Calvo, and R. Stevens, “Classification of EEG for Affect Recognition: An Adaptive Approach,” in AI 2009: Advances in Artificial Intelligence, vol. 5866, A. Nicholson and X. Li, Eds. Springer Berlin / Heidelberg, 2009, pp. 52–61. [75] C. Wohlin, Experimentation in Software Engineering: An Introduction. Springer, 2000. [76] “Biosemi EEG ECG EMG BSPM NEURO amplifiers systems.” [Online]. Available: http://www.biosemi.com/faq/file_format.htm. [Accessed: 09-Aug-2012]. [77] M. Ungureanu, C. Bigan, R. Strungaru, and V. Lazarescu, “Independent component analysis applied in biomedical signal processing,” Measurement Science Review, vol. 4, no. 2, 2004. [78] U. Nations, World Robotics 2004: Statistics, Market Analysis, Forecasts, Case Studies and Profitability of Robot Investment. United Nations, 2005. [79] Y.-H. Yu, P.-C. Lai, L.-W. Ko, C.-H. Chuang, B.-C. Kuo, and C.-T. Lin, “An EEGbased classification system of Passenger’s motion sickness level by using feature extraction/selection technologies,” in Neural Networks (IJCNN), The 2010 International Joint Conference on, 2010, pp. 1 –6.

60

[80] M. Macas, M. Vavrecka, V. Gerla, and L. Lhotska, “Classification of the emotional states based on the EEG signal processing,” in Information Technology and Applications in Biomedicine, 2009. ITAB 2009. 9th International Conference on, 2009, pp. 1–4. [81] W. Y. Huang, X. Q. Shen, and Q. Wu, “Classify the number of EEG current sources using support vector machines,” in Machine Learning and Cybernetics, 2002. Proceedings. 2002 International Conference on, 2002, vol. 4, pp. 1793–1795. [82] M. Tangermann, I. Winkler, S. Haufe, and B. Blankertz, “Classification of artifactual ICA components,” Int J Bioelectromagnetism Volume, vol. 11, no. 2, pp. 110–114, 2009. [83] P. Rani, N. Sarkar, C. A. Smith, and L. D. Kirby, “Anxiety detecting robotic system – towards implicit human-robot collaboration,” Robotica, vol. 22, no. 01, pp. 85–95, 2004. [84] K. Takahashi, “Remarks on SVM-based emotion recognition from multi-modal biopotential signals,” in Robot and Human Interactive Communication, 2004. ROMAN 2004. 13th IEEE International Workshop on, 2004, pp. 95 – 100. [85] H. A. Simon, “The functional equivalence of problem solving skills,” Cognitive Psychology, vol. 7, no. 2, pp. 268–288, Apr. 1975. [86] V. Goel and J. Grafman, “Are the frontal lobes implicated in ‘planning’ functions? Interpreting data from the Tower of Hanoi,” Neuropsychologia, vol. 33, no. 5, pp. 623– 642, May 1995. [87] Y. Xu and S. Corkin, “H. M. revisits the Tower of Hanoi Puzzle,” Neuropsychology, vol. 15, no. 1, pp. 69–79, 2001. [88] J. R. Anderson and S. Douglass, “Tower of Hanoi: Evidence for the cost of goal retrieval,” Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 27, no. 6, pp. 1331–1346, 2001. [89] N. Claus, “La tour d’Hanoi, jeu de calcul,” Science et Nature, vol. 1, no. 8, pp. 127– 128, 1884. [90] A..PDo m ro͡i da a dnA.P .Dm oroi︠ a︡ ,d Mathematical games and pastimes, vol. 10. Pergamon, 1964. [91] A. M. Hinz, “Shortest paths between regular states of the Tower of Hanoi,” Information Sciences, vol. 63, no. 1–2, pp. 173–181, Sep. 1992. [92] D. E. Egan and J. G. Greeno, “Theory of rule induction: Knowledge acquired in concept learning, serial pattern learning, and problem solving,” in Knowledge and cognition, Oxford, England: Lawrence Erlbaum, 1974, p. ix, 321. [93] J. R. Hayes and H. A. Simon, “Understanding written problem instructions,” in Knowledge and cognition, Oxford, England: Lawrence Erlbaum, 1974, p. ix, 321. [94] C. Scien, Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society. Routledge, 1995.

61

APPENDIX A: SELF-ASSESSMENT MANIKIN (SAM) AND INTERNATIONAL AFFECTIVE PICTURE SYSTEM (IAPS) Self-Assessment Manikin (SAM) and International Affective Picture System (IAPS) were used during the experiment for each subject. Table A below has list of IAPS used, aim emotion for them and rating by the subject. The rating scale was from 1 to 10 where towards 1 is low valence/arousal; towards 10 is the highest valence/arousal and 5 is taken as neutral.

IAPS 2220(Male face) 7100 7320 7820 7247 8466(Nudists) 5000(Flower) 2722(Jail) 3060(Mutilation) 4660(EroticCouple) 7325(Watermelon) 9331(Homeless man) 3080(Mutilation) 1450(Gannet) 3170(BabyTumor) 9220(Cemetery) 5621(Skydivers) 6350(Attack) 4659(EroticCouple) 8185(Skydivers) 2490(Man) 3010(Mutilation) 5800(Leaves) 9410(Soldier) 8186(Sky Surfer) 8030(Skier) 7900(violin) 2590(ElderlyWoman) 9360(EmptyPool) 5711(Field)

Aimed Emotion neutral neutral neutral neutral neutral neutral pos/calm neg/calm neg/arou pos/arou pos/calm neg/calm neg/arou pos/calm neg/arou neg/calm pos/arou neg/arou pos/arou pos/arou neg/calm neg/arou pos/calm neg/arou pos/arou pos/arou pos/calm neg/calm neg/calm pos/calm

Valence 4 5 6 5 5 5 7 5 1 7 9 5 7 7 1 3 8 3 7 7 6 1 8 1 8 7 6 6 5 8

Arousal 5 5 5 5 5 7 6 5 8 7 7 5 7 6 8 5 7 5 7 7 6 6 6 9 6 7 5 6 5 6

Table A: SAM for Subject 1 (used for each subject individually) and Description of IAPS

62

APPENDIX B: QUESTIONNAIRE The questionnaire below was used as Self-Assessment Manikin (SAM) during the experiment conducted under pilot study for each subject. Subject Number was anonymous number. Subject Number: ____________________________________________________________ Strategy Used to Solve Tower of Hanoi (Be Explicit): __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ The 4-Disc Problem was Hard? (1=Strongly Agree 7=Strongly Disagree) 1 2 3 4 5 6

7

The 5-Disc Problem was Hard? (1=Strongly Agree 7=Strongly Disagree) 1 2 3 4 5 6

7

63