Design and Development Methodology for the Emotional State Estimation of Verbs Georgios Kouroupetroglou, Nikolaos Papatheodorou, and Dimitrios Tsonos National and Kapodistrian University of Athens, Department of Informatics and Telecommunications, Athens, Greece
[email protected]
Abstract. The use of words and particularly the verbs in Human-Human Interaction reveals significant aspects of both human’s social and mental state. This work presents a novel methodology towards the emotional assessment of verbs by users. Essentially we would like to study whether the emotions that user experience are comparable with the corresponding results obtained through a mixture of natural language and statistical classifiers in SentiWordNet. Following the paper and pencil guidelines of the International Affective Picture System (IAPS) we have developed a web-based unsupervised version of the Self Assessment Manikin (SAM) test, designed for the emotional assessment of verbs in English and Greek language. Thirty five men and seventeen women participated in an internet survey version of the experiment. In the first part of the process, the participants had to assess their induced emotional state while reading a verb (totally 75 Greek verbs), on 5-point scales of “Pleasure”, “Arousal” and “Dominance”. The results comprise coherence and consistency. As a rule, all verbs obtained low to mid range scores on Arousal and Dominance axis and only on the Pleasure dimension scores are close to the edge. Keywords: verbs, emotional state, SentiWordNet, Self-Assessment Manikin test.
1
Introduction
Emotions are considered as an integral part of human existence and thus of our daily activity. The scientific domain of emotions, as an expression of human behavior, constitutes a multidisciplinary effort, with Human-Computer Interaction (HCI) to be one of its aspects. Similarly, it is not the first research area that investigates the way to make emotions experimentally available and conceptually compatible with existing research models. The definition of the word "emotion" is open to several interpretations. According to Scherer [1] emotion is defined as: "An episode of interconnected, synchronized changes in circumstances of all or most of the five organic subsystems in response to the evaluation of an external or internal stimulus on the key concerns of the organism". An alternative and also simpler definition given by Brave and Nass [2]: "The emotion is the reaction to events associated with the needs, goals or concerns of a person and includes the ingredients of physiology, influence, behavior and A. Holzinger et al. (Eds.): SouthCHI 2013, LNCS 7946, pp. 1–15, 2013. © Springer-Verlag Berlin Heidelberg 2013
2
G. Kouroupetroglou, N. Papatheodorou, and D. Tsonos
cognitive". Therefore, emotional behavior in HCI shows that expressiveness incorporates a significant amount of individual variability. This point out that people signifies their emotions with a variety of styles and in a rather wide range of intensity. This work aims to design, implement and test an online web-based tool for selfassessment of the induced emotional state of the participant when reading certain verbs. Existing similar studies have been conducted using semi-automated methods and algorithms (SentiWordNet) without being so far confirmed by users for their validity. The present study introduces a systematic way to study the emotional response of the reader by using a standard experimental procedure. The results can be incorporated to augment the users’ experience during Human-Computer Interaction (details presented in section 6). The basic requirement was that the methodology could be accessed by a large statistical sample of participants in different languages and cultures. For this reason, the focus was on developing a web application so that the research could be easily accessible by users with diverse interests, educational background or occupation, covering a wide range of ages. In this work, we first describe a SentiWordNet based methodological approach for the proper selection of the verbs. In section 3 we present an interactive webbased Self Assessment Manikin (SAM) Test we have developed for the emotional assessment of verbs along with the experimental setup. Last, the results of the study are presented and discussed, along with their possible uses in Human-Computer Interaction, focusing on augmenting acoustic interaction through Expressive Speech Synthesis.
2
Methodology for the Selection of Verbs
2.1
WordNet
The ontology based English WordNet of Princeton University [3] contains information about the conceptual meaning of verbs and much more. It incorporates a classification of the main parts of speech (i.e. nouns, verbs, adjectives and adverbs) into synsets, i.e. sets of semantically or conceptually equivalent words that express an individual concept. WordNet interconnects words not only morphologically, but also with the specific meanings of words. As a result, words that are in close proximity with others in the lexical base are not semantically ambiguous. In addition, WordNet characterizes the semantic relationships between words, in contrast to a dictionary which does not follow a clear pattern, except the concept of similarity. Relations among synsets create an interconnected network. Different senses of polysemous words are members of distinct synsets that are related to different synsets (i.e., occupy different locations in the network). For example, {stock. broth} has superordinate synset {dish} and {stock, breed} has superordinate {variety}. These different synsets are also linked to different parts/whole synsets.
Design and Development Methodology for the Emotional State Estimation of Verbs
2.2
3
SentiWordNet
Every synset in the SentiWordNet [4] lexical resource is correlated with three scores: Obj(s), Pos(s) and Neg(s), that indicate how much Objective, Positive, and Negative are the terms included in the specific synset [5], [6]. These scores take a numerical value between 0 to 1 and the sum of the three scores for each synset must be always 1. It is possible different senses of the same term to have different opinion-related properties. This supports the transition for terms to synsets. It is obvious that a synset may have all of its three scores with nonzero values. In that case, according to the sense indicated by the synset, the respective term has the three opinion-related properties only to a certain degree. Fig. 1 shows the graphical model representation of the three scores for each synset.
Fig. 1. Graphical model representation of the three scores for each synset
SentiWordNet provides scores for all the different meanings that a verb can have in a synset. For example, the verb “teach” under SentiWordNet corresponds to two different concepts: a) impart skills or knowledge of and b) accustom gradually to some action or attitude. For this reason, our study focused on the score of the first concept of each verb which is also the most common. Fig. 2 shows the two meanings of the verb “teach” and the respective ratings in the graphical model representation. 2.3
Selection of Verbs
The following methodology was applied for the selection of the appropriate verbs for the experiments. From the lexical database of the Corpus of Contemporary American English [7] a list of the 5.000 most frequently used words in the English language was originally determined. From this list we choose the most commonly occurring verbs in English. After comparing the lexical database of verbs in SentiWordNet to the corresponding database of the Corpus of Contemporary American English we selected the 1.000 most commonly occurring verbs in English along with their emotional annotation.
4
G. Kouroupetroglou, N. Papatheodorou, and D. Tsonos
Studying the emotion rating of these verbs, we noticed that a very large proportion of them were similarly rated in more than one axis (grade 0.8). As a result we got a list with the 25 verbs that have the highest scores in each group. Thus, we selected 75 verbs for the evaluation which were then translated into Greek. The translation was done in accordance with the widely known online dictionary Babylon English - Greek [8].
Fig. 2. The two different meanings of verb “teach” and their corresponding rating in the graphical model representation
Tables 1, 2 and 3 present the three classes (Positive, Neutral and Negative) respectively of the 75 verbs finally selected. The ranking of each verb in the list of the 5.000 most frequently English words is shown in the second column of the three tables.
3
Emotional Assessment of Verbs
To investigate the emotion that elicited each verb to the reader, we used the SelfAssessment Manikin (SAM) Test [9] [10]. The SAM method measures the emotional response, based on the dimensional approach of emotions [1] [11], giving the user the ability to avoid the verbal expression of emotions in the assessment. It implies a quick and easy procedure (Fig. 3). The tool has been designed to follow a visual, rather than verbal, self-assessment course of the emotions. This makes it easy to use and therefor fast in the application. One of its major advantages can be seen in its reproducibility which allows a comparison of studies from different laboratories or statistical samples. In addition it has been shown to be independent from cultural and linguistic characteristics, suitable for use in different countries and cultural groups [10] [12]. By applying the SAM Test, one can measure emotions in three axis: • “Pleasure” (also used as “evaluation”, “valence”). • “Arousal” (also used as “activation”, “activity”). • “Dominance” (also used as “power”, “potency”).
Design and Development Methodology for the Emotional State Estimation of Verbs
5
Participants in the SAM test can choose one from at least five figures (manikins). In the present study we used a 5-point scale (Fig. 4). To assess the emotional state of “Pleasure”, the extreme values are a smiling and a sad figure. In case of “Arousal” one pole is represented by a figure of great vigor and the opposite with a calm manikin, with eyes closed. Similarly, “Dominance” dimension is represented as controlled by a small manikin and non-controlled by a large one. When evaluating the results, users’ answers can be easily converted from scale points to a group of values which ranges in the interval [-1, 1] or [-100%, 100%]. The value "0" represents the neutral state in each of the three dimensions, whereas the middle values represent an intermediate state.
Fig. 3. Extracting the emotion under the SAM Test. Participants assess their emotional state using the manikins. The 5-point scale values are converted into a percentage scale and then mapped on to the emotion wheel for the verbal-semantic representation of the emotion.
By applying the SAM test on specific projected stimuli we can have two different results that are closely related: the dimensional perspective / modeling of both the emotional states and the emotions of the participants. An application system for the automated estimation of emotion primitives (the dimensional approach) from speech using acoustic features has been proposed recently [13], [14], [15]. Moreover, a study [16] on the automated detection of pleasant and unpleasant emotions in spoken dialogs derived from a call center has been presented.
6
G. Kouroupetroglou, N. Papatheodorou, and D. Tsonos
The SAM test has been used also by Busso et al. [17] in order to evaluate the modelling of the head motion sequences in an experiment of expressive facial animations analyzed in terms of their naturalness and emotional salience during perception. Table 1. The 25 positively classified verbs in English with their rank number according to Corpus of Contemporary American English and the corresponding Greek translation Positive Verbs Greek Translation
Frequency
Dispersion
Positive
Negative
Objective
prefer
προτιμώ
20946
0.96
0.875
0.0
0.125
fit
ταιριάζω
27875
0.96
0.75
0.0
0.25
1964
question
ερωτώ
17924
0.97
0.75
0.0
0.25
2244
deserve
αξίζω
14944
0.97
0.75
0.125
0.125
Rank
Word
1
1728
2
1356
3 4 5
3197
6
3753
7
3075
8
3305
ID
αρμόζω
9477
0.95
0.75
0.0
0.25
διευκολύνω
7571
0.94
0.75
0.0
0.25
qualify
δικαιούμαι
10090
0.95
0.625
0.0
0.375
admire
θαυμάζω
9016
0.96
0.625
0.125
0.25
suit accommodate
9
4125
please
ευχαριστώ
6565
0.95
0.625
0.0
0.375
10
4134
donate
δωρίζω
6606
0.94
0.625
0.0
0.375
11
391
love
αγαπώ
103681
0.95
0.5
0.0
0.5
12
546
teach
διδάσκω
72668
0.95
0.5
0.0
0.5
13
735
save
σώζω
52067
0.97
0.5
0.0
0.5
14
857
check
ελέγχω
45760
0.95
0.5
0.0
0.5
15
1043
tend
τείνω
38295
0.94
0.5
0.125
0.375
16
1214
define
ορίζω
33958
0.90
0.5
0.125
0.375
17
1319
contribute
συνεισφέρω
30090
0.92
0.5
0.25
0.25
18
2836
respect
σέβομαι
11083
0.97
0.5
0.0
0.5
19
2864
possess
κατέχω
11474
0.93
0.5
0.0
0.5
20
3026
satisfy
ικανοποιώ
10194
0.96
0.5
0.0
0.5
21
3066
rid
απαλλάσω
10104
0.95
0.5
0.0
0.5
22
3583
lend
δανείζω
7961
0.96
0.5
0.25
0.25
23
3852
rescue
διασώζω
7187
0.96
0.5
0.0
0.5
24
3868
diagnose
διαγιγνώσκω
7255
0.94
0.5
0.0
0.5
25
4286
instruct
καθοδηγώ
6117
0.95
0.5
0.0
0.5
Following the paper and pencil IAPS Guidelines [18] we have developed a webbased unsupervised version of the SAM test, designed for the emotional assessment of verbs in English and Greek language. This automated SAM test helps to create an easy to use experiment and the rapid collection and process of the results.
Design and Development Methodology for the Emotional State Estimation of Verbs
7
Table 2. The 25 neutrally classified verbs in English with their rank number according to Corpus of Contemporary American English and the corresponding Greek translation Neutral Verbs Word
Greek Translation
Frequency
Dispersion
Positive
Negative
Objective
18
do
κάνω
2573587
0.95
0.0
0.0
1,0
19
say
λέγω
1915138
0.95
0.0
0.0
1.0
35
go
πηγαίνω
1151045
0.93
0.0
0.0
1.0
4
37
can
μπορώ
1022775
0.98
0.0
0.0
1.0
5
45
make
κατασκευάζω
857168
0.98
0.0
0.0
1.0
6
47
know
γνωρίζω
892535
0.93
0.0
0.0
1.0
7
48
will
διαθέτω
824568
0.97
0.0
0.0
1.0
8
56
think
σκέφτομαι
772787
0.91
0.0
0.0
1.0
9
63
take
λαμβάνω
670745
0.97
0.0
0.0
1.0
10
67
see
βλέπω
663645
0.96
0.0
0.0
1.0
11
70
come
έρχομαι
628254
0.95
0.0
0.0
1.0
12
85
look
κοιτάζω
491707
0.93
0.0
0.0
1.0
13
92
use
χρησιμοποιώ
420781
0.96
0.0
0.0
1.0
14
95
find
βρίσκω
395203
0.98
0.0
0.0
1.0
15
103
tell
διηγούμαι
388155
0.94
0.0
0.0
1.0
16
117
work
εργάζομαι
318210
0.98
0.0
0.0
1.0
17
127
try
προσπαθώ
294023
0.96
0.0
0.0
1.0
18
131
ask
ζητώ
284632
0.96
0.0
0.0
1.0
19
139
become
γίνομαι
259102
0.97
0.0
0.0
1.0
20
150
leave
φεύγω
240482
0.96
0.0
0.0
1.0
21
151
put
θέτω
237480
0.96
0.0
0.0
1.0
22
154
mean
εννοώ
242198
0.93
0.0
0.0
1.0
23
156
keep
κρατώ
231760
0.96
0.0
0.0
1.0
24
159
let
αφήνω
240300
0.93
0.0
0.0
1.0
25
164
begin
αρχίζω
218617
0.98
0.0
0.0
1.0
ID
Rank
1 2 3
For the development of the application, PHP [19] was used on an Apache Web Server and MySQL [20]. PHP allowed us to develop dynamic web pages, for the presentation of the stimuli, and to automate the registration of the participants’ answers. The answers are stored in a database (MySQL). Moreover, technologies like HTML (Hyper Text Markup Language) [21], JavaScript [22] and CSS [23] were used in order to enhance the usability, administration and visual characteristics of the tool (e.g. to accurately control the projection time of the verb).
8
G. Kouroupetroglou, N. Papatheodorou, and D. Tsonos
Table 3. The 25 negatively classified verbs in English with their rank number according to Corpus of Contemporary American English and the corresponding Greek translation Negative Verbs
Rank
Word
ID 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Greek Translation
Frequency
Dispersion
Positive
Negative
Objective
0.96 0.97 0.93 0.97 0.96 0.94 0.98 0.93 0.91 0.94 0.95 0.97 0.98 0.95 0.96 0.96 0.97 0.94 0.97 0.95 0.93 0.96 0.97 0.94 0.91
0.0 0.0 0.0 0.0 0.0 0.0 0.125 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.25 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
0.875 0.75 0.75 0.75 0.75 0.75 0.625 0.625 0.625 0.625 0.625 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
0.125 0.25 0.25 0.25 0.25 0.25 0.25 0.375 0.375 0.375 0.375 0.5 0.5 0.5 0.25 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
1413
deny
αρνούμαι
26675
750
protect
προστατεύω
50649
1535
hate
μισώ
24921
1874
complain
παραπονούμαι
19102
3024
damage
ζημιώνω
10163
3778
abuse
καταχρώμαι
7554
1670
fear
φοβούμαι
21333
3508
murder
δολοφονώ
8462
3679
average
υπολογίζω
8106
3782
apologize
απολογούμαι
7485
4639
spare
εξοικονομώ
5492
284
lose
χάνω
134102
554
face
αντικρίζω
69493
912
imagine
φαντάζομαι
43487
973
worry
ανησυχώ
40210
1186
replace
αντικαθιστώ
32688
1382
ignore
αγνοώ
27023
1587
disappear
χάνομαι
23389
1786
warn
προειδοποιώ
19996
1798
steal
κλέβω
20296
2083
mind
νοιάζομαι
17378
2437
confront
αντιμετωπίζω
13757
3685
endure
αντέχω
7586
3755
injure
τραυματίζω
7574
3982
vanish
εξαφανίζομαι
7154
Design and Development Methodology for the Emotional State Estimation of Verbs
9
Fig. 4. The manikins of the 5-point scale SAM Test as presented during the test. The verbal expressions of “Pleasure”, “Arousal” and “Dominance” do not appear during the test.
Just before the tests, all the participants have to read a brief text explain the purpose of this research study as well as a short simplified introduction on the emotional states theory. Standard guidelines are followed for all participants during the experimental process [18]. Also, they have to fill in an electronic form with demographic information, for example about their age, education level, for any visual problems, and also if she/he agrees to participate in the test. The participants are familiarized with the SAM test before the measurements. Each stimulus is presented for a few seconds and then they are asked to select the manikins as presented in Fig. 4. By pressing the “continue” button, the next stimulus is presented to the user. Stimuli are given in a random sequence to all the participants. To avoid possible biased results, we have designed the manikins’ presentation layout on the screen in such a way to eliminate any interruption of the participants during the test. In order to evaluate the mental condition of the participants they have to fill in two questionnaires: a) the Symptom Check List-90-Revised (SCL-90-R) [24] to investigate the relationship between psychosomatic disorders and emotional verbal assessment and b) the Eysenck Personality Questionnaire (EPQ) [25].
10
4
G. Kouroupetroglou, N. Papatheodorou, and D. Tsonos
Experimental Setup
Fifty two native Greek speakers, 35 men and 17 women, university students (mean age = 21.7 years, SD = 3.5 years) participated in the experiment. The stimuli were 75 Greek verbs, 25 positively, 25 negatively and 25 objectively characterized (Tables 1, 2 and 3) selected with the methodology described in section 3. Each participant had to fill her/his demographic information and complete a consent form that she/he agrees to participate in the test. Then, they were familiarized with the experiment, through a demo of the SAM test (three stimuli). Certain guidelines were followed for all participants during the experimental process [18]. After the completion of the SAM test, they completed the SCL-90-R and EPQ questionnaires both standardized to the Greek population [26] [27].
5
Results
Figures 5, 6 and 7 present the average values of participants' responses for each verb, on each scale, namely “Pleasure”, “Arousal” and “Dominance”, respectively. Each verb is represented by an integer ID number in the interval [1, 25] (Tables 1, 2 and 3 show the matching ID number-verb). The results of the study comprise coherence and consistency. Specifically, observing Fig. 5, 6 and 7, we can assume that the verb “save” (Table 1, verbID=13 - “σώζω” in
Fig. 5. The graphical representation of the average rating (with the corresponding standard error of mean) of the 75 classified verbs on “Pleasure” dimension in [-1,1] scale. Positive values correspond to positively assessed emotions, while negative values correspond to negatively assessed emotions.
Design and Development Methodology for the Emotional State Estimation of Verbs
11
Greek) obtained the highest value on “Pleasure” and “Arousal” dimensions whereas the verb “murder” (Table 3, verbID=8 - “δολοφωνώ” in Greek) has the lowest value. Additionally, it is noticeable that among the negatively classified verbs, the verb “spare” (Table 3, verbID=11 - “εξοικονομώ” in Greek) show a very low score on the Arousal field in contrast to the verb “worry” (Table 3, verbID=15 - “ανησυχώ” in Greek) which held the lead in its classification. Finally, regarding Dominance it is obvious that the verb with the highest score is “apologize“ (Table 3, verbID=10 - “απολογούμαι” in Greek) suggesting a submissive emotion unlike the verb “know” (Table 2, verbID=6 “γνωρίζω” in Greek) which shows a strong dominant emotion (Fig. 7). We observe that all the 25 positively classified verbs recorded a positive score with only two verbs being at the very middle of the “Pleasure” axis (Fig. 5), namely “suit” (Table 1. verbID=5 - “αρμόζω” in Greek) and “lend” (Table 1, verbID=22 “δανείζω” in Greek). Regarding the 25 neutrally classified verbs, it is worth noting that the majority’s score is concentrated above the middle of the Pleasure axis (22 verbs) and below the middle of the Dominance axis (23 verbs). Furthermore, we observe that the 25 negatively classified verbs are spread out on all three axis. In general, we notice that the neutrally classified verbs show a significant approach towards the positively classified rather than the negative classified ones. As a rule, all verbs obtained low to mid range scores on Arousal and Dominance axis and only on the Pleasure dimension scores are close to the edge.
Fig. 6. The graphical representation of the average rating (with the corresponding standard error of mean) of the 75 classified verbs on “Arousal” dimension in [-1,1] scale. Positive values indicate higher intensity.
12
G. Kouroupetroglou, N. Papatheodorou, and D. Tsonos
Fig. 7. The graphical representation of the average rating (with the corresponding standard error of mean) of the 75 classified verbs on “Dominance” dimension in [-1,1] scale. Positive values correspond to submissive emotions, while negative values correspond to dominant emotions.
6
Conclusions
We have described a web-based Self-Assessment Manikin (SAM) Test [18] designed and developed for the emotional assessment of verbs. The selection of the verbs has been done according to SentiWordNet based methodology. Apart from the verification of the validity of the relative results obtained by SentiWordNet, our study makes a further step regarding the formalism and the correlation of the exported results. Obviously, the format (as shown in Fig. 1) that SentiWordNet has established in order to present the measured sentiment for each verb is very simple and lacks additional information, such as other factors that influence the final score (e.g. mental and psychological state, educational background), as well as supplementary scales which could highlight different interesting aspects of each verb. For instance, due to the alternative way we initially established for the evaluation of each verb, we managed to export the results in relation to three different scales (Pleasure, Arousal, Dominance). Consequently, the experiment offered enhanced information and showed the different influences that contributed to the final result, in comparison with SentiWordNet which characterizes each verb as positive, objective or negative.
Design and Development Methodology for the Emotional State Estimation of Verbs
13
With regard to the issues we faced during the development of the whole experiment, the application’s background response time tracking system which was implemented from the beginning, showed that some users showed lack of concentration after the first half of the experiment which suggests that the large number of verbs could be potentially decreased [31]. In addition, while this study was held we noticed that almost 50% of the users who started the test quit it after some minutes which emphasizes the assumption above. All these cases were excluded from the participants’ set. During the experiment, users were evaluated with regard to their emotional and psychological state (EPQ & SCL-90-R) at the very moment after the completion of the verbs evaluation. The addition of the above tests to our methodology, can lead to further statistical analysis of the exported results. For example, it is possible to identify users with certain emotional and psychological disorders, so as to distinguish them from the rest examined sample and determine whether the emotional and psychological state of a user has an impact on the sentiment evaluation or not and to what extent after a deeper analysis of that particular sample [31]. It should be noticed that the presented experimental procedure has the advantage of being developed and designed for a significant number of users as it is widely open to the public as opposed to SentiWordNet which started by a very small group of experts and maintained by semi-automatic algorithms later on. As a result, we have the capability to investigate a statistically significant population sample, depending on the experimental design and needs. An interesting future study would be to investigate whether emotion/emotional state semantics of the verbs are cross-cultural and/or language independent. Due to “manikin description” of emotional states during the experimental procedure, instead of using their verbal description, we can easily conduct a multilingual and/or multicultural research. There is a growing interest for the results of such kind of studies as they can be merged into software systems for the automated emotional annotation of text documents, e.g. using EmotionML [28]. This kind of information would very useful for systems that can automatically emotionally rank a document by its content in order to accomplish users’ information retrieval needs. Also, this kind of annotation can be used to augment expressions during Human-Computer Interaction. For example, combining the results of the current study with Expressive Speech Synthesis [29], we can augment the user experience during multimodal interaction, focusing on acoustic, as proposed in [30]. The experience obtained from the implementation of this particular study could probably set new standards regarding the sentiment evaluation of verbs and even expand relative research for the analysis of nouns, adjectives as well as adverbs. Consequently, the implemented methodology we used to analyze the available data from the web based application with the help of users’ interaction, can be proven valuable regarding its effectiveness in the opinion mining field of interest [32]. Moreover, the current design of the web application does not support interaction with blind users which leads to improve the user interface-interaction adding support for the acoustic and/or haptic modality. Finally, the support of touch sensitive controls is in our plans as it would make the experimental process easier and more directly straight forward.
14
G. Kouroupetroglou, N. Papatheodorou, and D. Tsonos
Acknowledgements. This research has been co-financed by the European Union (European Social Fund – ESF) and Greek national funds through the Operational Program "Education and Lifelong Learning" of the National Strategic Reference Framework (NSRF) under the Research Funding Project: “THALIS-University of Macedonia- KAIKOS: Audio and Tactile Access to Knowledge for Individuals with Visual Impairments”, MIS 380442.
References 1. Scherer, K.R.: What are emotions? And how can they be measured? Social Science Information 44(4), 695–729 (2005) 2. Brave, S., Nass, C.: Emotion in human-computer interaction. In: Sears, A., Jacko, J.A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 77–92. Taylor and Francis Group (2007) 3. WordNet, Lexical database of English, http://wordnet.princeton.edu/ 4. SentiWordNet, http://sentiwordnet.isti.cnr.it/ 5. Esuli, A., Sebastiani, F.: SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining. In: 5th Conference on Language Resources and Evaluation (LREC 2006), Genoa, Italy, pp. 417–422 (2006) 6. Esuli, A., Baccianella, S., Sebastiani, F.: SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. In: 7th Conference on International Language Resources and Evaluation (LREC 2010), Malta (2010) 7. Word frequency lists and dictionary from the Corpus of Contemporary American English, http://www.wordfrequency.info/intro.asp 8. Babylon 9 translation Software and Dictionary Tool, http://www.babylon.com 9. Lang, P.J., Bradley, M.M., Cuthbert, B. N.: International affective picture system (IAPS): Affective retings of pictures and instruction manual. Technical report A-8, University of Florida, Gainesville, FL (2005) 10. Bradley, M.M., Lang, P.J.: Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25(1), 49–59 (1994) 11. Russell, J.A., Mehrabian, A.: Evidence for a three-factor theory of emotions. Journal of Research in Personality 11(3), 273–294 (1977) 12. Morris, J.D.: Observations SAM: The Self-Assessment Manikin - An Efficient CrossCultural Measurement of Emotional Response. Journal of Advertising Research 35(8), 63–68 (1995) 13. Grimm, M., Mower, E., Narayanan, S., Kroschel, K.: Combining categorical and primitives-based emotion recognition. In: Proceedings of 14th European Signal Processing Conference (EUSIPCO), Florence, Italy (September 2006) 14. Grimm, M., Kroschel, K., Mower, E., Narayanan, S.: Primitives-based evaluation and estimation of emotions in speech. Speech Communication 49(10-11), 787–800 (2007) ISSN: 0167-6393 15. Grimm, M., Kroschel, K., Narayanan, S.: Support Vector Regression for Automatic Recognition of Spontaneous Emotions in Speech. In: IEEE International Conference on Acoustics, Speech and Signal Processing 2007 (ICASSP 2007), Hawaii, USA, April 15-20, vol. 4, pp. IV-1085-IV–1088 (2007) ISBN: 1-4244-0728-1 16. Chul, M.L., Narayanan, S.S.: Toward detecting emotions in spoken dialogs. Transactions on Speech and Audio Processing 13(2), 293–303 (2005) ISSN: 1063-6676
Design and Development Methodology for the Emotional State Estimation of Verbs
15
17. Busso, C., Deng, Z., Grimm, M., Neumann, U., Narayanan, S.: Rigid Head Motion in Expressive Speech Animation: Analysis and Synthesis. IEEE Transactions on Audio, Speech, and Language Processing 15(3), 1075–1086 (2007) ISSN: 1558-7916 18. Lang, P.J., Bradley, M.M., Culthbert, B.N.: International Affective Picture System (IAPS): Instruction Manual and Affective Ratings. Technical Report A-6, The Center for Research in Psychophysiology, University of Florida, U.S.A. (2005) 19. PHP Hypertext Preprocessor, http://www.php.net/ 20. Welling, L., Thomson, L.: PHP and MySQL Web Development, 4th edn. Addison-Wesley Professional (2008) 21. HTML Tutorial, http://www.w3schools.com/html/default.asp 22. JavaScript Tutorial, http://www.w3schools.com/js/default.asp 23. CSS Reference, http://www.w3schools.com/cssref/default.asp 24. Derogatis, L.R.: The Symptom Checklist-90-revised. NCS Assessments, Minneapolis (1992) 25. Corulia, W.J.: A psychometric investigation of the Eysenck Personality Questionnaire (revised) and its relationship to the I7 Impulsiveness Questionnaire. Personality and Individual Differences 8, 651–658 (1987) 26. Donias, S., Karastergiou, A., Manos, N.: Standardization of the symptom checklist-90-R rating scale in a Greek population. Psychiatriki (1), 42–48 (1991) 27. Demetriou, E.X.: The Eysenck Personality Questionnaire: Standardization to the Greek population, Adult and Junior. Engefalos 23, 41–54 (1986) 28. Emotion Markup Language (EmotionML) 1.0, http://www.w3.org/TR/emotionml/ 29. Campbell, N., Hamza, W., Hoge, H., Tao, J., Bailly, G.: Editorial Special Section on Expressive Speech Synthesis. IEEE Trans. Audio, Speech, and Language Processing 14(4), 1097–1098 (2006) 30. Kouroupetroglou, G., Tsonos, D.: Multimodal Accessibility of Documents. In: Pinder, S. (ed.) Advances in Human-Computer Interaction, pp. 451–470. I-Tech Education and Publishing, Vienna (2008) 31. Stickel, C., Ebner, M., Steinbach-Nordmann, S., Searle, G., Holzinger, A.: Emotion Detection: Application of the Valence Arousal Space for Rapid Biological Usability Testing to Enhance Universal Access. In: Stephanidis, C. (ed.) Universal Access in HCI, Part I, HCII 2009. LNCS, vol. 5614, pp. 615–624. Springer, Heidelberg (2009) 32. Petz, G., Karpowicz, M., Fürschuß, H., Auinger, A., Winkler, S.M., Schaller, S., Holzinger, A.: On Text Preprocessing for Opinion Mining Outside of Laboratory Environments. In: Huang, R., Ghorbani, A.A., Pasi, G., Yamaguchi, T., Yen, N.Y., Jin, B. (eds.) AMT 2012. LNCS, vol. 7669, pp. 618–629. Springer, Heidelberg (2012)
