Keyboard and Mouse Interaction Based Mood Measurement Using Artificial Neural Networks Mohammad Sohail Khan
Iftikhar Ahmed Khan
Muhammad Shafi
Department of Computer Software Engineering, University of Engineering and Technology Peshawar, Pakistan.
[email protected]
Department of Computer Software Engineering, University of Engineering and Technology Peshawar, Pakistan.
[email protected]
Department of Computer Software Engineering, University of Engineering and Technology Peshawar, Pakistan.
[email protected]
suddenly changes the behavior of the parents from being angry to very polite response. Many other such examples can easily be seen in our daily lives where the emotional state of another human being changes the way of our communication/interaction. Moods also have significant effect on the performance/ productivity of humans. Such studies have been conducted which have shown the effect of different moods on the performance of programmers [3] and testers [4]. Thus for a computer to communicate with humans in a more human like manner or to enhance its user’s productivity/performance, it is necessary that the computer can recognize and measure the different moods associated with human beings. Natural and social interaction between humans and computers is an active research topic of the Human Computer Interaction (HCI). Reeves and Nass, proposed that human-computer interactions have identical principles like recognition, interpretation and expressions of emotions as found in humanhuman interactions [1]. According to Zimmerman et al., a computer system which can recognize affects can communicate with humans in a more natural way, which can make the task easier for humans [2]. Studies such as [2] and [5] have been conducted in order to relate the current state of emotions and feelings of a computer user while interacting with computer systems.
Abstract—the study is based on an experiment to measure the affective states of computer users via their use of mouse and keyboard. The experiment was replicated from a previous study by Khan et al., [5] resulting in significant correlations between the computer users pattern of interactions and their valence, arousal ratings. This study utilized the same data set from [5] and re-confirmed its validity by training Artificial Neural Networks (ANN). The data was divided into two portions for each individual. A portion to train ANN on his/her patterns of interaction and other portion to test the ANN. The study resulted in an average recognition rate of 64.72 % for valence and 61.02 % for arousal ratings. The highest recognition rates for individual participants’ valence and arousal were 100% and 87% respectively. These figures suggest that ANN is a bright prospect for the measurement of affective states of individual computer users via their interaction with keyboard and mouse. Keywords-component; Mood; affects; recognition; Neural Network; Keybaord; Mouse.
I.
INTRODUCTION
Natural and social interaction between humans and computers is one of the active research topics of the Human Computer Interaction (HCI). Reeves and Nass, [1] proposed that humancomputer interactions have identical principles like recognition, interpretation and expressions of emotions as found in human-human interactions. According to Zimmerman et al, [2], a computer system which can recognize affects can communicate with humans in a more natural way, which can make the task easier for humans. Studies have been conducted in order to recognize the state of feelings and emotions of a human while interacting with computer systems like Khan et al., [3]. This study further investigates the efficacy of using Artificial Neural Networks for mood recognition. II.
III.
Affect recognition in humans is a complex process and various methodologies have been applied to enable computers to recognize the affective states of its users. Various methodologies have been summarized in the following paragraphs based on the technologies utilized to solve the problem of human affective state recognition. A. Sensors based affective computing Hardware equipment such as Galvanic Skin Response (GSR) meters to monitor skin temperature, body resistance and sensors for Blood Pressure (BP), Electrocardiograph (ECG) etc. have been used by various studies to monitor bio signals of the participants. The bio signals recorded through the
BACKGROUND AND SIGNIFICANCE
The natural ability of human beings to easily recognize the emotional state of each other is what makes them communicate among themselves so easily and diversely. When a kid does something wrong, the parents usually get angry but a slight feeling of remorse in the attitude of the kid
978-1-4673-4886-7/12/$31.00 ©2012 IEEE
AFFECT RECOGNITION METHODOLOGIES
130
sensors are then processed to relate to various moods and affective states of the participants at a given time. Nasoz et al. [7] used wireless sensors to recognize emotional states based on physiological signals.
A. Two Dimensional Mood Measurement Model The Two-Dimensional Mood Measurement Model utilizes valence and arousal states of a human to measure his/her mood. It got its name because of the two dimensional representation of valence and arousal on X-axis and Y-axis to form a mood plotting model as proposed by Thayer [6]. Valence has been defined as the degree of delight or grief by Sanchez [7] while he defines arousal as the feeling of being active or passive towards external stimuli. Figure 1 shows the two dimensional plot of valence against arousal and the sample moods that could result from the plots i.e. low valence and high arousal suggests restlessness or fear where a person is highly active towards external stimuli and simultaneously grieved.
B. Image processing based affective computing Extensive studies have been performed in order to recognize the affective/emotional state of a person by processing his/her image/image sequences. According to Ekman and Friesen [10], emotions are related directly to facial expressions such as happiness, anger, sadness etc. Ekman argues that emotions can be recognized by the muscular motions of the face and presented their Facial Action Coding System (FACS) to code facial expressions as a combination of facial movements termed as Action Units (AU). Similarly, Sebe et al., [11] and Yacoob [12], have also utilized image processing techniques to recognize emotions from facial expressions. C. User Interaction based affective computing All of the above mentioned techniques require participants to wear some sensing device for data collection or generate feelings of observations. Although these techniques are said to be non-intrusive during the experiment but they cannot be adopted in daily life of a computer user. As already discussed in the previous section, there is significant evidence from the literature which shows that moods/emotional states can also be recognized by recording the patterns of interactions for a given computer user and it has been shown that these patterns have significant correlations when plotted in terms of valence and arousal to measure the emotional states of that user. Studies such as [2, 3, 4 and 5], have utilized or supported this method to recognize/measure a person’s affective states through his/her patterns of interactions with the computer system.
Figure 1: Valence Arousal Model with samples of plotted moods.
Self-Assessment Manikins Model (SAM) is one of the famous subjective mood measurement scales. SAM was proposed by Lang [8] as a scale used for the purpose of measuring a person’s mood through self-reporting of his/her feelings. SAM has already been utilized by various studies and has been reported as a viable solution for problems concerning self-assessment of emotional states [9]. SAM is a non-verbal and inexpensive way to measure affective states by selfreporting. SAM represents Pleasure-Arousal-Dominance (PAD) in the form of graphical characters on a nine point scale. The person reporting about his/her affective state can choose one character according to his/her emotional state. Figure 2 shows the SAM usage in this study for the subjective assessment of the participant’s mood.
This study considers the same methodology for mood recognition. It utilizes the data from the computer keyboard and mouse while the user interacts through these devices. The data from these devices has already been shown to have positive correlations with the moods of the user at a specified time [5]. This study will be using Artificial Neural Network to be trained on a participant’s interaction patterns associated with participant’s mood in terms of valence and arousal ratings. This trained network will then be used to measure the mood of the same user by classifying his/her interaction data from keyboard and mouse usage. The setup will be fairly simple and can be integrated in software that computer users use in their daily lives. IV.
MOOD MEASUREMENT MODEL
This study considered well-known model for measurement known as the Two-dimensional measurement model.
mood mood
Figure 1: Self-Assessment Manikin (adopted from [5]) 978-1-4673-4886-7/12/$31.00 ©2012 IEEE
131
V.
B. Efficient mood measurement Artificial Neural Networks is one of the most tested and efficient pattern recognition tools. It has been used in image/ pattern recognition algorithms as well as for machine learning purposes. The study aims at incorporating ANNs for learning the various mood patterns of a user and then recognizing that person’s moods while he/she is working on the system. No specialized hardware is required for such a setup and therefore it is a very cost effective mood measurement technique.
PATTERN RECOGNITION
This study investigates the application of ANN to the data set compiled by Khan, [5]. ANNs have been used for pattern recognition since long and they are considered to be very efficient pattern recognizers. For an ANN to recognize or classify a pattern it should be presented with the pattern as inputs (values) to its input layer. An ANN can have one or more than one hidden layers. Based on the learning algorithm used for that ANN it adjusts the weight values associated with each input from one layer to another. Basically, there are two types of pattern learning paradigms for ANNs. These paradigms have been briefly explained in the following paragraphs.
C. Mood recognition and remedy The study aims at developing a system which can measure mood of a computer user while he/she is interacting with the computer system. Usually, people carrying out the same routine tasks like using word processors, spread sheets and other office automation tools, get bored which may interfere with their pattern of interaction with the computer as suggested by Khan [4]. Hence their productivity / creativity may reduce because of their mood shifts. In such a scenario a mood measurement system can help refreshing the user by suggesting activities [5].
A. Supervised Learning In supervised learning, ANN is first trained using a training data set and then it is tested with inputs other than the training data set. The training data set consists of patterns somewhat similar to the ones for which the ANN is being trained. Usually the training data set is a selection of input patterns for which the output is already known. These inputs are presented one by one to the ANN through its input layer neurons and the output of the ANN is compared with the desired output for that specific pattern. If the output of the ANN and the desired output differ then the weights of the input layer and the hidden layers of the ANN are adjusted in order to match the two outputs otherwise next pattern is presented to the ANN as input. This process is repeated until all the patterns in the training data set are correctly recognized by the ANN or a considerably low error level in the recognition of the ANN has been achieved. Once the training of the ANN is complete it is presented with patterns which are not part of the training data set and its accuracy is measured.
Similarly, programmers and/or testers can utilize the ANN based mood recognition system embedded in the Integrated Development Environment (IDE) in which they work. The system will be able to recognize the emotional state of programmer or tester while they work and can help them in getting refreshed whenever they feel bored or frustrated. VII. RESEARCH METHODOLOGY The study considers the user interactions with computer system via keyboard and mouse. Such data from mouse and keyboard interactions has been shown to be positively correlated with moods of the user by Khan, et al., [5]. This study recorded the keyboard and mouse data of 26 participants (13 programmers and 13 non-programmers but computer users). Custom-made software was installed on each participant’s computer. The software logged various categories of key press events like capital characters, lower case characters, numeric keys and special characters etc. Similarly, for mouse, it recorded left or right button press and number of button press events in a given time slot along with the window name.
This study utilizes the supervised learning technique to train the ANN on a portion of data from a single participant’s entries and then test its accuracy on the remaining data for that particular user. A separate ANN has been trained for every participant because affective states for people with varying personality might differ. B. Unsupervised Learning In unsupervised learning for ANN, the desired output of the network is not known in advance and the network is not trained through a training data set. In fact, it is presented with the patterns straight forward and the network adjusts itself in such a way that similar input vectors are assigned to the same category or clusters. Examples of such networks are SelfOrganizing Maps (SOM) etc. VI.
For subjective self-assessment of mood and the type of activity performed at the time of assessment, the participants were presented with a manual mood rating dialogue at a specified time interval (3 minutes for this experiment). The mood rating dialogue included SAM scale for valence and arousal ratings along with options for the type of activity being performed by the participants such as watching movie or word processing etc. The ratings were logged along with the data entered by the participant about the application in which he/she was working at the time of rating or any entertainment (music or games) he/she was enjoying before the mood rating process. In the specific data set used for this study, the SAM dialogue was customized to appear at an interval of 3 minutes during the
AIMS AND OBJECTIVES
A. Natural/Social way of Communication The main aim of the study is to enable a computer to recognize the mood of its user in a more feasible and cost effective way and respond in a way which is more natural and social rather than preplanned response.
978-1-4673-4886-7/12/$31.00 ©2012 IEEE
132
interaction of a participant with computer. The keyboard and mouse interaction data (key presses, mouse clicks etc.) around the SAM dialogue entry in the log file were averaged to form one instance for that specific participant.
9
56
89.28
73.21
10
18
33.34
83.34
11
69
44.93
86.96
On the average 77 such instances were formed for each participant. Each key press or button press on the mouse has been logged as an event. For all participants the entries with less than 10 events were filtered out. After filtering the data, the remaining entries of data averaged to 53.5 entries per participant. From these entries, 70 percent were used to train an ANN for each participant’s arousal and valence ratings respectively. The remaining entries were used to test the accuracy of the trained ANNs.
12
65
76.92
49.23
13
30
60.00
53.34
14
62
77.42
67.74
15
66
53.03
46.97
16
69
89.85
81.16
17
46
78.26
76.09
VIII. DATA PREPARATION
18
57
70.17
15.79
19
45
80.00
28.89
20
31
38.71
74.20
21
46
32.61
58.70
22
58
39.65
62.07
23
59
59.32
57.63
24
66
56.06
66.67
25
32
75.00
50.00
26
68
55.88
55.88
Valence and arousal values that the participants recorded at each interval of time were on a 9 point SAM scale. For each participant a separate ANN was trained using the interaction data of the participant and the SAM rating recorded by the participant in one interval as a single instance of input and target output pair. In order to make the ANN testing process simple, these 9 points were divided into 3 distinct categories (1-3 for high valence and arousal, 4-6 as neutral and 7-9 as low valence and arousal). These three distinct categories were used to check the output of the trained ANN during the testing process. The number of accurate outputs has been presented as percentage value for each participant in table given below. IX.
RESULTS
X.
The current analysis shows an average accuracy of 64.72 percent of the ANN for valence recognition and 61.02 percent for arousal recognition. The table given below shows the overall results for each participant. The table shows participant id, number of logged entries and the percentage of accurate recognitions for valence and arousal by the respective ANN. By plotting the outputs of these trained ANNs for a single participant using the valence arousal model in figure I, the ANNs are able to recognize up to 9 different moods for that specific participant and showed an average accuracy of 62.87 percent.
The study suggests that ANN can be utilized for the measurement of affective state of a computer user by utilizing his/her patterns of interaction with the computer mouse and keyboard. The results of the study show a considerable amount of accuracy for the data collected for another study [5]. The results can further be refined by arranging a specialized experiment to log data of a computer user while controlling his/her mood through induced environmental conditions or emotions instead of subjective ratings of moods by the computer user. The study considered Feed-Forward ANN for training and testing on the interaction patterns of computer users. Other ANN like Back Propagation neural network can also be utilized to test whether the recognition rate increases or not.
Table I: ANN Valence and Arousal recognition accuracy for each participant Participant ID
Entries Logged
ANN accuracy (Valence)%
ANN Accuracy (Arousal)%
1
73
72.60
67.12
2
49
73.47
73.47
3
72
69.45
72.23
4
68
64.71
70.59
5
46
76.09
80.43
6
67
68.66
56.72
7
19
47.37
26.31
8
54
100
51.85
978-1-4673-4886-7/12/$31.00 ©2012 IEEE
CONCLUSION AND FUTURE WORK
REFERENCES
133
[1]
Reeves, B., & Nass, C. “The media equation: How people treat computers, television, and new media like real people and places,” Stanford, CA: CSLI Publications, 1996.
[2]
Zimmermann, P., Guttormsen, S., Danuser, B. & Gomez, P. “Affective Computing - A Rationale for Measuring Mood with Mouse and Keyboard,” Internat. Journal of Occupational Safety and Ergonomics. Vol. 9, 2003, pp. 539-551.
[3]
[4] [5]
[6]
Thayer, R. E., Newman, J. R. and McClain, T. M. “Self Regulation of Mood: Strategies for Changing a Bad Mood, Raising Energy, and Reducing Tension,” Journal of Personality and social psy-chology, vol. 67, 1994, pp. 910-925.
[7]
Nasoz, F., Alvarez, K., Lisetti, C. and Finkelstein, N. “motion recognition from physiological signals using wireless sensors for presence technologies,” Congition, Technology & work, vol, 6, 2004, pp. 4-14. Lang, P.J. “Behavioral treatment and bio-behavioural assessment,” computer applications, Sidowski, J. B., Johnson, J. H. & Williams, T. A.
[8]
(Eds). Technology in mental health care delivery systems. Norwood, 1980. [9] Bradley, M. M., Lang, P. J, “Measuring emotion: the Self assessment manikin and the semantic differential,” Journal of Behavior Therapy and Experimental Psychiatry, vol. 25, 1994, pp. 49-59. [10] P. Ekman, W. V. Friesen. “Facial Action Coding System: Investigator’s Guide”. Consulting Psychologists Press, 1978. [11] N. Sebe, I. Cohen, A. Garg, M. Lew, and T. Huang. “Emotion recognition using a Cauchy Naive Bayes classifier,” ICPR, 2002. [12] Y. Yacoob and L.S. Davis. “Recognizing human facial expressions from long image sequences using optical flow,” IEEE Trans. Patt. Anal. Machine Intell., 1996, pp. 636–642.
Khan, I.A., Brinkman, W.P., & Hierons, R.M. “Moods and Programmers Performance,” 19th Psychology of programming workshop, Joensuu, Finland, 2007, pp. 3-16. Khan, I., Brinkman, W., Hierons, R. “Do moods affect programmer’s debug performance?”. Cognition, Technology & Work, 2010, pp. 1-14. Khan, I., Brinkman, W., Hierons, R. “Towards a computer interactionbased mood measurement instrument,” 20th Psychology of Programming Interest Group Conference, Lancaster, UK, 2008.
978-1-4673-4886-7/12/$31.00 ©2012 IEEE
134
