guiding me in my life choices. ... the most important person during this period, that did not just supported me but also ..... Eight possible actors' body position [30].
P OLITECNICO DI M ILANO D EPARTMENT E LECTRONIC , I NFORMATION AND B IOENGINEERING D OCTORAL P ROGRAMME I N 2012
T HEATRE B OT: S TUDYING EMOTION PROJECTION AND EMOTION ENRICHMENT SYSTEM FOR AUTONOMOUS THEATRICAL ROBOT
Doctoral Dissertation of:
Julian Mauricio Angel Fernandez
Supervisor:
Prof. Andrea Bonarini Tutor:
Prof. Francesco Amigoni The Chair of the Doctoral Program:
Prof. Carlo Fiorini
Year 2016 Cycle XXVIII
Acknowledgment
During these last years, I have met people that have contributed in one way or another to arrive to this point of my life: to finish my PhD. Although I would like to mention all of them, the pages and my memory are not enough. But I would like to express all my gratitude to my parents to be there when I needed them. Even though they were not physical present during these years, they were always supporting me and guiding me in my life choices. Also I would like to thanks my advisor Professor Andrea Bonarini for the continues support of my PhD study and all his patience. Beside my advisor, I would like to thank Professor Lola Cañamero for her insightful help and guidance during my period abroad in her lab. Last but not least, I would like to thank the most important person during this period, that did not just supported me but also was there in all the difficult times, and that I have to say that without her support I would not be where I am right now, and this person is Valentina Caradonio.
1
Abstract
Robots should be able to express emotional states to interact with people as social agents. Emotions are a peculiar characterization of humans and usually conveyed through face expression or body language, however there are cases where robots cannot reproduce anthropomorphic shape: they have to perform tasks which require a specific structure: this is the case of home cleaners, package carriers, and many others platforms. Therefore, emotional states need to be represented by exploiting other features, such as movements and shape changes. Therefore the work presented in this thesis studies emotion expression in non-anthropomorphic platform and how it is perceived by human: a set of case studies, aimed to identify specific values to convey emotion through changes in linear and angular velocities, that might be applied on different non-anthropomorphic platforms were done. The work takes into account some of the most considered emotion expression theories and human emotion coding. The results show that people can recognize some emotional expressions better than others, so some guidelines are proposed to generate emotions exploiting only movements. In parallel to these studies, it was developed an emotional enrichment system, which let the parametrization, modulation and expression of emotions through the change of actions’ parameters and addition of required actions. This system was used in the last two cases studies, where the emotions elicited by the robot were totally generated by it. Additionally a simple test were done with other platforms to test their interoperability . Moreover, it was explored the possibility to use robots in theatre to study further social capabilities in robotics. For this reason has been designed a specific software architecture which includes the emotional enrichment system and facilitate the use of robots in theatre.
I
Riassunto
I robot devono essere in grado di esprimere gli stati emotivi interagendo con le persone in un contesto sociale. L’espressione di emozioni è una caratteristica peculiare dell’essere umano, generalmente veicolata attraverso l’espressione del viso o il linguaggio del corpo. Nella maggior parte dei casi, però, i robot non presentano forma antropomorfa poichè devono svolgere compiti che richiedono una struttura specifica. Pertanto, gli stati emotivi hanno bisogno di essere rappresentati mediante altre caratteristiche, quali i movimenti e cambiamenti di forma. Il presente lavoro presenta una piattaforma robotica non antropomorfa, si concentra sulle modalità con cui può esprimere emozioni e come esse sono percepite e riconosciute dall’uomo. Di conseguenza sono stati svolti un insieme di casi di studio, i quali hanno analizzato e individuato i valori specifici da assegnare a cambiamenti di velocita lineare e angolare di piattaforme non antropomorfe per trasmettere certe emozioni. Il lavoro prende in considerazione le pi’note teorie sull’espressione di emozioni note e alcuni studi sulla codificazione delle emozioni umane. I risultati mostrano che alcune emozioni possono essere riconosciute facilmente dall’uomo ed è possibile definire delle linee guida da seguire per poter generare emozioni sfruttando unicamente i movimenti. Parallelamente a questi studi, è stato sviluppato un Emotional Enrichment System, che permette la parametrizzazione, la modulazione e l’espressione delle emozioni attraverso il cambiamento dei parametri di azioni e l’aggiunta di nuove azioni. Questo sistema è stato utilizzato nel corso degli ultimi due casi di studio durante i quali le emozioni espresse dal robot sono state generate autonomamente dal sistema. Inoltre è stato realizzato un test per potere valutare l’interoperabilita tra le diverse piattaforme. Da ultimo, è stata analizzata la possibilità di utilizzare i robot nel teatro per studiare ulteriormente le capacità sociali nel campo della robotica. Per questo motivo è stato progettato un’architettura software specifica che include l’Emotional Enrichment System e facilita l’uso di robot nel teatro.
III
Contents
1. Introduction 1.1. Main Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2. Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9 11 12
2. Theoretical Framework 2.1. Theatre . . . . . . . . . . . . . . . 2.1.1. Characteristics . . . . . . . . 2.1.2. Constraints . . . . . . . . . . 2.1.3. Actor Lessons . . . . . . . . 2.2. Emotional Models . . . . . . . . . 2.2.1. Circumplex Model of Affect 2.2.2. Tomkins’ Theory . . . . . . 2.3. Performance Robotics . . . . . . . 2.3.1. Interacting with the audience 2.3.2. No Audience Interaction . . . 2.4. Studies on Emotion Expression . . 2.4.1. Laban Effort System . . . . . 2.4.2. Human Studies . . . . . . . . 2.4.3. Robotic Studies . . . . . . . 2.5. Emotion Selection Systems . . . . . 2.6. Summary . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
13 13 13 14 14 15 17 17 18 18 19 20 21 22 22 27 29
3. Platform 3.1. Holonomic Platforms . . . . . . . . . . . 3.1.1. Kinematics . . . . . . . . . . . . . 3.2. Prototype . . . . . . . . . . . . . . . . . 3.2.1. Electronics and Mechanical Design 3.2.2. Software . . . . . . . . . . . . . . 3.3. Version 1.0 . . . . . . . . . . . . . . . . 3.3.1. Hardware Changes . . . . . . . . . 3.3.2. Software Changes . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
31 32 33 34 34 34 39 39 39
V
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
Contents
3.4. Version 1.1 . . . . . . . . . . . . . . . . 3.4.1. Hardware and Mechanical Changes 3.4.2. Software Changes . . . . . . . . . 3.5. Version 2.0 . . . . . . . . . . . . . . . . 3.5.1. Hardware and Mechanical Changes 3.5.2. Software . . . . . . . . . . . . . . 3.6. Summary . . . . . . . . . . . . . . . . . 4. Architecture 4.1. Simple and Compound Actions 4.2. Emotional Enrichment System . 4.3. Emotional Model . . . . . . . . 4.4. Action Decision . . . . . . . . . 4.5. Description . . . . . . . . . . . 4.6. Summary . . . . . . . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
39 39 42 42 45 45 47
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
49 51 53 53 53 54 56
5. Action Modulation 5.1. First Emotional Enrichment System . 5.2. Emotional Execution Tree . . . . . . 5.2.1. Simple and Compound Actions 5.2.2. Nodes Types . . . . . . . . . . 5.3. Emotional Enrichment System . . . . 5.3.1. Design . . . . . . . . . . . . . 5.3.2. Emotion Enrichment Process . 5.4. Summary . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
57 58 58 59 60 60 60 62 63
. . . . . .
. . . . . .
6. Emotion Selection 6.1. Stimulation Calculator . . . 6.2. Emotion Generator . . . . . 6.3. Design and Implementation 6.4. Results . . . . . . . . . . . 6.5. Summary . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
65 66 68 69 69 71
7. Pilot and First Case Study 7.1. Design and Setup . . 7.2. Emotion Description 7.3. Study . . . . . . . . 7.4. Results . . . . . . . 7.5. Summary . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
73 73 74 75 75 77
. . . . .
79 79 79 82 82 84
. . . . .
. . . . .
8. Second Case Study 8.1. Design and Setup . . . . 8.1.1. Scene Description 8.2. Study . . . . . . . . . . 8.3. Results . . . . . . . . . 8.4. Summary . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
VI
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Contents
9. Third Case Study 9.1. Design . . . . . . . . 9.2. Emotions Description 9.3. Study . . . . . . . . 9.4. Results . . . . . . . 9.5. Summary . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
10. Experiment 10.1.Design . . . . . . . . . . . . . . . . 10.1.1.Variables . . . . . . . . . . . 10.1.2.Independent Variables Values 10.1.3.Emotion Sequences . . . . . 10.1.4.Setup . . . . . . . . . . . . . 10.1.5.Tools . . . . . . . . . . . . . 10.1.6.Procedure . . . . . . . . . . 10.2.Study . . . . . . . . . . . . . . . . 10.3.Results . . . . . . . . . . . . . . . 10.4.Summary . . . . . . . . . . . . . . 11. Fourth Case Study 11.1.Design and Setup . . . . . 11.1.1.Emotion Perception 11.1.2.Scene . . . . . . . . 11.2.Description . . . . . . . . 11.2.1.Emotion . . . . . . 11.2.2.Scene . . . . . . . . 11.3.Study . . . . . . . . . . . 11.4.Results . . . . . . . . . . 11.5.Summary . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . .
. . . . .
87 87 88 89 89 93
. . . . . . . . . .
95 95 96 97 97 97 99 99 100 100 109
. . . . . . . . .
111 111 111 113 113 113 113 116 116 119
12. Case Studies Discussion
121
13. Conclusions and Further work
125
A. Experiment’s Additional Information A.1. Treatments Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2. Participants’ Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . A.3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
129 129 129 129
B. Fourth Case Study Additional Information B.1. Sequence Design . . . . . . . . . . B.2. Kalman Filter Implementation . . . B.2.1. Kalman Filter . . . . . . . . B.2.2. Implementation . . . . . . . B.3. Scene’s script . . . . . . . . . . . .
143 143 143 143 146 146
Bibliography
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
153
VII
List of Figures
2.1. 2.2. 2.3. 2.4. 2.5.
Stage division used by directors to give instructions [49]. . . . . . . . . Eight possible actors’ body position [30]. . . . . . . . . . . . . . . . . Mapping of emotion theories with the emotion elicitation phases [94]. . Russel’s circumplex model of affect [90]. . . . . . . . . . . . . . . . . Patterns for surprise-startle, fear-terror, interest-excitement, anger-rage, distress-anguish and enjoyment-joy, after Tomkins [48]. . . . . . . . .
14 15 16 17
3.1. Holonomic wheel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2. Holonomic platform with three motors in a Cartesian plane. m1 , m2 and m3 represent the motors. . . . . . . . . . . . . . . . . . . . . . . . . . 3.3. a) Platform’s base design with its respective measurements: L = 110mm, R = 35mm, side = 120mm. b) Prototype’s second layer design, seen from the top. c) Prototype’s design seen from the left side. . . . . . . . 3.4. Prototype based on the design depicted in Figure 3.3. . . . . . . . . . . 3.5. General distribution and communication scheme among the software components running on the Arduino. . . . . . . . . . . . . . . . . . . . 3.6. Design of the first version. a) Base platform, including wheel protectors and the foam. b) First layer, including battery slot. c) Second layer with the servos’ distribution. d) Lateral view of this version. . . . . . . . . . 3.7. a) Back of the platform without foam, each red arrow shows the movement of beams. The movement is controlled independently. b) Front of the platform without the blue cloth. The red rectangle highlights the space of the body that is moved, and the red arrow shows that is possible to move it in both directions. . . . . . . . . . . . . . . . . . . . . . . . 3.8. First version with the blue cloth. . . . . . . . . . . . . . . . . . . . . . 3.9. Final interface used to program the emotions conveyed by the robot. . . 3.10.Design of the 1.1 version. a) Base platform, this layer is used to carry the battery or batteries. b) First layer, which includes the Arduino and the H-Bridges to control the motors. c) Second layer with the OdroidU3. d) Third and last layer, with the servo motors distribution. e) Lateral view of this version. . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
1
18
33
35 36 38
40
41 41 42
43
List of Figures
3.11.Platform with changes in the upper part and addition of the aluminium base. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.12.Graphical interface used to control the platform version 1.1. . . . . . . 3.13.Third version of the platform. a) Platform without foam. The blue acrylic is used to give some structure to the upper part to improve the shape’s change. b) Platform with foam. . . . . . . . . . . . . . . . . . 3.14.Design of the third version. a) Base platform, this layer is used to carry the batteries. b) First layer, which includes the Arduino and the HBridges to control the motors. c) Second layer with the Odroid-U3 and the mechanical structure to support the upper part. d) Upper part, with the servo motors distribution. e) Lateral view of the version. . . . . . . 3.15.UML class diagram of Arduino’s Due code. . . . . . . . . . . . . . . . 3.16.General distribution and communication scheme among the diverse software components running on the whole system. . . . . . . . . . . . . . 4.1. TheatreBot’s architecture is composed of six sub-systems: Feature, Belief, Decision, Motivation, Description, and Emotion Enrichment. The full arrows represent the information that has to be used by the module. Dashed arrows mean that one model may influence the other. The influenced module can either accept or reject the suggestion given by the influencing one. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2. Desired positions to move in the stage. Where U means up, D down, R right, C center, and L left. These positions correspond to the 9 positions defined in theatre on stage. . . . . . . . . . . . . . . . . . . . . . . . . 4.3. Example of a compound action that is composed by another compound action (walk) and a simple action (speak). . . . . . . . . . . . . . . . . 4.4. Script example that shows possible flows of beat (B) with some alternative compound actions (CA), and a set of compound actions. If, for some reason, no compound action can be selected, a Panic action has been defined to try to overcome the problem. The dash arrows show that all beats or compound actions are connected with the panic action. . . 5.1. Emotional Execution Tree’s Model. The two enumerations restrict the values that could be given to the type of context and priority. . . . . . . 5.2. General system design. Each simple action corresponds to one ROS node, and there is just one node for the emotion enrichment system. The ovals represent the ROS topic parameters, rectangles represent black boxes, and texts outside containers represent input files that contain the system parametrization. . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1. General architecture of the system. The arrows show the information flow. The time difference between two images is used in the stimulation calculator module to calculate the stimulation. . . . . . . . . . . . . . 6.2. Behaviour of the increase function for different delays in the image using parameters base_increase = 30, d = 0.1, and increase_f actor = 10. . 6.3. Behaviour of the decrease function for decrease_f actor = −0.5 and different time delays. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
44 44
45
46 47 48
50
52 52
54 62
62
66 67 68
List of Figures
6.4. Emotional System UML Class Model. . . . . . . . . . . . . . . . . . . 6.5. Stimulation (continuous line) and events (dots in horizontal lines) obtained from the continuous comparison of two consecutive images. The y-axis on the left represents the stimulation level, while the one on the right represents the events generated from the slope analysis. . . . . . . 6.6. Intensity obtained by our system for the four emotions implemented: fear (blue), interested (purple), surprise (red) and relief (yellow). . . . .
69
70 70
7.1. Example of the sequence of movements done for the robot in its bottom part. The upper shows a sequence without any angular movement for a total of xmeters. The bottom shows the same displacement, but this time with an oscillation between [θ, −θ] and angular velocity ω . . . . . 7.2. First case study setup. . . . . . . . . . . . . . . . . . . . . . . . . . . .
74 75
8.1. 8.2. 8.3. 8.4. 8.5. 8.6. 8.7. 8.8. 8.9.
. . . . . . . . .
80 80 81 81 81 81 81 82 82
9.1. Setup of the third case study. . . . . . . . . . . . . . . . . . . . . . . .
88
10.1.Holonomic platform used in the experiments. . . . . . . . . . . . . . . 10.2.Example of the features used in the experiment. x represents the displacement in meters, ω is the angular velocity (rad/s) and θ the oscillation of the body around its center (rad). The upper sequence depicts a movement based only on linear velocity, while the bottom one shows a sequence with angular and linear movement. . . . . . . . . . . . . . . 10.3.Combination of direction and orientation. The crosses symbolize the possible initial points. a) Direction = 0 and Orientation = 0. b) Direction = 0 and Orientation = π. c) Direction = π and Orientation = π. d) Direction = π and Orientation = 0. e) Direction = −π and Orientation = 2 0. f) Direction = −π and Orientation = π . . . . . . . . . . . . . . . . . 2 10.4.Setup for the experiment. The crosses symbolize the possible starting points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5.Interface used in the experiment. Once a sequence is selected, the interface shows sequence’s values. Also, the interface give information about the current position of the robot and its velocity. . . . . . . . . . . . . . 10.6.Example of the questionnaire used in the experiment. . . . . . . . . . . 10.7.Alpha values for each treatment. . . . . . . . . . . . . . . . . . . . . . 10.8.Anger α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants. . . . . . . . . . . . . . . . . . . . .
95
Setup of the second case study. . . . . . . . . . Sequence of actions for the angry scene. . . . . Sequence of actions for the curiosity scene. . . Sequence of actions for the disgust scene. . . . Sequence of actions for the embarrassed scene. Sequence of actions for the fear-1 scene. . . . . Sequence of actions for the fear-2 scene. . . . . Sequence of actions for the happiness scene. . . Sequence of actions for the sadness scene. . . .
3
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
96
98 99
99 101 103
104
List of Figures
10.9.Excitement α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants. . . . . . . . . . . . . . . . . . 10.10. Happiness α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants. . . . . . . . . . . . . . . . . . 10.11. Sadness α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants. . . . . . . . . . . . . . . . . . . . . 10.12. Fear α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants. . . . . . . . . . . . . . . . . . . . . 10.13. Tenderness α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants. . . . . . . . . . . . . . . . . . 10.14. Other α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants. . . . . . . . . . . . . . . . . . . . .
104
104
105
105
105
106
11.1.Lower part of the platform’s version 2.0 used in the last case study. . . . 11.2.Environment setup for the fourth case study. . . . . . . . . . . . . . . . 11.3.Stage discretization used for the small scene. The blue squares correspond to the each zone, while the numbers correspond to the ID given to each zone. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4.Sequence of movements done by the robot. The red arrows show the trajectory done by the robot, while the numbers show the order among the movements. a) The first ten movements b) The last five movements 11.5.Graphical interface used to communicate with the Emotional Enrichment System. a) It is the interface used to send the actions sequence. b) It is the interface used to send a emotion and its intensity. . . . . . . . . 11.6.Fourth case study setup during the Researchers’ Night 2015. . . . . . .
115 116
B.1. General distribution and communication scheme among the diverse software components running on the whole system, including Kalman filter.
147
4
112 112
114
115
List of Tables
2.1. Comparison among the works done on performance robotics. Where NA means "it does not apply" and NS "not specified". . . . . . . . . . . 2.2. Comparison among works on emotion projection in robotics. NA = Not Available . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21 28
3.1. Comparison of the platform available when the decision of the platform was done. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
32
4.1. Description of the eight simple actions implemented, and their respective parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51
7.1. Features that could be perceived by the audience, their modalities. Where Embarr. is Embarrassment and Asy. is Asymmetric. . . . . . . . . . . . 7.2. Answers obtained the first case study. On each row is the emotion that was intended to express, and on the columns the reported emotions. . . 7.3. Pairwise comparison among all the implemented emotions in the first case study using Fisher’s exact test with α = 0.05. The * indicates that the p-value was adjusted using the Holm-Bonferroni correction for multiple comparisons. . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4. Example of table compiled for each emotion on the subjects that have been presented each emotion (here Happiness). . . . . . . . . . . . . . 7.5. Classification accuracy of the presented emotions by the single panels, computed as mentioned in the text, with corresponding 95% confidence interval, no-information rate, and p-value that accuracy is greater than the NIR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1. Answers obtained in the second case studies. On each row is the emotion that was intended to express, and on the columns the reported emotions. 8.2. Comparison of percentage of correct emotion perception in the first and second case study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
76 76
77 77
78 83 83
List of Tables
8.3. Classification accuracy of the presented emotions by the single panels, computed as mentioned in the text, with corresponding 95% confidence interval, no-information rate, and p-value that accuracy is greater than the NIR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1. Parameters for the implemented emotions, and general characteristics used to come up with the parameters, with relative references. The description of the features are the same as it was described in the original source. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2. Results obtained from the open questionnaire. . . . . . . . . . . . . . . 9.3. Answers obtained in the case study. . . . . . . . . . . . . . . . . . . . 9.4. Classification accuracy of the presented emotions by the single panels, computed as mentioned in the text, with corresponding 95% confidence interval, no-information rate, and p-value that accuracy is greater than the NIR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5. Pairwise comparison among all the implemented emotions using Fisher’s exact test for both questionnaires with α = 0.05. The * indicates that the p-value was adjusted using the Holm-Bonferroni correction for multiple comparisons. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1.Possible values for each one of the independent variables. . . . . . . . 10.2.Participants’ Country of origin. . . . . . . . . . . . . . . . . . . . . . . 10.3.Participants’ Career. . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4.Example of the table generated for each treatment. This table corresponds to the treatment with ID = 1. . . . . . . . . . . . . . . . . . . 10.5.Treatment top 10 for Happiness. This top list was generated by ordering from highest to lowest first by Happiness mean, then by Happiness α, and finally by Treatment α. . . . . . . . . . . . . . . . . . . . . . . . . 10.6.Top 10 treatments for Anger. This top list was generated ordering from highest to lowest first byAnger mean, then byAnger α, and lastly by Treatment α. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.Treatment top 10 for Fear. This top list was generated ordering from highest to lowest first byFear mean, then byFear α, and lastly by Treatment α. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.8.Treatment top 10 for Sadness. This top list was generated ordering from highest to lowest first bySadness mean, then bySadness α, and lastly by Treatment α. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.9.Example of of the contingency table generated for the treatment ID = 1 and happiness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.Parameters’ values selected from the experiment. . . . . . . . . . . . . 11.2.Summary of the answers obtained in the case study. . . . . . . . . . . . 11.3.Pair comparison among all the implemented emotions using Fisher’s exact test for both questionnaires with α = 0.05 for the fourth case study. The * indicates that the p-value was adjusted using the Holm-Bonferroni correction for multiple comparisons. . . . . . . . . . . . . . . . . . . . 6
85
88 90 91
91
92 97 100 102 103
106
107
107
108 108 114 117
117
List of Tables
11.4.Accuracy, precision and results of Pearson’s χ2 for each contingency matrix with α = 0.05 for the fourth case study. . . . . . . . . . . . . . 11.5.Classification accuracy of the presented emotions by the single panels, computed as mentioned in the text, with corresponding 95% confidence interval, no-information rate, and p-value that accuracy is greater than the NIR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6.Answers obtained for the small scene. . . . . . . . . . . . . . . . . . . 12.1.Emotions implemented and enlisted in first, third and fourth case study. 12.2.Features used in the implementations of Anger, Fear, Happiness, and Sadness for the four case studies. . . . . . . . . . . . . . . . . . . . . .
118
118 119 122 123
A.1. Treatments generated from the combination of the independent variables’ possible values. Part 1. . . . . . . . . . . . . . . . . . . . . . . A.2. Treatments generated from the combination of the independent variables’ possible values. Part 2. . . . . . . . . . . . . . . . . . . . . . . A.3. Treatments generated from the combination of the independent variables’ possible values. Part 3. . . . . . . . . . . . . . . . . . . . . . . A.4. Treatments generated from the combination of the independent variables’ possible values. Part 4. . . . . . . . . . . . . . . . . . . . . . . A.5. Treatments generated from the combination of the independent variables’ possible values. Part 5. . . . . . . . . . . . . . . . . . . . . . . A.6. Treatments generated from the combination of the independent variables’ possible values. Part 6. . . . . . . . . . . . . . . . . . . . . . . A.7. Treatments generated from the combination of the independent variables’ possible values. Part 7. . . . . . . . . . . . . . . . . . . . . . . A.8. Sequence of treatments generated for each participant. Where P ∗ is the participant’s number and S ∗ is the trial number. . . . . . . . . . . . . . A.9. Mean of the intensity for each emotion for all the treatments. Part I. . . A.10.Mean of the intensity for each emotion for all the treatments. Part II. . . A.11.Mean of the intensity for each emotion for all the treatments. Part II. . . A.12.Mean of the intensity for each emotion for all the treatments. Part III. . A.13.Mean of the intensity for each emotion for all the treatments. Part IV. .
136 137 138 139 140 141
B.1. List of the emotions presented to each group of participants and the order of the scene. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
144
7
129 130 131 132 133 134 135
CHAPTER
1
Introduction
Emotions and mental states are not just expressed through facial expression but also by body through postures and other features [25]. Many psychological studies have been focused on understanding the role of human face in emotion projection [32], [50] and mental states. This trend has been followed by the robotics community, where anthropomorphic faces (e.g., [33], [14]) and bodies (e.g., [9], [42], [27]) have been used to convey emotions. However in many situations the presence of anthropomorphic elements would be out of place and not justified by the main robot’s functionalities. Most of the current and future robotics platforms on the market will not require anthropomorphic faces [14] or limbs [64] , in some cases like, for instance, in floor cleaning robots, anthropomorphic characteristics could even be detrimental to robot’s task accomplishment. This generates the necessity to study other mechanisms that could help to project emotions, which could give people an idea about the robots’ state, and engage the user in long term relations. Has been noticed that the amount of works studying non-anthropomorphic features in robotics (e.g., [59, 78, 91, 95]) remains still small in comparison to those exploiting anthropomorphic features. Moreover, these works do not give specific range of values for the characteristics used to expressed emotions implemented. For example Suk and collaborators [76] in their study give specific values for acceleration and curvature, but their connection to specific emotions is not explicited. Rather, they establish a relationship between their values and pleasure/arousal dimensions. This could help to select specific features to convey certain emotions, but it would be even better to know the precise range of values to be assigned to each characteristics to avoid misinterpretation with wrong emotions. These values could be used in social robotics to show emotions and as consequence increase their acceptance. Robots that interact with humans, or social robots, indeed, do not only have to per9
Chapter 1. Introduction
form an assigned tasks, but they should also be accepted by humans. Building and testing a robot that have the following capabilities implies many challenges that should be faced simultaneously: the robot must identify objects and people, detect people’s intentions and emotions, and also have the expected social characteristics. The complexity of the environment, though, makes difficult the construction of a robot that satisfactorily performs in all the conditions. As a consequence, the use of a setting that enables the study of social interactions without losing the intrinsic dynamism in social interaction is needed. Emotion projection to the whole audience is the main component in theatre, actors must relay on their body, and not just in their faces to convey emotions to the whole audience [104]. Like theatrical actors, also social robots must act as they can interact socially with their audience and comfortably deal with them showing empathy (e.g. expressing emotions). Many studies have shown the relationship between the social presence projected by a robot and the grade of people’s acceptance [43], which increases when robots show high social presence. Social interactions are concerned with timing [45], showing intentions and affection [3, 59], and nonverbal communication [35, 52]. Theatre has many of the desired characteristics that allow researchers to study social interaction. The kind of situations that theatre tries to emulate are analogous to those where robots have to show social capabilities in service applications. As once said by Eugenio Barba, the Odin Theatre’s co-founder: "I love theatre since illusion repels me." [7]. In a theatre setting an actor knows in advance most of the social aspects that should be detected in a real world setting, and for this reason can act in a realistic way without need to simplify the interaction, as it often happens in real life, where most of the information available a priori in theatre, should be collected from interaction. Although the creation of a theatrical robot could be used in the entertainment industry, it also provides the possibility to study other characteristics, intrinsic to theatre, thus also to interpersonal interaction, such as timing, expressiveness and adaptiveness to situations. For this reason, it is not surprising that different researchers suggest theatre as a good place to test these capabilities [13,45,65,67,82]. Despite this, just few works have been developed by using theatre as environment [16,46,54,55,57,63,65,66,81,99,105]. Yet, these works have been focused on the use of theatre as settings where entertainment robots operate without implementing abilities to show autonomously emotion or social behavior. This thesis focuses on the identification of movement features that could be considered platform independent and therefore used to express emotions with a wide range of robots. It has been devised a very simple robot base, with no resemblance to humans or animals, to do diverse case studies which aim to identify labels that participants give to specific platforms’ movements. To automatize emotional expression and to let others researcher to embed emotion expression in their systems, it has been designed and implemented an emotional enrichment system, which blends actions with emotions through changes in actions’ parameters. These changes are described in files that are read by the system each time that is lunched to enable the possibility to make modification in the emotions’ description easily. To use the emotional enrichment system in a real context, a software architecture has been designed with the objective to enable robots to participate in theatrical performances with humans. The architecture is based 10
1.1. Main Contributions
on the actors’ training methods and takes advantage of the constraints given by theatre to reduce the complexity of some modules. The architecture has been designed as modular to enable the possibility to reuse modules, such us reactive emotion selection system, in other settings.
1.1 Main Contributions The main contributions of the present work are listed as follows. Emotional enrichment system. The Emotional Enrichment System (EES) modifies actions’ parameters and/or add actions to create the illusion of emotion expression in a robot. Although the EES was originally conceived to be used in an autonomously performing robot [4], its design was devised to make it extensible to other platforms and adaptable to new tasks. To achieve this goal, the system relies on an Emotional Execution Tree (EXT), including simple actions, sequential and parallel nodes. Additionally, the concept of compound action has been defined to group a bunch of nodes, which reduces the tree dimension and enables to reuse recurrent actions generated by specific combinations of simple actions and other nodes. Reactive emotional selection system. The reactive emotional system uses sensory data to select an emotion based on Tomkins’ theory [48, 98], the pre-selected emotions compete among them to be triggered. The system has been designed in a modular way, so to make it easy to combine it with other, more complex models such as the one suggested by Izard [47]. Theatrical architecture. A software architecture has been designed to allow robots to act with humans in theatrical plays. The architecture is based on theatre constraints and requirements and actors’ lessons to enable emotion expression, action selection without any need to make them explicit in the script, and also the architecture as thought as an extension of current action decision systems. This makes the architecture extensible to other fields, such as assistant robots, or robot games [72]. On the other hand, the script description uses theatre’s constraints to enable the robot to autonomously shift among emotional states which are shown as decided by the robot. Study on the contribution of movement features, such as linear and angular velocity, and oscillation angle, to express emotions. Four case studies and one experiment were performed with a non-anthropomorphic platform to study how the selected features can contribute to the perception of different emotions. The platform was created with the solely purpose to not resemble humans nor animals, and improved after the lessons learned in each case study. The results obtained from these case studies confirm the findings of others researchers in expressing emotions with non-anthropomorphic platforms , such as Saerbeck and Christoph [91], which found that acceleration is correlated with the perceived arousal. There are two main differences between these case studies and the previous works done on emotion expression. (i) These case studies were done during exhibitions, where people with different background and technology fluency participated. (ii) The values used in the emotions’ implementation are reported. 11
Chapter 1. Introduction
1.2 Thesis Outline Chapter 2 outlines the main concepts used in the thesis, which range from basic introduction to theatre characteristics to emotional models. Also, it describes relevant related work on performance robots, emotion expression and emotion selection. Chapter 3 introduces the platform, starting from the prototype and passing through all the diverse versions until the last version. For each version, including the prototype, measures, hardware and software details are provided. Chapters 4, 5, and 6 describe the theatrical architecture and its components. Chapter 4 introduces the general architecture and describes all the components without digging in all the details. Chapter 5 explains in more detail the Emotional Enrichment System and its implementation as well its interface with ROS [87] . The emotional selection system is described in the Chapter 6. Chapters 7, 8, 9 , 10 and 11 depict the four case studies and the experiment. For each one on these, design, setting, procedure, and obtained results are presented. Chapter B concludes the thesis, reviewing obtained results and outlining future improvements. Appendix A reports details about the experiment, Chapter 10. Appendix B explains the Kalman filter implementation used in the fourth case study, Chapter 9, as well the information used in this case.
12
CHAPTER
2
Theoretical Framework
2.1 Theatre Theatre is considered as lively art [104]. Thanks to its characteristics and constraints, it is an excellent framework to test sociability and expressiveness in robots; moreover, actor training systems (e.g., [21, 38, 96, 104] can inspire the development of expressive robots. In this section, are presented some of the relevant aspects of theatre that were considered in this work. 2.1.1.
Characteristics
People believe theatre is a repetitive show, but they forget about essential points that make theatre a lively art are often forgotten [104]: Opposite to television or movies, during a theatre performance actors do not have a second chance to perform in front of the same audience. If an actor fails remembering a line, or he/she does not show a believable character, the audience will get a bad impression of the play. Each performance is unique. No matter how much effort actors put to repeat each time the same performance, subtle changes could been perceived: actors’ and objects’ stage position, actors’ mood and, more importantly, audience’s attitude. Audience’s attitude influences actors. Actor could hear laughs, coughs, silence, and could even feel the tension in the audience. This could eager or discourage actors, affecting the whole performance. The performance outcome does not rely on one person. Good outcome comes from correct collaboration and coordination of playwriter, director, technical peo13
Chapter 2. Theoretical Framework
Figure 2.1: Stage division used by directors to give instructions [49].
ple and performers. In the specific performers’ case, they must work as a unity and show to the public a coherent story. Therefore, theatrical robot actors should have abilities similar to those of their human counterpart to be considered as actors and not as props. This makes it necessary that robot actors be expressive, social, and with enough autonomy to be able to face anything may happen on stage. 2.1.2.
Constraints
Theatre’s constraints make possible to focus research efforts on emotion expression and social behavior, considering that robots have already at least a general idea of what to do and how. These constraints are: The play script contains all the necessary information: actions, coordination cues, dialogues, and characters attitude. Since the script is known before any representation, rehearsals can be done to get used with objects’ and performers’ positions. The stage space is discretized to facilitate directors to give instructions, and actors to remember their positions (Figure 2.1). Actors should basically take one out of eight preset orientations during a performance (Figure 2.2). 2.1.3.
Actor Lessons
Human actors are demanded to do some actions that are beyond script descriptions as [21]: Actors must not underact neither overact. Underact makes audience lose interest about the play and may turn it boring. On the other hand, overact produces audience’s interest to focus on one actor, and forget others. 14
2.2. Emotional Models
Figure 2.2: Eight possible actors’ body position [30].
Actors must project emotions to the whole audience, not just to the front rows. A good actor should be able to project emotions primarily using his "magic triangle" which is the part of the body between the two shoulders and his belly button. "The audience sees everything" [21]. Although the actor could not be in a principal character during a scene, some people in the audience might look at him/her. Thus, actors should always do something, and this should look realistic and coherent. Sometimes actors are on the stage without an active role, but they still have to be there as characters, with their emotions and features. A character has a rhythm and tempo, which are established during rehearsals. The rhythm is related to the way the character moves at any time: it might be fast or slow, nimble or brisk, etc. The tempo is how the character changes within his rhythm based on the stimuli it receives.
2.2 Emotional Models There is not a commonly agreed definition of emotion [18, 83], however it is known that emotions are responses to stimuli and that very often they motivate actions. Emotions help humans as a mechanism for preservation and socialization. For instance, fear is one of the most common emotions related to self-preservation: when people are in dangerous situations, fear enable our body to get ready to run from the dangerous source. At the same time, emotions are used as social regulators. When two people are talking, both must show positive emotions to make the talk continue smoothly and effectively. Moreover, if someone has problems to assess the emotional state of others, he may act in a way that others may consider inappropriate, and he might be socially banned for his "misbehaviour". Emotions’ role in humans have attracted a variety of people to come with a theory about how emotions are triggered in humans. These theories differ in assumptions and in the components involved in the process. They can be classified in different ways. For example [71, 94] use the following categories: 15
Chapter 2. Theoretical Framework
Figure 2.3: Mapping of emotion theories with the emotion elicitation phases [94].
Adaptational: based on the idea that emotions are an evolving system used to detect stimuli that are of vital importance. Dimensional: organize emotions according to different characteristics, usually valence (pleasantness-unpleasantness) and arousal. One of the most widely used is the Russel’s circumplex model of affect [90]. Appraisal: argues that emotions arise from the individual’s judgment, based on its believes, desires, and intentions with respect to the current situation. This is a predominant theory among the others primarily because it makes explicit the link between emotion elicitation and response. EMA [40] and Fatima [29] frameworks fall in this category. Motivational: studies how motivational drives could generate emotions. Some theories then argued that emotion is a result of evolutionary needs that should be satisfied. Circuit: supports the fact that emotions correspond to a specific neuron path in the brain. Models that fall in this area use the assumption that emotions’ differentiation and the number of emotions are genetically coded as neural circuits. Discrete: are theories based on Darwin’s work, the expression of emotion in man and animals. These theories use as a pillar the idea of the existence of a basic emotions such as anger, fear, joy, sadness, and disgust. One of its most recent theorists is Paul Ekman, who postulated the existence of six basic emotions (anger, fear, happiness, sadness, surprise and disgust) that are ”cross-cultural” [32]. Other approaches are lexical, social constructivist, anatomic, rational, and communicative. In practice, these theoretical categories overlap. The difference among them is mainly in how the process, inputs and elicitation phases are considered in each one. This overlap goes along with the suggestion done by Izard [47], among others, that emotion elicitation can be performed at different levels, and at some of these levels people are not aware of the process. The overlapping among these theories and the relationship with the elicitation phases are depicted in Figure 2.3. In the next sections, more details about the theories use as reference for current work is given. 16
2.2. Emotional Models
Figure 2.4: Russel’s circumplex model of affect [90].
2.2.1.
Circumplex Model of Affect
The circumplex model of affect is a dimensional theory [70] that organizes emotions in a Cartesian plan, where each axis is an attribute associated to each emotion. One of the most used models is the one suggested by Russell [90], which organizes emotional states on two dimensions: valence and arousal. Valence is related to how much pleasant an emotion is, ranging from negative to positive, while arousal indicates the energy of emotion. The distribution of emotions in the Russel’s model is reported in Figure 2.4. 2.2.2.
Tomkins’ Theory
Tomkins’ theory [48, 98] integrates various perspectives. For Tomkins, the affect system evolved to solve the problem of overwhelming information present in the environment to which people are exposed. His theory states that people cannot manage consciously all the information available in the environment, therefore the affect system comes to select what information could be relevant to be aware in a given moment. For example, someone may be focusing on reading a book, ignoring the rest of events that are happening around, when, suddenly there might be a loud sound that gets his/her attention. This kind of behaviour could be obtained through the activation of different systems. Tomkins recognizes four systems closely related to affect: Pain is a motivator for very specific events that take place on our bodies. Drive deals with the basic needs that human body could need (e.g., eating, urination, breathing). Cognitive interprets the world and make inference from it. Affect focuses on driving person’s attention to specific stimuli. More importantly, Tomkins suggests that affect in certain situations could make pain and drive systems be omitted, while the affect and cognitive could work together. Because affection has a main role in human subsistence, he describes nine affects that 17
Chapter 2. Theoretical Framework
Figure 2.5: Patterns for surprise-startle, fear-terror, interest-excitement, anger-rage, distress-anguish and enjoyment-joy, after Tomkins [48].
could be triggered depending on brain activity and are organized in three categories. (i) Positive affects are those rewarding (i.e., Interest-excitement, enjoyment-joy). (ii) Negative affects are those punishing situations that could threaten people’s well-being (i.e. fear-terror, distress-anguish, anger-rage, disgust, dissmell, and shame). (iii) Neutral affects, which do not motivate people to do nothing (i.e. surprise-startle). Figure 2.5 shows activation patterns for surprise-startle, fear-terror, interest-excitement, angerrage, distress-anguish and enjoyment-joy. For instance, sustained low stimulation leads to distress-anguish, while a very highly increasing stimulation leads to fear-terror, and a less steep increase in stimulation leads to interest-excitement. Moreover, the time windows for these emotions are different. The remaining three affects are not triggered following a stimulation pattern, but rather they are triggered in specific situations that could be dangerous for a person. Finally, Tomkins suggested that these affects are the input to select an emotion. In his model, each person has an internal "script" that tends to map the affects to specific emotions. This mapping is generated by the personal experience of each person.
2.3 Performance Robotics Literature about robots as actors is not as wide as it might be expected. There are works where the robot interacts with the audience (like in a stand-up comedy), usually without any interaction with other actors, and others where robot perform, possibly with human actors, without any direct interaction with the audience, as typical of traditional theatre. Although tools and assumptions are different in each case, they have basic elements in common, such as the need for social abilities to delight the audience with a good play. 2.3.1.
Interacting with the audience
Interaction with audience might be as simple as the autonomous stage developed by Breazeal and collaborators [13]. They used one robot on the stage. The anemonelike robot developed for this work had few behaviours: watering plants, taking a bath, drinking water, following people movements, or getting scared when a person comes 18
2.3. Performance Robotics
too close. The robot was able to show some basic emotions (i.e., fear and interest), but it could not move on the stage. IbnSina Center [73] was an interactive theatre installation project. The installation was expected to have different kind of sensors, one screen to project virtual characters, and space where people could sit to enjoy the play or interact with the characters. Robots could be either tele-operated, semi-autonomous, fully autonomous, or scripted. A monologue script including one robot and virtual characters on the stage was developed. Knight [54,55] used the platform NAO to produce a sort of stand-up comedy. Before the robot’s presentation each person in the audience was given two paddles: a red one and a green one. These paddles were used to collect audience’s content or discontent to the joke, green and red paddle respectively. The audience’s response is used by the system to determine the next joke. Moreover, the time between two consecutive jokes is adapted to the audience’s laugh. If this timing is not respected by the system, the interaction would feel unnatural by the audience. Additionally, the robot performs basic actions to add some expressiveness to the joke, but it is not intended to project any emotion. 2.3.2.
No Audience Interaction
Performances without any direct interaction with the audience have been developed by more researchers. Commedia Dell’Arte [105] was a performance used to create a space to learn new knowledge in the process. The play script was adapted from Lazzo of the Statue, while the ”actors” were built using Lego’s platforms. Three robots were constructed, one for each character, but only the Arlequine robot had eyebrows movement, which enabled it to show some emotions. All of three robots lacked sensory inputs, which just gave them the possibility to repeat their pre-coded actions without any adaptation to the current situation. A software architecture to build a robotic actor was proposed in [16]. Each robot portrays a character, which was modelled using behaviours, actions, internal states, and inner obstacles. The last two are considered as aspects of the character personality. The sequence of actions, objects positions and other characters’ position were hard-coded. Once again, the robots just follow the scripts without correcting their movements in relation to other characters, neither have any capability to show emotions. Trying to add some theatrical realism, Breazeal and collaborators [46] designed and implemented a system to control a lamp. The main characteristic of this lamp was that it could be controlled by just one person, which could select the focus point where the lamp must look at. This simple ability improved the credibility that the lamp was listening to the person speaking to it. Roboscopie [57,63] had a simple story line involving one person and one robot. The robot could navigate autonomously in its environment, and build a 3D model of it. The human-robot coordination was implemented by tele-operation during the performance, and the positions of some objects were already known by the robot. The play took place in 2011, with public audience and it lasted 18 minutes. Fan and collaborators [65, 66] believe that humanoid robots are more suitable as theatre actors than any other kind of robots. Their robots are capable to perform autonomously many actions, such as: people drawing, jazz drumming, marionette oper19
Chapter 2. Theoretical Framework
ating, notation reading, and singing. However, they have problems with the amount of computers needed to control them. The high complexity to control humanoid robots, and the lack of expression of their body, makes this approach unsuitable to study emotion expression. They have done presentations with both humanoid and wheeled robots. More importantly, theatre has not been just used to create entertainment platforms, but it has been the center of study of people’s response to robots. As it is the case of a version of Shakespeare’s Midsummer Night’s Dream, where two different flying quadrotor platforms were used to portray fairs’ alter ego [75]. All the robots used in the play were tele-operated off-stage, but they interacted on stage with the actors. The findings reported about this experience, rather than technical, concerned audience and actor acceptance as well as their safety in the presence of the robots. With the idea to familiarize people with robots, Robots actors project was created [81, 99], where Wakamaru [74] and Geminoid F [?] robots were used in the play. Unfortunately, deeper information about this implementation is not available, although from videos is possible to infer that the play was designed for robots and robots seem remotely driven. Unfortunately, none of these works have used theatre to create robotic actors that could adapt to circumstances not described in the script or that could project emotions on the audience. Table 2.1 compares all these works by the following criteria: Platform is referred to the robotic platform used in the theatrical play; Autonomy tells whether the robotic platform decided autonomously the action to be performed, or it was tele-operated; Performance specifies in what kind of performance art the robot was used, such as: theatre, stand up comedy, and stage; Interaction with people tells if the robot interacts in some way with people, would them be actors or the audience; Adaptation to changes specifies if the robot could adapt to changes in the performance, such as changes in the object or characters on the stage; Emotion projection says if the robot was able to convey emotions to the audience or not; How is the emotion conveyed? In case that the robot can convey any emotion, this tells what kind of medium was used to convey the emotion. The mediums considered were: movement, body posture and face poses.
2.4 Studies on Emotion Expression Although it is not possible to apply a direct mapping from human studies to robots [9, 92], human studies provide guidelines about possible features and values that could be used to generate emotional motion. 20
2.4. Studies on Emotion Expression Table 2.1: Comparison among the works done on performance robotics. Where NA means "it does not apply" and NS "not specified".
2.4.1.
Work
Year
Platform
Interaction with Humans Yes (Audience)
Adaptation to Changes Yes
Emotion Projection
Emotions Implemented
Two
Fear and Interest
How is the emotion conveyed? Body posture
Breazeal and collaborators [13] Mavridis and collaborators [73] Knight [54, 55] Wurst [105]
2003
Autonomous
Stage
Autonomous
2010
Self Built (Anemonelike) Self Built (Human like) NAO
Stage
Yes (Audience)
Yes
No
NA
NA
Autonomous
2002
Lego
Stand-Up Comedy Theatre
Yes (Audience) No
Yes
No
NA
NA
Autonomous
No
NS
Face postures
Nomad Scouts
Autonomous
Theatre
No
No
Yes (Just one of three robots) No
Bruce and collaborators [16] Breazeal and collaborators [46] Lemaignan and collaborators [57, 63] Fan and collaborators [65, 66] Murphy and collaborators [75]
2002
NA
NA
2008
Self Built (Lamp)
Teleoperated
Theatre
Yes tor)
(Ac-
NA
Yes
NS
2012
PR2
Teleoperated
Theatre
Yes tor)
(Ac-
NA
No
NA
Body postures + Colors NA
2013
Self Built (Humanlike) Quadrocotor
Autonomous
Theatre
No
No
No
NA
NA
Teleoperated
Theatre
Yes (Actors)
NA
No
NA
NA
2009
2011
Autonomy
Performance
Laban Effort System
Some hints to define a formal model for this may come from models used to frame emotional human emotion. The Laban Effort System [58] is one of the more interesting models to code human movement. The dance director Rudolf Von Laban devised a framework to describe and represent artistic human body movement. His original idea was to create a system that could allow different dancers to perform a same ballet written by a choreographer using his notation. His coding system has not been used just in performance arts (e.g., ballet and theatre), but also in other fields (e.g., psychology [23, 41], and robotics [53, 95, 100]). His theory adopts the following four elements to describe human movements: Body describes which body parts are moving, and how each part influences the others; Shape describes the way the body changes during a movement; Space defines the connection between the movement and the environment; Effort focuses on the subtle changes in the movements while they are done w.r.t. an inner intention. Effort Factors
To achieve a correct representation on movements’ dynamics, Laban broke effort in four different factors. Each of these factors ranges between two boundaries as mentioned below. Weight is related to how the body weight is used in the movements, and it ranges from light to strong (delicate vs. powerful). Space defines the movements as indirect or direct. Time specifies the velocity during the movement, from quick to sustained. Finally, Flow defines the quality in the movements, which ranges from bound 21
Chapter 2. Theoretical Framework
to free. However, any further details to interpret the symbolic representations of these factors is given to the actor. Being effort related to the inner intentions, it is the most suitable element to describe emotions, which, in the case of acting people, are driven by inner intentions. 2.4.2.
Human Studies
Emotion plays an important role in human-human interaction, and can be expressed through diverse channels (e.g. body). Given the complex structure of human face, where more than 43 muscles exists, it has manly associate with emotion expression. As a consequence many works have focused on facial expression, mainly, but not only, influenced by the work done by Ekman [32]. Fortunately, the important role that body plays in emotions projection has been highlighted by some researchers [25, 101]. Nevertheless, the amount of works related to body expression of emotions is still small compared to studies on facial expression. Analysing some of few works that have studied human body expressiveness, it is possible to recognize two different methodologies to create the data base of movements to be assessed during the experiments. Applying a first methodology, human actors (either professionals, or amateurs) are asked to walk straight from point A to point B conveying specific emotions [24, 26, 101]. Each trial is recorded and later shown to each subject, who evaluates all the sequences. The second methodology uses virtual agents [89,100] to generate the very same set up for the experiments; the agent’s movements are generated from the data recorded from human actors. The work done by Wallboot [101] is a reference in studies on body emotion projection. His work addressed the question: Are specific body postures indicative of emotion or are they only used to indicate the intensity of the emotion? To answer this question, he recorded 224 videos for joy, happiness, sadness, despair, fear, terror, cold anger, hot anger, disgust, contempt, shame, guilt, pride and boredom and showed them to the subjects. His results reaffirm that movement and body postures are indicative of intensity for some emotions, while, for other emotions, these two characteristics seem to be enough to identify them. Complementary studies using virtual agents have been performed by Kluwer and collaborators [22], who studied the contribution of postures and angle view in the interpretation of emotions. They generated 176 static positions for happiness, anger, disgust, fear, sadness and surprise. Each image was later rendered from three different angles (front, left, and above and behind left shoulder) producing a total of 528 images. All the participants were exposed to all 528 images and were asked to label the image with the emotions that best represent it. Their results show that five of six emotions were highly recognized independently from the angle of view. However, disgust was in some postures confused with fear. The main drawback of all these projects is the use of video recorded sequences, which misses the impact related to a complete physical experience. 2.4.3.
Robotic Studies
The direct consequence of the abundance of works in face elicitation in humans is the amount of works done in Human-Robot Interaction (HRI) focused also on faces. 22
2.4. Studies on Emotion Expression
One of the most well-known expressive robots is Kismet [14], a robotic face able to interact with people and to show emotions. The face had enough degrees of freedom to portray the basic emotions suggested by Ekman [32] (happiness, surprise, anger, disgust, fear, and sadness) plus interest. The face’s physical design was done in way to encourage humans to treat it as if it was a social creature. The interaction studies done with this platform were conceived to consider a person as a caregiver and the robot as the receiver; it seems that the system was capable to engage people in a long term interaction. To achieve this goal, the system had six sub-systems: vision, hearing, motivational, behavioral, speaking, and emotion selection system. Despite the complex system behind Kismet, the emotion’s projection evaluation was done using videos with a very limited number of participants. In this experiment, each participant was asked to look at seven videos, one for each implemented emotion. After a video was projected, each participant selected the emotion from a list containing the seven implemented emotions, only. The results showed high percentage of recognition (over 57%). Saerbeck and Christoph [91] analyzed the relationship between robots’ motion characteristics and perceived affect. They first did a literature review to select characteristics that could be used to show affection. From their literature review, they decided to use acceleration and curvature on robot’s trajectory. Using these two characteristics, they did an experiment with eighteen participants. For the experiment they used two different platforms (iCat and Roomba) to verify if the embodiment had any impact on affection determination. For the Roomba researchers decided to use a circular trajectory in the room. While for the iCat two objects were place in front of it. It starts in a central position to then move to the left right and back to center. To reduce variables’ possibilities, they picked three definite values for each variable. Each participant was exposed to both embodiments and all possible variable combinations. To assess participants’ perception, they used PANAS [102] and Self-Assessment Manikin (SAM) [60]. Their results show that there was no significant difference between the two embodiments used and that participants were able to assign different emotions to the movement patterns shown to them. More importantly, they found out that acceleration is correlated with the perceived arousal but not with valence. As part of their work on detecting emotions in humans, with a case of use in robot games, Lourens and Barakova [6] implemented a set of behaviors to determine the emotion perceived from movement. The behavior parameters were selected based on the work done by Camurri et al. [19]. To focus participants’ attention on movement, they used the e-puck platform. Their experiment is not fully described, but authors highlighted that there were not given any list with possible emotions, rather they asked participants how they think that the robot was feeling. The results showed a very high recognition rate for the implementations of sadness, nervous and fear. The implementations for anger and happiness were confused between them. Barakova and collaborators used a closet robot [44], which does not any anthropomorphic resemblance, to study new possibilities in social interaction. This closet robot can perceive human presence and react to show behaviours that could be perceived as emotions or mental states. The closet robot had several sensors to detect human state and several lights to convey its behaviours. The robot’s behaviours were defined using the Interpersonal Behaviour Circle (ICB) [61], which is based on two dimensions (dominance-submission and hate-love). A pilot to test the five implemented behaviours 23
Chapter 2. Theoretical Framework
was made. The five behaviours correspond to two implementations of dominance, two of submission, and a neutral behaviour. These behaviours differ one from the other in the way that the light is turned on and its intensity. From pilot’s results, they selected two most convincing behaviours, one dominant and one submissive. Using these two behaviours, they did an experiment with three different scenarios, in which all the participants were exposed. To measure participants’ appreciations, they used the Social Dominance Orientation (SDO) [86] questionnaire and SAM. Their results suggest that people prefer systems that display submissive behaviors. More importantly, their findings suggest that electronic systems receive a different type of reactions than the one expected on theories of interpersonal communication. Using a humanoid platform NAO [88], Cañamero and collaborators [9, 10] studied the perception of key body poses designed to show emotions. They suggested that the techniques used to convey emotions in virtual characters could not be used in robots, due to the fact that virtual characters have no physical constraints, while robots are constrained by their physical capabilities [9,92]. They proposed a set of poses that could be used to express emotions with NAO, paying a particular attention to the contribution of the head. They did two experiments in which they show the participants different poses and each participant had to pick one emotion from the list given to them. The list use had six emotions: pride, happiness, excitement, fear, sadness and anger. Their results showed a recognition rate of 88%, 85%, 92%, 88%, 73%, and 73% for anger, sadness, fear, pride, happiness and excitement respectively. The major finding from their experiments is that moving the head up improved the identification of pride, happiness, and excitement. While moving the head down improved the identification of anger and sadness. However, these key poses are taken statically, without any displacement of the robot in the environment. Li and Chignell [64] used a teddy bear robot capable to move its arms and head to study their contribution on emotion projection. To achieve this, they did a total of four studies. In the first one, four participants were asked to create a gesture that the robot would do in one of the twelve scenarios presented to them. These gestures were recorded and used in the second study. The experiment consisted of two parts: one just showing the gestures, and the second one describing a scenario were the gestures were generated. The participants had to select from a list the emotion that they thought the robot was conveying. The list included basic emotions and mental states, which were selected from the comments obtained during the first study. The results showed that giving a context increased the rate of recognition. In the third experiment, they asked five novices and five puppeteers to create gestures for each of the six Ekman’s basic emotions. The gestures generated were recorded and used as input in their last test, where the subjects had to select an emotion from a list including the names of the six basic Ekman’s emotions. Their results showed that it is possible to convey emotions just using movements in the head and arms. Also they found out that the gestures made by the puppeteers had a better recognition for disgust and fear. Although they presented numerical information about their findings, this information is shown in a way that is not possible to discern the recognition rate of each emotion for each experiment. Daryl [33] is an anthropomorphic robot, without facial expression, nor limbs, but with mobility capabilities. This platform was used to test whether it is possible to project emotions without using cues used in a human-like platform (e.g., tail, ears and 24
2.4. Studies on Emotion Expression
head). This robot has head, ears, ability to generate colors in a RGB-LED set positioned in its chest, and a speaker system. The head had no capabilities to show facial expressions, but its movements and robot translations were used to show emotions. The emotions implemented were: happiness, sadness, fear, curiosity, disappointment, and embarrassment. During the experiments done by the researchers, special attention was put to the final distance between the robot and the subject, but the approaching velocity and the followed trajectory were not taken into consideration. The subjects were exposed to a sequence of movements. For each sequence the subject was asked to rate the intensity of each emotion enlisted in a five-point Likert scale questionnaire. The enlisted emotions were the six implemented plus anger, disgust and surprise. Their results showed that participants gave a higher intensity to the desired emotion, also similar ones, such as sadness and disappointment. Using the platform WABIAN-2R, Destephe and collaborators [28] studied the attribution of emotion to a robot’s gait. To obtain the robot’s movements, the researchers asked two professional actors to walk in a room conveying anger, happiness, sadness, and fear with different intensities (low, regular, normal, high, and exaggerated). All the actors’ walking were recorded using a Cortex motion capture system. These data were later reported to robot’s embodiment. To verify their data, the researchers first did a pilot study with just two emotions (happiness and sadness) and thirteen subjects [27]. Each subject was exposed to a series of videos. After each video, the subject was asked to select one emotion from a list (happiness, neutral, or sadness) and to state the intensity of the selected emotion. The videos showed were not made with the real robot, but with a virtual model of the robot. From the results obtained, they decided not to use the low intensity because it had a very low recognition rate. During the experiment they adopted the same procedure, but this time showing all the emotions and intensities, excluding low, to the participants. Their results showed that sadness was recognized 73.81% (average) of the times with an intensity of 21.43%, happiness was recognized 66.67% (average) of the times with an intensity of 30.95%, anger was recognized 61.9% of the times with an intensity of 26.19%, and fear was recognized 83.33% (average) of the times with an intensity of 28.58%. These results show that people could reasonably well recognize emotions from the robot’s gait. Lakatos and Collaborators [59] take inspiration from human-animal interaction to create behaviours that could enrich human-robot interaction. They conducted an experiment to analyse the recognition rate of emotion expression, using as inspiration the movements done by dogs to convey emotions. The experiment was done using a Wizard-of-Oz scene and its goal was to determine if people could distinguish two emotions (happiness and fear). They use a game approach to determine participant’s appraisal of the robot emotion. This was done in order to avoid asking the participants the emotion they believe the robot is eliciting. Therefore, in the experiment, each participant had two balls (yellow and black) and they could play with the robot using one of this balls at the time. However, one of the balls triggers robot’s happiness and the other fear. The dog’s favourite ball was changed from participant to participant to avoid any bias for the balls’ color. The results of this experiment showed that the participants decided to use more dog’s favourite ball to play with the dog, which is consistent with the idea that the participants could discern which ball produced ”happiness” in the robot. Brown and Howard focused their interest on head and arm movements using the 25
Chapter 2. Theoretical Framework
DARwIn-OP platform [15]. Their hypothesis was that using some principles, that they determined as important, it is possible to express happiness and sadness in a way that people could recognize. These principles establish that to show happiness is necessary to move robot’s head up, arms up, and it should be done fast. While to express sadness the head goes down, arms down, and it should be done slow. To test their hypothesis they conducted an experiment with thirteen participants. Each participant was exposed to fifteen sequences of poses and he/she had to answer for each sequence a five-point Likert scale (very-sad to very happy, with neutral in the middle). They obtained a 95%, 59%, and 94% of accuracy for happiness, neutral and sadness respectively. Although their results show that their key features are determinant to convey happiness and sadness, the questionnaire used and the quantity of participants bring doubts about the possibility to generalize their findings. Sharma and collaborators [95] used a quadrotor to study how different Laban’s effort [58] parameters could impact on the perception of affection. A professional Laban certified actor was asked to generate 16 different paths, for each one changing one of the four Laban’s parameters (space, weight, time, and flow). Each generated path was recorded using the Vicon motion-tracking system. Then, they did an experiment were people were asked to assess each path using the Self-assessment Manikin (SAM) [60]. In order to analyse the results, they mapped them on the circumplex model of emotion, and evaluated the contribution of each Laban’s parameter in the 2-D circumplex model (arousal and valence). The results show that it is possible to increase both valence and arousal by using a more indirect space, or by performing the motion more quickly, and to decrease valence or arousal by a more direct use of space, or a more sustained motion. Although they suggest the use of Laban’s description as a tool to specify affection movements in Human-Robot Interaction, how to use these parameters in the actual implementation remains an open question, since Laban defined them qualitatively, with reference to human people, and they leave very open questions about the most appropriate numerical values. Suk and collaborators studied the human emotional interpretation for speed, smoothness, granularity and volume of a non-human or animal like object [76]. Two experiments were designed to determine the relationship among these features and the emotional interpretation. To assess participants’ emotional response they used SAM. The first experiment was focused on speed and smoothness movement features, selecting five different values for each one of these two features. Each participant was exposed to twenty five movements. After a participant observed a movement, the participant marked the two SAM graphic figures (pleasure and arousal). The results from this first experiment show that the arousal increases as speed increases, but that there is not any clear tendency for smoothness. In the second experiment they evaluated the other two features (granularity and volume) using the same procedure followed in the first experiment. Their results show that granularity is positively correlated with pleasure and arousal. On the other volume is negatively correlated with pleasure and positively correlated with arousal. As overall result, they found a major contribution of speed on arousal and minor contribution of granularity and volume. To improve the coordination between humans and robots Novikova and Watss [78] proposed the use of emotion. They did studies using a own built platform called E4 to verify whether people could detect emotions from a non-human like platform. The 26
2.5. Emotion Selection Systems
E4 platform was constructed using a Lego Mindstorms NTX and it was based on a Phobot robot’s design . In all their experiments they used six emotional expressions (scared, surprised, excited, angry, happy, sad) and neutral. The emotion list used in their experiments was balanced using the 2-D circumplex model (arousal and valence), selecting three from each quadrant for a total of twelve emotions, plus the options of “other” and “don’t know”. They obtained a recognition rate of 52%, 42%, 41%, 36% and 15% for surprise, fear, sadness, happiness, and anger respectively. Table 2.2 summarizes all the robotics studies to convey emotion that we considered, highlighting the following characteristics: Embodiment tells if the robot is human-like, robot-like or neither of these two; Platform used gives the name of the platform used in the study; TheLocomotion criterion was added to put in evidence whether the platform uses displacement movement to express emotion; Emotion Evaluation Method refers to the methodology used to collect the data in the studies. Emotion presentation tells if a real robot was used in the study, or any other method; Emotions Implemented gives the list of emotions showed to the subjects; Type of test tells if it was done an experiment or a case study. How is the emotion conveyed? This tells what medium was used to convey the emotion. The media considered were: movement, body posture and face poses.
2.5 Emotion Selection Systems In this section is described systems created to let the robot select what emotion express. Cañamero and Fredslund developed the LEGO humanoid robot Feelix to expresses emotions on its face based on physical (tactile) stimulation [17]. A tactile sensor is used to determine the stimulation which could fall in one of the following cases: short (less than 0.4 sec), long (up to 5 sec), and very long (over 5 sec). The events generated from the stimulation are used to determine the emotion activation based on the state of a finite state machine that implements general emotion activation patterns (cf. Fig 2.5) drawn from Tomkin’s theory of emotions [98], that we have also used in our research. Feelix could detect stimulation patterns for and display the following emotions: anger, sadness, fear, happiness and surprise. MEXI is a robotic face that is capable to interact with people by exploiting emotions [34]. MEXI is capable to understand people emotions through image analysis of data coming from two cameras, and its speech recognition system. MEXI’s architecture lacks of any deliberative component, but it uses emotions and drives to control its behaviours. Its emotion system obtains information from the behaviour system and external perceptions to come up with the new values for each emotion. Each one is 27
Chapter 2. Theoretical Framework Table 2.2: Comparison among works on emotion projection in robotics. NA = Not Available Work
Year
Embodiment
Platform Used
Locomotion
Emotion Evaluation Method
Emotions Implemented
Type test
Happiness, surprise, anger, disgust, fear, sadness and interest NA
Experiment
How is the emotion conveyed? Face poses
Breazeal [14]
2002
Face
Kismet
NA
Questionnaire
Saerbeck and Christoph [91]
2010
iCat / Roomba
NA / Differential
PANAS SAM
Cañamero and collaborators [9, 10]
2010
Animal/ Non-Human like Human-Like
Experiment
Movement
NAO
Bipedal
Questionnaire
Real robot
Experiment
Body poses
Experiment
Body poses
Real robot
Anger, sadness, fear, pride, happiness, and excitement Random + Anger, disgust, fear, happiness, sadness, and surprise NA
Li and Chignell [64]
2011
AnimalLike
Teddy Bear Robot
NA
Questionnaire
Video (Real robot)
Barakova and collaborators [44]
2013
NA
Closet robot
NA
SDO and SAM
Experiment
Real Robot
Laban’s poles
Experiment
Bipedal
SAM + Questionnaire + Interview Questionnaire
Changing lights on and intensity Movement
Sharma and collaborators [95] Destephe and collaborators [28] Embgen and collaborators [33]
2013
NonHuman/Animal
Quadrotor
Aerial
2013
Human-like
WABIAN2R
Video (Virtual Robot)
Experiment
Daryl
Differential
Questionnaire
Real Robot
AnimalLike
Dog
Holonomic
Real Robot
2014
Human-Like
Bipedal
NonHuman/Animal
NA
SAM
Real Robot
Happiness and sadness NA
Experiment
2014
DARwInOP Self-made
Indirect (Interaction with the robot) Questionnaire
Fear, anger, happiness, and sadness Happiness, sadness, fear, curiosity, embarrassment, and disappointment Happiness and Fear
2014
Human-like
Lakatos and Collaborators [59] Brown and Howard [15] Suk and collaborators [76]
2014
Novikova and Watss [78]
2015
NonHuman/Animal
Lego
Differential
Questionnaire
Real Robot
Scared, surprise, excited, angry, neutral, happiness and sadness
Experiment
and
Emotion Presentation Video
Real Robot
Real Robot
of
Experiment
Experiment
Experiment
Movement (Gait)+Body Poses Movement + Body poses
Move + Body poses Body poses Speed, smoothness, granularity and volume Movement + Poses
associated to an intensity value between 0 and 1, updated according to the current perception. The considered emotions are: anger, happiness, sadness and fear. The architecture described in [62] uses a mixture of hard-coded emotions and emotions learned by association. Their emotion system uses inputs from the deliberative and reactive architectural layers to select one of the following emotions: fear, anger, surprise, happiness, and sadness. Each of these emotions is triggered according to perceived events, internal state, and goals of the robot in the current movement. The emotion selected by the emotion system affects the way each behaviour is performed. Lourens and Barakova proposed a framework to detect and express emotional states [6] based on Laban movement analysis with a case of use in robotic games. They use curvature/variability, velocity and acceleration to determine weight, time, and flow, which correspond to three out four Laban’s efforts. Once the value of these efforts is determined, they proceed to determine the person’s emotion that could be sadness, joy, fear or anger. In their study cases the emotion detected was then used as base to let the robot act accordingly, which could be seen as the emotion selection system that influences robots behaviour. The emotional model proposed by Malfaz and Salichs [69] uses appraisals to select an emotion. Happiness is related to the fact that something “good” happens to the agent (e.g., interpreted as the reduction of a need), and sadness to something “bad” 28
2.6. Summary
(e.g., interpreted as the increment of a need). Fear is related to the possibility that something bad happens to the agent and it is activated when something dangerous could be expected by the agent.
2.6 Summary In this chapter, the principal components that were taken into account to design the experiments and software have been introduced. To get a better understanding of theatre environment, theatre’s main concepts, constraints and actors’ lessons (Section 2.1) were first described. This information has been later used in the design of the architecture, and it is explained in Chapter 3. A brief introduction to different emotional models is given to contextualize the cirumplex model of affect and Tomkin’s theory. The first is used by some works and was used during the third case study to select the emotions’ values. The second is used as based ground to develop our first version of the emotion selection module. With all theoretical background given, the state of art in performance robotics was presented. This showed that most of the works have used theatre to build entertainment robots without considering this space to study timing and non-verbal expression in robotics. This finding reinforced our idea that the architecture should be modular and that it should be possible to use in different contexts, and not just in theatre. Moreover, it showed that none of the works paid attention about how emotions are conveyed, which is a necessary element for theatrical robot actors and in social environments. This brought me to look on studies on emotion projection and realized that most of the works done have been focused on humanoid platforms. Although the results obtained in these works reinforce the idea that it is possible to convey emotions with robotics platforms, it is still not clear what features could be used in a non-anthropomorphic platform, which could not even have shape change capabilities such as the Roomba platform. Therefore, it was decided to conduct case studies with a simple platform to determine what features and respective values could be used to convey specific emotions. Finally, some works on emotion selection system in robotics were presented to frame the state of the art and provide a yardstick for our approach, described in the next sections.
29
CHAPTER
3
Platform
To determine a suitable platform that could be use in the scope of determine the features and their values to project emotions. It was done a comparison of the available platforms in the market when the decision was made. Besides the requirements detected during the study of theatrical studies, in which actor must not rely on their faces to convey emotions, the following parameters were considered: Locomotion: possibility to move from one point to another; Shape: the aspect of the robot and its size should be compatible to what needed on the scene, and should allow for human-like performances; Control over the shape: possibility to change the robot’s shape, for example expanding or contracting robot’s extremities; Sensors: the type of sensors available on the platform should enable a reasonable perception of the scene; On-board processing: processing should be possible on-board. Communication: it should be possible to receive and transmit information to external devices for syncronization (Wifi, Bluetooth, and XBee); Maximum Velocity: the maximum linear velocity that could be reached by the platform is that of a fast pace, e.g., 1200 mm/s. Modular: possibility to add, change or delete hardware components without any major impact on the performance. 31
Chapter 3. Platform
The platforms used in the related literature have been analysed, as well as popular platforms available in the market, and the results could be seen in Table 3.1. Although NAO and Darwin platforms are widely used, these platforms have the following problems: Their maximum velocity is still too slow. Their gait is very clumsy. The results in the emotion expression study would be biased by the humanoid body. This is a problem because most of the current platforms and further platforms that could interact with humans might need to have a non human-like embodiment to perform their tasks: in many cases they will have a shape functional to their tasks, and different from the bio-inspired ones. The other platforms are too slow or do not have enough grades of freedom to change their direction while they are moving. As a consequence, it was decided to create a new platform that allows us to project emotions and have enough processing power. This platform was envisioned to be holonomic for two reasons. First, this platform could emulate human movements because they can change their orientation while they are moving without any constraint. Second, the research group has a long-term experience in this kind of platforms. Table 3.1: Comparison of the platform available when the decision of the platform was done. Platform
Locomotion
Shape
Control Shape
Sensors
Darwin-OP
Bipedal
Human-like
Extremities
MIC, Camera, and Speaker
NAO V4
Bipedal
Human-Like
Extremities
LEGO Mindstorms EV3
Configurable
Configurable
Configurable
Pioneer 3-DX
Differential
Non-human like
Not could changed
PLEO
Quadruped
Animal like
Extremities
TurtleBot
Differential
Non-human like
Not could changed
be
DFRobot HCR Mobile Robot
Differential
Non-human like
Not could changed
be
be
On-Board Processing Yes, ARM CortexM3
Communication WiFi
Maximum Velocity Very slow
MIC, Camera, and Speaker It should be bought
Yes, Intel Atom Yes, ARM9
WiFi
Very slow
No
Very slow
Ultra sonic, but it could be added others Touch sensor, RFID, Speaker, Camera Kinect but it could bought additional ones Ultra sonic, but it could be added others
Yes, Core
WiFi
Medium
It could be added XBee by hacking the robot WiFi
Very slow
No
Slow
It could be added parts on the top
It could be added
Very slow
It could be added parts on the top
Dual
Yes, ARM7
Yes, Asus computer Yes, Arduino Nano
Modular It could be changed the parts configuration No It could be changed the parts configuration It could be added parts on the top
This chapter describes the platform implemented and used in this research. The first part briefly introduces holonomic platforms, while the rest of the section describes the evolution of the platform.
3.1 Holonomic Platforms Holonomic platforms are characterized by the possibility to move in any direction without the necessity to have a specific orientation, i.e., they are free to move taking any desired orientation. This type of movement requires a specific kind of wheel, as the one that is shown in Figure 3.1. This kind of platforms require at least three motors and wheels to move. Figure 3.2 depicts a holonomic platform with three motors. In this configuration each motor is 32
3.1. Holonomic Platforms
Figure 3.1: Holonomic wheel.
Figure 3.2: Holonomic platform with three motors in a Cartesian plane. m1 , m2 and m3 represent the motors.
angle on a circular reference, and each wheel has a rotation of placed every 2pi 3 respect to the motor’s axis. 3.1.1.
pi 2
with
Kinematics
Using the configuration shown in Figure 3.2, it is possible determine the velocity of each motor given the desired velocity triplet < Vx , Vy , ω >, where Vx and Vy are the velocities in the x and y axis directions, respectively, and ω is the rotational velocity with respect to the center of the robot. Then, the system could be described as follows: cos(60) L − cos(30) −R m1 Vx R R cos(60) cos(30) L m2 = R − R Vy R 1 L m3 ω −R 0 −R
(3.1)
Where R is the wheel’s radius, L is the distance between the center of the configuration and a wheel. The contribution of each motor to each velocity component can be described as follows: 33
Chapter 3. Platform
cos(60) cos(30) L −1 − − Vx m1 R R R cos(60) cos(30) L Vy = R m2 −R R L ω m3 − R1 0 −R
(3.2)
3.2 Prototype A prototype with this kinematic characteristics was built to study if by using changes in oscillation angle, angular and linear velocity would be possible to convey emotions. It was used during a pilot trial done in the Researchers’ Night event, in 2013. In the following it is provided a detailed description of this prototype. 3.2.1.
Electronics and Mechanical Design
This prototype was done with a hexagonal acrylic base as it is depicted in Figure 3.3a. The following components were attached to the platform: Three metal gear motors with 64 CPR encoder [84]. Each motor is attached to an omni-wheel, enabling the robot to move at maximum linear velocity of 1.2m/s. One servo motor HS-475HB attached to a beam of 18cm. Three motor drivers [85], each one attached to a motor. One Arduino Mega 2560 [1]. X-Bee Pro to enable the communication between Arduino and an external computer. One Zippy Li-Po Battery, three cells (11.2 V) and 4000 mA/h. The distribution of all these components could be seen in Figure 3.3-b and c, and the mechanical prototype in Figure 3.4. 3.2.2.
Software
The software is divided in two parts. First one runs on Arduino, which is in charge to interpret the following commands: Simple command: tells the robot the angular velocity (Vω ), x and y velocities (Vx , Vy ). If the platform is executing a simple command and a new simple command arrives, the new command is executed. However, if a profile (see below) is in execution, the command is ignored. This command does not have any reply from the platform. Stop command: stops the platform even if it is executing a profile. This command does not generate any reply from the platform. 34
3.2. Prototype
Figure 3.3: a) Platform’s base design with its respective measurements: L = 110mm, R = 35mm, side = 120mm. b) Prototype’s second layer design, seen from the top. c) Prototype’s design seen from the left side.
35
Chapter 3. Platform
Figure 3.4: Prototype based on the design depicted in Figure 3.3.
Profile command: is a special command that loads in the platform an emotion profile. An emotion profile is a sequence of velocities in a straight line. Each distance where the velocity should be changed, is called point. To send a profile command first it is verified that the platform is not executing any other profile. If the platform is free, then the next point in the profile is sent, to be executed. After all the points are sent, the profile is executed. To achieve the execution of these three commands, the following modules were designed and implemented: Motor’s velocity calculation receives the pulses from the encoders and calculates the motor velocity V as follows: 2 ∗ π ∗ number_pulses rad V = pulses_per_turn ∗ ∆t s
(3.3)
Where number_pulses is the quantity of pulses counted during the time window, pulses_per_turn is the quantity of pulses that are necessary to do a whole turn in the wheel (1920 for the adopted configuration), ∆t is the time elapsed between two consecutive calculations (0.025 seconds). Replacing the variables with their values the following formula is obtained:
rad V = number_pulses ∗ 0.1309 s
(3.4)
Platform’s Velocity/Position calculation updates the platform’s current velocity and position. Replacing the variables in equation 3.5 for platform’s values, it is 36
3.2. Prototype
possible to have this information. It is important to notice that it was not used L 6= 125mm. The reason relies on the mechanical structure of the wheel, which has two contact points to the floor. Therefore, the circumference followed by the robot during rotation is determined by the inner contact point, which is approximately at L = 110mm. Although this difference does not look critical, the cumulative error generated using L = 125mm to calculate robot’s orientation would be greater than when L = 110mm is used. Therefore, the matrix for the velocity calculation looks like: Vx 11.6667 11.6667 −23.333 m1 h i mm 0 Vy = −23.2073 23.2073 m2 s ω −0.1061 −0.1061 −0.1061 m3
(3.5)
The Equation 3.5 is given in the robot’s frame at time t. Therefore, it is necessary to rotate this matrix to the reference frame to obtain absolute velocity and position. As a consequence, the matrix becomes: Vxr0 cos(θt−1 + r0 Vy = sin(θt−1 + ω r0 0
ω rt ∆t) 2 ω rt ∆t) 2
r −sin(θt−1 + ω2 t ∆t) 0 Vxrt r cos(θt−1 + ω2 t ∆t) 0 Vyrt ω rt 0 1
(3.6)
Where < Vxrt , Vyrt , ω rt > correspond to the values obtained in Equation 3.5, and Vxr0 , Vyr0 , ω rt > are the velocities in the reference frame. With these values, the position update is done as follows: r r 0 Vxr0 Xt 0 Xt−1 r0 r0 r0 Yt = Yt−1 + Vy ∗ ∆t[mm] r0 θtr0 θt−1 ω r0
(3.7)
Kinematics gets the desired velocity vector < DVxr0 , DVyr0 , Dω r0 > as input, and converts it to motors velocity. Replacing the variables in the Equation 3.1 becomes: m1 0.014 −0.025 −3.142 cos(θt ) sin(θt ) 0 DVxr0 0.025 −3.142 −sin(θt ) cos(θt ) 0 DVyr0 m2 = 0.014 m3 −0.286 0 −3.142 0 0 1 Dω r0 (3.8) PID ensures that the velocity of each motor is the expected one. The Arduino’s PID library [8] was used. The output then is modulated in PWM to manage the motor’s velocity. The P ID was tuned using the Ziegler-Nichols method [11]. The connection of all these modules is depicted in Figure 3.5. Moreover, a graphical interface to operate the platform was implemented. The interface was built in Java to enable the interoperability among operation systems. This 37
Chapter 3. Platform
Figure 3.5: General distribution and communication scheme among the software components running on the Arduino.
38
3.3. Version 1.0
interface is designed to describe the linear trajectory of the platform; this description is used to express an emotion profile. Each emotion profile is composed by one or more points, where the value of the variables that are controlled is given. The variables whose value is set at each point are: desired velocity, distance at which the velocity should be reached. This interface allows the user to add and delete points, modify each point, see each point in a line, save, modify the whole profile, and give a little description or comments related to each emotion profile. Each point could be seen in a graphical line, which could give the user an idea of its position. The distance used in the whole program is absolute, thus, if it is necessary to make the robot to go backwards, a negative distance should be introduced.
3.3 Version 1.0 From the comments received during the pilot, it was decided to modify the platform and the interface to add new features: upper movement and angular velocity. 3.3.1.
Hardware Changes
Two beams attached to a servo motor each, and another one to move a part of the body were added. The two beams could be controlled to obtain an asymmetric movement, to resemble the movement of the shoulders, or opening and closing at the same time. The design and dimensions for this version can be seen in Figure 3.6. The mechanical structure can be seen in Figure 3.7. In both cases, the desired angle can be configured in the last version of interface. The platform was covered with foam, and a light blue cloth covering the foam. The light blue color was selected because is known as neutral color, and it is not expected to generate any prejudice in the audience. The final version could be seen in Figure 3.8. 3.3.2.
Software Changes
The Arduino’s software was sightly changed to control the new servo motor set. But the general design remained the same as the previous one. The new variables that could be controlled in the interface are: angle restrictions, maximum angle, upper body position, and upper maximum angle. When the angle restriction is active, the rotational movement of the robot is going to be constrained to the maximum angle. When the upper maximum angle is positive, it determines the angle at which the beams are going to move, but when it is negative it gives the angle at which both beams are going to close. The interface could be seen in Figure 3.9.
3.4 Version 1.1 Although the version 1.0 showed to have great power to convey emotions, still it was not autonomous enough to host high level algorithms needed for localization. 3.4.1.
Hardware and Mechanical Changes
To improve the robot’s autonomy, it was added an Odroid-U3 Microcomputer. This microcomputer was attached to the Arduino Mega via ROS-Serial. The Odroid-U3 39
Chapter 3. Platform
Figure 3.6: Design of the first version. a) Base platform, including wheel protectors and the foam. b) First layer, including battery slot. c) Second layer with the servos’ distribution. d) Lateral view of this version.
40
3.4. Version 1.1
Figure 3.7: a) Back of the platform without foam, each red arrow shows the movement of beams. The movement is controlled independently. b) Front of the platform without the blue cloth. The red rectangle highlights the space of the body that is moved, and the red arrow shows that is possible to move it in both directions.
Figure 3.8: First version with the blue cloth.
41
Chapter 3. Platform
Figure 3.9: Final interface used to program the emotions conveyed by the robot.
addition, also enables the communication via WIFI. Small improvements were done in the upper part. Two reasons motivated these changes. First, the deformation mechanism was not changing the shape as expected. Second, the beams were damaging the foam. Therefore, the deformation mechanism was changed to have a lever, and a piece of rubber was attached to each beam. In addition to these changes, an aluminium base was added to give the robot more robustness. The new design is depicted in Figure 3.10 and its mechanical structure in Figure 3.11. 3.4.2.
Software Changes
The rudimentary system to convey emotions was changed from our first version of emotional enrichment system (See 5.1). This upgrade, brought the necessity to change the visual interface to enable the connection with the control movement system. The new interface could be seen in Figure 3.12, where it is possible recognize five different modules: Platform’s velocity and position in real time. Possible action to be executed, which is selected from a list of actions implemented. Action parameters, whose interpretation depends on the desired action. Possible emotion and its respective intensity. Panel with sent action, reset position, sent emotion, and stop buttons.
3.5 Version 2.0 Unfortunately, improving the autonomy of the robot showed a problem unnoticed before: a ”shaking” movement from each servo motor. This problem comes from the 42
3.5. Version 2.0
Figure 3.10: Design of the 1.1 version. a) Base platform, this layer is used to carry the battery or batteries. b) First layer, which includes the Arduino and the H-Bridges to control the motors. c) Second layer with the Odroid-U3. d) Third and last layer, with the servo motors distribution. e) Lateral view of this version.
43
Chapter 3. Platform
Figure 3.11: Platform with changes in the upper part and addition of the aluminium base.
Figure 3.12: Graphical interface used to control the platform version 1.1.
44
3.5. Version 2.0
Figure 3.13: Third version of the platform. a) Platform without foam. The blue acrylic is used to give some structure to the upper part to improve the shape’s change. b) Platform with foam.
limitation of Arduino Mega which has only one timer to calculate the velocity of each wheel and control accurately the servo motors. 3.5.1.
Hardware and Mechanical Changes
To overcome the shaking problem, the Arduino Mega was changed for Arduino Due [5]. This decision was made after comparing Arduino Due with other two possibilities used in robotics: Raspberry Pi, and BeagleBone. Raspberry Pi was discarded because it has few GPIO, that we needed to control the DC motor and servo motors. Although BeagleBone is a good board and it could fit easily our requirements, it was discarded, due to the time required to re-implement all the software into this board. The upper part was modified to increase the bending. To obtain this, a servo motor HS-765HB was attached to a beam, which was installed in the top part of the base, and two servo motors HS-475HB in the upper part of the beam. The design is shown in the Figure 3.14. The final version, with improved computational power (Arduino Due instead of Mega) is shown in Figure 3.13. 3.5.2.
Software
It was decided to give a better structure to the Arduino code to improve its readiness and allow additions in an easiest way. To achieve this, it was used the possibility to write Arduino code in a reduced version of C++, although in this configuration not all libraries and functionalities are available. However, our goal was to adopt the Object Programming Paradigm, which allow us to dive the code and express it in classes, each one with a specific role. The UML class diagram is depicted in Figure 3.15. Finally, the second version of the emotional enrichment system was used in this 45
Chapter 3. Platform
Figure 3.14: Design of the third version. a) Base platform, this layer is used to carry the batteries. b) First layer, which includes the Arduino and the H-Bridges to control the motors. c) Second layer with the Odroid-U3 and the mechanical structure to support the upper part. d) Upper part, with the servo motors distribution. e) Lateral view of the version.
46
3.6. Summary
Figure 3.15: UML class diagram of Arduino’s Due code.
platform (See 5.3). As a consequence, a new tab was added to enable the possibility to send compound actions. The distribution of all the modules are depicted in the Figure 3.16.
3.6 Summary This chapter has first shown the comparison done among different platforms available in 2013 to select a suitable for the study. The result showed that any of the platforms fulfilled all the desired requirements. Therefore, it was decided to create a holonomic platform to fulfill the requirements. The platform evolution (hardware and software) were described and the main characteristics were highlighted.
47
Chapter 3. Platform
Figure 3.16: General distribution and communication scheme among the diverse software components running on the whole system.
48
CHAPTER
4
Architecture
Performance arts are mostly characterized by emulation of daily situations, they aim at making the performance credible and enjoyable to viewers, involving them in a world created in the stage. Although nowadays there are a variety of performance arts, theatre is still considered as the lively art [104], since actors have to create lively situations and manage them in the best possible way, without compromising the quality of the play. Theatrical actors should not only learn their lines, but also need to make their performance and speech as natural as possible and coherent with the play situation. Unlike television programs and movies, theatre actors do not have the possibility to fix their mistakes during a performance. They must play believably and coherently every time the play is presented. The robot actor’s architecture has been designed to fulfil two main requirements. First, it should enable a robot to be an ”actor”, it should be capable to adapt to different situations, to be coherent with the social context, and to express emotions, in addition to the standard requirements for an autonomous robot. Second, the architecture should be general enough to be applied also in other domains. With these aims, the architecture was created as modular (see Figure 4.1). Each module can provide information to the others, but these last can decide whether to consider the incoming information. If none of the modules accepts suggestions, the system reduces to a traditional decision system, where decisions about actions are taken considering only information about the perceived physical world and the actions required by the script: actions are done without any emotion or expression. The architecture includes six main sub-systems: Description manages the information provided by designers and users: script, character description, action profiles, emotion profiles, and platform descriptions. Two kind of modules are in it: the ones specific to theatre (i.e., script and character) and the ones transversal to different ap49
Chapter 4. Architecture
Figure 4.1: TheatreBot’s architecture is composed of six sub-systems: Feature, Belief, Decision, Motivation, Description, and Emotion Enrichment. The full arrows represent the information that has to be used by the module. Dashed arrows mean that one model may influence the other. The influenced module can either accept or reject the suggestion given by the influencing one.
plications (i.e., platform description, action, and emotion profiles). Feature transforms the raw sensor data into information about the world (e.g., objects and other actors present on the stage and their positions). Belief builds and maintains the set of robot’s believes about the world and its state. They come either from the Feature sub-system (World Model) or from elaboration of what happens in the play (Social World Model). While the world model maintains information about objects and people, the social world model is a representation of social relationships that would allow the robot to act in a ”socially correct” way. In other words, the social world model has a representation of the relationship between the character and other people, and the current situation. This information helps the system to generate the best action to execute in a given moment. For example, people could not do some actions in their office, while they could do the same action in their house. Action Decision selects the next action to be executed according to the current situation. The selection depends on objects’ and characters’ actual position, social situation, and internal emotions, among others. Emotional Enrichment System (EES) blends the inner emotion, social awareness, and the decided action to produce an expressive action. Motivation describes needs and desires [14, 39] that may exist at any time based on the robot’s belief and influenced by its emotional state. 50
4.1. Simple and Compound Actions Table 4.1: Description of the eight simple actions implemented, and their respective parameters. Action Name
Do nothing Move body
Oscillate body Move shoulder
Oscillate shoulder Move torso Oscillate torso Look at
Description
Mandatory rameter(s)
Pa-
It waits a time t. It could be seen as a delay. It moves the platform from its current position a to a desired position b. It generates an oscillation in the whole platform by a θ angle. It moves the shoulder to a desired angle . It is considered as angular movement. It oscillates the shoulder by a given angle θ It moves the torso to a desired angle It oscillates the shoulder by a given angle θ It maintains the robot orientation towards a position, object or person
Time
Optional Parameter(s) null
Landmark or Point
Velocity
Amplitude
Velocity
Amplitude
Velocity
Amplitude
Velocity
Amplitude
Velocity
Amplitude
Velocity
Landmark or Point
Maintain
4.1 Simple and Compound Actions To achieve actions’ enrichment with emotions, two different types of actions are used: simple and compound actions. Simple actions are actions that could be considered as “atomic”; they can be enriched with emotion changing their specific parameters. For example, consider two simple actions: speak and move body. The human action of speaking is related to changes in the vocal cords to produce different sounds, while move body is directly related to the body movement. Therefore, for the speaking action the following parameters are expected: text, pitch and tone. On the other hand, action move body would have as parameters desired destination, velocity, and trajectory constraints. Since each action is characterized by different parameters, the parameters’ modifications to obtain an emotional action will be different for each of them. Simple actions to be implemented were selected by considering the platform’s capabilities and requirements to be fulfilled to create a theatrical robot. Eight actions were selected: move body, oscillate body, move shoulder, oscillate shoulder, move torso, oscillate torso, do nothing, and look at. Description for each action, its mandatory parameters and optional parameters are shown in Table 4.1. These simple actions enable the robot to express emotions by changing their parameters, such as velocity and acceleration. However, the small number of these actions could be seen as a strong restriction for the robot’s expressiveness. To overcome this limitation, the concept of compound actions is used. A compound action is created from other compound or simple actions. For example, suppose that it is required that the robot has to recite some dialogue and walk to a different position, while doing a trajectory as it is shown in Figure 4.2. The respective compound action would consist of a speak simple action in parallel to two consecutive walk actions, as shown in Figure 4.3. The action walk is also a compound action that is composed by three parallel actions: balance left arm, balance right arm, and move body. 51
Chapter 4. Architecture
Figure 4.2: Desired positions to move in the stage. Where U means up, D down, R right, C center, and L left. These positions correspond to the 9 positions defined in theatre on stage.
Figure 4.3: Example of a compound action that is composed by another compound action (walk) and a simple action (speak).
52
4.2. Emotional Enrichment System
4.2 Emotional Enrichment System The Emotional Enrichment System is divided in two parts: modulation and generation. The first part gets the desired action, emotion and character descriptions to modify the action’s parameters accordingly to the emotion description files. When all the changes are done, the enriched action is sent to the generation module to be executed. This generation module separates the emotional enrichment process to the platform where the actions should be executed. This enables the interoperability of the system among different platforms using the same approach. The details about the EES implementation used in this work could be found in the Chapter 5.
4.3 Emotional Model This module updates the system’s emotional state. This update could be done using one or a combination of the existent psychology theories (e.g., [47, 80, 93]). However, the hidden assumptions that authors make in their models [20, 71] emerge when trying to implement them in artificial agents and robots. It makes these models difficult to use in dynamic environments, as theatre is. The details about implementation of this module could be found in the Chapter 6.
4.4 Action Decision Humans have in mind an ideal flow of actions that they can perform in particular situations. If at some point they cannot continue with the successive action, they will select another action that they already know that could help them to continue doing their task. If they do not know any action that could help them, they will select a default action that could be just do an afraid face. Following this idea, this module starts reading the initial beat, as it is specified in the script. The script contains all the necessary information for the robot’s performance. The script is a directed weighted graph where each node is called beat, which is an envelope that adds meta-information and pre-conditions that should be fulfilled to execute the compound action inside the beat. The meta-information is related to the additional information that could be relevant for the system; for example, it could give information for the current situation, or the character’s relation with others. This approach allows us to generate responses to possible situations that could occur during a play. The weights on the arcs correspond to the desirability of the alternative actions. Every time that this module detects the end of an action, it evaluates the conditions for the ideal beat successor. If the conditions are verified, the beat is executed. However, if the current situation is not compatible with the ideal beat, the system will check if any of the alternatives suit in the current situation. If none of the alternatives is feasible, the system will execute a “panic beat”, which has no conditions. However, if more than one alternative is feasible, it is selected the one with higher desirability. A graphical example of a script is depicted in Figure 4.4, where an ideal flow of actions to be executes and different possibilities in different circumstances are shown. 53
Chapter 4. Architecture
Figure 4.4: Script example that shows possible flows of beat (B) with some alternative compound actions (CA), and a set of compound actions. If, for some reason, no compound action can be selected, a Panic action has been defined to try to overcome the problem. The dash arrows show that all beats or compound actions are connected with the panic action.
4.5 Description The description files allow the parametrization of the system and its adaptation to different circumstances. The system has the following parametric data: Emotion description gives information of the parameters that should be changed in all the simple actions to express the desired emotion. Therefore, for each combination emotion-action should be given a description of how the parameters are defined. If there is any action that does not have any specification, the system will not change its parameters. The following is the EBFN: hemotion descriptioni|= ’{’hnamei ’,’hobservationi ’,’hactioni’(,’hactioni)* ’}’ hnamei|= ’emotion:’hstringi hobservationi|=’observation:’hstringi hactioni|=hstringi’:{’hdescriptioni ’,’hactions affectedi ’}’ hdescriptioni|=’description:{’haction namei’,’hemotioni’,’ hparameter’s typei’}’ haction namei|=’emotionProfileAction:’hstringi hemotioni|=’emotionProfileEmotion:’hstringi hparameter’s typei|= ’movement_parameter’ hactions affectedi|=’actions:{’haction parameteri (’,’haction parameteri)*’}’ haction parameteri|= hstringi’:{’hreferencei’,’ hrepetitioni’,’hparametersi’}’ hreferencei|= ’reference:’hnumberi 54
4.5. Description
hrepetitioni|= ’repetition:’’yes’|’no’ hparametersi|= ’parameters:[’ hparameter descriptioni (’,’hparameter descriptioni)* ’]’ hparameter descriptioni|= hmovement parameter descriptioni|hnew parametersi hmovement parameter descriptioni|=’{time:’hnumberi’,space:’ hnumberi’}’ Character’s emotions parameters give the system information about character’s rhythm for each pair action-emotion. In other words, it is going to say how the emotional parameters for the desired action should be modulated. If there is not any information, the system will use the default emotional values for the action. The following is the corresponding EBFN: hcharacter emotion profilei|=’{’ hemotioni’,’hactionsi’}’ hemotioni|= ’emotion:’hstringi hactionsi|= ’actions:[’ hactioni(’,’hactions descriptori)* ’]’ hactioni|= ’{ action: ’ hstringi ’,’’bias:’ hnumberi ’,’ ’amplitude:’ hnumberi , ’long:’hnumberi } Emotional model parameters are used to change how the emotional system updates the emotions. hemotional modeli|= ’[’hemotion descriptori(’,’hemotions descriptori)*’]’ hemotion descriptori|= ’{ name:’hstringi ’,’’window_size:’hnumberi ’,’’increase_factor:’hnumberi ’,’’decrease_factor:’ hnumberi } Scene configuration gives information about the stage dimensions, as well as the expected objects’ localization on the stage. Script contains all the necessary information for the robot’s performance. The following is a script’s EBNF: hscripti|= ’{’ hinitial beati ’,’ hfinal beati ’,’ hpanic beati ’,’ hbeati (’,’ hbeati)* ’}’ hinitial beati|= ’initial:’ hIDi hpanic beati|=’panic:’ hIDi hfinal beati|= ’final:’’[’ hIDi(’,’hIDi)* ’]’ hbeati|= ’beat:{’ hIDi ’,’ ’[’hsuccessori’],’’[’hconditioni’],[’hactioni (’,’hactioni)*’]}’ hsuccessori|= ’ideal:’hIDi (’,’’[’halternativei(’,’halternativei)*’]’)* halternativei|= ’alternative: {’ hnumberi ’,’ hIDi ’}’ hconditioni|= ’position’|’landmark’|’emotion’ hactioni|= hsimple actioni|habstract actioni|hcontexti hsimple actioni|= ’{’ haction headeri ’,’’parameters:[’ hsimple action parametersi ’]}’ habstract actioni|= ’{’ haction headeri ’,’’parameters:[’ habstract action parametersi ’]}’ 55
Chapter 4. Architecture
haction headeri|= ’type:’ haction typei ’,’’name: ’ hstringi ’,’ his primaryi hsimple action parametersi|= ’{’hparameter headeri ’,’ hparameter descriptioni ’}’ habstract action parametersi|= hsimple action parametersi (’,’ hsimple action parametersi)* hcontexti|=’{’ haction typei ’,’ hemotion synci ’,’ haction synci ’,’ his primaryi ’,’ ’information:’ hstringi ’,’ ’actions: ’ hactioni ’}’ hparameter headeri|= ’type:’ hparameter typei ’,’ ’name:’hstringi hparameter descriptioni|= hparameter amplitudei|hparameter circlei|hparameter landmarki| hparameter pointi|hparameter speechi| hparameter squarei|hparameter timei hparameter amplitudei|= ’amplitude:[’hnumberi’,’hnumberi’,’hnumberi’]’ hparameter circlei|= hpointi(’,’hposei)?’,’’radio:’hnumberi hparameter landmarki|= ’landmark_id:’hIDi’,’’landmark_type:’ hlandmark typei(’,’hposei)? hparameter pointi|= hpointi (’,’hposei)? hparameter squarei|= hpointi (’,’hposei)? ’,’’delta:[’ hnumberi ’,’ hnumberi ’]’ hparameter timei|= ’time:’hnumberi’,’ ’emotional_time:’ hnumberi ’,’ ’repeat_emotional:’ (’yes’|’no’) hpointi|= ’point:[’ hnumberi ’,’hnumberi ’,’hnumberi’]’ hposei|= ’pose:[’hnumberi’,’hnumberi’,’hnumberi’,’hnumberi’]’ hlandmark typei|= ’Persona_Landmark’|’Object_Landmark’| ’Place_Landmark’ hparameter typei|= ’mandatory_parameter’|’optional_parameter’ his primaryi|= ’yes’|’no’ haction typei|= ’parallel_context’|’serial_context’|’simple_action’| ’composite_action’ hIDi|= hdigiti(hdigiti)*
4.6 Summary This chapter has introduced the architecture proposed to create theatrical robots. The architecture was designed with two main objectives: (i) let robots be an actors in theatre performance and (ii) reuse of diverse components in other projects or fields. To achieve these objectives, the architecture was divided in six-subsystems: description, feature, belief, action decision, motivation and emotional enrichment. Some modules suggested for these sub-systems are not required for the development of a theatrical robot actor, but those elements are considered as important on the study of features that could be improve the acceptance of social robots. For example the emotional model is not required in a theatrical presentation because the emotional state is given in the script. However, developing this module let towards the creation of social robots that could show empathy to humans. The following two chapters describe in more detail the action modulation sub-system and the emotion selection module.
56
CHAPTER
5
Action Modulation
The main idea of Emotional Enrichment System (EES) is to blend an emotion and action to produce an "emotional action". However this is not far from what previous works have done, therefore, the system has been envisioned to also allow: Interoperability among different platform. Introduction of new parameters and emotions. Interface with diverse action decision system. As a consequence, the system could be thought as a box that blends a desired action, action’s parameters and emotions together. This is done without paying particular attention to who or how the decision to execute a particular action with specific parameters were made. This is achieved through the use of text messages to describe actions that should be executed, its parameters, desired emotion and emotion’s intensity. To change the emotion without affecting the executing action, the information is divided into two messages: one for the action and its parameters, and other for the emotion and its intensity. Every time the system receives a new action, it verifies if the action has an implementation in the system and the parameters corresponds to the action. If these two conditions are met, then the system checks if the action is compound or simple. If it is a simple, the system proceeds to add all the required actions (emotive actions) and change their parameters to convey the desired emotion. On the other hand if the action is compound, it is first decomposed in simple actions (mandatory actions), which are then processed to add their emotive actions. However, this addition is done taking in consideration to not add any action that was previously added. Once all simple actions 57
Chapter 5. Action Modulation
and their parameters are determined, the system proceeds to verify the drivers’ existence to execute these actions in the desired platform. If the system finds out that one or more mandatory actions could not be executed, it triggers a message to inform that the desired action could not be executed. This description corresponds to our final model and implementation. A system’s exploratory implementation was done first, to verify if the design were suitable for the system’s requirements. The design, implementation and limitations are discussed in the next section. Then, the Emotional Execution Tree, simple and compound actions are formalized. Finally, a system’s implementation description is given.
5.1 First Emotional Enrichment System The system’s first implementation was done with the idea to verify if the design could fulfill the requirements. So, a UML description of the system was done. During this design phase, a special attention was given to parameters’ description, actions and emotions. They were modeled as abstract classes that should be implemented for each specific parameter, action or emotion such as it is suggested in the composite design pattern [37]. The system uses abstract classes and calls methods in the abstract class that creates a list with the simple actions that should be executed. This list is generated in abstract’s class instances. Once the list were generated, the actions contained in it are informed with the desired parameters, which were already modified to convey the selected emotion. Despite the good modeling, the design was changed due to the restrictions introduced by ROS. They had a big impact in the design because ROS messages could just be created from ROS basic types or other ROS messages. This close the possibility to share objects among ROS nodes. Hence, a ROS message with all parameters’ fields was created, which allows the use of one ROS channel to share the parameters. This modifications made the system to be highly integrated to ROS, which made the system dependent of ROS. Once the implementation was finished and tested, it was notorious that the current design did not allow the description of sequential actions or the use of compound actions that are not been already implemented. To overcome this limitation, the system’s uses cases were done again, this time considering the lessons learned in the first implementation. As a result of this analysis, it was decided to model compound actions as trees, which let the modification of actions’ parameters and addition of actions that should be used to express a desired emotion.
5.2 Emotional Execution Tree Emotional Execution Tree (EXT ) is a representation of sequence of actions that should be executed in a precise order. EXT is a connected acyclic graph G(V, E) with |V | vertices and |E| edges. The root and non-leaf nodes could be parallel or sequence type. The parallel node could be one out of four different sub-types: action and emotion synchronous, action synchronous and emotion asynchronous, action asynchronous and emotion synchronous, or action and emotion asynchronous. Sequential nodes could just be one of two sub-types: emotion synchronous or asynchronous. Action synchronous means that each time that a parallel node receives a finish notification (success or failure), it will notify others nodes. On the other hand, emotion synchronous means that 58
5.2. Emotional Execution Tree
each time that a node (sequence or parallel) receives an emotion synchronization message, it will propagate the message to the other nodes. Thus all the nodes that receive the message update their emotional parameters. This distinction creates the possibility to make emotional changes without affecting the actions’ execution. Finally, the leaf nodes could only be simple action (SA) nodes. All the nodes can be on one out of two levels: principal or secondary. If a node is principal, it will notify its predecessor about the messages that it has received, while the secondary cannot propagate any message to its predecessor. 5.2.1.
Simple and Compound Actions
To achieve emotional enrichment, it was necessary to formalize two parametric actions (simple and composite) introduced in Section 4.1. Simple actions (SA) are functions that map a set of parameters ppl i ∈ P to specific platform’s (pl) movepl pl pl pl pl ments (mj ∈ M ), which could be seen as (mpl for j = sa ({pi })). Each sa each specific action could have different implementations due to different reasons such as: type of platform, goal of the action, the mapping of inputs to outputs, among others. Therefore, two actions are considered different if (∀sai , saj ∈ SA, ∀q, t ∈ P L|P aram(saqi ) 6= P aram(satj ) ⇒ saqi 6= satj ), where P L is the set of all the possible platforms, P aram(·) returns the input parameter of a sa (e.g., linear_velocity, pitch,textT oSay). On the other hand, two actions are considered as equals if (∀sai , saj ∈ SA, ∀q, t ∈ P L|P aram(saqi ) = P aram(satj ) ∧ Obj(saqi ) = Obj(satj ) ⇒ saqi ≡ satj ), where Obj(·) returns the set of sa’s objectives. This SA equivalence allows to avoid the specification of all the platforms for all the actions that share the same objective and hide leaf nodes that do not give relevant information to understand the emotional tree. Now it is possible to define emotion enrichment as (∀sa ∈ SA, ∀e ∈ E, ∃ en | en = Enrichment(sa, e, P aram(sa), Intensity(e))), where Intensity(·) returns the intensity of a given emotion, and en can be a sa with different parameters than the original one, or even a set of sas. This definition goes along with the idea that new sas could be added to convey a specific emotion. Moreover, the Enrichment(·) function has the following properties: (∀sai , saj ∈ SA, ∀e ∈ E | sai ≡ saj ⇒ Enrichment(sai , e, ...) ≡ Enrichment(saj , e, ...)) (∀sai , saj ∈ SA, ∀e ∈ E | (sai 6≡ saj ) ⇒ Enrichment(sai , e, ...) 6≡ Enrichment(saj , e, ...)) Moreover, the enrichment can be done with any type of mapping, e.g., a linear function or an exponential function. SA enable the robot to convey emotions by changing their parameters, such as velocity and acceleration. However, the small number of these actions could be seen as a strong restriction on reusing complex actions, which could be a disadvantage in real applications where it could be tedious to always describe the same EXT . To overcome this limitation, the concept of composite action (CA) is used. A ca is an action that is created from other ca and sa. Similarly to sa, ca is defined as g = ca({pi }), where (pi ∈ P aram(sa)), (sa ⊆ SA), (g ∈ EXT ). It is important to highlight that ca is an abstraction that enables to group simple actions and improve the readability of EXT . 59
Chapter 5. Action Modulation
To illustrate the power of ca, let us consider the example shown in Figure 4.3. First, we should start by defining a set of SA like speak, move_body, balance_arm, where P aram(speak) = {text, pitch}, P aram(move_body) = {position, v, ω}, and P aram(balance_arm) = {id, angle, ω}. Second, the ca walk is described as follows: P aram(walk) = {position, v, angle}and P arallel(balance_arm(lef t, angle, ω1 ) balance_arm(right, −angle, ω1 ) move_body(position, v, ω2 )) where P arallel(·) means that the n actions given as parameters are executed in parallel. Finally, the compound action called move_talk is described as: P aram(move_talk) ={position1 , position2 , angle1 , angle2 , text} move_talk =P arallel(Sequence( walk(position1 , v, angle1 ), walk(position2 , v, angle2 )), speak(text, pitch) Where Sequence(·) means that the n actions are executed in sequence. 5.2.2.
Nodes Types
To made the implementation of the EXT there were identified four nodes: sa, sequence, parallel, and ca. The first three nodes could be either principal or secondary. If a node is principal, it will notify its predecessor about the messages that received, while the secondary cannot propagate any message to its predecessor. Algorithm 1 describes the execution of SA’s nodes. The sequence node executes the next action when the current principal action finishes, otherwise it will notify to its predecessor the error. Algorithm 2 describes how parallel nodes are executed. CA nodes are abstractions for sub-trees used to represent compound actions, which reduce the effort to re-write compound actions.
5.3 Emotional Enrichment System The final version of the emotional enrichment system uses the EXT to model any desired compound action. The definitions of compound actions are given in a JSON format according to the grammar defined section 4.5. 5.3.1.
Design
Abstract classes were used as base for concrete classes of parameters, simple actions, compound actions and execution tree. The difference respect the previous design 60
5.3. Emotional Enrichment System Data: Action parameters, Emotion parameters Result: Failure or Success while not Finish and not Failure do if has Emotion then modify action parameters; if has finished emotion step and is_principal then synchronize emotion; notifyPredecessor(synchronize emotion); end end execute action step; end if is_principal then notifyPredecessor(terminate action); end Algorithm 1: Simple Action Node. Data: Execution Sub-Tree Result: Failure or Success while !hasFinished(principal_branch) do if New Emotional synchronization Message then foreach branch do synchronize emotion; end if is_principal then notifyPredecessor(synchronize emotion); end end end foreach branch do terminate action; end if is_principal then notifyPredecessor(terminate action); end Algorithm 2: Parallel Node
is that the idea to use objects to share information in the system was discarded, instead JSON messages were used. As a consequence the parameters’ instances generate JSON messages that could be transmitted as string in the ROS messages. Therefore, most of the changes are done in the parameters’ implementation, which generate the JSON description to share among the ROS nodes. Composite pattern is used to describe the execution tree, such as is shown in Figure 5.1. Two enumerations were introduced to constraint and give additional information to people who would extend or use the system. The ContextPriority enumeration constrains the possibilities to the ones recognized . The other enumeration, ContextType, gives information about the type of actions’ execution (sequential or parallel). For this implementation, special attention was taken in the integration of the system with ROS. Therefore, a ROS node creates an instance of the system and calls the methods exposed by the system. As a consequence, additional processes to receive and 61
Chapter 5. Action Modulation
Figure 5.1: Emotional Execution Tree’s Model. The two enumerations restrict the values that could be given to the type of context and priority.
Figure 5.2: General system design. Each simple action corresponds to one ROS node, and there is just one node for the emotion enrichment system. The ovals represent the ROS topic parameters, rectangles represent black boxes, and texts outside containers represent input files that contain the system parametrization.
communicate with the other nodes are done in the ROS node such as is depicted in Figure 5.2. It also shows that the mediator class is used in each simple action node to hide the existence of diverse platforms. In other words, this mediator class delivers the message to the correct platform’s driver. If there is not any driver for the desired platform, the mediator will just ignore the execution of the action. To reduce the amount of topics generated for each action, just three topics were used to communicate action parameters, emotional modification parameters and synchronization messages. The use of two different topics instead of one to communicate parameters is based on the assumption that changing the emotional parameters should not affect the current execution. 5.3.2.
Emotion Enrichment Process
The emotion enrichment is obtained through the following phases: 1. Generation of emotional execution tree: this phase starts every time that a new ext is received, which must be given in a JSON format. The process begins by 62
5.4. Summary
parsing the format, verifying that the actions described on it exist in the system, and that the parameters correspond to the ones expected by the action. This is done using the action description associated to each action. When the verification is completed positively, each node in the ext is created and filled. 2. Emotion addition: uses the ext created in the previous phase. New sas are added to the ext and the sas’ parameters are modified following the emotion description, which is generated from files. This process is broken down in two steps. First, all the actions that are required to convey the desired emotion, and that are not yet present are added. Second, the emotional parameters are modulated based on the emotion’s intensity and character traits. 3. Execution: this is the last phase and it is done after the ext is ”coloured” with emotional characteristics (actions additions and emotional parameters). The algorithm 3 depicts the flow followed during the execution. The decision to have two different communication channels, one for action parameters and another for the action emotional parameters, was taken to enable the possibility to update the emotional parameters without interfering with the current execution. Data: Emotion Execution Tree (extt ) Result: Failure or Success do if extt != extt−1 then stop(extt−1 ); load(extt ); end if newSynchMessages() then if isEmotionalSynch() then propagateEmotionSynch(); end else if isActionSynch() then propagateActionSynch(); end end if updateActions() then sendActionParameters(); if hasEmotion(extt ) then sendEmotionalParameters(); end end while hasFinished(extt ); return state(extt ) Algorithm 3: Emotion enrichment execution.
5.4 Summary This chapter has described the Emotional Enrichment System, which has been designed and implemented to enrich robots’ movements with emotions. To achieve this, it was used an Emotional Execution Tree created from three different types of nodes: simple actions, parallel, and sequential. Simple actions are functions that map a set of 63
Chapter 5. Action Modulation
parameters to specific movements. Sequential executes in order the sequence of actions associated to this type of node, while parallel executes all the action at same time. To enable synchronization among simple actions, parallel nodes could be one of four different subtypes, while sequential just one of two different subtypes. A formalization of the Emotional Execution Tree and the principal consideration during the implementation of the system has also been provided.
64
CHAPTER
6
Emotion Selection
Social environments involve subtle interaction among people and the physical environment. These interactions, context in which they take place, and people’s mental perception of the world affect the emotions that arise. Different theories and models of how emotions arise have been proposed in psychology, such as [47, 80, 93]. Although these models seem acceptable and cogent to most of us, the hidden assumptions that authors make in their models [20, 71] emerge when trying to implement them in artificial agents and robots. Computational frameworks based on these “high-level” models have been implemented [29, 40], but they use abstract concepts that have to be defined in the system. As suggested by Izard [47], among others, emotion elicitation can be performed at different levels, and at some of these levels people are not aware of the emotional process. The system here described focuses on the “reactive”, “pre-aware” part of emotion elicitation (and selection) using as an input gray scale images from a web cam and it corresponds to the emotion selection module introduced in chapter 4. The system does not take in consideration any cognitive information from the environment; instead, it compares two consecutive images and calculates the percentage of different pixels. Algorithm 4 show how the this calculation is done. This percentage difference is given as input to the stimulation calculator to determine the “stimulation” that is later used by the emotion generator to update the intensity of each emotion. This update is done searching for the patterns suggested by Tomkins [48] (Figure 2.5). The previous process is always modulated by the time delay between the two images considered. This delay is of vital importance in the system because it could not be determined with certainty beforehand. Consequently, the system gives different values of “stimulation” depending on the delay between two consecutive images. 65
Chapter 6. Emotion Selection Data: Imaget , Imaget−1 , threshold, n and m Result: percentage percentage = 0 for i < n; + + i do for j < m; + + j do if |Imaget (i, j) − Imaget−1 (i, j)| > threshold then + + percentage end end end percentage = percentage n∗m return percentage Algorithm 4: Difference percentage calculation. Where n and m are the rows and columns of the image, respectively.
Figure 6.1 depicts the general process with all the subsystems. These subsystems were selected to enable upgrades in the system without the need to make considerable changes in the code. For example, the change detector subsystem could be changed to another subsystem that detects additional features from the images; if the new module maintains the post-condition, output must be percentage (value between zero and one), the rest of the system would work normally.
Figure 6.1: General architecture of the system. The arrows show the information flow. The time difference between two images is used in the stimulation calculator module to calculate the stimulation.
6.1 Stimulation Calculator This subsystem receives the percentage from the change detector and updates the new stimulation (stimulus(t)) based on the current change (s_increment), the last stimulation (stimulus(t − 1)), and a reduction value (s_decrement). Equation 6.1 66
6.1. Stimulation Calculator
presents the formula used to update the system. In addition to stimulus(t − 1), functions s_increment and s_decrement use the time delay (delay) as a parameter. stimulus(t) = stimulus(t − 1) + s_increment(percentage, delay) + s_decrement(stimulus(t − 1), s_increment(percentage, delay), delay, bias) (6.1) The s_increment function ranges on the percentage of change and the delay time, and it is computed as shown in equation 6.2. The s_increment uses an exponential function with desired base (base_increase) and displacement coefficient (d). This displacement coefficient is used to obtain values greater than one, but it also introduces a small bias that is corrected by the second part of the equation. The increase_f actor is a coefficient that modulates the gain of the function, which is used to obtain less or more stimulation. delay is a variable whose value comes from the time delay between the two pictures used to generate the percentage. s_increment(.) = ((base_increase)(percentage−d) − (base_increase)d ) ∗ increase_f actor ∗ delay | {z }
(6.2)
correction factor
Figure 6.2 illustrates the behaviour of s_increment(.), showing that this function produces greater values when the delay increases.
Figure 6.2: Behaviour of the increase function for different delays in the image using parameters base_increase = 30, d = 0.1, and increase_f actor = 10.
The s_decrement(.) function (Equation 6.3) uses s_increment(.), time delay, and a bias to determine the decreased value. The parameter bias is used to modify the lower output value of the system. Like s_increment(.), this equation depends on time to make the modulation. Figure 6.3 illustrates its behaviour with a decrease_f actor = −0.5 and with different time delays. 67
Chapter 6. Emotion Selection
s_decrement(.) = (stimulus(t − 1) + s_increment(.) − bias) ∗ decrease_f actor ∗ delay
(6.3)
Figure 6.3: Behaviour of the decrease function for decrease_f actor = −0.5 and different time delays.
6.2 Emotion Generator This subsystem was divided in two modules (event generator and pattern detection) to give the possibility of adding or deleting new emotion patterns, and of modifying the event characteristics. Event generator centralizes the process of detection of relevant events from the stimulation slope. The events considered are: null, small, medium, large, and huge slope. Except null slopes, the other events could be either positive or negative. A pattern detection module is implemented for each emotion that should be detected. Each pattern detection module considers a different pattern as well as the number of events to search for in the pattern. The emotions, their patterns, and their update functions are: Surprise is recognized when one of the following events are present in its time window: large or huge positive slope. Due of this strong constraint, every time that this pattern is detected, its intensity grows faster than that of other emotions. Fear is increased when three or more consecutive recent events have either large or huge positive slopes. Interest occurs when three or more consecutive events have either medium or small positive slopes. Relief works with negative slopes, in contrast with the other emotions, and its intensity increases when at least five negative slope events are detected. It was considered only emotions affected by stimulus change since the system is focused on the reactive processes to activate emotions, while emotions related to constancy of stimuli in Tomkin’s model are expected to be managed by the cognitive part of a larger system. 68
6.3. Design and Implementation
6.3 Design and Implementation The system design was done keeping in mind the goal of maintaining the modularity. Therefore, it was decided to define a class to make the interface with ROS, create an instance of the system and give the information to the system about the localization of the configuration files. This class was called NodeEmotionalModel. To maintain the possibility to integrate other methods for emotion selection, it was created the class EmotionModel, too. This class has the responsibility to create all the required instances to select an emotion. In the current implementation, this class were ReactiveModel and all the classes derived from the EventAnalyzer class. The ReactiveModel is in charge to determine the stimulation using the equation 6.1 and then to generate the event corresponding to this stimulation. This result is then given to all the implementations of the abstract class EventAnalyzer. Each implementation is in charge to detect a given pattern and update the current status of its emotion. With all of these results, the NodeEmotionalModel determines the final emotion and its intensity. The class diagram is shown in Figure6.4.
Figure 6.4: Emotional System UML Class Model.
The system was implemented in C++ and uses OpenCV [12] to analyse images. To facilitate easy parameter change, two configuration files were added: one related to all the general parameters (e.g., threshold and increasecoef f icient ) and the other to establish the increment, decrement, and time window (number of events to consider) for each of the implemented patterns.
6.4 Results The system was tested online with information coming from a Logitech CY270 Web-Cam. The intensity and events obtained are depicted in the Figure 6.5, where the relationship between the stimulation’s slope and the events can be seen. 69
Chapter 6. Emotion Selection
Figure 6.5: Stimulation (continuous line) and events (dots in horizontal lines) obtained from the continuous comparison of two consecutive images. The y-axis on the left represents the stimulation level, while the one on the right represents the events generated from the slope analysis.
Figure 6.6: Intensity obtained by our system for the four emotions implemented: fear (blue), interested (purple), surprise (red) and relief (yellow).
Figure 6.6 depicts the intensity obtained for each implemented pattern (fear, interest, surprise and relief), also showing that each pattern module updates its emotion intensity independently. This is clearly seen at second 120 when fear, interest, and surprise unevenly increase their respective intensity and after some time they also reduce their intensity unevenly. This (increase and decrease) unevenness is related to the the pattern configuration, which is not the same for each emotion. The presence of more than one emotion with a value different from zero suggests that a further mechanism should be used to determine which emotion should be elicited, for example taking the one with higher intensity or just modifying behaviour parameters proportionally to each intensity. In other words, our initial aim to use this system as first step to select an emotion in a non-structured context has been achieved. 70
6.5. Summary
6.5 Summary This chapter has introduced a reactive emotional system based on Tomkins’ theory. The system is modular to permit its integration with more complex systems and its configuration based on the output from the pattern detection modules. The system was implemented in C++ with interface to ROS to make it possible to use it in other models and in robotic platforms. Four patterns (fear, surprise, interest, and relief) were implemented and tested. The results show that the output compete with each other, and the emotion has to be selected in a further step with a logic that suits the specific purpose, which could be as simple as take the emotion pattern with higher intensity, or weight behaviours by the intensity of the corresponding patterns. Additionally, this reactive system could be used as complement for cognitive systems.
71
CHAPTER
7
Pilot and First Case Study
A pilot case was done using an informal procedure, by just showing people the robot’s movement and asking them which emotion the robot was trying to show. This was done during the 2013 Researcher’s Night, a European event with general public participation. The prototype platform (Figure 3.4) was used during this pilot. The emotions designed and showed in this phase were: happiness, anger, sadness and fear. People’s comments and perceptions were informally collected by direct questions. The main conclusion of this pilot was that it was necessary to add new features to express emotions like happiness and fear, which were not well perceived by people. With the updated version of the platform (Figure 3.7), it was decided to do a first case study at the Museum of Science and Technology in Milan, where high school students and families were coming for an event that lasted 4 days. The final features used on this case study are depicted in Figure 7.1. After these new features were added a first case study was done. In this case study a new version of the questionnaire were also used, which had ten emotions enlisted but just eight implemented. The results show that it is possible to convey emotions with quite simple and general features, but some effort is needed to tune the features so that the emotion projection could be more widely accepted.
7.1 Design and Setup Each group of participants was exposed to three rounds, in each of which the robot was performing a different emotion. The subjects were asked to mark which emotionrelated term best described what they believed the robot was trying to show, selecting among a set of nine possible options: anger, curiosity, disgust, embarrassment, fear, happiness, sadness, pride, and neutral. It was also added the option to answer: “Un73
Chapter 7. Pilot and First Case Study
Figure 7.1: Example of the sequence of movements done for the robot in its bottom part. The upper shows a sequence without any angular movement for a total of xmeters. The bottom shows the same displacement, but this time with an oscillation between [θ, −θ] and angular velocity ω .
known”. Although these ten options were enlisted, just the first eight emotions were implemented and presented. To avoid that spectators could be influenced by previous trials, different emotions were shown to each group. It is important to highlight that five out six Ekman’s basic emotions were implemented, while the other correspond to second level emotions. Additionally, it was decided to locate the robot 1.5m away from the participants and with its front facing them every time that a new emotion would be showed, as it is depicted in Figure 10.4.
7.2 Emotion Description Most of the emotion projection descriptions found in literature, summarized in [23, 51], are described using Laban’s theory [58]. This theory expresses human movements in fuzzy terms that could not be precisely described numerically. As a consequence, it was decided to follow an empirical design, consisting in taking into account the features that could be changed in the robot base and in composing them to express some basic emotions. The first implementations where tested and tuned basing on the emotions designed by the researchers and perceived by them from the robot’s movements. The emotions implemented in this case study correspond to five out six emotions suggested by Ekman [31] as basic emotions. These emotions are disgust, happiness, sadness and anger. The emotion surprise was not implemented because it was considered as an emotion that takes just few second to be elicited. Additional to these five emotions, it was implemented two secondary emotions: embarrassment and curiosity. The features that could easily be perceived by the observer are summarized in table 7.1, and each selected feature is described here below: Speed is the target speed for the robot during the movement. It was discretized on five values: very slow (100 mm/s), slow (200 mm/s), normal slow (300 mm/s), normal (400 mm/s), and fast (800 mm/s). 74
7.3. Study
Figure 7.2: First case study setup.
Front/Back represents the fact that the robot moves backwards at some point in its movement, before going forward again; the possible values are: “yes1”, which means that the robot just go back once, “yes2” when the robot goes back twice, and “no” if the robot goes only forward. Shoulder considers the movement of the upper part: “asymmetric” when the two beams move asymmetrically, alternatively one forward and the other backward, “close”, when the upper parts get close to each other, a combination of the two (“close-2”), and “none”. Shoulder Amplitude is the maximum angle that the two beams have to rotate. There are five possibilities: “none” (0◦ ), “small” (10◦ ), “medium” (30◦ ), “large” (50◦ ), and “huge” (70◦ ). Angular Velocity ω is classified as: “none” (0 mm/s), “slow” (300 mm/s), “medium” (500 mm/s), and “fast” (800 mm/s). Body Rotation Amplitude is the maximum rotation angle that the robot base can reach during the linear movement (it is a holonomic base); it could be: “none” (0) rad, “small” (0.1) rad, “medium” (0.3) rad, “large” (0.5) rad, and “huge” (0.7) rad.
7.3 Study There was no specific group size, since this was determined by the amount of people that got close to the booth. Therefore, there were groups with just one person and others with 24 persons. The total amount of volunteers was 154: 55 males, 26 females, and 73 that did not specify their gender. The average age was 21.95 with a standard deviation of 12.26, minimum age was 5, maximum 62.
7.4 Results Table 7.2 shows the results obtained in the first case study. 75
Chapter 7. Pilot and First Case Study Table 7.1: Features that could be perceived by the audience, their modalities. Where Embarr. is Embarrassment and Asy. is Asymmetric. Shoulder Amplitude Medium None
Angular Velocity
Body Rotation Amplitude
None Slow
Very Small Small
Small
None
None
No No
Close-2 + Asy. Asy. Asy.
Large Medium
Slow Fast
Small Small
No No No
Few Asy. Asy. Asy.
Small Medium Medium
None Slow None
Very Small Large None
Emotion
Speed
Front/Back
Anger Sadness
No Yes-1
Asy. Close-2
Yes-2
Embarr. Happiness
Fast Very Slow Normal Slow Slow Fast
Disgust Curiosity Neutral
Slow Normal Normal
Fear
Shoulder
Disgust
Embarr.
Fear
Happiness
Neutral
Pride
Sadness
Unk.
Tot.
Percentage
Anger Curiosity Disgust Embarr. Fear Happiness Neutral Sadness
Curiosity
``` ``` Reported ``` Presented ``
Anger
Table 7.2: Answers obtained the first case study. On each row is the emotion that was intended to express, and on the columns the reported emotions.
23 2 1 3 3 15 5 4
11 9 11 11 29 4 0 5
3 3 1 0 5 0 1 1
4 9 3 8 25 2 1 11
7 2 7 9 22 3 0 13
6 3 3 3 2 28 1 0
1 2 6 0 2 4 0 2
0 8 8 4 0 4 3 3
0 3 5 2 2 2 2 7
4 0 1 1 2 1 0 0
59 41 46 41 92 63 13 46
38.98% 21.95% 2.17% 19.51% 22.82% 44.44% 0% 23.81%
Table 7.2 shows that the best emotion perceived was Happiness and the worst one Disgust, with 44% and 2% respectively of accuracy. Moreover, some movements were designed to convey a specific emotion, but a different emotion was perceived. For example the intended Sadness was identified by 14% of subjects exposed to it, while 25.5% perceived it as Fear. The neutral behaviour was not much considered, possibly because people, in this setting, expected to see something specific, not a neutral movement. To analyse whether each emotional expression was misinterpreted with others a Fisher’s exact test was applied for each possible combination of emotions implemented. Additionally, a Holm-Bonferroni correction was applied for multiple comparisons to get better p-value estimation. The results showed in the Table 7.3 suggest that people perceived the following implementations as similar, having a p − value > 0.05: curiosity-disgust, curiosity-embarrassment, curiosity-sadness, disgust-embarrassment, disgust-sadness, embarrassment-fear, embarrassment-sadness, and fear-sadness. Moreover, an analysis was done per groups, for each presented emotion. Excluding unknown and uncertain answers, we have considered the answers of each subject. For each emotion we considered how many subjects in each group recognized it, how many identified a different emotion, how many identified the considered emotion when 76
7.5. Summary Table 7.3: Pairwise comparison among all the implemented emotions in the first case study using Fisher’s exact test with α = 0.05. The * indicates that the p-value was adjusted using the HolmBonferroni correction for multiple comparisons. Pair Compared Anger vs Curiosity Anger vs Disgust Anger vs Embarr. Anger vs Fear Anger vs Happiness Anger vs Sadness Curiosity vs Disgust Curiosity vs Embarr. Curiosity vs Fear Curiosity vs Happiness Curiosity vs Sadness Disgust vs Embarr. Disgust vs Fear Disgust vs Happiness Disgust vs Sadness Embarr. vs Fear Embarr. vs Happiness Embarr. vs Sadness Fear vs Happiness Fear vs Sadness Happiness vs Sadness
p-value 1.1e-6 3e-7 2.6e-4 1e-7 1.1e-5 5e-7 0.31 0.19 7.4e-5 5e-7 0.015 0.126 3.8e-6 2e-7 0.015 0.047 4e-7 0.139 1e-7 1.8e-3 1e-7
p-value* 1.43e-5 5.1e-6 2.34e-3 2.1e-6 1.21e-4 7.5e-6 0.504 0.504 7.4e-4 7.5e-6 0.105 0.504 4.6e-5 3.6e-6 0.105 0.235 6.4e-6 0.504 2.1e-6 0.0144 2.1e-6
Table 7.4: Example of table compiled for each emotion on the subjects that have been presented each emotion (here Happiness). ```
``` Reported Happiness ``` Presented `` Happiness 28 Other 30
Other 4 96
presented another one, and how many recognized an emotion different from the considered one when presented a different emotion. This lead for each presented emotion to a table like the one reported in Table 7.4 for Happiness. For each of these tables the classification accuracy and the no-information rate (NIR), i.e., the accuracy that had be obtained by random selection, have been computed using the R package CARET [36], as reported in table 7.5. It can be seen also from these data that some implementations of emotion expressions (i.e., Anger and Happiness) have been quite well recognized by the respective panels, while others are not. This may depend on the specific implementation of the emotion expression.
7.5 Summary The case study presented in this chapter studied empirical values for eight emotions. Five emotions were selected from the set of basic emotions suggested by Ekman [31]: anger, happiness, sadness, fear, and disgust. The other two emotions were selected from the set of emotions considered as secondary: embarrassment and curiosity. The last emotion was the neutral emotion. The values used for the implemented emotions 77
Chapter 7. Pilot and First Case Study
95% CI
No-information Rate
P-Value [Acc > NIR]
Anger Curiosity Disgust Embarassment Fear Happiness Neutral Sadness
Classification Accuracy
Presented emotion
Table 7.5: Classification accuracy of the presented emotions by the single panels, computed as mentioned in the text, with corresponding 95% confidence interval, no-information rate, and p-value that accuracy is greater than the NIR.
0.74 0.56 0.52 0.58 0.60 0.78 0.55 0.66
(0.65, 0.81) (0.45, 0.67) (0.40, 0.63) (0.47, 0.67) (0.53, 0.66) (0.71, 0.85) (0.32, 0.76) (0.56, 0.74)
0.61 0.61 0.54 0.60 0.58 0.63 0.55 0.62
1.0e-3 0.65 0.71 0.77 0.31 2.8e-05 0.59 0.25
were empirical selected, which was done through trial and error until the researchers thought that the movement was representing the desired emotion. The result shows that it is possible to convey some emotions using just movement and a non-bio-inspired embodiment. Moreover, showing emotional states in a context that puts in evidence the stimulus that produced them (as it would happen in a real application) might improve the perception of the emotions, as it is known also for human-human relationships, and this will be studied in a future trial.
78
CHAPTER
8
Second Case Study
Since the results of the first case study showed that the identification of most of the designed emotions was still quite poor, it was decided to try again with the same emotions, but this time a context to the emotional state of the robot was provided by playing a little scene. This is motivated by the fact that that emotions are responses to stimuli [18, 83].
8.1 Design and Setup The idea to introduce the emotion expression by a little scene comes from the fact that even people have a rough time trying to figure out others’ emotional state, if they do not know what happened to them. Thus, a little scene was created for each emotion. The procedure and questionnaire were the same as the one adopted for the first case. However, it was added a variant of the Fear emotion (Fear2), in which the robot finishes its trajectory far from the person. As done in the first case, each group was exposed to three emotions and the sequence presented to each group was different to eliminate the change to influence other groups. As in the first case study, the total distance covered by the robot remained in 1.5m, but this time the robot was facing the "actor" playing the scene. The distance from the participants to the robot and actor was 1.5m, as depicted in Figure 9.1. The "actor" was a non trained person in any performance art. 8.1.1.
Scene Description
It was created a total of each scenes, each one for the seven emotions implemented in the first case study 7. The last emotion corresponds to Fear2. The descriptions of the eight scenes are: 79
Chapter 8. Second Case Study
Figure 8.1: Setup of the second case study.
Figure 8.2: Sequence of actions for the angry scene.
Angry: the robot starts 1.5m far from the actor and it start to approach. The actor shows fear while the robot is approaching. The sequence of this scene is depicted in Figure 8.2. Curiosity: the robot starts 1.5m far from the actor and then approaches to him. The actor is down on his knees holding something in his hands. The sequence could be seen in Figure 8.3. Disgust: the robot is positioned near the actor giving him the ”back”. The robot moves away from the actor as depicted in Figure 8.4. Embarrassed: the robot is positioned near the actor giving him the ”back”. The robot moves away from the actor, who disproves the robot’s actions. The sequence is depicted in Figure 8.5. Fear-1: the robot starts 1.5m far from the actor and then approaches to him. The sequence could be seen in Figure 8.6. Fear-2: the robot starts 1.5m far from the actor and approaches to him. During the robot’s approach, the robot “reacts” to the actor’s movements getting away from the actor. The sequence could be seen in Figure 8.7. 80
8.1. Design and Setup
Figure 8.3: Sequence of actions for the curiosity scene.
Figure 8.4: Sequence of actions for the disgust scene.
Figure 8.5: Sequence of actions for the embarrassed scene.
Figure 8.6: Sequence of actions for the fear-1 scene.
Figure 8.7: Sequence of actions for the fear-2 scene. 81
Chapter 8. Second Case Study
Figure 8.8: Sequence of actions for the happiness scene.
Figure 8.9: Sequence of actions for the sadness scene.
Happiness: the robot starts 1.5m far from the actor and approaches to him. The actor waits the robot with his arms open. The sequence could be seen in Figure 8.8. Sadness: the robot is positioned near the actor giving him the ”back”. The actor ”scold” the robot. The sequence is depicted in Figure 8.9. The implementation of the emotions used in both case studies and the scenes could be seen in the video https://www.youtube.com/watch?v=AXAglJKLwbITheatreBot: Case Studies
8.2 Study The AIRLab Open Day in 2013 was used to perform the scenes and actions and to collect data. Eight groups participated to the study, each one made of approximately 20 volunteers for a total of 156: 51 males, 17 females, and 88 that did not provide their gender. The average age was 26.34 with a standard deviation of 12.44, and with a minimum age of 11 and maximum of 65.
8.3 Results Table 8.1 shows the results obtained in the second case study. In table 8.2, the comparison between the percentage of emotion recognition in both cases is reported. It shows that giving information about the context that produce the current emotional state of the robot improves its recognition. In the second case study, the results were considerably better, where four out of the nine showed emotions had a percentage of recognition higher than 60% (Anger, Happiness, Curiosity and Fear2). Another important point to highlight is that Sadness, Fear and Embarrassment were still perceived as different emotions. As it was done for the first case study, a Fisher’s 82
8.3. Results
Embarr.
Fear
Happiness
Neutral
Pride
Sadness
Unk.
Tot.
Percentage
Anger Curiosity Disgust Embarr. Fear Fear2 Happiness Neutral Sadness
Disgust
``` Reported ``` `
Presented
Curiosity
``` `
Anger
Table 8.1: Answers obtained in the second case studies. On each row is the emotion that was intended to express, and on the columns the reported emotions.
41 8 6 7 0 3 0 1 0
1 38 2 2 13 0 1 3 4
2 0 5 1 0 5 0 2 2
0 3 4 4 17 5 0 5 22
6 0 3 12 10 35 5 7 14
9 4 0 0 6 0 55 9 0
0 1 4 0 0 1 1 5 2
2 1 7 1 0 0 0 5 3
0 1 4 10 0 2 0 1 15
1 4 3 0 0 0 1 1 1
62 60 38 37 46 51 63 39 63
66.13% 63.33% 13.16% 10.81% 21.74% 68.63% 87.30% 12.82% 23.81%
exact test and Holm-Bonferroni correction were applied for each possible combination of implemented emotions. The results suggest that no implementation was considered as similar to others by the participants, since for all we have p−value < α. Comparing the percentage from both cases, it is possible to see that giving information about the context that produces the current emotional state of the robot improves its recognition. However, if the conveyed emotion does not match the scene, the audience does not recognize it. This shows that even giving information about the situation, and using a human actor to provide the context, people could not recognize the emotion if the correct features are not exploited. To verify this, it was done a comparison between the results obtained for each emotion in the first and second case doing a Fisher’s exact test. The results suggest that implementations of Disgust (p = 0.088), Fear (p = 0.206), and Sadness (p = 0.269) are perceived as the same in both cases. This would suggest that the combination of stimuli (scene + actor) and correct robot’s movements are necessary to the correct emotional interpretation. However, with the adopted setup, it was impossible to determine which factor was more relevant. Table 8.2: Comparison of percentage of correct emotion perception in the first and second case study. Tested Anger Curiosity Disgust Embarassment Fear Fear2 Happiness Neutral Sadness
First 38.98% 21.95% 2.17% 19.51% 22.82% 44.44% 0% 15.22%
Second 66.13% 63.33% 13.16% 10.81% 21.74% 68.63% 87.30% 12.82% 23.81%
Such as was done in the first case study, a contingency table for each emotion was elaborated. For each of these tables the classification accuracy and the no-information rate (NIR), i.e., the accuracy that had be obtained by random selection, were also computed. Table 8.3 shows the information for both case studies (First and Second). It can be seen also from these data that some implementations of emotion expressions (i.e., 83
Chapter 8. Second Case Study
Anger and Happiness in the first trial, and Anger, Curiosity, Fear2, and Happiness in the second trial) have been quite well recognized by the respective panels, while others are not. This may depend on the specific implementation of the emotion expression, as well as on the specific panels, and on the effectiveness of the contextual scene (in the second case study), and needs to be further investigated. For almost all the presented emotions, there has been an increment in the recognition rate from the first to the second case study, fact that supports our hypothesis that the context plays an important role to recognize an emotional act.
8.4 Summary This second case study if giving a context to the participants could improve the recognition rate of the emotions implemented during the first case study. A small scene for each emotion was created, which was performed by the robot and non-trained human actor. Although the results show that some emotions were recognized well than before, other emotions remained with similar or lower recognition rate. This opens the question about the influence that could have the human actor and the scene on the recognition rate. Although the study of their influence could be beneficial, it was decided to continue the studies on the values that could be used to express emotions. Therefore it was devise a third case study that uses values informed in the literature.
84
Presented emotion Case Anger Curiosity Disgust Embarassment Fear Fear2 Happiness Neutral Sadness
Classification Accuracy 1st 2nd 0.74 0.89 0.56 0.86 0.52 0.71 0.58 0.64 0.60 0.49 0.83 0.78 0.93 0.55 0.69 0.66 0.74
95% CI 1st 2nd (0.65, 0.81) (0.83, 0.93) (0.45, 0.67) (0.79, 0.91) (0.40, 0.63) (0.60, 0.81) (0.47, 0.67) (0.53, 0.73) (0.53, 0.66) (0.40, 0.57) (0.75, 0.88) (0.71, 0.85) (0.88, 0.96) (0.32, 0.76) (0.59, 0.79) (0.56, 0.74) (0.67, 0.80)
No-information Rate 1st 2nd 0.61 0.66 0.61 0.65 0.54 0.66 0.60 0.64 0.58 0.65 0.65 0.63 0.64 0.55 0.65 0.62 0.66
P-Value [Acc > NIR] 1st 2nd 1.0e-3 2.4e-12 0.84 9.5e-09 0.71 0.21 0.77 0.55 0.31 0.99 1.39e-06 2.8e-05 NIR]
Table 9.4: Classification accuracy of the presented emotions by the single panels, computed as mentioned in the text, with corresponding 95% confidence interval, no-information rate, and p-value that accuracy is greater than the NIR.
High/Low arousal Positive/Negative Anger Happiness Sadness Content
0.6931 0.5545 0.75 0.68 0.79 0.69
(0.62,0.756) (0.4831,0.6242) (0.687,0.81) (0.6142,0.7467) (0.7295,0.8458) (0.624,0.7559)
0.52 0.53 0.76 0.7129 0.7726 0.7525
8.349e-0.7 0.311 0.6646 0.8440 0.2819 0.977
91
Chapter 9. Third Case Study Table 9.5: Pairwise comparison among all the implemented emotions using Fisher’s exact test for both questionnaires with α = 0.05. The * indicates that the p-value was adjusted using the HolmBonferroni correction for multiple comparisons. Pair Compared Anger vs Happiness Anger vs Sadness Anger vs Content Happiness vs Sadness Happiness vs Content Sadness vs Content
List Questionnaire p-value p-value* 0.4903 0.9 2.85e-5 1.01e-5 1.2e-4 9e-5 7.81e-7 6e-7 2.2e-6 6e-7 0.7036 0.9
Open Questionnaire p-value p-value* 0.2879 0.492 6.09e-7 3.05e-6 7.9e-3 0.0239 1.8e-8 1.08e-7 3.4e-4 1.36e-3 0.246 0.492
Sadness and Content were associated with Fear, the last having 14 and 13 occurrences, respectively. Finally, the Content implementation was not correctly perceived, it had not more than four occurrences in second case. To verify this misinterpretation between implementations, a Fisher’s exact test was applied for each possible combination of emotions implemented, which gives a total of seven combinations. Additionally, a Holm-Bonferroni correction was applied for multiple comparisons to get a better p-value estimation. The results are shown in Table 9.5. As they show, the implementations of Anger and Happiness, and, respectively, Sadness and Content are interpreted as implementations that are similar, while the other implemented emotions are distinguished from each other. Moving forward from the results, it is evident that the current specification of Laban’s theory is not enough to be interpreted in a way that allows the generation of repeatable emotion movements. This is due to the fact that there is not a precise range of values for Laban’s effort descriptors for emotion, which leave the designer the interpretation of Laban’s effort. Moreover, the use of words to describe Laban’s variables, as it could be detected by a literature analysis, leave again to the designer the responsibility to interpret these words to give specific values that could be executed by a robotic platform. Also, the use of additional features could be required to convey the desired emotion. This is evident from the results obtained in this case study, where the p-values for all the four implementations were greater than 0.05, which means that the null hypothesis can not be rejected. In other words, the parameters used to convey the selected emotions do not correctly represent the emotions. This brings us to a crossroads with at least two possible ways to assess emotions in environments where time is a constrain, each one with pro and cons: List possible emotions. The main advantages of this approach are: possibility to assess big groups, instant response, and possibility to compare results among participants. Main disadvantage is that listing the motions constrains the participant to pick from emotions, which favour the expected results. However, this brings the following question up: how much the participant is influenced by the options? At a first glance, we could be tempted to conclude that it has a big influence, such as is suggested by the results of Anger, which recognition rate considerably increased when the list was used. But here, it is important to realize that there is not a body of knowledge that allows us to establish an underlying relationship among 92
9.5. Summary
the words, which is out of the scope of this work. For example the word power could be related to be angry because when someone tries to show to be powerful uses to express Anger. The participant can write a sentence or a word. The main advantage is that the participants are not driven to a desired result. On the other hand (disadvantage) the results could not be comparable due to the fact that people could use different words to describe the same object, but that these words could be interpreted differently, depending on many factors, included subjective ones.
9.5 Summary The third case study was done to assess the agreement among the participants to recognize the emotion presented to them by a non human-like robot. The emotions implemented and enlisted in the case study were selected as Sharma and collaborators [95] described. The study was conducted using two questionnaires: an open questionnaire where participants could write the emotion that they thought the robot was expressing, and the second questionnaire where they had to select an emotion from a list. The open questionnaire was used as a control mechanism to consider the perception of the participants without any possible bias that could be generated by an available list of emotions. The emotions enlisted in the multiple answer questionnaire were eight, two from each quadrant of the circumplex model. Regarding the methodology used to assess the emotional attribution, it could be concluded that the use of an open questionnaire is not going to give any significant insight because people will use many kinds of words (not only emotional) to describe the experience. Although somebody might say that these non-emotional words could be related to specific emotions, there are two reasons that may introduce errors in the results. First, there is not a precise guide that explains how to relate mental states and non-emotional words to emotions. Second, leaving the possibility to the researcher to interpret the data arbitrarily will close the doors to make comparisons with other works. The use of circumplex model of emotions showed to be useful in the design and analysis of the data. This model allowed us to detect that velocity is related to the arousal axis, information that could not been detected if we had used another mechanism to select the emotions presented by the robot and in the questionnaire (e.g., emotions randomly selected). However, the codification of emotional movements in Laban’s terms could give a general idea about the characteristics of movement to express an emotion, but they should be associated to precise values to lead towards the possibility to compare and replicate works in HRI.
93
CHAPTER
10
Experiment
An experiment was designed to assess precise values that could be used to express happiness, angry, fear and sadness, which correspond to four basic emotions suggested by Ekman [31]. The platform used in this experiment is platform presented in Figure 10.1, which corresponds to the lower part of the robotic platform 2.0 3.5. This decision was taken to reduce any possible contribution of the upper part. Moreover, the Emotional Enrichment System is used to project the desire movement.
10.1 Design The experiment was designed to get a better understanding about the contribution linear and angular velocity, oscillation angle, direction, and orientation to express happiness, anger, sadness and fear, which are four out of six emotions considered by Ek-
Figure 10.1: Holonomic platform used in the experiments. 95
Chapter 10. Experiment
Figure 10.2: Example of the features used in the experiment. x represents the displacement in meters, ω is the angular velocity (rad/s) and θ the oscillation of the body around its center (rad). The upper sequence depicts a movement based only on linear velocity, while the bottom one shows a sequence with angular and linear movement.
man [32] as basic emotions. 10.1.1.
Variables
Independent variables
The selected independent variables and their explanation are described as follows: The selected independent variables and their explanation are described as follows: Angular velocity: is the rotational speed (ω) of the robot in respect of its center. Oscillation angle: is the maximum extension in which the robot will rotate (θ). Linear velocity: is the rate of change of the position of the robot (V ). Direction in respect of participant’s perspective: is the angle generated from the participant’s point of view in respect of the robots trajectory. Orientation of the body in respect of participant’s perspective: is the robot’s angle in respect of robot’s trajectory. The three first variables are shown in the Figure 10.2. Dependent variables
Emotion: is the feeling perceived by the participants from the robots movement. From previous experiences, it was decided to give the participants a list with all the possibilities. The options include four emotions and two mental states that could be misinterpreted from these emotions and the option of ”other”, where participants could write their own interpretation. The four emotions were selected among basic emotions suggested by Ekman [32] and they are: happiness, sadness, fear and anger. The two states of mind are tenderness and excitement, which correspond to low and high arousal states. 96
10.1. Design
Emotion’s intensity: indicate the intensity that the emotion is perceived by the participant. This variable is measured through a ten point scale rate, ranging from 0 to 10. Where 0 means that the corresponding emotion is not perceived by the participant and 10 that the emotion is highly perceived by the participant. 10.1.2.
Independent Variables Values
Due to the independent variables are positive real, which means that they could get any value from 0 to motors’ physical capabilities. It was decided to select concrete values for these variables. Therefore, it was done first a simple test to evaluate when significant changes could be evident to the participant. The chosen values are shown in the Table 10.1. Table 10.1: Possible values for each one of the independent variables. ``` ``` Possibilities ``` First ``` Variable Angular Velocity (rad/sec) 0 Oscillation Angle (rad) 0 Linear Velocity (mm/sec) 0 Direction (rad) 0 Orientation (rad) 0
Second
Third
Fourth
1 0.087 200 π π
2 0.175 500
3 0.349 900
−π 2
To get a better idea about the variables of direction and orientation, the Figure 10.3 shows all possibilities for these two variables. The treatments, meant as desired procedures to compare [79],were generated from the combination of independent variables’ values for a total of 384 combinations. All the treatments that would not add any significant information to the experiment were deleted, such as treatments with θ = 0 and ω 6= 0, which reduced the total amount of treatments to 195. The final treatments could be seen in the tables A.1, A.2, A.3, A.4, A.5, A.6 and A.7. 10.1.3.
Emotion Sequences
It was decided that each participant will be just exposed to twenty over one hundred and ninety five possible treatments, which would be last from 10 to 15 minutes. This was decided because each participant was volunteer to participate and would not perceive any monetary remuneration. The twenty treatments were selected picking a number without replacement from 1 to 195. Every time that all the numbers were already selected, the procedure was repeated until the generation of 50 sequences. The treatments and emotion sequences are shown in Table A.8. 10.1.4.
Setup
The experiment’s setup and dimensions are shown in the Figure 10.4. The crosses symbolize possible starting points, which is selected depending on the direction’s value, such us it is showed in the Figure 10.3. 97
Chapter 10. Experiment
Figure 10.3: Combination of direction and orientation. The crosses symbolize the possible initial points. a) Direction = 0 and Orientation = 0. b) Direction = 0 and Orientation = π. c) Direction = π and Orientation = π. d) Direction = π and Orientation = 0. e) Direction = −π 2 and Orientation = 0. f) and Orientation = π Direction = −π 2 98
10.1. Design
Figure 10.4: Setup for the experiment. The crosses symbolize the possible starting points.
Figure 10.5: Interface used in the experiment. Once a sequence is selected, the interface shows sequence’s values. Also, the interface give information about the current position of the robot and its velocity.
10.1.5.
Tools
The lower part of the robotic platform 2.0 3.5 was decided to be used. Additionally, it was created a graphical interface 10.5 to reduce the possibility on introducing wrong values for a desired sequence. This interface loads the sequences from a .txt file and displays sequences’ numbers on it. Every time that a new sequence should be presented to a participant, the sequence’s number is selected in the interface, which will display sequence’s values. Once the robot has been positioned to the correct position, the execution message could be send to the robot clicking on send button. In case that the sequence’s execution should be aborted, it should be clicked the button stop. 10.1.6.
Procedure
The protocol is described below: 1. The subject is asked to fill out the following information: Sex Career 99
Chapter 10. Experiment
Age Country of origin. As it could be seen, this information maintains participants’ anonymity and makes impossible to trace back each participant. 2. The robot is shown to the subject and the experiment procedure is explained. 3. An example of the questionnaire is given beforehand and an example of the movement that the robot will perform is presented. 4. The subject is exposed to a specific treatment sequence according to the following steps: a) The subject is exposed to the performance with a configuration of values. b) The subject could use as much time as he or she needs to select the corresponding emotion. c) After the subject had completed his/her selection about the current treatment, the sequence is repeated for the rest of treatments.
10.2 Study The information about all the participants were collected using a Google form. The order of the enlisted options changed for each question in the questionnaire to prevent any kind of bias. Figure 10.6 shows an example of the questionnaire. The experiment was performed at Politecnico di Milano, campus Leonardo during the months of June and July of 2015. A total of 49 volunteers were involved: 12 female and 37 male. The average age of the participants was 25.28 with standard deviation of 2.8, with a minimum age of 20 and maximum of 32. The participants’ country of origin and their careers are shown in the Table 10.2 and Table 10.3, respectively. Table 10.2: Participants’ Country of origin. Country Albania Bosnia Brazil Colombia Germany Greece Iran Italy Moldova
Counting 1 1 2 4 1 1 5 33 1
10.3 Results A total of 980 answers were collected, with a minimum of 5 trials for each treatment. A table for each treatment was generated. Table 10.4 is an example of the generated tables. 100
10.3. Results
Figure 10.6: Example of the questionnaire used in the experiment.
For each table were calculated the mean, standard deviation, and median. These are shown in Tables A.9, A.11, A.12, and A.13. It was not possible to use ANOVA test over the data because the assumption of normality is not achieved in the collected data. This was check using the Shapiro-Wilk Test. Additionally, a contingency table for each emotion was generated in each treatment as it is depicted in Table 10.9, where the intensity for the other emotions is calculated as the mean of them, including the option of ”other”. For all tables, including the contingency, were calculated the Krippendorff’s alpha agreement [56] (α), which is a reliability coefficient to measure the agreement among different participants. Unlike other coefficients (Kappa), α is a generalization of several known reliability indices, and it applies to: Any number of observers. Any number of categories. Any type of data. Incomplete or missing data. Large and small sample sizes.
This calculation was done using the R package irr. To improve the table interpretation, it was decided to just record the emotions’ alpha values that had a mean greater than zero. The α for each treatment is shown in Figure 10.7. Also, graphs were generated for each contingency table with their respective mean. Figures 10.8, 10.9, 10.10, 10.11, 10.12, 10.13,a 10.14 correspond to Anger, Excitement,Happiness, Sadness, Fear, Tenderness, and Other, respectively. Unfortunately, these graphs could overwhelm with information and cannot been easily interpreted. Moreover the graphs did not give a clear idea about 101
Chapter 10. Experiment Table 10.3: Participants’ Career. Career Aeronautical Engineering Architecture Social assistance Automation Bio-medical Engineering Computer Science Electronic Engineering Mechanical Engineering Nursery Pedagogical Science Tourism
Counting 1 1 1 1 5 33 2 2 1 1 1
the relationship among the emotions. Therefore, tables with a top ten ranking have been set up. The raking considered: (i) the mean of the respective emotions, (ii) the alpha agreement for the respective emotion, and (iii) the alpha agreement for the treatment. The decision to give more importance to emotion’s alpha rather than intensity average was taken basing on the consideration that most participants agreed on their observation. From the resulting tables 10.5, 10.6, 10.8, and 10.7, it is possible to notice: Fear was the only emotion that had six over ten movements obtaining both general and specific alpha agreement over 0.41, which is the lower bound for moderate agreement [2]. Also, it was mostly selected (eight over ten) when direction = π and orientation = π. The angle and the angular velocity attributed six over ten times were 0.087 and 2, respectively. So it seems that people perceive as fear when the robot is looking at them and moving far from them fast. Sadness was mostly attributed to linear velocities of 0mm/s (two over ten) and 200mm/s (eight over ten), and angular velocities of 0rad/s and 1rad/s, with higher predilection of the second (eight over ten). It seems that people attribute sadness to slow velocities with slow angular velocity and small oscillation angle. Regarding the other two features, there is not a concrete pattern that could lead to make a generalization. Happiness is attributed to different values of the independent variables. But it is the only emotion that is highly perceived when the linear velocity is 0 with four over ten. However the oscillation angle for these four cases is equally divided between the values 0.174 and 3.49. So it seems that happiness is mainly attributed to fast angular velocities and big oscillations angles. However specific agreement among the other features is not present. Anger is highly perceived with an angle of 0.087 (seven over ten) and with linear velocities over 200mm/s. It seems that people attribute anger to fast velocities, both angular and linear, small angle of oscillation and the robot facing the person when it is approaching them.
102
10.3. Results
Sad
Other
29
Iran
0
2
8
0
0
0
0
25
Italy
0
0
6
0
0
8
0
24
Italy
0
0
0
8
0
7
0
26
Italy
0
0
0
0
0
0
7
24
Italy
4
8
0
0
0
0
0
Figure 10.7: Alpha values for each treatment.
103
Explain
Angry
Masculine
Scared
Feminine
Tender
Masculine
Excited
Feminine
Happy
electronic engineering Computer science Computer science Science pedagogiche Computer science
Country of Origin
Career
Masculine
Age
Gender
Table 10.4: Example of the table generated for each treatment. This table corresponds to the treatment with ID = 1.
Dubbioso
Chapter 10. Experiment
Figure 10.8: Anger α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants.
Figure 10.9: Excitement α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants.
Figure 10.10: Happiness α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants.
104
10.3. Results
Figure 10.11: Sadness α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants.
Figure 10.12: Fear α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants.
Figure 10.13: Tenderness α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants.
105
Chapter 10. Experiment
Figure 10.14: Other α and mean values for each treatment. The left axis corresponds to the α agreement, while the right axis corresponds to the mean of the intensity given by the participants.
Table 10.5: Treatment top 10 for Happiness. This top list was generated by ordering from highest to lowest first by Happiness mean, then by Happiness α, and finally by Treatment α. α Treatment
General
Happiness
Excitement
Tenderness
Fear
Anger
Sadness
Other
Happiness
Excitement
Tenderness
Fear
Anger
Sadness
Other
Mean
29 161 38 87 156 45 83 143 22 136
0.38 0.53 0.33 0.21 0.48 0.34 0.31 0.43 0.24 0.28
0.71 0.71 0.22 0.21 0.11 0.71 0.12 0.11 0.11 0.22
0 0 0 0 0.72 0 0 0 0 0
0 0 0.13 0.13 0.12 0 0 0 0 0
0 0 0 0 0 0.13 0.61 0 0 0
0.13 0 0 0.23 0.76 0.47 0 0.13 0 0
0 0 0 0.34 0 0.23 0 0 0 0
0.13 0 0 0 0 0 0.13 0 0.13 0
6.8 6.8 6.6 6.6 6.4 6 6 6 6 5.8
3 3.8 2.6 3.6 6.8 1.4 4.8 0 4 0.8
2.6 2 0.6 1.4 1.2 3.2 4.8 3.4 4.2 3
0 0 0 2 4 1.6 0.4 0 0 0
1.4 0 1.6 0.8 0.2 0.4 0 0.6 2.2 0
0 0 0 0.4 0 0.6 0 0 0 0
1.4 0 0 0 0 0 2 2 1.2 1.4
106
10.3. Results
Table 10.6: Top 10 treatments for Anger. This top list was generated ordering from highest to lowest first byAnger mean, then byAnger α, and lastly by Treatment α. α Treatment
General
Happiness
Excitement
Tenderness
Fear
Anger
Sadness
Other
Happiness
Excitement
Tenderness
Fear
Anger
Sadness
Other
Mean
4 34 186 39 173 144 17 31 30 141
0.19 0.29 0.60 0.44 0.41 0.44 0.13 0.54 0 0.30
0 0.13 0 0 0 0.13 0.13 0 0.13 0.13
0.13 0 0 0 0 0 0 0.16 0 0
0.13 0.76 0 0 0 0 0 0 0.13 0
0.13 0.13 0.13 0 0 0 0 0 0.13 0
0.22 0.21 0.73 0.12 0.11 0.43 0 0.16 0 0.21
0 0 0 0 0.47 0 0 0 0 0
0 0 0 0 0 0 0 0 0.13 0
3.2 1.6 0 3 0 0.8 1 2.4 1.6 1
1.4 3.8 1.6 5 4.8 2.6 0 6.4 3 1.8
1.2 0.2 0 0 2.6 0 0 0 1 0
1.8 1 1.4 2.8 0 1.4 2.4 0 1.6 3.4
6.4 6.4 6.2 6.2 5.8 5.2 5.2 5 5 4.8
0 0 0 0 0.4 0 0 0 0 0
0 0 0 0 0 0 1.8 0 1.8 0
Table 10.7: Treatment top 10 for Fear. This top list was generated ordering from highest to lowest first byFear mean, then byFear α, and lastly by Treatment α.
General
Happiness
Excitement
Tenderness
Fear
Anger
Sadness
Other
Happiness
Excitement
Tenderness
Fear
Anger
Sadness
Other
Mean
Treatment
Alpha
113 109 105 103 102 92 160 76 107 93
0.85 0.73 0.41 0.33 0.68 0.44 0.35 0.53 0.20 0.53
0 0 0.13 0.24 0 0.13 0.35 0 0.13 0
0 0 0 0.24 0 0.13 0.13 0 0.13 0
0.15 0.15 0 0 0 0 0 0 0 0.13
0.83 0.79 0.75 0.34 0.72 0.66 0.22 0.30 0.16 0.71
0 0 0 0.24 0.13 0 0 0 0.34 0.13
0 0.79 0 0 0 0 0 0 0 0.13
0 0 0 0.25 0 0 0 0 0 0
0 0 2 1.33 0 1 0.4 0 1.8 0
0 0 3.4 1.5 0 1 1.2 0 1.6 0
1.2 0.6 0 0 0 1.2 0 0 0 1
9.6 9.6 9 7.83 7.8 7.6 7.4 7 6.8 6.6
0 0 3.6 0.66 0.6 1.6 3 2.20 0.4 0.8
0 0.2 0 0 2.20 0 0 0 2.4 1.2
0 0 0 1.66 0 0 0 0 0 0
107
Chapter 10. Experiment
Table 10.8: Treatment top 10 for Sadness. This top list was generated ordering from highest to lowest first bySadness mean, then bySadness α, and lastly by Treatment α.
General
Happiness
Excitement
Tenderness
Fear
Anger
Sadness
Other
Happiness
Excitement
Tenderness
Fear
Anger
Sadness
Other
Mean
Treatment
Alpha
130 42 13 166 91 49 157 52 88 50
0.64 0.39 0.18 0.10 0 0.27 0.14 0.40 0.32 0.36
0 0 0.13 0.13 0 0.13 0 0 0.69 0
0 0 0 0.29 0 0 0.13 0.13 0.69 0
0 0.13 0 0 0.13 0 0.13 0 0 0
0 0 0 0 0.13 0 0.13 0 0 0
0.13 0 0.13 0 0.14 0 0 0 0 0
0.73 0.22 0.11 0 0 0.21 0.21 0.57 0.16 0.16
0.13 0 0 0 0 0 0.13 0.13 0 0
0 0 1.6 0.8 0 1.2 0 0 0.2 0
0 0 2 0.4 0 0 2 0.8 0.2 0
0 1.8 5.4 1.8 0.6 3.4 2 3 1.2 3.6
0 0 2.6 0 1.4 1.8 1.2 3 3.2 3.6
1.2 0 1.6 0 1.2 0 0 0 0 0
7.4 7.2 6 6 6 5.8 5.8 5.6 5.4 5.4
1.6 3.2 0 3.2 2.4 0 1.6 0.6 0 0
Table 10.9: Example of of the contingency table generated for the treatment ID = 1 and happiness. Participant 1 2 3 4 5
Happiness 0 0 0 0 4
108
Other 1.66 2.33 2.5 1.166 1.33
10.4. Summary
10.4 Summary The experiment presented in this chapter was designed to determine the contribution of angular and linear velocity, oscillation angle, direction and orientation on the perception of happiness, anger, sadness and fear. These emotions correspond to four enlisted by Ekman [31] as basic emotions. To reduce possible contributions of the upper part of the platform, it was decided to just use the platform’s lower part. The Krippendorff’s alpha agreement [56] was used to calculate the consensus for each treatment. Using the alpha and the average intensity was created a top 10 treatments for each one of the emotions.
109
CHAPTER
11
Fourth Case Study
A final case study was done at Researcher’s Night 2014 with two main objectives (i) cross-validate the findings obtained from the experiment, and (ii) use the Emotional Enrichment System to verify whether the participants would prefer scenes when the robot expresses emotions or rather moves without any emotion expression. In this case study was just used the lower part of the version 2.0 base (Section 3.5) as it is shown in Figure 11.1. Moreover during the whole case, it was used the Emotional Enrichment System (Chapter 5) to add emotion to the movements.
11.1 Design and Setup This case study uses the results obtained in the experiment (Chapter 10) and was designed to have two parts (i) emotion expression through changes in linear velocity, angular velocity, and oscillation angle. Figure 7.1 presents an example of these variables. (ii) Presentation of a small scene to verify whether the participants would prefer scenes when the robot shows emotions or not. the Emotional Enrichment System (Chapter 5) was used in both cases (with and without emotions). Two web-cams and eight Alvar tags were added to use Kalman filter to improve robots localization in the stage. The distribution of the web-cams and the tags are depicted in Figure 11.2. 11.1.1.
Emotion Perception
To verify the emotion perception, it was decided to use just two of the top ten treatments identified in the experiment for each emotion. Likewise, the same list of emotions used during the experiment is used in this case study. Therefore, the list has four emotions and two mental states. The four selected emotions are Anger, Happiness, Sadness and Fear, and the mental states are Excitement and Tenderness, which could 111
Chapter 11. Fourth Case Study
Figure 11.1: Lower part of the platform’s version 2.0 used in the last case study.
Figure 11.2: Environment setup for the fourth case study.
112
11.2. Description
be confused with emotions given that excitement is considered to have high arousal and tenderness low arousal. 11.1.2.
Scene
To verify people’s preferences and test the Emotional Enrichment System, a scene was designed, which was shown with and without emotion to the same group of subjects. Information regarding which scene is shown with or without emotion is not given to participants, but it is told that the scene is the same in both cases. To reduce the impact of external factors on the robot’s trajectory, such as friction coefficient, an Extended Kalman Filter (EKF) to improve the localization was implemented. The information needed for EKF comes from the two web-cams localized outside the stage and the on-board web-cam. The two external web-cams localize the robot through the detection of AR tag mounted on the top of the robot. On the other hand, the on-board web-cam gives information about the position of the observed landmarks positioned around the stage. These landmarks are small boxes holding AR tags. The detection of the AR tags is done through the use of the ROS package ar_track_alvar [77]. Further information about the Kalman filter and its implementation could be found in the Annex B.2.
11.2 Description As the case study was designed to have two different parts, this section describes the parameter values for both parts. Additionally, it is described the designed scene. 11.2.1.
Emotion
The parameters selected for each of the four emotions (Anger, Happiness, Sadness and Fear) are shown in Table 11.1. The main considerations to select the two implementations for each emotion were: (i) the linear velocity should be greater than 0. In other words the robot should show some linear displacement. And (ii) it should be in the top 10 list of the emotion, which were obtained in the experiment (Chapter 10). 11.2.2.
Scene
As it was already mentioned, actors should adapt to different circumstances during the performance. Following this approach, the stage was discretized in 9x9 matrix as is shown in Figure 11.3. Thus, the movements of the robot are given in terms of the matrix positions. This allows the adaptation to different stage dimensions because the robot’s final position is calculated by the Emotional Enrichment System during execution taken under consideration the stage dimensions. For instance, during the scene’s preparation in the laboratory the stage was 3 meters per 3 meters, but in the final presentation the stage was 2.5 meters per 2.5 meters. The scene’s description is the following: the robot starts in the middle of the stage to move to the upstage right (Figure 2.1), close to the right wing. Then, the robot moves to upstage right center and rotates by π/2 left (Figure 2.2). Next the robot moves to the right center to then go to the center. When it arrives there, it turns full back and 113
Chapter 11. Fourth Case Study
Angle (rad)
Treatment ID
Sadness
Angular Velocity (rad/s)
Fear
Linear Velocity (mm/s)
Anger
Orientation (rad)
Happiness
Direction (rad)
Emotion
Table 11.1: Parameters’ values selected from the experiment.
0 0 π 0 π π π 0
0 0 0 0 π π 0 π
500 900 500 900 900 500 200 200
3 3 3 1 2 2 1 1
0.349 0.174 0.087 0.087 0.174 0.087 0.349 0.349
29 38 144 31 113 102 130 52
Figure 11.3: Stage discretization used for the small scene. The blue squares correspond to the each zone, while the numbers correspond to the ID given to each zone.
114
11.2. Description
Figure 11.4: Sequence of movements done by the robot. The red arrows show the trajectory done by the robot, while the numbers show the order among the movements. a) The first ten movements b) The last five movements
Figure 11.5: Graphical interface used to communicate with the Emotional Enrichment System. a) It is the interface used to send the actions sequence. b) It is the interface used to send a emotion and its intensity.
move backwards to downstage center with a full front orientation. There, it turns full back to move to center. Finally the robot turns to profile right and it does a step back; then it goes to the upstage center and then upstage right. The sequence of movements programmed to the robot are depicted in Figure 11.4. The relation between emotion and movement is as follow: movements one, two, three, four and five are expressed without any emotion. Movements six, seven, eight, nine and ten show fear. Movement eleven depicts happiness, and the remaining movements depict sadness. The two scenes are executed by the Emotional Enrichment System using the same ”script” and the emotion selection is done manually via graphical interface (Figure 11.5).The ”script” were written in JSON using the language described in the Chapter 4. The final description could be found in the Annex B. 115
Chapter 11. Fourth Case Study
Figure 11.6: Fourth case study setup during the Researchers’ Night 2015.
11.3 Study This case study was done during Researchers’ Night, 2015. During a period of two days, people were asked to participate to this study. Each subject was exposed to two rounds, in each one the robot was performing a different emotion. And they were also exposed twice to a small scene, one with emotion and other without emotions. The emotions showed in each trial and the order of the scenes (with or without emotion) were generated randomly beforehand. The total number of volunteers was 256: 128 males, 126 females, and 2 that chose not to specify their gender. The average age was 27.29 years, with standard deviation of 16.58, minimum age was 4 and maximum 76. The Figure 11.6 shows the setup during the Researchers’ Night, where it is possible to observe the markers used to give information to EKF.
11.4 Results Table 11.2 summarizes the results obtained during the case study. It could be observed that the two implementations of Happiness were confused with Anger and Excitement. In a similar way, the first implementation of Anger was mostly confused with Excitement, which was voted twenty one over forty nine subjects that were exposed to the first implementation of Anger. The second implementation of Anger showed an improvement of perception from 10% to 38%. This implementation was perceived also as Happiness, Fear and Excitement. Both implementations of Fear had a high level of recognition 54 % and 50 % and mostly confused with Excitement, which was voted nine times for the first implementation and twenty times for the second implementation. Finally, the two implementation of Sadness was confused with Fear and Tenderness. To verify these misinterpretations among the implemented emotions, a Fisher’s exact test was applied for ten different combinations. Additionally, a Holm-Bonferroni correction was applied for multiple comparisons to get a better p-value estimation. The results are shown in Table 11.3. As this analysis suggest, the two implementation of 116
11.4. Results Table 11.2: Summary of the answers obtained in the case study.
Angle (rad)
Happiness
Anger
Fear
Sadness
Excitement
Tenderness
Other
Total
Sadness
Angular Velocity (rad/s)
Fear
Linear Velocity (mm/s)
Anger
Orientation (rad)
Happiness
Emotions
Direction (rad)
Presented/Reported
Features
0 0 π 0 π π π 0
0 0 0 0 π π 0 π
500 900 500 900 900 500 200 200
3 3 3 1 2 2 1 1
0.349 0.174 0.087 0.087 0.174 0.087 0.349 0.349
8 11 7 14 6 7 3 5
16 11 5 29 2 3 5 5
7 6 6 13 28 37 17 15
4 2 2 2 1 2 14 28
16 19 21 13 9 20 5 6
4 3 7 3 6 4 16 15
7 1 1 2 0 1 5 7
62 53 49 76 52 74 65 81
Table 11.3: Pair comparison among all the implemented emotions using Fisher’s exact test for both questionnaires with α = 0.05 for the fourth case study. The * indicates that the p-value was adjusted using the Holm-Bonferroni correction for multiple comparisons. Pair Compared Happiness 1 vs Happiness 2 Anger 1 vs Anger 2 Anger 2 vs Happiness 1 Anger 2 vs Happiness 2 Fear 1 vs Fear 2 Sadness 1 vs Sadness 2 Fear 1 vs Sadness 1 Fear 1 vs Sadness 2 Fear 2 vs Sadness 1 Fear 2 vs Sadness 2
p-value 0.38 7.3e-4 0.137 0.157 0.74 0.665 8.35e-5 5e-7 2e-7 1e-7
p-value* 1.0 4.4e-3 0.69 0.69 1.0 1.0 5.8e-4 4e-6 1.8e-6 1e-6
Anger were perceived as two different emotions. Also shows that the two implementation of Happiness were perceived to be similar to the second implementation of Anger. An analysis was done for each emotion, therefore it was created a contingency matrix such as was done in the previous studies. For each of these tables, the positive predictive value, accuracy and a Pearson’s χ2 were computed. The results are shown in table 11.4. They show that there is significant evidence to conclude that second implementation of Anger, both of Fear and Sadness have an impact in the perception of the emotion and they are considered as different implementation respect the rest of implementations. While both implementation of Happiness and first of Anger are considered as similar to the other implementation.
117
Chapter 11. Fourth Case Study Table 11.4: Accuracy, precision and results of Pearson’s χ2 for each contingency matrix with α = 0.05 for the fourth case study. Presented Emotion
Positive Predicted Value 0.13 0.21 0.1 0.38 0.54 0.5 0.22 0.35
Happiness 1 Happiness 2 Anger 1 Anger 2 Fear 1 Fear 2 Sadness 1 Sadness 2
Accuracy
χ2 (1)
p-value
0.79 0.81 0.8 0.81 0.8 0.78 0.85 0.85
0.11 3.7 3.8e-29 34.4 36.2 35.8 27.4 72.9
0.74 0.054 1 4.47e-9 1.8-e9 5.3e-10 1.63e-7 2.2e-16
Table 11.5: Classification accuracy of the presented emotions by the single panels, computed as mentioned in the text, with corresponding 95% confidence interval, no-information rate, and p-value that accuracy is greater than the NIR.
Angle (rad)
Classification Accuracy
95% CI
No-Information Rate
P-Value [Acc > NIR]
Sadness
Angular Velocity (rad/s)
Fear
Linear Velocity (mm/s)
Anger
Orientation (rad)
Happiness
Direction (rad)
Presented Emotion
Features
0 0 π 0 π π π 0
0 0 0 0 π π 0 π
500 900 500 900 900 500 200 200
3 3 3 1 2 2 1 1
0.349 0.174 0.087 0.087 0.174 0.087 0.349 0.349
0.79 0.81 0.8 0.89 0.79 0.78 0.85 0.85
(0.75,0.82) (0.77,0.84) (0.76,0.83) (0.76,0.84) (0.75,0.83) (0.73,0.81) (0.81,0.88) (0.81,0.88)
0.89 0.88 0.88 0.83 0.88 0.83 0.84 0.81
1.0 1.0 1.0 0.95 1 0.99 0.47 0.035
For each of the contingency tables the classification accuracy and the no-information rate (NIR), i.e. the accuracy that had been obtained by random selection, are reported in table 9.4. The results reveal that the only implementation with enough statistical evidence is the second implementation of Sadness. Nevertheless, it is important no notice that the results were obtained using the lower part of the robot without any change in shape. Another important factor to highlight is the impact words enlisted in the questionnaire have on the perception rate. As it was expected in the experiment, Excitement and Tenderness were confused with other emotions with similar arousal level. In this precise case the emotions Anger and Happiness were confused with Excitement, and Sadness and Fear emotions were confused with Tenderness. Despite the bias generated by the two mental states enlisted in the questionnaire, the recognition rate of five out of eight implementations was over 35%, being the two implementation of Fear the implementations with the higher recognition rates (54% for the first and 50% for the second). The results obtained form the small scene are presented in Table 11.6. A chi-squared test with one degree of freedom with an alpha of 0.5 was done to verify if there was 118
11.5. Summary Table 11.6: Answers obtained for the small scene. Gender Male Female
With Emotion 84 81
Without Emotion 43 45
Total 127 126
enough statistical evidence to accept our hypotheses: (i) people prefer scenes with emotions and (ii) gender has no impact on the preference. The results of the tests show that there is enough statistical evidence to accept our first hypothesis and reject the second one, with p-values of 1.42E − 6 and 0.85, respectively. Additionally, the Emotional Enrichment System was used in the two parts of the case used. Although, there was not done any measure of any variable of the system, two things could be said about the system. First it enables the possibility to adapt same script to different stage measures with any impact in the script. Second, that it does not block the execution of an action when an emotion is changed.
11.5 Summary The case study presented in this chapter was done to cross validate the findings in the experiment. For each one of the four emotions studied in the experiment were selected two set of parameters. The results show that both implementations of happiness were confused with anger and excitement, while one implementation of anger was just confused with excitement. Both implementations of sadness were confused with tenderness and fear. Both implementations of fear had a recognition rate over 50%. Additionally to the cross validation, it was done a small scene to check if people have preference to scenes where emotional movements are presented or not. The results show that people prefer scenes with emotional movements.
119
CHAPTER
12
Case Studies Discussion
One pilot, four case studies and an experiment were done to study the use of linear, angular velocity and angle to convey specific emotions. The pilot was done to verify if through the change of angular, linear velocity and some shape’s modification was possible to convey emotions. Considering results of the pilot, a first case study was done. The parameters’ values for the desired emotions were selected empirically. The recognition rate for the implemented emotion were low, thus it was decided to verify a context given would improve it. Therefore a second case study was done using the same emotions’ description and questionnaire, but emotions were presented in a little scene. The results suggest that a context given improves the recognition rate, but also opened the question about the impact that have other participants (actors) and the environment in the recognition rate. After the second case study, it was decided to use values from the literature in the next experiment which usually are coded using Laban’s coding system. This brought the issue related to the interpretation of the coding system, which is described in human terms without precise values. It was decided to make an experiment to determine the contribution of linear and angular velocity and oscillation angle in conveying Anger, Happiness, Sadness and Fear. To analyze the possible misinterpretation with mental states in the questionnaire were enlisted two of them that could be easily confused with the objective emotions. To cross validate the results obtained in the experiment a fourth case study were done. An inexpert person would try to compare the result obtained among all these case studies. However a comparison is not possible. Excluding the first and second case study, the other two cases have different emotions enlisted in the questionnaire as well implemented emotions are different. To understand their differences, table 12.1 presents the emotions implemented and enlisted in case study one, three and four. As it could be seen in the third and fourth case studies, there are words that could bias par121
Chapter 12. Case Studies Discussion
ticipants’ perception. For instance happiness and anger were considered as excitement in the last case study. This misinterpretation should not be a surprise given the fact that there is not a unique definition of emotion [18, 83], and each person would interpret a situation differently, so they will give a different label to the presented movement. Besides this fact, a repetitive pattern could be detected in the first, third and fourth case studies: the misinterpretation between happiness and anger. This could suggest that additional features (e.g. trajectory or shape) should be added to increase the differentiation between these two emotions. For example, Venture and collaborators [100] had found out that in human bodies the recognition rate of anger and fear are increased when the torso and head are downwards. On the other hand, they found that happiness perception is increased when the torso and head are move upwards. This example could bring some insight to possible body changes that could occur in non-human like bodies, but it should tested in this kind of platforms to confirm if the same impact is reached. Table 12.1: Emotions implemented and enlisted in first, third and fourth case study. Case Study Fist Case
Shape Change Yes
Emotions Implemented Anger
Emotions Enlisted in the Questionnaire Anger
Curiosity
Curiosity
Disgust
Disgust
Embarrassment
Embarrassment
Fear
Fear
Happiness
Happiness
Sadness
Sadness
Pride
Pride Neutral
Third Case
Not evident
Anger
Unknown Anger
Happiness
Happiness
Sadness
Sadness
Content
Content Frustrated Bored Astonished
Fourth Case
No+ Part
Lower
Anger
Tired Anger
Happiness
Happiness
Sadness
Sadness
Fear
Fear Excitement Tenderness
The features used in the implementation of Anger, Fear, Happiness and Sadness for the four case studies are summarized in the table 12.2. As it could be seen to convey Anger the robot should be approaching a person with linear velocities around 800 − 122
900mts/s or high angular velocities with small oscillation angles. To convey Fear the robot should get far from the source of stimulus and it should be facing the source with high linear velocity and angular velocity. Happiness was best shown when the velocity was 500mts/s with an angle of 0.08rad and angular velocity of 0.8rad/s or angle of 0.26rad with angular velocity of 3mts/s. And finally Sadness is well represented with linear velocities around 200 − 300mts/s and approaching to the person. Table 12.2: Features used in the implementations of Anger, Fear, Happiness, and Sadness for the four case studies.
Emotion
Anger
Fear
Happiness
Sadness
Features
First
Linear velocity Angular velocity Oscillation angle Orientation Direction Linear velocity Angular velocity Oscillation angle Orientation Direction Linear velocity Angular velocity Oscillation angle Orientation Direction Linear velocity Angular velocity Oscillation angle Orientation Direction
800 0.8 0.1 0 0 500 0 0 0 0 500 0.8 0.08 0 0 100 0.3 0.1 0 0
Case Studies Second Third First Second 800 300 0.8 3.5 0.1 0.08 0 0 0 0 500 500 0 0 0 0 0 π 0 π 500 800 0.8 3 0.08 0.26 0 0 0 0 100 300 0.3 0 0.1 0 0 0 π 0
Fourth First Second 500 900 1 2 0.87 0.87 0 0 π 0 900 500 2 2 0.174 0.087 π π π π 500 900 3 3 0.349 0.174 0 0 0 0 200 200 1 1 0.349 0.349 0 π π 0
After the execution of these four case studies there are some aspects that should be addressed by the HRI community: Although experiments are very important and they could give many insights, there is also need to test robots in environments where people are going to create their judgment at first glance. This means that it is necessary to create a procedure that allow people to create case studies that could be comparable to other works. This process should include information about which emotions enlist in the questionnaire, how to conduct the case to not influence participants, and finally the possible pitfalls that could be incurred in during the data analysis phase. Giving the participants the possibility to write their interpretation of the robot’s movements was not going to be helpful to assess what emotion they thought the robot was conveying during case studies. Although there are plenty of procedures that have been proposed to determine the emotion that a person could be thinking at (e.g., SAM), they could just be used in a controlled environment where the experimenter could spend long periods of time with the subject. This is not the case of exhibitions where people would agree to stop just for a couple of minutes. 123
Chapter 12. Case Studies Discussion
On the other side, exhibitions make it possible to collect a large number of subjects representative of a varied population. Establish common features values that could be used to implement common emotions, which will enable to compare in mode detail works that involve emotions. Although Laban’s effort code could be helpful to share finding in the community, these coding should come with specific ranges that allow their implementation in robots.
124
CHAPTER
13
Conclusions and Further work
The present thesis has studied the contribution of linear and angular velocities and oscillation angle in the perception of emotions using a non-anthropomorphic platform. To achieve this objective four case studies and one experiment were done. The specific feature values for each emotion implemented in all the case studies are reported, which let others to replicate the emotions. The first case study used an empirical approach to select the values for each of the emotions. The emotions implemented in this case study correspond to five out of six emotions suggested by Ekman [31] as basic emotions. These emotions are disgust, happiness, disgust, sadness and anger. The emotion surprise was not implemented because it was considered as an emotion that takes just few second to be elicited. Additional to these five emotions, two secondary emotions were implemented: embarrassment and curiosity. To understand if a context given can improve emotion perception, a second case study was done with the same emotions but adding a little scene were a human actor was playing to contextualize each implemented emotion. The results showed a perception improvement, but the question about how much this was influenced by the actor and the performed scene remains open. With these results, it was then designed a third case study, where one emotion from each circumplex model affect quadrant was implemented. The values for each emotion were selected from the literature. Without any considerable improvement respect the first case study, it was then decided to set up an experiment to understand the contribution of the selected variables in the perception of anger, happiness, sadness and fear. To cross validate the results, a fourth case study was done. This case study used the same questionnaire than the one used in the experiment. The emotions presented in this case study correspond to the best two configurations detected in the experiment for each emotion. The results show that including some mental states could bias participants’ voting and that this could influence the emotion recognition. 125
Chapter 13. Conclusions and Further work
All of these case studies provide a detailed description of platform and the methodology used. Also the specific parameter values used in each emotion were reported. This might allow other researchers to replicate emotions used in this work. Moreover, results obtained from these case studies confirm the findings from previous works done with non-anthropomorphic platforms (e.g. Saerbeck and Christoph [91]) but they also suggest possible values that could be applied to implement emotions. In addition the results suggest that angular velocity and oscillation angle could be used to convey certain emotions, as it was evident through all case studies. Although linear and angular velocities, and oscillation angle could be used to convey emotions, other features should be added to differentiate among them. For instance, if the robot gets far from the subjects this increases the perception of fear w.r.t when it goes in the subject’s direction. This tendency was seen during the experiment, where participants perceived as fear when the robot was getting far from them. Unfortunately, the recognition rates obtained in all these case studies could not be compared among them or with other previous works following a similar approach. The reason is the high impact that emotions enlisted in questionnaire play in the emotion evaluation. For instance, in the last case study participants were biased to select Excitation when high arousal emotions (i.e., Anger and Happiness) were presented. This misinterpretation should not be a surprise given the fact that there is not a unique definition of emotions [18, 83], and each person would interpret a situation according to her/his experience, and may give any label. Besides this fact, a repetitive pattern could be detected in the first, third and fourth case studies: the misinterpretation between happiness and anger. This could suggest that additional features (e.g., trajectory or shape) should be added to increase the differentiation between these two emotions. In parallel to these cases studies, an Emotional Enrichment System has been designed and implemented to automatized emotional expression and let others researcher embed emotion expression in there systems. To give a contribution, respect previous works that have work on emotion expression, the system has been envisioned to allow: (i )Interoperability among different platforms. (ii) Introduction of new parameters and emotions in the system without affecting current implementation. (iii) Interface with diverse action decision systems. These objectives have been fulfilled through an iterative approach, which consisted of design, implementation, and test. A first version was designed between the second and third case study and used during the third case. This version was written in C++ and connected with ROS. After the case study, some possible improvements were identified, which brought as result the definition of an emotional execution tree. This version was then used in the fourth case study, where a small scene was played using this system. Additionally, a software architecture was proposed to let robots be used in theatrical presentations. Although there are works that have created theatrical robots [16, 46, 57, 63, 65, 66, 81, 99, 105], they focus on tele-operated or semi-autonomous robots which cannot interact autonomously with human actors nor exploit theatre’s characteristics to study social aspects in robotics. In other words, they omitted theatre’s most important features, which apply to human robot studies such as timing, expressiveness and adaptation to situations. With this main objective, the architecture has been design to be modular to allow the introduction of new modules or use them in other applications, where they could improve the user experience. The emotional enrichment system was 126
integrated into the system as a module in the architecture. A first version of the emotion selection system was implemented. This system was designed to be modular to enable its integration with more complex systems and its configuration was based on the output from the pattern detection modules. The system was implemented in C++ with interface to ROS to make it possible to use it in other models and robotic platforms. Four patterns (fear, surprise, interest, and relief) were implemented and tested. The results showed that the outputs compete with each other, and the emotion has to be selected in a further step with a logic that suits the specific purpose, which could be as simple as take the emotion pattern with higher intensity, or weight behaviours by the intensity of the corresponding patterns. Although this system is simple in comparison to more complex cognitive systems, it could be integrated with these complex system as reflexive emotion selection. Although the work done in this thesis is still insufficient to let a robot be a theatre actor, it was possible to see the benefits to use theatre as a test bed. Theatre enables researchers to focus on testing models that require complex information to be used. This allowed us to focus our efforts to start creating components that could be used in other contexts and that could be beneficial for social robots, such as the emotional enrichment system. The theatrical system and emotions’ parameters values still require work before a robot could perform autonomously a character in a theatrical play. It is still necessary to finalize all the software architecture’s modules, which includes the construction of modules with high level of complexity, such as social relational model and social world model. As it has been done so far, the missing modules should be implemented in a way that could be extrapolated to other contexts. Moreover, the emotion selection system should be improved to take advantage of the social model to implement emotional cognitive models. Although these modules could be given to the robot via script, the real challenge is to determine ways to update all of them from the sensory input. This would require the joint work with researchers that are already working on detecting these characteristics, or the integration of available systems. Regarding the emotional enrichment system, this should be tested in other platforms and new actions should be added to adopt it in others fields. Finally, the features and values to express emotions should be verified on other platforms, to determine whether the values found in this thesis or other works are independent from platforms’ shape or they should be determined for each platform. Moreover, further studies about how the context could improve the recognition ratio should be done. These studies should take into account the influence that an actor or scene could have to get a better understanding on the real significance of each factor.
127
APPENDIX
A
Experiment’s Additional Information
This appendix reports all the additional information related to the experiment done during June and July of 2015 at Politecnico di Milano, Milan - Italy.
A.1 Treatments Design Tables A.1, A.2, A.3, A.4, A.5, A.6 and A.7 show the values of each treatment used in the experiment. Table A.1: Treatments generated from the combination of the independent variables’ possible values. Part 1. ID
Direction (rad)
Orientation (rad)
1 2 3 4 5
0 0 0 0 0
0 0 0 0 0
Linear Velocity (mm/sec) 0 0 0 0 0
Angular Velocity (rad/sec) 1 1 1 2 2
Angle (rad)
0.087 0.174 0.349 0.087 0.174
A.2 Participants’ Sequence A.3 Results Tables A.9, A.11, A.12, and A.13 show the mean of the intensity for each emotion for all the treatments. 129
Appendix A. Experiment’s Additional Information
Table A.2: Treatments generated from the combination of the independent variables’ possible values. Part 2. ID
Direction (rad)
Orientation (rad)
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 π
Linear Velocity (mm/sec) 0 0 0 0 200 200 200 200 200 200 200 200 200 200 500 500 500 500 500 500 500 500 500 500 900 900 900 900 900 900 900 900 900 900 0
130
Angular Velocity (rad/sec) 2 3 3 3 0 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3 1
Angle (rad)
0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0.087
A.3. Results
Table A.3: Treatments generated from the combination of the independent variables’ possible values. Part 3. ID
Direction (rad)
Orientation (rad)
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π
Linear Velocity (mm/sec) 0 0 0 0 0 0 0 0 200 200 200 200 200 200 200 200 200 200 500 500 500 500 500 500 500 500 500 500 900 900 900 900 900 900 900
131
Angular Velocity (rad/sec) 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2
Angle (rad)
0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349
Appendix A. Experiment’s Additional Information
Table A.4: Treatments generated from the combination of the independent variables’ possible values. Part 4. ID
Direction (rad)
Orientation (rad)
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110
0 0 0 π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π
π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π
Linear Velocity (mm/sec) 900 900 900 0 0 0 0 0 0 0 0 0 200 200 200 200 200 200 200 200 200 200 500 500 500 500 500 500 500 500 500 500 900 900 900
132
Angular Velocity (rad/sec) 3 3 3 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3 0 1 1
Angle (rad)
0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174
A.3. Results
Table A.5: Treatments generated from the combination of the independent variables’ possible values. Part 5. ID
Direction (rad)
Orientation (rad)
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π π
π π π π π π π 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Linear Velocity (mm/sec) 900 900 900 900 900 900 900 0 0 0 0 0 0 0 0 0 200 200 200 200 200 200 200 200 200 200 500 500 500 500 500 500 500 500 500
133
Angular Velocity (rad/sec) 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3
Angle (rad)
0.349 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174
Appendix A. Experiment’s Additional Information
Table A.6: Treatments generated from the combination of the independent variables’ possible values. Part 6. ID
Direction (rad)
Orientation (rad)
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170
π π π π π π π π π π π
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
−π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2
Linear Velocity (mm/sec) 500 900 900 900 900 900 900 900 900 900 900 0 0 0 0 0 0 0 0 0 200 200 200 200 200
134
Angular Velocity (rad/sec) 3 0 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 0 1 1 1 2
Angle (rad)
0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087
A.3. Results
Table A.7: Treatments generated from the combination of the independent variables’ possible values. Part 7. ID
171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195
Direction (rad)
Orientation (rad)
−π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2 −π 2
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Linear Velocity (mm/sec) 200 200 200 200 200 500 500 500 500 500 500 500 500 500 500 900 900 900 900 900 900 900 900 900 900
135
Angular Velocity (rad/sec) 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3 0 1 1 1 2 2 2 3 3 3
Angle (rad)
0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349 0 0.087 0.174 0.349 0.087 0.174 0.349 0.087 0.174 0.349
P ∗ /S ∗ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
1 3 96 53 137 194 126 47 122 135 33 169 133 158 193 139 5 174 130 177 129 103 55 99 89 48 122 149 183 191 10 178 132 30 32 9 93 115 103 186 177 1 22 100 48 61 90 160 161 149
2 139 51 128 69 172 183 138 117 167 190 28 138 11 100 22 107 77 21 67 13 125 74 138 169 45 69 181 65 175 101 189 42 153 147 76 107 92 130 5 195 87 24 36 51 155 67 104 81 138
3 16 41 5 78 45 27 72 13 151 58 153 173 8 184 113 186 110 146 111 82 111 18 73 174 105 182 6 160 167 35 158 116 69 88 157 117 161 97 152 52 66 37 35 132 184 136 77 146 153
4 93 18 179 155 34 25 4 80 108 189 24 15 69 179 120 96 78 74 9 31 154 186 72 15 119 85 3 189 157 46 56 38 128 71 3 180 150 94 123 39 88 175 182 137 124 15 72 109 126
5 57 87 63 67 120 42 159 125 185 142 166 191 93 90 123 170 172 171 94 25 23 19 24 52 129 127 88 147 61 17 165 129 55 29 40 14 122 100 188 169 117 6 89 172 108 97 69 11 166
6 23 76 97 84 7 82 83 71 114 187 181 106 62 20 134 92 86 194 176 27 27 132 168 32 98 104 56 156 180 89 86 1 79 106 20 149 19 155 135 25 181 33 121 105 65 189 130 129 139
7 75 161 99 136 162 17 65 106 140 119 83 165 155 33 58 152 160 71 81 156 107 38 75 143 192 173 12 40 71 119 78 46 151 35 131 81 185 28 98 120 41 193 168 164 142 2 143 159 134
8 113 141 111 175 22 149 79 130 91 147 39 40 18 48 23 1 141 102 61 56 113 178 11 152 54 162 106 146 49 80 10 187 65 87 140 90 170 163 102 27 70 58 79 115 106 45 154 56 173
9 49 163 55 59 103 60 43 90 169 110 29 116 185 145 164 65 115 159 30 54 148 44 64 29 26 2 92 142 171 4 162 26 47 126 17 167 83 177 143 60 75 20 152 187 71 57 21 26 163
10 48 148 29 109 92 146 101 94 153 176 6 188 125 91 76 121 114 178 32 50 42 57 190 8 161 31 188 20 145 181 44 112 49 61 75 96 138 179 50 107 111 194 165 23 123 113 78 12 140
11 195 192 56 61 77 115 73 180 24 171 75 167 57 68 89 84 168 95 157 141 166 68 60 164 94 95 158 195 185 183 8 51 43 60 109 121 172 133 127 17 5 93 82 178 162 4 50 55 191
12 143 9 170 10 11 19 165 20 6 121 64 127 66 137 162 175 37 41 14 13 81 37 7 87 102 150 170 22 76 176 168 91 62 12 111 74 82 34 125 91 180 7 38 84 96 76 186 119 8
13 50 64 21 112 86 38 157 12 133 88 149 51 143 126 132 182 154 147 7 58 140 130 120 121 179 36 131 51 5 195 25 113 6 63 105 22 192 144 154 13 95 122 28 86 192 167 118 14 94
14 134 145 81 1 107 44 89 158 173 31 36 85 55 16 73 109 117 161 70 14 67 159 66 123 109 184 93 115 96 139 64 99 174 52 124 13 7 84 16 103 102 114 74 98 99 31 9 49 83
15 181 150 95 191 70 129 46 35 15 37 128 87 72 98 150 10 108 140 2 4 91 100 43 117 83 114 133 90 177 164 194 142 120 48 67 114 27 104 66 170 30 46 63 42 183 101 133 34 147
16 40 186 105 102 156 68 123 26 188 35 151 80 124 190 79 119 142 148 88 176 163 84 165 136 134 9 155 151 112 191 193 11 166 190 41 110 18 134 59 73 53 3 148 16 19 54 185 176 118
17 166 28 74 32 182 131 52 144 62 183 136 99 163 105 118 101 12 187 195 50 193 41 187 79 86 16 108 153 78 58 145 15 31 108 68 57 137 23 2 171 125 80 127 43 85 188 29 174 103
18 36 8 168 98 14 116 104 127 66 131 180 43 47 135 53 103 34 144 42 39 77 118 172 53 34 139 137 110 1 159 146 95 45 54 182 72 171 184 37 18 156 44 150 157 32 112 158 151 147
19 178 118 30 152 174 160 132 85 100 38 192 17 97 3 104 60 19 26 189 124 59 82 21 80 25 70 97 135 144 156 53 101 24 175 36 169 173 148 39 190 116 10 135 68 47 145 110 131 16
20 193 177 2 54 39 154 164 124 184 122 52 44 49 59 4 112 63 46 45 63 62 47 33 194 126 30 28 128 116 21 85 136 118 160 33 73 77 141 70 64 40 62 179 92 144 141 59 128 71
Appendix A. Experiment’s Additional Information
Table A.8: Sequence of treatments generated for each participant. Where P ∗ is the participant’s number and S ∗ is the trial number.
136
A.3. Results
Table A.9: Mean of the intensity for each emotion for all the treatments. Part I. Treatment 1 2 3 4 5 6 7 8 9 10
Happy 0.8 1.8 0 3.2 2 0.8 2.6 2.4 2.6 0.4
Excited 2 0.6 0 1.4 0.8 0 5 3.6 3.6 0
Tender 2.8 2.4 4.5 1.2 1.6 3 0.8 0.4 3.8 4.2
137
Scared 1.6 2 1.4 1.8 2.8 0.8 2 1.2 0 0.2
Angry 0 1 4.4 6.4 3.4 0 2.8 2.4 0 0
Sad 3 1.2 2.6 0 1.8 5 0.4 0 0 2.4
Other 1.4 1.6 0 0 1.2 1.4 1.6 0 0 2
Appendix A. Experiment’s Additional Information
Table A.10: Mean of the intensity for each emotion for all the treatments. Part II. Treatment 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
Happy 0.6 1.6 1.6 2 0 2.5 1 2.4 1 0 3.2 6 3.6 3 0.8 4.2 3.2 4.2 6.8 1.6 2.4 2.8 3.2 1.6 4.2 0 1.2 6.6 3 2 0.4 0 2.8 3.8 6 0 5.6 4.5 1.2 0 0 0 0 3.8 0
Excited 1.2 1.2 2 7.8 3.8 0.8 0 1 2 1.2 1.6 4 2.4 6.6 3 3.6 5.4 1.6 3 3 6.4 3.2 2.4 3.8 6.4 3 4.5 2.6 5 2 0.8 0 4 2.4 1.4 3.4 6.6 3.8 0 0 0.4 0.8 0.6 3.4 0
Tender 3.2 3.8 5.4 1.4 0.4 0 0 0.4 0 2.8 1.8 4.2 2.4 0.8 1.2 1.2 0 0 2.6 1 0 0.4 0 0.2 0 0 0 0.6 0 1.6 0.4 1.8 2.2 0 3.2 0 2.8 2 3.4 3.6 4.4 3 0 3.6 0.8
138
Scared 1 0 2.6 1.4 1.2 0 2.4 0 0 1.6 0 0 0 1.4 2.6 0 3.4 0.2 0 1.6 0 1.6 0 1 1 0.8 1.8 0 2.8 1.4 4 0 3.8 1.8 1.6 3 0 0.6 1.8 3.6 0 3 6 1.6 2.8
Angry 2.2 0 1.6 4.2 0.4 4.3 5.2 1.2 0.4 1 3.4 2.2 0 3.8 1.8 2.4 1.8 3.8 1.4 5 5 3 4.4 6.4 1.6 4.8 4 1.6 6.2 0 0 0 1.2 0 0.4 0 1.4 3.4 0 0 0 0 1.6 0 0
Sad 3.8 3.4 6 0 0 4.1 0 0.6 1.6 0 0.2 0 3.4 0 0.6 0 0.2 0 0 0 0 0 0 0 0 0.4 0 0 0 3 2.8 7.2 0 1.2 0.6 0 0 2 5.8 5.4 5 5.6 0.4 1 3
Other 0 1 0 0 0 0.8 1.8 0 1.2 1.2 0 1.2 0 0 2 3.4 0 0 1.4 1.8 0 2 0 0 0 0 0 0 0 1.4 0 3.2 0 0 0 1.6 0 0 0 0 0 0.6 0 0 3.2
A.3. Results
Table A.11: Mean of the intensity for each emotion for all the treatments. Part II. Treatment 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
Happy 1.8 1.2 0 0 1 2.2 2.6 1.2 2.6 2.4 1 0 0 0 2.2 1 2.6 1.8 1.6 0 0 1 1.6 2.2 0.2 2.8 4.2 6 5 3.2 5.4 6.6 0.2 0 1 0 1 0 1.2 3.2 1.4 0.2 0 0 0
Excited 4.8 1.6 3.6 0 2 2.4 0.8 4 3.6 1.2 3.8 0.6 0 0.8 1.8 0.8 2.4 2.6 2.2 0.8 0 4 1 0.2 1.6 1.2 4.5 4.8 3.8 5.8 6.8 3.6 0.2 0 0.8 0 1 0 0.6 4.2 4.2 0.4 0 1.4 0
Tender 0 2.2 0.6 0 0 0.8 1.6 1.8 1.2 1.2 0.8 0 0 1.2 0.6 0 1.8 1.2 0 0 0 0.4 1.8 3.2 1.2 6.6 0 4.8 2 1.4 3.8 1.4 1.2 2 3.2 0.6 1.2 1 2.6 0.8 0 0.8 0.8 0 1.2
139
Scared 4.5 3 3.8 5 3.4 4 3.4 0 2 3 3.6 5.2 6.6 1.2 2.6 3.8 1.4 6.2 2.8 3.8 7 3.8 4.5 0.6 0 0 3 0.4 0 1.8 0.8 2 3.2 4.5 3 1.4 7.6 6.6 2.4 2.4 5.6 4.4 5.8 3.4 1.8
Angry 2 0 1.4 0 1.6 1 0 3.2 0 2.4 1.8 1.6 2.2 4.4 3 2.3 2.2 0.8 0.6 3 2.2 2 1.6 0.4 0 0 4.4 0 0 3.2 1 0.8 0 0 0 1.2 1.6 0.8 0 1.2 2.2 0.2 0.2 0 0.6
Sad 0 0.8 1.4 1.4 0.8 0.4 2.2 0.6 1.4 0 0 0 0 0 1.6 0 1 0 0 0.6 0 0 0 2.8 2.4 2.2 0.6 0 1.8 0 0.6 0.4 5.4 4 3.8 6 0 1.2 1.6 0 1.2 1 2.6 0 4.5
Other 1.4 0 0 0 0 2 1.4 0 0 2 0 0 0 1.6 0 1.6 0 0 2 0 0 0 1.8 3.2 3.2 5 0 2 0 0 1.6 0 0 0 0 2.4 0 0 3.6 0 0 0 0 1.4 1.4
Appendix A. Experiment’s Additional Information
Table A.12: Mean of the intensity for each emotion for all the treatments. Part III. 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
1.4 0 1.3 0 2 2.2 1.8 0 0 2.8 0 0 0 0.4 0 0 0.8 1.3 0.8 0.6 0 2.8 1.6 1.6 2.8 2.2 0 3 2.8 0 2.2 0.4 2.6 4.5 2 5.8 0 3.2 0 3.8 1 2.6 6 0.8 0
0 0 1.5 1.4 3.4 2.2 1.6 0 0 0 3.6 3.6 0 1.2 4.5 3.2 1 0 0 0 6.2 3 1.8 6.6 4.8 3.2 1.4 0 1.8 0 0.6 1.2 0 7.8 1.6 0.8 1.8 2.8 1.6 0 1.8 0.8 0 2.6 0
0.4 0 0 1.2 0 0.6 0 0 0.6 2.2 0 0 1.2 0.6 0 0 0.6 2.6 3.8 2.6 0 1.6 0.8 2 0.2 0.4 2.2 3.2 0.6 0 0.6 3 2.6 0 0 3 0 1.6 0 0.4 0 2.2 3.4 0 0
5 7.8 7.8 3.6 9 4 6.8 5.2 9.6 5.2 5.2 5.4 9.6 5.8 5.4 6.2 6.2 3.1 0 2 2.6 0 0.2 1.2 0.6 1.8 2.2 3.2 1.4 0 3.6 2 1.8 2.2 1.6 0 3.6 1.4 0 0 3.4 0 0 1.4 2.8
140
0 0.6 0.6 4 3.6 0.4 0.4 2.6 0 0 2.4 0.4 0 1.2 3.6 0 0 0.3 1.4 1.6 3.4 1.4 0 0.8 2.6 2.6 0 0 2.6 1.2 1.2 1.8 0 0.8 2.6 0 0 1.2 4.2 0 4.8 0 0.6 5.2 1.4
0 2.2 0 0.8 0 0 2.4 1.4 0.2 0.8 0.4 0.2 0 1.6 0 0 1.6 1.3 3.6 3.8 1 1.8 0.4 0 0 1.6 3.6 2.4 5.2 7.4 0 0.6 0.4 0 0 0 0 0 3 1.8 0 1.2 0 0 0
1.4 0 1.6 0 0 0 0 0.8 0 0 1 1.4 0 0 0 0 0 1.6 2 1.2 0 1.2 3.2 0 0 0 1 0 0 1.6 0 0 1.6 1.6 1.2 1.4 1 0 2 3 0 1.4 2 0 0
A.3. Results
Table A.13: Mean of the intensity for each emotion for all the treatments. Part IV. 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195
1.6 2.8 1.8 3.6 3.2 4.4 4.5 0.6 4.5 3.6 6.4 0 1.4 0 0.4 6.8 3.4 1.6 4.8 1.6 0.8 2.8 0 0 2 2.8 1.4 0 5.2 3.8 1.6 4.8 4 2.2 3.2 0.4 3.2 1 4.4 3.8 0 5 2.4 3.2 2.8 2.8 1.8 4.8 4 4
0 2.8 4 5.4 4.4 6 7.6 3.4 4.5 3.2 6.8 2 0 0 1.2 3.8 2.8 5.2 5 0.8 0.4 1.6 0 0 4 2.4 1 4.8 5.6 1.8 1.8 2.2 2.6 2.6 4.8 1.6 2.6 6.2 3.8 2.2 1.6 5.2 5 3.2 3.6 1.8 4.8 7.6 6.2 5.6
0 1 1 0 0 0 0.2 0 0 0 1.2 2 0.6 0.8 0 2 2 0 0 1.8 1.8 2.2 1.2 2.2 0 0 4.5 2.6 0.4 2.6 3.2 0 1.4 2 0 0.4 4.2 0 0 0.8 0 0 0 0.4 1 0 0 0 0.2 0
2 1 1.4 1.8 3.4 3.2 1.4 3.4 2 5 4 1.2 3.2 2.8 7.4 0 0 0 2 0 0 1.4 0 2.4 0.8 0 0 0 0.8 0 1.4 1.8 0 0 0 0 0 1.6 0.4 0.2 1.4 1.4 1 0 2 5 0 1.4 4.2 3.4
141
0 1.3 4.4 1 0 1.2 0 0 1.4 0.2 0.2 0 0 0 3 0 1.2 1.2 0 2.8 0 0 0 1.2 1.4 3.2 0.6 5.8 3.8 1.6 0 1 2 0 3.6 3.4 1 1.6 2.4 0 6.2 2.6 3.2 0 1.8 2.8 3.2 1.6 3.2 1.6
2.8 0.5 0.2 0 0 0 0 1.2 0 0 0 5.8 2 4.5 0 0 0 0 0 1 6 2 3.8 2.4 0.4 1.4 3.6 0.4 1.8 0 0.8 0 2.6 0.6 0 1 1 1 0.4 1.8 0 0 0 0 0 0 0 0 0 0
2.6 2.8 0 0 0 0 0 3.2 0 0 0 1.6 1.4 0 0 0 2 0 1.6 1.4 3.2 0 3.2 1.6 1.6 1 0 0 0 0 1.6 0 0 3.8 0 1.6 2 0 1.2 0 0 0 0 3.4 0 0 1 0 1.2 0
APPENDIX
B
Fourth Case Study Additional Information
This appendix reports all the additional information related to the case study done during the Researchers’ Night, 2015.
B.1 Sequence Design Table B.1 shows the sequence of emotions and scene order (with or without emotion) generated for each group of participants.
B.2 Kalman Filter Implementation A implementation of the Extended Kalman Filter was implemented to improved the localization of the robot, which reduce the variability among the scene representations. B.2.1.
Kalman Filter
Kalman filter (KF) is one of the most studied approach to implement Bayes filter [97]. Therefore, KF could be thought as a set of mathematical equations that provides a recursive approach to estimate the state of a system given some inputs [103]. Given a system that is controlled by a linear equation: xt = Axt−1 + But−1 + wt−1
(B.1)
Where xt is the state of the system in time t; xt−1 is the state of the system in time t − 1; ut−1 is the control variables given in the time t − 1; wt−1 is the noise of the system that is assumed to be white noise and with normal probability distribution described by p(w) N (0, Q), where Q is the system noise covariance matrix; the matrix A relates the state xt−1 with the current state; and B relates the control variables with the current 143
Appendix B. Fourth Case Study Additional Information
Table B.1: List of the emotions presented to each group of participants and the order of the scene. Group ID 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Treatment ID 144 52 113 102 102 113 29 52 113 29 31 102 130 113 102 31 29 38 31 102 52 31 38 102 31 130 29
Treatment ID 29 38 130 31 130 38 31 144 52 144 130 38 52 38 29 144 113 144 130 52 144 29 113 130 102 52 113
144
1st Scene Without emotion Without emotion Without emotion With emotion With emotion With emotion With emotion With emotion Without emotion With emotion Without emotion Without emotion Without emotion With emotion With emotion With emotion Without emotion With emotion Without emotion Without emotion Without emotion With emotion With emotion With emotion Without emotion Without emotion With emotion
2nd Scene With emotion With emotion With emotion Without Without Without Without Without With emotion Without With emotion With emotion With emotion Without Without Without With emotion Without With emotion With emotion With emotion Without Without Without With emotion With emotion Without
B.2. Kalman Filter Implementation
state. The measurement vector is described by a linear equation: zt = Hxt + vt
(B.2)
Where vt is a measurement noise that is assumed to be white noise and with normal probability distribution described by p(v) N (0, R), whereR is the measurement noise covariance matrix; H relates the state to the measurement; and it is updated every time that new measures are obtained. The EK could be describe in two phases: 1. Prediction: xt = Axt−1 + But−1 Σt = AΣt−1 AT + Q
(B.3) (B.4)
Kt = Σt H T (HΣt H T + R)−1 xt = xt + Kt (zt − Hxt ) Σt = (I − Kt H)Σt
(B.5) (B.6) (B.7)
2. Correction
Where xt is the priori estimate of the true state x∗t and xt is the posteriori estimate. Extended Kalmand Filter
The assumptions of linear transitions and measurements are very strong and it real cases most of the time broken. To overcome the linearity assumption, it was created the extended Kalman Filter (EKF). Here the assumption is that the next state is governed by nonlinear function g and measurement by h: xt = g(ut , xt−1 ) zt = h(xt )
(B.8) (B.9)
Where the function g replaces the matrices A and B, and h replaces H. In the EKF the two phases could be seen as: 1. Prediction: xt = g(xt−1 , ut−1 ) Σt = GΣt−1 GT + Q
(B.10) (B.11)
Kt = Σt H T (HΣt H T + R)−1 xt = xt + Kt (zt − h(xt )) Σt = (I − Kt H)Σt
(B.12) (B.13) (B.14)
2. Correction
The matrix G and H are obtained linearized functions g and h , respectively, as follows: 145
Appendix B. Fourth Case Study Additional Information
G is the derivative of g respect to state xt−1 H is the derivative of h with respect to state xt The disadvantage of linearized KF is that the system could diverge [68]. As a consequence, EKF is used in short time missions. Therefore, it has been used new methods of lin B.2.2.
Implementation
The Kalman implementation was done in three different kind of nodes: (i) an ar_track_alvar type, which is instanced one for each camera used in the whole system. This node is in charge to get the raw data given by the camera and detect the position of all the AR tags in the image. (ii) One node merging the information from all the global cameras and bringing it as unique message, this is done with the idea to reduce the computational overhead due the matrix calculations that should be done. (iii) node is in charge to collecting the information from the global information and the local information. The distribution of the software and the communication among them is shown in the Figure B.1 Global
This node has two main responsibilities (i) transform the information coming from the two global cameras to the robot’s frame of reference, and (ii) merge the data from both cameras in just one ROS message. This node reads the file robot_initial_position_position.json, which contains the initial position of the robot in the stage, and the camera configuration files, which are the cameras’ parameters. Filter
This node receives the information from the ar_track_alvar node, which gives the position of each AR tag detected with the local camera, the global node and the robot’s position estimation. As a result, the node communicate to Arduino correction vector. This vector corresponds to the adjustment that should be done to the current position and it could be written as: correctiont = [∆correction_xt , ∆correction_yt , ∆correction_θt ]T This correction vector is calculated every ∆time. While this time pass, all the data is received but just the most recent is saved, discarding the previous one. The process to obtain the correction vector is the same as it is used by Thrun [97] with additional considerations suggested by Lui [68].
B.3 Scene’s script The scene’s script were described in JSON and using the language described in the Chapter 4: { "type" : "serial_context", "emotion_synch" : "yes", 146
B.3. Scene’s script
Figure B.1: General distribution and communication scheme among the diverse software components running on the whole system, including Kalman filter.
"action_synch" : "yes", "is_primary" : "yes", "information" : "", "actions" : [{ "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 72, "pose" : [0.0,0.0,0.0,1.0] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", 147
Appendix B. Fourth Case Study Additional Information
"parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 64, "pose" : [0.0,0.0,0.38268,0.92388] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 57, "pose" : [0.0,0.0,0.38268,0.92388] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 40, "pose" : [0.0,0.0,0.38268,0.92388] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : 148
B.3. Scene’s script
[ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 40, "pose" : [0.0,0.0,1.0,0.0] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 41, "pose" : [0.0,0.0,1.0,0.0] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 41, "pose" : [0.0,0.0,0.0,1.0] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ 149
Appendix B. Fourth Case Study Additional Information
{ "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 44, "pose" : [0.0,0.0,0.0,1.0] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 44, "pose" : [0.0,0.0,1.0,0.0] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 40, "pose" : [0.0,0.0,1.0,0.0] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { 150
B.3. Scene’s script
"type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 31, "pose" : [0.0,0.0,0.70711,-0.70711] } ] }, { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 28, "pose" : [0.0,0.0,0.70711,-0.70711] } ] } , { "type" : "simple_action", "is_primary" : "yes", "name" : "move_body", "parameters" : [ { "type" : "mandatory_parameter", "name" : "parameter_landmark", "landmark_type" : "Place_Landmark", "landmark_id" : 72, "pose" : [0.0,0.0,0.7011,-0.70711] } ] } ] }
151
Bibliography
[1] Arduino mega 2560. http://arduino.cc/en/Main/arduinoBoardMega2560. [2] Understanding Interobserver Agreement: The Kappa Statistic. Family Medicine, 37(5):360–363, May 2005. [3] Expressing Thought: Improving Robot Readability with Animation Principles, Lausanne, CH, 2011. ACM Press. [4] Julian M. Angel F. and Andrea Bonarini. Towards an autonomous theatrical robot. In ACII’13, pages 689– 694, 2013. [5] Arduino. Arduino due. https://www.pololu.com/product/1212. [6] Emilia I. Barakova and Tino Lourens. Expressing and interpreting emotional movements in social games with robots. Personal and Ubiquitous Computing, 14(5):457–467, 2010. [7] Eugenio Barba. The house of the origins and the return. Speech on the occasion of conferring the title of Doctor honoris causa by the University of Warsaw 28.5.2003, 2003. [8] Brett Beauregard. Pid library. http://playground.arduino.cc/Code/PIDLibrary. [9] Aryel Beck, Antoine Hiolle, Alexandre Mazel, and Lola Cañamero. Interpretation of emotional body language displayed by robots. In 3rd ACM Workshop on Affective Interaction in Natural Environments, pages 37–42, 2010. [10] Aryel Beck, Lola Ca namero, and Kim Bard. Towards an affect space for robots to display emotional body language. In IREE RoMan Conference, 2010. [11] James Bennet, Ajay Bhasin, Jamila Grant, and Wen Chung. Pid tuning classical. https://controls. engin.umich.edu/wiki/index.php/PIDTuningClassical, October 2006. [12] G. Bradski. The OpenCV Library. Dr. Dobb’s Journal of Software Tools, 2000. [13] C. Breazeal, A. Brooks, J. Gray, M. Hancher, C. Kidd, J. McBean, D. Stiehl, and J. Strickon. Interactive robot theatre. In Proceedings of IEEE/RSJ Int. Conf. on Intelligent Robots and Systems(IROS 2003), volume 4, pages 3648–3655, 2003. [14] Cynthia Breazeal. Designing Sociable Robots. MIT Press, Cambridge, MA, USA, 2002. [15] LaVonda. Brown and AAyanna M. Howard. Gestural behavioral implementation on a humanoid robotic platform for effective social interaction. In Robot and Human Interactive Communication, 2014 RO-MAN: The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pages 471–476, Aug 2014. [16] Allison Bruce, Jonathan Knight, and Illah R. Nourbakhsh. Robot improv: Using drama to create believable agents. In In AAAI Workshop Technical Report WS-99-15 of the 8th Mobile Robot Competition and Exhibition. AAAI Press, Menlo, pages 27–33, 2000. [17] Lola Cañamero and J. Fredslund. I show you how I like you-can you read it in my face? IEEE Transactions on Systems, Man and Cybernetics, Part A, 31(5), 2001. [18] J.T. Cacioppo, L.G. Tassinary, and G.G. Berntson. Handbook of Psychophysiology. University Press, 2000.
153
Bibliography [19] A. Camurri, I. Lagerlöf, and G. Volpe. Recognizing emotion from dance movement: comparison of spectator recognition and automated techniques. 59(1-2):213–225, 2003. [20] Lola Cañamero. Emotion understanding from the perspective of autonomous robots research. Neural Networks, 18(4):445 – 455, 2005. Emotion and Brain. [21] J. Cavanaugh. Acting Means Doing !!: Here Are All the Techniques You Need, Carrying You Confidently from Auditions Through Rehearsals - Blocking, Characterization - Into Performances, All the Way to Curtain Calls. CreateSpace, 2012. [22] Mark Coulson. Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior, 28(2):117–139, 2004. [23] ElizabethA. Crane and M.Melissa Gross. Effort-shape characteristics of emotion-related body movement. Journal of Nonverbal Behavior, 37(2):91–105, 2013. [24] N. Dael, M. Mortillaro, and K. R. Scherer. Emotion expression in body action and posture. Emotion, 12:1085– 1101, 2012. [25] Beatrice de Gelder. Why bodies? twelve reasons for including bodily expressions in affective neuroscience. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 364(1535):3475–3484, 2009. [26] Marco de Meijer. The contribution of general features of body movement to the attribution of emotions. Journal of Nonverbal Behavior, 13(4):247–268, 1989. [27] Matthieu Destephe, Takayaki Maruyama, Massimiliano Zecca, Kenji Hashimoto, and Atsuo Takanishi. Improving the human-robot interaction through emotive movements: a special case: walking. In HRI, pages 115–116, 2013. [28] Matthieu Destephe, Massimiliano Zecca, Kenji Hashimoto, and Atsuo Takanishi. Perception of emotion and emotional intensity in humanoid robots gait. In Robotics and Biomimetics (ROBIO), 2013 IEEE International Conference on, pages 1276–1281, 2013. [29] Joao Dias, Samuel Mascarenhas, and Ana Paiva. Fatima modular: Towards an agent architecture with a generic appraisal framework. In International Workshop on Standards for Emotion Modeling, 2011. [30] Artopia Drama. Learning about the stage. [31] Paul Ekman. Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage (Revised and Updated Edition). W. W. Norton & Company, 2 rev sub edition, September 2001. [32] Paul Ekman. Emotions Revealed : Recognizing Faces and Feelings to Improve Communication and Emotional Life. Owl Books, March 2004. [33] Stephanie Embgen, Matthias Luber, Christian Becker-Asano, Marco Ragni, Vanessa Evers, and Kai O. Arras. Robot-specific social cues in emotional body language. In Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN’12), pages 1019–1025, USA, September 2012. IEEE Computer Society. [34] Natascha Esau, Lisa Kleinjohann, and Bernd Kleinjohann. Emotional communication with the robot head MEXI. In Ninth International Conference on Control, Automation, Robotics and Vision, ICARCV 2006, Singapore, 5-8 December 2006, Proceedings, pages 1–7, 2006. [35] Terrence Fong, Illah Nourbakhsh, Kerstin Dautenhahn, Terrence Fong A, Illah Nourbakhsh A, and Kerstin Dautenhahn C. A survey of socially interactive robots: Concepts, design, and applications, 2002. [36] Max Kuhn. Contributions from Jed Wing, Steve Weston, Andre Williams, Chris Keefer, and Allan Engelhardt. caret: Classification and Regression Training, 2012. R package version 5.15-044. [37] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns: Elements of Reusable Object-oriented Software. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1995. [38] S. Genevieve. Delsarte System of Dramatic Expression. BiblioBazaar, 2009. [39] Álvaro Castro González, Maria Malfaz, and Miguel Angel Salichs. Learning the selection of actions for an autonomous social robot by reinforcement learning based on motivations. I. J. Social Robotics, 3(4):427–441, 2011. [40] Jonathan Gratch and Stacy Marsella. A domain-independent framework for modeling emotion. Journal of Cognitive Systems Research, 5:269–306, 2004. [41] M. Melissa Gross, Elizabeth A. Crane, and Barbara L. Fredrickson. Methodology for assessing bodily expression of emotion. Journal of Nonverbal Behavior, 34(4):223–248, 2010.
154
Bibliography [42] Markus Häring, Nikolaus Bee, and Elisabeth André. Creation and evaluation of emotion expression with body movement, sound and eye color for humanoid robots. In IEEE International Symposium on Robot and Human Interactive Communication (Ro-Man), 2011. [43] Marcel Heerink, Krose Ben, Vanessa Evers, and Bob Wielinga. The influence of social presence on acceptance of a companion robot by older people. Journal of Physical Agents, 2(2), 2008. [44] Liang Hiah, Luuk Beursgens, Roy Haex, Lilia Perez Romero, Yu-Fang Teh, Martijn ten Bhömer, Roos van Berkel, and Emilia I. Barakova. Abstract robots with an attitude: Applying interpersonal relation models to human-robot interaction. In IEEE International Symposium on Robot and Human Interactive Communication, IEEE RO-MAN 2013, Gyeongju, Korea (South), August 26-29, 2013, pages 37–44, 2013. [45] Guy Hoffman. On stage: Robots as performers. RSS Workshop on Human-Robot Interaction: Perspectives and Contributions to Robotics from the Human Sciences, 2009. [46] Guy Hoffman, Rony Kubat, and Cynthia Breazeal. A hybrid control system for puppeteering a live robotic stage actor. In Martin Buss and Kolja Kühnlenz, editors, RO-MAN 2008, pages 354–359. IEEE, 2008. [47] Carroll Izard. Four systems for emotion activation: cognitive and noncognitive processes. Psychological review, pages 68–90, 1993. [48] Vernon Kelly. A primer of affect psychology, 2009. [49] Musical Theatre Kids. Stage directions for actors. [50] A. Kleinsmith and N. Bianchi-Berthouze. Affective body expression perception and recognition: A survey. Affective Computing, IEEE Transactions on, 4(1):15–33, 2013. [51] Andrea Kleinsmith and Nadia Bianchi-Berthouze. Affective body expression perception and recognition: A survey. IEEE Transactions on Affective Computing, 4(1):15–33, 2013. [52] Mark Knapp. La Comunicacion No Verbal: El Cuerpo y El Entorno. Ediciones Paidos, Barcelona - Buenos Aires, 1982. [53] H. Knight and R. Simmons. Expressive motion with x, y and theta: Laban effort features for mobile robots. In Robot and Human Interactive Communication, 2014 RO-MAN: The 23rd IEEE International Symposium on, pages 267–273, Aug 2014. [54] Heather Knight. Heather knight: la comedia de silicio. TED Ideas Worth Spreading, December 2010. [55] Heather Knight, Scott Satkin, Varun Ramakrishna, and Santosh Divvala. A savvy robot standup comic: Online learning through audience tracking. TEI 2011, January 2011. [56] Klaus Krippendorff. Computing Krippendorff’s Alpha Reliability. Technical report, University of Pennsylvania, Annenberg School for Communication, June 2007. [57] LAAS-CNRS. Roboscopie, the robot takes the stage! roboscopie.
http://www.openrobots.org/wiki/
[58] Rudolf Laban and Lisa. Ullmann. Modern educational dance. Praeger New York, 2d ed., rev. by lisa ullmann. edition, 1968. [59] Gabriella Lakatos, Marta Gacsi, Veronika Konok, Ildiko Bruder, Boroka Berczky, Peter Korondi, and Adam Miklosi. Emotion attribution to a non-humanoid robot in different social situations. PLos ONE, December 2014. [60] Peter J. Lang, Margaret M. Bradley, and B. N. Cuthbert. International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8, The Center for Research in Psychophysiology, University of Florida, Gainesville, FL, 2008. [61] Timothy Leary. Interpersonal Diagnosis of Personality: Functional Theory and Methodology for Personality Evaluation. Ronald Press Company: New York., 1957. [62] Christopher P. Lee-Johnson and Dale Anthony Carnegie. Emotion-based parameter modulation for a hierarchical mobile robot planning and control architecture. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, October 29 - November 2, 2007, Sheraton Hotel and Marina, San Diego, California, USA, pages 2839–2844, 2007. [63] Séverin Lemaignan, Mamoun Gharbi, Jim Mainprice, Matthieu Herrb, and Rachid Alami. Roboscopie: a theatre performance for a human and a robot. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, HRI ’12, pages 427–428, New York, NY, USA, 2012. ACM. [64] Jamy Li and Mark H. Chignell. Communication of emotion in social robots through simple head and arm movements. I. J. Social Robotics, 3(2):125–142, 2011.
155
Bibliography [65] Chyi-Yeu Lin, Li-Chieh Cheng, Chun-Chia Huang, Li-Wen Chuang, Wei-Chung Teng, Chung-Hsien Kuo, Hung-Yan Gu, Kuo-Liang Chung, and Chin-Shyurng Fahn. Versatile humanoid robots for theatrical performances. International Journal of Advanced Robotic Systems, 2013. [66] Chyi-Yeu Lin, Chang-Kuo Tseng, Wei-Chung Teng, Wei chen Lee, Chung-Hsien Kuo, Hung-Yan Gu, KuoLiang Chung, and Chin-Shyurng Fahn. The realization of robot theater: Humanoid robots and theatric performance. In International Conference on Advanced Robotics, 2009. ICAR 2009., 2009. [67] David V. Lu and William D. Smart. Human-robot interactions as theatre. In RO-MAN 2011, pages 473–478. IEEE, 2011. [68] Yong Lui. Robot Vision, chapter Navigation and Control of Mobile Robot Using Sensor Fusion. InTech, 2010. [69] M. Malfaz and M. A. Salichs. A new approach to modeling emotions and their use on a decision-making system for artificial agents. IEEE Transactions on Affective Computing, 3(1):56–68, 2012. [70] Stacy Marsella, Jonathan Gratch, and Paolo Petta. A Blueprint for Affective Computing, chapter Computational Models of Emotion, pages 21–46. Oxford University Press, 2010. [71] Stacy C. Marsella, Jonathan Gratch, and Paola Petta. Computational Models of Emotion. In K. R. Scherer, T. Banziger, and Roesch, editors, A blueprint for an affectively competent agent: Cross-fertilization between Emotion Psychology, Affective Neuroscience, and Affective Computing. Oxford University Press, Oxford, 2010. [72] Diego Martinoia, Daniele Calandriello, and Andrea Bonarini. Physically interactive robogames: Definition and design guidelines. Robotics and Autonomous Systems, 61(8):739–748, 2013. [73] Nikolaos Mavridis and David Hanson. The ibnsina center: An augmented reality theater with intelligent robotic and virtual characters. In RO-MAN 2009, pages 681–686. IEEE, 2009. [74] Mitsubishi. wakamaru. http://www.mhi.co.jp/en/products/detail/wakamaru.html. [75] Robin Murphy, Dylan Shell, Amy Guerin, Brittany Duncan, Benjamin Fine, Kevin Pratt, and Takis Zourntos. A midsummer night’s dream (with flying robots). Autonomous Robots, 30(2):143–156, 2011. [76] Tek-Jin NAM, Jong-Hoon LEE, Sunyoung PARK, and Hyeon-Jeong SUK. Understanding the relation between emotion and physical movements. International Journal of Affective Engineering, 13(3):217–226, 2014. [77] Scott Niekum. ar_track_alvar. http://wiki.ros.org/ar_track_alvar. [78] Jekaterina Novikova and Leon Watts. Towards artificial emotions to assist social coordination in hri. International Journal of Social Robotics, 7(1):77–88, 2015. [79] G.W. Oehlert. A First Course in Design and Analysis of Experiments. W. H. Freeman, 2000. [80] Andrew Ortony, Gerald L. Clore, and Allan Collins. The cognitive structure of emotions. Cambridge Univ. Press, Cambridge, New York, 1994. Autre tirage : 1999. [81] Zaven Paré. Robot drama research: from identification to synchronization. In Proceedings of the 4th international conference on Social Robotics, ICSR’12, pages 308–316, Berlin, Heidelberg, 2012. Springer-Verlag. [82] Claudio Pinhanez. Computer theater. In Proc. of the Eighth International Symposium on Electronic Arts (ISEA’97), 1997. [83] Robert Plutchik. The Nature of Emotions. American Scientist, 89(4):344+, 2001. [84] Pololu. 30:1 metal gearmotor 37dx52l mm with 64 cpr encoder. product/1443.
https://www.pololu.com/
[85] Pololu. Mc33926 motor driver carrier. https://www.pololu.com/product/1212. [86] Felicia Pratto, Jim Sidanius, Lisa M Stallworth, and Bertram F Malle. Social dominance orientation: A personality variable predicting social and political attitudes. Journal of personality and social psychology, 67(4):741, 1994. [87] Morgan Quigley, Ken Conley, Brian P. Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y. Ng. Ros: an open-source robot operating system. In ICRA Workshop on Open Source Software, 2009. [88] Aldebaran Robotics. Nao. http://www.aldebaran-robotics.com/en/. [89] Claire L. Roether, Lars Omlor, Andrea Christensen, and Martin A. Giese. Critical features for the perception of emotion from gait. Journal of Vision, 9(6):15, 2009.
156
Bibliography [90] J. A. Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39:1161–1178, 1980. [91] Martin Saerbeck and Christoph Bartneck. Perception of affect elicited by robot motion. In Proceedings of the 5th ACM/IEEE International Conference on Human-robot Interaction, HRI ’10, pages 53–60, Piscataway, NJ, USA, 2010. IEEE Press. [92] Martin Saerbeck and Albert J. N. van Breemen. Design guidelines and tools for creating believable motion for personal robots. In RO-MAN, pages 386–391, 2007. [93] K. R. Scherer. Appraisal considered as a process of multi-level sequential checking., pages 92–120. New York and Oxford: Oxford University Press, 2001. [94] K. R. Scherer. A Blueprint for Affective Computing, chapter Theoretical approaches to teh study of emotion in humans and machines, pages 21–46. Oxford University Press, 2010. [95] Megha Sharma, Dale Hildebrandt, Gem Newman, James E. Young, and Rasit Eskicioglu. Communicating affect via flight path: Exploring use of the laban effort system for designing affective locomotion paths. In Proceedings of the 8th ACM/IEEE International Conference on Human-robot Interaction, HRI ’13, pages 293–300, Piscataway, NJ, USA, 2013. IEEE Press. [96] Constantin Stanislavski. An Actor prepares. Routledge, New York, NY, 1989. [97] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, 2005. [98] Silvan S Tomkins. Affect theory. Approaches to emotion, 163:195, 1984. [99] Ida Torres. Robots share the stage with human actors in osaka university’s ’robot theater project’. Japandaily Press, February 2013. [100] Gentiane Venture, Hideki Kadone, Tianxiang Zhang, Julie Grèzes, Alain Berthoz, and Halim Hicheur. Recognizing emotions conveyed by human gait. International Journal of Social Robotics, 6(4):621–632, 2014. [101] Harald G. Wallbott. Bodily expression of emotion. European Journal of Social Psychology, 28(6):879–896, 1998. [102] D. Watson, L. A. Clark, and A. Tellegen. Development and validation of brief measures of positive and negative affect: the panas scales. Journal of Personality and Social Psychology, 54:1063–1070, 1988. [103] Greg Welch and Gary Bishop. An introduction to the kalman filter. Technical report, Chapel Hill, NC, USA, 1995. [104] E. Wilson and A. Goldfarb. Theatre: The Lively Art. McGraw-Hill Education, 2009. [105] Karl R. Wurst. I comici roboti: Performing the lazzo of the statue from the commedia dell’arte. In AAAI Mobile Robot Competition, pages 124–128, 2002.
157
