PUTTING ON A BRAVE FACE WITH INSTANT MESSAGING
Marc Fabri Leeds Met University Innovation North Leeds LS6 3QS United Kingdom
[email protected]
David Moore Leeds Met University Innovation North Leeds LS6 3QS United Kingdom
[email protected]
ABSTRACT Instant Messaging (IM) tools such as Microsoft® MSN Messenger or Yahoo!® Messenger enable people to communicate in real-time, via text-chat, over a distance. Typically emotional icons, or emoticons, can be embedded into messages as intentional communicative signals. Recent IM versions also allow display of a picture representing oneself, and in turn one may see a picture of the interlocutor. This paper applies the Instant Messaging paradigm to communication in collaborative virtual environments (CVE). The IM interface is embedded into virtual space and three-dimensional, animated avatars represent the interlocutors. In addition to simply seeing each other, users can control their avatar's facial expressions via emoticons. We outline the design considerations for such a messaging tool and investigate whether and how the introduction of emotionally expressive characters enriches the individual's experience. We define what we mean with richness of experience and present a methodological framework for evaluation. Keywords Instant Messaging, Emotion, Facial Expression, Virtual Reality, Evaluation, Presence
1. INTRODUCTION Instant Messaging is a relatively new technology for realtime, computer-mediated communication. It increasingly permeates both the private and professional life of the people who use it [18]. IM allows the quick exchange of
David Hobbs University of Bradford School of Informatics Bradford BD7 1DP United Kingdom
[email protected]
questions and facts, it is perceived as being less intrusive than a phone call and likely to produce a faster reply than email [18]. It is unique in that it allows the creation of a persistent connection over many hours with sometimes only few sporadic exchanges, thereby creating a virtual common space in which communications can be advanced at any time [9]. Today's IM tools allow not only the exchange of text chat, but also support embedding cartoon-style emoticons into messages. This is an advancement of text-based emoticons traditionally used in Multi User Dungeons of the 1990s, for example :-) to express happiness, or :-( to express sadness. Further, more recent IM versions also allow display of a picture representing oneself, and in turn one may see a picture of the interlocutor.
2. EXPERIMENTAL RATIONALE We now apply the Instant Messaging paradigm to communication in Collaborative Virtual Environments (CVEs). CVEs are multi-user, real-time, networked virtual reality systems where inhabitants are virtually in the same space and aware of each other. Inhabitants are typically represented by three-dimensional humanoid avatars. In our setup, these avatars are animated and their facial expressions can be controlled via emoticon buttons. When communicating with each other the avatar, and in particular the avatar's face, becomes an important non-verbal interaction device. In our experiment, then, we investigate whether and how the introduction of facial expressions affects the participants' experience. Two conditions were created, one with emotionally expressive avatars, and the other with non-expressive avatars. Participants are exposed to the conditions in a between groups design. They are given a task to complete collaboratively, the characteristics of which are described in detail below. Participants can express the following universal emotions by clicking the relevant emoticon on the interface: happiness, sadness, surprise, anger, fear, and disgust [5]. Previous work has shown that these can effectively be
visualised using a relatively simple animated 3D head model [6]. Indeed, whilst muscle activity in the human face is usually coded using 58 Action Units [5], we were able to reduce this to 12 Action Units for the virtual head model whilst maintaining high distinctiveness and recognisability of facial expressions. This paper describes work in progress and focuses on design considerations for the virtual instant messaging tool, discussing how traditional usability evaluation has influenced the process. A framework for evaluating the participants' experience is developed, and in this context the concept of richness is introduced and defined. Further, indicative results obtained from a series of pilot studies are presented. At the time of the conference we will report complete results.
3. THE SCENARIO An interaction scenario had to be created that allows studying the behaviour and subjective experience of participants. It needed to be sufficiently complex to warrant a stimulating discussion taking place, as well as being focussed whilst providing potential for diversion. Previous, related studies (cf. [12][1]) have successfully used negotiation scenarios in which the advantages and disadvantages of different options had to be weighed up and a joint solution be negotiated. We chose a classical survival exercise as this was thought to provide enough complexity with regards to the verbal dialogue and the emotional context alike. Several scenarios are available and a typical one involves two people who are stranded in a remote and hostile area, for example after a plane crash or after having broken down with their car. They salvage a number of items from the wreckage before it becomes inaccessible. They now have to rank these items in order of importance for their survival, first individually and then together. A consensus should be reached for the final list, which will eventually decide the virtual fate of the two participants. Survival exercises are often used to provide a structured experience in individual and group decision-making [16]. The scenario may naturally elicit emotional responses during the debate, and the virtual IM tool enables participants to display such emotions. Emotions can then be used to emphasise a statement or help to pursue a set goal, making them intentional communicative acts.
4. DESIGNING AND REFINING The setup for a virtual Instant Messaging tool is effectively a face-to-face conversation conducted in virtual space, with verbal (text-chat) and non-verbal (facial expressions) means of communication. To maximise positive transfer of skills, the chat components were designed very similar to their two-dimensional counterpart. Interlocutors are represented by three-dimensional animated avatar. Any message that is "said" by an avatar is displayed in the chat log and also in a speech bubble above the avatar's head.
To support the chosen scenario, the virtual IM tool had to display the items salvaged from the wreckage. Inspection, selection and ranking of items happens intuitively and by direct manipulation, within the virtual space - a familiar interface metaphor commonly used in GUI applications. 4.1 Prototype evaluation Prototypes of the virtual IM tool were subjected to a heuristics evaluation [11] as well as controlled interface trials with test users. The main issues that arose were related to the way items could be selected and re-ranked. Since it was felt that both participants should have access to the same list but not simultaneously, a reliable locking mechanism needed to be in place. This ensured that whenever one participant picked an item up and moved it around, the other participants could see it hovering and changing position, but could not manipulate it. In an early study on attention and action focus in CVEs, Benford et al [2] identified this as being important for creating a consistent focus of interaction, engendering mutual awareness and avoiding conflicts. To further emphasise the fact that a participant currently manipulates an item, their avatar’s gaze, and to some extent the entire head, follows any item moved about. Figure 1 shows the final interface version:
Figure 1: Interface of the virtual Instant Messenger tool
The items to be ranked are positioned on the left, with the standard chat interface in the centre of the screen. In addition to seeing the interlocutor's avatar representation, participants can observe a small mirror image of themselves. This enhancement was made following feedback from the pilot studies. Users can choose from a selection of six avatars (3 male, 3 female).
5. HOW WAS IT FOR YOU? From the real world, we know that others’ emotions influence us in our decisions and our own emotional state [13]. They are an important factor in problem solving, cognition and intelligence in general [3]. Emotions can motivate and encourage, they can help us achieve things.
Introducing emotional expressiveness into real-time CVE communication is likely to affect the individuals' experience, and we believe it results in a richer experience. By richness we mean the quality of the experience, manifesting itself through five observable characteristics: 1. 2. 3. 4. 5.
The degree of involvement in a given task Enjoyment of the experience The sense of presence during the task Agreement and harmony with the interlocutor Task performance
No comparable study combining all five characteristics is yet available. However, various researchers have looked at these and related characteristics of richness in isolation, and their interpretations informed our definitions and the choice of evaluation tools. 5.1 Involvement Involvement in the task is an objective measure of the number of user-initiated actions taking place. These are automatically recorded in an interaction log, consisting of communicative acts – text and emotion display – as well as manipulation of objects in the environment. 5.2 Enjoyment Enjoyment is a concept attracting increasing attention in the VR community (cf. [1],[15]). Measuring instruments are typically adapted from related disciplines, such as product design. Designers of consumer products have long been aware of the potential that quantifying enjoyment and pleasurability of use can yield for a product's success. Jordan [7] provides a pleasurability questionnaire, while Desmet and Hekkert [4] developed an interactive selfreport tool that lets people select what emotions they feel when considering a given product. The psychological basis of these instruments makes them potentially useful in many other, related domains, especially where the focus is on subjective attitudes to a system or product. In our study, we used a 12-item mood adjective checklist by Nichols [10]. Mood adjective checklists are typically employed in psychological studies as a self-report measure, for example to evaluate experienced contentment vs. distress. Nichols' list was created specifically for interaction in virtual reality systems. Participants indicate how much they felt certain moods, represented by adjectives such as exhilarated, safe, motivated or happy. 5.3 Presence VR Researchers have long considered presence a crucial element of the virtual world experience. However, measuring presence is not a trivial task. The nature of presence, what elicits it, and how it can be identified and quantified is disputed, sometimes heavily [14]. There are several instruments available to measure presence: The Witmer and Singer Presence Questionnaire [19] takes a
predominantly technological approach, revealing the three clusters involvement/control, naturalness and interface quality. The Slater-Usoh-Steed questionnaire [14], on the other hand, takes a more psychological approach, focusing on internal states and introspection as a means of detecting presence. More recently, Lessiter et al [8] offered an alternative instrument. The 43-item ITC-Sense of Presence Inventory (ITC-SOPI) is a carefully piloted and psychometrically sound questionnaire. ITC-SOPI identifies four distinct factors of presence: engagement, physical space, naturalness, negative effects. Unlike other questionnaires, it is not limited to a single type of virtual reality medium but valid across various media, including immersive VR, desktop VR, first-person 3G games and large projection screen cinema. This cross-media validity makes ITC-SOPI the appropriate tool for the current investigation. 5.4 Agreement Agreement between the participants is measured by analysing the interaction logs. These reveal how many times participants changed their mind on where particular items should be ranked, and show differences between original individual rankings with the final joint ranking. 5.5 Performance Overall performance is measured by comparing the joint ranking list with an ideal ranking list, which is available. The results are treated with care though as the correctness of the ranking list cannot automatically be considered the result of a good (or maybe even a poor) debate - it may simply indicate that a participant had relevant prior knowledge. For that reason, performance results will be considered only anecdotal evidence.
6. PILOT STUDY A pilot was performed to validate experimental design and procedure. Further, participants could comment on any other usability issue they felt could be improved. Six participants took part in the pilot in three sessions. Two sessions featured animated emotionally expressive avatars, while one session featured non-expressive avatars. Three participants commented that the speech bubble above the avatar’s head was disappearing too quickly, and that a history window may help setting utterances into context. Instant messaging tools such as Microsoft® MSN Messenger or Yahoo!® Messenger were given as examples that had an effective history feature. Three participants felt that it would be useful to get visual feedback on emotions expressed via one’s own avatar. First person computer games were given as examples, where such a mirror view is a common feature. Both the history window and a mirror view of oneself were added to the final interface. Most participants commented that the questionnaire was long, although they felt the questions were relevant.
7. DISCUSSION One important observation during the pilot study is that the extensive use of questionnaires, in particular long questionnaires with many items, is problematic. While we believe that the investigated characteristics of richness are the correct ones, and that the methods employed to measure these characteristics are valid, we have doubts that these methods employed are the only ones, or in fact the best ones. The results of this study and any future study have to be looked at individually and the use of evaluation tools considered carefully. There have as yet been few reported contributions to the structured design and evaluation of virtual environments. Not surprisingly, many of the issues and challenges reported here can also be found in Tromp et al [17], a reflection on the framework used for designing and evaluating the COVEN project. A notable indicative result from the pilot study is that the use of emotions often occurs in waves. After an initial period of text-chat only, participants seem to discover the potential of using facial expressions to complement or even fully replace their verbal statements. This then abates, only to re-occur when triggered by the use of an emotion in what may be a key moment of the conversation. This happened several times during the pilots and more data as well as comparison with general studies on dialog systems and traditional instant messaging tools is necessary before any conclusions can be drawn. To conclude, we expect to develop the outcomes of the study into guidelines for the design of effective and efficient user representations in virtual Instant Messaging tools, and for the design of interaction paradigms based on emotional expressiveness.
8. REFERENCES [1] Bartneck C. (2002). eMuu – An Embodied Emotional Character for the Ambient Intelligent Home, PhD thesis, Universiteit Eindhoven [2] Benford S., Bowers J., Fahlén, L., Greenhalgh, C. and Snowdon D. (1995). User Embodiment in CVEs. CHI'95 Proceedings, Denver/Colorado, ACM Press [3] Damásio A. (1994). Descarte’s Error: Emotion, Reason and the Human Brain, Avon Books
[4] Desmet P. and Hekkert P. (2002). The basis of product emotions. W. Green and P. Jordan (eds) Pleasure with Products, beyond usability. Taylor&Francis. 60-68 [5] Ekman P. (1999). Facial Expressions. T. Dalgleish and M. Power (eds). Handbook of Cognition and Emotion, New York. Wiley & Sons [6] Fabri M., Moore D., and Hobbs D. (2004). Mediating the Expression of Emotion in Educational CVEs: An Experimental Study. Int. J. of Virtual Reality. 7(2) [7] Jordan P. (2000). Designing Pleasurable Products. Taylor & Francis. London [8] Lessiter J., Freeman J., Keogh E., and Davidoff J. (2001). A Cross-Media Presence Questionnaire: The ITC Sense of Presence Inventory. Presence. 10(3) [9] Nardi B., Whittaker S. and Bradner E. (2000). Interaction and outeraction: Instant Messaging in action. CSCW 2000 Proceedings. ACM Press. 79-88 [10] Nichols S (1999). Virtual Reality Induced Symptoms and Effects. Phd Thesis, University of Nottingham [11] Nielsen J. (1994). Heuristic evaluation. Nielsen and Mack (eds) Usability Inspection Methods. Wiley, NY [12] O’Reilly W. (1996). Believable Social and Emotional Agents. PhD Thesis. Carnegie Mellon University [13] Picard R. (1997). Affective Computing. MIT Press [14] Slater M. (1999). Measuring Presence: A Response to the Witmer and Singer Presence Questionnaire. Presence 8(5). 560–565 [15] Stedmon A., D'Cruz M., Tromp J. and Wilson J. (2003). Two Methods and a Case Study: Human Factors Evaluations for Virtual Environments. HCI2003 International. Crete, Greece [16] Stuart R. (1998). Team Development Games for Trainers. Gower Publishing, Aldershot UK [17] Tromp J., Steed A. and Wilson J. (2003). Systematic Usability Evaluation and Design Issues for CVEs. Presence 12(3). 241-267 [18] Walther J. (2002). Time Effects in Computer-Mediated Groups. P. Hinds and S. Kiesler (eds). Distributed Work. MIT Press, 235-257 [19] Witmer B., Singer M. (1998). Measuring Presence in Virtual Environments. Presence 7(3). 225-240
