distributed VEs, focused on the way people handle computer interfaces and graphical ..... the relatively cheap, Desktop systems and their use for unstructured ..... to compare exactly, component by component, two laptop computers from two.
The International Journal of Virtual Reality, 2007, 6(3):45-54
45
The Impact of Social Interaction on Usability for Distributed Virtual Environments I. Heldal1
Abstract—Distance collaboration has long been a research area for distributed Virtual Environments (VEs). In these environments people interact with the technology and each other in a computer-generated synthetic world. The aim of this paper is to examine how social interaction influences the usability of distributed VEs, focused on the way people handle computer interfaces and graphical representations. The collected data come from two experiments for various problem-solving tasks. The first experiment examined collaboration in networked immersive projection technology and the second collaboration during repetitive usage of personal computers. When observing social interaction in VEs, certain sequences of actions, were identified that supported-and others that disturbed-the flow of collaboration. A main observation is that usability problems are of three different types: 1) problems that are or cannot be observed by the users, 2) problems that have impact only on some users, and 3) problems influencing all users. As the findings illustrate, these problems have different characteristics, and have to be considered differently during usability evaluation in distributed VEs. The results focus on examining social interaction sequences and discuss their resulting effects on the usability and design. Index Terms—Collaboration, distributed interaction, usability, virtual environments.
I.
work,
social
INTRODUCTION
Organizations and private people use computer networks on an everyday basis today. In relation to other information and communication technologies distributed Virtual Environments (VEs) provide sharing and direct manipulation of images. The users in these environments are free to navigate through the virtual space, “encountering each other, artefacts and data objects, and are free to communicate with each other using verbal and non-verbal communication through visual and auditory channels” (p.5) [1]. This also means sharing social cues [2] and creating an illusion of presence and collaborative presence. Distributed VEs bring together people with the same interests, who are collaborating over space. They are useful for making prototypes (e.g. [3]), training (e.g. [4, 5]), treating phobias (e.g. [6, 7]), rehabilitation (e.g. [8, 9]), education (e.g. [10, 11]), architecture (e.g. [12, 13]), etc. New technologies with new interfaces, often with innovative devices, appear which can focus better on certain problems (e.g. Manuscript Received on June 26, 2007. Ilona Heldal is an assistant professor at the Division for Technology and Society, Chalmers University of Technology. Her research area is human computer interaction, computer supported collaborative work, with focus on communication by using virtual reality technologies. E-Mail: ilohel@mot. chalmers.se.
[13, 14]). Besides the more specialized systems, like Powerwalls, head-mounted displays (HMDs), and CAVE-type systems [15], high-speed Desktop computers, capable of rendering complex 3D models, have become more common and allow real-time distance work [14, 16]. For designing new environments with more suitable functionalities, appropriate usability-evaluation methods and guidelines are needed [17]. An important characteristic that differentiates collaborative distributed VEs from single-user virtual environments is the presence of social interaction. Usability guidelines considering social interaction have been defined and already show their benefits for other technologies, for example in video conferences [18]. In design of new VEs such guidelines has rarely been taken in consideration [19]. Reasons can be, for example: the immaturity of the technology, or the complexity and non-deterministic character of social interaction and group communication [20, 21]. The aim of this paper is to examine how people interact in distributed VEs with special focus on social interaction, and to investigate how social interaction influences the way people handle interface technologies, virtual representations and problem-solving. This is done in order to identify social interaction characteristics which can be generalized and contribute to usability development for VEs. In order to obtain generalizable results across technologies and applications, the collected data come from two experiments that examined long-term use of distributed VEs. The first examined collaboration in networked immersive VEs; the second, collaboration during repetitive usage of low-tech equipment –personal computers, i.e. Desktop systems. For both experiments, several applications were studied – open-ended and with clear goals, structured and non-structured, abstract and concrete. When observing social interaction in these environments, certain sequences of actions were identified that supported, and others that disturbed the flow of collaboration. The paper presents such social interaction sequences and examines their impact on usability development for VEs. We begin the paper with a brief discussion of related research on group collaboration issues for usability. This is followed by hypotheses and study design. After this, we present the input experiments and the applied methodology examining interaction in distributed VEs. Chapter Ⅴ includes examples on social interaction sequences, and Chapter Ⅵ lessons leant from these. Finally a broader discussion is given on larger issues contributing to understanding usability and collaboration when using new technologies, concluding with recommendations for future design and evaluation of distributed VEs. The immersive projection technology system used here is a
The International Journal of Virtual Reality, 2007, 6(3):45-54
46
CAVE™-type system [15]. This work will employ the term “immersive projection technology” (IPT) for such systems. The Desktop computers are personal computers used in networked settings, collaborating via Internet. This paper investigates only small group studies in real-time collaborative settings. II.
RELATED RESEARCH
To monitor what is happening when people interact in distributed VEs is hard [21]. The collaborators have to handle the technology, know what they can expect from their partner, and understand whether they share a common aim or are working with individual aims. The results depend on the underlying social interaction and the performed interaction via technology. Each step is influenced by both technical and social aspects [20]. What collaboration means is an important topic for discussion in relation to networked VEs[19]. There are discrepancies already on the definition level for the term “collaboration”, depending on the technology involved (or whether technology is excluded from consideration), the ease of conversation, the kind of cooperation, the communication modalities, and whether the humans concerned are small groups or bigger groups etc. There is a distinction between whether people choose to collaborate by their own initiatives, whether they have to collaborate for work, and whether they are asked to collaborate for experiments in laboratory settings. Purely technical issues, for instance how the interaction can be transmitted by the user interface, may be involved in experiencing collaboration [21]. This is also true for social issues [20, 22]. According to some studies, the context of the environment [23], the flow of conversation [24], the representation of the other person [25] also influences the course and the outcome of the collaboration. This paper focuses on social interaction at micro level. At this level, the following main dimensions can influence it: language, group inclusion (e.g. role-taking), trust, security regarding facts, interpretations, categorizations, and rituals [26]. We examine these aspects from the perspective of an external observer. For the tasks examined here, people solve problems in distributed VEs. A definition of problem-solving considered here is that individuals encode a problem into their working memory, search for algorithms and heuristics to solve it in their long-term memory, execute these, and compare the new state with their original goals [27]. When people solve problems, there is a tension between the individual problem-solving and the group [20]. Since the individual members solve problems through several internal processes, these do not necessarily are known by others. To understand the different phases of the problem-solving in the group, the members have to externalize these internal processes [28]. This is done through social interaction and interaction via technical interfaces. It may be impossible to correctly divide the relevant social and technical interaction during the analysis of a certain activity in VEs [20]. Here, the intention is to focus only on the main, most representative characteristics of interaction. Social interaction sequences are originated from the social context.
Examples are short spoken sentences, actions, movements showing the personal enjoyment, the ease or difficulty of working with strangers, wellness, cultural differences, previous knowledge, current health status and so on. Similarly, technical interaction sequences are those which are determined by the equipment and the software used, i.e. the technical context. Examples are distractions observed during movements, navigation, orientation in the surrounding physical space, awkward devices, slow software responses, or network disturbances and delays. Since all distributed VEs have to consider interaction via technology, a first consideration for development guidelines may be the impact of this on the user interface [21, 29]. For using Desktop-based VEs, Hindmarsh et al. recommend examining major types of actions (looking, speaking, pointing, grasping) for object-focused manipulation in an environment by rethinking the representations of bodies, targets, and environments [30]. Based on this work and enhancing the role of social interaction Heldal suggests examining the flow of social interaction and problem-solving besides interaction via technology [20]. So far, there is only one longitudinal study that considers social interaction for defining usability guidelines for collaborative VEs[31], further revised by Tromp and colleagues [21]. They consider observations and measurements of human behavior following a limited number of predefined activities. This work aims to find the most representative actions before defining improved usability guidelines or evaluation methods. III.
STUDY DESIGN
Based on the presented background we can make the following hypotheses: z
z
Social interaction does influence usability for distributed VEs. Acknowledging this first hypothesis is not surprising. However, to connect it with theories and systematically examine social interaction, pointing to benefits for usability, is new. There can be defined such guidelines that are not application dependent and that can be used to predict how social interaction influences usability.
To examine the hypotheses we apply mainly qualitative approaches. For the background data we use two studies. In these, the basic data were collected via observations and thereafter lessons were drawn. The observations examine social interaction sequences, i.e. the way people handle their partners during conversation (paragraph 5.1) and communication (paragraph 5.2), and to what degree one is aware of the other and how this conversation is influenced by the surrounding environment (paragraph 5.1) dimensions indicated by Preece [32]. We used these observations and apply to the methodology defined by Hindmarsh [30] and further revised by Heldal [20] as input for this study. Via repeated observations, again, we relate these interaction sequences to each other, and compare and discuss what can be learned for further usability development.
The International Journal of Virtual Reality, 2007, 6(3):45-54 3.1 The experimental studies The first experimental study focuses on networked immersive environments (for detailed information see [33]) and the second on Desktop systems (for basic information see [34] and [35]). 1) Long-term use of IPT technologies This study is based on data from an experiment that examined long-term collaboration of couples on different tasks in networked IPTs. Six pairs worked together on (1) a Rubik type puzzle, (2) exploring a Landscape together, (3) Cluedo, solving a murder mystery by finding the killing weapon and identifying murderers, (4) solving word Puzzles, and (5) building a Lego-like construction in the Modeling World (for further information see [33]). For each task, object-focused applications were examined, where the users had to manipulate objects in order to solve the problems (e.g. Puzzle, Modeling), as well as applications where the users had to explore environments (e.g. Landscape, Cluedo). There were applications with clear goals (e.g. Puzzle, Cluedo), and others that were open-ended (e.g. Landscape, Modeling), abstract (e.g. Puzzle), and concrete (e.g. Poster). Each participant was portrayed to the other by the use of a simple avatar with a jointed left or right arm. Navigation was effected by a “move in the direction of gaze” metaphor. The object manipulation metaphor was a ray-casting technique but using a short ray. The subjects could talk to and hear each other by using a wired headset with microphone as well as earphones. The pairs spent at least 210 minutes each, doing five tasks together in the IPT VEs over the course of a day. The times that pairs spent for each task session were between a minimum of 25 minutes and a maximum of 70 minutes. For many other research studies performed in these fields, the users do not spend more than 15-30 minutes continuously in the virtual environments. The collaboration was video recorded and the conversation also audio recorded. 2) Long-term use of Desktop system based CVEs Having in mind to explore the advantages and limitations of the relatively cheap, Desktop systems and their use for unstructured meetings, we present a study that took place over the course of nearly three months. Four participants took part in ten, minimum one-hour long, meetings in a Desktop system based virtual environment in ActiveWorlds (AW, see www.activeworlds.com). TABLE 1: THE VIRTUAL SESSIONS DURATION TIME (T) IN MINUTES AND MEETING PROFILE (M). M: P = PLAN, L=LEARN, PS = PROBLEM SOLVING OR E=EXPLORE 1 2 3 4 5 6 7 8 9 10 T
90
M P
70
70
100
90
60
70
60
70
120
P,
P,
P,
PS
PS
PS,
PS
E
P,
L
PS
PS
E
L
For people collaborated during 10 virtual meetings (see TABLE 1) and 6 real meetings. Only to support the distributed meetings when it was considered necessary and time-saving, a real-life meeting has been inserted between two distributed
47
meetings. There were real-life sessions for 120 minutes before and after the virtual meetings, and for 30-60 minutes after the first, second, fifth and ninth virtual meetings. The data were collected from questionnaires, including individual evaluations and results reported both by subjective comments and on a 7-point scale after each session, and observations. Two different types of goals are distinguished for this study, viz. the goal of a meeting occasion, as defined for each meeting, and the goal of the entire experimental study, i.e. to manage to reduce the real meetings that support the virtual meetings. As an example, learning how to build was a goal for two meetings (according to TABLE 1, and for a constructed house see Fig. 1).
Fig. 1. Goals of meeting were, for example, to learn how to build (L) or to build (PS) a house.
Some goals for the meeting occasions corresponded to the goals for tasks from the previous study, e.g. for puzzle solving, for navigation in landscapes, and for building with different shapes. The results of this study showed that positive experiences in collaborating contribute to higher effectiveness, independently of the tasks. Several properties, like the structure of a meeting, how concrete the actual goals were and how well the roles in the group were defined, have been identified as factors influencing use and usability. Additionally, two processes of adaptation have been identified, one to the system (the technology) and another to the group (between the users). IV.
EXAMINING INTERACTION IN DISTRIBUTED VES
This is the first paper that uses all observation results from the previously described studies. To relate the lessons on overall interaction experiences from the two experimental studies, we will separately examine three underlying activities: (1) social interaction, (2) interaction via technical interfaces and (3) problem-solving, as presented previously [20]. This paper focuses on social interaction at the first instance. To understand this method, we first present how technologies support interaction during distributed collaboration. This is followed by presenting the way social interaction was examined.
The International Journal of Virtual Reality, 2007, 6(3):45-54
48
4.1. Experiences and interaction in distributed VEs Seamless collaboration requires non-problematical quick social interaction, via technology, and people’s ability to choose, understand, and approach the common goals. A team needs to collaborate at a social level in order to agree upon goals, methodology and roles. However, the richness of communication of social cues, such as non-verbal communication, can impact interaction, especially when some of the cues are lost. There are social cues that are filtered out by the medium [19], but also technical limitations that hinder transmitting all interaction [14], as TABLE 2 summarize. TABLE 2: EXAMPLES OF ACTIVITIES THAT ARE TRANSMITTED AND THAT ARE NOT TRANSMITTED VIA THE DISTRIBUTED SETTINGS. Examples Social Interaction Problem-solving Interaction via Technology Transmitted via technology
Voice
Movements
Tracked movements
Not transmitted via technology
Feeling sick
Manipulating virtual objects Failures with devices
Social cues Interruption from people outside the VE
Problems with the partner’s technology
What one says and does via social interaction or interaction via technology When one is doing nothing, thinking, bored, or waiting for something, etc. If one is confused
Orientation
Activities done in relation to problem-solving are mainly internal [28]. A given member of the group does not necessarily know what the others want or how they solve the problems to begin with. Following how the other person solves a problem can also be difficult [22, 30]. This is dependent on how active the person is, for example, whether she expresses thoughts by means of action or by speaking. The partner may misinterpret one’s intention if one is only thinking quietly, for example. However, this also depends on the accuracy and level of detail to which the technology captures, communicates and represents human movement [14]. What one user knows about the other and the other’s intention is only the transmitted information, e.g. voice, text, tracked gestures, and updates in the environment. Consequently, there is a remaining part of social and technical interaction that is not transmitted but can influence the overall outcome. Problem solving has to be exteriorized to inform one’s partner. This exteriorization occurs via social and technical interaction [21, 29]. Accordingly, problem solving activities are only partly transmitted via technologies, since social interaction and interaction via technologies are only partly transmitted. However, external observers may be able to see more than the transmitted information, i.e. everything what is visible in the watched settings. They can observe activities that the partners, from the other distributed settings, may ignore [22, 30]. Therefore observations, even if they are time demanding, are important for examining interaction in distributed environments [33].
4.2. Observing activities in distributed VEs In order to relate the results from the two experiments to each other, we reused the observations from the described two studies, while now focusing on social interaction. Data that also support this work are on several film sequences or in texts from notations. We also considered some assessments from the questionnaire in order to argue for results. As we pointed out earlier, the original idea of studying small interruptions in the flow of collaboration comes from Hindmarsh et al. They defined “fragments”, i.e. interruptions that disturbed the seamless interaction. These were mainly technical problems [30]. For this paper, these small sequences of actions: 1) can also support interaction, 2) not only occur during problematical interaction via technology, and 3) can also take a little longer time. This work mainly focuses on those activity sequences that occur during social interaction. One of the advantages of analyzing recorded interaction sequences are that we can see where people’s focus of attention is directed and how, for example, they cope with the environment and the objects in it (including each other’s avatars). A second advantage is that these sequences can be analyzed in great detail, including making sense of the whole scene and the dynamics within it. And finally, the recordings can be made available for re-analysis, which can reduce the errors of one researcher’s transcription and interpretation of the data. V.
RESULTS
Here we present social interaction sequences that illustrate their disturbing or supporting effects on the outcome of collaboration. We focus on the flow of conversation and communication, and to what degree one subject is aware of the other, based on the categories given by Preece and her colleagues [32]. 5.1 Conversation 1) Using networked IPTs Small appreciative phrases, e.g. “Great”, “Precise”, “Cool”, “Fine”, “Perfect” etc., effectively supported collaboration. When one subject in the collaborating couple encouraged the other, the collaboration went well, even though the partners seldom reacted to these phrases. People did not necessarily react to short acknowledging sentences, like “this is nice/hard/right”, “I’m here”, etc. or questions like “How are you?”, “All well?”, but these had an important role in keeping the conversation going. Similarly, people who often complained how difficult the task was, that they had never done such tasks before, and that they did not think they could do it, or who were silent or acted negatively etc., disturbed the collaboration, even though their partner did not necessarily answer these complaints directly. The example below shows how silence together with helpless laughter disturbs collaboration: A: I still haven’t learned how to operate. How to get them… B: [silence] A: Aha. What happened with that one? B: [silence]
The International Journal of Virtual Reality, 2007, 6(3):45-54 A: Do you think we’ll make this in 20 minutes? B: Hah [strange laugh]. No. I don’t think so! A and B had overall a hard time keeping collaboration going. A generally felt dispirited by B’s silence, and an observer could notice that B was annoyed by listening to his constantly complaining partner. How quickly one answered the other was very important and influenced the results directly. Long periods of silence can confuse the partners because one does not know whether the other is listening at all, or is ignoring comments (intentionally or not). A user from another pair continually made greater efforts that kept the conversation going, according to the example below:
49
Conversation played different roles in the different phases of problem-solving. To suggest strategies, to ask the partner about the suggestions, to acknowledge the partner’s suggestions quickly, and to show that one is active and interested in the task were especially important in the introductory phases for each task. The pair who did the tasks best had intense conversation periods, with a lot of questions at the beginning of each task. Frequently asking questions, acknowledging, rapid turn-taking had, in general, a positive effect on collaboration. However, the quick comments sometimes resulted in confusion. By not having gestures, facial expressions or eye glances, the conversation was occasionally stopped by unnecessary questions on what the other user mean.
A: We are going to run out of the cubes soon. [He tries to be funny] B: I think we may have different colors on each side. A: Yes, I think so. B: [silence] A: OK, black is no color. Maybe you tried to tell me that? B: Hmmm [no real answer]: A: “Ah.” The collaboration went much better for them working on the Puzzle (see Fig. 2). The conversation was problematical for this pair, but A always pushed it forward. B was passive but he did not avoid answering with at least “OK”, “Sure”, “Right”. As a result, they were one of the two pairs who succeeded (correctly or not) in putting together all the 11 sayings for the word puzzle tasks (Poster). For another pair, even if one had problems with the language, he never asked his partner (A) for help. The following conversation sequences show how a subject gets frustrated because of his passive partner (B) for the Cluedo application: A: OK. Ah…Last time, when you clicked the rooms, what criteria did you follow to click on the rooms? B: Ah, Ah… A: You see, you have done it well. So?… I don’t know…what did you do with the rooms? B: Last time? Do you mean last game? A: Yapp. B: [do not respond]. A: Yes? B: Do you mean…? [short pause]. A: Yes. [upset]. B: ...what I did with the rooms last game? [pause]. A: Yees! [more upset]. B: Jaha. Here I can see you. I’m right behind you. A: OK... [he gives up]. B: I can see the dead body again. [pause] I just clicked on the posters. B had additional problems with using the devices, seeing objects in 3D that made him talk in Swedish (during the tasks to himself or to the operator about his problems), which upset A on several occasions, since A did not spoke Swedish.
Fig.2. Puzzle.
Speaking at the same time, over each other’s words, was not disturbing but, on some occasions, supported collaboration. For the Posters, the people often read the words at the same time, by dividing the posters between themselves, and each subject read the words from the posters in her or his part of the room). 2) Using networked Desktop systems The conversation in the group can be considered to have been chaotic at the beginning. During the first session the participants had several problems, with usage of the technology and with each person having to focus on the three others all the time in the virtual world, looking at the text messages and checking whether they matched what could be heard via the audio. The novices had to handle two technologies, not only ActiveWords, but also VoiceCreator at the same time. Already after the first chaotic meeting, a few minutes at the beginning of each next meeting were spent on structuring the activities and conversation routines. The second meeting in real life (which followed the first meeting in the virtual environment) treated questions on structure. Questions like “What is the quickest way to get attention? What should someone do if she or he would like to speak, but the audio is busy? How to design a training session in AW?” was discussed. During this meeting one expert participant noted: “Please note that each comment is valuable. Please comment, ask, or just let the others know what he or she cannot see.” The novices were interested in whether there existed a “list of social rules” for AW in relation to how to handle situations when a participant had disappeared to another place in AW. The answer at that stage was no. The communication, and in general the collaboration, was
50
The International Journal of Virtual Reality, 2007, 6(3):45-54
evaluated as being worst during the fourth session, although the audio worked well. It took one more than one hour to decide what to do. One participant mentioned after the fourth meeting: “I changed my way of looking upon the others”. This person did not think that enough feedback or attention was gotten from the others. 5.2 Communication 1) Networked IPTs For the IPT experiment, it is hard to completely differentiate communication from conversation, awareness, or problem-solving that supports collaboration. Furthermore, treating the other’s avatar can often be regarded as communication. However, people could speak in the environment without communicating mutually: e.g. a person sometimes spoke Swedish, or with the operator; another person sometimes spoke with the objects; people described certain movements without being interested in whether the partner answered or not. One could be aware of the other although not necessarily communicating with her or him, for example when one subject became frustrated that the other was too quiet or inactive, as we showed previously. Dealing with the other’s avatar is not necessarily communication, as is demonstrated by the fact that a subject just walks through it several times without mentioning it by the time she or he solves the problem. In the beginning of the study, people were eager to know more about their own avatars, such as whether they had similar ones. The users discussed how the avatars, the objects or the environment look like. Besides speaking to each other, the subjects could easily see their partner’s location in the model during most of the tasks and notice what she or he was doing. By taking a short glimpse of the partner, one could ascertain that everything was all right. They could relate to their partners, refer to objects in relation to the partner’s position, and judge body sizes. Showing and pointing was important for collaborating effectively on the tasks, so as to help each other if needed. We present here some examples of these activities. Seeing what the other was doing and speaking at the same time enabled one subject to help or assist the other in handling the technical devices. The real-time communication also favored possibilities to discuss technical inconsistencies in the environment. Even though the subjects could not know whether their partner’s environment and the devices used were exactly the same, they assumed it. On several occasions for the Landscape environment and for Cluedo, a person went out of the simulated environment without wishing to. If somebody was “away” for a time, their partner usually tried to help when they proceeded to come back. The users often helped each other. The following example presents a case when a subject asked her partner for help to move an object and, in addition, instructed the partner to put it in the right place with the right position. A: Could you please help me with the red one? This is one that I definitely cannot move. Please put it round this one [showing it with the wrong hand]. B: [Follows the instructions given by A.]
A: A little more… A little more on my side…. Now straight ahead… Straight… A little more…Yes! Here we should emphasize – in addition to the observations regarding the frequently used non-tracked hand, described in the section on interaction via technology [22, 30, 33] – that the partners jointly used their non-tracked hand for pointing in all tasks, from the beginning of the experiment until the end. Sometimes they observed their mistakes and commented on them (in the example from the Puzzle below, B’s comment is ironic): A: Here, here... against me... Do you see my... Of course, you can’t see my hand... Damn, I’m waving with my real hand, hmmmm... B: Sure, yes... If you point with the joystick hand I’ll see it. 2)
Networked Desktop systems During this experiment several different communication possibilities were used, e.g. audio, text, and non-verbal gestures, and they have been shown to be used differently for different tasks. Waving, for example, could also be used to signal problems if the audio got disconnected. The communication was evaluated as quite stable during all the trials. There were no extreme values or major differences in the reported values. Surprisingly, contrary to the expectations from the design, no covariation was found between ratings for collaboration and ratings for communication. The reason for this may be the use of different communication modalities, of which at least one (text or voice) worked during the 10 sessions. There were individual differences on how people experienced communication. An example regarding the individual differences for feedbacks may be that one of the novices complained constantly of missing feedback, and the others learned to handle this (responding more often to this novice) over the course of the meetings by using different signs or gestures. Perhaps not surprisingly, a novice mentioned after the ten sessions: “I thought that communication was more easily maintained at the end.” How often a person needs feedback cannot therefore depend on expertise in this case, since one expert mentioned: “I don’t care if people hear me or not when I’m speaking. I only care about the feedbacks… This you learn to handle with time. Now you need to wait. Now to repeat… Now to take it all from the beginning…” 5.3 Awareness of the partners 1) Networked IPTs The partners were aware of each other since they helped each other directly (when one asked for help) or indirectly (when they were looking around and monitoring what the other was doing or what was happening). For the Puzzle and the Modeling applications, since the environments were small and open, each user could constantly see the partner. The only exceptions were for the participants in the IPT system in London, who would occasionally face out of the blank wall. However, this did not cause major disturbances in collaboration. For the other applications, there were times when one could not
The International Journal of Virtual Reality, 2007, 6(3):45-54 see the other’s avatar. For the Cluedo, one was often in some different room; for the Poster, one sometimes just left the virtual room; and for the Landscape, one was long out of sight of the partner. In these cases, the subjects often reported to each other in words about where they were and what they were doing (for example, working in the application Cluedo). Almost all subjects in the experiment helped their partners occasionally during the tasks. For the Puzzle task, the subjects often asked their partners about the colors of the hidden sides, i.e. the sides which they were not facing but saw that their partners faced. Even though manipulation with respect to the three axes of the coordinate system is intuitive in the IPT systems, the subjects could get a quicker sense of the colors of objects by asking their partners than by doing the manipulation themselves. Especially when they were looking for sides with special colors, and their partners faced many cubes, it was easier to ask for, say, a “blue-red” cube.
Fig..3. Shaking hands in the Modeling task.
Many of the subjects verbally described what they were doing in the environment. They followed their own movements and thoughts with a “track of words”, i.e. describing with words what they were doing. Such behavior was quite typical for this application, probably because the subjects here often were in different virtual rooms and could not see each other. “Now I’m in the … room” was a typical phrase in Cluedo. Being aware of the partner, and doing things together with her or him, seemed to be important. However, when people got to know certain benefits of the partner’s technology that they did not have, often in the middle of a task, they became frustrated. Almost all subjects in Gothenburg observed at a late stage that they could see the posters on the walls just by rotating themselves. When they mentioned it to the partners from London, who usually knew it already, they often felt discriminated and grew frustrated. As we pointed out earlier, there was a difference in how one handled the other’s avatar during the different stages of collaboration (introductory phase, proper collaboration, end phase). During the introductory phase they often briefly discussed each other’s appearance, the colors of their clothes, the sizes of their avatars, and the name tags on the top of the other’s head. They were also interested in their own appearance, and whether they looked as “slim” as their partners. They often
51
combined this short information with “real information”-where the other was situated physically, how the weather was in that city, or about their occupations. Previously it was described that when people crossed the other’s avatar - early in the introductory phases -, they mentioned it briefly. Many couples thought that this was fun and they did it on purpose. During the proper collaboration, when each subject was focused on solving the task, they did not care about their avatars except as references. If they stopped solving the task and discussed with each other – not necessarily task-focused issues, or in the introductory or final phase – they handled the others’ avatars nearly like real people, for example facing each other when speaking to each other, looking at each others gestures, or even trying to shake each other’s hands (see Fig. 3). In the intermediate phase the users crossed each others avatar more often. There were instances for most of the tasks when one crossed the other’s avatar twenty times during one minute, or also worked by staying for several seconds in the other person’s avatar. In these cases they mentioned the other’s avatar only if it disturbed problem-solving, e.g. blocked the view. 2) Networked Desktop systems In the experiment with Desktop systems, the adaptation process observed was not only about the technology, but also a social adaptation between the users. The novices knew at the end of sessions how to handle each other, and how to handle the experts, even if at the end of the 10th session they still could not consider themselves to be experts. Many usability problems originating in social interaction problems regarding the “use” and choice of avatars were observed throughout the sessions. For example, there were problems regarding recognition, feedback interpretation, especially for manipulating in 3D when the representations were in 2D, or due to inconsistencies between using audio (VoiceCreator) and changing locations. For exploration it was difficult to orient oneself and to follow another avatar from a constant distance, and to have the same view as the other’s all the time. Problems were also reported which originated in different collaborative actions. An example was the difficulty of “doing something else than the others” (allowing the individual to remain peripherally aware of the others’ activities and to return), and of “contributing to main conversation and subset conversation”. Both of these problems were seen during session 4, when one participant forgot the group for a while by engaging in a conversation with a visitor (a foreign AW user’s avatar) in the virtual environment, or another participant left the discussion in AW for a visitor in real life (in his physical office). VI.
LESSONS
In this section we discuss some of the direct implications of the identified sequences for collaboration. Small conversational sequences may be ignored, but play an important role in use. They were more often identified as benefits – small social links that kept the conversation going. Encouraging each other and acknowledging often may have a positive effect on problem-solving. We also showed that
52
The International Journal of Virtual Reality, 2007, 6(3):45-54
speaking to each other could help, although missing communication cues can confuse the other, as one does not know who should speak next. Long periods of silence were confusing. People could not know if the others were thinking, had problems, or just left. Saying things loudly together helps problem-solving. In a way similar to how non-symmetrical technical settings disturbed collaboration, different knowledge of the language used does so as well. People get disturbed because the partner speaks a language better; they can also be annoyed by the partner’s clumsiness or unwillingness. However, good intentions and effort, or time, can promote the collaboration. Here we define aspects that influence problem-solving. By taking into account in design those aspects that keep conversation going (for example, not letting long periods occur without any feedback, especially in the beginning of conversation), and by further studying what kind of feedbacks can compensate for conversation problems, it should be possible to increase effectiveness for problem-solving. A frequent pattern of behavior we saw in the networked IPT system was occasional glimpsing at the other user just to maintain awareness. Between communication sequences, we as many previous studies (e.g. [21, 22, 30]), presented the importance of showing, pointing and helping the partner. This will immediately affect the interaction via technology and problem-solving. This is much more difficult for interaction in Desktop-based VEs [22]. People have to know about, or learn, certain conventions in order to understand each other’s movements, gestures or in general the other’s intentions correctly. We showed that the subjects are aware of their partners: they ask where the other is if they do not see her or him, discuss their appearances mainly in the beginning, apologize if they collide with the other’s avatar, and mention when they see that the other crosses through them. They solve problems together, do things together, face each other sometimes when speaking together, help the other, and try to be polite in some cases when the other’s unwillingness to collaborate is upsetting. One can even adapt to the other’s behavior, if she or he always complains because of technical disturbances or, for example, speaking a different language. This means that the awareness of the other partner, and of what one must do to have suitable contact with the other, influences both problem-solving and technical interaction. To be aware and avoid frustration, a first step would be to make known already from the beginning the differences in technology and also the main differences in each other’s backgrounds. It would not be hard to allow use of a dictionary, or to make translation available for the tasks. Simply letting the partner know at the outset that the other needs more help with the language, for example, would counteract unnecessary frustration during problem-solving. Accordingly, it should be an essential design issue how extensive and intensive a social or technical interaction the application should support. Some suggestions for this are to: z
z
Use more suitable feedbacks for the social interaction, based on phases in collaboration (initial, intermediate, final). Use more proper feedbacks based on phases in
z
problem-solving (understanding the problem, proper problem solving when differentiating the phases for finding strategies, and following already defined strategies, help for technical interaction). Make a list of possible problems so as to find ways of working around them if they cannot be prevented.
There is an urgent need to build up a library with sequences on “successes and failures” in collaboration. Even though critical cases of successes and failures in collaboration are possible to identify via observations, the length of time during which the sequences should be observed remains to be defined. Examining activity sequences can be easier if one knows how much time is needed to observe a certain phenomenon, i.e. where one should start and stop observing these small sequences. The video recordings were important in this regard, since some sequences were examined on several occasions. These also showed that certain types of sequences are very rapid and some others take a longer time. Some quick activity sequences or problems may not disturb the collaboration directly, and people do not even mention them to their partners, precisely because they are brief and the users adapt to tolerate them – e.g. small technical disturbances – but they occur often and the number of occurrences influences the efficiency of collaboration indirectly. Having measurable ways to identify how many such activity sequences are acceptable for a certain type of application by the users would also contribute to better collaboration. When discussing usability problems in VEs we observed that there were three different types of these: z
z
z
Problems that were not observed by the users at all. These were, for example, small interruptions, discontinuities, or problems that affected in general the smoothness, the quality of interaction. Problems that had an impact only on one user, and not on her or his partners. These were smaller problems with one’s devices, but also devices that in general, through the whole process, were more difficult to use. Problems regarding or influencing individual learning can also be included in this category. Problems influencing all collaborators. Besides major usability problems, many of these problems originated in activities during social collaboration. Some of the problems could be solved together with the collaborating partners.
The usability problems from the different categories should be treated differently in developing usability guidelines for distributed VEs, since their effects on the users are different, as we showed through several examples in the previous section. VII.
DISCUSSION
Just as it would be extremely hard to compare exactly, component by component, two laptop computers from two different manufacturers, one cannot compare immersive and non-immersive VEs. However, their main features in relation to
The International Journal of Virtual Reality, 2007, 6(3):45-54 overall performance, costs, and usage scenarios can be compared. Therefore, to examine future directions of the new technologies, comparisons were made with the Desktop system, as a system that is well established and widely used. At the same it does not require departing from the existing settings in the use of computers. Many of the social interaction sequences present for immersive systems were found to occur for Desktop VEs too. However, these were found, in general, to be much more problematic to use for these easy tasks than immersive VEs. On several occasions people used not only speech but also natural body movements to acknowledge and point at each other. This is acknowledged by a range of studies that examine how natural interaction can be experienced and used in immersive environments [14, 20, 21, 22, 33]. As we showed, sometimes the differences with the different technologies can be treated as quite evident; at the other times they can easily be oversimplified. Accordingly, for example, one can claim that the surrounding representations in immersive technologies obviously yield a more realistic experience than on the screen of a Desktop system – or in order to interact with objects, one can maintain that it is easier to learn to handle 2-3 buttons in an IPT system than to learn to handle more complex commands with a Desktop system. To use laboratory settings can lead to divergence from conditions where end-users employ similar applications for non-experimental reasons. However, the fact that there are no well-established areas where VE applications have proved to be practically useful, especially in comparison with other settings, is a consideration that justifies our settings. It would be hard to choose representative applications from usage areas that are not clearly defined. In future work it will be useful to define more systematically the relative weight or importance of social interaction sequences, and to examine how the kinds of sequences identified here can be generalized further. It would also be valuable to study the role of social interaction for different other tasks, various users, and for other technologies and settings. An additional aspect will be to see whether these activity sequences can be considered exhaustive and no distinctive features are left out of consideration, as well as whether and how far they need to be analyzed in conjunction with technical interaction sequences. VIII.
CONCLUSIONS
Through a large amount of examples this study has demonstrated that social interaction does influence usability (hypothesis 1). By defining a method to examine social interaction (defining “interaction sequences” and showing its generalizability), we illustrated the benefits of this method for designing more usable VEs. Interaction sequences can enable higher efficiency by examining and making each fragment more effective (supporting successes and helping with problems) as well as making efficient interaction sequences occur more often. An example of the latter is providing information or initiating discussions during long silence periods. This work highlights an interconnection between social
53
interaction and usability development that has a major impact on effectiveness and experiences in VEs. By differentiating three types of usability problems from the collaborators perspective, we acknowledged hypothesis 2, e.g. there are group dependent usability problems in distributed VEs. We also exemplified how interaction can be better supported for designing new VEs for collaboration, whether immersive or non-immersive. The major benefits of interaction sequences may lie in the design of feedbacks to support encouragements and acknowledgements, and to deal with silence and with differences in the technology and between partners. By demonstrating how people treat the virtual representations, additional opportunities have been revealed for investigating the role and the realism of simulation for better collaboration in VEs. A broader reflection is that there are links between social interaction and technical development that outside observers can easily become aware of through analysis. However, by being a user or developer one can easily miss the clues. The users are not necessarily aware of them – as has been seen in the examples when the participants successfully cope with the various “unnatural” aspects of settings, and when they fail to overcome problems that they could easily avoid if they made themselves more aware of their situation in the settings. We showed that the usability of shared VEs can be enhanced not only by improving the systems and features of the environment, but also by improving the awareness of the users regarding the activities of their partners in the different settings. These reflections indicate further need for guidelines on social interaction for better technical development of shared VEs. ACKNOWLEDGEMENT I would like to thank Lars Bråthe for his continuous support and constructive ideas. Thanks to Ann-Sofie Axelsson, Alexander Nilsson, Ralph Schroeder, Maria Spante and Anthony Steed for their efforts with the previous experiments. REFERENCES [1]
[2] [3] [4] [5] [6] [7]
D. Snowdon, E. F. Churchill and A. J. Munro. Collaborative Virtual Environments: Digital Spaces and Places for CSCW: An Introduction, in Collaborative Virtual Environments: Digital and Places Spaces for Interaction, D. Snowdon, E. F. Churchill, A. J. Munro, Eds, Springer, London, 2001, pp. 3-20. S. Kiesler, J. Siegel and T.W. McGuire. Social psychological aspects of computer-mediated communication, American Psychologist, vol. 39, no. 10, pp. 1123-1134, 1984. K. Kuutti et al. Virtual prototypes in usability testing, in Proc. of the Hawaii International Conference on System Sciences, 2001. P. McGeorge et al. Using virtual environments in the assessment of executive dysfunction, in Presence-Teleoperators and Virtual Environments, vol. 10, no. 4, pp. 375-383, 2001. Zhigeng Pan et al. EasyBowling: A Virtual Bowling System based on Virtual Simulation, Computers & Graphics, vol. 27, no. 2, pp. 231-238, 2003. B.O. Rothbaum et al. Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia, American Journal of Psychiatry, vol. 152, pp. 626-628, 1995. V. Simon, C. Sík-Lányi and L. Simon. Using virtual public transport for treating phobias, International Journal on Disability and Human
54
[8] [9] [10] [11] [12] [13] [14]
[15] [16] [17]
[18]
[19]
[20] [21]
[22] [23] [24] [25]
[26] [27] [28] [29]
[30]
The International Journal of Virtual Reality, 2007, 6(3):45-54 Development, Spec. issue on Disability, virtual reality and associated technologies, vol. 4, no. 3, pp. 211-215, 2005. C. Sik Lányi et al. Computer-controlled cognitive diagnostics and rehabilitation method for stroke patients, ERCIM News, vol. 61, pp. 53-54, 2005. C. Sik Lányi and Á. Tilinger. Multimedia and virtual reality in the rehabilitation of autistic children, In: Lecture Notes in Computer Science LNCS 3118, pp. 22-28, 2004. Zhigeng Pan et al. Interactive Learning of Computer Garphics in Networked Virtual Environments, Computers & Graphics, vol. 29, no. 2, pp. 273-281, 2005. C. Sik Lányi et al. Virtual reality in special needs early education, Virtual Reality, vol. 5, no. 4, pp. 55-68, 2006. M. Billger, et al. Perception of color and space in virtual reality: a comparison between real room and virtual reality models, in Proc. Human Vision and Electronic Imaging IX, 2004. I. Heldal. Supporting participation for planning new roads by using virtual reality systems, Virtual Reality, Special Issue for E-society, Springer Verlag, vol. 11, no. 2, pp. 145-159, 2007. D. Roberts et al. Factors influencing flow of object-focused collaboration in collaborative virtual environment, Virtual Reality, Special Issue for Collaborative Virtual Environments for Creative People, vol. 10, no. 2, pp. 119-133, 2006. C. Cruz-Neira, D. Sandin and T. DeFanti. Surround-screen projectionbased virtual reality: The design and implementation of the CAVE, in Proc. of SIGGRAPH 93, pp. 135-142, 1993. Zhigeng Pan, J. Shi, Mingmin Zhang. Distributed Graphics Support for Virtual Environment, Computers & Graphics, vol. 20, no. 2, pp. 191-197, 1996. D. Bowman, J. Gabbard and D. Hix. A survey of usability evaluation in virtual environments: Classification and comparison of methods, Presence: Teleoperators and Virtual Environments, vol. 11, no. 4, pp. 404-424, 2002. D. H. Sonnenwald et al. Collaboration in the large: Using video conferencing to facilitate large group interaction, in Knowledge and Information Technology in 21st Century Organizations: Human and Social Perspectives, A. Khalil, Ed., Idea Publishing Co., Hershey, pp. 136-155, 2002. R. Schroeder, I. Heldal and J. Tromp. The usability of virtual environments and methods for the analysis of interaction, Presence: Teleoperators in Virtual Environments, MIT Press, vol. 15, no. 6, pp. 655-667, 2006. I. Heldal. The usability of collaborative virtual environments: Towards an evaluation framework, Ph.D. dissertation, Dept. Tech. and Soc., Chalmers University of Technology, Gothenburg, 2004. J. Tromp, A. Steed and J. Wilson. Systematic usability evaluation and design issues for collaborative virtual environments, Presence Teleoperators and Virtual Environments, vol. 10, no. 3, pp.241-267, 2003. I. Heldal et al. Successes and failures in copresent situations, Presence: Teleoperators in Virtual Environments, vol. 14, no. 5, MIT Press, vol. 14, no. 5, pp. 563-569, 2005. E. Goffman, Frame Analysis: An Essay on the Organization of Experience. Boston, Northeastern Univ. Press, 1986. G. Riva. Virtual reality as communication tool: A socio-cognitive analysis, Presence: Teleoperators & Virtual Environments, 1999. M. Garau et al. The impact of avatar realism on perceived quality of communication in a shared immersive virtual environment, in Proceedings of Conference on Human Factors and Computing Systems, pp. 529-536, 2003. J. H. Turner. A Theory of Social Interaction. Stanford, California, Stanford University Press, 1988. A. Nevell and H. Simon. Human Problem Solving. Englewood Cliffs, NJ: Pretince-Hall, 1972. Y. Engeström and D. Middleton (Eds.) Cognition and Communication at Work. Cambridge University Press, 1996. D. A. Bowman. Interaction techniques for common tasks in immersive virtual environments. Design, evaluation, and application, Ph.D. dissertation, Dept. Computer Science, Georgia Institute of Technology, 1999. J. Hindmarsh et al. Fragmented interaction: Establishing mutual orientation in virtual environments, in Proc. CSCW' 98, pp. 217-226, 1998.
[31] J. Tromp. Systematic usability design and evaluation for collaborative virtual environments, Ph.D. dissertation, Dept. Computer Science and Information Technology, Nottingham, 2001. [32] J. Preece, Y. Rogers and H. Sharp. Interaction Design: Beyond Human-Computer Interaction, John Wiley & Sons, 2002. [33] I. Heldal et al. Analyzing fragments of collaboration in distributed immersive virtual environments, in Avatars at Work and Play, R. Schroeder and A. Axelsson, Eds, Springer Verlag, pp. 97-130, 2006. [34] A. Nilsson et al. The long-term uses of shared virtual environments: An exploratory study, in The Social Life of Avatars, R. Schroeder, Editor, Springer, London, 2002. [35] I. Heldal, L. Bråthe and R. Schroeder. Collaboration and effectiveness for distributed meetings, in Proc. the 12th International World Wide Web Conference, 2003. Ilona Heldal, PhD, is an assistant professor at the Division for Technology and Society, Department of Technology Management and Economics, Chalmers University of Technology. Her research aims to define social and technical preconditions in which people can use information and communication technology better. Her research area is human computer interaction, computer supported collaborative work, with focus on communication by using virtual reality technologies.
