Supporting Content and Process Common Ground in Computer ...

CHI 2009 ~ Computer Mediated Communication 2

April 9th, 2009 ~ Boston, MA, USA

Supporting Content and Process Common Ground in Computer-Supported Teamwork Gregorio Convertino (1), Helena M. Mentis, Mary Beth Rosson, Aleksandra Slavkovic* , John M. Carroll College of Information Sciences and Technology & Department of Statistics* The Pennsylvania State University, University Park, PA, USA {gconvertino, hmentis, mrosson, jcarroll}@ist.psu.edu, [email protected]* ABSTRACT

We build on our prior work with computer-supported teams performing a complex decision-making task on maps, where the distinction between content and process common ground is proposed. In this paper we describe a distributed geo-collaboration software prototype. The system design rationale was gleaned from fieldwork, literature on team cognition, and an earlier lab study introducing a reference task with face-to-face teams. We report on a controlled experiment that evaluates this design rationale. Distinct sets of measures show that that the prototype supported both content and process common ground, offsetting the costs imposed by the distributed setting. We interpret the results in relation to prior work on common ground and draw implications for moving beyond current models of sharing and coordination. Author Keywords

Common ground, CSCW, design, prototype ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

The process of building common ground in the context of complex team activities is quite different from the knowledge sharing process that supports common ground in conversations [7]. Teamwork involves not only the exchange and management of information but also the coordination of actions and the generation of solutions. For example, when specialized experts work in teams on a complex task like emergency management planning, intelligence analysis, or pioneer scientific missions, they must not only communicate efficiently but also coordinate the work process and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2009, April 4–9, 2009, Boston, Massachusetts, USA. Copyright 2009 ACM 978-1-60558-246-7/09/04…$5.00.

generate solutions ex novo. Each expert must individually parse, filter, and make sense of the content; as a team, members exchange and co-analyze content, and develop a strategy of action. In the course of the work process they generate and accumulate judgments or syntheses in order to construct a solution for the task. Note that in such cases the work process is managed without the aid of a pre-existing shared script, as is typical for routine human conversations. A central implication of these characteristics of complex teamwork is that support tools must meet needs that may not be present in communication-oriented activities. These needs include managing large amounts of content in a short time, including content that is tied to team members’ taskrelated roles; supporting both individual and collaborative analysis and syntheses; minimizing coordination costs due to distribution of labor across a set of diverse experts; and enabling low-cost integration of knowledge and solutions. In this paper, we describe a new phase of our research program that has been studying the development of common ground in emergency management planning tasks. In prior work we have reported on field studies of real world emergency management teams [21] and a lab study designed to define and validate a reference task and associated procedures and measures [2, 7]. These lab study investigations helped us to define materials, methods, and measures that are useful in evaluating collaboration in a complex geospatial planning tasks; we focused in particular on charting the development of common ground in teams performing the task [7]. Working from our fieldwork results, the lab study, and published literature, we developed and implemented design ideas for a prototype system to support distributed emergency management planning. The present paper summarizes the rationale behind the design of the software prototype, and the details of a lab study of this software that uses the earlier reference task in a distributed setting (henceforth the Dis/SW study). We discuss how the results from this second study – in combination with those from the prior FTF/Paper study – can be used to assess the efficacy of our design reasoning. _____________________________________

(1)

Gregorio Convertino moved to the Palo Alto Research Center, Augmented Social Cognition Group, Palo Alto, CA ([email protected]).

2339

CHI 2009 ~ Computer Mediated Communication 2

April 9th, 2009 ~ Boston, MA, USA

In this paper, we first introduce the task of emergency management planning, drawing from both field and lab work. Next, we introduce the collaborative geospatial prototype that we developed, with design rationale drawn from our empirical work as well as related research on team cognition, computer-mediated communication (CMC), and computer-supported cooperative work (CSCW) (see reviews in [8, 18]). Then, we summarize the study of the prototype system, focusing on findings that speak to the design rationale of the system. Finally, we draw implications for moving beyond a few unnecessarily restrictive notions about sharing in computer-supported teamwork. Emergency Management Planning

Our initial field study of two local communities in Central Pennsylvania focused on the planning phase of the emergency management work conducted by these communities [21]. The planning phase occurs prior to any actual emergency disaster and allows stakeholders to discuss and practice emergency management strategies. During this phase, we found that emergency management planning teams periodically engage in activities that involve geocollaboration. A central activity was the tabletop exercise, where team members walk through a scenario and reconsider existing response procedures on a shared map in a meeting table setting. The team rehearses interdependencies and builds a shared planning experience; this experience supports improvisation during actual crises. An important need for these teams is to have tools that enable efficient and unbiased sharing (e.g., processing large amounts of information and balancing the contributions from different experts) for flexible knowledge sharing and coordination across different agencies. Team members’ ability to build these shared experiences together was constrained by their availability for co-located meetings; there was little to no support for distributed work. When meeting in person, their ability to share and manage large amounts of content was limited by the lack of interactivity in the physical tools they used. Maps and paper-based annotations do not support exploration and retrieval of information, distributed annotations, re-editable shared annotations, or reuse of prior annotations or their categorizations. Another key issue was the absence of explicit acknowledgements or reinforcements of the roles that are adopted by members representing different agencies (e.g., police, civil government) and expertise (e.g., healthcare, engineering). The roles, which consist of sets of agency-specific competences and responsibilities, are a central organizer of emergency management planning. The absence of roleoriented tools limited on the one hand their ability to select the right content to share (e.g., what should a civil engineer contribute at any given point) and on the other hand their ability to implicitly share the act of sharing itself. By this we mean coming to a common understanding of how shared data, judgments, or actions are related to roles (e.g., agency-specific data, judgments, or actions), as well as the

strategy for tackling the overall task (e.g., first sharing relevant data and then discussing it by area). These observations suggested to us a central requirement for design of role-defining and role-supporting features. Building from the fieldwork, we began to investigate regional emergency planning more systematically and to design a system that supports more flexible and distributed forms of these planning activities. This required us to first develop a reference task for studying complex emergency planning processes in a lab setting. In the field we had observed the practice of the tabletop exercises [21] during the planning phase. We developed a simulated version of this task as our reference task and we used it in a lab study of emergency planning that took place in a FTF setting, using paper maps, post-it notes, and markers. In the FTF/Paper study we validated the task materials and procedures, developed measures that characterize the process and product of the teamwork [2, 7]. Content and Process Common Ground

A central contribution of the initial lab study was the articulation of two types of common ground – process and content [7]. While content coordination requires an abundance of grounded content, process coordination requires a continual updating of grounded how-to conventions also referred to as procedural and strategic knowledge. Clark and Brennan offered a general distinction between content and process common ground, relying on classic examples of collaboration [5]. However, note that their investigations focused on formation of common ground in conversations (e.g., [18]) and not on generative group work in complex tasks. This theory of human language relies on the analogy of communication as a form of collaboration. It involves coordination of the content exchanged and coordination of the process of communication itself. We identify and study this distinction in the context of teamwork. Here, with "process" we mean more broadly the process of group work and not only the mechanics of communication [16]. In our conceptualization, content common ground encompasses "I know that you know that I know what." This is the shared understanding on the subject and focus of work. It is what is commonly referred to as common ground (e.g., the drugstore scenario). This results from exchanging content and mutually checking and signaling understanding. In contrast, process common ground encompasses "I know that you know that I know how." This is the shared understanding of the rules, procedures, timing, and manner in which the interaction will be conducted. This is more akin to a blind pass in basketball. Not only do team members share an understanding of how to do the work but they also learn the signals of when to apply different shared tactics. In a blind pass, I see my teammate looking towards the right but moving the ball to her left. Thus, I prepare myself to catch the ball as opposed to moving down the court.

2340

CHI 2009 ~ Computer Mediated Communication 2

April 9th, 2009 ~ Boston, MA, USA

Reflecting back to the field study findings, we see that content and process common ground are related to our observations of the emergency management planning teams. Content common ground is related to the need to selectively share information: team members must know what the group is discussing, why they are discussing it, and what information is known by themselves and others, if they are to make task-relevant contributions. At the same time, process common ground is related to the need to implicitly share information: team members can be more effective in their information sharing once they know how the group is working to achieve its goal. This process knowledge helps them to know when they should perform specific actions that can help to make progress toward the shared goal. SYSTEM AND DESIGN RATIONALE

The prototype (Figure 1) features a team map on the right and a role-specific map on the left. Each map displays multiple layers of geographical data. The team map is a shared object that is used collaboratively by all team members. The role-specific maps contain unshared data layers that are used privately by each user. The toolbar has tools for navigation within both maps (e.g., zoom-in and zoom-out), as well as tools for annotation (e.g., add a note, scribble tool). The team members can communicate freely through a shared audio channel. More details about the software architecture and the user interface can be found in [21, 12]. The design rationale for this system was grounded in our fieldwork, prior research on team cognition [8], and our FTF/Paper lab study [7]. We focus, below, on the two aspects of common ground: what knowledge is critical to share (design rationale for supporting content common ground), and how it is convenient or effective to share it (design rationale for supporting process common ground).

Thus, to provide support for development of content common ground, we introduced the following two features: (1) A prominent distinction between role (private) and team (public) spaces. The primary function of the software is to coordinate multiple unshared role-specific views of the map with one shared team view of the map. (2) Role-specific indicators to communicate individual team member’s actions on the shared map; the pointers are color-coded to specify owner as well as the highlighting around each team members’ annotations. By constantly presenting to team members the contrast of private role-specific and shared general information views, we hoped that the software would reinforce the importance of role-specific information, instilling in members a more proactive approach to their responsibilities in the task (see [27]). To make the sharing of role-specific information as easy as possible, we included a low-cost function for sharing (see the Copy-To button in the middle of Figure 1). By identifying all shared information in two ways (during the act of sharing via a color-coded telepointer and selection emphasis, and afterwards via color-coded annotations), we hoped that members would be continually reminded that different team members (i.e., roles) were contributing different types of information in addition to what specific information had been contributed by each of their partners during discussion. This should help them to generate a sense of ‘who knows what’ that would be useful in organizing and managing their shared information space. Working from our reference task and associated measures, we translated these design goals into specific expectations about team members’ behavior, in the Dis/SW study: 

As team members become more sensitized to their own role-specific information, they should be more likely to “push” information they hold into the discussion, rather than wait to have it “pulled” from others. Taking the initiative to share rather than waiting to be asked is indicative of greater content common ground. We have already seen this trend in the FTF/Paper study, but we expect it to be greater in this study because of the roleemphasizing features of the software prototype.



On the shared map, team members can always see what information has been shared by which team member. In the original FTF/Paper study we found that recall of this information (i.e., a measure of shared knowledge) increased as teams gained greater content common ground. We expect this effect to be even greater in the software-supported teams, because we have included explicit support for identifying these contributions.



The combination of improved role-specific sharing and better knowledge of what was shared should enhance team members’ perceptions of the quality of their team’s collaboration (i.e., relative to FTF/Paper teams).

Design Rationale for Content Common Ground

The development of content common ground in teamwork can be enhanced by enabling team members to be selective in determining what knowledge should be shared. Early research on team decision-making operationalized shared knowledge as team mental models (knowledge structures held in common by members), and focused on how shared mental models predicted team performance (e.g., [4]). More recently, researchers of team decision-making and collaborative technology have shifted attention from team mental models to transactive models of sharing - knowledge of who knows what within a team. Transactive models appear more appropriate for group tasks involving interdependency and role specialization; researchers have argued that the work of teams making complex decisions is more accurately represented by such models (e.g., [13, 17]). Therefore, the ability of multi-role teams to share large amounts of content could be productively augmented through tools that afford selective sharing of information when it is needed during the collaboration process as opposed to sharing all information up front.

2341

CHI 2009 ~ Computer Mediated Communication 2

April 9th, 2009 ~ Boston, MA, USA

Figure 1: Software Prototype User Interface.

Design Rationale for Process Common Ground

Our second general consideration in the design of the distributed system is that process common ground can be supported by enabling implicit forms of sharing. In other words, we wanted to reduce team members’ need for overt sharing and negotiation of coordination strategies (see "implicit coordination" in [15]). Research on teams in dynamic situations has suggested that to perform effectively the team members need to share preexisting strategic knowledge, which allows them to develop an understanding of cue/action sequences (i.e., which cue should trigger which action), cue patterns and their significance (i.e., cue patterns associated with task strategies), team resources and capabilities (i.e., roles, resources and expertise), and strategies most appropriate for the current task (see review in [8]). For example, again consider a welltrained basketball team, where a successful blind pass from one player to another requires both to simultaneously assess a pattern of cues in the shared dynamic environment of the game, and promptly respond to the current situation using an agreed tactic with no need for explicit communication. To provide support for the development of process common ground, we introduced the following two features:

(1) We used several indications of a member's role (e.g., color-coding, labels) that were associated to object selections and updates and other actions taking place as part of the team’s interaction on the shared team map. These indicators provide a trace of the collaboration that is taking place - visual support for an implicit sharing process. (2) We ensured that there was low-cost but explicit mechanism for moving information from role-specific to team maps (the Copy-To button). This provides a lightweight punctuation to the act of sharing; it also gives each member explicit control over their acts of sharing, reinforcing to each one their goals and responsibilities in deciding to add information to the shared map. Kraut and collaborators [e.g., 24] found that a shared visual space improves not only communication efficiency (i.e., content management) but also the knowledge of the task structure and situation awareness (i.e., useful to managing the process), especially in complex problem solving tasks. Similarly, we expected that affordances of the prototype that give more ‘visibility’ to the process of role-specific sharing would help team members to more quickly develop a shared understanding of when and how to share their individual pieces of knowledge; or to further annotate content already shared. The indicators of a member’s role on the

2342

CHI 2009 ~ Computer Mediated Communication 2

April 9th, 2009 ~ Boston, MA, USA

shared map would act as awareness-supporting tools (i.e., visual cues), enabling sharing at a tacit level. These design goals were translated into more specific expectations: 



If the role-related identifications of content on the shared map makes team members tacitly aware of partners’ sharing actions, then the team will require fewer explicit dialog acts that would otherwise be used to synchronize communication and sharing. They should also require fewer acts related to regulation of the team’s progress toward a goal. Conversely, more acts would focus on judgments. Thus, we should see less Checking and Management but more Judgment acts. If the software system promotes more rapid development of process common ground (relative to FTF teams using paper), then the team’s communication will become more efficient. As more process common ground is shared, fewer communication acts are needed overall, because the members have less need to explicitly manage the work process and can focus more on assessing and sharing information to complete the task.

More generally, we can summarize our design goals and expectations for both content and process common ground in this way: our essential design strategy is to make otherwise tacit aspects of individual and group activity explicit, visible, and permanent (e.g., through color-coded annotations). Such tracking and recording of the shared work should promote distributed cognition and transactive memory, by making selected aspects of past activities a tangible resource rather than a burden to short-term memory. SOFTWARE PROTOTYPE EVALUATION

After implementing a software prototype that embodied the design goals outlined above, we set out to test the effectiveness of our design (i.e., the Dis/SW study). We used the identical reference task, map content, experiment design, and measures of the earlier FTF/Paper study. This allowed investigating not only how common ground developed over the three runs in the Dis/SW teams, but also how this compared to the results of FTF/Paper teams (see [7, 10]). The reference task is similar in topic and types of information as that used by emergency management teams. But to systematically investigate sharing process and decision outcomes the information was distributed as described below. Collaborative Task and Roles

The teams were asked to generate the best plan for evacuating a family from a flooded area to an appropriate shelter. Each member played the role of an expert (Public Works, Environmental, or Mass Care). Because the participants were not actual experts we had to impose the beliefs and knowledge possessed by an expert. Each participant was given a detailed description about his/her role and rolespecific background information. They were also given the following information for the task: role-specific maps, information sheets with role-specific and shared information, and a shared task scenario with background information.

Each team was directed to develop plans for three alternative scenarios of the same type of task; members played the same roles in each scenario. All of the scenarios were similar in that there was a family in need of rescue in a flood plain (map center); there were four possible shelters (periphery of the map) and only one route to each shelter. Each scenario was of equivalent complexity and the order was counterbalanced across the sixteen groups in the sample. Of the four shelters to choose from, one is optimal (i.e., it has the least number of problems, here called cons), but the information provided to each member is biased toward one of the three other suboptimal solutions. But if the team pools its information across roles, then the fourth shelter is clearly the best alternative: it has 4 cons as opposed to 7 cons for each of the other three shelters. Procedure

Participants were seated at three different workstations in three adjoining rooms. At each workstation, a participant had a Dell Optiplex with a 19” widescreen LCD, a microphone, and set of speakers for verbal communication between team members and with the experimenters. They then read descriptions of their individual roles and the task scenario and read the role-specific information sheet relating each piece of information to their role-specific map. At this point, the participants began to collaborate on the planning task. They were instructed to share information with the team by copying information onto the shared map. When they reached a decision, they wrote down the final plan along with three alternatives in order of preference in a final plan document. Teams were given about 20 minutes to complete this task. At the end, the participants completed (1) a questionnaire that asked them to rate the quality of the groups’ process and performance; and (2) a set of openended questions that assessed the participants’ recall of the solutions generated, and the information considered for each solution. This process was repeated three times, with new scenarios and information presented each time. Participants

The study was run with 16 teams. Forty-eight university students were recruited from a large northeastern United States university and were assigned to teams. Same-gender teams, eight male and eight female were formed. The tasks were performed in English and all participants were fluent English speakers, about 28% of the participants were nonnative speakers. About 70% had undergraduate degrees and the remaining had graduate degrees (with age range from 20 to 40 years). The participants had little prior experience with emergency management planning or operations. Data Collection Post-Task Questionnaire and Cature Software

In prior work [9] we presented a questionnaire that produces seven self-reported indices of group process (gain of shared knowledge; quality of communication; communica-

2343

CHI 2009 ~ Computer Mediated Communication 2

April 9th, 2009 ~ Boston, MA, USA

tion means; understanding & expression; ease of referencing & planning; interpersonal awareness; and awareness over time) and indices for perceived team performance and satisfaction. We administered the questionnaire after each run, resulting in three sets of the nine indices (see [26]). The interaction of the distributed teams via the software prototype was recorded through tools provided by Noldus Information Technology [28]: uLog 2.0 (with Camtasia by TechSmith) for logging keystroke-level events and The Observer XT 7.0 for integrating the multiple data records (keystroke-level logs and videos) from each team and run.

Recall of Task Information

A third set of measures assessed group members’ retention of task-relevant information. After each session, we probed their memories for the solution chosen as the rescue plan and the information contributed by themselves and their partners. The expectation is that as common ground increases, the members will recall more about the solutions discussed, including the information provided by partners (e.g., [19]). A measure of recall was obtained by measuring how many available cons were remembered by the members after each run [see 9]. Class Transfer Info

Dialogue Patterns

We coded the content of the transcripts, using an adaptation ([7]) of the Conversation Game Analysis method ([23, 20]). This scheme classifies the communicative functions of dialogue acts (what the speaker is attempting to achieve) rather than their linguistic form or meaning. The method is relevant because it was specifically validated on dialogs in map-based collaborative tasks [23, 20]. We adapted it to our task domain [7]. Table 1 summarizes the codes and descriptions. Two coders were trained in parallel to use the coding scheme. For better reliability they coded the dialogues separately while reviewing the video of the interaction together. For every few minutes of a session, they viewed the video and coded the transcript, then compared their codes. At each such review step they negotiated and agreed about any conflicting codes, referring to the coding scheme. Three samples of coded transcripts were compared. The level of inter-coder agreement was about 82%. The cases of disagreements were discussed and resolved.

(to coordinate sharing)

Check Understanding Manage Process & Decision

We transcribed the communication records of the 16 teams during Runs 1 and 3. We adapted the analysis scheme used by Sellen [22], which breaks a dialogue into turns and pauses. It also codes simultaneous speech, including speech that causes an interruption (SSI, taking the floor), and noninterruptive simultaneous speech (SSNI). We compared counts and durations for runs 1 and 3, normalizing for the length of the run (see [7]).

(to perform task)

As common ground increases, communication becomes more efficient, because the shared understanding means that conversation topics need less introduction or clarification. Conversational turns occur more rapidly and utterances are more compact (e.g., more turns, fewer words, more synchronicity). Researchers have relied on these measures to assess the effects of different communication settings on communication efficiency and amount of common ground [e.g., 20, 22]. We expected to observe similar trends, with our teams becoming more efficient in their communication as common ground increased.

(to share)

Communication Structure

Dialogue Act

Description

Add Info (AI)

Provides new information, not elicited.

Query (Q)

Question used to elicit new information.

Reply (R)

Reply to query to provide new information.

Check (CH)

Verify own understanding, refers to information previously presented by others.

Align (AL)

Verify partner’s understanding, refers to information previously presented to others.

Clarify (CL)

Clarifies or restates information already presented.

Acknowledge (AC)

Signals receipt of information, understanding.

Manage (MN)

Instruction, command, direct or indirect request for action; orchestrating strategy, how to do the work

Judge (J)

Individual judgment, opinion, or preference.

Summarize (SA) Confirm (CO)

Summarizes information previously presented.

Agree (AG)

Indicates approval for a prior judgment or decision.

Requests partners’ agreement on a proposed decision.

Table 1: Dialogue Act Codes and their Description RESULTS Selective Sharing & Content Common Ground

Our first set of results addresses our assertion that a prominent distinction in the system between role (private) and team (public) spaces and role-specific pointers in the shared workspace supports content common ground, which affords selective sharing of knowledge in the distributed team. Setting Matters in Selective Information Sharing

We expected and found extra costs introduced by the distributed medium which undermined the development of content common ground and thus hindered selective information sharing. In the comparison of post-task questionnaire results from run 1 of the Dis/SW study to run 3 of the

2344

CHI 2009 ~ Computer Mediated Communication 2

April 9th, 2009 ~ Boston, MA, USA

FTF/Paper study, we found that FTF/Paper participants (36 participants, 12 teams) gave slightly higher ratings than the Dis/SW participants (48 participants, 16 teams) for quality of process and outcome (MANOVA test F9, 66=3.9, p