Why Some GSS Meetings Just Don't Work: Exploring

In: Galliers, R., Carlsson, S, Loebbecke, C., Murphy, C., Hansen, H.R., O’Callaghan, R. (eds) Proceedings of the 7 th European Conference on Information Systems (ECIS), Cork, Ireland, Vol. III, pp. 1266-1285, June 1997.

Why Some GSS Meetings Just Don't Work: Exploring Success Factors of Electronic Meetings Gert-Jan de Vreede Delft University of Technology The Netherlands [email protected]

Peter Muller Groningen University The Netherlands [email protected]

ABSTRACT Few studies have been reported on application failures of GSS and the lessons that may be learnt from them. This case study compared four GSS meetings that went awry to 46 meetings that were considered successful. Four indicators were used to describe meeting success: perceived outcome quality and commitment, and perceived process quality and commitment. Our results suggest that a perceived low outcome quality is the strongest indication for a poor meeting. Factors contributing to GSS meeting failure are overloaded agendas, little discussion time, diverging perceptions of meeting goals, knowledge gaps among the participants, and applying the technology without special provisions in conflict situations.

1. INTRODUCTION Group Support Systems (GSS) are information systems that aim to make group meetings more productive by offering electronic support for a variety of meeting activities. Since their first appearance in the late 1960s and early 1970s, GSS have grown to become a popular area for researchers in a variety of disciplines. As a result, a substantial body of knowledge emerged about the effects of GSS on group processes and outcomes. Most of this knowledge stems from experimental, or laboratory studies. Although, over the past few years, GSS have been moving out of the laboratory settings into real world application areas, there are still few examples of real GSS applications compared to the number of laboratory experiments [Pervan 1994; Pinsonneault and Kraemer 1989; Zigurs 1993]. Notwithstanding the relative small number of reported field studies, their results paint a positive picture. Moreover, it can be argued that field studies are predominantly reported as ‘success stories’. This can be seen in the overview presented in table 1, in which a number of field studies and their findings are summarized. Note that this overview does not claim to be exhaustive. As can be seen in table 1, a large majority of GSS field studies report successful applications of GSS. They suggest that teams using GSS to support

Table 1 Overview of GSS field studies and their findings. Source

Context

Nunamaker et al. 1987 Nunamaker et al. 1989 George et al. 1992

Findings

100 planners in 7 groups Decreased idea generation inhibition, anonymity separated status, authority and roles from 3 organizations from comments, equal participation opportunity, high participant satisfaction. Various IBM meetings Higher perceived and measured meeting effectiveness and efficiency, improved meeting outcome quality, high participant satisfaction. Tucson office of the Despite successful GSS introduction, GSS adoption failed: the facility was dismantled after Indian Health Service 9 months due to lack of use. Post 1992 Various Boeing meetings Higher perceived and measured meeting effectiveness and efficiency, higher quality of meeting results, high participant satisfaction. Tyran et al. 1992 8 strategic management Higher participation, higher perceived meeting efficiency, more equality of participation, cases in 5 organizations little evaluation apprehension and cross-hierarchical communication support. 1 Vician et al. 1992 A Texaco team using Maintained member attention, less emotional discussions, more negotiation, participation, GSS for TQM activities open communication, systematic problem solving, perceived sense of accomplishment over time, positive perception of technology impact. Dennis 1994 10 meetings involving 5 Higher perceived meeting effectiveness and efficiency, high participant satisfaction. organizations Emery 1994 GSS for IS requirement Perceived efficiency improvement without sacrificing effectiveness. determination Krcmar et al. 1994 50 meetings with various Parallelism perceived to be most useful, divergent perception of usefulness anonymity, organizations perceived correlation between meeting success and task clarity, equal participation, and meeting room comfort. Vreede and Wijk 9 Nationale-Nederlanden Higher perceived and measured meeting effectiveness and efficiency, higher perceived 1997 Insurance meetings quality of results, high participant satisfaction. 1

Used SAMM, all others used GroupSystems.

creative problem solving may have more effective and efficient meetings than teams that use manual processes. This has been reported to result in labor cost reductions averaging 50% and reductions of project calendar days averaging 90%, see e.g. [Nunamaker et al. 1989; Post 1992]. The success of GSS meetings relative to non-GSS meetings is often attributed to specific GSS characteristics: • Anonymity. By being able to enter ideas and votes anonymously, silent or shy participants are more encouraged to participate. Group members are hindered from dominating by position or personality. Ideas appear to be judged on their merit, not on the personality of the person that submitted it. • Parallelism. By generating ideas and communicating them in parallel, participants get equal ‘air time’, preventing production blocking so that participants can spend more time on generating new ideas. Also, working in parallel allows groups to generate more ideas. • Group memory. During an electronic meeting, all ideas and votes are stored electronically. Hence, little time is needed to produce meeting minutes and previous meeting results are readily available in follow-up meetings. Moreover, the meeting record is untainted in nature and also describes the evolution of a group’s position over time. However, these perceived beneficial characteristics are not necessarily an unmixed blessing. There may be some situations in which these characteristic can actually hinder or deteriorate the meeting process. Anonymity may turn out to be an obstacle if submitted information needs clarification and the owner is reluctant to reveal its identity. Also, we personally met with a substantial number of participants who wanted to ensure that their contributions would be identifiable so that nobody else gets the credit. Parallel communication may increase meeting productivity to such an extent that it results in information overload (see e.g. Dennis [1994]) and leads to a disproportionate amount of effort to organize generated information. Group memory may make people hesitant to contribute. People's positions on certain topics may either become more rigid or less outspoken for fear of not having the possibility of changing one's point of view later on in the process. These cautious arguments are not often encountered in reports of field studies and experiments. In fact, some authors do point out success factors of GSS meetings, such as meeting preparation and appropriate facilitation see e.g. [Nunamaker et al. 1995]. But these statements are seldom based on meetings that did not yield successful results or process. Rather, they hint at reasons why the GSS meetings were a success in the first place. This does, however, not necessarily imply that the meetings would have been a failure if the factor would not have been present. (This phenomenon is also referred to as the ‘fallacy of sampling on the dependent variable’). What we propose in this paper is the idea that a lot can be learnt about designing and conducting GSS meetings by studying unsuccessful GSS meetings. We believe there are a number of valid reasons for taking a close look at ‘bad’ GSS meetings: 1. One can learn more from unsuccessful experiences by comparing them to successful experiences than from successful experiences alone.

2. Lessons learnt from bad meetings can have serious implications for GSS meeting design and for the design of GSS meeting facilities. 3. Studying bad meetings enables us to explore the contours of the applicability of GSS to group work. We know that GSS seem to be working quite well in a number of situations. What we do not know much about yet are the situations in which their support is questionable or even harmful. In this paper, we present a case study on bad GSS meetings. The research question for this study is twofold. First, we want to know What causes the failure of some GSS meetings? Second, we want to find out What could have be done to preclude this failure? We will describe and analyze a number of meetings that did not yield the desired results and left the participants unsatisfied. The research objective of our study is to identify a number of success factors for designing and conducting GSS meetings. In pursuing this objective, the contribution of our study is twofold. First, it aims to further the knowledge about the perceived practical value of GSS in organizations. Second, it attempts to provide practical considerations for the application of GSS in real organizational settings. The remainder of this paper is structured as follows: The next section discusses the nature of electronic meetings in terms of ‘good’ and ‘bad’ meetings. The research approach and design of the study are described in Section 3. In Section 4, the results of the study are presented and subsequently discussed in Section 5. The paper concludes with a summary of the most important findings and a discussion of the limitations of the study.

2. GOOD AND BAD MEETINGS Reports of GSS meetings that went awry are hard to find. As stated in the previous section, most field studies report positive findings. Unsuccessful meetings in these studies were in any case not elaborated upon. We propose that there are three likely reasons why unsuccessful meetings are not reported in the literature: 1. Seductive study results. It may be easier to report on successful GSS field studies than on meetings that did not yield the desired outcomes. Especially at the formative stages of a research area, academic journals may not be enthusiastic to publish research that conveys the message that there is a wonderful new technology, but it does not work. 2. No studies done. There are, as far as we know, no experimental or field studies that specifically investigated GSS meetings that failed. Lab experiments with GSS normally focus on issues such as improving group productivity or enhancing the quantity and quality of meeting results. 3. Difficult design of experimental studies. We suggest that it will be hard to replicate a bad meeting in a constructed, experimental setting. First, it is not clear yet what critical factors may negatively impact meetings. Second, it may be too difficult to manipulate subject behavior and context variables to cause a bad meeting to occur. An other reason is that most often GSS just work. At this point, we want to stress that the arguments presented above are not meant to imply that GSS actually do not enhance group meetings in terms of productivity and satisfaction. On the contrary, we find that the research findings reported so far have convincingly shown that GSS have a genuine potential to improve collaborative work. However, to further the understanding of what

causes this improvement and how it can be maximized, we argue that it is worthwhile to take a closer look at unsuccessful applications. So what is a bad GSS meeting? This question does not have a straightforward answer. One can define a bad GSS meeting as a GSS meeting that is not good. But then, what is a good meeting? There are various possible answers to this question. A good meeting is a meeting where the participants feel that they had a good meeting. Or, a good meeting is a meeting in which the goals of the meeting were achieved. People may also find a meeting good if some unexpected insight occurred, in other words, if they take something else out of the meeting except for the results that were beforehand aimed for. Below, we argue that there are two constructs that help us determine whether a GSS meeting is good or bad: the quality of a meeting, and the commitment of the participants in the meeting. Further, we distinguish between two parts of a GSS meeting: the meeting process and the meeting outcomes.

2.1 Meeting quality We propose that the sense of whether a GSS meeting is good or bad can be expressed in terms of the quality of a GSS meeting. Quality is a relative construct. The quality of the outcome of a GSS meeting depends on what the purpose of the meeting was. In other words, in order to determine the quality of the meeting’s outcomes, they have to be compared with the meeting’s goals that were set beforehand. In this sense, meeting quality is the same as meeting effectiveness, where effectiveness is defined as the ratio between the expected results and the realized results of an activity [Veld 1992]. Determining the quality of the GSS meeting process in the same way may be more difficult. It would require us to compare the meeting process to another meeting process that had the same goals and the same outcomes. As it is unlikely to encounter such a situation, it is more practical to assess the meeting’s process quality by inquiring the perception of the people involved on it in relation to the meeting’s outcomes. Following this discussion, we argue that GSS meeting quality also is a subjective construct. It depends on the perception of people; people that involved in the meeting or people that set the standards to which the quality of the meeting is compared. Normally there are three types of people in a meeting that have a perception on the meeting’s quality: the facilitator, the participants, and the initiator, i.e. problem owner. The primary indication for the quality of the meeting outcomes is provided by the initiator of the meeting, as he or she sets the goals for the meeting and is able to identify to what extent these goals have been achieved. However, the participants’ perception of the outcomes’ quality may be completely different from the initiator’s, see e.g. [Vreede and Wijk 1996]. Participants may hold a number of additional, personal goals that can even conflict with the initiator’s goals. The primary indication for the quality of the meeting process must be provided by the participants, as they “make” the process by working together as a team, supported by the facilitator. Again, the perceptions of the initiator and participants may vary. Participants may value the process for different reasons than the initiator. For example participants may like it that everyone had an equal opportunity to be heard,

whereas the initiator may perceive the focus of the process on the task to be more beneficial.

2.2 Meeting commitment The second construct that relates to the success of a GSS meeting is the commitment of the participants and initiator with the process used and outcomes achieved. It is not enough for a meeting to be a success if the people involved find the process and outcomes of satisfactory quality. The larger organizational problem solving process of which the GSS meeting is but a portion, usually is not yet finished after the meeting is over. The decisions and actions agreed upon in the meeting have to be carried out by the group members or have to be conveyed to others. In other words, the outcomes of the meeting have to be adopted or implemented by the group members, or further elaborated. To this end, the group’s commitment to the meeting outcomes, i.e. their acceptance, is crucial [Maier 1970 cf. Dickson et al. 1993]. Commitment to the outcomes of a GSS meeting is especially important in situations were the meeting participants originate from different parts of an organization or different organizations all together that do not have control over each other’s actions. If a person is committed to the outcomes of a GSS meeting, it means that he or she can identify himself or herself with these outcomes. The outcomes are subscribed to; they represent the thoughts, ideas, and preferences of the person. If the meeting task concerns a decision for which there is no correct answer (a so-called non-intellective or preference task [McGrath 1984]), the participants’ commitment to this decision can be measured by assessing the group’s consensus [Dickson et al. 1993]. It has been shown that reaching consensus implies greater acceptance and commitment to the decisions made [McGrath 1984 cf. Dickson et al. 1993], and greater acceptance facilitates the actual implementation of the decision [Ginzberg 1981 cf. Dickson et al. 1993]. However, group consensus cannot be meaningfully measured for all types of GSS meeting tasks, e.g. information generation tasks. It is, then, more practical to refer to the individual perceptions of the group members on their commitment to the meeting outcomes. If a person is committed to the process of a GSS meeting, it means that he or she subscribes to the agenda and procedures used in the meeting. The basic assumption of meeting process commitment is that if the participants buy into the process then they will be more inclined to buy into the outcomes that the process produces. Also, in order to keep participants involved, motivated, and focused to contribute to the meeting, it is undesirable that they spend (too) much time and effort on questioning the process instead of executing it. Commitment to meeting process is especially important if a the problem solving process of a group spreads out over more than one meeting. The group members’ commitment to a meeting’s process can be indirectly determined by inquiring whether they consider the meeting process appropriate for similar future group activities.

2.3 Good or bad?

In this section we argued that the extent to which a meeting can be considered a success is a subjective judgment that depends on the initiator’s and participants’ perception of the meeting in terms of the quality of the meeting’s process and outcomes and the commitment to the meeting’s process and outcomes. A bad meeting does not necessarily have a negative perception with respect to all four indicators, just as a good meeting does not require all four indicators to achieve a positive evaluation. A group might, for example, wrest a product out of a terrible process by brute force, and finish feeling satisfied with the product but angry at the inefficiency and degradation imposed by a process. Likewise, a team may stumble badly during a good process, and wind up producing a poor product. Or, a process might lead to a bad product, but the team might like the process because it meets other tangible or intangible goals, such as team cohesiveness, belonging, recognition, or remuneration. Therefore, we propose that apart from the four indicators, also a more situation independent criterion has to be used to determine whether a meeting can be considered good or bad. This criterion is the relative success of a GSS meeting. It is measured by comparing the perceptions on the four indicators of a particular meeting to those of a number of other meetings that were perceived successful. If the meeting scores significantly lower than the others, it can be considered to be a bad meeting.

3. Research approach and design To answer our research question we carried out a case study [Benbasat et al. 1987; Yin 1989] because “bad GSS meetings” is yet too complex a phenomenon to emulate in a constructed setting and to study with objective instruments, and because as far as we know there have been no similar studies investigating GSS meeting failures. To design our study we applied the case study design guidelines as defined by [Benbasat et al. 1987; Yin 1989]: Goal of the study, unit of analysis, number of sites, criteria for site selection, determination of data sources, and method of analysis. The goal of the study was stated in Section 1. The other guidelines are addressed below.

3.1 Unit of analysis The unit of analysis in our study is a face-to-face GSS meeting. We collect data with respect to individual meetings. Some of the meetings that are investigated are part of the same project, as shown in the next section. However, we are not investigating the success rate of the project, but the perceived failure of individual GSS meetings.

3.2 Number and selection of meetings The meetings that we analyzed in this study were selected from all meetings that the institution of the first author organized since 1993. We used the following four selection criteria: 1. The meeting data should be made available. 2. The meeting should involve real organizations.

3. The meeting should address a “real” issue, important to the initiator. 4. The meeting should not be perceived to be a great success by the initiator and/or participants. In total, 306 meetings were organized over the period 1993 until June 1996, 55 of which met the first three criteria. In all 55 meetings GroupSystems was used as a GSS to support a group of Dutch participants from organizations in the Netherlands. Based on feedback from the initiators and participants, we selected three cases that involved bad meetings: 1. The 3DOME case involved a consortium from academia and business that aimed to develop the functional specifications of a new Internet service directed at the market for 3D design. This service would enable artists, movie makers, architects, and so on, to buy or license existing 3D objects from an online 3D object library. To this end, six GSS meetings were held. During the first five meetings a functional specification was created, that was evaluated in the last meeting. For a more detailed description, see [Muller et al. 1995]. 2. The Vice case involved a multi disciplinary police team that was responsible for implementing a municipal policy with respect to prostitution. Working together on a daily basis, the team encountered a number of practical problems with this policy. In order to discuss these problems, to focus future team activities, and to sustain the work motivation of the team members, a single GSS meeting was organized. During this meeting, the current and future status of the municipal policy were put on the agenda. 3. The Climate case involved a diverse group of industry and government representatives that attended a conference on climate control issues hosted by the Ministry of Housing, Spatial Planning, and the Environment. Being increasingly an important issue, the Ministry wanted to create awareness and establish commitment to climate control. Two GSS meetings were organized for two different mixed industry and government groups to discuss the effects and practical applicability of various climate control measures.

3.3 Data sources As we argued in Section 2, our measurement of GSS meeting success is based on subjective judgments. Therefore, we decided to use both qualitative and quantitative data sources to create a rich representation of what went on in the meetings. We felt that too limited a data collection would have been achieved if we had used e.g. questionnaires or field notes exclusively. Qualitative data sources included semi-structured and open interviews and our own observations. Interviews were held immediately after or up to a couple of weeks after the meetings. Quantitative data sources included questionnaires completed by the participants immediately after each meeting and the electronic logs of the meetings. All participants were willing to answer our questions.

3.4 Method of Analysis We analyzed the selected GSS meetings both on their own and in contrast to other meetings that were perceived successful. For both analyses we looked at the constructs

discussed in Section 2. Also, we collect data on various other meeting aspects in order to be able to describe the context of the meetings. To guide the data collection on all GSS meetings involved, we used a model based on the well-known descriptive GSS research model by Nunamaker et al. [1991], see figure 1. This model consists of four elements: 1. Meeting setting characteristics, such as group size, group experience, organizational context, and task characteristics. 2. GSS characteristics, such as anonymity, group memory, media characteristics, and user friendliness. 3. GSS meeting process, including issues as facilitation, process structure, process quality, and commitment to process. 4. GSS meeting outcomes, including issues as quantity and quality of outcomes and commitment to outcomes.

Meeting setting characteristics GSS meeting process

GSS meeting outcomes

GSS characteristics

Figure 1 Research model for GSS meetings. As this model is used as a basis for a lot of GSS studies, using it will facilitate the comparison of our findings to those of other studies. Although this model does not have explanatory power in itself, it offers a meaningful way of organizing empirical data on the meetings involved and compare and contrast this data. We pursued explanations for certain measured perceptions by combining the data from the sources identified above. The aspects that were investigated with respect to each of the four model elements are introduced during the presentation of the meeting evaluation results in the next section.

4. RESULTS Below, the results of the study are presented with respect to each of the elements of the research model depicted in figure 1. One remark: only two meetings of the six 3DOME meetings (meetings 3 and 5) were included in the analysis. Although all six can be considered to be far from successful, these meetings had the lowest average score on the questionnaires overall. The other 3DOME meetings were not part of the benchmark meetings.

4.1 Meeting setting characteristics

Table 2 describes the context in which the meetings took place. The average group size was 8, compared to 11.24 for the 46 other meetings. The group cohesiveness was low for the 3DOME and Climate groups, as they met for the first time. The Vice group was working as a team for quite some time prior to the meeting. The composition of all groups was heterogeneous. The group members came from different organizations (3DOME and Climate), from different management levels (Climate), from different disciplines (3DOME and Vice), and pursued different organizational objectives (3DOME, Climate). The amount of knowledge on the meeting topic was varied for the 3DOME and Climate groups, as they included both experts and people with a more general understanding of the meeting issues. Working together in the same area, the Vice members had an equal understanding of the meetings issues. The task complexity (the perceived extent to which the task was easy to carry out) and task uncertainty (the perceived insight into what exactly had to be done) were high for all meetings because of the little understanding of possible solutions and effects of proposals for the meeting issues. Only in the Vice meeting the issues were somewhat clear, although the way to address them was not.

4.2 GSS characteristics With respect to the GSS characteristics, we first measured the participants’ perceptions on the ability to contribute anonymously to the meeting. We also asked the extent to which they found the GSS interesting to use and whether it supported the meeting activities and was user friendly. The results are described in table 3, using a 5 point scale, 5 being the most positive. The ‘n’ in the benchmark meetings describes the number of observations on each aspect. Note that not every meeting included an inquiry into the identified aspects.

Table 2 Meeting setting characteristics. Aspect 3DOME-3 3DOME-5 Vice Climate 1 Group size 5 9 8 10 Group cohesiveness low low high low Group composition heterogeneous heterogeneous heterogeneous heterogeneous Group knowledge level varied varied equal varied Task complexity high high medium high Task uncertainty high high high high 1 The Climate data in tables 2 to 6, describes both meetings in this case. The groups consisted of different participants engaging in the same activity.

Aspect Anonymity GSS is interesting to use GSS supports meeting activities GSS is user friendly

Table 3 GSS characteristics measurements. 3DOME-3 3DOME-5 Vice µ σ2 µ σ2 µ σ2 2.0 1.0 2.9 1.3 2.9 1.6 2.6 1.3 3.4 1.7 3.9 0.8 2.8 1.3 3.3 0.9 3.3 1.0 3.2 0.8 3.7 1.2 3.9 0.6

Climate µ σ2 3.0 1.2 4.1 1.1 3.7 1.1

µ 3.9 4.5 3.9 4.4

Benchmark σ2 n 1.0 253 0.7 254 0.8 85 0.7 268

The first remarkable observation can be made with respect to anonymity. Compared to the benchmark meetings, the participants’ perception of anonymity is very low, even negative in three of the four meetings. Participants appeared to find anonymous discussing to be too impersonal, too sterile. As one participant said: “I am very much in favor of direct person to person communication and discussion of issues” (3DOME-5). Also, anonymity was sometimes considered to harm the quality of information exchange. Some participants in the Climate meeting felt that interaction would become more serious if one had to identify one self. Others in the Vice meeting took remarks (too) personal: “Who submitted this???”, was often heard during this meeting. Finally, anonymity was sometimes felt to be an obstruction to clarifying issues and ideas, especially during oral discussions, especially in the 3DOME and Climate meetings. Regarding their interest in using the GSS, the participants were a bit less negative, especially in the Vice and Climate meetings. Only in the 3DOME-3 meeting, there was a negative perception of this aspect. It appeared that this had to do with the level of support the GSS was able to give the participants at that point in the project, said one participant “We have to design a system and define the project. I do not think that GroupSystems is a helpful tool at this stage.” The participants in the Climate meetings found the system especially useful for the identification of issues, not for detailed discussions: “The system is good for brainstorming, but not for decision making.” Finally, the user friendliness of the GSS was perceived somewhat lower than in the benchmark meetings, especially in the 3DOME-3 meeting. We expect that the low evaluation of this aspect can be mainly attributed to the frustration of the participants that the GSS did not seem to support their meeting activities very well. In other words, they found it unfriendly because it could not do what they needed it to do.

4.3 GSS meeting process With respect to the meeting process, we surveyed the participant with respect to (1) their overall satisfaction with the meeting process, (2) the way in which the available meeting time was spent, (3) the time available for clarifying discussions, (4) the influence of the facilitator on the meeting process, and (5) the extent to which communication in the group was encouraged. Also, we investigated the participants’ commitment to the meeting process by asking whether they felt the process could be repeated. The results are presented in table 4, using the same five point scale. Overall, the participants had a mildly positive perception of the meeting process. The difference with the benchmark meetings was not very large, except for the 3DOME-3 meeting. A couple of issues appeared to damp the overall process satisfaction, such as typing proficiency (said a Climate participant: “Typing proficiency is too determinant for the amount of information one can contribute.”), available time (said a number of other Climate participants: “We

Aspect

Table 4 GSS meeting process measurements. 3DOME-3 3DOME-5 Vice µ σ2 µ σ2 µ σ2

Quality Meeting process was satisfactory? Time spent well, focused? Enough room for discussion? Encourages communication? Facilitator had positive influence? Commitment Meeting can be repeated?

Aspect

µ

Benchmark σ2 n

3.0 2.6 3.6 2.6 4.0

0.7 1.1 1.1 0.9 0.7

3.6 3.2 3.9 2.7 3.9

0.7 0.8 1.1 1.2 0.6

3.4 3.6 3.0 2.4 3.3

0.9 1.0 1.4 1.1 0.8

3.4 3.2 2.4 2.5 3.7

0.9 1.0 1.1 1.2 0.9

4.0 3.6 3.3 3.7 3.7

0.9 1.0 1.1 1.0 0.9

404 304 182 171 265

3.0

0.7

2.8

1.1

4.3

1.0

-

-

4.2

0.9

342

Table 5 GSS meeting outcomes measurements. 3DOME-3 3DOME-5 Vice µ σ2 µ σ2 µ σ2

Quality Quality of results? Productivity compared to manual meeting? Commitment Outcomes represent your ideas

Climate µ σ2

Climate µ σ2

µ

Benchmark σ2 n

2.6 3.0

0.5 1.0

3.0 3.4

0.5 1.3

2.5 2.8

0.9 0.8

2.7 3.3

0.9 1.1

3.6 4.2

0.9 0.9

332 391

3.8

0.8

3.4

0.5

3.5

1.0

3.9

0.9

3.9

0.9

235

just need more time.”), and process structure (said a Vice participant: “The process was too chaotic. The participants have to learn how to discuss and react on each other’s comments.”). The perception of the way in which the meeting time was spent, deviated not very much from the benchmark meetings, except for the 3DOME-3 meeting. Similar to the discussion in the previous section, this meeting’s participants felt that the system and the process did not come at the right time during the project. A process aspect that is usually critically perceived concerns the available time for discussing intermediary results. This can be seen from the relatively low benchmark score. It is interesting to see that the two 3DOME meetings received a higher evaluation of this aspect. Most complaints came from the Climate meetings: “There was none!”, “Too little time to react to each other.”, and “Too little time for discussion.”. Feedback suggested that the participants felt that the agenda had too many items. In other words, the agenda was too full. With respect to the extent to which the GSS encourages communication, there is a big difference between the bad meetings and the benchmark meetings. In the bad meetings, this aspect was evaluated considerably worse. Feedback from the participants suggested that they just did not feel comfortable communicating via GroupSystems. This was evident in the Vice meeting, during which the group regularly pushed the keyboards aside and started discussing issues orally. The Climate meetings yielded a negative evaluation as well. “It was not really invoked,” said one participant. “We did not really get around to that. Too little time,” said another. Again, this perception was attributed to the full agenda. Although the meeting process as a whole and with respect to various aspects did not receive a positive evaluation, the participants did not seem to hold the facilitator responsible. The facilitator’s influence on the process was evaluated very positively, in the 3DOME meetings even higher than in the benchmark meetings. Only in the Climate meetings, some of the participants seemed to hold the facilitator responsible for the “overloaded” agenda. With respect to the commitment to the meeting process, two remarkable observations can be made. First, in 3DOME-3 the commitment to the process was equal to the overall satisfaction with the process, while in 3DOME-5 the process commitment was (a) much less than the overall satisfaction, and (b) less that the commitment in 3DOME-3. This is especially surprising considering the somewhat less negative perception of the 3DOME-5 participants with respect to the other process aspects. One possible explanation is that in 3DOME-3 the participants still expected that the GSS meeting process would prove to be valuable in later meetings. This expectation may have subdued by the time they finished 3DOME-5. Second, considering their perception on the overall satisfaction and other process aspects, the high process commitment of the Vice participants is unexpected. Feedback from the participants revealed that this perception was mainly based on the considerable number of other successful meetings that they had previously participated in. In other words, they kept faith in GSS and electronic meetings and hoped that it would go better next time.

4.4 GSS meeting outcomes For the meeting outcomes, we surveyed the participants’ perception on the quality of the results and the productivity of the meeting. Commitment to the outcomes was investigated by asking whether the outcomes represented the participants’ ideas. Table 5 presents the results, using the same five point scale. The quality of meeting outcomes is believed to be relative to the meeting goal. This was painfully confirmed in the questionnaires and interviews with the participants in the various meetings. Said one 3DOME-3 participant: “To me is it still insufficiently clear what the 3DOME group aims to achieve.” One Vice participant was even more straightforward: “How can I evaluate the quality of the outcomes if I don’t know what purpose they serve?” Another issue that seemed to negatively impact the perception on outcome quality relates to high quality ideas getting swamped into the abundance of information. This is especially illustrated by a remark from a Climate participant: “Really interesting ideas get lost. We need more time to discuss and select quality ideas.” The productivity of the bad meetings was evaluated considerably less than that of the benchmark meetings. This is remarkable in the sense that this aspect normally scores relatively high. Feedback from the participants suggested that one explanation was that is was not clear what the exact purpose of the meeting was and where the meeting would get them. In other words, generating dozens of ideas in a short period of time does not seem to impress the participants if the ideas do not clearly relate to the meeting goals. Finally, regarding the commitment to the outcomes it is interesting to note that in three of the four meetings, the commitment to the outcomes is much higher than the perception on the quality of the outcomes. Especially compared to the benchmark meetings. This suggests that although the participants felt that the outcomes represented their ideas, they considered the outcomes’ quality not as high as it should or could be.

5. Discussion The results of our study showed that the four meetings experienced various problems. Going back to the four indicators for the success of a GSS meeting, we see that compared to the benchmark meetings, the four meetings can indeed be considered to be far less successful, see table 6. This is consistent with the signals we received from the initiators and participants involved. Table 6 Summary of the success indicators. Indicator 3DOME-3 3DOME-5 Vice Climate Benchmark Process quality 3.0 3.6 3.4 3.4 4.0 Process commitment 3.0 2.8 4.3 4.2 Outcome quality 2.6 3.0 2.5 2.7 3.6 Outcome commitment 3.8 3.4 3.5 3.9 3.9 It appears that outcome quality is the strongest indicator of a bad meeting. It received a negative evaluation in all meetings, except for the 3DOME-5, which was just neutral. Also in the benchmark meetings, this indicator received the lowest evaluation. In this context, it

is remarkable that the outcome commitment in the bad meetings is so high, especially compared to the benchmark meetings. Maybe the participants felt they made quality contributions in contrast to the contributions of others. Or, they felt they could and did make the contributions they wanted, but there were too many distracting, non-functional contributions made by others. Nevertheless, it seems that the participants are hesitant to do it again, except for the Vice participants who did not loose faith in GSS meetings. (In the Climate meetings, process commitment was not measured with a questionnaire, but oral feedback suggested that process commitment was low in these meetings.) There are various explanations why the meetings went awry, both from the initiators and participants’ perspective. The 3DOME meetings took place in a multi-organizational setting in which the critical aspects were team building, learning, information exchange, and creativity. Over the course of the project, most participants displayed a negative perception of the GSS support, the process, and the outcomes of their meetings. For this, there are three likely explanations. First, based on the participants’ feedback, there appeared to be diverging perceptions of the meetings’ goals. Second, there was a huge knowledge gap among the group members. Knowledgeable participants may have had a hard time conveying all their knowledge to the group, while some participants may not even have expected or desired to discuss topics outside their field of expertise. In fact, the most knowledgeable participants were most dissatisfied with quality of the meeting process. They felt the GSS did not enable them to discuss issues, only to put them on the table. Nevertheless, the initiator indicated that a ‘traditional’ meeting would probably have been dominated by these same knowledgeable participants. Third, coming from various organizations, some participants may have felt obstructed in defending their interests. Thus, the democratizing effect of GSS (anonymity) may have hindered their personal negotiating capabilities. The Vice meeting took place during a time in which the team was going through a crisis situation with respect to their work situation. There was some conflict within the Vice group as well as between the group and other actors that were involved in the group’s work situation. The intention of the initiator was to discuss the work situation and find ways to improve it. However, the participants felt that the right actors were not invited. As a result, it appeared that the participants took a less serious approach to the meeting, engaged in some finger pointing, and spent less time on discussing the issues at hand constructively. The anonymity of the GSS may have “invited” them to display this behavior. In addition, the initiator aimed for team building, but some of the group members indicated that the opposite had happened. The Climate meetings took place in a setting in which the participants and initiator seemed to have diverging interests: The initiator wanted to get as much information as possible and show the participants as much of the technology as possible. This resulted in a full agenda. Too full according to the participants, who appeared to prefer focusing on less issues but in more depth. In addition, the GSS’ anonymity feature was perceived to obstruct quick enlightening, verbal discussions. As a result, the participants did not think that they got very much out of the meeting compared to what they, in their perception, could have.

6. CONCLUSIONS As the GSS research area is growing more mature, opportunities arise to study applications of GSS that failed and learn from those failures. In this paper, we compared a number of meetings that went awry with a number of benchmark meetings that were perceived as successful. We presented a number of indicators that helped us to determine the (un)successfulness of GSS meetings. These indicators do not explain why a meeting was bad. As (perceived) meeting outcomes are contingent upon a balance of process gains and losses, which vary in strength depending upon the situation [Nunamaker et al. 1991], we used multiple sources of situational evidence to get insight into possible explanations for the failure of the meetings concerned. Our study has three limitations that have to be taken into account when interpreting the results. First, only a small sample of bad meetings was investigated, mainly because bad GSS meetings are much less common than successful ones. Second, we used a subjective measure of meeting success. There is evidence that suggests that poor participant satisfaction does not necessarily imply having a bad meeting [Connolly et al. 1990]. However, it may be difficult to assess meeting success in a more objective manner, as standards for meeting success have to be developed and agreed upon. Finally, meeting success is a multi faceted notion. We choose to focus on perceived quality and perceived commitment to describe success. Obviously, there are other ways of describing meeting success, that may also bring about insight in why meetings worked out the way they did. The results of our study lead to a number of insights, lessons learnt, and recommendations for meeting design and facilitation that can be summarized as follows: • Outcome quality seems to be key to the perceived success of a meeting. Bad outcome quality may dampen process commitment i.e. the enthusiasm to participate in similar GSS meetings. • The results from the participants’ questionnaires in this study illustrate that electronic meeting participants are critical when the situation requires it. In other words, “overcooperative” attitudes of participants are not a likely explanation for the positive findings in other field studies. • It seems unwise to overload a meeting agenda and as a result have little or no time for (verbal) discussions of intermediary results. Although GSS may speed up rough information collection considerably, most participants seemed to feel oral discussion are key to exploring and gaining insight into the meeting outcomes. Not having enough discussion time seems to decrease participant satisfaction with the process dramatically. As discussions can take considerable time, the meeting process and GSS application may have to be more consciously tailored for discussion needs than for “quick and dirty” information collection. In other words, focus on fewer issues in more detail than on a large number of issues superficially. • It is often pointed out that the main advantage of GSS is the high productivity resulting from entering ideas in parallel; many more ideas can be generated per hour. However, the results of our analysis suggest that participants do not rate productivity on the number of ideas per hour, but also on the number of ideas per hour that served the meeting goals. In other words, participants first consider a GSS activity to be

productive if it is addressing the meeting’s goals, and second if it is addressing these goals by using available time efficiently. • Diverging perceptions or unfamiliarity with the meeting’s goals seems to be another important explanation for perceived dissatisfaction with meeting outcomes and process. It is, therefore, recommendable to consciously discuss and agree on the meeting’s goals with the whole group before starting a meeting. • Equal participation that may result from GSS use, is believed to enhance decision quality [Dennis et al. 1988]. However, our results suggest that this should not be a goal in itself, specifically in situations where there is great difference in knowledge among the meeting participants with respect to the issues at hand. This may result in an uneven motivation to actively contribute to a meeting. • Employing GSS technology in conflict situations should be handled with care. The results from the Vice meeting suggest that this requires an extraordinary stress on clearly briefing consciously planning the meeting and clearly briefing the participants beforehand. If no agreement can be reached on the objective and desired outcome of a meeting in such a situation, it may be better to cancel it. Also, in conflict situation make a conscious decision whether to allow anonymous communication or not; anonymity may invoke unwanted and uncontrollable finger pointing. • It is often claimed that preplanning GSS meetings is key to their success. Our study illustrates that preplanning is, however, no guarantee for success. The 3DOME and Climate meetings were carefully preplanned and yet they were not successful. Hence, preplanning may be required but it is not sufficient. This study illustrated a number of meetings that did not yield what was hoped for by the initiators and participants. Due to the descriptive nature of the study, we are only at the beginning of understanding why GSS meetings can go wrong. However, the results indicate that important lessons can be learnt from investigating unsuccessful meetings and comparing them with successful ones. For future research efforts, we propose that closer attention is paid to unraveling the relationships between factors in a GSS meeting that cause the participants to have a negative or positive perception of the overall meeting success and of the four indicators that we described. The results from our study may be used as a first basis on which to develop a “success scale” for GSS meetings, and a model that helps to measure and explain the “success” of a GSS meeting.

ACKNOWLEDGMENTS The authors want to acknowledge Sajda Qureshi and Bob Briggs for their comments on a draft of this paper, and Kees Wim van der Herik for his insights on the issue of meeting quality.

REFERENCES BENBASAT , I., D.K. GOLDSTEIN, M. M EAD (1987), The case research strategy in studies of information systems, MIS Quarterly, 11(3), 369-386. CONNOLLY, T., JESSUP , L.M., VALACICH, J.S. (1990), Effects of anonymity and evaluative tone on idea generation in computer-mediated groups, Management Science, 36, 689-703. DENNIS, A.R. (1994), Electronic Support for Large Groups, Journal of Organizational Computing, 4(2), 177-197. DENNIS, A.R., GEORGE , J.F., JESSUP , L.M., NUNAMAKER, J.F. JR., AND VOGEL (1988), D.R., Information Technology to Support Electronic Meetings, MIS Quarterly, 12(4), 591-624. DICKSON, G.W., J.E. LEE-PARTRIDGE , L.H. ROBINSON (1993), Exploring Modes of Facilitative Support for GDSS Technology, MIS Quarterly, 17(2), 173-194. EMERY, C.C. JR. (1994), Reengineering the Requirements Process for Executive Support System Development Using an Electronic Meeting System, Proceedings of the 27th Hawaiian International Conference on System Sciences, Volume IV, 41-50, IEEE Computer Society Press, Los Alamitos CA. GEORGE , J.F., NUNAMAKER, J.F. JR., VALACICH, J.S. (1992), Electronic Meeting Systems as Innovation, Information & Management, 22(3), 187-195. GINZBERG, M.A., Key Recurrent Issues in the MIS Implementation Process, MIS Quarterly, 5(2), 47-60, 1981. KRCMAR, H., LEWE , H., SCHWABE , G. (1994), Empirical CATeam Research in Meetings, Proceedings of the 27th Hawaiian International Conference on System Sciences, Volume IV, 31-40, IEEE Computer Society Press, Los Alamitos CA. M AIER, N.R.F. (1970), Problem Solving and Creativity in Individuals and Groups, Brooks/Cole Publishing Company, Belmont, CA, 1970. M CGRATH, J.E. (1984), Groups: Interaction and Performance, Prentice Hall, Inc., Englewood Cliffs, NJ. M ULLER, P.C., ENGELEN, J.M.L. VAN, TERLOUW , P., VREEDE, G.J. DE (1995), Computer Supported Cooperative Design of a New Internet Service, working paper, University of Groningen. NUNAMAKER, J.F. JR., A PPLEGATE, L.M., KONSYNSKI, B.R. (1987), Facilitating Group Creativity: Experience with a Group Decision Support System, Journal of MIS, 3(4), 5-19. NUNAMAKER, J.F., BRIGGS, R.O., M ITTLEMAN , D. (1995), Ten Years of Lessons Learned with Group Support Systems, in: D. Coleman, R. Khanna (eds), Groupware: Technologies and Applications, 146-193, Prentice Hall, Saddle River, NJ.

NUNAMAKER, J.F. JR., A.R. DENNIS, J.S. VALACICH, D.R. VOGEL, J.F. GEORGE (1991), Electronic Meeting Systems to Support Group Work, Communications of the ACM, 34(7), 40-61. NUNAMAKER, J.F. JR., D.R. VOGEL, A. HEMINGER, B. M ARTZ, R. GROHOWSKI, C. M CGOFF (1989), Experiences at IBM with Group Support Systems: A Field Study, Decision Support Systems, 5(2), 183-196. PERVAN, G.P. (1994), A Case for More Case Study Research in Group Support Systems, in: Glasson, B.C., I.T. Hawryszkiewycz, B.A. Underwood, R.A. Weber (eds.), Business Process Re-engineering: Information Systems Opportunities and Challenges, Elsevier Science Publishers (North-Holland), 469-479. PINSONNEAULT , A., K.L. KRAEMER (1989), The impact of technological support on groups: An assessment of the empirical research, Decision Support Systems, 5(2), 197-216. POST , B.Q. (1992), Building the Business Case for Group Support Technology, Proceedings of the 25th Hawaiian International Conference on Systems Sciences, IEEE Computer Society Press, Los Alamitos, CA. TYRAN, C.K., A.R. DENNIS, D.R. VOGEL, J.F. NUNAMAKER JR. (1992), The Application of Electronic Meeting Technology to Support Strategic Management, MIS Quarterly, 16(3), 313-334. VELD, J. IN ‘T , Analyse van Organisatieproblemen, Stenfert Kroese, 1992. (In Dutch) VICIAN, C., G. DESANCTIS, M.S. POOLE, B.M. JACKSON (1992), Using Group Technologies to Support the Design of "Lights Out" computing Systems, in: Kendall, K.E. et al. (eds.), The Impact of Computer Supported Technologies on Information Systems Development, Elsevier Science Publishers, North-Holland. VREEDE, G.J. DE, W IJK, W.B. VAN (1997), A Field Study into the Organizational Application of Group Support Systems, Proceedings of the ACM SIGCPR 1997 Conference, San Francisco. YIN, R.K. (1989), Case Study Research: Design and Methods (revised edition), Sage Publications, Newbury Park. ZIGURS, I. (1993), Methodological and Measurement Issues in Group Support Systems Research, in: Jessup, L.M., J.S. Valacich (eds), Group Support Systems - New Perspectives, Macmillan Publishing Company, New York.