A Review of CRIWG Research Pedro Antunes1, José A. Pino2 1
University of Lisbon, Department of Informatics of the Faculty of Sciences, Campo Grande, Lisbon, Portugal
[email protected] 2 Universidad de Chile, Department of Computer Science, Avenida Blanco Encalada 2120, Tercer Piso, Santiago, CP 837- 0459, Chile
[email protected]
Abstract. This paper presents a meta-analysis of the CRIWG conference. The study is organized in three main sections: bibliometric analysis, analysis of references and subject analysis. The bibliometric analysis indicates that CRIWG is significantly above the average citation index of similar papers published in LNCS. The analysis of references shows a significant dependence on ACM papers and very low cross-referencing between CRIWG papers. The subject analysis reveals that CRIWG slightly favors positivist evaluations, although almost half of the papers do not present any type of evaluation. We conclude this study with a discussion over strengths, weaknesses, opportunities and threats. Key words: Meta-analysis, CRIWG review.
1. Introduction This paper is intended for the 16th edition of the CRIWG1 conference. As a round number (24), it may trigger an appropriate time to reflect on the past contributions of this conference and the outlook for the future. The authors would then like to present some data gathered from the previous proceedings and elaborate some analysis and discussion. We expect this to be the starting point of a rich and controversial interchange of viewpoints during the conference itself. CRIWG started in Lisbon, Portugal, in September 1995. It was initially thought to be a meeting to exchange research approaches in the field of Groupware for a few groups. Instead of establishing an informal gathering, a decision was made to make it scientifically valid through the commonly accepted peer-reviewing process, using anonymous submissions and reviews, and circulating the papers in proceedings distributed by international scientific publishers. The conference has kept this tradition up to now. CRIWG has some unique features when compared with other conferences in the field. It has just one track, with full papers and work-in-progress ones. The 1
Which now is an acronym for the Collaboration Researchers International Working Group on Groupware.
Antunes, P. and J. Pino (2010) A Review of CRIWG Research. Groupware: Design, Implementation, and Use. 16th CRIWG Conference on Collaboration and Technology, Maastricht, The Netherlands, vol. 6257, pp. 1-15. Heidelberg, Springer-Verlag.
presentation of the papers provides ample time for discussion, giving thus authors very valuable feedback. The conference also encourages social interaction among participants. Finally, the conference has been organized in interesting places around the world, but this has not been obstacle to sustained presence by participants to all sessions. We will present some data extracted from the proceedings, we will analyze it and try to make suggestions for the future of the conference. The analysis, of course, is the authors’ responsibility and it does not represent an official statement from the conference committees.
2. Methodology Most of this review is based on information provided by Thomson Reuters ISI Conference Proceedings Citation Index (ISI, for short). We analyzed all CRIWG papers published between 2000 and 2008. Our intention was to cover the set of proceedings published by LNCS and IEEE between 1999 and 2009. However, for some unknown reason the joint SPIRE/CRIWG 1999 conference was not found in ISI; and the 2009 papers were not yet available in December 2009, when the data was gathered. Gathering the list of papers from ISI was not completely straightforward, requiring combined searches using the CRIWG acronym and the Groupware keyword, plus manual inspection to remove spurious references to other journal and conference papers. The consolidated data set used in most of our analysis consists of 246 papers. The review was separated in three main goals: bibliometric analysis, analysis of references and subject analysis. The bibliometric information, such as the Hirsch index (h) [1], was automatically produced by ISI and gives a summative assessment of CRIWG. Our main purpose to analyze the CRIWG references to other papers was to understand how CRIWG views and positions itself relatively to other research fields. The references were automatically obtained from ISI, exported to Endnote and exported again to Excel, which was then used to discover the main referenced papers, authors and sources. It should be emphasized the references exported by ISI present some shortcomings. For instance, they do not identify all authors. Furthermore, conferences and journals are formatted with multiple short names. A considerable effort was done to normalize the data and obtain the consolidated information reported in section 4. The subject analysis follows a method that has been adopted by similar reviews (e.g., [2-4]). The method uses qualitative data analysis techniques to code the data set using multiple tags [5]. Unlike some reviews that use a predefined set of tags, we adopted a grounded approach [6] where the keywords emerge as the analysis progresses [5]. The main categories considered during the grounded coding were: research objective, research topic and type of evaluation. Two coding rounds were performed to ensure consistency.
The coding process was applied to the 246 paper abstracts obtained through ISI. This might be viewed as a controversial decision, since coding could instead be applied to the full paper bodies. The basis for our decision to only code abstracts was: (1) the abstracts should be considered accurate and concise summaries of the authors’ research, done by the persons most fit to accomplish them, i.e. the authors themselves; (2) abstracts have an implicit structure that is totally aligned with the coding categories mentioned above; and (3) this structure is typically checked and enforced by peer reviewers. During the coding process we confirmed the type of information we were reviewing was available in the abstracts, which made the access to the remaining information unnecessary. We nevertheless point out the search for more fine-grained information, including for instance reviewing which specific techniques, tools and algorithms were researched, would necessarily mandate a full body analysis. We finally note the trend lines shown in the paper are all polynomial functions. The R2 appearing near the trend lines was automatically calculated using Apple’s Numbers.
3. Bibliometric Analysis The total number of CRIWG papers considered in the bibliometric analysis is 246. The total number of citations to these papers is 336, which gives an average citation per paper of 1.37. The obtained h-index is 6. The distribution of citations per year is shown in Figure 1. After an expected ramp up of citation activity from 2002 to 2005, the number of citations has stabilized since 2006 at an average of 48 cites/year.
Fig. 1. Distribution of citations per year
Fig. 2. Distribution of citations per paper (number of citations in horizontal axis)
As shown in Figure 2, 132 papers (53%) have not received any citation. This indicates that CRIWG, although having a selection of good papers, has to improve the selection process.
3.1 Comparison with other conferences The information conveyed above might be difficult to analyze without a frame of reference. In order to create such reference we tried to compare CRIWG with other conferences related with CRIWG and also reported by ISI. The following conferences were selected: COOPIS (Conference on Cooperative Information Systems, now part of OTM - On The Move Confederated Conferences), WET-ICE (Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises), DSV (International Workshop on Design, Specification and Verification of Interactive Systems) and CONTEXT (International and Interdisciplinary Conference on Modeling and Using Context). Furthermore, we also contrasted the CRIWG bibliometrics with a broad set of papers collected from ISI using a search for “LNCS” and “Conference”. Table 1. Comparison with other conferences Name Years Nr. papers Times cited Av. citation
COOPIS 01-07 379 703 1.85
DSV 98-07 163 237 1.45
CRIWG 01-08 246 336 1.37
CONTEXT 00-07 194 235 1.21
WET-ICE 96-06 524 274 0.52
LNCS 90-09 8746 9103 1.04
The results shown in Table 1 indicate that CRIWG, in terms of quality measured by citation indexes, is above average when compared with the other selected conferences (1.37 against an average of 1.28) and significantly above the average citation index of a large collection of papers published in LNCS. 3.2 Top 5 cited papers The top 5 cited papers are shown in Table 2. Table 2. Top 5 cited papers (number of citations on the left) 16 9 9 8 7
Rosa MGP, Borges MRS, Santoro FM (2003) A conceptual framework for analyzing the use of context in groupware, 9th International Workshop on Groupware, 2003, Autrans, France, LNCS, vol. 2806, p. 300-313 Haake JM, Schummer T, Haake A, et al. (2003) Two-level tailoring support for CSCL, 9th International Workshop on Groupware, Autrans, France, LNCS, vol. 2806, p. 74-81 Collazos CA, Guerrero LA, Pino JA, et al. (2002) Evaluating collaborative learning processes, 8th International Workshop on Groupware, La Serena, Chile, LNCS, vol. 2440, p. 203-221 Neyem A, Ochoa SF, Pino JA (2006) Supporting mobile collaboration with serviceoriented mobile units, 12th International Workshop on Groupware, Medina del Campo, Spain, LNCS, vol. 4154, p. 228-245 Moran AL, Favela J, Martinez-Enriquez AM, et al. (2002) Before getting there: Potential and actual collaboration, 8th International Workshop on Groupware, La Serena, Chile, LNCS, vol. 2440, p. 147-167
3.3 Top 10 authors Table 3 presents the list of authors according to the number of papers published in the proceedings. Table 3. Top 10 authors (number of papers on the left) 19 17 15 14 14
Pino JA Antunes P Borges MRS Favela J Collazos C
12 Ochoa S 12 Guerrero L 10 Baloian N 9 Lukosch S 7 Vreede GJ 7 Fuks H
3.4 Country distribution A total of 29 countries have been represented in the conference series. Figure 3 indicates the internationalization ratio has slightly increased between 2000 and 2004 and slightly decreased in 2007 and 2008. Table 4. Country distribution 44 Brazil 34 Chile 30 Germany 27 Portugal 24 Spain 22 France 22 USA 21 Mexico 13 Netherlands 8 Colombia 6 Argentina 6 Canada 4 China 4 Korea
3 3 3 2 2 2 2 1 1 1 1 1 1 1 1
Costa Rica Finland Japan England Greece Norway Taiwan Australia Austria Belgium Luxembourg Romania Scotland Sweden Switzerland
Fig. 3. Internationalization (countries/year)
3.5 Collaborative research As mentioned in the introduction, CRIWG aimed to promote the participants’ socialization. This should allow researchers to meet potential partners for future projects. These researchers will probably return to the CRIWG conference to present the new joint results. A reasonable hypothesis then is to assume the proportion of papers presented by researchers from two or more institutions would increase in time.
Fig. 4. Percentage of collaborations by year
Figure 4 shows the evolution of the number of collaborations for the whole CRIWG period (1995-2009), represented as a percentage of the number of papers published in each year. This data was manually gathered from the proceedings. It shows the proportion of papers presented by two or more research groups slightly growing in time (groups from different departments of the same university were considered as just one group). Of course, the contribution from the conference may not be the only reason for this increase. Other factors may influence this result, like appearance of grants supporting research from more than one country, etc. However, in our own personal experience, CRIWG helped us finding research partners. 3.6 Special issues in ISI Journals Table 5 summarizes the special issues in ISI Journals published with extended versions of papers presented at CRIWG conferences. The total number of papers is 46 (18.7% of all papers published by the Proceedings in the 2000-2008 period). The number of citations is 99. The average citation index is 2.15, which is significantly higher than the one obtained by the proceedings. There is also in press one special issue of the Group Decision and Negotiation journal containing extended versions of papers presented at the 2008 conference. Table 5. Special issues in ISI journals Journal name Int. J. of Cooperative Information Systems Int. J. of Human-Computer Studies Journal of Universal Computer Science Computing and Informatics Group Decision and Negotiation (*) Multimedia Tools and Applications (*) The 2008 issue is in press.
No. of issues 2 1 2 1 2 1
No. of papers published 10 5 10 5 12 4
Besides the special issues in ISI-indexed journals, there have been special issues in other journals, as reported in Table 6. Table 6. Special issues in other journals Journal name Int. J. of Computer App. and Technology International Journal of e-Collaboration e-Service Journal Journal of CLEI
No. of issues 1 1 1 1
No. of papers published 9 4 5 1
4. Analysis of References The analysis of references gives a good indication of how CRIWG perceives its research community. The references were automatically obtained from ISI using the set of 246 papers published between 2000 and 2008. References to technical documentation and web sites were manually removed from the data set. Table 7. Analysis of references Total number of cited papers: 4524 Average number of references per paper: 19.15 (stdev: 8.6) References to ACM papers: 653 (14%) ACM Transactions: 98 Communications of ACM: 134 Proceedings of ACM: 338 Proceedings of ACM CSCW: 187 Proceedings of ACM CHI: 61 References to LNCS papers: 229 (5%) References to IEEE papers: 196 (4.3%) References to Thesis: 112 (2.5%) References to HICSS papers: 90 (2%) References to CRIWG papers: 84 (1.9%) References to ECSCW papers: 35 (0.8%) References to LNAI papers: 14 (0.3%)
The results summarized in Table 7 indicate a significant dependence on ACM papers. The data also indicate very low cross-referencing between CRIWG papers. In the one hand, this shows there is very low inbreeding in the CRIWG community, but in the other hand it also points out a lack of community building. The number of references to thesis might be interpreted as indicating a focus on exploratory rather than summative research. 4.1 Most cited references The references most cited by CRIWG are shown in Table 8.
Table 8. Most cited references (number of citations on the left) 15 12 12 10 10 8 8 7 7 7 6 6 6
ELLIS CA, 1991, COMMUN ACM, V34, P38 BRIGGS RO, 2003, J MANAGE INFORM SYST, V19, P31 GAMMA E, 1995, DESIGN PATTERNS ELEM FJERMESTAD J, 1999, J MANAGEMENT INFORMA, V15, P7 NUNAMAKER JF, 1991, COMMUN ACM, V34, P40 ROSEMAN M, 1996, ACM T COMPUTER HUMAN, V3, P66 SCHUCKMANN C, 1996, P ACM 1996 C COMP SU, P30 DOURISH P, 1992, P ACM C COMP SUPP CO, P107 GUERRERO LA, 2001, INFORM SOFTWARE TECH, V43, P457 MALONE TW, 1994, ACM COMPUT SURV, V26, P87 DEVREEDE GJ, 2006, INT J COMPUTER APPL, V25, P140 GRUDIN J, 1994, COMMUN ACM, V37, P92 CHABERT A, 1998, COMMUN ACM, V41, P69
4.2 Most referenced first authors The researchers most cited by CRIWG are shown in Table 9. Only the first authors are taken into account, since ISI does not produce the full reference records. Table 9. Most referenced authors (number of citations on the left) 50 46 42 35 34 34 26 23
C. Gutwin R. Briggs S. Greenberg C. Ellis P. Dourish G. Kolfschoten P. Dillenbourg D. Pinelle
23 22 22 21 21 21 20 20
L. Guerrero M. Borges G. Stahl J. Nunamaker J. Haake T. Malone M. Roseman B. Myers
4.3 Main referenced research areas The seven categories shown in Table 10 emerged after systematic data analysis of the data set using manual search. These categories may again be categorized by order of importance, where CSCW (Computer Supported Cooperative Work) emerges as the most important research area, followed by Decision Support Systems (DSS) and Computer Supported Collaborative Learning (CSCL). It should however be noted that these categories cover a small percentage of the data set (more precisely, 25.6%). The remaining 3/4 of data could not be associated to a category. Table 10. Main research areas referenced by CRIWG 1 – About 10% 2 – About 5% 3 – About 1-2%
413 (9.1%) 238 (5.3%) 230 (5%) 75 (1.7%) 58 (1.2%) 105 (2.3%) 45 (1%)
CSCW DSS CSCL Software engineering Human Computer Interaction Business Process Management Artificial Intelligence
5. Subject analysis 5.1 Research objectives
Fig. 5. Distribution of papers by research objective
As shown in Figure 5, nine different research objectives were codified during the data analysis. The most prevailing research objective is prototype development, followed by design and theory/model building. Interestingly, although a major focus is on prototyping and designing collaborative systems and tools, CRIWG has not focused on building guidelines for developers. 5.2 Research topics The data set was manually coded with the purpose to uncover the main research topics. The coding process was done in multiple rounds for consistency. Thirty-two research topics emerged after consolidation. Figures 5 and 6 show the most and least significant topics, respectively. Learning stands out as the most prevailing research topic, considered by 51 papers (20%).
Fig. 6. Research topics (most significant)
Fig. 7. Research topics (least significant)
We emphasize the coding process was performed at various conceptual levels before the main research topics were completely settled. Learning, for instance, is a
code that actually encompasses the following sub-codes: classroom activities/ composition, online materials/courses, teaching, reflection, infrastructure/platform, scenarios, processes, discussion forums, modeling knowledge, interdependencies/ mediation, performance, attitudes, dialectic reasoning, participation and selforganization. Awareness, in turn, includes the following sub-codes: semantic awareness, situation awareness, document awareness, awareness model, group awareness, presence awareness, workspace awareness, and change awareness. After the coding process, the research topics were aggregated in main research areas. Five main areas emerged this way, covering the foundations of collaboration support, application areas, group decision-making, system development issues, and communities. Table 11 shows that collaboration support is the main research concern expressed by CRIWG (35% of papers). It is also interesting to note that infrastructural issues related with collaboration support have not received sizeable attention from CRIWG researchers (10% of papers). Table 11. Main research areas 128 (35%) 98 (27%) 70 (19%) 36 (10%) 31 (8.5%)
Collaboration support (awareness, coordination, context, tailoring, flexibility, notification, performance, shared workspaces, communication, editing, document management) Applications areas (learning, workflow, handhelds, healthcare) Decision making (conflict, decision making, emergency management, group support systems, negotiation, knowledge management, collaboration engineering) Systems development (integration, peer-to-peer, reusability, XML, patterns, software engineering) Communities (virtual communities, communities of practice, virtual worlds, online communities)
5.3 Evaluation methods
47.5% - No evaluation 25.2% - Positivist (laboratory, survey, empirical, sample application, formative, simulation, analytic) 16.3% - Interpretivist (case study, case illustration) Fig. 8. Distribution of papers per evaluation method
Almost half of the papers (118 papers, 47.5%) do not present any type of evaluation. Of the adopted evaluation methods, case studies are the most prevalent one. CRIWG balances positivist and interpretivist evaluations, with a slight advance given to positivist evaluations. Interpretivism addresses questions of meaning while positivism addresses questions of cause and effect [7].
Trying to study the 118 papers without any evaluation, we found they are distributed according to Table 12. Table 12. Subject addressed by papers without any evaluation (number of papers on the right) Propose a framework/architecture Describe a prototype Concern design issues Address implementation issues (e.g., flexibility, synchronization, heterogeneity, interoperability) Propose a model Concern workflow Concern decision making Concern knowledge management Concern a methodology Concern software engineering
39 29 18 16 12 3 3 2 2 3
The large number of papers with no evaluation perhaps is related to one category of papers the conference has: the Work-in-Progress class. These papers, short in length, are supposed to present initial ideas and thus may tend to be speculative, reporting initial stages of research projects. We could not confirm this hypothesis with the available data, since the proceedings do not distinguish the CRIWG type of paper. 5.4 Trends in research objectives
Fig. 9. Framework
Fig. 10. Architecture
Fig. 11. Theory/models
Fig. 12. Prototype
Fig. 13. Design
The trend lines shown in Figures 9-13 account the different research objectives according to year of publication. The framework and architecture topics exhibit trend lines showing that, after an increase of importance, the pursuit of these research lines is in decline (note also that R2 is high). The other categories do not exhibit any definite trend. 5.5 Trends in research topics
Fig. 14. Support
Fig. 15. Application areas
Fig. 16. Decision making
Fig. 17. Application development
Fig. 18. Communities
The trends shown in Figures 14-18 indicate a clearly increasing interest over the communities theme (comprising issues such as virtual communities, communities of practice, virtual worlds and online communities). The CRIWG interest over application development (including various issues such as integration, peer-to-peer, reusability, XML and patterns) seems to persist as rather low when compared with the other categories. 5.6 Trend in the use of the “groupware” term As mentioned in the Introduction, the initial goal of the Conference was to ease collaboration of researchers working in the Groupware field. We have now the feeling that groupware is being phased out in favor of other terms, including “collaborative technology”, “group support”, or more specific keywords. Figure 19 shows that the percentage of occurrences of the keyword in the proceedings of each year is declining, thus supporting our hypothesis. The percentages were calculated using Word Counter 2.10 to scan PDF files, count words and convert PDF to text; and then using TextEdit to count how many times the groupware keyword appears in the text. To avoid counting occurrences in the papers’ references sections, we manually removed them from the PDF files.
Fig. 19. Number of “groupware” words in the conference proceedings per mil
6. SWAT Analysis A traditional Strengths/Weaknesses/Opportunities/Threats analysis is provided in Table 1. The entries are by no means definitive truths, but rather the authors’ interpretations of the data elements presented above; they are subject to contrast with other viewpoints.
7. Conclusions Several conclusions are already embedded in the SWAT analysis. However, the authors would like to emphasize a few of them. The CRIWG conference is a well-established conference with a positive SWAT chart. Of course, weaknesses should be faced and maybe they can be corrected. Although the average number of cites received is higher than the average LNCS conference, it certainly would be desirable to raise that figure. Table 13. SWAT analysis of CRIWG. Strengths
Weaknesses
- Average citation index within range of other reputed conferences and above LNCS average. - More than 18% papers were extended and republished in ISI journals. - Variety of evaluation methods. - Significant focus on prototyping development and design issues. - Balance between positivistic and interpretivist evaluation methods. - High number of collaborations (currently at 57%). - Long-term agreement to publish proceedings in LNCS series. - ISI visibility.
- Lack of cohesive research topics and trends. - Small focus in communities of practice (although it is increasing). - Small number of references to CRIWG papers. - Dependence on ACM conferences. - About half of the papers do not have any impact. - Small overall h-index. - About half of the papers do not address evaluation.
Opportunities
Threats
- The largest number of papers comes from Brazil. This country also has a strong local conference on CSCW. A possible synergy? - Emergent application areas outside the core of the conference could be used to attract interesting papers. - Some journals already know the conference and accept special issues; it could be possible to associate the conference to a specific prestigious journal.
- Dependence on a small set of authors. - Great challenges in the field already worked out (e.g., awareness). - A relatively small number of accepted papers may not make economically viable to organize the conference in the future.
Perhaps the opportunities can be taken. The authors particularly suggest the CRIWG Steering Committee should discuss with the Brazilian members the advantages and disadvantages of organizing the conference more often in Brazil. For example, the CRIWG conference could eventually merge with the Brazilian CSCW local event; the conference could be held in Brazil and abroad in alternating years. This is especially relevant since the 2011 conference will be held in Brazil. Another suggestion to the CRIWG conference would be to explore possibilities of long-term association with prestigious journals in the area. Finally, the incorporation of new promising research fields in areas of interest of the conference could also be considered. Acknowledgements. This paper was supported by the Portuguese Foundation for Science and Technology (PTDC/EIA/102875/2006) and Fondecyt (Proj. 1080352).
References 1. Hirsch, J.: An Index to Quantify an Individual's Scientific Research Output. Proceedings of the National Academy of Sciences of the United States of America 102, 16569-16572 (2005) 2. Pervan, G.P.: A Review of Research in Group Support Systems: Leaders, Approaches and Directions. Decision Support Systems 23, 149-159 (1998) 3. Pervan, G.P., Atkinson, D.J.: GDSS Research: An Overview and Historical Analysis. Group Decision and Negotiation 4, 475-483 (1995) 4. Serenko, A., Bontis, N.: Meta-Review of Knowledge Management and Intellectual Capital Literature: Citation Impact and Research Productivity Rankings. Knowledge and Process Management 11, 185-198 (2004) 5. Miles, M., Huberman, A.: Qualitative Data Analysis. SAGE, Thousand Oaks, California (1994) 6. Glaser, B., Strauss, A.: The Discovery of Grounded Theory Strategies for Qualitative Research. Aldine, Chicago (1967) 7. Davison, R.: An Action Research Perspective of Group Support Systems: How to Improve Meetings in Hong Kong. PhD Dissertation. Department of Information Systems, City University of Hong Kong (1998)
