A Decade of Database Conferences: A Look Inside The Program ...

A Decade of Database Conferences: A Look Inside The Program Committees Sherif Sakr1 and Mohammad Alomari2 1

National ICT Australia University of New South Wales, Australia [email protected] 2 School of Information Technologies University of Sydney, Australia [email protected]

Abstract. Database management technology has played a vital role in facilitating key advancements of the information technology field. Database researchers - and computer scientists in general - consider prestigious conferences as their favorite and effective tools for presenting their original research work and for getting good publicity. With the main aim of retaining the high quality and the prestige of these conference, program committee members plays the major role of evaluating the submitted papers and deciding which submissions are to be included in the conference programs. In this paper, we study the program committees of four top-tier and prestigious database conferences (SIGMOD, VLDB, ICDE, EDBT) over a period of 10 years (2001 - 2010). We report about the growth in the number of program committee members in comparison to the size of the research community in the last decade. We also analyze the rate of change in the membership of the committees of the different editions of these conferences. Finally, we report about the major contributing scholars in the committees of these conferences as a mean of acknowledging their impact in the community.

1

Introduction

Database management technology has played a vital role in the advancements of the information technology field. It has matured after extensive research and development efforts and has very successfully created a large market and solutions in different business domains. Database researchers - and computer scientists in general - consider top-tier and prestigious conferences as their favorite tools for presenting original research work in contrast to the general case of many other scientific disciplines where journal papers are routinely considered to be superior than those of conference papers [2, 10]. For example, it has been shown that the two top database conferences (SIGMOD and VLDB) receive many more citations per paper than the two top database journals (TODS and VLDB J.) [16]. In practice, the general culture in the computer science community is that journal papers are used to present deeper versions of papers that already have been

presented at conferences [2, 10]. One of the main reasons behind this is that the review process of journal papers are usually very long. The turnaround time (the interval between the submission date of a manuscript and the date of having the editorial decision) for conferences is often less than a third of that of journals [18]. Since the field of computer science research tends to be fast-paced, conferences provide a great chance for timestamping the latest research findings earlier which allows the knowledge to be publicly shared more rapidly. In general, we are witnessing a continuous growth in the database field [17]. That is mainly due to the continuous introduction of new application domains (e.g. web applications, mobile applications, cloud computing, sensor networks) with varying features and requirements on their data management aspects. In practice, data has become mobile, flexible, mirrored in a variety of logical and physical forms, evolving, being concurrently modified and replicated, dynamically generated and later reintegrated in very large repositories for further analysis and processing [9]. Therefore, there are many more researchers are entering the field to tackle these challenges, more research papers are being submitted to the top-tier conferences for getting publicity and hence more effort in evaluating the quality of these research contributions is exerted [5]. In principle, the members of the program committee of any conference play the main role of evaluating the submitted papers and deciding which submissions to accept for inclusion in the conference programs. Therefore, on the one hand, their decisions play the main role of retaining the high quality and the prestige of these conferences. On the other hand, these decisions reward the research scholars of the accepted papers such that they gain good publicity of their research contributions [16]. Thus, membership of the program committee of the top-tier conferences is considered as one of the main factors for judging the impact of the scholars in the research community. In a previous work [17], we studied the publications of the major database research venues over a period of 10 years (2001 - 2010). In this study, we reported about: the scholars with the highest number of publications in these venues and the most successful co-authorship relationships in the database research community in the last decade. In addition, we analyzed the growth in the number of research publications and the size of the database research community. Complementary to our previous work, in the current study, we take another angle for analyzing the database research community by focusing on studying the program committees of four top-tier and prestigious database conferences (SIGMOD, VLDB, ICDE, EDBT) [3, 16, 17] over the last decade (2001 - 2010). In particular, in this paper we make the following main contributions: – We analyze the growth rate on the size of the program committees of the major core database technology conferences in comparison to the size of contributing members in the core database research community and the number of core database research publications of these conferences in the last decade. – In order to spot on the diversity of the influencing community members, we analyze the overlap in the membership of the program committee of the different editions of the same conference in addition to the overlap in the

Table 1. Total Numbers of Distinct PC Members in The Major Database Conferences In The Last Decade

Total Distinct PC Members

VLDB SIGMOD ICDE EDBT Combined 688 451 845 349 1308

membership of the program committee of the different conferences in the same years. – We report about the scholars with the highest number of membership in the committee of the major core database technology conferences separately and when combined. The input data of this study has been manually collected, cleaned and verified from the official proceedings of the studied conferences1 . We made the detailed results of our study accessible on the web2 .

2

Statistics on Conferences

As we have previously mentioned, the members of the program committee of any conference are the key group of people who are responsible for selecting the submitted research contributions for the inclusion in the conference programs. Hence, they play the main role of retaining the high quality and the prestige of the top-tier conferences [20]. In this study, we focus on analyzing the membership of the program committees of the four major core database technology conferences (VLDB, SIGMOD, ICDE, EDBT) over the last decade (2001 - 2010). In this paper, we present the most important results of our study. For full detailed results, we refer the reader to the web page of this study. Some key remarks about the results of our study are given as follows: – Table 1 shows the number of distinct scholars that have been participating in each of the major database conferences separately and when combined over the last decade. The numbers show that the ICDE conference had the largest community membership with 845 scholars and the EDBT conference had the smallest program committee community with 349 members. However, it should be noted that while the VLDB, SIGMOD and ICDE conferences had 10 editions in the last decade, the EDBT conference had only 6 editions (2002, 2004, 2006, 2008, 2009, 2010). The EDBT conference used to be a bi-annual conference before turning to be an annual conference starting from 2008. In addition, the number of presented papers in the editions of the EDBT conference is the smallest in comparison to the corresponding editions of the other conferences in the corresponding years. For example, the results of [17] show that the number of published papers in the EDBT editions of the last decade is on average half of the number of publications 1 2

A set of Python scripts has been prepared to help achieving this task http://www.cse.unsw.edu.au/∼ssakr/PCStatistics/index.html

0.26 0.24 0.22 0.20

Year 2

0.18 0.16

2001 2002 2003 2004 2005 2006 2007 2008 2009

0.28 0.26 0.24 0.22 0.20 0.18

Year 2

0.28

0.14 0.12

0.16 0.14 0.12

0.10

0.10

0.08

0.08

0.06

0.06

0.04

0.04

0.02 2001

2002

2003

2004

2005

2006

2007

2008

2009

0.02 2001

2010

2001 2002 2003 2004 2005 2006 2007 2008 2009

2002

2003

2004

2005

(a) VLDB

0.20 0.18 0.16

Year 2

0.14 0.12

2007

2008

2009

2010

2008

2009

2010

(b) SIGMOD

2001 2002 2003 2004 2005 2006 2007 2008 2009

0.30 0.28 0.26 0.24 0.22 0.20

Year 2

0.22

2006

Year 1

Year 1

0.10

0.18 0.16 0.14

2001 2002 2003 2004 2005 2006 2007 2008 2009

0.12 0.08

0.10 0.08

0.06

0.06

0.04 0.02 2001

0.04 2002

2003

2004

2005

2006

Year 1

(c) ICDE

2007

2008

2009

2010

0.02 2001

2002

2003

2004

2005

2006

2007

Year 1

(d) EDBT

Fig. 1. Percentage of overlap between the different editions of the major database conferences

for the corresponding VLDB editions. When combined, 1308 distinct scholars have participated in the program committee of the major core database technology conferences over the last decade. This number gives an indication on the reasonably big size of the contributing scholars in the database research community over the last decade. – In general, it is quite unhealthy to have a fixed or slightly different list of members in the program committees for the different venues (different editions of the same conference or different conferences) of a research community [8, 20]. Having large numbers of the same scholars as frequent members in the program committees of different venues may have intended or unintended negative effects in the fairness of evaluating the research contributions or in the quality and variability of the conference programs. For example, a specific scholar might be not convinced with the importance of a specific research idea or with the quality of work of a specific research group. This scholar might abandon the visibility of this research idea or the work of this group in all the venues where he is involved in the program committee while

70

VLDB SIGMOD ICDE EDBT

Percentage of PC Members (%)

60

50

40

30

20

10

0 0

1

2

3

4

5

6

7

8

9

Number of Participations

Fig. 2. The total frequency of participation in the program committees of the major database conferences over the last decade

other scholars in the same research community may have different views in the evaluation of that research idea or the work of that group [6, 13, 15]. Therefore, it is quite healthy for any conference to have a dynamic membership for the program committees of its different editions. It is also quite healthy for any research community to have a large pool of trustable scholars from which the members of the program committees of the top-tier conferences can be recruited. This large pool of candidates serves the key role of easing the job of the chairs and organizers of these conferences in forming the committee that can achieve the goals of such prestigious conferences. Figure 1 illustrates the percentage of overlap between the different editions of the major core database technology conferences over the last decade. The Xaxis and the Y-axis represent the editions of the conference. The percentage of overlap between two editions of a conference is computed by dividing the total count of the common scholars in the program committees of these two editions over the total count of the distinct scholars in the program committees of the same two editions. On average, the percentage of overlap between the different editions of the VLDB conference is 14%, of the ICDE conference is 12%, of the EDBT conference is 10% and of the SIGMOD conference is 9%. The highest percentage of overlap between two different editions of the EDBT conference is 30% (2008 and 2009), of the SIGMOD conference is 26% (2006 and 2007), of the VLDB conference is 25% (2008 and 2009) and of the ICDE conference is 21% (2009 and 2010). The percentage of overlap between the two years 2001 (beginning of the decade) and 2010 (end of the

Table 2. Yearly Number of Distinct PC Members in The Major Database Conferences In The Last Decade 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Distinct PC Members 203 258 236 316 281 311 399 444 435 491

decade) of the VLDB conference is 8%, of the SIGMOD conference is 5%, of the EDBT conference is 5% (the first edition held in 2002) and of the ICDE conference is 3%. – Figure 2 illustrates the percentages for the frequency of scholars’ participation in the program committees of the major core database technology conferences. Out of the six editions of the EDBT conference in the last decade, 69% of the total distinct program committee members (349 in Table 1) have participated only once. Out of the ten editions of the ICDE, SIGMOD and VLDB conferences in the last decade, the percentages of the program committee members who have participated only once are 59%, 56% and 40% respectively. For the EDBT conference, only one scholar (Tore Risch) has participated in the program committees of the six editions of the decade with a percentage of participation (100%). In addition, nine scholars have participated four times (67%). For the EDBT conference, only one scholar (S. Sudarshan) has participated in nine editions with a percentage of participation (90%) while ten scholars have participated seven times (70%). For the SIGMOD conference, Minos Garofalakis has participated in seven program committees (70%), H. Jagadish and Kyuseok Shim have participated six times (60%) while twelve scholars have participated 5 times (50%). For the ICDE conference, four scholars (Christian Jensen, Goetz Graefe, Sunil Prabhakar and Kevin Chang) have participated eight times (80%) while other seven scholars have participated seven times (70%). – Table 2 shows the yearly number of distinct scholars that have been participating in all of the major database conferences in combined. The numbers show that core database technology conference had 201 distinct scholar in their committees in the 2001 editions. This number had been gradually increasing until it reached its maximum of 491 by end of the decade (250% increase). Figure 3 illustrates the percentages for the yearly frequency of scholars’ participation in the program committees of the major core database technology conferences. In 2001, 89% of the total distinct program committee members (203 in Table 1) have participated in only one of the major database conference while the 2010 editions had 72% of the total distinct program committee members participating only once. The 2001 editions of the major core database technology conferences had 10% of the the total distinct program committee members participating in two different conference while the corresponding percentage of 2010 has increased to 23%. In 2001, Daniela Florescu and Matthias Jarke have been participating in all editions of the major database technology conferences of the year. Other scholars had the same full participation of the editions of the following years as follows. 2002:

1 2 3 4

0.9

Percentage of PC Members (%)

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Year

Fig. 3. The yearly frequency of participation in the program committees of the major core database technology conferences over the last decade

Daniel Keim; 2003: Masaru Kitsuregawa; 2004: Timos Sellis; 2005: Bernhard Mitschang, Jeff Naughton, Johann-Christoph Freytag and Sam Madden; 2006: Kyuseok Shim and Minos Garofalakis; 2008: Elisa Bertino, Karl Aberer, Martin Kersten and Sihem Amer Yahia; 2009: Christian Jensen, Fabio Casati and Yufei Tao; 2010: Angela Bonifati and Wen-Syan LI. 2007 had the largest number of scholars (13 scholars) with full participation in the program committees of all conferences in the year: Dimitris Papadias, Elena Ferrari, Ihab Ilyas, Jignesh M. Patel, Juliana Freire, Magda Balazinska, Meral Ozsoyoglou, Nick Koudas, Paolo Atzeni, Phillip Gibbons, Raghu Ramakrishnan, Rajeev Rastogi and Sunil Prabhakar. The results of Figure 3 reflect the high volatility and dynamics in the yearly constitution of the program committees of the database conferences. On average, 80% of the program committee members participate in one conference per year, 15% participate in two conferences per year while only 5% who may participate more than twice! – The topics of the database field is continuously growing. Therefore, there are more researchers who are entering the research community and more research papers are being published [4]. One of the main reasons behind the increase in the number of publications in the database community is the continuous introduction of new research challenges which is relevant to the scope of the community. For example, XML has started to be introduced as a hot research topic for the database research community in the early

Number of PC Members

300

SIGMOD VLDB ICDE EDBT

250

200

150

100

50

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Year Fig. 4. Number of program committee members of the major core database technology conferences over the last decade

period of the last decade. Moro et al [14] referenced a list of more than 100 publications in a survey paper that provides an overview of some of the work that have been done in different aspects for XML data management. Recently, the topic of large scale data management on cloud computing and parallel data processing (e.g. MapReduce) have been introduced and they attract a lot of interest from the database research community [1]. As a consequence, a new series of research conferences, the ACM Symposium on Cloud Computing, has been started in 2010 [11]. The results of our previous study [17] have shown that the number of research publications and unique authors in the major database technology conferences have on average nearly doubled in number over the last decade. Figure 4 illustrates the number of program committee members for the different editions of the major core database technology conferences over the last decade. The results of this figure is compatible with the results of our previous study as it shows that the number of program committee members of these conferences have doubled on average as well.

3

Statistics on Scholars

Research is a competitive endeavor. Research scholars usually have multiple goals to achieve and it is therefore reasonable that their impact must be judged by multiple criteria. For example, one way of measuring the impact of research scholars in the community is their number of publications in top-tier research

0

(c) ICDE Kyu-Yong Whang

1

Thomas Seidl

2

Krithi Ramamritham

3

Nick Koudas

4

Walid Aref

5

Beng Chin Ooi

(a) VLDB

Martin Kersten

Hank Korth

Jiawei Han

Gustavo Alonso

Jeff Naughton

Wenfei Fan

Dimitrios Gunopulos

Dimitris Papadias

Alon Halevy

Martin Kersten

Divesh Srivastava

Christian Jensen

0 Alfons Kemper

1

Masaru Kitsuregawa

2

H. Jagadish

3

Kyuseok Shim

4

Minos Garofalakis

5

Number of Participations in PC

6

Paolo Atzeni

6

Number of Participarions in PC

Beng Chin Ooi

Gerhard Weikum

Jignesh M. Patel

Daniel Keim

Masaru Kitsuregawa

Krithi Ramamritham

Timos Sellis

Tiziana Catarci

Christian Jensen

Carlo Zaniolo

S. Sudarshan

Number of Participations in PC 7

Tore Risch

Minos Garofalakis

Elisa Bertino

Beng Chin Ooi

Nick Koudas

Jian Pei

Jiawei Han

Alexandros Labrinidis

Kevin Chang

Sunil Prabhakar

Goetz Graefe

Christian Jensen

Number of Participations in PC 9

8 7

6

5

4

3

2

1

0

(b) SIGMOD

8

7 6

5

4

3

2

1

0

(d) EDBT

Fig. 5. Top participating scholars in the program committees for each of major database conferences over the last decade

venues and the number of citations they receive, i.e. how frequently their publications are referenced by other publications (e.g. h-index1 [12], g-index2 [7]). Rahm et al. [16] presented a study where they considered ranking the database research scholars based on their citation counts. In our previous work [17], we presented a study where we focused on measuring the number of publications in top-tier publication venues as one of the main indicators to evaluate the impact of a research scholar in the community and the quality of his research production. In the current study, we focus on measuring the participation on the program committees of the top-tier conferences as another important factor in measuring the impact of research scholars in the community. Figure 5 illustrates the top participating scholars in the program committees for the major core database technology conferences over the last decade: VLDB 5(a), SIGMOD 5(b), ICDE 5(c) and EDBT 5(d). The results of these figures show that two scholars have appeared in the top participating scholars of the program committees for three major database conferences namely: Christian Jensen (VLDB, SIGMOD, ICDE) and Beng Chin Ooi (VLDB, ICDE, EDBT). Six scholars have appeared in the top participating scholars of the program committees for two major database conferences namely: Minos Garofalakis (SIGMOD, ICDE), Jiawei Han (SIGMOD, ICDE), Martin Kersten (SIGMOD, EDBT), Masaru Kitsuregawa (VLDB, EDBT), Nick Koudas (ICDE, EDBT) and Krithi Ramamritham (VLDB, EDBT). Looking back to the results of our previous study [17] where we presented the list of the scholars with the highest number of publications (top publishers) in the major core database technology conferences over the last decade, we found some overlaps as follows. Nick Koudas appears as one of the top publishers in the four conferences (VLDB, SIGMOD, ICDE, EDBT) in addition to his appearance in the list of scholar with the highest number of participation in the program committees of two conferences (ICDE, EDBT). Divesh Srivastava is a top publisher in 3 conferences (SIGMOD, ICDE, EDBT) and a top program committee participant in SIGMOD. Jiawei Han is a top publisher in 3 conferences (VLDB, SIGMOD, ICDE) and a top program committee participant in SIGMOD and ICDE. H. Jagadish is a top publisher in 3 conferences (VLDB, SIGMOD, EDBT) and a top program committee participant in SIGMOD. Beng Chin Ooi is a top publisher in 3 conferences (VLDB, SIGMOD, ICDE) and a top program committee participant in ICDE and EDBT. Dimitris Papadias is a top publisher in 2 conferences (SIGMOD, ICDE) and a top program committee participant in SIGMOD. Jeffrey F. Naughton is a top publisher in 2 conferences (SIGMOD, VLDB) and a top program committee participant in SIGMOD. Elisa Bertino is a top publisher in EDBT and a top program committee participant in ICDE. Wenfei Fan is a top publisher in SIGMOD and a top program committee 1

2

A scientist has index h if h of his Np papers have at least h citations each, and the other (Np − h) papers have at most h citations each Given a set of articles ranked in decreasing order of the number of citations that they received, the g-index is the (unique) largest number such that the top g articles received (together) at least g 2 citations

Table 3. Aggregate list of scholars with the highest number of participation in the program committees of the major core database technology conferences Scholar PC Participation h-index. g-index. Current Affiliation Christian Jensen 23 46 87 Aarhus University, Denmark Masaru Kitsuregawa 21 25 46 University of Tokyo, Japan Beng Chin Ooi 21 42 69 National University of Singapore, Singapore Minos Garofalakis 20 20 39 Technical University of Crete, Greece Martin Kersten 20 27 60 CWI, Netherlands Nick Koudas 20 41 78 University of Toronto, Canada Goetz Graefe 19 36 79 HP Labs, USA Kyuseok Shim 19 33 94 Seoul National University, Korea Timos Sellis 18 14 37 National Technical University of Athene, Greece Jignesh M. Patel 18 29 61 University of Wisconsin, USA Krithi Ramamritham 17 56 100 IIT Bombay, India Dimitris Papadias 17 43 79 HKUST, Hong Kong Tore Risch 17 23 51 Uppsala University, Sweden Karl Aberer 17 39 71 EPFL, Switzerland Paolo Atzeni 17 25 50 University of Roma, Italy H. Jagadish 17 58 112 University of Michigan, USA Alfons Kemper 16 32 53 Technical University of Munich, Germany Tiziana Catarci 16 22 43 University of Roma, Italy Walid Aref 16 35 67 Purdue University, USA Divesh Srivastava 16 54 98 AT&T, USA Surajit Chaudhuri 15 49 110 Microsoft Research, USA Kyu-Young Whang 15 21 36 KAIST, Korea Daniel Keim 15 38 93 Konstanz University, Germany Johannes Gehrke 15 46 122 Cornell University, USA Carlo Zaniolo 15 46 74 UCLA, USA Jiawei Han 15 93 210 University of Illinois at Urbana-Champaig, USA Gerhard Weikum 15 49 84 Max-Planck-Institut fr Informatik, Germany Elisa Bertino 15 60 104 Purdue University, USA

participant in VLDB. Jian Pei is a top publisher in EDBT and a top program committee participant in ICDE. Table 3 shows an aggregate list of scholars with the highest number of participation in the program committees of the major core database technology conferences over the last decade. The record of each scholar represents his h-index, his g-index 3 and the current affiliation. Christian Jensen comes alone on the top of this list with a total of 23 memberships out of a total of 36 venues (64%) in the program committees of the major core database technology conferences over the last decade. Masaru Kitsuregawa and Beng Chin Ooi had 21 memberships (58%). Minos Garofalakis, Martin Kersten and Nick Koudas had 20 memberships (55%). The reported affiliations in Table 3 show that 36% (10 out of 28) of the scholars with the highest number of memberships in the program committees of the major core database technology conferences are USA-based scholars. In another back-to-back comparison on the results of our previous study [17] where we presented a list of the scholars with the highest number of publications in the database research venues over the last decade, we identified the scholars who appear in both of that list and the current list of the scholars with the highest number of participation in the program committees of the top-tier database con3

The reported h-index and g-index information is based on the Publish or Perish software (http://www.harzing.com/pop.htm)

ferences. We indicated those scholars (12 out of 28 - 43%) with the italic font in Table 3. This percentage that represents the joint membership of the two lists is expected to be higher if we increase the length of both lists.

4

Conclusions

The database management technology has played a vital role in the advancements of the information technology field. The database research community is a big community which has been matured over the last three decades after conducting extensive research and development efforts and have very successfully created a large market and solutions for the database technology in different business domains. In a previous study [17], we analyzed the growth rate on the size of contributing database research community and the number of database research publications in the top-tier database conference over the last decade. We also ranked the scholars according to their number of publications in these prestigious venues. In the current study, we analyzed the database research community from another angle. In particular, we presented a detailed study for the program committees of four major core database technology conferences (SIGMOD, VLDB, ICDE, EDBT) over the last decade. The results of the two studies are compatible in indicating that the database research community size and the number of research publications have nearly doubled through the last decade. The results of the current study indicate that the program committees of the major core database technology conferences are quite dynamic and healthy. First, the yearly number of distinct scholars who are participating in the program committees of all the major database conferences is reasonably large. Second, the percentages of overlap between the program committees of the top venues are considerably low. Third, the percentages of scholars with very frequent memberships in the program committees are also very low. These three properties give a very good chance of pumping new research ideas into the community, a reasonably open environment of evaluating the different research directions and a good possibility for accepting new members who are conducting high quality research. The results of the current study also show a considerable overlap between the the list of scholars with the highest number of memberships in the program committees of the major core database technology conference and the list of scholars with the highest number of publications in these conferences. In general, the quality of a conference is correlated with that of its program committee member. Therefore, this overlap is to be expected as the scholars with good track records in the top-tier conferences are naturally good candidates for being acknowledged to serve as members of the program committees and play an effective role in retaining the high quality and the prestige of these venues [8]. As a future work, we plan to further study the database research community over the last decade from other two different angles: 1) The collaboration pat-

terns between the research institutes and their impact [19]. 2) The trend of the most contributing countries in the research community.

References 1. Divyakant Agrawal, Sudipto Das, and Amr El Abbadi. Big Data and Cloud Computing: New Wine or just New Bottles? PVLDB, 3(2):1647–1648, 2010. 2. Anastassia Ailamaki, Laura M. Haas, H. V. Jagadish, David Maier, M. Tamer ¨ Ozsu, and Marianne Winslett. Time for Our Field to Grow Up. PVLDB, 3(2):1658, 2010. 3. David Aumller and Erhard Rahm. Affiliation Analysis of Database Publications. SIGMOD Record, 40(1):26–31, 2011. 4. Philip A. Bernstein, David J. DeWitt, Andreas Heuer, Zachary G. Ives, Christian S. ¨ Jensen, Holger Meyer, M. Tamer Ozsu, Richard T. Snodgrass, Kyu-Young Whang, and Jennifer Widom. Database Publication Practices. In VLDB, pages 1241–1246, 2005. 5. Ken Birman and Fred B. Schneider. Program committee overload in systems. Commun. ACM, 52(5):34–37, 2009. 6. Fabio Casati, Maurizio Marchese, Azzurra Ragone, and Matteo Turrini. Is peer review any good? a quantitative analysis of peer review. Technical Report DISI09-045, University of Trento, Italy, August 2009. 7. Leo Egghe. Theory and practise of the g-index. Scientometrics, 69(1):131–152, 2006. 8. Ergin Elmacioglu and Dongwon Lee. Oracle, where shall I submit my papers? Commun. ACM, 52(2):115–118, 2009. 9. Daniela Florescu. Database Research for the Current Millennium. In ICDE, page 866, 2004. 10. Lance Fortnow. Viewpoint - Time for computer science to grow up. Commun. ACM, 52(8):33–35, 2009. 11. Joseph M. Hellerstein, Surajit Chaudhuri, and Mendel Rosenblum, editors. Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC 2010, Indianapolis, Indiana, USA, June 10-11, 2010. ACM, 2010. 12. Jorge Hirsch. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102(46):16569–16572, 2005. 13. Stefano Mizzaro. Quality Control in Scholarly Publishing: A New Proposal. American Society for Information Science and Technology, 54(11):989–1005, 2003. 14. Mirella M. Moro, Vanessa P. Braganholo, Carina F. Dorneles, Denio Duarte, Renata de Matos Galante, and Ronaldo dos Santos Mello. XML: some papers in a haystack. SIGMOD Record, 38(2):29–34, 2009. 15. Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, and Maurizio Marchese. A Quantitative Analysis of Peer Review. In Proceedings of the 13th Conference of the International Society for Scientometrics and Informetrics (ISSI), 2011. 16. Erhard Rahm and Andreas Thor. Citation analysis of database publications. SIGMOD Record, 34(4):48–53, 2005. 17. Sherif Sakr and Mohammad Alomari. A decade of database research publications: a look inside. Scientometrics, 88(2):521–533. 18. Richard T. Snodgrass. Journal Relevance. SIGMOD Record, 32(3):11–15, 2003. 19. Jiann wien Hsu and Ding wei Huang. Correlation between impact and collaboration. Scientometrics, 86(2):317–324, 2011.

20. Ziming Zhuang, Ergin Elmacioglu, Dongwon Lee, and C. Lee Giles. Measuring conference quality by mining program committee characteristics. In JCDL, pages 225–234, 2007.