A Toolset for Web-based Peer Review of Scientific

System Demonstration: A Toolset for Web-based Peer Review of Scientific Testimony Mark Deckert, Abram Stern, Warren Sack Digital Arts & New Media Program Digital Arts Research Center University of California 1156 High Street Santa Cruz, CA 95064

{mdeckert,aphid,wsack}@ucsc.edu ABSTRACT By connecting a set of web-based tools to scientific video in government, we demonstrate the potential for peer reviewers to interact within a highly configurable context of academic literature, citation networks and time-linked commentary. A new way of viewing and understanding information rich media in a web context emerges.

Categories and Subject Descriptors H.5.3 [Information Interfaces and Presentation]: Group and Organization Interfaces - asynchronous interaction, collaborative computing, computer-supported cooperative work, web-based interaction.

Keywords Video, Open Source Software, Social Web, Web 2.0, Online Information Systems, Multimedia Information Systems

1.

INTRODUCTION

In most government bodies there are long-standing “interfaces” that exist between elected officials and the experts of science, technology and engineering. These “interfaces” are places and institutions where science and technology meet with citizens, society, and government. Our research seeks to develop computational technologies to strengthen these “interfaces” and thereby improve the architecture of existing, democratic institutions. One example of such an interface is the President’s Council of Advisors on Science and Technology (PCAST) which, as noted by co-chair Harold Varmus, has a tradition of covering important and controversial topics. Fig. 1 shows a video of Professor Robert Paarlberg of Wellesley College giving a talk to PCAST. In this talk, Paarlberg draws a clear picture of the current controversy between what he terms the “Green Revolution Model” and the sustainable, organic,

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Dg.o’11, June 12-15, 2011, College, USA. Copyright 2011 ACM 978-1-4503-0762-8/11/06. . . $10.00.

agroecological model [9, 1]. Paarlberg claims that the solution to hunger in Africa is to deploy modern agricultural techniques of monocropping, genetically modified seeds, and petroleum based fertilizers and pesticides [12]. Paarlberg is quite eloquent in his presentation and makes an excellent case for what he terms “Science Intensive Farming.” What Paarlberg fails to mention is the unstated assumption that the external inputs which the “Green Revolution Model” relies upon will continue to remain available and economical [7, 6]. There is a large camp of scientists who back a concept called “Peak Oil,” which indicates that fossil fuels, the external input that modern agriculture is largely dependent on, will become increasingly more expensive and less available in the coming years [8, 2, 11]. By creating a set of web-based tools that build a network of academic information around the video, we allow a peer reviewer to place the video and slides into their larger academic context, helping to map out controversies such as the one Paarlberg is taking a position on. In addition, a commenting system which is connected to the reddit.com online community allows to reviewers to interact and discuss specific sections of testimony. The minimizeable, scalable, and movable tools include a video viewer, academic literature finder/scraper, inline PDF viewer, commenting panel, and slide viewer. A notable piece of related work, Videolyzer, is being developed at Georgia Tech [5]. A work-in-progress demonstration is available at [4].

2.

SPEAKER AWARE TOOLS

The context awareness of our toolset is facilitated by a project called popcorn.js which is an open source event framework that works with the native html5 video tag [10]. By integrating a speaker list generated from video transcripts, each tool can provide context aware information like academic papers by the current speaker and comment timestamps. The “citations” panel, which contains papers by the current speaker, is connected to custom scraping code which gathers available papers from Google Scholar. These papers can then be viewed inline through the Google document viewer panel by simply clicking the paper in question, putting relevant academic literature side-by-side with video and slides. This allows the reviewer to understand how the speakers comments connect with what has been said in science previously.

3.

CONTEXTUAL COMMENTING

Reddit is a social news website that has a strong discussion community and a highly diverse set of user-creatable topic based “subreddits.” Aside from its proven user interface [3], Reddit has a number of features which make it an ideal deliberative discussion platform: • Voting: every story (or piece of testimony) and comment can be voted upon by each user. Voting information is used to control the display of stories and comments. • Subreddits: content is organized into subreddits. Membership to subreddits can be controlled through both user subscriptions and moderation of “private” subreddits. • Karma: voting scores for both a users comments and submissions are tallied and provide a record of past performance. One of the key tools is a comment box which connects the user to a discussion on Reddit. This comment box tracks when the user started typing and when s/he clicked “submit” and includes this information in a link back to the relevant video along with an optional quote of the relevant section of transcript within the reddit comment it generates. By conducting peer review in the context of a reddit discussion on the web, certain changes to the common one-way process of written evaluation are realized. First, by subjecting the comments of experts to voting from their peers, they are no longer treated as infallible and instead develop a sort of “batting average” over time in the form of their cumulative “Karma” score. Second, by letting experts see reviews from other experts and respond to them over time in the context of information that can be pointed at via the internet, complex arguments and carefully researched refutations become possible. Finally, while privacy may be a necessity with regards to research funding, in a venue such as PCAST, where the president is ostensibly getting his scientific information, openness should be a given. Reddit’s comment interface is designed for efficient consumption through its sorting methods and thread control, making for effective openness.

4.

CONFIGURATION

Users are able to move, resize and minimize the various panels. Instead of watching a talking head that contains little information, the video panel can be minimized, allowing the user to listen in while watching the slides, reading relevant literature or viewing comments on the testimony.

5.

CONCLUSION

The scientific and technical information provided to the U.S. Government through expert testimony has a profound affect on policy, funding, and legislation. By building a webbased technology to facilitate peer review of scientific testimony, we seek to improve the quality of information the government receives from the Executive Branch’s Federal Advisory Committees, specifically, the President’s Council of Advisors on Science and Technology. If the project is successful, we will have a set of concrete, implemented proposals to show how the Executive Office of the President

Figure 1: PCAST video with several contextual tools and reddit commentary and the communities of science, technology, and engineering could be better integrated using tools and techniques of information technology.

6.

REFERENCES

[1] M. Altieri. The ecological role of biodiversity in agroecosystems. Agriculture, Ecosystems & Environment, 74(1-3):19–31, 1999. [2] U. Bardi. Peak oil: The four stages of a new idea. Energy, 34(3):323–326, 2009. [3] A. Clement, T. Costantino, D. Kurtz, and M. Tissenbaum. Participatory design and web 2.0: the case of PIPWatch, the collaborative privacy toolbar. In Proceedings of the Tenth Anniversary Conference on Participatory Design 2008, pages 51–60. Indiana University, 2008. [4] Demo. http://redd.it/gpzlz or http://metaviddemo01.ucsc.edu/citations/, March 2011. [5] N. Diakopoulos, S. Goldenberg, and I. Essa. Videolyzer: quality analysis of online informational video for bloggers and journalists. In Proceedings of the 27th international conference on Human factors in computing systems, pages 799–808. ACM, 2009. [6] M. Giampietro. Sustainability and technological development in agriculture. BioScience, 44(10):pp. 677–689, 1994. [7] M. GiamPietro and D. Pimentel. Energy efficiency: Assessing the interaction between humans and their environment. Ecological Economics, 4(2):117 – 144, 1991. [8] R. Hirsch, R. Bezdek, and R. Wendling. Peaking of world oil production. Impacts, Mitigation & Risk Management, 2005. [9] R. Lal. Soils and sustainable agriculture: A review. Sustainable Agriculture, pages 15–23, 2009. [10] Mozilla. http://popcornjs.org, March 2011. [11] P. Newman, T. Beatley, and H. Boyer. Resilient cities: responding to peak oil and climate change. Island Pr, 2009. [12] R. Paarlberg. Starved for science: How biotechnology is being kept out of Africa. Harvard Univ Pr, 2008.