Supporting Browsing-Specific Information Needs: Introducing the Citation-Sensitive In-Browser Summariser Stephen Wana , C´ecile Parisa, Robert Daleb b Centre
a ICT Centre, CSIRO, Sydney, Australia for Language Technology, Faculty of Science, Macquarie University, Australia
Abstract Practitioners and researchers need to stay up-to-date with the latest advances in their fields, but the continual growth in the amount of literature available makes this task increasingly difficult. In this article, we describe the CitationSensitive In-Browser Summariser (CSIBS), a new research tool to help manage the literature browsing task. The design of CSIBS was based on a user requirements analysis which identified the information needs that biomedical researchers commonly encounter when browsing through academic literature. CSIBS supports researchers in their browsing tasks by presenting both a generic and a tailored preview about a citation at the point at which they encounter it. This information is aimed at helping the reader determine whether or not to invest the time in exploring the cited article further, thus alleviating information overload. Feedback from biomedical researchers indicates that CSIBS facilitates this relevancy judgement task, and that the interface and previews are easy to use and informative. Key words: Text Summarisation, User-centric Design 1. Introduction Researchers regularly browse through large repositories of online academic literature, often to update their existing knowledge or to quickly familiarise themselves with a new topic. However, with a near exponential growth in available publications and scientific material, the task of browsing through and staying up-to-date with academic literature is a particularly challenging one, especially for the increasingly time-poor researcher. Given that a significant amount of research time is dedicated to the task of gathering and reviewing information (Hersh et al. [4] suggests as much as 18%), there is a need to develop new tools to support researchers in this task. The work presented in this paper caters to situations in which the researcher, who is already reading an article, wishes to browse through the related cited material. Typically, we expect to find additional important information by this browsing task. Indeed, in a survey of biomedical researchers, we found that two thirds of researchers regularly browsed scientific literature in this manner. In this scenario, as one reads a document (the citing document), a large number of citations may be encountered. Each citation points to a cited document and is embedded within a citation context. Some tools can already help with the straightforward task of automatically downloading cited documents. For example, reference managers Email addresses:
[email protected] (Stephen Wan),
[email protected] (C´ ecile Paris),
[email protected] (Robert Dale) Preprint submitted to Elsevier
such as JabRef 1 and Sente 2 can perform this ‘blind downloading’. However, this does not address the question of document relevancy at the time of reading the citing document. When relying on just these tools, the original reading context explaining why the document is potentially relevant is typically lost. We propose that, to make relevance judgements, information about the cited document is needed most when the citation context is first encountered. That is the moment at which the researcher’s curiosity in the cited document is first triggered. Given the potentially large number of possibly interesting citations, this results in the following browsing-specific problem: how can we help a user quickly determine whether the cited document is indeed worth downloading, perhaps paying for, and reading? To address this problem, we present the Citation-Sensitive In-Browser Summariser (CSIBS), a tool developed as part of the Elsevier Grand Challenge.3 CSIBS supports readers in deciding which cited documents they should read. We designed the tool in consultation with potential users, researchers in the biomedical domain, from the intended demographic so that CSIBS could facilitate their day-today research activities. Over the course of the semi-final and final rounds, feedback from users was used to focus research and development, the outcome of which is the focus of this paper. We first began with a user requirements analysis, which 1 jabref.sourceforge.net 2 www.thirdstreetsoftware.com 3 www.elseviergrandchallenge.com
November 10, 2009
revealed two prominent tasks: appraising the cited document, and relating the citation context to specific content in the cited document. Ultimately, in performing these two actions, the reader is trying to decide whether or not to read the cited document in full. It is this overall task that CSIBS strives to facilitate. CSIBS provides researchers with information which is useful at the point at which citations are encountered. The application provides information about the cited document to the user, based on his or her current reading context, within the same browser in which the citing document is being viewed (for example, Adobe Acrobat Reader or a web browser), thus avoiding an interruption to the primary reading activity. The key observation here is that the reading context can indicate why the reader might be interested in the cited document. In addition to metadata about the cited document and its abstract, CSIBS therefore shows a contextualised preview. This contextualised preview contains important sentences from the cited document that are related to the reading context. Our hypothesis is that the content of a citation context enables CSIBS to determine what information in the cited document will be of most value, given what has just been read by the reader in the citing document. This portion of the preview is built using automatic text summarisation methods that exploit the links and anchor texts in hypertext documents [15]. The preview is shown as a pop-up text box within the document browser. An example showing the pop-up, as viewed using a web browser, is presented in Figure 1. CSIBS previews can also be viewed in PDF documents using Adobe Acrobat, as presented in Figure 2. The integration of this support into the browser, overlaying summaries on top of the citing document, enables readers to maintain their focus of attention while providing relevant snippets tailored to their information needs. CSIBS thus helps the reader avoid following less relevant citations, by indicating whether or not additional information about the citation context exists in the cited document. The present paper is structured as follows. We first present related work in Section 2. Section 3 briefly describes the user study we carried out to determine the primary information needs of readers as they encounter citations; this study was carried out in the biomedical domain. Section 4 describes the design and implementation of the CSIBS system, based on the results of our user study. Section 5 discusses our preliminary evaluation study. Finally, Section 6 presents our current on-going and future research interests. Concluding remarks are provided in Section 7.
Figure 1: A sample pop-up with an automatically generated preview. This pop-up appears in a web browser when the user mouses over the citation. Words that match the citation context are coloured and emboldened. The figure also illustrates the presentation of relevant section titles, used to group extracted sentences. This grouping provides a better organisation of the citation-sensitive preview.
about cited documents. The tasks of users in academia, specifically in the humanities [2] and in mathematics [17], have been studied previously, revealing a number of teaching and research activities that might be supported by technology. Task-based analyses in the biomedical domain have been carried out by Bartlett and Neugebauer [1], and by Tran et al. [14]. Their analyses, like ours, use qualitative studies to uncover the underlying uses of information. However, the tasks outlined in these related works are focused on a specific set of information needs in a research area: for example, the determination of a functional analysis of gene sequences. Our work differs in that we wish to take a higher-level view in order to elicit information needs that occur in scientific research more broadly, at least in the domain of the biomedical sciences. Li et al. [5] analysed the usage of scientific text, specif-
2. Related Work The ways in which technology can help readers of academic texts have been scoped by Shotton et al. [12], who describe what might be possible in the future of academic publishing. Our work in automatically generating previews is related but focuses on the provision of information 2
Figure 2: A preview pop-up available within a PDF document. The image on the right is a magnified version of the pop-up. It contains an automatically generated preview which is made available by double-clicking the citation in Adobe Acrobat. This pop-up interface allows the researcher to select and copy the text in the preview.
ically focusing on the types of queries made when searching over large-scale scientific literature repositories. Our analysis differs in that we seek to understand the information needs and the underlying tasks being performed by researchers in the context of browsing, as opposed to searching, through the literature. To facilitate these tasks, we investigate technologies that are able to locate and present useful information from within scientific documents. Other research has addressed similar goals: the extraction of tabular information from within scientific documents has been explored by Liu et al. [6]; and key phrases from scientific publications are identified by Nguyen and Kan [9], who do so by taking account of document structure. In this paper, we are interested in extracting sentences that justify the citation context, thus helping the reader make a relevance judgement about the cited document. Sentence-related extraction technologies have been explored within the field of automatic text summarisation, although with a different application perspective than ours. The detection of sentences containing citation information has been studied in the context of finding patent information in Japanese text [7]. In earlier work by Nanba et al., such sentences were used to automatically compile survey
articles [8]. Using co-citation information for summarisation has also been explored. An overview is provided in Elkiss et al. [3]; in that work, the authors analyse the similarity between citation contexts for a particular document. Similarly, approaches for summarising an academic document based on the citations of that document found in the literature at large have been studied [10]. Citation contexts have also been classified according to the rhetorical role that they play in the document that is currently being read [13]. Given this classification, cited documents are summarised by showing sentences that support the specific rhetorical role. These approaches are similar to ours in that they exploit the use of citations between documents as linkages. However, except in the case of Teufel et al. [13], these approaches only produce summaries that are independent of the reading context, while our approach deliberately tailors a preview to the citation context that has just been read. Our work differs from that of Teufel et al. [13] in that we tailor the preview to the content of the citation context and not to the rhetorical role of the citation. There are also a number of commercial products that address the needs of users in academia. Knowledge extrac3
tion systems focusing on biomedical text, such as BioMed Experts,4 Illumin8,5 and ConceptWeb,6 can automatically extract facts or highlight linkages (for example, by using clustering approaches) between researchers and published research. Often these tools provide improved search interfaces by suggesting ways in which to expand a query. These tools are complementary to browsing-oriented tools like CSIBS, which focuses on the under-explored browsing scenario. For example, once a document is retrieved via one of these search engines, a researcher can use CSIBS to navigate further through the space of document citations.
to here as the citation-focused task. Specific examples of the latter include: explaining why a document was cited; selecting new baselines for the design of new experiments; advancing one’s understanding of the topic of the citation context; investigating the details of the citation context further; and justifying the citation context. The underlying contentoriented information need is triggered by the text that the participant has just read. Both the appraisal and the content-focused tasks serve to help make a relevance judgement about the cited document. 2. The main difficulty that participants reported was in performing citation-focused tasks. Intuitively, this accords with one’s expectations given that, ordinarily, a citation-focused task would require manually scanning through the full cited document to search for specific information. 3. Searches within the cited document to find passages of text that match the citation can be automated, guided by key words from the citation context, provided that the results are presented in such a way that makes the reasons for information selection understandable and transparent to the user.
3. User Requirements Analysis To determine what readers of scientific literature require of cited documents, we conducted a user requirements analysis. We focused on users of biomedical literature because we had access to an extensive corpus of material in this domain as participants in the Elsevier Grand Challenge. However, our study and the questions we asked of participants were not specific to this domain. We began by interviewing subjects from an appropriate user demographic and recording their verbal descriptions about a real scenario situated in their day-to-day activities. Following this, we designed a questionnaire for wider participation which presented scenario-based questions attempting to uncover participants’ information needs and tasks. Participants were asked to provide free text answers. The responses were then collated and analysed for commonalities, bringing to the fore those issues that were salient across the participants. We recruited 24 users with a background in biomedical life sciences. All 24 participants started the questionnaire, and 18 completed it. Of the 24 participants, two-thirds indicated that they browsed through academic literature at least once a week. The main section of the questionnaire consisted of a series of questions, corresponding to the issues we wanted to explore. Principally, we wanted to know what information needs researchers have of a cited document, and what specific tasks this information serves. We analysed the freetext responses, checking for repeated terms and concepts that could form the basis of clustering. Salient information needs were matched to corresponding tasks. The most salient groupings of responses revealed the following:7
4. Citation-Sensitive In-Browser Summarisation We took the three main outcomes of the user requirements analysis and used them (1) to decide what kinds of information to present in the pop-up preview, and (2) to determine what interface issues were pertinent in this scenario. In this section, we provide an overview of the outcome, CSIBS. Specifically, we focus on how the preview is generated and delivered on multiple presentation media. 4.1. The CSIBS System Our system architecture is presented in Figure 3. CSIBS produces a preview-annotated version of a published academic document, rendered either as an Adobe Portable Document Format (PDF) document or a HyperText Language (HTML) web page. The annotated document is a modified version of the original article with CSIBS previews inserted into the appropriate citation contexts. With the annotated document, the reader can activate the previews with mouse interactions in the appropriate browser. We now describe the process involved in generating the annotated document using data retrieved from databases maintained by Elsevier, including the full text of the cited documents. CSIBS is first provided with an identifier for the document that the reader has requested. This information is easily obtained from portals housing scientific data, such as ScienceDirect.8 The system retrieves the
1. There were two main types of information needs, met respectively by meta-data and content-oriented information. These correspond to the two tasks of appraising the cited document (making a value judgement about it) and relating content in the cited document to the citation context, a general task referred 4 www.biomedexperts.com 5 www.illumin8.com 6 www.wikiprofessional.org 7 A paper describing the full details of the user requirements analysis was published in [16].
8 www.sciencedirect.com
4
Figure 3: A system architecture diagram. CSIBS produces preview-annotated PDF documents and HTML web pages.
XML version of this document.9 All citation instances are identified in the XML representation; these are then extracted and matched with the full references at the end of the document. Meta-data for each of the cited documents is retrieved from the EMBASE web service.10 This provides, amongst other things, the Elsevier-specific Publication Item Identifier (PII), which is used to retrieve the XML representation of each cited document from the Elsevier XML Repository.11 Each citation instance, its meta-data and the XML for the corresponding cited document is then passed to the summarisation engine, which computes the contents of the pop-up. The resulting contextualised preview contains a subset of the meta-data for appraising the cited document, the abstract, and a citation-sensitive preview consisting of sentences extracted from the cited document that match the citation context. The meta-data corresponds to a generic view of the cited document and is independent of the citation context, while the citation-sensitive preview will differ depending on what is said about the cited document in the citation context. Finally, the preview is inserted into the appropriate place in the annotated version of the published article by inserting dynamically-produced JavaScript code at each citation point. To facilitate the researcher’s literature browsing activities, CSIBS can be deployed as a web service attached to an existing publications portal (provided that appropriate permissions to databases are in place), so that the portal
can provide the preview-annotated documents. 4.2. Meta-Data Information CSIBS coordinates the retrieval of different meta-data, returned from queries to an online publications databases. This currently includes: • The full reference: This is useful in providing readers with information about the publication. • Author information: CSIBS includes data to help the reader establish a level of trust in the citation, primarily focusing on the author’s affiliation. • The citation count for the cited document: This is extracted to help users appraise the citation. These pieces of information were commonly identified as useful in helping readers make value judgements about the cited work. For example, having the full reference allows the reader to make informal judgements based on information about the host journal, such as its impact factor. We found that such judgements were frequently made in the biomedical domain. 4.3. Citation-Sensitive Previews The module that generates the citation-sensitive preview requires, as input, the citation context and the text from the cited document. The latter is obtained by extracting the text from the XML representation of the document and performing sentence segmentation. As output, the module produces a short list of sentences that helps the reader perform the citation justification task. These
9 While
the functionality of CSIBS can be ported to other document collections, the current system assumes the availability of XML data. 10 www.embase.com 11 http://labs-repo.elsevier.com/repo
5
are extracted based on locating matching words. The process of finding matching words excludes stop words, words with minimal content including, for example, prepositions, determiners and auxiliary verbs. We cap the number of extracted sentences at a predefined limit, n, currently set to four. To select sentences, we represent the citation context and each sentence in the cited document as a vector, where each dimension represents a term in the vocabulary, and the value for that position in the vector is the term frequency. We use vector space methods [11] to find the best n sentences that match the citation context, based on the cosine similarity metric. This metric measures the cosine of the angle between two vectors: one representing a sentence and the other representing the citation context. If the vectors are orthogonal, the cosine will be zero, which is interpreted as indicating that the sentence is unrelated to the citation context. The attractiveness of this approach lies in its speed and simplicity. This is a crucial design point for an application which we intend to be used over the web. For such applications, interaction times must be sufficiently low. Previews can be computed in approximately 0.03 seconds, making the process amenable to batch processing of multiple documents or, in the future, live generation of previews at runtime. The vector space approaches were chosen as they allow a precise matching of words in the citation context, thus minimising the risk of exacerbating any existing information overload problems by further burdening the reader with poorly matching sentences. To help the user, word matches are highlighted, allowing the reader to understand why particular sentences were extracted. Users are thus able to determine which retrieved sentences are useful. Of course, false positives are always possible. However, with this interface, users can see when sentences are only marginally related, given the lack of informative words that are highlighted, and thus they can ignore any false positives that do occur. In our user requirements analysis (see Finding 3 in Section 3), we found transparency in determining results to be a desirable trait for an automated system. Relevant section headings from the cited document are also provided as a means of grouping extracted sentences to improve the organisation of the citation-sensitive preview. When a sentence is chosen, we locate the innermost section to which it belongs and retrieve the section heading. Section headings are then used to group multiple sentences belonging to the same section. Sections and their sentences are presented in the same order in which they appear in the cited document. The structure of the cited document was also reported by participants in the user study as being useful, since it provides a snapshot of the relevant parts of a document and allows readers to ascertain the scope of relevant material ‘at a glance’. One key feature of the contextualised preview is that the information presented should differ depending on the
underlying information need represented by the citation context. Intuitively, one would expect that a preview of an article cited in the “introduction” section should be different from a preview triggered by a citation in the “methods and materials” section, for example. Figure 4 presents two citation-sensitive previews side by side, each generated for the same cited document but triggered by different citation contexts. These examples are taken from a published article, for which a preview-annotated ScienceDirect webpage version was created by CSIBS. Since the citation contexts are different — coming from the “methods and materials” and “discussion” sections, respectively — the resulting citation-sensitive preview changes to adapt to the context. In this example, the preview on the left presents information relating to a method, specifically the procedure which is not documented in the citing document. The second preview on the right presents information about the conclusions drawn from the work described in the cited document, which is more relevant to the discussion at the end of the citing document, where this second citation occurs. 5. The Utility of CSIBS We provide here a brief overview of a preliminary qualitative evaluation. The goal of this evaluation was to examine how participants would react to the pop-up previews. The feedback allows us to further clarify our analysis and subsequent development. Three biomedical researchers, all of whom had taken part in our original user requirements analysis, participated in the evaluation. Each participant was shown nine different passages containing a citation context, each situated in a different FEBS Letters 12 publication (which was also presented in full to the participants). Subjects were asked to make relevance judgements on the cited document given the citation context and a preview. In this evaluation, the actual judgements are not as important as the self-reported levels of confidence which we asked the participants to provide, on a 3-point Likert scale. The small sample size does not permit hypothesis testing. However, we are encouraged by the positive gains in self-reported confidence scores when participants were shown the CSIBS previews, compared to simply showing the full reference, which is currently the only citation information available to researchers. Since both the abstract and citation-sensitive previews evoked comparable positive gains in self-reported confidence (+1.2 and +2.2, respectively), we assume that these types of information facilitated the relevance judgements. Participants also reported that, for the citation-sensitive preview, two thirds sentences were found to be useful on average. The participant feedback also indicated support for the CSIBS interface and the preview information presented. 12 The Journal of the Federation of Europeans Biochemical Societies.
6
Figure 4: Although both pop-ups display citation-sensitive previews for the same cited document, the contents are not the same because the citation contexts, displayed underneath each pop-up, are different.
One participant made some particularly interesting observations regarding the selected sentences and the structure of the cited document. Specifically, useful sentences tended to be located deeper in the cited document, for example in the methods sections. This participant suggested that, for an expert user, showing sentences from the earlier sections of a publication was not useful; for the same reason, the abstract might be too general and not helpful in justifying a citation. Finally, one participant remarked that, in those situations where each document downloaded from a proprietary repository incurs a fee, the citation-sensitive previews would be very useful in deciding whether to download the document. In a subsequent evaluation, we tested CSIBS with 26 biomedical researchers who had not participated in our user requirements studies. We found that results again supported the utility of CSIBS. Due to space constraints, we focus on the reactions of the participants to the interface, as represented by their feedback comments. Note that a recurrent theme is the usefulness of preview information, which participants described as helping the relevancy task they were asked to perform:
• “The ability to preview RELEVANT [sic] information contained in references is brilliant and would be an enormous help for my work. Easy access to such a tool in combination with easy access to the references in the citations would certainly change the way I would read papers.” (Researcher) • “I think the preview is a great idea, it would save a lot of time in terms of looking up entire articles to determine their relevance.” (PhD Student) • “The preview information was informative, and helps one to decide whether or not the cited article is worth reading.” (Post-Doctoral Researcher) The following comments about the citation-sensitive preview show that our interface decisions in presenting information (for example, using structural information) were found to be effective: • “The preview information which was presented in a structured way, with subtitles, paras etc was good and easier to understand.” (Post-Doctoral Researcher) 7
• “In general it gave you a snap shot of what to expect in the article. Often this would be enough to know whether the article was worth reading.” (PostDoctoral Researcher)
References [1] J. C. Bartlett, T. Neugebauer, A task-based information retrieval interface to support bioinformatics analysis, in: IIiX ’08: Proceedings of the second international symposium on Information interaction in context, ACM, New York, NY, USA, 2008. [2] N. J. Belkin, Design principles for electronic textual resources: Investigating users and uses of scholarly information, in: Current Issues in Computational Linguistics: In Honour of Donald Walker.Kluwer, Kluwer, 1994. [3] A. Elkiss, S. Shen, A. Fader, G. Erkan, D. States, D. Radev, Blind men and elephants: What do citation summaries tell us about a research article?, J. Am. Soc. Inf. Sci. Technol. 59 (1) (2008) 51–62. [4] W. R. Hersh, Information Retrieval: A Health and Biomedical Perspective, Springer, 2008. [5] H. Li, W.-C. Lee, A. Sivasubramaniam, C. Giles, Workload analysis for scientific literature digital libraries, International Journal on Digital Libraries 9 (2) (2008) 139–149. [6] Y. Liu, K. Bai, P. Mitra, C. L. Giles, Tableseer: automatic table metadata extraction and searching in digital libraries, in: JCDL ’07: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, ACM, New York, NY, USA, 2007. [7] H. Nanba, N. Anzen, M. Okumura, Automatic extraction of citation information in japanese patent applications, International Journal on Digital Libraries 9 (2) (2008) 151–161. [8] H. Nanba, N. Kando, M. Okumura, Classification of research papers using citation links and citation types: Towards automatic review article generation, in: The 11th SIG Classification Research Workshop, Classification for User Support and Learning, 2000.11, in Chicago, USA, The American Society for Information Science (ASIS), 2000. [9] T. Nguyen, M.-Y. Kan, Keyphrase extraction in scientific publications, in: Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers, 2007. [10] V. Qazvinian, D. R. Radev, Scientific paper summarization using citation summary networks, in: The 22nd International Conference on Computational Linguistics (COLING 2008), Machester, UK, 2008. [11] G. Salton, M. J. McGill, Introduction to modern information retrieval, McGraw-Hill, New York, 1983. [12] D. Shotton, K. Portwin, G. Klyne, A. Miles, Adventures in semantic publishing: exemplar semantic enhancements of a research article., PLoS computational biology 5 (4) (2009) e1000361+. [13] S. Teufel, M. Moens, Summarizing scientific articles: experiments with relevance and rhetorical status, Computional Linguistics 28 (4) (2002) 409–445. [14] D. Tran, C. Dubay, P. Gorman, W. Hersh, Applying task analysis to describe and facilitate bioinformatics tasks, Studies in Health Technology and Informatics 107107(Pt 2) (2004) 818– 22. [15] S. Wan, C. Paris, In-browser summarisation: Generating elaborative summaries biased towards the reading context, in: Proceedings of ACL-08: HLT, Short Papers, Association for Computational Linguistics, Columbus, Ohio, 2008. [16] S. Wan, C. Paris, M. Muthukrishna, R. Dale, Designing a citation-sensitive research tool: An initial study of browsingspecific information needs, in: Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries, Association for Computational Linguistics, Suntec City, Singapore, 2009. [17] J. Zhao, M.-Y. Kan, Y. L. Theng, Math information retrieval: user requirements and prototype implementation, in: JCDL ’08: Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries, ACM, New York, NY, USA, 2008.
Preview information was found to be useful not only for the relevancy task but also to help researchers acquire new knowledge: • “In some cases, the preview gave a useful expansion of the highlighted sentence. This feature would be particularly helpful when learning about a new research area.” (Post-Doctoral Researcher) 6. Future Work In future work, we will research methods for further improving the tailoring of citation-sensitive previews to the context. Additionally, in our work to date, we have demonstrated CSIBS on one scientific discipline, that of biomedicine. We aim to apply the techniques to other disciplines; we expect the transfer to other scientific domains to be straightforward given the domain-independent design of CSIBS. 7. Conclusion We presented a new research tool aimed at supporting researchers as they acquire knowledge through the perusal of scientific literature. The Citation-Sensitive InBrowser Summariser (CSIBS) was designed and developed by taking into account real tasks and information needs of academic researchers, particularly biomedical researchers. CSIBS helps readers determine whether or not a cited document is worth reading further, addressing two types of activities: article appraisal and citation-focused tasks. These tasks are facilitated by providing readers with documents annotated with previews. Our evaluations of CSIBS indicated that the preview information is helpful for making relevance judgements, and that the CSIBS interface is useful and intuitive. We conclude that the contextual previews naturally complement the document’s own abstract, given that generated previews can present detailed information from the body of the cited document that is more relevant to the specific citation context. 8. Acknowledgments We would like to thank the participants in our user studies for their time and insightful comments. We would also like to thank Elsevier for permitting access to their data via the Elsevier Grand Challenge. Finally, we would like to thank Julien Blondeau, Michael Muthukrishna and Ilya Anisimoff for their efforts in performing engineering tasks and HCI studies.
8
