arXiv:cond-mat/9907038v2 [cond-mat.dis-nn] 10 Sep 1999 The diameter of the world wide web Despite its increasing role in communication, the world wide web (www) remains the least controlled medium: any individual or institution can create websites with unrestricted ... Cited by 2497 - Related articles - View as HTML - All 52 versions
Self-similarity in World Wide Web traffic: evidence and possible causes
from psu.edu Find it @ Oxford [PDF]
ME Crovella, A Bestavros - Networking, IEEE/ACM ", 2002 - ieeexplore.ieee.org Abstract—Recently, the notion of self-similarity has been shown to apply to wide-area and local-area network traffic. In this paper, we show evidence that the subset of network traffic that is due to World Wide Web (WWW) transfers can show characteristics that are consistent ... Cited by 3122 - Related articles - BL Direct - All 56 versions
World-wide web: The information universe
[PDF]
from cern.ch
T Berners-Lee, R Cailliau, JF Groff, B " - Internet ", 2010 - emeraldinsight.com Purpose – The World-Wide Web (W 3 ) initiative is a practical project designed to bring a global information universe into existence using available technology. This paper seeks to describe the aims, data model, and protocols needed to implement the “web” and to compare them ... Cited by 1919 - Related articles - Check Availability - BL Direct - All 26 versions
Exploring the Web with OXPath∗ [BOOK]
Information architecture for the world wide web
[HTML]
from informationr.net
L Rosenfeld, P Morville - 1998 - portal.acm.org Information Architecture for the World Wide Web is for webmasters, designers, and anyone else involved in building a web site. It's for novice web designers who, from the start, want to avoid the traps that result in poorly designed sites. It's for experienced web designers who have ... Cited by 1030 - Related articles - Library Search - All 19 versions
Weaving the Web: The original design and ultimate destiny of the World Wide Web by its inventor [BOOK]
Tim Furche, Georg Gottlob, Giovanni Grasso, Christian Schallhart, Andrew Sellers T Berners-Lee, M Fischetti - 1999 - portal.acm.org Tim Berners-Lee, the inventor of the World Wide Web, has been hailed by Time magazine as one of the 100 greatest minds of this century. His creation has already changed the way people do business, entertain themselves, exchange ideas, and socialize with one another.. " ... Cited by 983 - Related articles - Library Search - All 13 versions
Oxford University Computing Laboratory, Wolfson Building, Parks Road, Oxford OX1 3QD
[email protected] [PDF]
Data preparation for mining world wide web browsing patterns
[PDF]
from psu.edu
R Cooley, B Mobasher, J Srivastava" - Knowledge and Information ", 1999 - Citeseer ... The World Wide Web (WWW) continues to grow at an astounding rate in both the sheer volume of traffic and the size and complexity of Web sites. The complex- ... Data Pr eparati on for Mining Wo r ld WideWeb Br owsing Patt e rns ... T able 1. Common Characteristics of Web Pages ... Cited by 1267 - Related articles - View as HTML - Check Availability - All 38 versions
Consumer reactions to electronic shopping on the World Wide Web
OXPath is a careful extension of XPath that facilitates data extraction from the deep web. It is designed to facilitate the large-scale extraction of data from sophisticated modern web interfaces with client-side scripting and asynchronous server communication. Its main characteristics are (1) a minimal extension of XPath to allow page navigation and action execution, (2) a set-theoretic formal semantics for full OXPath, (3) and a sophisticated memory management that minimizes page buffering. In this poster, we briefly review the main features of the language and discuss ongoing and future work.
{click /}
ABSTRACT
SL Jarvenpaa, PA Todd - International Journal of Electronic ", 1996 - portal.acm.org Much fascination and speculation surrounds the impact of the World Wide Web on consumer Web Images Videos Maps News Shopping Gmail more !
[email protected] | shopping behavior. At the same time, there is little empirical evidence underlying all this speculation. This article provides one such data set. It reports on factors that consumers ... Advanced Scholar Sear Search Cited by 719 - Related articles Check Availability - BL Direct 2 versions Web - Images Videos Maps News - All Shopping Gmail more !
[email protected] Search within articles citing Albert: The diameter of the world wide Advanced Scholar Search Searching the world wide web [PDF] from psu.edu world wide world wide webweb Search 2 1 Scholar All Create email S Lawrence, CL Giles - Science, 1998 - sciencemag.org Find itanytime @ Oxford include citations cited_by The coverage and recency of the major World Wide Web search engines was analyzed, yielding some surprising results. The coverage of any one engine is significantly limited: No Emergence single engine of scaling in random networks title: Scholar and patents include citations Create email alert indexes morepaper than about one-third of theArticles "indexable Web," the coverageanytime ofauthors: the six ... AL Barabási, R Albert - Science, 1999 - sciencemag.org Cited by 1067 - Related articles - BL Direct - All 71 versions Page 1. DOI: 10.1126/science.286.5439.509 , 509 (1999); 286 Science et al. Albert-László title: The diameter of the world wide web Barabási, Emergence of Scaling in Random Networks This copy is for your personal, R Albert, Jeong, AL Arxiv preprint cond-mat/9907038, 1999 -use arxiv.org Distributions ofauthors: surfers' paths Hthrough theBarabási world -wide web: Empirical characterizations from psu.edu non-commercial only. . clicking here[PDF] colleagues, clients, or customers by ... arXiv:cond-mat/9907038v2 [cond-mat.dis-nn] 10 Sep 1999Cited The by diameter the world wide web PLT Pirolli, JE Pitkow - World Wide Web, 1999 - Springer Find- All it @ 8027 - of Related articles - BL Direct 57 Oxford versions Despite its increasing role in communication, the worldamong wide web (www) remains the least Surfing the World Wide Web (WWW) involves traversing hyperlink connections medium: any individual or many institution can create websites with unrestricted ... documents. The ability to controlled predict surfing patterns could solve problems facing[PDF] producers The structure and function of complex networks Cited by We 2497 - Related articles - View - All site, 52 versions and consumers of WWW content. analyzed WWW server logsasforHTML a WWW ... MEJ Newman - Arxiv preprint cond-mat/0303516, 2003 - Citeseer ck /} Cited by 118 - Related articles - BL Direct - All 18{cli versions Inspired by empirical studies of networked systems such as the Internet, social networks, and 3 Self-similarity in World Wide Web traffic: evidence and possible bio- logical networks,causes researchers have in recent years developed a variety of techniques and ME Crovella, A Bestavros - Networking, IEEE/ACM ", web 2002 - ieeexplore.ieee.org Web mining: Information and pattern discovery on the world wide from psu.edu models to help us understand or predict[PDF] the behavior of these systems. Here we review ... the -notion of self-similarity has been shown to 4908 apply- to wide-area and- View as HTML - BL Direct - All 140 versions R Cooley, B Mobasher, J Abstract—Recently, Srivastava - ictai, 1997 computer.org Cited by Related articles local-area network traffic. this paper, show evidenceasthat the subset of network traffic that Abstract Application of data mining techniques to theInWorld Widewe Web, referredto Web is of due to World Wide Web (WWW) characteristics that are consistent ... mining, has been the focus sever al recent research projectstransfers and papcan ers.show However, there [PDF] Surfing the p53 network Citedleading by 3122 Related articles - BL Directr -esearch All 56 versions is no established vo cabulary, to -confusion when comparing efforts.The ... B Vogelstein, D Lane, AJ Levine" - Nature, 2000 - cmbi.bjmu.cn Cited by 861 - Related articles - Check Availability - All 55 versions … Tumour-suppressor genes are needed to keep cells under control. Just as a car's brakes regulate World-wide web: The information universe its speed, properly func- tioning tumour-suppressor genes act as brakes to the cycle of cell T Berners-Lee, R Cailliau, JF Groff, B " Internet ", 2010 - emeraldinsight.com Create email alert growth, DNA repli- cation and division into two new cells. When these genes fail to ... Purpose – The World-Wide Web (W 3 ) initiative is a practical project bring a- global Cited by 2928designed - Relatedtoarticles View as HTML - All 10 versions information universe into existence using available technology. This paper seeks to describe the aims, data model, and protocols needed to implement the “web” and to compare them ... [CITATION] Evolution of networks Cited by 1919 - Related articles - Check versions 1 Availability 2 3 4 5 6- BL 7 Direct 8SN9 Dorogovtsev, 10- All 26Next Result Page: JFF Mendes - 4+ Advances in Physics, 2002 - Taylor & Francis Cited by 2846 - Related articles - Library Search - BL Direct - All 54 versions [BOOK]
Information architecture for the world wide web
L Rosenfeld, P Morville - 1998 - portal.acm.org
Categories and Subject Descriptors H.5.4 [Information Interfaces and Presentation]: Hypertext/Hypermedia—navigation
2
4
General Terms
6
Languages, Algorithms
8
Keywords
The large-scale organization of metabolic networks
Information Architecture for the World Wide Web is for webmasters, designers, and anyone else doc("scholar.google.com")/descendant::field()[1]/{"world..."} Ê- nature.com H Jeong, from B Tombor, R Albert, ZN Oltvai, AL Barabási - Nature, 2000 involved in building a web site. It's forwide novice world webweb designers who, Search the start, want to avoid In a cellweb or microorganism, processes that generate mass, energy, information transfer and the traps that result in poorly designed sites. It's for experienced designers who the have ... /following::field()[1]/{click/} cell-fateË specification are seamlessly integrated through a complex network of cellular constituents Cited by 1030 - Related articles - Library Search - All 19 versions and reactions 1 . However, despite the key role of these networks in sustaining cellular ... /(//a[contains(string(.),’Next’)]/{click/}) *Í [BOOK] Weaving the Web: The original design and ultimate destiny of the World Wide Web Go to Google Home - About Google - About Google Scholar inventor _r:[.//h3:] //div.gs ©2010 Google [.//*.gs_a:] [.//a[.#=’Cited by’]/{click /}Ì //div.gs_r:[.//h3:] [.//*.gs_a:]]
Figure 1: Scholar
Finding an OXPath through Google
Web extraction, web automation, XPath, AJAX
1.
INTRODUCTION
In this paper, we present details of the OXPath language, a formalism for specifying data extraction from modern web applications. XPath has become the de facto standard for querying single XML and HTML documents. However, modern web applications often require navigation, form filling, and other (single or multi-page) interactions to expose the data of interest. OXPath is an extension of XPath to allow page ∗The research leading to these results has received funding from the European Research Council under the European Community’s Seventh Framework Programme (FP7/2007– 2013) / ERC grant agreement no. 246858. The views expressed in this article are solely those of the authors.
navigation, form filling, interactions, and the extraction of multiple data items in a single expression. For space reasons, we focus here on presenting the language and point the interested reader to [1] for discussion of OXPath’s semantics, evaluation algorithm, and formal properties.
2.
3. (c) 2011 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the U.S. Government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. LWDM 2011, March 25, 2011, Uppsala, Sweden. Copyright 2011 ACM XXXX ...$10.00
EXAMPLE
Figure 1 shows an OXPath expression for extraction of papers, authors, and citations from Google Scholar: Lines 1–2 fill and submit the search form. Line 3 realizes the iteration over the set of result pages by repeatedly clicking the “Next” link. Lines 4–5 identify a result record and its author and title, lines 6–8 navigate to the cited-by page and extract the papers.
OXPATH: THE LANGUAGE
OXPath extends XPath with (1) a new axis for selecting nodes based on visual attributes, (2) a new node-test for selecting visible fields, (3) a new kind of location step for actions and form filling, and (4) a new kind of predicate for marking data to be extracted. For page navigation, we adapt the notion of Kleene star over path expressions from [2]. Nodes and values marked by extraction markers
are streamed out as records of the result tables. For efficient processing, we cannot fix an apriori order on nodes from different pages. Therefore, we do not allow access to the order of nodes in sets that contain nodes from multiple pages.XPath expressions are also OXPath expressions and retain their same semantics, computing sets of nodes, integers, strings or Booleans. Style Axis and Visible Field Access. We introduce two extensions for lightweight visual navigation: a new axis for accessing CSS DOM node properties and a new node test for selecting only visible form fields. The style axis is similar to the attribute axis, but navigates dynamic CSS properties instead of static HTML properties. For example, the following expression selects the sources for the top story on Google News based on visual information only:
extracts from Google News a story element for each current story, containing its title and its sources, as in: 2
doc("news.google.com")//*[style::color="#767676"]
Actions. For explicitly simulating user actions, such as clicks or mouse-overs, OXPath introduces contextual action steps, as in {click}, and absolute action steps with a trailing slash, as in {click /} . Since actions may modify or replace the entire DOM, OXPath’s semantics assumes that they produce a new DOM. Absolute actions return DOM roots, while contextual actions return those nodes in the resulting DOMs which are matched by the action-free prefix of the performed action: The action-free prefix afp(action) of action is constructed by removing all intermediate contextual actions and extraction markers from the segment starting at the previous absolute action. Thus, the action-free prefix selects nodes on the new page, if there are any. For instance, the following expression enters “Oxford” into Google’s search form using a contextual action—thereby maintaining the position on the page—and clicks its search button using an absolute action. 2
doc("google.com")/descendant::field()[1]/{"Oxford"} /following::field()[1]/{click /}
Extraction Marker. Navigation and form filling are often means to data extraction: While data extraction requires records with many related attributes, XPath only computes a single node set. Hence, we introduce a new kind of qualifier, the extraction marker, to identify nodes as representatives for records and to form attributes from extracted data. For example, : identifies the context nodes as story records. To select the text of a node as title, we use :. Therefore, 2
doc("news.google.com")//div[@class~="story"]: [.//h2:] [.//span[style::color="#767676"]:]
The nesting in the result above mirrors the structure of the OXPath expression: An extraction marker in a predicate represents an attribute to the (last) extraction marker outside the predicate. Kleene Star. Finally, we add the Kleene star, as in [2], to OXPath. For example, we use the following expression to query Google for “Oxford”, traverse all accessible result pages, and to extract all contained links. 2
The style axis provides access to the actual CSS properties (as returned by the DOM style object), rather than only to inline styles. An essential application of the style axis is the navigation of visible fields. This excludes fields which have type or visibility hidden, or have display property none set for either themselves or an ancestor. To ease field navigation, we introduce the node-test field() as an abbreviation. In the above Google search for “Oxford”, we rely on the order of the visible fields selected with descendant::field()[1] and following::field()[1]. Such an expression is not only easier to write, it is also far more robust against changes on the web site. For it to fail, either the order or set of visible form fields has to change.
Tax cuts ... Washington Post Wall Street Journal ...
4
doc("google.com")/descendant::field()[1]/{"Oxford"} /following::field()[1]/{click /}/ ( descendant::a.l:/ancestor::* /descendant::a[contains(.,’Next’)]/{click /} )*
To limit the range of the Kleene star, one can specify upper and lower bounds on the multiplicity, e.g., (...)*{3,8}. Also note that OXPath adopts the class selector from CSS.
4.
FUTURE WORK
To the best of our knowledge OXPath is the first web extraction system with strict memory guarantees. These memory guarantees reflect strongly in our experimental evaluation. Just as important, OXPath is built on standard web technology, such as XPath and DOM, so that it is familiar and easy to learn for web developers. We believe that it has the potential to become an important part of the toolset of developers interacting with the web. To further simply OXPath expressions and enhance their robustness, we plan to investigate additional features, such as more expressive visual language constructs and multi-property axes. A strength of OXPath is that it is focused and easily embeddable. We want to exploit that potential by realizing OXPath in a variety of contexts, but will next focus on deploying OXPath in a cloud for large-scale extraction. OXPath is designed for highly parallel execution: the host language can assign different bindings for the same variable to create multiple OXPath queries. These queries can be processed with separate web sessions hosted on separate computing instances. We think this approach to parallel decomposition is ideally suited for the share-nothing nature of computing instances in elastic computing environments. The design of a host language for the cloud requires careful consideration: in particular, most existing web programming languages, such as XQuery, do not provide access to a dynamic DOM. Beyond variable bindings, any useful host language almost certainly requires aggregation and subquerying capabilities. Adapting OXPath to a cloud-based environment raises the question of how to optimize the evaluation of several parallel OXPath expressions in order to minimize expensive browser instantiations, unnecessary replication of common action sequences, and to best consolidate output extracted by OXPath expressions running in multiple instances.
5.
REFERENCES
[1] T. Furche, G. Gottlob, G. Grasso, C. Schallhart, and A. Sellers. Finding an OXPath to Cherries Hidden in the Scripted Web. Tech. rep., diadem-project.info/oxpath. [2] M. Marx. Conditional XPath. TODS, 2005.
