Application of Metadata Concepts to Discovery of Internet Resources

Application of Metadata Concepts to Discovery of Internet Resources Mehmet Emin Kucuk1 , Baha Olgun2 , and Hayri Sever2 The Department of Library Science Hacettepe University 06532 Beytepe, Ankara, Turkey The Department of Computer Engineering Hacettepe University 06532 Beytepe, Ankara, Turkey 1

2

Abstract. Internet resources are not yet machine-understandable re-

sources. To address this problem a number of studies have been done. One such a study is the Resource Description Framework (RDF), which has been supported by World-Wide Web (WWW) Consortium. The DC (Dublin Core) metadata elements have been de ned using the property of extensibility of RDF to handle electronic metadata information. In this article, an authoring editor, called H-DCEdit, is introduced. This editor makes use of RDF/DC model to de ne contents of Turkish electronic resources. To serialize (or to code) a RDF model, SGML (Standard Generalized Markup Language) has been used. In addition to this work, a possible view of RDF/DC documents is provided using Document Style Semantics and Speci cation Language (DSSSL) standard. H-DCEdit supports use of Turkish language in describing Internet resources. Isite/Isearch system developed Center for Networked Information Discovery and Retrieval (CNIDR) organization with respect to Z.39.50 standard is able to index documents and allows one to query the indexed terms in tagged elements (e.g., terms in RDF/DC elements). In the scope of our work, the localization of this Isite/Isearch system is completed in terms of sorting, comparison, and stemming. The feature of supporting queries over tags provides basis for integrating H-DCEdit authoring tool with Isite/Isearch search engine. Keywords:

ing editor.

Discovery of Internet Resources, Web mining, RDF/DC author-

1 Introduction As it is witnessed by everybody, the Internet has been growing rapidly since its inception in December 1969 and we are experiencing a boom in electronic information availability on the Internet. 1996 statistics showed that the number of Web sites were doubled in 100-125 days. Again, 1997 statistics proved that the number Web sites doubled in every 4 months. According to a survey of

the Internet Valey Inc., in January 1998, there were 29.67 million hosts on the Internet with 2.5 million domains and 2.45 million Web sites [2]. In year 2000, it was estimated that 50 million computers are connected to the Internet, there are 304 million Internet users and 200 million Internet documents and 7 million Web sites are available on the Internet(Nua surveys. [online] http://www.nua.ie/ [2000, May 5]). Obviously, organizing this huge amount of information sources and retrieving the desired information are not easy matter. Although the Internet and its most commonly used tool WWW represent signi cant advancement to retrieve desired information, we are experiencing some retrieval and information dissemination problems on the Internet. The major search engine companies have often claimed that they can keep up with the size of the Web, that is, that they can continue to index close to the entire Web as it grows. The reality however shows dierent story: according to a survey on the search engines, Northern Light covers 16covers 15covers 2.5that the entire Web is not covered and indexed by the engines [5]. In addition to the coverage, precision and recall rate are the other problematic area. Recall and precision factors of engines frequently involve precision factors of much less than 1 percent. For example, a search of the WWW using search engine ANZWERS on acronym "IETF" (which stands for Internet Engineering Task Force) retrieved 100,888 match on 12 April 2000, 91,017 match on 5 August 1999, 896,354 match in 1998. Every Web page which mentioned the IETF in an incidental way retrieved by this search. This example illustrates that search engines can return a lot of irrelevant information because they have no means (or very few means) of distinguishing between important and incidental words in document texts [3]. 1.1

What is Metadata ?

There are many de nitions of the term metadata. Literally meaning information about data, the most simpli ed and referred de nition is data about data. One recognizable form of metadata is the card catalogue in a library; the information on that card is metadata about a library material (e.g., book, journal, proceeding, etc.) Regardless of the context being used for publishing metadata, the key purpose remains the same which is to facilitate and improve the retrieval of information. In an environment such as the traditional library, where cataloging and acquisition are the sole preserve of trained professionals, complex metadata schemes such as MARC (Machine Readable Cataloging) are, perhaps, acceptable means of resource description [6]. 1.2

Metadata Standards

There are variety of metadata standards such as Content Standards for Digital Geospatial Metadata, Encoded Archival Description, Text Encoding Initiative, Categories for the Description of Works of Art, Dublin Core (DC) and etc. Some standards have been developed to describe and provide access to a particular type of information resource, such as geospatial resources. Other standards such

as DC, have been developed to provide a standard way of describing a wide range of dierent types of information resource, allowing these diverse types to be retrieved through a single searching process. There will always be variety of metadata standards. However, DC is the most commonly used and promoted metadata standard since it aimed all types of documents in any subject. The DC (http://purl.org/dc/) metadata element set was developed during 1995 and 1996 and has been supported by the W3 Consortium. DC Element Set includes 15 data elements which are listed as Title, Author or Creator, Subject and Keywords, Description, Publisher, Other Contributors, Date, Resource Type, Format, Resource Identi er, Source, Language, Relation, Coverage, and Rights Management. 1.3

Tools for Metadata Creation

Metadata creation tools can be categorized as editors and generators. The rst type, loosely labelled as editor, allows the creation of metadata by providing a template for entering new metadata content. The supporting software places the content into the HTML tag, which may then be cut and pasted into a document 1 . The second type, loosely labelled as generator, extracts metadata from existing HTML- encoded documents and places the content into the HTML tag. The generators have a choice of outputs: for example, some produce HTML v3.2, or HTML v4.0 and some generate XML for immediate inclusion in a repository (Meta Matters: Tools [online] http://www.nla.gov.au/meta/tools.html [2000, July 25]).

2 H-DCEdit DC element set was translated into 22 languages including Turkish 2 [10]. However, translating DC element set into a dierent language without supporting this particular language by a metadata creation tool does not make a signi cant improvement in identifying and retrieving the documents in this particular language. As a response to the need to improve retrieval of electronic information in Turkish, H-DCEdit Software based on RDF/DC3 standard was developed as a part of KMBGS (Kasgarl Mahmut Bilgi Geri-Getirim Sistemi) research project at the Department of Computer Engineer, Hacettepe University in 1999. Our objection with the H-DCEdit editor is to de ne the metadata content of Internet sources using RDF model and DC elements. As shown in Figure 1, the It is important to note that the metadata language is an internal representation used by the system, not for end users to specify directly. This is similar to PostScript, which is used to specify page layouts for printers. Users do not generate PostScript directly, instead they use a graphical application to automatically generate it [9]. 2 Currently, Turkish version of DC has only Turkish counter-parts of some DC elements. 3 RDF/DC stands for Resource Description Framework on top of which the controlled vocabulary of Dublin Core has been de ned.

1

SGML Declaration

RDF/DC Document Type Definition

DSSL Declaration Style Declaration

RDF Parser RDF/DC Style Definition SGML Parser

Output of Parsing

Editor of DC Elements

SGML Document

DSSL Engine (jade)

SGML Document

Local Auxilary Programs (vi, netscape, xview)

Document in the Form of html, rtf, TeX, etc.

Fig. 1. Functional View of H-DCEdit System Model. H-DCEdit editor utilizes the SGML/XML 4 concrete syntax notation with the version of ISO 8879. The functionality of H-DCEdit editor depends on SGML declaration including name spaces of XML. We have used core RDF structs excluding the rules for statements about statements, where the predicate eld of triple concerns with creation of sets or bags objects of the subject. The components lled with gray color in Figure 1, indicate names of freeware modules, which are available for Unix platform and incorporated into implementation of H-DCEdit editor whose components are explored in the rest of this section. The SGML declaration was designed in such a way that it is not only consistent with the reference concrete syntax, but also supports use of Turkish characters. The eight-bit character set of document is based on two character sets: ISO646 and ECMA-128 (ISO Latin 5) for the positions of 0-127 and 128255, respectively. In the RDF model, XML namespaces are used for schema speci cation as well as for control of vocabulary. An element name is associated with a XML namespace; hence, its structure and correct interpretation become possible to software agents (or robots), even if namespaces are nested in describing an Internet resource5. This kind of associations in XML is denoted in the form of and Since, as discussed later, SGML's type declaration was designed in consistent with some XML notation, it is called SGML/XML, instead of SGML. 5 any object that has a valid URI address is called Internet resource.

4

becomes part of document type de nition by introducing the following clause to the NAMING section of SGML declaration. LCNMSTRT UCNMSTRT LCNMCHAR UCNMCHAR

"" "" "-.:" "-.:"

The LCNMSTRT and UCNMSTRT articles de ne lower and upper letter to be used for naming characters. By adding the characters of -.: into both articles respectively, these characters are conceived as naming characters. The document type de nition of RDF/DC notation is provided using SGML declaration. On other words, both XML compatibility and Turkish character support is also provided to RDF/DC notation for the reason that XML is used as a serialization language of the RDF/DC model that is, in turn, represented by acyclic and directed graph. Critical sections of RDF/DC document type de nition is given as follows.



rdf:RDF - - ( rdf:Description )* > rdf:RDF CDATA "http://www.w3.org/RDF/" CDATA "http://purl.org/DC/"

This recursive coding (Kleene closure of ) provides building blocks for document content enclosed by and < =rdf:RDF> tags. The attributes of element are listed as and with contents of http://www.w3.org/RDF/ and http://purl.org/DC/, respectively. The content model of would be any property as shown below.


DC:TITLE DC:CREATOR DC:SUBJECT DC:DESCRIPTION DC:PUBLISHER DC:CONTRIBUTOR DC:DATE DC:TYPE DC:FORMAT DC:IDENTIFIER DC:SOURCE DC:LANGUAGE DC:RELATION

-

-

%dccontent; %dccontent; %dccontent; %dccontent; %dccontent; %dccontent; %dccontent; %dccontent; %dccontent; %dccontent; %dccontent; %dccontent; %dccontent;

> > > > > > > > > > > > >


- - -

%dccontent; %dccontent;

> >



This parametric property provides us to de ne DC elements as properties of RDF elements, but at the same time, allows possible extensions in terms of new element de nitions that might be needed in the future. The attribute list of is declared as follows.

The above attributes describe (or identify) the rdf objects as discussed in[7]. Let us give a simple RDF model of an Internet resource in Figure 2. Here, the source object, sgml.cs.hun.edu.tr is described by RDC/DC elements in the form of . For this example, the output of HDC-Edit is given in the Appendix (GUI based user interfaces were developed using MOTIF on Unix platform. Because of the size limitation of the paper, we refer the reader to home page of KMBGS Project for appearances of these interfaces [8]. The RDF/DC parser component incorporates with the SP package (SGML parser) via API (application program interface) to parse an SGML document complying with type de nition of RDF/DC. The compilation process yields an intermediate output that is in suitable form to the RDF/DC editor whose browser module in turn displays that output. DSSSL standard contains three main sections, namely transformation, style, and query. The HDC- Edit uses the style utiliy of DSSL engine which was embedded into the SGML processing system. Note that the SGML declaration and document type definition is also valid for the DSSSL engine. The SGML document generated by RDF/DC editor can be translated to well-known languages like RTF, TeX, HTML along with its CSS (cascaded style sheet).

3 Application In this section, we introduce a traditional way of exploiting metadata. Speci cally we use Isite/Isearch search engine, which is a freeware software developed by CNIDR (Center for Networked Information Discovery and Retrieval). Database of information items given in Figure 4 consists of RDF/DC documents generated by H-DCEdit editor (strictly speaking this is not a true picture, because RDF/DC documents were

Baha Olgun

Creator

Title

SGML Türkiye

Publisher sgml.cs.hun.edu.tr

Hacettepe Üniversitesi

Contributor

. . .

Type Hayri Sever

text/SGML

Fig. 2. Content representation of a simple Internet resource by RDF model translated to HTML using DSSSL engine of SGML processing system). This back-end database can be populated by remote Internet resources using a Web crawler. Note that the choice of Isite/Isearch is twofold. The rst one is that it allows users to query over contents of tagged elements, which is something we need for DC elements. The other is that it operates on Z.39.50 networked information retrieval standard whose support to DC model (in addition to MARC records) is imminent. As a simple application we have cataloged our small departmental library using H-DCEditor and indexed using Isite system. We encourage the reader to take a look at the query interface and view source of documents located at http://ata.cs.hun.edu.tr/~km/gerceklestirim.html. We will not go through details of the integration of Isite/Isearch system with SGML processing system, but, instead, give a brief description of this search engine along with the design of a Web crawler. 3.1

Web Crawler

A Web crawler has been designed and implemented to retrieve documents from the World Wide Web and create a database. The implementation has been done in JAVA to eectively utilize the power of its platform independence, secured access and powerful networking features. As seen in Figure 3, the Web crawler agent is composed of three tightly-coupled submodules, namely Downloader, Extractor, and Parser. Initially the downloader starts with a seed (root) URL and then navigates Web catalog by a breadth- rst expansion using the queue for URLs to be traversed. It retrieves Web documents via hypertext transfer protocol (http) and in the process, passes the HTML document to the extractor. The detection of bad links are handled by the downloader by trapping the HTTP status code returned by the HTTP server of the URL to be visited. The hyperlink extractor detects the hyperlinks in the web documents, extracts them and passes

Document Files

Topology File

Seed URL HTTP Download

Hyperlink Extractor

Hyperlink Parser

URL Traversed and to be Traversed

Queue for URLs to be Traversed

Comparator

Fig. 3. The View of Web Crawler them to the parser for further processing. It searches for HTML tags of the form < a href = "http : == " > or < frame src = " " > : The parser converts these relative URLs to absolute URLs following the Internet Standards (RFC 1738, 1808) drafted by the Network Working Group. No parsing is necessary for absolute URLs. The host name (more speci cally, the domain name of the network host) of each of these URLs is then compared with the host name of the seed URL and only those URLs whose host name match with that of the seed URL are added to the queue. Care should be taken so that any new URL that is added to the queue is not a repetition of any of the URLs that has been already traversed or will be traversed. Thus the Web crawler retrieves all the web documents within the site speci ed by the root URL. URLs added by the parser to the queue are restricted to certain speci c types only and any URL to postscript (:ps), image (:gif; :jpeg; :jpg; etc.), compressed (:gz; :tar; :zip; etc.) les are not added.




3.2






Retrieval Engine

Functional diagram of descriptor-level retrieval is shown in Figure 4. The retrieval of documents using the keywords and matching them to descriptors is called descriptorlevel retrieval. For descriptor-level document/query processing we use the Isite/Isearch system. Isite database is composed of documents indexed by Iindex and accessible by Isearch. Isite/Isearch allows one to retrieve documents according to several classes of queries. The simple search allows the user to perform case insensitive search on one or more search elements, e.g. the use of titles/headers in matching search terms with document terms. Partial matching to the left is allowed. The Boolean search allows the user to compose a two-term query where the two terms are related by one of the Boolean operators AND,OR, and ANDNOT. "Full Text" is the default search domain unless the user selects a particular element for a term from the term's pull-down menu.

Client

Z.39.50 Network Gateway and Server

Server

Submission of User Query in HTML Form

Connection to Z.39.50 Server

Connetction to Information Retrieval System

Display of Retrieval Output in HTML Form

Merge HTML Documents with RDF/DC Content

Describe Resulting Information Items by HTML

Isite System Evaluate the User Query via Iindex and Isearch Components

Database of Information Items

Fig. 4. Functional View of H-DCEdit System Model. The advanced search form accepts more complex Boolean queries that are formed by nesting two-term Boolean expressions. To narrow a search domain from Full Text to a single search element, the term is pre xed with the element name and a forward slash, e.g., DC:TITLE/metadata. Format and size of the information items listed in a retrieval output may be speci ed by choosing target elements from a pull-down menu. Isearch is capable of performing a weighted search that employs the inverse document frequency of terms. As a nal note, we have made some patching to Isite/Isearch to enable indexing of Turkish documents (in terms of stemming, sorting and comparison) as well as use of Turkish characters in expression of user queries[1]. A new module of stemming based on G. Duran's graduate thesis was added, which can be considered as a supplementary module to Isite/Isearch[4].

Acknowledgment This work is partly funded by T.R. State Planning Organization under the project number of 97K121330 (please, refer to the home page of KMBGS Project at http://www.cs.hacettepe.edu.tr/~km for detailed information).

Appendix: The output of HDC-Edit for the example in Figure 2 http://sgml.cs.hun.edu.tr Baha Olgun SGML Türkiye Kullan&iwhdot;c&iwhdot;lar&iwhdot; SGML Türkiye SGML Türkiye Web Sayfas&iwhdot; Hacettepe Üniversitesi Hayri Sever Her Hakk&iwhdot; Sakl&iwhdot;d&iwhdot;r text text/sgml tr

References 1. Akal, F. Kavram Tabanl Turkce Arama Makinas. Yuksek Muhendislik Tezi. March 2000. Fen Bilimleri Enstitusu, Hacettepe Universitesi, 06532 Beytepe, Ankara, Turkey. 2. Chowdhury, G.G. The Internet and Information Retrieval Research: A Brief Review. Journal of Documentation. March 1999, 55 (2): 209-225. 3. Connolly, D. Let a thousand owers bloom. In interview section of IEEE Internet Computing. March-April 1998, pp. 22-31. 4. Duran, G. Govdebul: Turkce Govdeleme Algoritmas. Yuksek Muhendislik Tezi. Fen Bilimleri Enstitusu, Hacettepe Universitesi, 06532 Beytepe, Ankara, Turkey, 1997. 5. Lawrence, Steve and C. Lee Giles. Searching the World Wide Web. Science. 3 April 1998, 280: 98-100. 6. Marshall, C. Making Metadata: a study of metadata creation for a mixed physicaldigital collection. In Proc. of the ACM Digital Libraries'98 Conf., Pitsburgh, PA (June 23-26, 1998), pp. 162-171. 7. Miller, E. An introduction to the resource description framework. D-Lib Magazine. May 1998. 8. Olgun, B. Dublin Core Ustveri Elemanlari Editoru. Yuksek Muhendislik Tezi. Fen Bilimleri Enstitusu, Hacettepe Universitesi, 06532 Beytepe, Ankara, Turkey. 1999. This thesis is also available at http://www.cs.hacettepe.edu.tr/~km. 9. Singh, N. Unifying Heterogeneous Information Models. ACM Comms. May 1998, 41(5):37-44. 10. Weibel, S. The state of the Dublin Core metadata initiative. D-Lib Magazine. April 1999.