ing commercial software, we applied well-known data mining techniques. (association rules and clustering) to analyze access log records collected on a web ...
Mining Web Access Logs of an On-line Newspaper Paulo Batista and M´ ario J. Silva Departamento de Inform´ atica, Faculdade de Ciˆencias – Universidade de Lisboa Campo Grande 1749-016 Lisboa Portugal {pb,mjs}@di.fc.ul.pt
Abstract. With the explosive growth of data available on the Internet, personalization of this information space become a necessity. An important component of web personalization is the automatic knowledge extraction from web log files. However, analysis of large web log files is a complex task not fully addressed by existing web access analyzers. Using commercial software, we applied well-known data mining techniques (association rules and clustering) to analyze access log records collected on a web newspaper. This paper identifies several reading patterns and discusses approaches for mining this data.
1
Introduction
The evolution of the Internet has lead to an enormous proliferation of the available information and the personalization of this information space has become a necessity. The knowledge obtained by learning web users preferences can be used to improve the effectiveness of their web sites by adapting the web information structure to the users behavior. Automatic knowledge extraction from web log files can be useful for identifying such reading patterns and infer user profiles. However, it is hard to find appropriate tools for analyzing raw web log data to retrieve significant and useful information. There are several commercially available web log analysis tools [1], but most of them are disliked by their users and considered too slow, inflexible, expensive, difficult to maintain or very limited in the results they can provide [18]. Recently, the advent of data mining techniques for discovering usage patterns from web data (a.k.a. web log mining or web usage mining) made it possible to mine typical user profiles from the vast amount of access logs. Web usage mining can be viewed as the extraction of usage patterns from access log data containing the behavior characteristics of users. Web usage mining can help in addressing some of the shortcomings of the standard approaches for web personalization. However, the discovery of patterns from usage data is not by itself sufficient for performing the personalization tasks. Nevertheless, it is an important component for effective derivation of ”actionable” user profiles derived from web access patterns [11]. The learned knowledge
can also be used for other applications, such as improving site usability, business intelligence, and usage characterization. In this paper, we present initial work on web usage mining, describing the use of data mining techniques to analyze web log records collected from Publico On-Line [15], a web newspaper. Using commercial data mining software (SPSS Clementine [7] and IBM Intelligent Miner [9]), we have identified several web access patterns by applying association rules induction and clustering techniques to the access logs of this digital publication. The remaining of this paper is organized as follows: in section 2, we present the web usage mining concept. Our access logs processing architecture is presented in section 3. Then, in section 4, we present and analyze the results of the data mining work on Publico On-Line access data. Finally, section 5 summarizes our conclusions and presents directions for future work.
2
Mining Web Usage Data
Data mining efforts associated with the Web, called Web Mining, can be broadly categorized into three areas of interest based on which part of the Web to mine: web content mining, web structure mining, and web usage mining [10]. Web content mining focuses on techniques for searching the web for documents whose content meets web users queries. Web structure mining is related to the analysis of the link structure of the web, in order to identify relevant documents. Web usage mining is defined as the process of applying data mining techniques to the discovery of usage patterns from web logs data, to identify web users behavior [16]. Web content and structure mining are beyond the scope of this work, despite we used some sort of structure mining in our preprocessing phase. In Web mining, data can be collected at the server-side, client-side, proxy servers, or a consolidated web/business database. In [16], the authors present a more detailed description of these data sources. To summarize, (i) Web server logs explicitly records browsing behavior of site visitors, (ii) Client-side data collection can be implemented by using a remote agent or by modifying the source code of an existing browser (iii) and Web proxies act as an intermediate level of caching between client browsers and Web servers. The information provided by the data sources described above can be used to construct several data abstractions, namely users, page-views, click-streams, and server sessions [17]. A user is defined as a single individual that is accessing file Web servers through a browser. In practice, it is very difficult to uniquely and repeatedly identify users. A user may access the Web through different machines, or use more than one browser at one time. A page-view consists of every file that contributes to the display on a user’s browser at one time and is usually associated with a single user action such as a mouse-click. A click-stream is a sequential series of page-views requests. Note that any page view accessed through a client or proxy-level cache will not be recorded on the server side. A server session (or visit) is the click-stream for a single user for a particular Web
site. The end of a server session is defined as the point when the user’s browsing session at that site has ended. The process of Web usage mining can be divided into three phases: preprocessing, pattern discovery, and pattern analysis [16]. Preprocessing consists of converting usage information contained in the various available data sources into the data abstractions necessary for pattern discovery. Another task is the treatment of outliers, errors, and incomplete data that can easily occur due reasons inherent to web browsing. The data recorded in server logs reflects the (possibly concurrent) access of a Web site by multiple users, and only the IP address, agent, and server side click-stream are available to identify users and server sessions. However, it is important to notice that the data collected by server logs may not be entirely reliable because some page views may be cached by the user’s browser or by a proxy server. In a Web server log, all requests from a proxy server have the same identifier, even though the requests potentially represent more than one user. Also, due to proxy server level caching, multiple users throughout an extended period of time could actually view a single request from the server. The Web server can also store other kinds of usage information such as cookies, which are markers generated by the Web server for individual client browsers to automatically track the site visitors. After each user has been identified (through cookies, logins, or IP/agent analysis), the click-stream for each user must be divided into sessions. As we cannot know when the user has left the Web site, a timeout is often used as the default method of breaking a user’s click-stream into sessions. The next phase is the pattern discovery phase. Methods and algorithms used in this phase have been developed from several fields such as statistics, machine learning, and databases. This phase of Web usage mining has three main operations of interest: association (i.e. which pages tend to be accessed together), clustering (i.e. finding groups of users, transactions, pages, etc.), and sequential analysis (the order in which web pages tend to be accessed). The first two are the focus of our ongoing work. Pattern analysis is the last phase in the overall process of Web usage mining. In this phase the motivation is to filter out uninteresting rules or patterns found in the previous phase. Visualization techniques are useful to help application domains expert analyze the discovered patterns.
3
Access Logs Processing Architecture
Publico On-Line is a daily online newspaper. Each edition is constructed by a generation program that collects all articles, applies formats and constructs a navigable Web structure with articles grouped in thematic sections. We have defined a general architecture for web access mining (see Figure 1), using the site’s Web server logs as data source. The preprocessing phase includes initial preparation tasks that are included in a processing agent system [13]. This system performs the following tasks: noise filtering (i.e. removing irrelevant data like access errors or images requests),
Site Files
Server Logs
Preparation tasks: Data Cleaning User and Session Identification
FREQUENT ITEM SETS DISCOVERY
access repository
Session Files
Data Preprocessing Session Identification Format transformation Usage Statistics
STATISTICAL ANALYSIS: - data distribution estimation
CLUSTERING: - session clusters
Usage Mining
WEB ACCESS PATTERNS & USER PROFILES
Fig. 1. Overview of a general architecture for Web Access Mining.
sessions identification, and storage in a repository. Session identification consists of grouping all page-view records from a given IP address collected during user activity periods (we define inactivity as a period of 30 minutes or higher for which we have no registered accesses to the Web server). For each valid pageview (a news article) the agent assigns the corresponding news section based on site structure information present on the page’s URL. The conceptual schema of the repository is illustrated in Figure 2. Each article (artigo) is associated to one section (seccao) and a reader (cliente) accesses one or more articles during a session. The sessions identified by the processing agent as described above are called short sessions. When allowed by the user agent, the web server also registers a cookie that is accepted by that agent. We call long sessions to the set of short sessions that share the same cookie (accumulation of the user access transactions grouped by cookie). To adapt this data to the data structures of the data mining algorithms used, we transformed log access tables into numerical and Boolean matrices, where each column corresponds to a newspaper section and each row represents a session. In numerical matrices, each matrix cell contains the quantity of articles accessed on each pair (session, section); in Boolean matrices a cell is True when at least one article is accessed in that (session, section) pair. We examined the aggregated data matrices through a set of basic statistical functions that help in obtaining a preliminary view about the data. For numeric variables we have observed the maximum, minimum, mean, and standard deviation; for Boolean variables we obtained the frequencies (see Figure 3). These statistics show that the matrices are very sparse, that is, for each session we have a small number of articles and a small number of sections accessed. For example, 82.8% of the sessions do not have any accessed articles from the
Fig. 2. Conceptual schema of the access record repository: readers (cliente) access articles (artigo) during a session (sessao); each article belong to a section (seccao).
Science section, and in the remaining 17.2% we have an average of 2.3 accessed articles.
Name
Minimum Value
Maximum Value
Mean
Standard Deviation
Na me
Mo dal Va lue
CIENCIAS CULTUR A DES PORTO ECONOMIA INTERNACIONAL LOCAL_LISBOA LOCAL_PORTO POLITICA SOCIEDADE EDUCACAO
1 1 1 1 1 1 1 1 1 1
97 208 318 258 208 460 256 208 367 90
2.30343 3.78779 5.69846 3.93347 3.38226 5.68833 7.59835 3.35767 4.26733 2.64958
2.81841 5.97421 10.836 7.23418 5.55397 11.5647 13.2351 5.41012 7.9853 3.29088
CIENCIAS CULTURA DE SPOR TO EC ONOMIA EDUCAC AO INTERNAC IONAL LO CAL_LIS BOA LO CAL_PO RTO PO LITICA S OCIEDADE
F F F F F F F F F F
Mo dal Fre que ncy(%) 8 2.80 8 3.12 6 7.87 7 6.84 8 4.98 6 9.59 7 7.81 8 6.47 7 0.86 7 1.70
Fig. 3. Analysis of short sessions. The dominant values of both the numerical matrix (left) and the Boolean matrix (right) show that most users access a very small number of articles. Identical results were obtained for long sessions.
4
Mining Publico On-Line Access Data
To study the identification of associations between sections, we used typical data mining modelling operations. We view our problem of analyzing patterns of access to groups of news sections as a Market Basket Analysis problem [4].
Discovery of frequent itemsets is one of the techniques used in this kind of problem. It’s aim is to find groups of items that are frequently referred together in transactions. In our problem, transactions are the web accesses and items the news sections. 4.1
Discovering Frequent Itemsets
Groups of items occurring frequently together in many transactions are referred to as frequent itemsets [2]. Generally, a support threshold is specified before mining and is used by the algorithm for pruning the search space. The itemsets returned by the algorithm satisfy this minimum support threshold. We have identified frequent sets on Boolean data, defining weak associations as those below 5% of the total number of occurrences, and heavy associations as those above 10%. We have chosen these values based on a previous study [6].
Fig. 4. Discovery of frequent sets applied to short sessions. Strong associations have a heavy line.
Analysis of the results shows that strong associations on short sessions also exist on long sessions. This is an expected result, as long sessions accumulate accesses made in short sessions. For example, we have identified strong associations between Politics (Politica) and Society (Sociedade), Politics (Politica) and International News (Internacional), and between Society (Sociedade) and International News (Internacional), among other strong associations (see Figure 4). 4.2
Clusters Identification
Groups of sections obtained by frequent itemsets analysis gives us some interesting associations. However, it shows ”dependencies” among news sessions independently of the type of users preferences. Identification of groups of users with
identical preferences requires the extraction of different kind of access patterns. We searched groups of sessions (clusters) that were similar in the sections accessed. We had two approaches for clustering available on the used data mining tools: demographic clustering (based on Euclidian similarity metrics) and neural clustering (namely Kohonen self organizing maps) [9]. Figure 5 shows the largest clusters obtained in the analysis of numeric and Boolean short and long sessions, using both approaches.
Session type
Measure data type
Demographic Clustering
numerical
61% all sections except International
short boolean numerical long boolean
14% Sports
13% Internat.
75% all sections except Science 13% Internat.
12% Sports
Neural Clustering 15% Economy, Science LocalPorto, Educat. 15% Sports 12% Internat. 14% Sports
13% Internat. 12% Sports 13% Internat.
Fig. 5. Largest clusters for short and long sessions in Publico access log data.
Clustering on numerical data shows no evidence of clear reading patterns. We suspect that we have a very significant number of irregular sessions (outliers) with sporadic accesses without a defined pattern. This issue will be studied in future research. Both approaches for clustering Boolean data show similar reading patterns in short and long sessions. The most frequent clusters are those that group the accesses to Sports and International sections.
5
Conclusions and Future Work
In this paper we discussed the application of data mining technology to the analysis of access log records collected from a newspaper web site. Using commercial data mining software systems, we have identified and characterized several reading patterns within the news site. These patterns will define user profiles which integrate a news recommendation system based on web user preferences. Frequent sets and clustering produce different patterns. Frequent sets show groups of sections that are more frequent together, independently of the user profiles, and clustering show groups of sections that define similar web usage. Clustering of Boolean and numerical data lead to different results. While for Boolean data results are similar in both kinds of sessions and clustering approaches, we obtained different reading patterns for numerical data, or no reading patterns at all. We detected that a very significant number of sessions consist of a single page-view referred from a site external to the online newspaper. This may explain why we were able to identify patterns in Boolean data and had more
difficulty when dealing with numerical data. This suggests that to find more interesting patterns it is necessary to remove these sessions from the repository. The clustering results on numerical data may also be an outcome of the Euclidean distance-based similarity measures that are not adequate for mining our web access data. Previous research indicated that access data to digital libraries follows a Zipf-like distribution [14]. Commonly used clustering algorithms such as K-means, were developed for data samples from gaussian populations [3]. As future work, we plan to study more appropriate methods for analyzing web log data, using different similarity metrics (Minkowski distances, cosine measure and extended Jaccard similarity), and taking account the data distribution function.
References 1. Access Log Analyzers, http://www.uu.se/Software/Analyzers/Accessanalyzers.html 2. R. Agrawal, R. Srikant, Fast Algorithms for Mining Association Rules, Proc. of the 20th VLDB Conference, 1994. 3. P. S. Bradley, U. M. Fayyad, Refining Initial Points for K-Means Clustering, Proc. of the 15th International Conference on Machine Learning, Morgan Kaufmann, 1998. 4. M. Berry, G. Linoff, Data Mining Techniques - For Marketing, Sales and Customer Support, John Wiley & Sons, 1997. 5. Brian F. J. Manly, Multivariate Statistical Methods, Chapman & Hall, 1986. 6. P. Batista, M. Silva, Prospec¸c˜ ao dos Dados de Acesso a um Servidor de Not´ıcias na Web, 2 Conferˆencia sobre Redes de Computadores, Portugal, Outubro 1999. 7. Clementine User Guide, Version 5, Integral Solutions Limited, 1998. 8. R. Cooley, B. Moshaber, J. Srivastava, Data Preparation for Mining World Wide Web Browsing Patterns, Knowledge and Information Systems, 1(1), 1999. 9. Using the Intelligent Miner for Data, IBM Corporation, 1998. 10. R. Kosala, H. Blockeel, Web Mining Research: A Survey, SIGKKD Explorations, 2(1), July 2000. 11. B. Mobasher, H. Dai, T. Luo, N. Nakagawa, Y. Sun, J. Wiltshire, Discovery of Aggregate Usage Profiles for Web Personalization, Proc. of the Web Mining for E-Commerce Workshop (WebKDD’2000), August 2000. 12. B. Moshaber, R. Cooley, J. Srivastava, Automatic Personalization Based on Web Usage Mining, Communications of the ACM, 43(8), 2000. 13. N. Maria, P. Gaspar, N. Grilo, A. Ferreira. M. Silva, ARIADNE - Digital Library Architecture, Proc. of the Second European Conference on Research and Advanced Technology for Digital Libraries, Springer, 1998. 14. J. E. Pitkow, Summary of WWW Characterization, WWW Journal, 2(1), 1999. 15. Publico On-Line, http://www.publico.pt 16. J. Srivastava, R. Cooley, M. Deshpande, P.-N. Tan, Web Usage Mining: Discovery and Applications of Usage Patterns from Web Data, SIGKKD Explorations, 1(2), Jan 2000. 17. WWW Committee Web Usage Characterization Activity, http://www.w3.org/WCA, Web Characterization Terminology & Definitions Sheet, W3C Working Draft, May 1999. 18. O. R. Zaiane, M. Xin, J. Han, Discovering Web Access Patterns and Trends by Applying OLAP and Data Mining Technology on Web Logs, Proc. of Advances in Digital Libraries Conference (ADL98), April 1998.
