Automated vs. Manual Classification: A Multi-Methodological Set of Web Analysis Components Christian Bauer* Alan Parkinson* Arno Scharl** *
Department of Information Systems Curtin University of Technology Perth, Western Australia
[email protected] [email protected]
**
Department of Management Information Systems Vienna University of Economics and Business Administration Vienna, Austria
[email protected] Abstract We describe the design and initial development of a tool to automatically classify Web information systems. The applications of such a tool could be found both at the macro-level of economic observation and the micro-level of industry and competitive environment analysis. Components of the tool include mirroring, script-based feature extraction, and five different analysis techniques. The methodological framework comprises statistical clustering, unsupervised neural networks, supervised neural networks, textual analysis, and manual classification. Preliminary results are presented for the prototype implementation and prove encouraging. Even at this early stage they provide a concrete response to open questions about how to capture and analyse the content of WIS, its structure, and its dynamic change. Keywords World Wide Web, Web Information Systems, Classification, Clustering, Automated Tools, Content Analysis
INTRODUCTION Since the first public appearance of the World Wide Web in 1991, the deployment of publicly available, commercial Web Information Systems (WIS) has become one of the cornerstones of corporate communication strategies. Despite the substantial investments and the strategic importance, the manageability and measurement of WIS success has not been satisfactory. The ultimate goals behind this research are the development of autonomous software tools to capture the characteristics of WIS, store them in a database, determine their specific importance, and build a categorisation based on their attributes. The utilisation of dedicated software components to examine WIS is more efficient and immune against inter- and intrapersonal variances than traditional human evaluation. Thus the inclusion of thousands of WIS becomes feasible compared to tens (Selz et al., 1998; 1999) or hundreds (Psoinos and Smithson, 1999; also compare http://www.excite.com) in previous efforts. Naturally, these advantages come at the expense of sacrificing non-quantifiable, frequently recipientdependent information (“soft” attributes).
Proc. 10th Australasian Conference on Information Systems, 1999
54
Generally, a distinction between manual and automated approaches to analyse (positivistic aspect) and evaluate (normative aspect) on-line resources has to be pointed out. In most cases, manual evaluation relies on the judgements of individual analysts regarding certain WIS. Even if a group of experts is involved, the evaluation takes place with varying degrees of subjectivity regarding the process itself and the captured data structures. Rigorous structure is assured when evaluation is done automatically with software tools. However, this advantage of consistency may also lead to reduced relevance for specific domains of knowledge. Application Areas Of An Automated Classification Tool If successful, this research will provide a consistent analysis and evaluation framework of publicly accessible hypertext structures. Both the framework and the software prototype imply the definition of measurable, operational criteria. These criteria will be investigated and preprocessed by the tool for each WIS separately. Possible applications of such a tool fall into three areas: •
Snapshot analysis (static): Analysis of a large number of WIS at a given time allows comparison of individual criteria (e.g., means, variances, or other statistical parameters) and the clustering of WIS, for example by industry, by company size, or by provided support for the different phases of electronic market transactions (compare Schmid and Lindemann, 1998).
•
Longitudinal analysis: A defined set of WIS can be documented and analysed over a longer period of time; i.e. a series of snapshot analyses are conducted on the same sample. Trends, turning points and marginal changes are monitored for specific industries and other sub-categories. These complex developments can hardly be observed and quantified in a traditional way. Nevertheless, the generated indicators are required by both industry and academic research.
•
Comparative analysis: The analysis of their own industry and competitors' efforts provides decision makers with reliable benchmark data about relevant differences and the relative performance of their own WIS. Benchmarks indicating weaknesses should trigger and guide the immediate redevelopment and optimisation of deployed WIS. Comparative analysis as a feedback channel is envisaged as part of a tool-based methodology for the evolutionary development of WIS, which can be seen as four sequential phases arranged in an iterative cycle: Design, implementation, usage, and analysis (Bauer and Scharl, 1999). The WebAnalyzer as an effective means to assess and compare the competitive situation and user perception provides invaluable help for decision-makers responsible for WIS design and implementation.
PREVIOUS CLASSIFICATION APPROACHES Several WIS classification schemes have been previously introduced. Many of these classification frameworks lack a clear and consistent distinction between identified categories as many of these approaches are based more on anecdotal than empirical evidence. Most of the content-based classification schemes rely on an underlying model of attributes, which are hard to operationalise on a numeric scale. Available automated tools are rarely designed for a general classification model and are not yet ready for a comprehensive coverage of WIS. The majority of past efforts are driven by scholarly desire for typology. Many of them have a strong foundation in advertising and are utilised as guidelines for strategic decision making and for defining the marketing goals of on-line businesses. Hoffman et al. (1995), for example, identify six distinct categories for commercial WIS based on a functional typology: •
On-line Storefronts offer direct sales, electronic catalogues, and substantial customer support.
55
•
An Internet Presence can take the form of flat ads (single documents), image sites (emotional consumer appeal), or detailed information sites.
•
Content Sites are characterized by their funding model, which can be either fee-based, sponsored, or a searchable database.
•
Electronic Malls typically feature a collection of on-line storefronts.
•
Incentive Sites make effective use of the specific technical opportunities of the World Wide Web for advertising purposes.
•
Search Agents identify other WIS.
A more generalised commercial WIS classification with similar intentions, but a mutually exclusive typology, has been presented by Hansen et al. (1996). It comprises two dimensions, content and interactivity, and defines five distinct types of WIS (Electronic Business Cards, Advertainments, Electronic Brochures, Electronic Catalogues and Web-Services) in the context of these dimensions. Similar schemes can be found for industry-specific WIS classification and have been applied in the retail banking industry for building reference models (Bauer, 1998), observation of industry evolution (Mahler et al., 1996), and commercial WIS analysis (e.g. Booz-Allen and Hamilton, 1998). With regard to WIS adaptability, Scharl and Brandtweiner (1998) offer a more technology-oriented classification scheme and distinguish between five different levels of Web-based customization (also compare Hansen and Tesar, 1996; Bauer and Scharl, 1999a; 1999b). For commercial scenarios, Selz et al. (1997; 1998; 1999) propose a Web assessment model to identify and evaluate successful commercial applications. While the resulting model offers a detailed analysis of these solutions, the time-consuming method requires access to company information frequently not available. So far, automated methods for the classification of Web sites appear to be taking a development-oriented approach. On the one hand, the use of meta data promises to resolve some of the problems with the information overflow that is characteristic for the Internet. Semantic labels – e.g., Platform for Internet Content Selection (PICS); http://www.w3.org/PICS/ – appear to be the most promising technology for content determination. The Extensible Markup Language (XML; http://www.w3c.org/xml/) as a semantic markup languages promises to resolve confusion caused by the mixture of content, navigational and structural information in HTML (Hypertext Markup Language) markup. On the other hand, several tools try to assure the quality of a particular Web site by investigating its structure. Such tools range from relatively simple approaches like the HTML Validation Service (http://validator.w3.org) of the World Wide Web Consortium (W3C) to more sophisticated on-line tools like the Web Site Garage (http://www.websitegarage.com), FritzService (http://www.fritz-service.com), or Bobby (http://www.cast.org/bobby) with multidimensional ratings of investigated WIS. Right from the beginning of the World Wide Web, search engines and meta-indices were categorising Web sites in URL collections, e.g., Yahoo! (http://www.yahoo.com) or Excite (http://www.excite.com). Various criteria are employed including subject and geographical area. The perceived value of such collections can be further improved by adding detailed reviews of indexed systems.
CLASSIFICATION PROCESSES AND METHODS This paper describes an automated, tool-based approach to analyse a wide variety of Web Information Systems. The classification process is broken down into three clearly defined phases as presented in Figure 1 (to be read bottom-up: stages A, B and C): A) WIS Mirror: Automatically downloads HTML files from publicly accessible WIS and stores the retrieved information in a local database.
56
B) WebAnalyzer: Extract structural information and other attributes from the mirrored files and produces a numeric record (vector) for each WIS. It also provides the content information from the complete WIS in a meaningful format for the textual analysis. C) Classification Methods: Five different methods have been identified as being particularly suitable for classification purposes in this context: Statistical validation, unsupervised learning in neural networks, supervised learning in neural networks, textual analysis and manual classification.
Figure 1: Mirroring (A) and analysing (B) the WIS are prerequisites for the subsequent evaluation methods (C) Each of the three phases has a clearly defined input and output interfacing the user and subsequent phases. In reality, the three phases can be run rather independently from each other and future research may choose to employ a different implementation of one particular component without affecting the other phases. Alternatively, the five suggested methods in phase C may be added to or reduced as appropriate for different application domains.
57
Phase A: WIS Mirror Currently, a limited number of WIS are mirrored to a CD-ROM archive with the tool HTTrack (http://www.ensicaen.ismra.fr/~Eroche/httrack.html) running on a Linux operating system. For the requested sites, which are to be included into the classification, a short descriptor (usually the organisation’s name), the publicly accessible URL and the local subdirectory for the mirror location have to be specified. Upon successful completion HTTrack creates the requested files as well as an event- and error-log. For the purpose of this research, the following mirroring options were chosen for the command line interface: -
p1: Save only HTML files.
-
R5: Relative Links.
-
c10: Number of multiple connections (10).
-
I0:
-
C0: No cache.
-
M10485760: Maximum overall size that should be downloaded (10 Megabytes).
-
m100000,100000: Maximum file length for non-HTML and HTML files.
Do not make an index.
The limitation to a maximum of 10 Megabytes facilitates the comparison of WIS of different size and is a necessary condition to manage the available local storage space. Phase B: Web Analyzer The WebAnalyzer is currently available as a prototype, which recursively parses all HTML files in the subdirectories of the mirrored site for HTML markup tags. All tags related to the variables in Figure 2 result in an update of the record for this particular WIS. The list of variables in Figure 2 is directly derived from the identification of three primary classification criteria: content, interactivity, and navigation. The listed criteria try to advance previously introduced classification schemes. Specifying these criteria is the first step towards any consistent framework based on empirical evidence (in contrast to conceptual frameworks). Considering technical feasibility, the three categories of Figure 2, which are characterised by varying degrees of measurability, evolved from reviewing previous approaches and exploratory case studies. The WebAnalyzer has been implemented in a Linux environment on a Perl-5 interpreter. Input and output into the WebAnalyzer are managed through commadelimited text files, a file format that ensures interoperability with most applications. Additionally, the textual content of the WIS is written into one single ASCII file for further processing with the WordSmith tool set (see below).
58
Criteria Content
Variables Criteria # Documents Navigation [total] Kbytes downloaded [total | text only] # File Types [distinct extensions] # Images [total | distinct] Interactivity # Forms [total | distinct | fields] # Documents with JavaScript [total] # Java Applets [total | distinct] # MailTo-Links [total | distinct]
Variables Frames [yes/no] # Internal Links [total | distinct | broken] # External Links [total | distinct | broken] # Anchors [total | distinct | broken] # Links to Anchors [within | between documents]
Figure 2: Structural classification criteria for the WebAnalyzer Phase C: Classification Methods Several classification methods can be identified, which may complementarily contribute to a successful classification of commercial or non-profit WIS (compare Figure 1). Each of these methods has certain inherent strengths and weaknesses, suggesting a multi-methodological approach. Additionally, the comparison of results allows for assessment of the validity of each individual method. Two issues can be distinguished here: establishing which categories are appropriate for the data, and subsequently assigning new WIS to the set of established categories. Alternatively, the category sets could be updated on the occurrence of new WIS that do not fit into the existing classification. Unsupervised Learning In Neural Networks This is our first choice for determining the appropriate categories. Following early results (Kohonen, 1988), which successfully identified phonemes from continuous speech, the selforganizing map has been used for such diverse purposes as estimation of real estate values (Carlson, 1991) or process control (Hyotyniemi, 1993). The Kohonen algorithm effectively projects the original data onto a subspace of (usually) two dimensions, such that similar data are placed close to one another. This representation is referred to as a topographical map. The topographical map leads to the clustering of categories by grouping together neighbouring WIS. In contrast to supervised neural networks, the Kohonen algorithm does not require the definition of categories and pre-assigned WIS. Statistical Methods There are a number of standard statistical techniques for identifying clusters within data, and we make use of these as a way of validating the neural network results. The Kohonen competitive learning algorithm has well documented similarities with more conventional statistical techniques (see Ueda and Nakano, 1994), so we would expect substantial agreement between the clusterings obtained by the two methods. For the purposes of this preliminary investigation, we used the K-means clustering technique. The K-means algorithm usually requires input factors with interval- or ratio-scales and advance specification of the desired number of categories and outputs cluster centres and group memberships.
59
Supervised Learning In Neural Networks Once clusters have been established, it becomes feasible to develop a neural network using a supervised learning algorithm such as standard backpropagation or its more advanced variations. Although a self-organizing map can be used for classification into previously established clusters, they are not optimal for that application (Kohonen, 1991). A properly trained feed-forward network would be expected to be faster and/or more accurate. Hybrid approaches, which combine the adaptive features of neural networks with the modeling flexibility of fuzzy logic, have become increasingly popular. Especially for non-numeric classification criteria this seems to be a promising alternative and would eventually allow the integration of manual and automated classification methods. This is of particular interest with regards to the textual and manual analysis (see below), which require a much more resourceconsuming classification process. Therefore, the authors are currently in the process of testing the domain-specific suitability of a neurofuzzy-shell (FuzzyTech 5.01; Inform, 1997), which employs a supervised neural component to specify the rules and membership functions of the fuzzy inference engine. Textual Analysis Exploratory textual analysis is based on linguistic units like words or lemmas (Aarseth, 1994; Pearce and Miller, 1997). At the moment, the commercially available tool “WordSmith” developed by Mike Scott (http://www.liv.ac.uk/~ms2928/index.htm) is used for processing the textual output of the WebAnalyzer (see screenshot in Figure 3). From a linguistic point of view, analysing WIS is mainly concerned with syntax (relationship between words in sentences), semantics (meaning of the words), and pragmatics (associations between utterances and the circumstances of communicating), neglecting other established linguistic fields like phonetics, lexicology, or morphology. Automatic indexing and frequency computations ignore information of semantic or syntactic nature that usually is available to any reader – e.g., synonyms (two strings that have identical meaning), homonyms (one string that has more than one meaning), or the particular order of words within a text (Lebart, 1998). On the basis of this word list, hierarchical cluster analysis and correspondence analysis are used to visualize the emergence and decay of topics of interest, both for longitudinal studies and for industry-based comparisons.
Figure 3: Screenshot of The WordSmith Tools 3.0 (http://www.liv.ac.uk/~ms2928/index.htm) Manual Classification Any manual classification will be more content-oriented and domain-specific than automated evaluations. The classification scheme in Figure 4 was developed for WIS of environmental
60
non-profit organisations and that orientation is reflected in its attributes. The first attribute strategy - is based on the organisation's approach of how to change the world, either aggressively by putting pressure on industry, or cooperatively by seeking mutual beneficial situations and joint projects. As far as the organisation's goal is concerned, informational WIS have to be distinguished from motivational ones, which are mainly concerned with activating the reader. Not surprisingly, interactivity plays a crucial role as well, especially for motivational WIS. Independent of the organisation's goal, the wealth of provided information as well as its (graphical) appearance represent two additional criteria to classify environmental WIS. Last, but not least, the structure of an organisation - i.e., funding or control as part of a larger organisation, via corporate sponsoring, or independently via direct membership programs - are directly reflected in the content of the WIS and also have some indirect influence on the other dimensions. We regard the manual classification approach as a temporary expedient to assist in validating the other clustering methods and to relate the results to meaningful properties of the application domain. As a long-term strategy, manual classification suffers from the shortcomings of being extremely labour-intensive and essentially subjective in nature. As outlined above, the application of supervised neural networks may solve the dilemma between inclusion of qualitative data analysis and assessment efficiency. Strategy
AGGRESSIVE
NEUTRAL
COOPERATIVE
INFORMATIONAL
BALANCED
MOTIVATIONAL
Interactivity
LOW
MEDIUM
HIGH
Wealth of Information
LOW
MEDIUM
HIGH
Appearance
AMATEUR
SEMIPROFESSIONAL
PROFESSIONAL
Organisational Structure
AFFILIATED
SPONSORED
INDEPENDENT
Goal
Figure 4: Classification Scheme of Environmental WIS
PRELIMINARY RESULTS At the early stages of research and system development, the efforts have been concentrated on the architecture of the tool, and on preliminary efforts to establish suitable pre-processing techniques and identification of appropriate clusters in the data. Consequently, the emphasis of this paper has been on the research process and potential applications. Results for supervised training of neural networks and textual analysis have to be omitted at this stage. As noted earlier these methods depend on results from the other methods, and would therefore be too speculative at this point of time. Unsupervised Learning In Neural Networks With some rare exceptions, the input data for neural networks will need pre-processing. The Kohonen network, for example, requires normalised inputs to function correctly (Parkinson and Parpia, 1998). Accordingly, the measures from WebAnalyzer were first scaled down to the range 0 - 1 (independently for each measure), then the vector for each site was normalised to a Euclidean length of one. The same pre-processed data was used for the statistical analysis (since K-means clustering has similar requirements). The publicly available (ftp://ftp.informatik.uni-stuttgart.de/pub/SNNS/) Stuttgart Neural Network Simulator (SNNS) was used to test this part of the system. It is capable of simulating
61
a wide variety of neural networks, including a number of recurrent and feed-forward types. For this study the Kohonen SOM implementation was applied to the clustering problem. Using the normalised data the optimisation of the clustering process required many trials varying the size of the resulting map as well as various learning parameters. The clustering obtained was robust in the face of these changes, which serves as an indication of some stability in the data. A total of eight clusters (with a possible ninth consisting of a single site) were identified. Statistical Validation The K-means clustering method in SPSS Version 8 (set to eight clusters in accordance with the previous, unsupervised neural network interpretation) was employed to verify the neural network results. As expected on theoretical grounds, there was close agreement between the two methods, with only a few borderline sites moving from one cluster to another. In all such cases, the clusters concerned were adjacent on the Kohonen topological map. Manual Classification Manual classification was performed using the dimensions described above by one of the authors. This assessment was done by accessing the original sites (the mirror does not contain multimedia data). It would seem that some refinement of the five dimensions used might be required. Although clusters were identifiable (using K-means) they did not seem very robust and corresponded only poorly with those obtained by structural analysis. These preliminary results indicate that there may be some predictive ability in the structural clusters but it is too early to say if the effect will be strong enough to be useful in practice. The prognostic power of a supervised neural network with the (quantitative) structural information from the WebAnalyzer as the input vector and the (qualitative) manual parameters as the output vector will be of particular interest and a validity test of the underlying assumptions of this research.
CONCLUSION AND FURTHER RESEARCH The exploratory analysis and evaluation approach described in this paper would enable developers and decision-makers to effectively track dynamic trends within and between different industries by repeatedly assessing thousands of WIS without human intervention. With the Kohonen self-organizing map as the inductive validation method, the common mistake of arbitrary definitions during the initial specification of categories is avoided and will lead to a more realistic model appropriate for Web information systems. At the same time, the linguistic component is used to visualize the emergence and decay of topics of interest. While fuzzy modeling maintains the transparency of the system's internal calculations, the neural component ensures maximum flexibility and continued optimization, eliminating the need for predefined and often questionable mathematical relationships. The preliminary results offer some encouragement as to the feasibility of the proposed architecture. Early components are now performing reliably. Our next goal is to tune the neural network clustering tools and to relate the clusters to some meaningful properties of the application domain by either refining the manual classification approach or (ideally) extending the textual analysis to the semantic domain.
REFERENCES Aarseth, E.J. (1994). Nonlinearity and Literary Theory. Hyper/Text/Theory. G. P. Landow. Baltimore, Johns Hopkins University Press: 51-86. Bauer, C. and Scharl, A. (1999a). Advanced Design Of Web Information Systems Based On Dominant And Emerging Web Communication Patterns. The Fifth Australian World Wide Web Conference (AusWeb'99), Ballina, Australia, 74-85.
62
Bauer, C., Glasson, B. and Scharl, A. (1999b). Evolution of Web Information Systems: Exploring the Methodological Shift in the Context of Dynamic Business Ecosystems. Doing Business On The Internet: Opportunities and Pitfalls. C. Romm and F. Sudweeks. London, Springer: In print. Bauer, C. (1998). Internet und WWW für Banken: Inhalte, Infrastrukturen und Erfolgsstrategien. Wiesbaden, Dt. Univ.-Verl., Gabler. Booz-Allen & Hamilton (1996). Consumer Demand for Internet Banking. Unpublished report, New York. Carlson, E. (1991). Self-organizing feature maps for appraisal of land value of shore parcels. In: Artificial Neural Networks, T. Kohonen, K. Maksira, O. Simula, J. Kangas (eds) Elsevier Science (North-Holland) 1991 Hansen, H.R. et al. (1996). Klare Sicht am Infohighway – Geschäfte via Internet & Co. Orac, Vienna. Hansen, H.R. and Tesar, M.F. (1996): “Die Integration von Masseninformationssystemen in die betriebliche Informationsverarbeitung”, “Data Warehouse”, Gerhard Mercator University Duisburg. n.a. Hoffman, D.L. et al. (1995). Commercial Scenarios for the Web: Opportunities and Challenges. Journal of Computer-Mediated Communication. 1 (3), http://jcmc.huji.ac.il/vol1/issue3/hoffman.html (11/98). Hyotyiemi H.(1993). Optimal control of dynamic systems using self-organizing maps. Proceedings of International Conference on Artificial Neural Networks (ICANN-93) Springer-Verlag, 850-853. Inform (1997). FuzzyTech 5.0 User's Manual. Inform Software Corporation, Aachen. Kohonen, T. (1988). The “neural” phonetic typewriter. Computer 21(30), 11-22. Kohonen, T. (1991). Self-Organizing Maps: Optimization Approaches In: Artificial Neural Networks, T. Kohonen, K. Maksira, O. Simula, J. Kangas (eds) Elsevier Science (NorthHolland) 1991 Lebart, L. et al. (1998). Exploring Textual Data. Dordrecht, Kluwer Academic Publishers. Mahler, A. and Göbel, G. (1996). Internetbanking: Das Leistungsspektrum. Die Bank. 1996 (8), 488-492. Parkinson, A. and Parpia, D.(1998). Intensity encoding in unsupervised neural nets 1. Neural Networks 11, 723-730. Pearce, C. and Miller, E. (1997). The TELLTALE Dynamic Hypertext Environment: Approaches to Scalability. Intelligent Hypertext - Advanced Techniques for the World Wide Web. C. Nicholas and J. Mayfield. Berlin, Springer. 1326: 109-130. Psoinos, A. and Smithson, S. (1999). The 1999 Worldwide Web 100 Survey. London, London School of Economics. http://is.lse.ac.uk/press_release.htm (05/99). Scharl, A. and Brandtweiner, R. (1998). A Conceptual Research Framework for Analyzing the Evolution of Electronic Markets. International Journal of Electronic Markets, 8 (2), 39-42. Schmid, B.F. and Lindemann, M.A. (1998). Elements of a Reference Model for Electronic Markets. 31st Hawai'i International Conference on System Sciences (HICSS-31), Hawaii, USA, IEEE Computer Society Press.
63
Selz, D. and Schubert, P. (1997). Web Assessment – A Model for the Evaluation and the Assessment of successful Electronic Commerce Applications. International Journal of Electronic Markets. 7 (3), 46-48. Selz, D. and Schubert, P. (1998). Web Assessment – A Model for the Evaluation and the Assessment of Successful Electronic Commerce Applications. 31st Hawai‘i International Conference on System Sciences (HICSS-31). Volume IV: Internet and the Digital Economy. IEEE Computer Society, Los Alamitos, 222-231. Schubert, P. and Selz, D. (1999). Web Assessment – Measuring the Effectiveness of Electronic Commerce Sites Going Beyond Traditional Marketing Paradigms. Proceedings of the Thirty-Second Annual Hawaii International Conference on System Sciences (HICSS-32), Maui, Hawaii, USA, IEEE. Ueda, N. and Nakano, R. (1994). A New Competitive Learning Approach Based on an Equidistortion Principle for Designing Optimal Vector Quantizers. Neural Networks 7(8), 1211-1227.
COPYRIGHT Christian Bauer, Alan Parkinson, Arno Scharl 1999. The authors assign to ACIS and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ACIS to publish this document in full in the Conference Papers and Proceedings. Those documents may be published on the World Wide Web, CD-ROM, in printed form, and on mirror sites on the World Wide Web. Any other usage is prohibited without the express permission of the authors.
64
