website. We introduce the proposed architecture and its applicability across separately hosted open-source Web-based. Content Management Systems (WCMS) ...
2013 IEEE/WIC/ACM International Conferences on Web Intelligence (WI) and Intelligent Agent Technology (IAT)
Towards Cross-Site Personalisation Kevin Koidl, Owen Conlan, Vincent Wade, Knowledge and Data Engineering Group (KDEG), Trinity College Dublin, School of Computer Science, Dublin, Ireland {Kevin.Koidl, Owen.Conlan, Vincent.Wade}@cs.tcd.ie
(1) To assist the user across different websites in a non-intrusive manner by retaining the users browsing freedom.
Abstract—Personalisation on the web is mostly confined to websites of online content providers. The main drawback of this approach is the missing consideration of the users previous crosssite browsing experience resulting in an often fragmented browsing experience. This paper introduces a service driven architecture for user-centric personalisation in online cross-site tasks. The goal of the architecture is twofold: (1) to provide nonintrusive personalised recommendations to the user by not interfering in their browsing freedom and (2) to introduce crosssite personalisation in a non-invasive manner, thus not interfering with the design and functionality of the overall website. We introduce the proposed architecture and its applicability across separately hosted open-source Web-based Content Management Systems (WCMS). We also discuss encouraging results of an initial task-based experiment concentrating on the user’s perception of non-intrusive personalisation within and across different websites.
(2) To apply cross-site personalisation in a noninvasive manner by limiting the impact on the design and functionality of the website. The remainder of the paper is structured as followed: Section 2 discusses related work in cross-site personalisation, which is followed by an outline of the service based UNITE (UNIfied Task-based browsing Experience) architecture with the goal to provide cross-site personalisation. Finally implementation details and the results of an initial evaluation are provided. II.
In recent years several research groups have introduced different approaches addressing user-centric personalisation within and across different websites. Among others these are Adaptive Hypermedia Systems and Web-based Content Management Systems.
Keywords—Personalized Websites, Cross-System Personalization, Adaptive Web Interfaces, Adaptive Hypermedia, Content Management Systems
I.
INTRODUCTION
A. Web Recommender Systems Web Recommender Systems (WRS) guide users to artifacts that are related to resources the user is currently interested in, such as related web resources or items [1]. The majority of WRS can be categorized in collaborate recommender systems and content-based recommender systems. The former generates recommendations based indicated interest of other users such ratings and purchase. The latter uses mostly keyword-based algorithms to identify artifacts related to the current resource of interest. In recent years content-based recommendation systems have been introduced, addressing a more open and cross-site browsing experience ranging from browser side plug-ins such as Letizia [2] to proxy-server based approach such as Quickstep [3], an on-line research papers recommender. Other examples are Personal Web Watcher [4], WebMate [5] and Amalthaea [6].
Personalisation on the web is mostly confined to content provider websites and not cross-site. It can be argued that the main reason for this shortcoming is the need of content providers to encourage users to remain within their website as long as possible. The expected outcome of a prolonged user visit is an increase in revenue either during the visit or in future visits through intensifying customer loyalty. The user on the other hand benefits from website specific personalisation by receiving personalised recommendations, such as purchase related items, that may be of interest to the user. However this approach cannot assist the user in online tasks, which require cross-site browsing such as exploring product related information across enterprise and user generated websites. The result of this fragmented browsing experience can increase user frustration through repetitive query usage within the different websites or misaligned recommendations by not reflecting that do not reflect the overall cross-site browsing experience of the user. To unify the fragmented browsing experience both the need of the user (freely browsing across the web) and of the content provider (encouraging the user to stay on the website as long as possible) has to be addressed.
B. Content-based Recommender Systems For content-based recommendation systems to work effectively the elicitation of content-based terms is essential. It allows the creation of recommendations related to the current resource of interest. To identify relationships between resources the most common approaches are keyword based text analysis and ontology based semantic analysis. In addition
This paper discusses a third-party service approach addressing both needs by concentrating on following twingoals: 978-1-4799-2902-3/13 $31.00 © 2013 IEEE DOI 10.1109/WI-IAT.2013.76
REALTED WORK
542
recently folksonomy and social web related approaches have emerged providing tag based keyword clouds for mostly unstructured User Generated Content (UGC) [1]. Keyword based approaches are useful in matching resourced based on their syntactical dimension e.g. by using cosine similarity (TFID) or Bayesian classifiers. However keyword-based approaches fail in addressing semantic challenges, such as the same keyword related to different topics e.g. the term keyboard could relate to content about a musical instrument or a computer device. Common approaches in semantical analysis are the use structured ontologies to train text classifiers or use extended knowledge sources such as WordNet1 or DBPedia2 to identify meaning based on the contextual relationships of the extracted keywords [7].
only consider service side considerations. For cross-site personalisation to be applied on the open web client-side considerations need to be taken into account. The discussed cross-site personalisation approach facilitated by the UNITE architecture extends the notion of AFS by addressing client side considerations; thus the needs of the users and of the content providers likewise [15]. III.
However, even though content-based recommender systems have proven successful in specific domains such as music, news and print media they show clear limitations assisting complex tasks common in learning or knowledge acquisition. It can be argued that the main reason for these shortcomings is that most recommender systems rely purely on content related tags. However, complex tasks require a more user-centric approach to avoid recommending resources that may not be suitably to the current knowledge state of the user. C. User-centric approaches User-centric approaches have been introduced by several Adaptive Hypermedia Systems (AHS). In comparison to content-based approaches the approach taken by AHS is to incorporate different models reflecting the user, the overall content domain and adaptive strategies such as device or location specific information [8].
A. Term Identification Service: The main purpose of the Term Identification Service is to identify terms related to the current webpage the user is viewing. Terms can either be retrieved directly from the interfacing website (e.g. though an existing taxonomy or folksonomy) or through term extraction tools such as Yahoo JQL table3. In a second stage the extracted terms can be used to receive related terms through external knowledge services such as WordNet4 or openCalais5. Once an initial term-based taxonomy is created the service can train text analytics tools such as Weka6 to create related taxonomy terms for websites which either have no related terms or for which the terms are not in the scope of the previously collected terms. B. User Model Repository: The User Model consists of terms related to the current task of the user and that were identified by the Term Identification Service. The User Model can be enriched with properties from external User Modeling Services such as preferred content type or language. C. Strategy Repository: Depending on the task and User Model specific properties the Strategy Model Repository can identify suitable strategies to create cross-site personalised recommendations. An example would be the use of different strategies depending on the device the user is currently.
Even though AHS have proven successful, especially in complex learning related tasks [9], it s to be argued that the complex interplay of the various different models have lead to Adaptive Hypermedia Systems which need to consider the adaptive features during design-time; thus limiting their usability in broad and dynamic information spaces such as the open web. This limitation has been identified by several AH research groups as open corpus problem and addressed by introducing more flexible and distributed AHS [10]. Among others successful distributed AHS are APeLS, Personal Learning Assistant (PLA) [11], AHA! [12] and ADAPT² [13]. The main advantage of distributed AHS is the outsourcing of responsibility to different services, such as User Modeling services and Domain Modeling Services. However in most distributed AHS the individual services are tightly coupled and designed specifically to provide adaptively within specifically designed portals. Recently more flexible approaches have been introduced defined as Adaptive Functionality Service (AFS) [14] concentrating on providing adaptive recommendation through flexible interfaces to external services that do not have to be considered during design-time. However, even though AFS are a promising approach to provide cross-site personalisation it has to be argued that they
3
http://developer.yahoo.com/search/content/V1/termExtraction.html http://wordnet.princeton.edu/ 5 http://www.opencalais.com/ 6 http://www.cs.waikato.ac.nz/ml/weka/ 4
1 2
THE UNITE ARCHITECTURE
The UNITE architecture is designed to assist users in complex online tasks which are short-term and cross-site, such as in Online Customer Care (e.g. solving PC security issues) through a third-party Adaptive Feature Service (AFS) based architecture. This is achieved by utilizing a unified term space based on the cross-site browsing experience of the user. The main advantage for the user of choosing a service and not client side approach is to receive personalised cross-site recommendations both browser and device agnostic. The advantage for the content provider is a central interface to request personalised cross-site recommendations. The overall architecture utilizes a number of service side repositories and services as well as an extendible interface layer and client side Web-based Content Management Systems modules (Figure 1).
http://wordnet.princeton.edu/ http://dbpedia.org/
543
Figure 1 Architecture for Unified Task-based browsing Experience (UNITE) UNITE service and different website clients is based on the aforementioned RESTful architecture and implemented through HTTP GET and HTTP POST. Clients can use the HTTP GET method to request resources such as terms related to the previous browsing experience of the user to provide cross-site personalisation. The HTTP POST method is used to send terms related to the current content viewed by the user back to the service.
D. Scrutiny Interface: To ensure user trust in the recommendations provided the user view all models and adaptive decisions. E. RESTful Service Layer: The UNITE architecture implements a RESTful service layer for client communication. F. WCMS Module Extensions: Based on the mostly flexible and simple extensibility mechanisms of Webbased Content Management Systems different models are offered to be deployment at run-time to enable cross-site personalised recommendations. IV.
The terms sent to the service are added via the Term Identification Service to a unified term space. In order to ensure that the terms in the taxonomy are unified the service may request webpages to extract further terms. Once the service has created an initial taxonomy text analytic tools such as Weka can be trained to provide a statistical model assisting term unification across the browsing space of the user. The Term Identification Service implements a Jena Semantic Framework (v.2.63)9 exposing a SPARQL based API. This API is used by the User Model Repository to create a user specific term based model representing the unified term space of the user related to the current tasks. In addition the User Modeling Repository can interface with third-party User Modeling Services to receive properties such as preferred content type or language preference. The repository stores the user model in a mySQL database10. Finally the Strategy Repository, which implements as JBoss Drools11 Business
IMPLEMENTATION
The UNITE architecture is base on the Representational Stare Transfer (REST) architecture. The implementation is divided in service and client side. A. Server Side Implementation On the service side several different repositories and services interface with the architecture. A central piece is the UNITE service which is the central access point and which manages the interaction between the different services and repositories. It is implemented as RESTful service based Java based JAX-RS 1.1 (Jersey) library7 and deployed on a Glassfish application server8. The interface between the
9
http://jena.sourceforge.net/ http://www.mysql.com/ 11 http://www.jboss.org/drools
7
10
http://jersey.java.net/ 8 http://glassfish.java.net/
544
V.
Logic Rule Engine to manage the creation of cross-site personalisation recommendations.
EVALUATION
The goal of this evaluation was to assess both the perceived usefulness of non-intrusive cross-site recommendations and the level of invasiveness in introducing cross-site recommendations in websites during run-time.
B. Client Side Implementation To investigate the perceived usefulness of non-intrusive cross-site recommendations different client-side implementations were investigated. The goal of this investigation was to identify website platforms that allow the adding of cross-site personalisation in a non-invasive manner in run-time avoiding re-designing or re-deployment [15]. WebSystems fulfilling these requirements are Web-based Content Management System (WCMS) such as Drupal, Mediawiki, Plone and Wordpress. The main advantages in relation to cross-site personalisation are that most WCMS provide simple extensibility mechanism allowing third-party services to influence the WCMS in a non-invasive manner. Based on its current popularity and simple extensibility mechanism the Drupal v.6.1412 WCMS was chosen.
The evaluation was based on real world Online Customer Care tasks related to the Symantec's Norton 360 product range. The evaluated systems consisted of the third party UNITE web-service and two independently hosted Drupal web-based Content Management Systems (WCMS); one hosting Symantec Norton 360 structured manual content and the second hosting related User Generated Content (UGC). An initial term taxonomy was extracted semi-automatically from the structured corporate content [16]. The unstructured UGC was retrieved by crawling the online Symantec Norton 360 User Forum14 with the Heratrix15 web crawler. Finally the text analytic tool Weka16 was trained with the structured content and the initial taxonomy to generate a statistical model allowing unified term identification across the UGC. In both Drupal deployments the Drupal taxonomy module17 was used to associate extracted terms with content nodes. In addition to hosting different content types on two separate Drupal deployments different design templates were chosen to ensure a real world cross-site browsing experience. For model scrutiny both websites were able to display the underlying term model at any time. The assessment of a strategic model was not part of this evaluation.
To inform the user about related content based on the previous cross-site browsing experience the adaptive hypermedia technique of link annotation was used [17]. For this three different Drupal modules were developed. The first and most important module utilizes the Drupal hook_nodeapi13 allowing the module to hook into the call stack of the Drupal environment. Every time a user accesses a content node in Drupal the module sends the related terms to the UNITE service via HTTP POST. The two remaining modules are designed to provide a cross-site personalised experience to the user by adding an image ( ) beside any link that is relevant to the current task based on the previous browsing experience. Both modules used the same image to provide a coherent cross-site experience. Based on the open-source nature of the module the image can be altered by the content provider deploying the module. The first module adds link annotations to the forum thread list (example Figure 2) and the second module adds the same link annotations to the search result list on a separate second site. In addition to the image the module ) that indicates the terms also adds a tooltip box ( that match both the terms of the underlying content behind the link and the terms of the previous cross-site browsing experience provided by the third-party UNITE service.
A. Use Case 1) User browses to a website hosting structured corporate content. 2) Website sends UNITE service a user ID and terms related to viewed web pages. 3) UNITE service queries the Term Identification Service (TIS) to retrieve related terms that are then stored in the User Model Repository (UMR). 4) User browses to a different website hosting UGC. 5) Prior to visualisation the Website requests cross-site terms related to all links within the currently accessed web-page. 6) UNITE service queries TIS and UMR and sends matching terms back to the website for cross-site link annotation based on the user’s previous browsing experience.
Figure 2 Cross-Site Link annotation in Forums
14
12
http://community.norton.com/t5/Norton-360/bd-p/Norton_360 http://crawler.archive.org/ 16 http://www.cs.waikato.ac.nz/ml/weka/ 17 http://drupal.org/documentation/modules/taxonomy
http://drupal.org (At the point of development Drupal 7 was not available as stable deployment) 13 http://api.drupal.org/api/drupal/developer--hooks-core.php/function/hook_nodeapi/6
15
545
Term Identification Service
task a second task was displayed based on the same procedure illustrated above. To avoid a learning effect task selection (two out of four) and system sequence (personalised or nonpersonalised system pair) were determined by the Latin Square design. Finally two questionnaires related to usability were presented; one comparative and one generally about personalisation on the web. The final two questionnaires were based on a 4-point LIKERT scale (1=strongly disagree, 2=disagree, 3=agree, 4=strongly agree). Even though not common practice, the four-point scale was chosen to force users to provide either positive or negative feedback. As incentive all participating users were added to a draw for an electronic device.
User Model Repository
3
6 UNITE RESTful Web Service
2 WCMS Client hosting structured Norton 360 Content
5 WCMS Client hosting UGC Norton 360 Content
1
C. User Performance Findings The overall task completion time of the cohort evaluating the CSP enabled system setting was slightly better with 35512 seconds (SD 433). The CSP disabled cohort had an overall task completion time of 36545 seconds (SD 534).
4
An analysis of the p-values resulted in a non-significant difference between the task completion times of both system settings (p-value of 0.19)18. A possible reason for the high pvalue may lie in the fact that the difference between both systems was very subtle with a small visual icon being the only difference. Even though the overall mean comparison is encouraging, a higher number of participants would be required to ensure the mean measurement is significant.
Figure 3 Use Case Overview
B. Evaluation Setup To address both hypotheses a user-study consisted of four real world Online Customer Care tasks related to different features of the Norton 360 product range was conducted. Each task had up to six questions ranging from general questions such as “What are the main features of the Norton 360 backup function?” to specific questions such as “How do I get the Norton toolbar back?”. In each task users were asked to browse within and across two separate WCMS Drupal instances; one hosting structured manual content, the other user generated forum content. To evaluate the effect of non-intrusive cross-site recommendations each WCMS instance was deployed with and without personalisation modules.
A secondary argument can be made in relation to the nature of the task and the application space of CSP. The task was constructed to assess if CSP can assist users in addressing tasks that require browsing unrelated websites. It can be argued that a task that requires browsing is related to fact-gathering and not fact-finding (found in Information Retrieval related tasks). Therefore the participants might have different ways of assessing content. This individual preference can lead to different reading times not assessed in this experiment.
The evaluation was conducted online with 36 volunteering participants from Trinity College Dublin and the University of South Australia. The pre-questionnaire indicated that most users rated their knowledge low in Norton 360 products. Furthermore most users stated that they are consulting online manual and forum information on a regular base and using search engine technology frequently in searching for task related information. Finally most users indicated none to very limited experience in the use of personalised systems.
An analysis of the cumulated clicks per system setting indicates the same findings. For the CSP Enabled System setting 455 clicks (SD 4.78) were recorded. For the CSP Disabled System setting 383 clicks (SD 4.13) A slightly higher click rate in the CSP enabled cohort can be interpreted as positive due to the users exploring more content. This exploration aspect is important in relation to browsing assistance. However, similar to the user’s task completion time, it does highlight critical questions, such as the relation between clicks and time. It can be argued that the increase in clicks may not be interpreted as positive due to the argument that effective guidance should prompt lesser clicks. Both arguments are valid; however, on the contrary it can be argued that by injecting link annotation based on the user cross-site information needs the user becomes more active and motivated to explore additional content related to the task. For example the user might click on links that are not annotated
The evaluation was initialized by sending login and evaluation details to the individual participants. Once logged in, the participant was asked to fill out a consent form and an initial questionnaire assessing prior knowledge in the Norton 360 product range and personalised systems. The participant was then provided with a short introduction related to the first system pair. The next page displayed two out of the four tasks and two links, each pointing to a separate WCMS instance. One link pointed to the WCMS hosting the manual content, the second link pointed to the WCMS hosting user generated forum content. The user was asked to open both WCMS in separate tabs or browser windows. Once the user completed the first task across both WCMS a SUS questionnaire was presented. After finalizing the SUS questionnaire for the first
18
546
Throughout the discussion of both evaluations all data used in ttests was checked for normal distribution.
and then decide to also click on the annotated links that would have not been clicked and explored otherwise.
clearly described and demonstrated in an introduction video presented to the participant prior to the evaluation. This observation is supported by comments provided by the participants, such as "the main problem i had was that i didnt notice the adaptive hints or terms at all really" or "the face icon only appeared a couple of times and was always at the page I had already clicked on from the search results".
To gain further insight in the impact of CSP on the user’s browsing, the following the self-reported findings of the participants are discussed in the following. D. Self Reported Findings The resulting SUS score of both system settings (CSP enabled websites resulted in 68.67 with 31 user responses and the CSP disabled websites in 62.58) indicated that there were no significant usability issues in any one of the two system settings. Even though SUS does not focus specifically on CSP the SUS score provided certainty that the overall website was not designed poorly and therefore did not distract the user with the effect to distort the findings.
In conclusion it can be stated that the introduced adaptive hint may have been too subtle for some of the participants to notice. Furthermore, with the experiment being conducted online it was not possible to identify if all participants read the introduction text as well as viewed and understood the introduction video. To investigate CSP further, the participants were asked if CSP would reduce the feeling of being lost. Even though over 30% of the participants disagreed, almost 40% agreed and 30% strongly agreed. This result is encouraging and indicates that the majority of the participants perceived CSP as reducing the feeling of being lost when browsing.
To assess the perceived relevance of the CSP approach the participants were asked to indicate the relevance of the adaptive hints when browsing both websites. Most participants agreed (approx. 60%) that the adaptive hints were relevant (Figure 4).
Finally, the participants were asked about their overall impression. More than half of the participants agreed that they were satisfied with the performance, assistance and guidance of the CSP enabled websites (19% strongly agreed, 56% agreed, 22% disagreed and 3% strongly disagreed). This encouraging result was backed up by over 60% of the participants wanting to try the CSP approach in an open web domain (30% strongly agreeing). To gain more insight into the acceptance of the approach the participants were asked if they would have privacy concerns if used on the open web. The answers were provided in free text with some interesting comments mainly indicating the wish to disable CSP if necessary, and the requirement that the data is to be collected anonymously. Sample answers were “I would if it could be deactivated. I would prefer the system to be open source to see what it is actually doing” and “As long as it collects data anonymously (no personal information just browsing behaviors and interests) and it cannot be trace back to me, it's ok.”
Figure 4 Relevancy of adaptive hints
In relation to the non-intrusiveness notion of the approach Interestingly more than 60% of users disagreed that they were constantly aware of the adaptive hints on both websites (Figure 5).
The participants were also asked what they most liked and disliked about the approach. Users liked the notion of personalisation based on cross-site browsing behavior. Surprisingly many users indicated that they would welcome more intrusive personalisations. The main drawback participants indicated was that once they arrived on a webpage and after clicking a recommended link they were left on their own without any further indication on what section of the website is relevant. Other unpopular features were related to the small amount of hints at the start of the browsing process (cold start problem). The outcome of the first experiment that focused on evaluating the initial CSP prototype was encouraging. The main reason for encouragement was that it was possible to provide a CSP Service to communicate with two separate websites and provide consistent user assistance. Further reasons for encouragement were: (1) Users indicated the relevancy of the provided personalisations; (2) A measurable difference in the
Figure 5 Awareness of adaptive hints (non-intrusive)
Based on this finding, it can be argued that some users may have not been aware of the adaptive hints even though it was
547
performance of both systems pairs was recorded ; (3) Users indicated strong interest in using the approach in open domains. However, less encouraging observations where: (i) Users perceived the introduced personalisations as too subtle; (ii) Users indicated privacy concerns; (iii) Task design and experiment setup requires more concentration on exploration and may require lab based assessment to allow open questioning of the user after completing the task. VI.
[3]
[4] [5]
[6]
CONCLUSION AND FUTURE WORK
This paper discussed a novel approach for cross-site personalisation. The main challenges of this approach were discussed and addressed by considering both the user and content provider needs. To illustrate the functionality and usefulness of the approach an initial implementation was presented together with evaluation results. The results indicated a clear acceptance of the approach. Moreover the impact of the approach on the underlying website can be concluded as minimal based on the use of the extensibility mechanism of WCMS allowing a simple and non-invasive applicability of cross-site personalisation to websites during run-time.
[7]
[8]
[9]
The evaluation has provided important insights resulting in three areas to be addresses in further work. First, the application of personalisation strategies needs to be introduced to allow more user-centric and useful task-specific recommendations. Second, the user has to be made aware that the website is personalised either by a tool bar or by subtle explicit feedback if recommendations are not followed. Finally, further modules for different WCMS, such as MediaWiki and Plone, need to be developed.
[10]
[11]
[12]
ACKNOWLEDGMENT
[13]
This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localization (www.cngl.ie) at Trinity College Dublin. We would like to acknowledge the contributions made by Symantec in relation to providing Customer Care Content, especially Fred Hollowood.
[14] [15]
[16]
REFERENCES [1]
[2]
P. Lops, M. Gemmis, and G. Semeraro, “Content-based Recommender Systems: State of the Art and Trends,” Recommender Systems Handbook, Springer US, 2011, pp. 73-105. H. Lieberman, “Letizia: An Agent That Assists Web Browsing,” International Joint Conference On Artificial Intelligence, 1995, pp. 924– 929.
[17]
S.E. Middleton, N.R. Shadbolt, and D.C.D. Roure, “Ontological user profiling in recommender systems,” ACM Transactions on Information Systems, vol. 22, 2004, pp. 54–88. D. Mladenic and B.A.P. Webwatcher, Machine learning used by Personal WebWatcher, 1999. L. Chen and K. Sycara, “WebMate: A Personal Agent for Browsing and Searching,” In Proceedings of the Second International Conference on Autonomous Agents, ACM Press, 1998, pp. 132–139. A. Moukas and P. Maes, “Amalthaea: An Evolving Multi-Agent Information Filtering and Discovery System for the WWW,” Autonomous Agents and Multi-Agent Systems, vol. 1, Mar. 1998, pp. 5988. M. Sah, W. Hall, and D.D. Roure, “SemWeB Semantic Web Browser ? Improving Browsing Experience with Semantic and Personalized Information and Hyperlinks,” InterFace 2009: 1st National Symposium for Humanities and Technology, 2009. O. Conlan, V. Wade, C. Bruen, and M. Gargan, “Multi-Model, Metadata Driven approach to Adaptive Hypermedia Services for Personalized eLearning,” Proceedings of Second Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, SpringerVerlag, 2003, pp. 100-111. Conlan, O and Wade, V., “Evaluation of APeLS - an adaptive eLearning service based on the multi-model, metadata-driven approach,” Lecture Notes in Computer Science in proceedings of the Third International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Eindhoven,The Netherlands: Springer, 2004, pp. 291 - 295. P. Brusilovsky and N. Henze, “Open Corpus Adaptive Educational Hypermedia,” The Adaptive Web, Springer Berlin / Heidelberg, 2007, pp. 671-696. P. Dolog, N. Henze, W. Nejdl, and M. Sintek, “Personalization in distributed e-learning environments,” WWW Alt. '04: Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, New York, NY, USA: ACM, 2004, pp. 170–179. P.D. Bra, A. Aerts, B. Berden, B.D. Lange, B. Rousseau, T. Santic, D. Smits, and N. Stash, “AHA! The adaptive hypermedia architecture,” Proceedings of the fourteenth ACM conference on Hypertext and hypermedia, Nottingham, UK: ACM, 2003, pp. 81-84. M. Yudelson and P. Brusilovsky, “Adaptive Link Annotation in Distributed Hypermedia Systems: The Evaluation of a Service-Based Approach,” Proceedings of the 5th international conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Hannover, Germany: Springer-Verlag, 2008, pp. 245-254. M. Yudelson, Providing Service-Based Personalization In An adaptive Hypermedia System, PhD Thesis, University of Pittsburgh, 2010. K. Koidl and O. Conlan, “Engineering Information Systems towards facilitating Scrutable and Configurable Adaptation,” Hannover, Germany: Springer, 2008, pp. 405 - 409. M. Sah and V. Wade, “Automatic metadata extraction from multilingual enterprise content,” Proceedings of the 19th ACM international conference on Information and knowledge management, Toronto, ON, Canada: ACM, 2010, pp. 1665-1668. P. Brusilovsky, “Adaptive Hypermedia,” User Modeling and UserAdapted Interaction, vol. 11, 2001, pp. 87-110.
[18] l Writer’s Handbook. Mill Valley, CA: University Science, 1989.
548
