Univ Access Inf Soc DOI 10.1007/s10209-008-0116-1
LONG PAPER
User profiling and virtual agents: a case study on e-commerce services Giovanni Semeraro Æ Verner Andersen Æ Hans H. K. Andersen Æ Marco de Gemmis Æ Pasquale Lops
Springer-Verlag 2008
Abstract The main contribution of this work is the design of an application framework based on both conversational agents and user profiling technologies for the development of e-commerce services. User profiles are exploited by conversational agents to help customers in retrieving potentially interesting products from a catalogue. Three techniques were used for collecting data for a usability test: eye-movement tracking, questionnaire, and recording the user-system dialogue. The main outcomes of the experimental sessions are: (1) the dialogue capabilities of the agent facilitate the interaction between the user and the e-commerce site; and, (2) user profiles improve the retrieval capabilities of the agent. Finally, some limitations of the user profiling techniques adopted in the framework are discussed and a more sophisticated content-based profiling technique is proposed. Keywords Personalization E-commerce Evaluation methodology
G. Semeraro M. de Gemmis (&) P. Lops Dipartimento di Informatica, Universita` degli Studi di Bari, Via E. Orabona, 4, I70126 Bari, Italy e-mail:
[email protected] V. Andersen Systems Analysis Department, Risoe National Laboratory, Frederiksborgvej 399, Build. 110, P.O. 49DK-4000, Roskilde, Denmark H. H. K. Andersen National Research Centre for the Working Environment, Lersø Parkalle´ 105, 2100 Copenhagen Ø, Denmark
1 Introduction In Business to Consumer (B2C) e-commerce, the process of buying products and services often implies a high degree of complexity and uncertainty, and it is very time consuming for customers. Customers have started to require personalised support responding to their specific needs, in order to avoid access to electronic shops that often appear as a warehouse, where one must know exactly what to buy and where to find it [2, 15, 28, 30]. An effective solution for these issues should combine the usefulness of an added-value service with a high degree of usability, as well as dedicated measures to build up trust and confidence in inexperienced users [12]. To meet these conditions, interaction should, at the same time, be as natural as possible, thus enabling users to rely on their communicative skills, convey precise and relevant information, and address the personal background of the individual user. The interface should use best practice solutions to achieve a high degree of dialogue intelligence, and employ an appropriate graphical design. The solution discussed in this paper is based on an ‘‘intelligent personalised agent’’, which represents a virtual assistant achieving personalization by modelling the ability to support customers. There are many possible applications for virtual assistants. They could instruct customers in the use of a website, point out new offers, help sift through products, and other support. A chat robot (‘‘chatterbot’’) is a program that attempts to simulate the conversation or ‘‘chatter’’ of a human being [8]. It can be thought of as the spokesperson for an artificial intelligence system and usually consists of a collection of dialogue management rules, which might use different techniques for processing the user’s input. These techniques may range from simple keyword-based text parsing
123
Univ Access Inf Soc
to more complex logic-oriented techniques based on inference mechanisms. There have already been some efforts made in developing chatterbots based on expert systems. Chatterbots such as ‘‘Eliza’’ [36] and ‘‘Parry’’ [11] are well-known early attempts of creating programs that might at least temporarily fool a real human being into thinking they were talking to another person: recently, Artificial Linguistic Internet Computer Entity (Alice, http://www. alice.org) has gained a lot of popularity on the World Wide Web: hundreds of people around the world have contributed to the success of Alice because the source code is freely available under the GNU license statement. The basis for Alice’s behaviour is Artificial Intelligence Markup Language (AIML), an XML specification for programming chatterbots. It follows a minimalist philosophy based on simple stimulus–response algorithms, allowing programmers to specify how Alice will respond to various input statements. Simple chatterbots only simulate conversation without utilising any knowledge about the individual users and their actual behaviour during online sessions. In order to achieve an adequate non-stereotypical repertoire of reactions, the individual dialogue situation must be interpreted. This kind of dialogue intelligence is based on elaborated dialogue rules allowing the system to interpret a wide variety of situations that may occur. Whereas an increase in general dialogue intelligence can be achieved by elaborate rule sets, the naturalness of the dialogue depends on the degree to which the system is able to adapt to individual users, whether it is able to learn about their preferences and attitudes during the dialogue, and memorise them for later use. For this purpose, learning mechanisms can be exploited to extract features of a given user from the dialogue; of course, the user must consent in advance to this, and will be given an opportunity to inspect and change the data [19]. This helps content providers to tailor their offers to the customers’ needs, and can be used to generate assumptions about new users, when they start to converse with the system. Published research to date shows that further development of personalised interfaces into more flexible dialogue-oriented interfaces could increase the acceptance of such personalised agents [3, 6, 9, 10, 32]. This work presents an agent-based application, designed according to the issues previously discussed, that supports users of e-commerce websites. In order to find out appropriate ways to adjust the system parameters, the Bertelsmann On Line website (BOL— http://www.bol.com) for selling books, music, gifts, etc., has been utilised as a test site, and the technical development has been accompanied and heavily influenced by in-depth evaluations of both the individual components as well as the system as a whole.
123
The results of the empirical study are discussed with the aim of evaluating the impact of the system on the user behaviour. The paper is organised as follows. Section 2 discusses the main issues related to intelligent dialoguing agents and personalization techniques, particularly in the area of electronic commerce. Section 3 introduces the approach undertaken in the COGITO project in order to design advanced solutions for personalised information access, and describes the overall architecture of the system. Section 4 presents a thorough description of the empirical study carried out with the specific aim of evaluating the impact of the system on the user’s behaviour. In Sect. 5, some limitations of the user profiling techniques adopted in the framework are discussed, and a more sophisticated content-based profiling technique is proposed. Finally, conclusions are drawn in Sect. 6.
2 Background The main problem of most of today’s Internet services is that they offer manifold navigation options and (usually simple) search functions, but leave it up to users to find their way through the many interface functions, understand them and interrelate them cognitively [7]. Beginners and occasional users are often daunted by the complexity of today’s services and thus need ‘‘proactive’’ support or advice from the system in order to fully exploit the range of available functions. Intelligent agents can converse with the user in written natural language and can be regarded as an alternative interface metaphor to the contemporary GUI technology. Natural language dialogues are easy to understand, the users may express their needs in an unrestricted form, and it is possible to produce sophisticated system behaviour. Moreover, largest e-commerce sites offer millions of products and are visited by users having a variety of interests. Considering the vast selection of offerings, it is of particular interest to provide the users with personal advices, which reflect their individual needs and interests [25]. For example, searching the product database can be overwhelming, thus contributing to the user’s sense of information overload. What users really need is a ‘‘personalised’’ search process that takes into account their tastes, in order to sift through large amounts of retrieved information. Personalization could be obtained by learning about users’ preferences and attitudes during the dialogue and memorising them in personal profiles for later use in the searching process. Thus, agents and personalization become the key factors to enhance human–computer interaction for Internet services.
Univ Access Inf Soc
2.1 Intelligent agents The expressive visualisation of a virtual advisor—e.g., as an animated cartoon ‘‘Persona’’—can be a direct and useful complement to the proposed dialogue approach [14]. By being able to take the initiative, rather than simply reacting to user input and requests, a system can take on the role of an independent agent during dialogue [23]. To make this role of a true counterpart more transparent, it is helpful to visualise the agent—thus the agent is also visually present and can go beyond the communicated contents to express functional aspects of its dialogue contributions (questions, recommendations, warnings, etc.) by means of mimic and gestures. Moreover, the Persona can also visually express emotional aspects of the interaction (interest, warmth, humour), thus contributing significantly to a relaxed atmosphere and increased attractiveness of the service. A chatterbot is a software system capable of engaging in conversations (in written form) with a user, often entertaining the user with some ‘‘smalltalk’’—sometimes accompanied by cartoons expressing emotions. In most applications, chatterbots are used as guides who can show the user around on a website. This can be a stereotyped ‘‘guided tour’’, allowing only few deviations; however, this concept has to be abandoned when the website is too large to be explored by navigation, or contains too many offers. This is the case in e-commerce applications, where pages are generated on demand by retrieving data from a product database and assembling the result into HTML pages, which present the hit lists of searches. Virtual assistants must be capable of flexible behaviour if they are to be acceptable to users on a long-term basis [16]. Simple chatterbots, such as the first system of this type, ELIZA [36], and most of its successors (for an overview, see http://www.botspot.com) only simulate conversation without exploiting any knowledge about the individual users and their actual behaviour during online sessions. Such simple chatterbots are not powerful enough to serve as a medium for customer advice. This means that, in addition to some of the abilities already available (e.g., help question answering controlled by simple event-action rules), a further reaching dialogue management will be needed to help accomplish two major goals. First, in order to achieve an adequate, nonstereotypical repertoire of reactions, the individual dialogue situation must be interpreted, and second, dialogues that are more complex allow goal-directed strategies to be pursued (e.g., cooperative behaviour and convincing argumentation). This kind of dialogue intelligence will be based on elaborated dialogue rules allowing the system to interpret a wide variety of situations that may occur. Dedicated editor tools will support the construction of these rule sets.
While important, the increased ergonomic usability and personalization of chatterbots are only preliminary steps. Therefore, in the developed system, a component for intelligent access to the supplier’s repository was added, which act as a ‘‘Prompter’’ helping the chatterbot in problematic retrieval situations (too many, too few hits, etc.). It relies on a repository of search heuristics, and exploits the personal profiles as well as domain knowledge provided by the content manager. The latter is capable to harvest a supplier’s XML-based website and extract structural and semantic information. Since any automatic assistance is limited, a gateway to the supplier’s call centre is important. This again should contribute to increase the consumer’s trust and confidence. 2.2 Personalisation Personalization has become an important business strategy in the e-commerce field, where a user explicitly wants the site to store information such as preferences about herself [29]. Exploiting the underlying one-to-one marketing paradigm is essential to be successful in the increasingly competitive Internet marketplace [26]. A key issue in personalization of a website is the automatic construction of accurate machine processable user profiles [20]. This process is called user modelling and consists of ascertaining a few bits of information about each user, processing that information quickly and providing the results to applications, all without intruding upon the user’s consciousness [35]. The final outcome of the process is the construction of a user model or a user profile [24]. Any application that behaves differently for different users employs a user model. The models themselves can be big or small, complex or simple, rich or sparse. They often have different names: personality profiles, psychographics profiles, or consumer databases. They are collections of information about an individual. Such collections of information are at best embryonic precursors of an ideal user model, which would possess an intimate and thorough knowledge of the user it refers to. In short, the user model should be able to recognise the user, know why the user did something, and guess what she wants to do next. Profiles could be used to deliver personalised content to the user, fitting her personal choices. The main advantages of using the oneto-one personalization paradigm based on user profiling are [33]: •
Making the site more attractive for users. A website that takes into account user preferences is able to make recommendations reflecting user needs. Specifically, in the e-commerce area, this will probably turn a
123
Univ Access Inf Soc
•
•
significant part of site visitors into buyers, thus increasing the conversion rate; Obtaining trust and confidence. Users will not be requested to explicitly insert information concerning their preferences, tastes, etc., but they will be able to participate in the management and update of their personal profile. This will result in an increase of trust and confidence in a system able to automatically collect data about their preferences; Improving loyalty. The effectiveness of a personalization system improves in the long run. Every time a user interacts with the website, the personalization mechanism collects new data about her preferences, so that a more and more satisfactory service can be offered. Specifically, in the e-commerce area, switching to the competition is often unfavourable for a customer. In fact, even if a competitor uses a personalization system, it has to learn a lot of information about the new customer to be able to offer the same satisfactory service.
3 The COGITO project The COGITO project, funded by the EC in the 5th Framework Programme, Key Action 2: New Methods of Work and Electronic Commerce, aimed at bringing together appropriate technologies from different branches of artificial intelligence, information retrieval, electronic publishing and commerce, and human factors research to realise infrastructures for advanced applications. COGITO has developed a unique combination of technologies, including: •
•
•
User guidance in conversational interaction: the system is proactive and capable of engaging in a goal-directed conversation with the user, in order to interpret the user’s behaviour and support the accomplishment of a specific task the user intends to fulfil, e.g., by recommending search strategies. Personalization and profile extraction: knowledge about the user behaviour is learned during the interaction and stored in personal profiles exploited in the retrieval process. Intelligent retrieval: in order to support a user in retrieving appropriate database entries, the retrieval process involves a mechanism of automatic query expansion accessing different knowledge sources, such as the profile repository.
This approach was used to improve the usability of the BOL German website. The system has been used to provide access to relevant product information as well as to help the user in finding appropriate offers.
123
3.1 An overview of the COGITO architecture The general architecture of the COGITO system includes the following macro modules: BOL Web-Server, chatterbot, Profile Extractor, Prompter, XML Content Manager. The system architecture is centred on a chatterbot capable of engaging conversations with users. The system invokes appropriate rules from a Chat Rule Base, which trigger internal processes such as database access, and generate the system response in written language. The graphical user interface (GUI) incorporates an interactive chatbox allowing for two-way conversational interaction in written natural language (in German and English), and employs a Visual Persona visualizing the virtual dialogue partner/sales assistant. In COGITO, XML technologies play a relevant role for representing and exchanging information in homogeneous and structured format. The XML Content Manager is a complete web-enabled platform responsible for the management of XML data. A large part of the project is concerned with further additions to the chatterbot, through the implementation of a context-dependent Prompter and a Profile Extractor. The Profile Extractor is the module that implements the user profiling mechanism of the system, while the Prompter is an intelligent information retrieval component invoked in situations which are problematic, in the sense that the general problem solving competence embedded in the chatterbot’s rule base is not enough to meet a user request, as in situations caused by unknown phrases, highly specialised vocabulary or ambiguous words [34]. 3.2 Profile extraction and retrieval process In B2C e-commerce, items are often grouped in a fixed number of categories. For example, at BOL.com books in the catalogue are classified in many subject categories in order to make searching easier (Fig. 1). However, users are hardly ever interested in all these categories: their preferences focus only on a small subset. A simple approach to acquire the preferred categories is to build a profile of interests by filling in an initial form, in which users are asked for explicit preferences among the list of categories available in the store. The main problem of this process lies in its static nature, since it depends on whether the users are willing to give and update their preferences. In the COGITO system, the use of automatically inferred user profiles was combined with a virtual dialoguing agent in order to improve the usability of the search function of the BOL website.
Univ Access Inf Soc Fig. 1 Book categories at BOL.com
The Profile Extractor employs supervised learning techniques to dynamically discover users’ preferences from interaction data recorded during past visits to the e-commerce website. Following an approach similar to that adopted by Adomavicius and Tuzhilin [1], preferences are stored in the customer profile. In the above context, the problem of learning user’s preferences can be cast to the problem of inducing general concepts from examples labelled as members (or nonmembers) of the concepts [21]. In this context, given a finite set of categories of interest C ¼ fc1 ; c2 ; . . .; cn g, the task consists in learning the target concept Ti ‘‘user interested in the category ci’’. In the training phase, each user represents a positive example of users interested in the categories she likes and a negative example of users interested in the categories she dislikes. Moreover, an operational description is chosen of the target concept Ti, using a collection of rules that match against the features describing a user in order to decide if she is a member of Ti. In the COGITO project, the set of categories of interest on which the system has been tested (translation in English of the German words is reported in parenthesis) is C = {Belletristik (Fiction), Computer und Internet (Computer and Internet), Kinderbu¨cher (Childrens), Kulturund Geschichte (Culture and History), Nachschlagewerke (Reference Books), Reise (Travel), Sachbuch und Ratgeber (Monographs and Guidebooks), Schule und Bildung (School and Education), Wirtschaft und Soziales (Economics and Law), Wissenschaft und Technik (Science and Technique)}. The members of C are the ten main book
categories the BOL product database is subdivided into, and represent the preferences of the users accessing the BOL website. The system exploits data collected in the course of past dialogues with the chatterbot, as well as interaction data, that represent the features used for describing the users (for example, number of searches and purchases in a specific category). By learning from features, the Profile Extractor is able to recognise the product categories preferred by a buyer. Each customer represents an instance of the learning problem. A subset of the customers (training set) is chosen to train the learning system and is labelled by a domain expert, who classifies each customer as member or non-member of each category. Training instances are processed by the Profile Extractor, which induces a classification rule set for each book category. Rules are used to classify a new customer (not included in the training set) as interested or not interested in each product category, on the basis of her features. For example, the ‘‘Belletristik’’ rule set contains rules able to classify a new user, on the ground of her features, as interested or not interested in that category. After the learning phase, the classification process is repeated for each book category. All these classifications, together with the interaction details, are gathered to form the user profile (Fig. 2), composed of two main frames: factual and behavioural. The factual frame contains transactional data extracted from both the dialogues and session logs (number of searches and purchases in a specific category, etc.), while the behavioural frame contains the book categories ranked according to the degree of interest computed by the learning system.
123
Univ Access Inf Soc Fig. 2 An example of user profile
The name of the chatterbot integrated in the COGITO system is Susanna. A user is known by Susanna if she completes the BOL registration procedure. This step allows the system to provide each customer with a personal identification number. This is necessary to recognise a user accessing to the online media shop as well as to collect data about her preferences for generating/ updating the personal profile. The profile of a user is then exploited to identify the book category with the highest degree of interest, which can be enclosed in a query for getting more specific results. This mechanism, called query expansion, is applied to the search engine of the BOL website. The value added by user profiles to the retrieval process is instantiated in the following scenarios: 3.2.1 Scenario 1: unknown user In this scenario, an unknown user asks the chatterbot for a book written by an author named King. Susanna finds several books written by an author with this last name through a remote call to the search engine available at the BOL website, and displays them as shown in Fig. 3. Notice that the books that rank at the top are authored by Stephen King. Books by other authors are found further down the list, which means that the user should scroll down a long list if she is not looking for a book by Stephen King. The customer not looking for a book written by Stephen King can now choose to either refine the search by using an advanced search function or continue to chat with Susanna about different fields of interest.
123
3.2.2 Scenario 2: registered user In the second scenario, the user has already been chatting to Susanna about some of her interests. Therefore, a profile of this user is available to the system, which can exploit it to accomplish a more precise search in the product database. Let consider the case that the profile of the user accessing the BOL website is the one presented in Fig. 2 and that the query submitted by the user is the same as in the previous scenario. The first book displayed in the result page is a book about Windows 2000 co-authored by Robert King (Fig. 4). This result is due to the fact that the original query about King has been automatically expanded into ‘‘Author: King, Genre: Computer and Internet’’ (highlighted by the circle in Fig. 4), since Computer_und_ Internet is the category with the highest degree of interest in the profile of the user (Fig. 2). In the new result set, books belonging to the user’s most preferred category are ranked before books belonging to other categories (other search criteria being equal). These scenarios highlight the degree of personalization of the interaction, because of the dependence of the result set on the profile of the user that issued the query. 4 Empirical evaluation methods The evaluation of the COGITO system is based on an analysis of the agent–user dialogue, eye-tracking measures related to the use of GUI elements, and detailed questionnaires. The evaluation groups of test persons executed
Univ Access Inf Soc Fig. 3 Susanna offers a long list of books by authors whose last name is ‘‘King’’ belonging to different categories
Fig. 4 Susanna offers a list of books written by authors whose last name is ‘‘King’’ and belonging to the category ‘‘Computer and Internet’’
various tasks where they used the full range of features offered by Susanna to search for general information or specific products on the BOL website. In the analysis of the user–agent dialogue, several measures were performed, including the length of sentences utilised by the users indicating a real conversation in contrast to using just a search-engine technique, the use of stereotypical sentences by the agent, and the number of fall-back sentences indicating a missing interpretation of the request from the user. Using the eye-tracking measure, it was possible to measure the time the user spent looking at the agent, the area presenting the answers given by the agent, or the BOL website itself. Furthermore, the evaluation is based on the users’ subjective assessment of using the system based on their fulfillment of detailed questionnaires. Using
questionnaires allowed exploring users’ general impressions and understandings of the agent, when it reacted to user requests and presented its suggestions to solutions through links to the BOL website. Therefore, during the session the visual perceptions of the test persons are monitored, and following the session the eye-tracking data and the log files of the communication between the user and the agent have been analysed to test, respectively, the visual attraction of various parts of the screen and the user–agent dialogue related to various supporting features. The eye movements were monitored using a headset-free eye tracking equipment in order not to distract the test person and thereby influence the performance. Finally, the test persons were requested to complete a questionnaire revealing their satisfaction with the system
123
Univ Access Inf Soc
and the agent. The design of the questionnaires has been inspired by the questionnaires developed by Crowder et al. [13], which were considered as especially helpful towards evaluating web based intelligent agents (see Error! Reference source not found.1). Crowder et al.’s questionnaire shares some resemblance to the software usability measurement inventory presented in [18, 27]. In order to test—for various levels of users—the benefit from the agent experienced by the users compared to their previous experience from using the Internet, the test group was divided into two groups, including users less and more experienced in using the Internet, respectively. After being introduced to the system, the test person was asked to deal with a number of tasks prepared for the test session. The tasks included getting an overview about BOL, following a guided tour of the site, finding a specific book or books about a topic, and searching for information about how to order and pay, or about the security in using credit cards. The questionnaire has been divided into two parts. One part was meant to reveal background information about the test person herself, and the other for expressing the personal indication of the test person regarding various features of the agent and the site, as well as the overall feelings about the system with respect to expectations. 4.1 Evaluation of test sessions Two groups of eight persons each were recruited for the test sessions, one group of less experienced and one group of experienced users of the Internet, in order to test the validity of Susanna for each of these types of end users. Even though groups of eight persons are not representative of the population as a whole, this group composition was a reasonable compromise between the available resources and the need of getting a first impression of the functionality and validity of Susanna. The evaluation sessions were performed as follows: Table 1 shows the scales used in the evaluation questionnaire and their description (inspired by Crowder et al. [13])
• •
the less experienced group using the system, the experienced users using the system.
4.2 Background of respondents It was attempted to obtain a reasonable distribution between students and employed group participants. The real numbers of the test groups, less experienced and experienced, in relation to students/employed were 6/2, and 7/0, respectively. One of the recruited persons did not show up. However, due to the fact that the average age of the groups was all in the middle of twenties (26 for less experienced and 24 for experienced), and nearly all the participants were skilled persons, the students/employed relation is not expected to be of vital importance. For the gender distribution, the respective numbers for the same sequence of groups are 6/2 and 3/4. The criteria for being placed in the group of less experienced or experienced users are related to the experience concerning use of computers as well as experience in using the facilities on the Internet or using it for buying purposes. The experienced users have double or more computer experience as compared with the less experienced ones, even though they all had reasonable experience. Likewise, the experienced users reported a more frequent use of the Internet at home and, in contrast to the less experienced, they also use the Internet during working hours. Furthermore, they reported a higher variety in using the facilities on the Internet. Finally, the experienced users have much more experience in buying various items via the Internet than the less experienced, who were very limited in their purchases, if buying at all. Previous experience with agents on the Internet was not seen as an important criterion, and the background of the test persons showed that none of the less experienced had any experience with agents, whereas a few of the experienced users had worked with agents one or more times (Table 1).
Scale
Description
Impression
The user feeling good, warm, happy or the opposite as a result of interacting with the agent
Command
The feeling the user has that the agent is responding in a normal and consistent way to input
Effectiveness
The feeling that the user has that the agent is enabling the task to be performed in a quick, effective and economical manner The degree to which the user perceives that the agent communicates in a helpful way and assists in the resolution of problems
Aidability Learnability
The degree to which user feels that it is relatively straightforward to become familiar with using the agent
Navigability
Was the user able to move around using the agent?
Comprehension Did the user understand the interaction with the agent?
123
Univ Access Inf Soc Table 2 The results of the analysis of the agent–user dialogue log Agent–user\dialogue
Correct output
Fall-back sentences
Proactive sentences
Positive search queries
Percentage
61
27
29
61
All numbers are averages shown in percentages
4.3 Analysis of the user–agent dialogue During the evaluation session, the dialogue between the user and agent was recorded. The analysis encompasses 368 sentences created by Susanna through the dialogue with the users. The analysis of the log served the purpose of measuring the quality of the user–agent dialogue based on measuring the number of correct text output, fall-back sentences, proactive sentences and positive search queries (Table 2). As seen in Table 2, the agent produced ‘‘correct output’’ in 61% of the 368 sentences recorded during the evaluation sessions, which is a positive result in terms of precision and transaction success of the system. The measure ‘‘correct output’’ is based on a manual analysis and interpretation of elements of the agent–user dialogue, consisting of one user text input string, e.g., a request for information or a search query, and one agent output text string, e.g., delivering a meaningful answer and/or requesting further information from the user. A proactive sentence produced by the agent is counted every time the agent records a word or a string of words and constructs a contextually meaningful response to user input. Examples of a proactive and a nonproactive sequence taken from the actual user–agent dialogue recorded during the evaluation sessions are reported below, respectively: Proactive sequence: User: Agent:
Please tell me something about Microsoft Office? Which Office software do you most often use?
Non-proactive sequence: User: Agent:
What can I do online? How can I be of further assistance to you?
In addition, the quality of the dialogue was also analysed using two measures: the proportions of fall-back sentences and the various sentence categories applied by the agent. On the one hand, a large occurrence of stereotypical fallback sentences could impact the quality of the user–agent dialogue in a somewhat negative way. That is, there is a risk that the users will get frustrated if the agent ‘‘speaks’’ in phrases or cliche´s, which means that the users experience a rather stereotypical dialogue. On the other hand, using the proportion of fall-back phrases as a quality measure may be somewhat dubious. For example, a sentence produced by the agent ‘‘please, reformulate your
input’’, which we counted as a fall-back sentence, may be seen as a correct input related to, e.g., a typing error by the user. In this sense, the fact that the agent produced fallback sentences in 27% of the 368 sentences recorded during the evaluation sessions may have had an impact on the quality of the interaction seen from the perspective of the user, but from the perspective of the agent a fall-back sentence could prompt the user to give more input. That is, using fall-back sentences as a quality measure must be done with great care, and, for example, should take into account the broader context in which the user–agent dialogue is situated. In addition, the agent produced variation in its part of the dialogue by applying nine different categories of fall-back sentences in the user–agent dialogues such as: ‘‘interesting expression’’, ‘‘never heard it before’’ and ‘‘what do you really mean by that?’’. The search queries generated by the agent in prompting the BOL search engine were also analysed. In this respect, a query is counted as positive every time the agent, on the basis of selected user input plus the added search terms created by the query expansion processes of the agent, prompts the BOL search engine with queries that produce a meaningful list of search results in terms of relevance during a given user situation (task). For this purpose, all the search queries listed in the data log were repeated using the conventional BOL search machine, and the result were analysed in relation to the users tasks (see Table 2). It is well known that query length and search effectiveness are interrelated in experimental interactive information retrieval systems [6]. In traditional nonexperimental interactive information retrieval systems, query length is typically relatively short, in the range of two or three words [5]. Jansen et al. [17], for example, in an analysis of queries posed by users at a major Internet search service, found that Web queries contained 2.21 words on average. In the study presented here a user query contained 5.05 words on average. This finding is in line with the findings of Belkin et al [6], who established that adding specific techniques to interactive information retrieval systems results in longer queries than that of standard query elicitation technique. One of the main reasons for introducing intelligent agents on the Web is to overcome some of the obstacles of traditional information retrieval, such as the use of Boolean operators, and, at the same time, to allow users in a natural way to type their queries in a dialogue-based manner. Since
123
Univ Access Inf Soc Fig. 5 A snapshot of BOL site equipped with Susanna as it appeared during the evaluation. The red colour ‘‘boxes’’ (dark for black and white copies) mark the examined areas of interest
the conducted study is based on data from 15 respondents, it is difficult to generalise to the studies mentioned above. Since only one web-query was studied and results from only 15 persons are available, no general conclusions can be drawn based on the comparison between a traditional web search engine and the agent other than that a slight difference was found in the observed query length. More respondents would be needed to figure out if the differences show a tendency that is significant. 4.4 Eye-tracking analysis A remote eye-tracking system was used to measure the respondent’s visual behaviour during the evaluation session. This device is non-intrusive, and the respondents can behave as they normally would in front of a computer display. The eye-tracking system samples the eye-movements at 50 Hz to a data file. In addition, eye-movements were videotaped together with the graphic signal from the computer. The data is sampled only during respondents task solving, i.e., data were recorded from the point in time where the respondent has finished reading the task situation out loud and until she has finished the task, gives up, or is stopped by the moderator. The screen was divided into 5
so-called ‘‘areas of interest’’ (AOI, see Fig. 5): (1) the Agent torso, which shows the animation of the agent; (2) the Agent text output field, where text from the agent is displayed; (3) the User input field, where the user can type, e.g., requests for information; (4) the BOL site where, e.g., search results are displayed; (5) the Right lower corner, that shows the background of the screen. Then the amount of visual attention paid to each of these AOI in percentage of all viewing time during the task situations was calculated (Table 3). The smallest amount of viewing time has been spent looking at nothing at the right corner of the display. This is rather common to normal viewing behaviour, when people is either daydreaming or solves cognitive tasks that do not require any new visual input. In addition, a small amount of viewing time has been spent on the visualisation of the agent. With respect to the agent part of the system, most viewing time has been spent at the text output field. This is not surprising, as reading text takes time. In addition, some users had to scroll back to read longer paragraphs since the text ‘‘ran’’ too fast. Much viewing time was spent outside the display. This is not surprising, since the agent requires input in terms of written text using the keyboard. Not looking at the display
Table 3 Results of eye-tracking analysis for all respondents for the agent Areas of interest
Agent torso
Agent text output
User input field
BOL site
Right lower corner
Outside display
All figures in %
1.6
12.6
4.1
18.6
0.8
62.1
Numbers are averages in percentage
123
Univ Access Inf Soc
means that no eye-data are available. This means that either the respondent blinks, or eye-data is lost while the respondent looks at the screen due to a less optimal calibration, or the person looks outside the display at the keyboard, at the task description or at the moderator. From the video of the respondents, it is clear that approximately 40% of the viewing time outside the display is spent on typing at the keyboard. One lesson learned from the performed analysis is that care should be taken in selecting an appropriate user– agent interaction technique according to the purpose of the system. If the purpose is to attract the users’ attention to the web-page content, keyboard input might not be the best technique to choose. Another lesson learned from the eye-movement tracking analysis is that, if the attraction of the agent is important, the appearance of the agent plays a major role. In the specific case, the agent hardly attracted any attention, which not necessarily is a negative result. In daily living, we are used to interpret people based on short eyesights. The same thing could be the case regarding the agent. That is, the communication that one wants to convey through the visual appearance of a webbased agent should be meticulously composed, since the user might only look at the agent for a very short period of time. 4.5 Questionnaire analysis The members of the test groups indicated their overall impression about the agent by filling in a detailed questionnaire concerning seven measures: impression, command, effectiveness, navigability, learnability, aidability, and comprehension. A few examples of the outcome concerning these criteria are presented below. The result of the ‘‘Impression—users feelings or emotions’’ is presented in Table 4 (for a full presentation of the evaluation see [4]). The questions related to the impression of the agent are based on the agent being enjoyable or a bit awkward to use,
and if the user would recommend use of the agent to colleagues. The group of less experienced users had rather negative feelings for the agent in this respect, probably because they expect that an agent—when being available— should act unimpeachably in all situations. The experienced users, however, are aware of the need of a period for maturing a new product, and, in fact, the satisfaction for Susanna among these users has been stated as 61%, as shown in Table 4 by summing the columns ‘‘satisfied’’ and ‘‘very satisfied’’. Another example is comprehension, shown in Table 5. In this class, the questions were related to the user’s personal feeling of understanding of the information provided by the agent, the action of the agent, the interaction between the agent and the BOL site, and about how to operate the agent in relation to this site. Similarly to impression, satisfaction of the less experienced is less than the satisfaction of the experienced users. The agent appeals to the users in this respect, with 44% of the less experienced satisfied or very satisfied with their understanding of the interaction, while a similar feeling of understanding holds for 72% of the experienced users. However, for learnability (Table 6), related to the information needed in beforehand in order to be able to act with the agent, as well as the acquaintance with the agent based on just short time of experience, the analysis indicated 55% satisfaction for the less experienced. Once again, the experienced users were even more satisfied, indicating a satisfaction of 70%.
5 Beyond COGITO: from coarse-grained profiles to fine-grained profiles This section describes a new piece of work, which has been carried out in order to overcome the limitations of the COGITO search engine integrated with the conversational agent.
Table 4 User feelings or emotions Impression: the users feelings or emotions when using the software (%)
Very unsatisfied
Unsatisfied
Satisfied
Very satisfied
Less experienced users
16.67
47.92
22.92
12.50
Experienced users
11.11
27.78
50.00
11.11
Numbers are shown in percentage of the total number of user ratings in each category
Table 5 How well did the users understand the interaction between the BOL site and the agent Comprehension: the user understood the interaction with the application (%)
Very unsatisfied
Unsatisfied
Satisfied
Less experienced users
13.89
41.67
36.11
8.33
1.96
25.49
58.82
13.73
Experienced users
Very satisfied
123
Univ Access Inf Soc Table 6 Learnability of the prototypes Learnability: easy to become familiar with (%)
Very unsatisfied
Unsatisfied
Satisfied
Very satisfied
Less experienced users
16.13
29.03
48.39
6.45
8.70
21.74
65.22
4.35
Experienced users
The profiling process adopted in the COGITO project exploits supervised learning techniques to infer coarsegrained profiles, containing the product categories the user is interested in. Coarse-grained profiles are inferred from the analysis of transactional data, that is to say, the browsing and purchasing history of customers, and are exploited to refine the original query issued by the user. The main limitation of the coarse-grained profiles is that the learned model is too shallow for those information access scenarios that cannot be solved through a straightforward matching of queries and items in the repository. For example, a user looking for ‘‘interesting books about criminal minds’’ or ‘‘interesting movies about serial killers’’ cannot easily express this form of information needed as a query suitable for search engines. Even if the user can easily formulate the query ‘‘criminal minds’’ or ‘‘serial killers’’, other elements typically influence the relevance of the retrieved results, such as the plot of the movie or the nature of the committed crime. These elements are strictly related to the content of the items. This situation creates many new challenges for personalised search. Recent developments at the intersection of Information Retrieval, Information Filtering, Machine Learning, User Modelling and Natural Language Processing offer novel solutions for personalised information access. Most of this work has focused on the content-based information recommendation paradigm [22] that exploits textual descriptions of the items to be recommended in order to infer a profile exploited to select items of interest. For example, book descriptions at BOL.com usually consist of a set of features such as title, authors, price, editorial reviews, etc. Another relevant piece of information that can be exploited for personalised marketing are the customers’ opinions on the various products in the catalogue. For this reason, e-commerce sites generally allow customers to rate products through a scaled voting system. The COGITO profile generation process described in Sect. 3.2 has been improved by refining coarse-grained profiles through the adoption of a content-based approach that builds fine-grained profiles from the textual descriptions of the items rated by the user in each category in which she is interested. The preferred product categories stored in coarsegrained profiles are exploited to expand the query issued by
123
the user, while fine-grained profiles are used to filter the items in the result set. In short, if the coarse-grained part of the user profile contains, for example, the book category ‘‘Horror’’, the fine-grained part is exploited to filter ‘‘Horror’’ books by ranking first the most preferred ones, according to a preference score computed by analysing their content. The following sections first describe how fine-grained profiles are inferred from book descriptions, and subsequently how they are involved in the recommending process. 5.1 Learning Bayesian user profiles for content-based filtering Fine-grained user profiles are built by ITem Recommender (ITR), a system able to learn from textual descriptions of the items and the related users’ ratings. The system implements the naı¨ve Bayes classifier [21], a popular algorithm in text classification applications. The system classifies an item as interesting or not interesting for a particular user by exploiting a probabilistic model, learned from training examples (preclassified instances—the items rated by that user in the past). The system learns the content-based profile of a user in the form of a binary text classifier able to assign a new (unseen) item with a classification score, which can be used to decide whether the item should be recommended or not to the owner of the profile. For example, a profile for a user interested in ‘‘Horror’’ books is learned by ITR from previously rated ‘‘Horror’’ books and is exploited to recommend new ‘‘Horror’’ books to her. Given a set of classes C ¼ fc1 ; c2 ; . . .; cx g, the goal of the classification task is to assign an item I, described by a tuple of independent (the naı¨ve assumption) attributes ða1 ; a2 ; . . .; an Þ, to one of the elements of C. In a machine learning perspective, the objective is determining the best hypothesis from the space C, given an item I. According to the Bayes theorem, the posterior probability of each candidate hypothesis P(cj|I) is calculated as Q Pðcj Þ ni¼1 Pðai jcj Þ ð1Þ Pðcj j IÞ ¼ PðIÞ Since, for any given instance, P(I) is a constant with respect to cj, this factor can be ignored in calculating
Univ Access Inf Soc
Eq. (1), because it is only needed to find the hypothesis with the highest posterior probability—maximum a posteriori hypothesis—rather than a probability estimate. In the task of recommending an item I, only two alternative hypotheses are considered in C: (1) the user likes the item I (class c+), (2) the user dislikes the item I (class c-). In the learning problem, each instance (item) is represented by a set of slots. Each slot, corresponding to a specific feature of an item, is a document that contains a collection of stemmed words—bag of words (BOW)—processed by considering their occurrences in the original text. Thus, each instance is represented as a vector of documents. Specifically, in the scenario of book recommending, items are represented as sets of three slots: title, authors and annotation (‘‘Editorial Reviews’’ of the book). Given this representation of the items, the posterior probability of a category cj, given an item Ii, is computed using the formula: QjSj Qjbim j Pðcj Þ m¼1 k¼1 Pðtk jcj ; sm Þnkim ð2Þ Pðcj j Ii Þ ¼ PðIi Þ where S is the set of slots, bim is the BOW in slot sm of item Ii, and nkim is the number of occurrences of token tk (word) in bim. To calculate (2), the probability terms P(cj) and P(tk|cj, sm) are estimated from the training data TR as described in Ref. [31]. The final outcome of the learning process is a probabilistic model used to classify a new item in the class c+ or c-. This model is the user profile, which includes those tokens that turn out to be most indicative of the user preferences. To sum up, the naı¨ve Bayes learning method involves a learning step in which model parameters are estimated
Table 7 Interests of user 39 and user 40 in books belonging to the category ‘‘Computing and Internet’’ User ID
Interests
39
Machine learning, data mining, artificial intelligence, decision support systems
40
Web programming, XML, databases, e-commerce
based on word frequencies in the training data. Parameters are included in the user profile and used to classify each new item by applying Eq. (2). 5.2 Exploiting profiles to personalise recommendations Fine-grained profiles are exploited to provide users with personalised recommendations delivered using the ranked list approach (items are ranked according to a relevance score). A usage scenario of the ITR system, in which two users with different interests in books belonging to the category ‘‘Computing and Internet’’ submit the same query to the ITR search engine, is presented below. The explicit interests of the two users are reported in Table 7. The profiles of both users, inferred by ITR, are depicted in Fig. 6. Those profiles show some (stemmed) keywords in the slot title, which are most indicative of user preferences, ranked according to their predictive power (depending on the conditional probabilities computed in the learning step [31]).
Fig. 6 Profiles of user 39 and user 40 learned by the ITR system.
123
Univ Access Inf Soc Fig. 7 Books recommended by ITR to user 39, who issued the query ‘‘programming’’
When a user submits a query q, the books bi in the result set Rq are ranked by the classification value P(c+|bi), bi [ Rq, computed according to Eq. (2).(The exact posterior probabilities are determined by normalising P(c+|bi) and P(c-|bi), so that their sum is equal to 1. The result set retrieved by ITR in response to the query q = ‘‘programming’’, submitted by user 39, is presented in Fig. 7. The first book displayed is ‘‘Expert Systems in Finance and Accounting’’, in accordance with the interests contained in the user profile. Conversely, if another user submits the same query, the books in the result set are ranked in a different way, due to the fact that this user has a different profile (user 40 in Fig. 6). In this case, the first book displayed is ‘‘Java Professional Library’’ (Fig. 8). These scenarios highlight the effect of the personalization on the search process, where the result set depends on the profile of the user who issued the query. An empirical evaluation has been carried out with the aim of comparing the accuracy of different search results obtained by using coarse-grained or fine-grained user profiles. Results demonstrate the effectiveness of adopting more refined content-based profiling techniques in the retrieval process [31].
6 Conclusions The amount of information available on the Web, as well as the number of e-businesses and web shoppers, is growing exponentially. The Web has quickly become a global marketplace, and enterprises are developing new
123
business portals, providing their customers with large amounts of product information. Choosing among so many options is very time consuming; therefore, customers have started asking for online personalised recommendations, in the same way a salesperson provides information in a traditional shop. This problem requires solutions that show a certain degree of autonomy, personalization, and ability to react to specific circumstances. This paper has presented a modern architecture for e-commerce that integrates a conversational agent and personalization technologies. The architecture has been designed taking into account the requirements analysis made for BOL.com, a real dot com company offering books, music, and gifts via the Internet. Personalization is obtained by machine learning techniques that discover knowledge about user preferences from interactions. The result of the learning phase is exploited to build personal profiles that can be used in the retrieval process as a source of additional terms to be enclosed in a user-defined query. The profiles inferred by the COGITO system are coarsegrained, as they contain the book categories preferred by a user. A technique to infer more accurate fine-grained user profiles is also proposed in order to take into account users’ preferences in each category. Fine-grained user profiles are able to distinguish between interesting and not interesting products in each category, in order to achieve more precise recommendations. The performance of the agent has been evaluated in order to be able to understand how well the goals of a system are being achieved. The agent has been evaluated by two groups of test persons, less experienced and
Univ Access Inf Soc Fig. 8 Books recommended by ITR to user 40, who issued the query ‘‘programming’’
experienced Internet users, to look at the agent’s capability to facilitate the interaction between the user and an e-commerce site. The lesson learned from analysing the agent–user dialogue is that the precision of the agent in terms of correct output leaves a positive impression. On the other hand, the quality regarding the flow of the dialogue measured in terms of the proportion of fall-back sentences produced by the agent during the evaluation sessions might show a less positive result. However, the result of measuring the flow of the dialogue using fall-back sentences must be interpreted with care. For example, the broader context in which the user–agent dialogue is situated is not taken into account when counting fall-back sentences at ‘‘face value’’. That is, fall-back sentences might result in useful information for the user in some cases. A last lesson learned from analyzing the agent–user dialogue is that a slight difference was found in the query length observed, compared with a traditional web search engine, in favour of the agent system. Since the conducted study is limited in terms of number of respondents, and the results were compared only with two other web-query studies, care must be taken in drawing any general conclusions on this finding. Regarding the eye-movement tracking analysis, it was found that only a very small amount of viewing time was dedicated to the appearance of the agent, while much of viewing time has been spent outside the display. In the light of the results of the evaluation process, it can be concluded that the agent offers a reasonable satisfying support to customers during interaction, providing personal recommendations and helping users in problematic situations during search. However, based on the analysis of the questionnaire, it seems that more experienced users possess a better capability of benefiting from the qualities of the agent, as the claim of being satisfied or very satisfied with the agent is
about 70% higher for the experienced users than for the less experienced users, considering all the evaluated aspects: impression, control, effectiveness, navigability, learnability, aidability, and comprehension of the agent. On the other hand, the relative satisfaction of the less experienced users with the learnability of the agent is important, as one of the goals of the agent is to facilitate the access of new users to the Internet.
References 1. Adomavicius, G., Tuzhilin, A.: Using data mining methods to build customer profiles. IEEE Comput. 34(2), 74–82 (2001) 2. Allen, C., Yaeckel, K.D., Yaeckel, B.: Internet world guide to one-to-one web marketing. Wiley, New York (1998) 3. Allen, J., Ferguson, G., Stent, A.: An architecture for more realistic conversational systems. In: Proceedings of International Conference on Intelligent User Interfaces (IUI), pp. 1–8 (2001) 4. Andersen, V., Andersen, H.H.K.: Evaluation of the COGITO system. Deliverable 7.2, COGITO IST-1999–13347. Risoe National Laboratory, Denmark (2002) 5. Belkin, N.J., Cool, C., Kelly, D., Kim, G., Kim, J.-Y., Lee, H.-J., Muresan, G., Tang, M.-C., Yuan, X.-J.: Query Length in Interactive Information Retrieval, SIGIR’03, Toronto (2003) 6. Belkin, N.J., Cool, C., Stein, A., Thiel, U.: Cases, scripts, and information seeking strategies: on the design of interactive information retrieval systems. Expert Syst. Appl. 9(3), 379–395 (1995) 7. Borchers, A., Herlocker, J., Konstan, J., Riedl, J.: Ganging up on information overload. IEEE Comput. 31(4), 106–108 (1998) 8. Cassell, J., Sullivan, J., Prevost, S., Churchill, E.: Embodied Conversational Agents. MIT Press, Cambridge (2000) 9. Chai, J., Horvath, V., Nicolov, N., Stys-Budzikowska, M., Kambhatla, N., Zadrozny, W.: Natural language sales assistant— a web-based dialog system for online sales. In: Proceedings of the 13th Innovative Applications of Artificial Intelligence Conference, pp. 19–26. AAAI Press, Menlo Park (2001) 10. Chai, J., Horvath, V., Nicolov, N., Stys, M., Kambhatla, N., Zadrozny, W., Melville, P.: Natural language assistant. A dialog system for on-line product recommendation. Artif. Intell. Mag. 23(2), 63–75 (2002)
123
Univ Access Inf Soc 11. Colby, K.M.: Artificial Paranoia. Pergamon Press, New York (1975) 12. Cooperstein, D., Delhagen, K., Aber, A., Levin, K.: Making Net Shoppers Loyal. Forrester Research, Cambridge (1999) 13. Crowder, R., Wills, G., Hall, W.: Computers in industry evaluation of a hypermedia maintenance support application. Comput. Ind. 51(3), 327–344 (2003) 14. Ekman, P.: Emotion in the Human Face. Cambridge University Press, New York (1982) 15. Hof, R., Green, H., Himmelstein, L.: Now it’s YOUR WEB. Business Week (October 5), 68–75 (1998) 16. Jameson A.: Adaptive interfaces and agents. In: The Human– Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. Lawrence Erlbaum Associates, Inc., Mahwah (2002) 17. Jansen, B.J., Spink, A., Saracevic, T.: Real life, real users, and real needs: a study and analysis of user queries on the web. Inf. Process. Manag. 36, 207–227 (2000) 18. Kirakowski, J., Corbett, M.: SUMI: the software usability measurement inventory. Br. J. Educ. Technol. 24(3), 210–212 (1993) 19. Kobsa, A.: Tailoring privacy to users’ needs. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) Proceedings of User Modelling 2001, LNAI, vol. 2109, pp. 303–313. Springer, Berlin (2001) 20. Kobsa, A.: Generic user modeling systems. User Model. Useradapt. Interact. 11(1–2), 49–63 (2001) 21. Mitchell, T.: Machine Learning. McGraw-Hill, New York (1997) 22. Mladenic, D.: Text-learning and related intelligent agents: a survey. IEEE Intell. Syst. 14(4), 44–54 (1999) 23. Ogden W.C., Bernick P.: Using natural language interfaces. In: Helander M. (ed.) Handbook of Human Computer Interaction. Elsevier, North-Holland (1996) 24. Pazzani, M., Billsus, D.: Learning and revising user profiles: the identification of interesting web sites. Mach. Learn. 27(3), 313– 331 (1997) 25. Peppers, D., Rogers, M.: The One to One Future: Building Relationships One Customer at a Time. Currency Doubleday, New York (1993)
123
26. Peppers, D., Rogers, M.: Enterprise One to One: Tools for Competing in Interactive Age. Currency Doubleday, New York (1997) 27. Porteous, M., Kirakowski, J., Corbett, M.: SUMI User Handbook. Human Factors Research Group, University College Cork, Ireland (1993) 28. Resnick, P., Varian, H.R.: Recommender systems. Commun. ACM 40(3), 56–58 (1997) 29. Riecken, D.: Personalized views of personalization. Commun. ACM 43(8), 27–28 (2000) 30. Schafer, J.B., Konstan, J.A., Riedl, J.: E-commerce recommendation applications. Data Min. Knowl. Discov. 5(1/2), 1115–153 (2001) 31. Semeraro, G., Degemmis, M., Lops, P.: (2002) User profiling to support Internet customers: what do you want to buy today? Inf. Int. J. Comput. Inf. 26(4), 407–418. Informatika Society ISSN: 0350–5596 32. Stein, A., Gulla, J.A., Thiel, U.: User-tailored planning of mixed initiative information-seeking dialogues. User Model. User-adapt. Interact. 8(1–2), 133–166 (1999) 33. Tasso, C., Omero, P.: Personalization of web content: e-commerce, i-access, e-government. Franco Angeli (in Italian) (2002) 34. Thiel, U., Everts, A., Lutes, B., Tzeras, K.A.: Logic-based approach to search in digital libraries. In: Fankhauser, P., Ockenfeld, M. (eds.) Integrated Publication and Information Systems, 10 Years of Research and Development, pp 169–186. GMD- Forschungszentrum Informationstechnik, Sankt Augustin (1998) 35. Webb, G.I., Pazzani, M.J., Billsus, D.: Machine learning for user modeling. User Model. User-adapt. Interact. 11(1-2), 19–29 (2001) 36. Weizenbaum, J.: ELIZA—a computer program for the study of natural language communication between man and machine. Commun. ACM 9(1), 36–45 (1966)
