A Study into the Usability of E-encyclopaedias Ruth Wilson
Julie Shortreed
Monica Landoni
Department of Computer and Information Sciences University of Strathclyde Glasgow, UK +44(0)141 548 3705
Information Services University of Abertay Dundee UK +44(0)1382 308836
Department of Computer and Information Sciences University of Strathclyde Glasgow, UK +44(0)141 548 4949
[email protected]
[email protected]
[email protected]
ABSTRACT This paper describes a usability study performed on three different e-encyclopaedias, all of which were available on the web, contained similar content and targeted similar audiences, but differed in their styles of presentation. The purpose of the study was to discover the effectiveness of each design in terms of overall usability and subjective satisfaction, to extract general guidelines for the successful design of e-encyclopaedias, and to examine the requirements of different user groups.
Keywords electronic encyclopaedias, e-encyclopaedias, usability, evaluation
1. DESIGN OF EDUCATIONAL MATERIAL FOR THE WEB Educational websites and electronic textbooks are becoming progressively similar in terms of both content and presentation, as scholars mount their course material online in increasingly sophisticated ways and print resources are redefined for electronic delivery. Textbooks need no longer be supplied in their entirety to students; rather, selected chapters or sections can be provided electronically, alone or in a “package” with chunks of other textbooks or complementary material. The notion of the book as a static set of pages bound together by a cover is fading, as more and more of its traditional features are transformed. Educational websites, at the same time, are becoming more advanced, bringing together rich, scholarly material and employing many of the same design elements as electronic textbooks, such as internal and external hyperlinks and multimedia. Based on this observation, that the “look and feel” of educational websites and electronic textbooks are overlapping and the boundaries between them blurring, the EBONI (Electronic Book ON-screen Interface) Project [1] asked if design principles which have been proven to succeed in one area could be borrowed by the other.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SAC 2003, Melbourne, Florida, USA © 2003 ACM 1-58113 -624-2/03/03…$5.00.
The design of material for the web is a well-researched area, and principles are now well-established [2, 3, 4], whereas e-book design is still in its infancy and many of the usability studies which have taken place have focused on handheld devices [5, 6, 7]. EBONI investigated the area of on-screen e-book design, taking Morkes and Nielsen’s web usability guidelines as a starting point [8], and evaluated their applicability in a series of e-book evaluations involving over 100 students, lecturers and researchers from a variety of disciplines in UK Higher Education [9]. It was discovered that certain web design principles could indeed be borrowed by the field of e-book design, but that some traditional features should also be retained. Specifically, in the resulting set of Electronic Textbook Design Guidelines [10], two main ingredients for a successful electronic book are proposed: adherence to certain familiar aspects of the paper book metaphor, such as page length, tables of contents and indexes; and “scannability”, a notion inherited from web design principles which encourages the use of extra headings, coloured text and images to break up long streams of text [11]. During the summer of 2001, the project also conducted a study into the usability of electronic encyclopaedias (e-encyclopaedias) [12]. As resources which were previously published in print but which comprise short “bites” of non-linear content, these sit somewhere between electronic books and web sites and, as such, presented interesting design questions to the project. Whereas many books have arguably been comfortable within the “limitations” of print, encyclopaedias have been constrained by a linear, alphabetical structure since their beginnings in the 18th century, hindering access to the rich organisation and interconnections among articles [13]. It was just over a decade ago that a seamless “circle of learning” started to become a real possibility with hypertext, which can take the reader “out of a linear model and into a connected network of thought” [14]. The electronic medium, therefore, presents great opportunities for encyclopaedias, if designed well. EBONI’s experiment aimed to define what is meant by a well-designed e-encyclopaedia by examining the requirements of a range of users, and to determine the relationship between e-encyclopaedia, e-book and web usability. This paper describes the evaluation of three web-based encyclopaedias, asking to what extent design principles from the e-book and web usability arenas are applicable, and attempting to understand this dynamic, evolving genre that is highly adapted to the online environment.
2. E-ENCYCLOPAEDIA EXPERIMENT 2.1 Hypotheses The study involved three e-encyclopaedias, all of which were available via the web, contained similar material and targeted similar audiences but exhibited different styles of presentation. The purpose was to discover the effectiveness of each design in terms of overall usability and subjective satisfaction, and to extract general guidelines for the successful design of eencyclopaedias. The selected material ranged from an e-encyclopaedia which closely resembled its paper counterpart in terms of appearance to a highly interactive, complex hypermedia one which had a variety of added value content and bore little, if any, resemblance to a print encyclopaedia. As identified above, the web offers particular opportunities to eencyclopaedias in the form of interactivity and added value content. The first aim of the experiment was to measure the impact of these factors on usability; as such, the degree to which these elements were utilised by each e-encyclopaedia was measured: •
Interaction (the degree of interaction available to the user).
•
Added value (the degree of added value content available to the user).
The overall criteria for judging the effectiveness of these elements were: •
“Sense of directness”, which measures the user’s ability to learn and internalise the interface.
•
“Sense of engagement”, which measures the level of interest induced in the user and is related to the level of interaction available and the novelty of the system [15].
The following hypotheses were put forward: •
•
Hypothesis 1. A high degree of interaction and added value content will not have a positive effect on sense of directness (the added complexity will increase users’ confusion and disorientation and greater cognitive effort will be required to achieve a satisfactory level of performance). Hypothesis 2. A high degree of interaction and added value will have a positive effect on sense of engagement (users will experience more pleasure and satisfaction and their level of interest will be higher).
A secondary aim of the study was to investigate how age and educational background affect the way people interact with the selected material. If e-encyclopaedia publishers are to successfully tailor content and design to suit the needs of their diverse audience, then an awareness of how age and background can influence use is required. The methodology was based on EBONI’s e-book evaluation framework, specially developed by the project to provide cohesiveness to all its individual experiments [16]. It comprises several stages: selection of material, selection of actors, selection of tasks, and selection of techniques.
2.2 Selection of Material Three free e-encyclopaedias were selected for evaluation, all of which differed in terms of presentation, navigation and added value content. They were analysed in detail with reference to a range of current web usability guidelines [17, 18, 19, 20]. Britannica.com (Britannica) [21] was the least interactive of the three in terms of its functionality, and added value content was limited to links to external resources. It is a search-dominant resource, with a simple interface. The 6th edition (2001) Concise Columbia Encyclopaedia (Columbia) [22] bore most resemblance to its print counterpart in terms of content, layout and appearance, albeit enhanced by a search function and hyperlinked cross-references. Articles were indexed alphabetically and the page layout was consistent with the paper book metaphor. The third encyclopaedia, Microsoft’s Encarta (Encarta) [23] was the most interactive in terms of functionality and content and offered the most added value content including multimedia. It was also the largest and most complex encyclopaedia of the three.
2.3 Selection of Actors 36 participants took part in the experiment. An important aim of the study was to investigate how age and educational background would affect the way people interact with the selected material. To this end, the 36 participants belonged to two distinct groups: 1.
“School group”. This comprised a class of 5th and 6th year pupils from Perth Grammar School sitting their Higher Computing exams. All 15 participants were aged 16 or 17 and had obtained either a Standard Grade in Computing or Information Systems Intermediate 2 as entrance to the Higher course.
2.
“Online group”, composed of 21 postgraduate students and mailing list respondents, ranging from 19 to over 50 years of age. The postgraduate students were studying for a Postgraduate Diploma or Msc in the Department of Information Science, University of Strathclyde. The mailing lists were: a.
JISCmail lis -e-books, aimed at those in the academic community with an interest in e-books.
b.
The eBook Community mailing list, for the general discussion of e-books and the e-book industry.
It was assumed that online participants would all have a qualification or professional interest in information science in general and e-publishing in particular. Postgraduate students were motivated to complete the survey by a desire to help the evaluator (a fellow-student) and an interest in the subject; mailing list respondents’ motivation stemmed from their professional interest in the outcome of the study and the advancement of research into e-books. 75% of online respondents used the Internet daily, and 80% of school participants used it at least two or three times a week; the Head of Computing confirmed that all pupils were familiar with the Internet and used it on a regular basis as part of the school curriculum. In both groups, the majority of respondents had not used an e-encyclopaedia before, and none used one on a regular basis.
One evaluator coordinated all aspects of the study, and collaborated with the Head of Computing at Perth Academy with regard to task development and assessment.
2.4 Selection of Tasks and Techniques Tasks and techniques were inherited from EBONI’s earlier experiments. Scavenger Hunt [24] and memory tasks were used as a means of measuring accuracy, recognition and recall, and were designed with two aims in mind:
•
To ensure that participants used all the features of the encyclopaedias, in terms of presentation, navigation and content. To ensure that neither group was disadvantaged due to differences in age and education. Participants were asked about astronomy, a topic of general interest, and tasks were designed to be challenging enough to require both groups to use the encyclopaedias in order to discover the answers, yet not to be beyond the capabilities of either group.
Subjective satisfaction questionnaires were completed by all participants after the tasks. Due to the limitations of time (school group) and resources (online group), interviews and think-aloud sessions were not conducted. However, the evaluator was able to observe the school pupils during the evaluation.
2.5 Procedure A between-subjects design was adopted, with each participant evaluating only one encyclopaedia. All participants were asked: •
To provide details about themselves regarding age, and their experience of the Internet and electronic encyclopaedias.
•
To visit a particular e-encyclopaedia and complete some tasks.
•
To complete a subjective satisfaction questionnaire.
The tasks and subjective satisfaction questionnaires were identical for both groups; however the school group’s evaluation was conducted in a controlled environment with a fixed time limit, while the online group’s evaluation was conducted over the Internet, unsupervised and with no time limit.
2.6 Measures The measures and criteria for evaluation were derived from EBONI’s e-book evaluation methodology, and gathered information about participants’ sense of engagement and sense of directness with the e-encyclopaedias: •
Criteria
Measures
Satisfaction
Quality Ease of use Likeability User affect
Memorability
Recognition Recall
Sense of directness
Usability
Task success
Data collection methods Subjective satisfaction questionnaires (qualitative and quantitative) Memory tasks (quantitative) Scavenger Hunts (quantitative)
By incorporating tasks which require a range of cognitive skills, and questionnaires designed to measure user satisfaction, the experiment aimed to measure usability comprehensively and at a variety of levels. Moreover, by adopting a methodology that was consistent with EBONI’s, it could be “mapped” and compared to past and future evaluations in the field.
2.7 Results Data was analysed individually for each measure, and compared across the three encyclopaedias and the two groups of participants. Figure 1 shows participants’ overall sense of directness and sense of engagement with the three encyclopaedias. 100 90 80 70 60 50 40 30 20 10 0
Britannica Columbia Encarta
Sense of directness
Sense of engagement
Two criteria comprised sense of engagement: o
o •
Metacriteria Sense of engagement
percentage
•
Table 1. Techniques used in the e-encyclopaedia experiment, and their relationship to the evaluation criteria.
Satisfaction, including quality, ease of use, likeability and user affect, and measured via the subjective satisfaction questionnaire. Memorability, including recognition and recall, measured via the memory tasks.
In this study, the criterion for sense of directness was usability, and participants’ success at Scavenger Hunt tasks was used to measure the usability of each encyclopeadia.
The various tasks and techniques, their measurements and their relationship to the criteria for evaluation are outlined in Table 1 below.
Figure 1. Overall sense of directness and sense of engagement Hypothesis 1 was validated. Interaction and added value content did not have a positive effect on sense of directness (Encarta was the most interactive and had most added value content). When users were task driven and when the tasks involved specific factfinding, there was a clear indication that a design retaining elements of the book metaphor (Columbia), in terms of both functionality and presentation, made it easier for users to learn and internalise the system. Indeed, Columbia users scored significantly higher than Encarta users in the Scavenger Hunts (p
