Web usability testing â CARE methodology

Web usability testing – CARE methodology Arthi Anandhan, Senthil Dhandapani, Hassan Reza, Karthik Namasivayam Department of Computer Science, University of North Dakota, Box 9015, Grand Forks, ND 58202-9015 {[email protected]}

Abstract Usability testing is a technique that is used to elicit the quality of systems and sites by regular tests carried with the potential users. Testing websites for usability requires an understanding of the sites’ audience, category, content, usability goal and how to measure to achieve these goals. There is a whole variety of usability testing methods available for web based testing. But our main objective was to make a pick of those right methods that could be implemented to test usability of any web application given a short time, small budget and fewer resources. This paper describes how a combination of some of the existing methods resulted in CARE (Cheap, Accurate, Reliable, Efficient testing), a new methodology. CARE could be engaged to surmount the constraints on cost, time and resources. Key Terms – Usability, evaluation, questionnaire, user testing, thinking aloud protocol, question asking protocol, co-discovery method

1. Introduction Through past decades, it became obvious that one of the most important qualities of software product would be the ease by which the end user can learn and interact with the product. In general, the quality attributes of a system (e.g., reliability) are the ones that are blamed for a system reengineering, and they are independent from system functionality [10]. Software quality includes reusability, reliability, usability, etc. It is often noticed that people spend huge amount of money on fancy design for an application rather than spending comparatively less to check this quality. Usability of a system is considered as a key quality because it contributes to user dissatisfaction

and frustration that eventually will result in the total abandonment of the system. With the increase in web activities like e-commerce, e-banking etc., the focus on the usability of web applications (web usability) becomes essential. The test for web usability is a fact-finder revealing the extent to which the product is found to be useful by specified users in achieving specific goals. As an outcome of the evaluation and usability tests, come adjustments that improve the system. The CARE methodology was an outcome of our intention to perform usability test on Electronic Review System (ERS), a website that was at the verge of being released. The developer had not sought any input from the users during the creation of this website. So a quick round of pre-launch usability testing of the website was seen as a requisite. The methodologies of usability evaluation can be divided into two broad categories: those that gather data from actual users and those that can be applied without actual users present. We wanted to concentrate more on the data from actual users thus leading to more methods that involve actual users. Usability testing like any component of development, involves a certain amount of planning, thought, process development and execution. Combination of different factors that affect the user's experience with the product or system include ease of learning, efficiency of use, memorability, error frequency and severity and subjective satisfaction. The sections that follow discuss the related work in usability testing, the approach (general steps) to test usability, our contribution towards usability testing and the steps in conducting the CARE method to test for usability. The implementation of the CARE method on a product is presented as a case study in the later section and concludes with the result of the test and future work.

Proceedings of the Third International Conference on Information Technology: New Generations (ITNG'06) 0-7695-2497-4/06 $20.00 © 2006

IEEE

Usability & Data/Information Requirements

participant is thinking on the spot. Being exhaustive for the users to verbalize a thought process throughout testing is a drawback.

Plan & prepare the evaluation - Measures - Observations - Lab configuration - Protocols - Participant type & no. - Tasks / scenarios

Question asking protocol [8] takes thinking aloud one step further where testers prompt users by questioning about the product. A better understanding of the user's mental model and interaction with the product along with focus on specific areas makes the method advantageous. On the other hand, the method often causes the users to modify their performance.

Analyze data & report results

Revise Interface

Recruit & schedule participants

Run test sessions in configured labs

Re - Test

Figure 1. Structure of usability testing

2. Related work Jakob Neilson has done study on implementing usability test in a short time. But we wanted to create a methodology to conduct the test not only in a short period but also with fewer resources and less budget. The different methods available for usability testing are divided into three main categories: ‘Inquiry’, ‘Inspection’ and ‘Testing’. Consequently, the job of selecting relevant methods for evaluating a particular type of interface can be daunting. We avoided the mode of testing through inspection mainly because it does not involve real users but goes by the experts’ opinion and it costs a lot to hire experts. A study of some of the other methods, left us with the list of the following 5 methods: User Testing [1,7] is a typical approach, where participants (target audience), one at a time or two working together, use the web application to perform predefined tasks, while one or more observers watch, listen, and take notes. Eliciting affirmative results that are more convincing to the experts is the advantage of this method in spite of the fact that it is expensive. Thinking aloud protocol [4] is also a popular technique where the users vocalize their thoughts, feelings, and opinions while interacting with the product. The result is valuable insight into what the

Co-discovery method [9] is one that has two participants helping each other in the same manner they would if they were working together on a common goal. Its interactive nature potentially produces many relevant issues and insights while it is difficult to watch two people working and turns out to be expensive. Questionnaire [3] is defined, in a more structural way as "a method for the elicitation, and recording and collecting information". The relatively inexpensive method allows flexible administration and describes characteristics of a large population but is devoid of direct observation. With the summary of the pros and cons of various methods, we envisioned our methodology.

3. Our Approach The focus of a usability test is the user's experience with a site and is mainly used for improving websites. Figure 1 illustrates the general steps involved in usability testing. Some basic steps described for usability testing are: Select a small group of people who are representative of the real audience. Watch the user performing assigned tasks on the website, and note where they fail. Examine the data gathered from direct observation of users; figure out why the subjects fail to perform tasks. Explain the problem areas to the designers and the site owners, and take corrective actions. Finally, start over testing the supposedly corrected site again with a new group of users.


IEEE

4. Evolution of CARE method A full-blown usability test tends to be timeconsuming (usually a month or more) and expensive (needing the use of a usability testing lab, several professionals and a paid group of volunteers) which are the major disadvantages. As a remedy, we wanted to design CARE, a combination of some popular and effective methods.

5. Testing usability with CARE method The steps in conducting the CARE method to test usability of any web application are discussed below. The case study section describes a module from ERS that was tested using the CARE method. 5.1 Requirements The main objective in designing CARE method was to conduct effective usability testing in spite of constraints on time, cost and resources. So the method had relatively very few requirements. The steps in applying CARE method to test usability are as follows: Step 1: Problem definition Identifying the problem statement or objective of the test is the key element of designing an effective usability test. These objectives guide the rest of the test and hence need to be properly defined. Step 2: Preparation of task list A set of predefined tasks are prepared for the users to perform during usability testing. Recognizing exactly which part of the application needs testing and what concerns one has about the application that is to be tested helps in the preparation of task list. It is important to select relevant tasks for users to try. Step 3: Selecting sample users It is essential that the sample users recruited accurately represent the current or potential users. Identifying the target audience and then recruiting representatives from the audience to participate in the usability test is beneficial. Step 4: Schedule and conduct the test The test is scheduled according to the users’ convenience and conducted accordingly. A formal test is always accompanied by a trained facilitator to

interact with the user. The observers watch, listen and take notes concentrating on observations of behavior rather than on inferences. During the test, the participants are encouraged to express their reactions as they work. Step 5: Data collection The data (performance, remarks and suggestions from the users) that is collected during the test helps in ascertaining the usability level of the application. Based on the analysis made on the collected data, the changes are made to make the application more effective. Step 6: Result Analysis The main objective of testing is to see if the users feel the application efficient and easy to use. So, the users’ comments, remarks and suggestions make together the test results. After the test, the data from all participants gets compiled and the problems that participants had are listed. Based on the priority and frequency, the problems are sorted and solutions developed. If the solutions are not obvious to fix the problems, expert advice is sought. Proper result analysis helps in making right decisions and solutions

6. Case Study Electronic Review System (ERS) was the website that we intended to test for usability. ERS is an attempt to automate the existing manual system of reviewing manuscripts and technical papers before publishing. The main objective of the application is to expedite the evaluation process of manuscripts. The users of this website are in two forms: Authors – who submit their papers for reviews. Reviewers – who review and evaluate the technical papers submitted and aid in improvising the paper if necessary. The main processes that take place in the website are submitting papers for review and submitting reviews for the submitted papers. Authors post their papers in the website. People involved in the field of research studies review and suggest any changes before the manuscript could be published. Step 1: Problem definition Our purpose of conducting the usability testing was to determine if the ERS website is easy to use even for a novice. The data collected within the test was expected to facilitate in making the website


IEEE

easier to use, finding information fast and less frustrating for users. Step 2: Preparation of task list The developer of ERS website defined a list of tasks (scenarios) to be performed by the users during the test. The list contained the tasks that were determined to be the most commonly performed in the website. Some of the scenarios that the developer defined for testing the site were creating and activating a user account in the website, logging into the account, submitting manuscripts for review, reviewing/rating the submitted manuscripts etc. Step 3: Selecting sample users Pre-test questionnaire is the screening questionnaire that was used as a method for identifying potential users. The answers to the questions were taken into consideration when selecting the participants. The Electronic Review System (ERS) is not bound to any single department or subject. It can be commonly used by anyone involved in research field regardless of their expertise. Hence our sample users contained people who are involved in research, who are not involved in research, who have good computer skills and people with less computer skills from different departments. Abiding to the testing requirement [2], 7 sample users were selected for testing the ERS website. We chose 2 faculties, 1 doctorate student and 4 graduate students from different departments and background. Step 4: Schedule and conduct the test A schedule was prepared and confirmed according to the convenience of the students and the faculties who were to participate in the test and the availability of the lab. It was taken care to schedule one or two tests a day. To keep within the budget, the test was conducted in the computer lab that is available in our department. Before the test the user was given instructions about the test. Throughout the test, the user was assisted by an observer who is the web developer in our case. Step 5: Data collection The participants were given a list of predefined tasks during the test. At the end of each task, the user was asked to fill a questionnaire. This data was very useful in identifying the difficult areas of operation.

During the test, the user was encouraged to give comments as they interacted with the website. A note of their activities and their comments were made by the observer. It facilitated in capturing the users’ preference. The main concern of the test was to see if the users apprehend the terminologies, menus and navigation through different web-pages. The web page response time was also noted. At times, the observer also probed the users in order to get more information about the website and the users’ overall opinion of the site. At the end of the test, the data collected was tabulated by the administrator.

Figure 2. Step 1 of 4 for submitting papers for review.

Step 6: Result Analysis As the main purpose of the website is to allow users to electronically submit manuscripts and reviews, we had tested modules for submitting manuscripts for review and submitting reviews for manuscripts. For illustrative purpose, out of a set of tasks defined, we consider just one: to submit a manuscript for review. However, in order to submit the manuscript, the user needs to gain entry by logging on via an ERS account which handles the access permissions. Thus the task sequence, starting from the ERS home page was to: Log in to the user account, Identify and click the link to submit a manuscript (fig.2) Execute the 4 steps of the process Successful submission of the manuscript Logout Assuming the user has rightly identified the link and has arrived to the main page (as shown in fig.2),


IEEE

we see that this page has many features: the menu on the side and the links at the top are accessible from every ERS page; there is information about the ERS website statistics showing the number of authors and reviewers currently checking the website, the hits made on the website; a column with the latest news scrolling. We considered user goals, actions and feedback for each of these features.



As a first step of the process, information about the author and the manuscript is collected (fig2.). Some of the information is details of the author submitting the manuscript, type of manuscript submitted, title of the manuscript, the abstract, and the number of coauthors (if any). The information is used to sort the manuscripts while submitted for review. A manuscript is sometimes written by more than one author. In such cases, details about the coauthor(s) are collected in step 2 (fig.3). Uploading the manuscript is step 3 (fig.4). This step allows the users to upload their papers. But the files were to be of .doc or .PDF format only. The final step (fig.5) allows the user to verify the information provided before confirming the submission of the manuscript. Findings and discussion: Our main concern in this scenario was to see if the users are able to complete the process by following the 4 steps. Some of the issues that were identified during the test were:

Users preferred separate link for submitting manuscripts rather than then combined link (fig.2). Users could not make out what certain terminologies like ‘specialty’ and ‘subspecialty’ meant (fig.2). The term ‘Co-author’ was preferred to ‘Contributing author’ (fig.3). Users suggested that more file formats could be allowed for uploading in step 3 (fig4). Fonts in some pages were reported to be small and difficult to read because of the use of light colors. Users appreciated the 4 step process to be very helpful. The logout button was not easily identified.

Outcome: The observations were made on each user’s response, comments, actions and feedback. The test results were analyzed and brought in various changes and decisions. It was decided to give users the option of uploading any file format which will be automatically converted to PDF format. Terminologies difficult to understand were changed. Font size and colors were changed according to the users’ need. Some of the issues identified from other scenarios are Difficulty in navigating to the right page for review submission. When the user attempts to submit a review for a manuscript already reviewed by him, an error message is given only at the end of a complete long process which seemed frustrating to the user.


IEEE

end of usability testing. Usability could be achieved starting from the design phase itself which is bound to reciprocate a cheap and efficient result, the study of which is planned for the future.

8. References [1] Jakob Nielsen, “Designing Web Usability: The Practice of Simplicity” New Riders Publishing, Indianapolis ISBN 1-56205-810-X [2] Nigel Bevan, Carol Barnum, Gilbert Cockton, Jakob Nielsen, Jared Spool, Dennis Wixon, “Accepted Panels: The "magic number 5": is it enough for web testing?”,USA, Year of Publication: 2003 ISBN:1-58113-637-4.

Figure 5. Step 4 of 4 for submitting papers for review

7. Conclusion and future work

Low Cost

Reliable

User Testing Thinking aloud protocol Question asking protocol Co-discovery method Questionnaire method CARE Methods

X X X X

9 9 9 X 9 X

Less Resource

Methods

Less Time

Reports show that at least a month’s time with big budgets and highly equipped test as requirements for an effective usability testing. But we took an estimated time of about three weeks to research, design and perform the test.

X

X X X X

9 X 9 9 9 9

9 9

X

Table 1. Comparison of different usability methods with CARE method

Table 1 shows a comparison of some of the methods and the CARE method. Though our work proves that usability testing can be done successfully with limitations in resource, budget and time, the result in choosing other methods involving specialized equipments and facilities would differ. Failing to involve users in the early stages of the development cycle had led to various changes at the

[3] Bartek, V. and Cheatham, D. “Experience Remote Usability Testing, Part 2: Examine the Benefits and downside of Remote Usability Testing.” Retrieved from http://www-106.ibm.com/developerworks/ web/ library/wa-rmusts2.html [4] S. E. Ransdell, “Generating thinking-aloud protocols: Impact on the narrative writing of college students,” Amer. J. Psychol., vol. 108, no. 1, pp. 89– 98, 1995. [5] Jakob Nielsen and Kara Pernice Coyne, “A Useful Investment: Usability testing costs - but it pays for itself in the long run”, Published by CIO article in 2001. [6] Czaja, Ronald, and Johnny Blair, 1996, “Designing Surveys”. Pine Forge Press. ISBN 08039-9056-1. [7] Tom Brinck, Darren Gergle, Scott Wood, “Usability for the Web: Designing Web Sites that Work”, Publisher: Rebound by Sagebrush (October, 2001) ISBN: 0613920546 [8] Kato, T (1986). What "question-asking protocols" can say about the user interface, International Journal of Man-Machine Studies, Vol. 25,659-673. [9] Wilson, C. Pros and cons of co-participation in usability studies. Usability Interface 4, 4 (April 1998). [10] M. Jazayeri, A. Ran, and F. Linden. Software Architecture for Product Families, Addison-Wesley, 2000.


IEEE

Web usability testing â CARE methodology - IEEE Xplore

Web usability testing â CARE methodology - IEEE Xplore

Suggest Documents

Methodology of testing Electromagnetic Interference ... - IEEE Xplore