a method of evaluating online usability research: a ...

IADIS International Conferences Interfaces and Human Computer Interaction and Game and Entertainment Technologies 2013

A METHOD OF EVALUATING ONLINE USABILITY RESEARCH: A CASE STUDY OF THE ITAÚ BANK SITE Fabiane Rodrigues Fernandes and Luis Carlos Paschoarelli PPGDesign – Pós Graduação em Design – UNESP/Bauru

ABSTRACT Through an assessment of the Itaú Bank website, this paper presents a methodology, based on ISO 9241 (NBR 9241-11:2002), to evaluate the effectiveness and efficiency of and satisfaction with online usability research. The final data were calculated as percentages based on score scale of SUS (ALBERT & TULLIS, 2008). Eight individuals participated in this study. Three were women and five were men, and their average age was 31 years. The results indicate that the interaction with the interface is satisfactory, but its efficacy must be improved. This paper also presents a simple method of measuring the level of performance of the interface. KEYWORDS Method, evaluating, HCI, usability.

1. INTRODUCTION Some time ago, banks began to offer online services to their customers, thus making banking far more convenient, since it was no longer necessary to go to an ATM or a bank branch. It is also true that such websites are still regarded by many as complex because of the amount of information that they provide online. Therefore, the investigation of the usability of banking websites has generated interest. Specifically, it is important to better understand the interaction between human beings and interfaces in order to improve the experience and satisfaction of users. Usability is a common term in the field of human-computer interaction (HCI), which has a particular interest in the design of products that better ensure human needs and welfare. Thus, ergonomic design seeks to create products that are more suited to human capacities. The Itaú Bank website is a channel of communication, information, and service. Through such websites, one can perform banking transactions and access financial credits and other information. At the end of 2012, the Itáu Bank revamped its website, making the homepage simpler by arranging information in blocks of distinct categories. It is important to determine if this change has bought significant improvements in user interactions with the website. Thus, through an assessment of the Itaú online banking website, this study presents a methodology, based on ISO 9241 (NBR 9241-11:2002), for evaluating the effectiveness and efficiency of and the satisfaction with online usability research.

2. ONLINE USABILITY RESEARCH Several methods of usability evaluation exist that fit into two categories: Inspection methods are assessments made by experts that employ specific criteria and guidelines, such as heuristic evaluations; empirical methods consists of system or product assessments by users who execute predetermined tasks. In the empirical method, a researcher normally carries out observations and record data. However, it is also possible to conduct usability evaluations through online studies. Albert et al. (2010) believe that the recent emergence of online testing of usability has resulted in savings of time and money, while offering valuable insights into user experiences. These authors argue that “Online usability studies are probably the most efficient way to collect detailed and scalable usability metrics” (ALBERT et al, 2010).

235

ISBN: 978-972-8939-90-8 © 2013 IADIS

Another advantage of online usability studies is that may be held in the participant's natural environment without a moderator's presence and influence. This feature prevents participants from saying what they think the moderator wants to hear.

2.1 Score Scale In an accepted standard scoring scale (BANGOR et al., 2009), all products with score more than 90% are considered exceptional, those with scores equal to 80% and below 90%, good, and those with scores between 70% and 79% acceptable. Any score below 70% indicates usability problems and the products that obtain them should be viewed with concern.

2.2 ISO 9241 (NBR 9241-11:2002) The International Organization for Standardization (ISO 9241-11) defines usability as the use of a product in a specified context by specific users who seek to achieve specified goals with effectiveness, efficiency, and satisfaction; with this definition, it is possible to make an overall evaluation of the interface. The ISO 9241 (constant in Brazilian standard NBR 9241-11:2002) is a tool for usability evaluation. This standard contains several topics related to the ergonomic aspects of software and hardware. In topic 11, the norm explains the measurement of overall usability through effectiveness (objectives successfully achieved), efficiency (tasks completed per unit of time or other variables), and satisfaction (a questionnaire). Sauro and Lewis (2012) find no specific guidelines for the measurement of effectiveness, efficiency, and satisfaction, but most tests involved a combination of completion rates, errors, times of tasks, satisfaction levels of tasks, satisfaction levels of interfaces, aids, and lists of usability problems, which often include frequency and severity.

2.3 Number of Participants Some researchers, such as Nielsen and Landauer (1993), believe that with five survey participants, “the magic number” (Albert et al., 2008), it is possible to identify 80% of the quality problems posed by an interface. Albert et al. (2008) argue that five appears to be a satisfactory number of participants, since in most of the usability tests conducted by them, the most significant problems show up by the fourth or fifth participant, regardless of the total number. Thus, these authors believe that the “magic number” can be used when the scope of a review (has a limited set of functions) and when the target audience is well defined and representative.

3. CASE STUDY – ITAÚ BANK WEBSITE 3.1 Method and Material This study is transversal and complies with ethical principles (Resolution 196/96--CNS / MS and Code of Ethics of Certified Ergonomist --ABERGO, 2003). It employs an online survey of usability, meaning that users interacted with the Itaú Bank website and provided information on this interaction through an online form. The study was based on statistical inference and an online protocol with five steps was applied: reading and accepting the terms of consent; completing an identification questionnaire, filling out an Internet use questionnaire; taking an “Interaction test” with three tasks; and completing a satisfaction survey. The tasks for the “Interaction test” consisted of requests, such as “Supposing that you want information about ... search for it on the website.” It was necessary to indicate the link found as a result of this search (the site has no search engine and no site map, being possible just one path to accomplish tasks and find information. For this reason, the individual was asked to paste the page link as a result found for tasks), the time needed to complete the task (less than 1 minute, between 1-2 minutes or more than 2 minutes), and if the task was completed with easily, with difficulty, or not completed.

236


For questions on the perception of and satisfaction with use, we utilized a questionnaire with 11 questions. Issues 01-03 had 3-point scale (good, neutral, or bad); issues 04-09 a 5-point scale (strongly disagree, disagree, neutral, agree, or strongly agree); and the issues 10-11, a 2-point scale (yes or no). To evaluate interaction, effectiveness was measured in terms of (tasks completed successfully, that is, correctly). A score of 0 equaled “not completed” and 1 “completed” (Table 1). The result is presented as a percentage, after its calculation with a formula inserted into Excel® software (=AVERAGE [E3: E11] * 33.3333333/100, in which E3: E11 is the sum of points of the tasks performed by each participant). Table 1. Scores for Subsequent Calculations of Effectiveness Tasks Completed correctly Not completed correctly

T1 Score 1 0

T2 Score 1 0

T3 Score 1 0

Overall 100% 0%

To measure efficiency (correctly completed tasks within the variables of time and degree of difficulty), a rating scale for each task was employed, as reported in Table 2. The result is shown as a percentage, calculated through a formula inserted into Excel® software (=AVERAGE [E3: E11]* 5.55555556/100, in which E3: E11 is the sum of points of the tasks performed by each participant). Table 2. Scores for Subsequent Calculation of Efficiency Task

T1

Description

Not Completed correctly

Completed with difficulty (more than 2 minutes)

Completed with difficulty (between 1-2 minutes)

Score

0

1

2

Completed with difficulty (less than 1 minute)

3

Completed with facility (more than 2 minutes)

4

Completed with facility (between 12 minutes)

Completed with facility (less than 1 minute)

5

6

For the measurement of satisfaction, a questionnaire containing 11 questions was employed (Table 3). The result is shown as a percentage, calculated through a formula inserted into Excel ® software (=AVERAGE [B13: I13}* 3.125/100, in which B13: I13 is the sum of the points of the questions of each participant). Table 3. Scores for Subsequent Calculation of Satisfaction Questions Description

Score

Questions 1-3 bad 0

neutral 1

good 2

Questions 4-9 strongly disagree 0

disagree 1

neutral 2

agree 3

strongly agree 4

Questions 10-11 no Yes 0 1

3.2 Results and Discussion Eight individuals participated in this study. Three were women and five were men; their average age was 31 years. Six of these individuals had completed college and two had not. Before the test of interaction, the participants were asked two questions about their perceptions of the interface before using it: the first asked if the site appeared was easy or difficult (easy, difficult, or no answer); and the second queried if compared to other bank websites that of Itaú website appeared easier or hard to use (easier, harder, or don’t know). The results, which are presented in Table 4, show the participants’ positive impression of the interface before using it.

237

ISBN: 978-972-8939-90-8 © 2013 IADIS

Table 4. Presentation of Impressions before Use (Itaú Bank Website) Impressions of the website before use Easy to navigate Don't know Not easy to navigate

P1 P2 1 0 0

Impressions in comparison to similar websites before use Easier to navigate Don't know Not easier to navigate

1 0 0

P1 P2 0 0 1

1 0 0

P3

P4

P5

P6

P7

P8

Overall

%

1 0 0

1 0 0

1 0 0

1 0 0

1 0 0

1 0 0

8 0 0

100 0 0

P3

P4

P5

P6

P7

P8

Overall

%

1 0 0

1 0 0

1 0 0

1 0 0

1 0 0

0 1 0

6 1 1

75 12,5 12,5

The effectiveness rating was 71% (Table 5), a score calculated from the average of the percentages of completed tasks of the scores shown in Table 1. The result is within the range considered satisfactory (Bangor et al., 2009). It is noteworthy that only two individuals failed to complete the three tasks, and only one task was completed correctly by all participants. Table 5. Presentation of Effectiveness Result Participants P1 P2 P3 P4 P5 P6 P7 P8 Effectiveness

T1 1 1 1 1 1 1 1 1

Tasks T2 0 1 0 0 0 1 0 0 Average

T3 1 1 1 1 1 1 1 0

OVERALL 2 3 2 2 2 3 2 1 71%

Efficiency attained a score of 60% (Table 6), calculated from the average of the percentage of tasks completed within the variables and the scores of Table 2. The efficiency results can be considered unsatisfactory (Bangor et al., 2009). Thus, the interface is effective because the participants were able to achieve a goal but inefficient because they realized it with difficulty and by taking too much time. Table 6. Presentation of Efficiency Result Participants P1 P2 P3 P4 P5 P6 P7 P8 Efficiency

T1 5 6 6 6 6 6 6 6

Tasks T2 0 5 6 0 0 5 0 0 Average

T3 1 5 2 6 6 2 1 0

OVERALL 6 16 14 12 12 13 7 6 60%

The score for satisfaction was 82% (Table 7) calculated from the average of the total scores of each participant in Table 3. The result, which can be considered good (Bangor et al., 2009), demonstrated that the participants are satisfied with the new interface of the Itaú Bank website. All participants felt that the website achieved its goal, and they recommend its interface. Issues 01-03 (its navigation, menu layout, and information) were rated on a 3-point scale (bad, neutral, or good); issues 04-09 (aspects of its fonts, colors, and other visible elements) were gauged on a 5-point scale (strongly disagree, disagree, neutral, agree, or strongly agree); finally, issues 10-11 (achievement of goals and recommendations of website) were rated on a 2-point scale.

238


Table 7. Presentation of Satisfaction Results Issues

P1

P2

P3

P4

P5

P6

P7

P8

Evaluation of menu layout information inside the menu nice interface consistent areas legible fonts good contrast clear label/title easy to assimilate information reached goal recommends use

1 1 1 3 1 3 4 2 2 1 1

2 2 2 3 4 4 3 3 3 1 1

1 2 1 4 4 3 4 2 2 1 1

2 2 1 4 4 4 2 3 3 1 1

1 2 2 3 1 3 2 3 3 1 1

2 2 1 4 4 4 3 3 4 1 1

2 1 2 4 4 3 3 4 4 1 1

2 2 2 4 4 3 3 4 4 1 1

25

27

22

29 29 82%

30

Overall SATISFACTION

20

28 Average

4. FINAL CONSIDERATIONS Although this study is preliminary, was conducted with a small number of participants, and was based on few tasks, it demonstrates that the Itaú website satisfied those who used it. The results for effectiveness, efficiency, and satisfaction are for specific tasks, participants, and contexts. This finding may be related the presentation of the homepage, whose interface the participants found easier to navigate than similar ones. In this regard, all participants concluded after interaction that the interface full fills its role and is worth recommending. Other usability tests, such as heuristic evaluations, ergonomic criteria assessments, and measures of effectiveness, efficiency, and satisfaction with more accurate and detailed tasks should be conducted on banking websites.

ACKNOWLEDGMENT This study was supported by CAPES/Brazil.

REFERENCES ABERGO, 2003. Associação Brasileira de Ergonomia. Norma ERG BR 1002 - Código de Deontologia do Ergonomista Certificado. Available from: http://www.abergo.org.br/arquivos/Norma_ERG_ BR_1002_-_Deontologia.pdf [Accessed at 08.03.2005]. ASSOCIAÇÃO BRASILEIRA DE NORMAS TÉCNICAS, 2002. NBR 9241/11: requisitos ergonômicos para trabalhos em escritórios com computadores: orientações sobre usabilidade. Rio de Janeiro. Albert, Bill; Tullis, Tom; Tedesco, Donna, 2010. Beyond the Usability Lab: Conducting Large-scale Online User Experience Studies. Burlington, MA: Morgan Kaufmann. Bangor, A., Kortum, P.T., Miller, J.T., 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. In Journal Usability Studies, Vol. 4, No 3, pp.114–123. Albert, Bill; Tullis, Tom., 2008. Measuring User Experience: Collecting, Analyzing, and Presenting Usability Metrics. San Francisco, CA: Morgan Kaufmann. Sauro, Jeff; Lewis, James R., 2012. Qualifying the User Experience: Practical Statistics for User Research. Waltham, MA: Morgan Kaufmann.

239