Using the Results from a Systematic Mapping ...

Using the Results from a Systematic Mapping Extension to Define a Usability Inspection Method for Web Applications Luis Rivero and Tayana Conte Instituto de Computação, Universidade Federal do Amazonas (UFAM) Manaus, AM - Brazil {luisrivero,tayana}@icomp.ufam.edu.br

Abstract— Usability is one of the most crucial factors in Web applications, allowing the successful usage of such systems. Many Usability Inspection Methods (UIMs) have been proposed to guarantee that Web applications provide a friendly, direct and easy to understand interface to their users. Nevertheless, some of these methods are not being used due to the lack of information about them. In this paper we describe the actual state of UIMs for Web applications through the extension of a systematic mapping study about Usability Evaluation Methods. Besides providing background knowledge to UIMs, our results showed that in order to meet the actual needs of the software development industry, emerging UIMs should: (a) find problems in early stages of the development process; (b) find specific problems and suggest solutions; and (c) provide automation. Using these features and usability criteria from other UIMs, we developed the Web Design Usability Evaluation (Web DUE) technique. The Web DUE aims to improve the quality of Web applications by allowing the usability evaluation of paper based prototypes by using pieces of Web pages called Web page zones. We have provided a proof of concept of the Web DUE by evaluating a paper based mock-up of a real Web application. Keywords-Usability Inspection Methods; Web Applications; Systematic Mapping Extension

I. INTRODUCTION In recent years, Web applications development demand has grown considerably [1]. These applications are currently the backbone of business and information exchange, and are being used to present products and services to potential customers [2]. According to Matera et al. [6], the acceptability of such applications is determined by their degree of usability. If a Web application possesses poor usability, it will be quickly replaced by a more usable one as soon as its existence becomes known to the target audience [7]. Usability Inspection Methods (UIMs) have emerged as a cost-effective way to improve the usability of such systems [14]. According to Matera et al. [6], the software development industry has been investing in the development of a variety of UIMs to address Web usability issues. Nevertheless, companies are not using these methods [4]. This lack of usage can be the cause of low quality regarding the usability aspect within Web applications [9]. In [12], we extended a systematic mapping study on usability evaluation methods for Web applications by selecting and thoroughly analyzing the papers that addressed new UIMs. This systematic mapping extension allowed us to

identify the state of art of UIMs for Web applications and to provide researchers and practitioners with a knowledge background for choosing a determined UIM. In this paper, we have used the findings presented in [12] to suggest a set of desirable features that a new UIM should possess in order to meet the actual needs of the software development industry. Based on these features, we have proposed the Web Design Usability Evaluation (Web DUE) technique. This technique aims to guide inspectors through the evaluation of paper based prototypes by dividing Web pages into Web page zones. According to Fons et al. [3], Web page zones are pieces of Web pages with specific types of contents. We crafted the Web DUE by selecting usability criteria from UIMs within the studies in [12] and relating them to the Web page zones. Furthermore, this paper also presents a proof of concept of the Web DUE technique by using it to evaluate the usability of a prototyped Web page of the Journal and Event Management System (JEMS1). This paper is organized as follows. Section II presents the background to Usability Inspection Methods. In Section III we summarize the planning and execution of the systematic mapping extension on UIMs for Web applications. Section IV shows the results from this extension, our findings and suggests a set of desirable features for emerging UIMs for Web applications. In Section V we show the Web DUE technique proposal, while Section VI provides a proof of concept by using it to evaluate the usability of a paper based prototype of a real Web application. Finally, Section VII presents our conclusions and future work. II. USABILITY INSPECTION METHODS The term usability is defined in the ISO 9241-11 [5] as “the extent to which a product can be used by specified users to achieve specific goals with effectiveness, efficiency and satisfaction in a specified context of use”. Regarding Web applications, usability is one of the most relevant quality aspects because of its own features, as cited in [10]: “Web applications are interactive, user-centered, hypermedia-based applications, where the user interface plays a central role”. Many usability evaluation methods (UEMs) have been proposed in the technical literature in order to improve the usability of different kinds of software systems [4]. According to Rocha and Baranauska [14], UEMs are procedures 1

https://submissoes.sbc.org.br/

composed by a set of well-defined activities that are used to evaluate the system’s usability. UEMs are divided into two categories: (a) user testing, in which empirical methods, observational methods and question techniques can be used to measure usability when users perform tasks on the system; and (b) inspections, which make use of experienced inspectors to review the usability aspects of the software artifacts [2]. In this research, we focus on usability inspections as they can lower the cost of finding usability problems since they do not need any special equipment or laboratory [14].

inspection methods for Web applications. From this initial set of papers we only selected the studies that thoroughly described UIMs at a mature stage. Consequently, we discarded papers that met at least one of the following exclusion criteria:

The main generic UIMs that can be used to increase the system’s usability and therefore its quality are: the Heuristic Evaluation [8], the Cognitive Walkthrough [11], and the Perspective-based Usability Inspection [15].

Categorization of Studies: We created a set of research sub-questions to better address the state of art of UIMs for Web applications. We used the answers to these sub-questions to categorize the analyzed papers. Table I shows our research sub-questions, their motivation, and the possible answers that can be obtained when analyzing a selected research paper.

The Heuristic Evaluation, proposed by Nielsen [8], assists the inspector in usability evaluations using guidelines. The evaluation process consists of a group of evaluators who examine the interface using heuristics, which are rules that seek to describe common properties of usable interfaces. The Cognitive Walkthrough, proposed by Polson et al. [11], is a method in which a set of reviewers analyze if a user can make sense of interaction steps as they proceed in a predefined task. In order to identify usability problems the inspectors ask questions to answer if: (a) the action is sufficiently evident; (b) the action will be connected with what the user is trying to do; and (c) if the user will understand the system’s response. According to Zhang et al. [15], it is difficult for an inspector to detect all kind of problems at the same time. Consequently, they proposed a usability inspection technique based on perspectives (Usability Based Reading - UBR). In the UBR, the inspector focuses in a sub-set of questions according to the usability perspective to find problems. In the following Section we show how we carried out the extension of the systematic mapping on UIMs for the Web. III.

EXTENSION OF THE SYSTEMATIC MAPPING ABOUT UIMS FOR WEB APPLICATIONS Fernandez et al. [2] presented a systematic mapping on UEMs for Web applications. However, in order to thoroughly describe how UIMs for the Web had been applied, it was necessary to independently analyze them. In this Section, we briefly explain the planning and execution of the extension of the systematic mapping in [2]. In this extension, we extracted specific information from [2] regarding the new UIMs for Web applications. We used the obtained results to answer the following research question: “What new Usability Inspection Methods have been employed to evaluate Web artifacts and how have these methods been used?” Readers must take note that a thoroughly described version of the execution process of this extension can be found in [12]. Selection Process: Fernandez et al. [2] analyzed 206 papers about usability evaluation methods for Web applications and classified them into categories. We have used this classification as a starting point in the selection of papers. We selected papers that, according to [2], described new

  

Papers presenting usability problems and no methodology to identify them. Papers describing only ideas for new research fields. Papers presenting techniques with no description of their execution process.

TABLE I.

RESEARCH SUB -QUESTIONS, POSSIBLE ANDSWERS AND MOTIVATIONS FROM THIS SISTEMATYC MAPPING EXTENSION.

Research Sub-Questions and Answers

Motivation

Q1. Theoretical Basis Method: (a) Heuristic Evaluation (b) Cognitive Walkthrough (c) Perspective Based (d) Other Basis Q2. Type of Evaluated Artifact: (a) HTML code (b) Model (c) Application/Prototype Q3.- Type of Application Evaluated by the Inspection Method: (a) Generic (b) Specific Q4.- Use of Inspectors in the Inspection Process: (a) Yes (b) No

To discover whether the Usability Inspection Methods for the Web have been developed considering well known Generic Usability Inspection Methods or whether they have been using new basis. To discover which is the most commonly evaluated artifact in Usability Inspection Methods for the Web. To discover whether the Usability Inspection Methods for the Web have been crafted to find generic usability problems or usability problems of a specific type of Web application. To discover whether the Usability Inspection Methods for the Web have been automated to a point where inspectors are no longer necessary.

Execution: Fig. 1 shows how we executed this systematic mapping extension. From the initial set of 206 papers in Fernandez et al. [2], we selected 37 papers that, according to the classification in [2], presented new usability inspection methods for Web applications. However, as 5 papers were unavailable for download, we reduced the initial set to 32.

Figure 1. Execution process of this Systematic Mapping Extension.

After reading each study, we discarded 6 studies for meeting the exclusion criteria we defined in the selection process stage. The Selected Primary Studies List in this paper shows the 26 selected papers that we analyzed in this literature review. In the next Sections we explain how we used the data

obtained from the categorization of studies, to address the current stage of UIMs for the Web and suggest a new UIM. IV.

RESULTS DISCUSSION AND IMPLICATIONS FROM THE SYSTEMATIC MAPPING EXTENSION This Section summarizes the principal findings from the analysis of the results of this literature review. Furthermore, we used these findings to propose a set of desirable features for emerging UIMs seeking to meet the actual needs of the software development industry. A. Results and Principal Findings Table II shows the classification of primary studies according to the research sub-questions we defined above. Readers must take note that the summation of the percentages of sub-questions Q1 and Q2 is over 100% as a paper can be classified in one or more answers. In this sub-section we will discuss our findings regarding each of the sub-questions. Q1 - Theoretical Basis Method: Around 60% of the reviewed papers based the new UIM for Web applications on already known usability inspection methods. UIMs based on Nielsen’s [8] Heuristic Evaluation (27%), mainly focused on better describing or self explaining how or in which situation each heuristic could be applied. Regarding the use of the Cognitive Walkthrough (19%), two new inspection methods were developed: Blackmon’s CWW described in papers S06, S07 and S08; and Filgueiras’ RW described in S14. Studies S09, S12, S23 and S25 (15%) made use of perspectives to help focus on each usability attribute when carrying out the inspection. Moreover, the remaining techniques (58%) are being based on heuristics specifically proposed for the Web. UIMs for the Web are based on: (a) generic usability inspection methods; and (b) new specific evaluation criteria for the Web. However, none of them can address all circumstances and types of Web artifacts. A combination of these methods can be used to enhance the evaluation results. Q2 - Type of Evaluated Artifact: Around 77% of the reviewed papers reported UIMs analyzing prototypes/systems. Inspectors carry out the evaluation process by analyzing the interaction provided by the prototype or product while executing a task on it. Moreover, 15% (S02, S21, S23 and S26) of the analyzed papers describe automated techniques in which HTML code was verified. Regarding Model analysis, 15% (S04, S18, S21 and S23) of the studies evaluated if the model met interaction rules within the Web domain. The main artifacts used during the inspection process are: models, HTML code and prototype/application. Our results show that prototype/application is the most common evaluated artifact. However, the evaluation is being held with functional prototypes, which means that the cost of correcting usability problems is high. Therefore, there is a shortage of UIMs for the Web able to identify usability problems during the initial stages of the development process. Q3 - Type of Web Application Evaluated by the Inspection Method: The results revealed that around 88% of the UIMs could be applied to any Web application. The remaining studies (S01, S05 and S24), around 12% of the

selected UIMs, focused on a specific type of Web application. Allen et al. (S01) describe a paper based technique for medical Web applications. Basu (S05) proposes a new framework to evaluate e-commerce applications. In paper S24, Thompson and Kemp evaluated Web 2.0 applications. Generic UIMs focus on finding usability problems that can be applied to every Web application. However, most of them do not provide feedback on how to treat a violation of usability. On the other hand, UIMs that evaluate specific types of Web applications provide evaluators with more data regarding that type of application. Our results show that there is a higher number of UIMs for generic Web applications compared to the number of UIMs for specific Web applications; and that there is a need for UIMs that suggest solutions for the identified problems. TABLE II.

RESULTS OF THE CLASSIFICATION ANALYSIS . Q1

Q2

Q3

Q4

a b c d a b c a b a b Paper X X X X S01 X X X X S02 S03 X X X X X X X X S04 X X X X S05 S06 X X X X X X X X S07 X X X X S08 S09 X X X X X X X X X S10 X X X X S11 X X X X X S12 X X X X S13 X X X X S14 X X X X S15 X X X X S16 X X X X S17 X X X X S18 X X X X S19 X X X X X S20 X X X X X S21 X X X X S22 X X X X X X S23 X X X X S24 X X X X X S25 X X X X S26 Total 7 5 4 15 4 4 20 23 3 24 2 Studies % 26.9 19.2 15.4 57.7 15.4 15.4 76.9 88.5 11.5 92.3 7.7 Categorization of Primary Studies Q1.- Theoretical Basis Method: (a) Heuristic Evaluation (b) Cognitive Walkthrough (c) Perspective Based (d) Other Basis Q2.- Type of Evaluated Artifact: (a) HTML code (b) Model (c) Application/Prototype Q3.- Type of Application Evaluated by the Inspection Method: (a) Generic (b) Specific Q4.- Use of Inspectors in the Inspection Process: (a) Yes (b) No

Q4 - Use of Inspectors in the Inspection Process: Our results show that the automation of the inspection process is not yet possible in techniques involving judging and human interaction. Consequently, techniques using model and prototype analysis are not being automated (92%), but enhanced by using tools to provide inspectors with means of reducing evaluation effort. However, 8% of the reviewed studies described UIMs that did not use any inspectors at all.

There is a relationship between the UIM’s evaluated artifact and the degree of automated process. UIMs evaluating HTML code are being fully automated. Nevertheless, their evaluated usability aspects are less than the usability aspects of UIMs that make use of inspectors. B. Meeting the needs of the Software Development Industry We used the sub-questions from this systematic mapping extension to suggest three features an emerging UIM should possess in order to meet the actual needs of the software development industry. We did not consider sub-question Q1 (Theoretical Basis Method) because its goal was to explore the bases of new UIMs for Web applications, rather than identifying research opportunities. We present three features and their relationship to our research sub-questions as follows: Feature 1 - Ability to find problems in early stages of the development process: The results for sub-question Q2 indicated that there is a need for UIMs evaluating the usability of artifacts related to the early stages of the development process. Consequently, in order to reduce the cost of correcting usability problems new UIMs should be able to evaluate models or prototypes. Feature 2 - Ability to find specific problems and suggest solutions: Research sub-question Q3 indicated that most Generic UIMs do not provide feedback on how to treat a usability problem once it is found. Consequently, emerging UIMs should be able to aid in both the identification and solution of usability problems. Feature 3 - Automation: Our results for sub-question Q4 indicated that there is a shortage of automated UIMs. Therefore, in order to enhance the performance of the evaluation, new UIMs should be automated or provide assistance by means of a tool. Automated UIMs reduce the cost of carrying out inspections. Nevertheless, not every UIM can be automated. In this case, the UIM should provide means to reduce the inspector’s effort. V. USING FINDINGS TO DEFINE A NEW UIM The Web Design Usability Evaluation (Web DUE) technique is an inspection method that proposes to meet the needs of the software development industry regarding UIMs for Web applications. Therefore, the Web DUE technique was crafted by adopting the suggested features obtained from this systematic mapping extension. In this Section, we relate each of the characteristics of the Web DUE technique to the suggested features.

Evaluation – WDP [1] technique. The purpose of this usability verification items is to address usable characteristics within each of the Web page zones. Some of these verification items are shown in Table IV. Readers must note that the usability verification items also include examples in order to aid inspectors in the identification and solution of the identified problems. The complete list of the usability verification items per Web page zone can be found in [13]. Once the inspector verifies which Web page zones are contained within the paper based prototypes, he/she uses the usability verification items to find specific problems affecting the Web page zones that are related to the evaluated Web application (Feature 2). Furthermore, the inclusion of the violated usability verification items within the paper based prototypes can aid in the correction of the encountered usability problems. TABLE III.

WEB PAGE ZONES USED BY THE WEB DUE TECHNIQUE BASED ON [3].

Zone Navigation System’s State

Contents Navigation Links Information about the application's state and how we got there. Information Information from the application's data base. Services Access to functionalities related to the information zone. User Information Information about the logged user. Institution Information about the institution that is responsible for the Web application. Direct Access Common Web functionalities (Login, Logout, Home). Data Entry Provides a form to input data to execute certain operations. Custom Domain-independent content Help Information about how to use the application and how it works.

A simplified version of the evaluation process of the Web DUE technique can be seen in Fig. 2. Initially, inspectors must divide the provided mock-ups into their respective Web zones (Stage 1). For each Web page zone the inspector verifies if the application meets usability rules by checking the technique’s usability verification items (Stage 2). Finally, the non conformity of any verification item implies in a usability problem (Stage 3). In order to correct usability problems, the Web application must consider including the violated items.

The Web DUE technique evaluates the usability of paper based low-fidelity prototypes (or mock-ups). This means that the evaluation can be carried out in the early stages of the development process (Feature 1). The Web DUE technique guides the evaluation through Web page zones. Web page zones are pieces of Web pages with specific types of contents [3]. Table III shows the Web page zones used by the Web DUE technique and their contents. For each of the Web page zones we crafted a set of usability verification items based on the Heuristic Evaluation [8] and the Web Design Perspectives-Based Usability

Figure 2. Simplified inspection process of the Web DUE technique.

Another important feature is that the Web DUE technique makes use of tools to reduce inspectors’ effort. As it involves inspectors’ judgment in order to identify usability problems, the Web DUE is not fully automated. However, it aids inspectors during the evaluation process by using a tool

that automatically presents the usability verification items checklists to aid inspectors during the inspection of paper based prototypes (Feature 3). VI. APPLYING THE WEB DUE TECHNIQUE In this Section we provide a proof of concept by evaluating the usability of a paper based prototype using the Web DUE technique. Readers must take note that we only show part of the example due to lack of space. Part 1 of Fig. 3 shows the paper based prototype of a Web page of the Journal and Event Management System (JEMS). This Web page is used in the JEMS system to edit user data. In Fig. 3 part 2 we have identified all Web page zones: System’s State Zone, Data Entry Zone and Navigation Zone. In Fig. 3 part 3 we have also zoomed in some of the components within the Web page zones of the mock-up. These components have been augmented because they represent usability problems within the paper based prototype of the Web page.

before filling the user’s country does not follow a logical order (see Fig. 3 part 3 element B). Finally, in the Navigation zone we encountered nonconformity 03. This verification item indicates that the symbols used within the navigation zone are difficult to understand. A user would find it difficult that the “globe” symbol would leave to the JEMS portal (see Fig. 3 part 3 element C). TABLE IV. USABILITY VERIFICATION I TEMS THAT THE EVALUATED MOCK-UP DOES NOT MEET WITHIN THE WEB DUE TECHNIQUE.

N° 01

Web Page Zone System’s State

02

Data Entry

03

Navigation

Usability Verification Items The System’s State is naturally and logically presented to the user. For example, when visualizing the system’s state the user must be able to understand what path led him to that state. The data to be input by the user are requested in a natural and logical order. In other words, when inputting data the sequence of input data must be logical. It is easy to understand the words and symbols used in the system. In other words, the user must be able to link the information being showed (labels or images) with what he/she is trying to do.

We have shown the simplified inspection process of the Web DUE technique by evaluating a low fidelity prototype of a real Web page. Readers must note that in order to evaluate the entire Web application, all Web pages within the Web application must be evaluated. Furthermore, in order to correct these problems, the prototype must integrate the violated verification items from Table IV.

Figure 3. Example of the Web DUE’s Evaluation Process using a Mock-up.

In Table IV we present some of the usability verification items of the Web DUE technique and their corresponding Web page zones. These verification items are nonconformities regarding the evaluated mock-up in Fig. 3. In other words, the paper based prototype of the Web page has not met these usability verification items. If we look at Fig. 3 part 3 and Table IV simultaneously, we can relate the nonconformity of the usability verification items in Table IV with the augmented elements A, B and C in Fig. 3. part 3. Readers must take note that we have shown one augmented element per Web page zone. We will address each of the encountered usability problems as follows. In the System’s State zone, the usability verification item 01 indicated that, despite showing the actual state of the system, the prototype does not show it logically (see Fig. 3 part 3 element A). In other words, the prototype does not show how the user reached that state. Furthermore, in the Data Entry zone, we identified nonconformity 02. This usability verification item indicates that the mock-up does not request data in a logical way. Asking for a country’s state (even if this input data must be filled only for members from the US)

VII. CONCLUSIONS AND FUTURE WORK This paper has discussed the results from a systematic mapping extension addressing UIMs for the Web. The analysis of 26 studies from [2] showed that in order to meet the actual needs of the software development industry, emerging UIMs should posses the following features: (a) find problems in early stages of the development process; (b) find specific problems and suggest solutions; and (c) provide automation or assistance to reduce the inspector’s effort. We used these features to propose the Web DUE technique that guides inspectors through the inspection process using Web page zones. The Web DUE also seeks to evaluate usability attributes in early stages of the development process by evaluating paper based prototypes. We have presented a proof of concept by evaluating a mock-up of a real Web page. We hope that our findings will be useful to provide an outline to which usability evaluation methods can be applied. We also hope that the set of desirable features for emerging UIMs become adopted in new UIM proposals for Web applications. Moreover, as future work, we pretend to carry out empirical studies in order to evaluate the feasibility of the Web DUE technique, and verify if the tool support influences in a positive way in the results of the evaluation. Furthermore, we pretend to use the results of the studies to refine the technique and understand how it will be used by inspectors in the context of a real development environment.

ACKNOWLEDGMENTS We would like to acknowledge the support granted by CAPES process AEX 4982/12-6. REFERENCES [1]

[2]

[3]

[4]

[5]

[6]

[7]

[8] [9]

[10] [11]

[12]

[13]

[14] [15]

T. Conte, J. Massollar, E. Mendes and G. Travassos, “Web usability inspection technique based on design perspectives,” IET Software, Volume 3, Issue 2, 2009. A. Fernandez, E. Insfran and S. Abrahao, “Usability evaluation methods for the Web: A systematic mapping study,” Information and Software Technology, Volume 53, Issue 8, 2011. J. Fons, V. Pelechano, O. Pastor, P. Valderas, and V. Torres, “Applying the OOWS model-driven approach for developing Web applications: The internet movie database case study,” In: G. Rossi, D. Schwabe, L. Olsina and O. Pastor, Web Engineering: Modeling and Implementing Web Applications, Springer, 2008. E. Insfran and A. Fernandez, “A Systematic Review of Usability Evaluation in Web Development,” Proc. Second International Workshop on Web Usability and Accessibility, New Zealand, 2008, pp. 81-91. International Organization for Standardization, ISO/IEC 9241-11: Ergonomic Requirements for Office work with Visual Display Terminals (VDTs) – Part 11: Guidance on Usability, 1998. M. Matera, F. Rizzo and G. Carughi, “Web Usability: Principles and Evaluation Methods,” In: E. Mendes and N. Mosley, N, Web Engineering, Springer, 2006. E. Mendes, N. Mosley and S. Counsell, “The Need for Web Engineering: An Introduction. Web Engineering,” In: E. Mendes and N. Mosley, Web Engineering, Springer, 2006. J. Nielsen, “Finding usability problems through heuristic evaluation,” Proc. CHI’92, UK, 1992, pp. 373-380. J. Offutt, “Quality attributes of Web software applications,” IEEE Software: Special Issue on Software Engineering of Internet Software, Volume 19, Issue 2, 2002. L. Olsina, G. Covella and G. Rossi, “Web Quality,” In: E. Mendes and N. Mosley, Web Engineering, Springer, 2006. P. Polson, C. Lewis, J. Rieman and C. Wharton, “Cognitive walkthroughs: a method for theory-based evaluation of user interfaces,” International Journal of Man-Machine Studies, Volume 36, Issue 5, 1992. L. Rivero and T. Conte, “Characterizing Usability Inspection Methods through the Analysis of a Systematic Mapping Study Extension,” Proc. IX Experimental Software Engineering Latin Workshop, Argentina, 2012. L. Rivero and T. Conte, “Web DUE technique: Usability Verification Items per Web Page Zone,” Technical Report RT-USES-2012-001, 2012. Available at: http://www.dcc.ufam.edu.br/uses/index.php/ publicacoes/cat_view/69-relatorios-tecnicos. H. Rocha and M. Baranauska, “Design and Evaluation of Human Computer Interfaces,” (Book), Nied, 2003. (in Portuguese) Z. Zhang, V. Basili and B. Shneiderman, “Perspective-based Usability Inspection: An Empirical Validation of Efficacy,” Empirical Software Engineering, Volume 4, Issue 1, 1999.

SELECTED PRIMARY STUDIES LIST [S01] M. Allen, L. Currie, S. Patel and J. Cimino, “Heuristic evaluation of paper-based Web pages: A simplified inspection usability methodology,” Journal of Biomedical Informatics, Volume 39, Issue 4, 2006. [S02] D. Alonso-Rios, I. Vazquez, E. Rey, V. Bonillo and B. Del Rio, “An HTML analyzer for the study of Web usability,” Proc. IEEE International Conference on Systems, Man and Cybernetics, USA, 2009, pp. 1224-1229. [S03] C. Ardito, R. Lanzilotti, P. Buono, and A. Piccinno, “A tool to support usability inspection,” Proc. Working Conference on Advanced Visual Interfaces, Italy, 2006, pp. 278-281. [S04] R. Atterer and A. Schmidt, “Adding Usability to Web Engineering Models and Tools,” Proc. 5th International Conference on Web Engineering, Australia, 2005, pp. 36-41.

[S05] A. Basu, “Context-driven assessment of commercial Web sites,” Proc. 36th Annual Hawaii International Conference on System Sciences, USA, 2003, pp. 8-15. [S06] M. Blackmon, P. Polson, M. Kitajima and C. Lewis, “Cognitive walkthrough for the Web,” Proc. SIGCHI Conference on Human Factors in Computing Systems, USA, 2002, pp. 463-470. [S07] M. Blackmon, M. Kitajima and P. Polson, “Repairing usability problems identified by the cognitive walkthrough for the Web,” Proc. SIGCHI Conference on Human Factors in Computing Systems, USA, 2003, pp. 497504. [S08] M. Blackmon, M. Kitajima and P. Polson, “Tool for accurately predicting Website navigation problems, non-problems, problem severity, and effectiveness of repairs,” Proc. SIGCHI Conference on Human Factors in Computing Systems, USA, 2005, pp. 31-40. [S09] D. Bolchini and F. Garzotto, “Quality of Web Usability Evaluation Methods: An Empirical Study on MiLE+,” Proc. International Workshop on Web Usability and Accessibility, France, 2007, pp. 481-492. [S10] C. Burton and L. Johnston, “Will World Wide Web user interfaces be usable?,” Proc. Computer Human Interaction Conference, Australia, 1998, pp. 39-44. [S11] J. Chattratichart and J. Brodie, “Applying user testing data to UEM performance metrics,” Proc. of the Conference on Human Factors in Computing Systems, Austria, 2004, pp. 1119-1122. [S12] T. Conte, J. Massollar, E. Mendes and G. Travassos, “Web usability inspection technique based on design perspectives,” IET Software, Volume 3, Issue 2, 2009. [S13] M. Costabile and M. Matera, “Guidelines for hypermedia usability inspection,” IEEE Multimedia, Volume 8, Issue 1, 2001. [S14] L. Filgueiras, S. Martins, C. Tambascia, and R. Duarte, “Recoverability Walkthrough: An Alternative to Evaluate Digital Inclusion Interfaces,” Proc. Latin American Web Congress, Mexico, 2009, pp. 71-76. [S15] P. Fraternali, and M. Tisi, “Identifying Cultural Markers for Web Application Design Targeted to a Multi-Cultural Audience,” Proc. 8th International Conference on Web Engineering, USA, 2008, pp. 231-239. [S16] Y. Habuchi, M. Kitajima and H. Takeuchi, “Comparison of eye movements in searching for easyto-find and hard-to-find information in a hierarchically organized information structure,” Proc. Symposium on Eye Tracking Research & Applications, USA, 2008, pp. 131-134. [S17] S. Kirmani, “Heuristic Evaluation Quality Score (HEQS): Defining Heuristic Expertise,” Journal of Usability Studies, Volume 4, Issue 1, 2008. [S18] F. Molina and A. Toval, “Integrating usability requirements that can be evaluated in design time into Model Driven Engineering of Web Information Systems,” Advances in Engineering Software, Volume 40, Issue 12, 2009. [S19] M. Moraga, C. Calero and M. Piattini, “Ontology driven definition of a usability model for second generation portals,” Proc. 1st International Workshop on Methods, Architectures & Technologies for e-Service Engineering, USA, 2006. [S20] A. Oztekin, A. Nikov and S. Zaim, “UWIS: An assessment methodology for usability of Web-based information systems,” Journal of Systems and Software, Volume 8, Issue 12, 2009. [S21] L. Paganelli and F. Paterno, “Automatic reconstruction of the underlying interaction design of Web applications,” Proc. 14th International Conference on Software Engineering and Knowledge Engineering, Italy, 2002, pp.439-445. [S22] P. Paolini, “Hypermedia, the Web and Usability issues,” Proc. IEEE International Conference on Multimedia Computing and Systems, Italy, 1999, pp. 111-115. [S23] O. Signore, “A comprehensive model for Web sites quality,” Proc. 7th IEEE International Symposium on Web Site Evolution, Hungary, 2005, pp. 30-36. [S24] A. Thompson and E. Kemp, “Web 2.0: extending the framework for heuristic evaluation,” Proc. 10th International Conference NZ Chapter of the ACM’s Special Interest Group on Human-Computer Interaction, New Zeland, 2009, pp. 29-36. [S25] L. Triacca, A. Inversini and D. Bolchini, “Evaluating Web usability with MiLE+,” Proc. 7th IEEE International Symposium on Web Site Evolution, Hungary, 2005, pp. 22-29. [S26] J. Vanderdonckt, A. Beirekdar, and M. Noirhomme-Fraiture, “Automated Evaluation of Web Usability and Accessibility by Guideline Review,” Proc. 4th International Conference on Web Engineering, Munich, 2004, pp. 28-30.