An Evaluation of Web Accessibility Metrics based on their Attributes André P. Freire, Renata P. M. Fortes
Marcelo A. S. Turine, Debora M. B. Paiva
Mathematics and Computer Science Institute University of Sao Paulo P.O. Box 668 Sao Carlos,SP, Brazil
Depart. of Computer Science and Statistics Federal University of Mato Grosso do Sul P.O. Box 549 Campo Grande, MS, Brazil
[email protected],
[email protected]
{mast, debora} @dct.ufms.br
measurements of quality attributes of products and process can provide important foundations for process improvement activities.
ABSTRACT Accessibility is a concept related to providing access to Web content to people with different abilities and people using different devices. A number of metrics have been proposed in the last years to help to obtain quantitative Web accessibility levels. Each Web development project has different aspects, and the choice of a given Web accessibility metric should be carefully considered according to the needs of the project. In this paper, we present a review on the main existing Web accessibility metrics, emphasizing the main features and comparing their correlations from experimental data obtained from large scale evaluations. Project managers and evaluators should consider specific features for each given project to effectively choose a proper metric. Needs to adjust customized coefficients or to adhere to automatic evaluation tools are important issues to be observed.
The interest in defining and applying metrics for measuring Web accessibility has spread out in last years. Guidelines provide important information to help design, implementation, and evaluation of Web content, regarding many elements, for example, presentation of graphics, video, and audio. In this paper, we present a review and a comparison of existing Web accessibility metrics published in last years. The main contribution is to indicate the state of art in this context and to provide indications of the main features of each metric according to important attributes and to experimental data. With these data we aim to help researchers and practitioners interested in Web accessibility to define or establish their measurement process. The paper is organized as follows: in Section 2 we present an overview about Web accessibility, focusing on evaluation aspects. In Section 3 we discuss background issues, such as the importance of measurement and Web Engineering metrics. In Section 4 we describe Web accessibility metrics presented in the literature. In Section 5 we compare such metrics and in Section 6 we present the final remarks and conclusions.
Categories and Subject Descriptors H.5.4 [Information Interfaces and Presentation]: Hypertext/Hypermedia; D.2.0 [Software]: Software Engineering; K.4.2 [Computers and Society]: Social Issues
General Terms Human Factors
2. WEB ACCESSIBILITY AND ACCESSIBILITY
Keywords Information Accessibility, Accessibility accessibility, Web Engineering
Metrics,
The growth in the number and variety of Web applications has placed the Web as one of the most important technologies to spread information. The use of standard technologies, accessible from a number of devices and by people with different abilities, makes the Web a promise for providing universal access to information.
Web
1. INTRODUCTION Software measurement is recognized as a means to understand, monitor, predict and control software development [2]. The objective of measurement process is to define and apply a set of metrics, according to guidelines and procedures to collect and analyze them [11]. When properly conducted, continuous
However, if information contained in Web applications is not properly designed, it may lead to many access barriers. These barriers may prevent users with special needs from accessing Web content and performing communication using it. Tim Berners Lee4 has stated that “The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect".
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGDOC’08, September 22-24, 2008, Lisbon, Portugal. Copyright 2008 ACM 978-1-60558-083-8/08/0009…$5.00.
Many techniques have been proposed to help the development of accessible Web applications. Freire et. al [9] has surveyed techniques for developing accessible content in Web applications. 4
73
W3C Director and inventor of the World Wide Web
These techniques were classified according to the process of ISO IEC 12207 standard [13].
possible); (3) cost effective; and (4) informative (ensure that changes to metric values have meaningful interpretations).
In the context of Web development, accessibility should be considered an important quality attribute, which should be evaluated and verified [23]. The main goal of evaluation methods is to identify and report problems to be solved in order to improve user experience. Similarly to usability evaluation, there are several accessibility evaluation methods available.
Particularly, in terms of Web Engineering, few studies about definition and validation of metrics can be found. In general, researchers indicate some extension of traditional Software Engineering metrics. Mendes et al [18] proposed a set of metrics for Web applications development. Metrics were organized into five categories: (1) length size; (2) reusability; (3) complexity size; (4) effort; and (5) confounding factors. Length size metrics include, for example, number of HTML files, media files, JavaScript files, and Java applets; duration of audio, video, and animation; and size of media files (Kbytes). Reusability metrics include, for example, number of reused or modified media _les and total space allocated for all reused media _les. Complexity size metrics included, for example, number of links per page and number of graphics, audio, video, and animations used in page. Effort metrics included, for example, estimated elapsed time taken to test all media in application and estimated elapsed time taken to digitize media. Confounding factors include authoring/design experience of subject and type of tool used to author/design Web pages.
Usability evaluation methods may be classified as usability inspections or usability tests [6]. Inspection methods usually involve tests that are carried out by the expert with no participation of users. Heuristic evaluations, Guideline review and Cognitive walkthrough are examples of inspection methods. Evaluators are supposed to inspect a user interface in order to find usability problems and to report them. User tests involve the definition of experiments in which real users experience the interface. Inspection methods are very important to identify usability and accessibility issues. However, it is crucial to perform tests with real users, and to consider users with different abilities to accurately find problems [21].
Ivory et al [15] have investigated how to provide empirical evidence about the importance of Web metrics. They have identified a number of Web metrics, for example: (1) total words on a page; (2) total graphics on a page; (3) text areas highlighted with color, bordered regions, rules or lists; and (4) total links on a page. In addition, they have collected data for 1,898 Web sites regarding the set of metrics. Ivory et. al have stated that, as a result, it was possible to determine the significance of the metrics in predicting good versus not-good pages. It is important to notice, however, that additional study is required due to the simplicity of metrics.
One of the most popular accessibility methods is the guidelines review. The World Wide Web Consortium (W3C) has defined a set of guidelines named Web Content Accessibility Guidelines (WCAG) [26] in 1999. Since then, this document has become a reference to accessibility guidelines, and is the main evaluation guidance. These guidelines set some recommendations, such as the use of alternative representations of audio and visual content, to enable alternative input methods, and others. Each guideline is divided into checkpoints, and each checkpoint is assigned a different priority, according to its severity.
Furthermore, studies related to metrics focusing on Web usability and Web accessibility have been published. In general, the concept of barriers to use is the main factor observed and metrics indicate how barriers free Web sites are or not. This is the main topic of this paper and a detailed discussion is presented in Section 4 and Section 5.
The proper understanding of accessibility evaluation methods is very important to set up an environment to collect quantitative data for accessibility metrics.
3.METRICS IN WEB ENGINEERING Fenton and Pfleeger [7] states that the measurement process is related to set values (numbers, symbols) to entities aiming at describing them according to well-defined rules. As it is expressed by Pfleeger [22], if we are able to quantify actions and consequences using a mathematical language, we can evaluate our progress. In other words, measurement is important because specific processes and products features can be clearly observed. The importance of measurement and data, according to Kan [16], is closely related to the progress of science and engineering: without verification by data and measurement, theories and propositions may not be applied.
4.WEB ACCESSIBILITY METRICS Metrics are very important to help understanding, controlling, and improving products and processes in software development [7]. In terms of accessibility, metrics can be especially useful in two situations: companies may improve the accessibility of final products and companies beginning the software development can introduce accessibility issues in the development process. Many researchers have investigated accessibility metrics for Web systems. The main goal of such metrics is to establish values summarizing results of accessibility evaluations based on guidelines. In next sub-sections we present a review about the main accessibility metrics found in the literature.
In spite of the importance of measurement, defining metrics my be a difficult task, depending on the attributes to be measured [17]. For example, ISO/IEC 9126 standard [14] has evolved significantly last years to obtain metrics for software product attributes. Important results have been reached. However, in several cases, the subjective opinion of users of metrics (potential evaluators of software product) is considered among the components to define the metric results. In this sense, in order to help the definition of good metrics, Daskalantonakis [5] identified important characteristics of useful software metrics. According to Daskalantonakis [5], software metrics must be: (1) simple to understand and precisely defined; (2) objective (as much as
Each important concept in the context of metrics was represented according to the named variables described in Table 1.
74
In this metric, the final value represents a probability of finding a barrier in a Web site that could prevent a user from completing a task. This metric also applies the concepts of potential problems and weights for barriers. However, the use of user-derived metrics is proposed. Weights for barriers should be defined after carrying out experiments with users with different disabilities. In the complete form of the metric, it also considers the concept of “evaluator confidence”, which is related to the reliability of the evaluation.
Table 1. Symbol standards used in the formulas Symbol
Meaning
B
Number of barriers or problems in a Web pages
P
Number of potential barriers in a Web page
W
Coefficient for a barrier weight
NP
Number of pages evaluated
NT
Number of tests carried out
Although this metric has very interesting features, there is no report on experimental derived weights for barriers. In the available documentation, the value Wi = 0:05 is used for all barriers5. Following, we show the simplified formula for a single page, which does not consider the test confidence:
4.1.Failure rate The first published study about accessibility measurement was purposed by Sullivan and Matson [24]. This metric considers the ratio between the total number of points of failure encountered in a page (Bp) and the total number of potential barriers (Pp). For example, an image without alternative text is considered an accessibility barrier. Thus, any image in a page may be considered a potential point of failure, as it may be an accessibility barrier if an alternative text is not properly defined. The formula for this metric is as follows:
Ip =
UWEM =
4.5.Improved Aggregation formula - A3 Buhler et. Al [2] proposed a couple of improvements on the metric UWEM [4]. They have used probability properties and aggregated some issues related to the complexity of the page, considering the number of violations of a given checkpoint in relation to the total number of violations (Bn). The formula of the A3 is denoted as follows:
Bp Pp
4.2.Metric for blind users
n
A3 ( p ) = 1 − ∏ (1 − Wi )
The concept of “weight" for each barrier is another important concept for accessibility metrics. Gonzales et. Al [12] proposed an accessibility metric based on the model “WebQEM” (Web Quality Evaluation Method) [19]. In this metric, a global ratio (GP) is calculated for a given page based on the percentage of accessibility problems (Bi) related to their respective potential problems (bi) for each barrier. This value is later multiplied by a weight, defined according to the impact of each barrier (Wi).
4.6.Web Accessibility Quality Metric -WAQM The metric WAQM (Web Accessibility Quality Metric) [25] considers the guidelines from WCAG 1.0 [26] classified according to the principles: Perceivable, Operable, Understandable and Robust from WCAG 2.0 [27]. Different from other metrics, this metric also considers problems identified as warnings by evaluation tools.
Bij
∑∑ ( P )(W ) i
WAB =
j =1 i =1
Bi Bi + ) Pi B p
Buhler et. al [3] performed a preliminary experimental study, using the value Wi = 0.05 for all barriers. This study involved the evaluation of six Web pages by a group of six users with different types of disabilities. In the experiment, quantitative values for the use of the pages were defined. Afterwards, the authors of the paper compared these values with values obtained from A3, UWEM and the metric of potential problems. The metric A3 had the best performance in this experiment.
Parmanto & Zeng [20] have proposed the metric WAB (Web Accessibility Barrier). This metric also adopts the concepts of potential problems and weights for barriers. Besides, it also takes into account the total number of pages that a given site contains (NP). The weights are defined as the inverse of the priority of each WCAG [26] checkpoint, which are defined as 1, 2 or 3. The formula for calculating WAB is as follows: n
(
i =1
4.3.Web Accessibility Barriers (WAB)
NP
B 1 n 1 − ∏ (1 − pj Wb ) ∑ n j =1 Ppj b
Two different approaches are considered for the calculus, using a hyperbole approximation. A decision based on experimental values must be taken in order to choose which formula should be used. The summarized formula is illustrated as follows. The variable Nx denotes the number of checkpoints for a given principle, or error, for example, Np is the number of checkpoints related to the principle perceivable. The weights Wz were experimentally defined as W1 = 0.80, W2 = 0.16 and W3 = 0.04. The values a and b were also defined experimentally. The authors of WAQM have set them as a = 0.3 and b = 20. This metric ranges from 0 to 100, where 100 correspond to accessible sites,
ij
NP
4.4.Aggregation formula - UWEM Another important metric was defined in the UWEM (Unified Web Evaluation Methodology) Project [4]. The UWEM was developed in the context of WAB Cluster, a joint effort that involves 23 European organizations in three European projects combined into a cluster. They have developed UWEM to ensure that large scale monitoring and local evaluation are compatible and coherent among themselves and with the Web Content Accessibility Guidelines from W3C/WAI.
5
75
This value was used do that the sum of weights for all barriers is equal to 1, considering twenty barriers.
•
different from the other metrics that assign higher values to more inaccessible pages.
WAQM =
Axyz
1 NP NTx ∑ ∑ NP j =1 x∈{ p , o, u , r } NT
NTxy y∈{e , w} NTx
∑
•
∑W A z
xyz
x∈{1, 2 , 3}
•
Bxyz Bxyz − 100 a − 100 ( P )( b ) + 100, if P < a − 100 / b xyz xyz = − a ( Bxyz ) + a, otherwise Pxyz
In Table 2, a summary of the main features of each metric analyzed in this study is provided. It is possible to observe that half of the metrics use WCAG 1.0 as the base for metric computation. The metric for blind users is based on a customized set of guidelines. UWEM and A3 are very flexible to work with different sets of guidelines. WCAG is strongly oriented to adhere to automatic evaluation tools.
5.COMPARISON AND EVALUATION OF WEB ACCESSIBILITY METRICS
Considering the coupling level of the metrics, metrics WAB and WAQM are strongly coupled to WCAG guidelines and priority levels to define barriers weights. The metric for blind users is also very linked to the set of guidelines defined by the authors.
Each particular Web development project has its own particularities and needs. Considering which metric to use for each particular project is a key issue for success.
Most of the metrics were designed to support evaluation from automatic evaluation tools. Metrics UWEM and A3 provide more detailed information on how to use these metrics with manual expert inspection methods.
In Section 4 we have presented the main Web accessibility metrics found in the literature. It is possible to notice that each metric has different features, such as strict orientation to guidelines use, flexibility to use customized coefficients, adherence to automatic evaluation tools and others.
The most recent metrics all have included in some way a consideration of the complexity of the Web sites, usually related to the influence of the number of the Web pages in the whole Web site.
When choosing one or more metrics to be used in a Web Engineering process, it is important to perform a careful comparison between the metrics. Managers are supposed to consider their specific needs before choosing a given metric.
The use of barriers weights coefficients was first used in the metric for blind users [12], and all metrics published after have used this concept. WAB and WAQM consider weights based on WCAG barriers. UWEM and A3 suggest the use of coefficients based on the impact on the user of each given barrier.
For example, if a given Web application is supposed to be in conformance with WCAG, a strict guidelines based metric may fit well for the process. However, if the main goal is to perform evaluations based on user testing, it would more interesting to use a flexible metric, which could accommodate different guideline sets.
However, as it may be noticed from Table 5, these metrics do not define any default values obtained from user tests. Metrics WAB and WAQM use fixed values related to the priority of each checkpoint in WCAG. The metric for blind users use metrics related to the impact of each checkpoint on blind users and other related to the percentage of blindness.
In this section, we present an evaluation of the metrics discussed in Section 4, comparing them according to some important attributes. Following, we describe the main attributes used in the comparison, based on good accessibility attributes to good metrics proposed by Zeng [28] and other attributes proposed by the authors of the present paper: • • • • • •
Validation with users: describes whether there is some report on any kind of the metric when compared with the evaluations involving real users; Automated tool: automated tool for metric computation, if any; Use in large scale: use of a metric in large scale evaluations (more than 1,000 pages evaluated).
Only metric A3 [3] has an associated report on a validation with users. An evaluation of six Web pages involving users with disabilities were carried out, and the impressions obtained from users were compared to metrics obtained from inspection based evaluations, using the metrics failure rate, UWEM and A3. The metric A3 showed to be more close to user impressions obtained from user tests.
Guidelines set: set of guidelines used to the metric, such as WCAG 1.0, self-defined guidelines set or customized set of guidelines; Coupling level with guidelines: how coupled the metric is with the guidelines set, or in other words, if it is easy or not to change the guidelines set; Type of evaluation: type of the evaluation methods that a metric is supposed to support, such as automatic guidelines review, manual inspections or user testing; Considers site complexity: the metric considers or not the complexity and size of the site, usually in number of pages; Type of barriers weights coefficients: use of predefined coefficients based on the priority of barriers in a given set of guidelines, user-derived coefficients or other approaches; Default coefficients: default values for barriers weights coefficients;
As most of the metrics were used in the context of automatic evaluation, the majority of them presented an associated automated tool to compute the metrics based on accessibility evaluations. Most of the papers describing the metrics have reported on the use of the metrics in large-scale evaluations. Only the metric for blind users did not. It may be explained by the fact that the main goal of this metric was to support evaluations in an end-user tool for blind users. Besides the comparison using the attributes discussed above, we have also performed a large-scale accessibility evaluation of Brazilian state government Web sites, in order to verify the
76
suitability of the most recent metrics defined in the literature to
perform this type of analysis.
Table 2: Comparison of different Web accessibility metrics Attributes Guidelines set Coupling with guidelines Type of evaluation Considers complexity Barriers coefficients
Web Accessibility Failure Metric for blind Barrier (WAB) rate [24] users [12] [20] WCAG 1.0 Self-defined WCAG 1.0 High
Low
Low
High
Automatic
Automatic
Automatic / Manual
Automatic / Manual
Automatic
Automatic
No
No
Yes
Yes
Yes
Yes
None
Specific for blind WCAG priorities
Based on user impact
Based on user impact
WCAG priorities
Not defined
Not defined
0.8 for priority 1, 0.16 for priority 2 and 0.04 for priority 3
No
Yes
No
EIAO (European Internet Accessibility Observatory) Yes
EIAO (European Internet Accessibility Observatory) Yes
1 priority
Validation with users
No
No
Automated tool
-
KAI (Kit for Accessibility)
-
Use in large scale
Yes
No
Yes
No
• • • • • •
EvalAccess Yes
0.0236, for priority 2 checkpoints, the value was set to 0.0157 and for priority 3 checkpoints, the value was 0.0079.
The evaluation was done using the tool Hera Metrics [10] [8] to compute the accessibility metrics for each Web site. The implementation of Hera Metrics was performed based on the features of the automatic evaluation tool Hera [1]. The following checkpoints were considered in the automatic evaluation:
• •
Web Accessibility Quality Metric (WAQM) [25] WCAG 1.0
High
Default coefficients -
•
Improved Aggregation formula (A3) [3] Customized
Low
Most related to % of blindness
•
Metrics Aggregation formula UWEM [4] Customized
In metric WAQM, the same values define by Vigo et. al [25] were applied. The weight Wz = 0.80 was set for priority 1 checkpoints, Wz = 0.16 for priority 2 checkpoints and Wz = 0.04 for priority 3 checkpoints.
Guideline 1: provide equivalent alternatives to auditory and visual content: checkpoints 1.1 and 1.5; Guideline 3: use markup and style sheets and do so properly: checkpoints 3.2, 3.3, 3.4 and 3.5; Guideline 4: clarify natural language usage: checkpoint 4.3; Guideline 5: create tables that transform gracefully: checkpoint 5.5; Guideline 6: ensure that pages featuring new technologies transform gracefully: checkpoints 6.2, 6.3 and 6.5; Guideline 7: ensure user control of time sensitive content changes: checkpoints 7.4 and 7.5; Guideline 9: design for device-independence: checkpoints 9.5; Guideline 10: use interim solutions: checkpoints 10.2, 10.4 and 10.5; Guideline 11: use W3C technologies and guidelines: checkpoints 11.1 and 11.2; Guideline 12: provide context and orientation information: checkpoints 12.1, 12.2, 12.3 and 12.4.
In order to get a more accurate value for accessibility barriers level, we have performed the evaluation over the main page of each URL plus all pages in the second navigation level of each page, that is, pages that are linked by the main page. A total of 1,232 pages was evaluated in the whole process. For each metric and URL, the average of the accessibility barriers levels of all the pages was calculated. In Figure 1, we show the dispersion graphic (below section) and the Pearson correlation coefficients (above section) for each combination of metrics. It is possible to notice from Figure 1 that although Buhler et. al [3] propose improvements for the aggregation formula of UWEM [4], the correlation coefficient between the two metrics was 1.0, indicating that they are strongly correlated. In fact, the major difference between the two metrics is that they had a distinction between the value ranges. It is possible that this situation may be different for other kind of evaluations. However, this value gives strong indications of the correlation between the metrics.
After computing the number of potential violations and the number of the barriers, Hera metrics computes the barrier levels for the most recent metrics presented in Section 4. For the metrics A3 and UWEM, as experimental values for weights are not still defined, fixed values based on the priority of each checkpoints were used. For priority 1 checkpoints, the value of Wi was set to
77
6.CONCLUSIONS AND FUTURE WORK Web accessibility has become a very important quality attribute within Web Engineering. Providing accessible applications is a key issue to enable a more inclusive Web and to broaden access to everyone, regardless of disabilities. Measuring and monitoring accessibility has been a very important requirement to support Web Engineering processes. Accessibility metrics are important contributions to enable to perform quantitative analysis regarding accessibility. In this paper we have discussed the importance of considering different project aspects when choosing an accessibility metric. Each accessibility metric may be more suitable for different projects according to the needs of such projects. The main contribution of this work was to present a review on the main published works on Web accessibility metrics and the presentation of the main features of each metric. The main attributes of the metrics were discussed to provide means for project managers and developers to ponder about which metric may be more suitable for a given Web development project. Figure 1: Dispersion graphic and Pearson correlation coefficients for metric based accessibility evaluation of Brazilian state government Web sites.
Metrics WAB and WAQM are very coupled with WCAG guidelines. WAQM was designed to strictly adhere to automatic evaluation tools. A3 and UWEM propose the use of user-derived coefficients for barriers weights. However, no indication of a predefined set of coefficients was found.
The correlation between metrics A3 and WAB was 0.86 and the correlation between UWEM and WAB was 0.87. This shows that metrics A3, UWEM and WAB have a high correlation between them. The explanation for this fact may rely on the use of WCAG priorities for barriers weights in the three metrics.
We have also presented the correlations between the most recently published metrics, obtained from a large-scale evaluation of governmental Web pages in Brazil using such metrics.
The metric that differed the most in relation to the others was WAQM. The correlation between A3 and WAQM was -0.023, the correlation between UWEM and WAQM was -0.065, and the correlation between WAB and WAQM was -0.27. The negative value of the correlations is due to the fact that high values in WAQM metric correspond to more accessible pages, while in the other metrics more accessible pages have lower scores. However, the correlation values are indeed very low. Other explanations for such differences may be on the facts that the metric WAQM also uses a not linear formula for calculation of the values and different scales for weights between priority 1, 2 and 3 guidelines.
As future work, we intend to conduct tests comparing the results of evaluations using different metrics to results of evaluation with user testing, considering a wide range of users with different types of disabilities. We also intend to check the effectiveness of accessibility metrics to improve accessibility inside organizations involved in process improvement to adhere to accessibility requirements. Acknowledgement: we would like to acknowledge CNPq (process 551017/2007-4) and INEP/CAPES (WebPIDE project) for funding this research. We also thank CNPq for funding author Andre Freire. Thanks to FAPESP for funding TIDIA-Ae Project, developed in cooperation with this project. We also thank Prof. Mario de Castro for his valuable support in the statistical analysis.
It is worth knowing about the difference between the range of values and of the behavior of accessibility metrics. It is interesting information to take into account when choosing metrics to use in Web Engineering processes. Different metrics with diverse approaches may contribute a lot for obtaining good evaluations. However, it is important to know when comparing values from different metrics do not make sense.
7.REFERENCES [1] C. Benavídez, J. Fuertes, E. Gutiérrez, and L. Martínez. Semi-Automatic Evaluation of Web Accessibility with HERA 2.0. In Proceedings of the 10th International Conference on Computers Helping People with Special Needs (ICCHP 2006), pages 199-206. Springer, 2006
It is not possible to state which metric could be more effective in general cases. Each metric may be more suitable for different projects, according to their needs. The most effective approach for better choosing a good metric is to perform user tests and comparing the results with different metrics obtained from inspection evaluations.
[2] L. C. Briand, C. M. Differding, and H. D. Rombach. Practical Guidelines for Measurement-Based Process Improvement. Software Process Improvement and Practice Journal, 2(4), 1997. [3] C. Buhler, H. Heck, O. Perlick, A. Nietzio, and N. UlltveitMoe. Interpreting results from large-scale automatic evaluation of web accessibility. In Computers Helping
78
People with Special Needs, volume 4061/2006, pages 184191. Springer Berlin / Heidelberg, 2006.
[18] E. M. Mendes, N. Mosley, and S. Counsell. Web Metrics Estimating Design and Authoring Effort. IEEE Multimedia, 8(1):50-57, 2001.
[4] W. W. A. B. Cluster. Unified Web Evaluation Methodology (UWEM 1.0), 2006. Available at http://www.wabcluster.org/uwem/. Last access on January/2008.
[19] L. Olsina and G. Rossi. Measuring Web Application Quality with WebQEM. IEEE MultiMedia, 9(4):20-29, 2002. [20] Parmanto and X. Zeng. Metric for Web accessibility evaluation. Journal of the American Society for Information Science and Technology, 56(33):1394-1404, 2005.
[5] M. K. Daskalantonakis. A Practical View of Software Measurement and Implementation Experiences within Motorola. IEEE Transactions on Software Engineering, 18(11):998 - 1010, 1992.
[21] H. Petrie, F. Hamilton, N. King, and P. Pavan. Remote usability evaluations with disabled people. In CHI '06: Proceedings of the SIGCHI conference on Human Factors in computing systems, pages 1133-1141, New York, NY, USA, 2006. ACM Press.
[6] Dix, J. E. Finlay, G. D. Abowd, and R. Beale. HumanComputer Interaction. Prentice Hall, 3a. edition, 2003. [7] N. E. Fenton and S. L. Pfleeger. Software Metrics: A Rigorous and Practical Approach. PWS Publishing Co., Boston, MA, USA, 1998.
[22] S. L. Pfleeger. Software Engineering: Theory and Practice. Prentice-Hall, Upper Saddle River New Jersey, 2a edition, 2001.
[8] P. Freire, T. J. Bittar, and R. P. M. Fortes. An approach based on metrics for monitoring web accessibility in brazilian municipalities web sites. In Proceedings of the 23rd Annual ACM Symposium on Applied Computing, pages 2420-2424. ACM Press, 2008.
[23] I. Spinuzzi. Learning our abcs: accessibility, bottlenecks, and control in an organized research unit's website. In SIGDOC '07: Proceedings of the 25th annual ACM international conference on Design of communication, pages 201-206, New York, NY, USA, 2007. ACM.
[9] P. Freire, R. Goularte, and R. P. de Mattos Fortes. Techniques for developing more accessible web applications: a survey towards a process classification. In SIGDOC '07: Proceedings of the 25th annual ACM international conference on Design of communication, pages 162-169, New York, NY, USA, 2007. ACM Press.
[24] T. Sullivan and R. Matson. Barriers to use: usability and content accessibility on the Web's most popular sites. In CUU '00: Proceedings on the 2000 conference on Universal Usability, pages 139-144, New York, NY, USA, 2000. ACM Press.
[10] P. Freire, E. H. Tanaka, S. M. A. de Lara, H. V. Rocha, and R. P. M. Fortes. Comparing accessibility in web based elearning environments: a metric based approach. In Proceedings of the XIII Brazilian Symposium on Multimedia and the Web (Webmedia), pages 49-56. ACM Press, 2007.
[25] M. Vigo, M. Arrue, G. Brajnik, R. Lomuscio, and J. Abascal. Quantitative metrics for measuring web accessibility. In W4A '07: Proceedings of the 2007 international crossdisciplinary conference on Web accessibility (W4A), pages 99{107, New York, NY, USA, 2007. ACM Press.
[11] Fuggetta, L. Lavazza, S. Morasca, S. Cinti, G. Oldano, and E. Orazi. Applying GQM in an Industrial Software Factory. ACM Transactions on Software Engineering and Methodology, 7(4):411-448, 1998.
[26] W3C. Web Content Accessibility Guidelines 1.0, 1999. Available at http://www.w3c.org/TR/WCAG10. Last access at February/2008. [27] W3C. Web Content Accessibility Guidelines 2.0 - Working Draft, 2007. Available at http://www.w3c.org/TR/WCAG20. Last access at February/2008.
[12] J. Gonzalez, M. Macias, R. Rodriguez, and F. Sanchez. Accessibility metrics of web pages for blind end-users. In Proceedings of the 2003 International Conference on Web Engineering (ICWE), pages 374{383, Oviedo, Spain, 2003. Springer Berlin / Heidelberg.
[28] X. Zeng. Evaluation and Enhancement of Web Content Accessibility for Persons with Disabilities. PhD thesis, University of Pittsburgh, 2004.
[13] ISO - International Standard Organization, New York, NY. (ISO/IEC 12207) Standard for Information Technology Software Lifecycle Processes, 1998. 85 S. [14] ISO - International Standard Organization. (ISO 9126-1) Software Engineering - Product Quality-Part 1: Quality Model. Geneva, 2001. [15] M. Y. Ivory, R. R. Sinha, and M. A. Hearst. Empirically validated web page design metrics. In CHI '01: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 53{60, New York, NY, USA, 2001. ACM. [16] S. Kan. Metrics and Models in Engineering. Addison Wesley, 1995.
Software Quality
[17] L. Lavazza and G. Barresi. Automated support for processaware definition and execution of measurement plans. In ICSE '05: Proceedings of the 27th international conference on Software engineering, pages 234-243, New York, NY, USA, 2005. ACM.
79
80
