The current issue and full text archive of this journal is available at www.emeraldinsight.com/1066-2243.htm
INTR 19,1
Homepage not enough when evaluating web site accessibility Stephanie Hackett and Bambang Parmanto
78 Received 16 March 2008 Accepted 28 September 2008
Health Information Management Department, School of Health and Rehabilitation Sciences, University of Pittsburgh, Pittsburgh, Pennsylvania, USA Abstract Purpose – The purpose of this paper is to determine if the homepage of a web site is representative of the whole site with respect to accessibility. Design/methodology/approach – The paper presents an intraclass correlation (ICC) between homepage web accessibility barrier (WAB) scores and the WAB scores of web site levels 1 through 3 for 33 popular web sites. Findings – The paper finds that the homepage is not sufficient to detect the accessibility of the web site. ICC of the homepage and average of levels 1-3 is 0.250 (p ¼ 0:062) and ICC of levels 1, 2, and 3 is 0.784 (p , 0:0001). Evaluating the homepage and first-level pages gives more accurate results of entire site accessibility. Originality/value – This is first study correlating homepage accessibility with web site accessibility. Keywords Worldwide web, User studies, Disabilities Paper type Research paper
Internet Research Vol. 19 No. 1, 2009 pp. 78-87 q Emerald Group Publishing Limited 1066-2243 DOI 10.1108/10662240910927830
Introduction Nielsen (2000) argues that the homepage is the gateway to the web site and therefore sets the tone of the entire web site for the user. There are many studies evaluating the accessibility of the web for people with disabilities that have included only the homepages (Davis, 2002; Flowers et al., 1999; Lazar et al., 2003; Lazar and Greenidge, 2006; Loiacono and McCoy, 2006; Klein et al., 2003; Paris, 2006; Spindler, 2002; Yu, 2002), a selected handful of pages (Diaper and Worman, 2003; Sloan et al., 2002; Thompson, 2003), or limit the levels of pages of the web site being evaluated (Jackson-Sanborn et al., 2002; Hackett et al., 2004). One can look at the homepage of a site as being the “doorway” into the site as was done in Yu (2002). Yu (2002) looked at four sets of homepages for each of the 108 California Community Colleges, where available, including the homepages of the college, library, distance education, and disabled student programs and services. The rationale behind looking at only homepages is that the initial web pages are gateways to major resources; if these pages are free from accessibility errors, they provide successful pathways to the information (Yu, 2002). Or, as stated in Klein et al. (2003), if the home page is not accessible, the rest of the site probably is not accessible. Similarly, Loiacono and McCoy (2006) felt that homepage evaluation was appropriate because visitors are most likely to enter through a homepage and, if the homepage is not accessible the accessibility of the remainder of the site becomes irrelevant.
If the homepage is a true representation of the entire site, studies looking at only the homepage are valid; however, if not, these studies have no consequence. This study asks the question of whether the homepage is enough when evaluating a web site for accessibility for people with disabilities, in particular visual impairments, and hypothesizes that homepage accessibility is indicative of the accessibility of the entire web site. Previous studies evaluated the accessibility of web sites and not the correlation of accessibility scores between the levels of a web site. The findings from this study will guide future evaluations of web accessibility and may also be relevant to other types of web site evaluation. Background Web accessibility The term accessibility, as applied to the Internet, means that anyone can equally access the information presented, regardless of device and/or personal limitations. Persons with disabilities, however, often find that the web is far from accessible. Some persons with disabilities use assistive technologies to aid them in accessing Internet information. A person with low vision can use a screen magnifier to enlarge the text on the computer, use a larger font, or increase the contrast between background and foreground colors (Godwin-Jones, 2001). A person who is blind can use a text-to-speech program that reads aloud the text on the screen via a voice synthesizer or use a refreshable Braille display to obtain a tactile output of the information. Web designers must take into consideration both the limitations experienced by individuals with disabilities and the limitations that coincide with their use of computer-related assistive technologies. Section 508 of the Rehabilitation Act of 1973 (29USC§794d, 1986 (A-Prompt, n.d.)) is probably the most important piece of legislation regarding accessibility for persons with disabilities and the Internet. Section 508 requires all electronic technology developed or purchased by federal agencies to be accessible to persons with disabilities as of June 2001, unless this would pose an undue burden. In the event of an undue burden, the agency must provide alternative access to the information. Section 508 requires that federal agencies and departments follow accessibility regulations when procuring, developing, using, or maintaining electronic and information technology (USAccessBoard, 2004), however it does not require manufacturers to develop accessible technologies. To develop accessible web sites, designers can utilize the World Wide Web Consortium’s (W3C) Web Content Accessibility Guidelines (WCAG) (Chisolm et al., 2001), a voluntary set of guidelines to be followed when designing accessible web pages. WCAG 1.0 is composed of 14 guidelines, each with several checkpoints (65 checkpoints in all), addressing various aspects of accessibility to persons with physical, visual, hearing and cognitive/neurological disabilities. Importance of web accessibility Early studies on information technology for persons with disabilities found that computer and Internet access can increase levels of independence and have a positive impact on the academic progress and career success of individuals with disabilities (Coombs, 1991; Burgstahler, 1992). It can also bring a sense of community that has not been experienced before (Ritchie and Blanck, 2003; Hillan, 2003; Sanyal, 2006).
Homepage not enough
79
INTR 19,1
80
However, web content must be accessible to persons with disabilities if it is to have positive outcomes. The web has become less accessible for users with disabilities (Amtmann et al., 2000; Bucy et al., 1999; Hackett, 2004; Heim, 2000; USDOJ, n.d.), as web page design has evolved from a text-based environment to one that includes images, frames, tables, animated Java applications, and streaming audio and video. Currently, the web is three times less usable by persons with disabilities than those without (Nielsen, 2001). However, accessibility is not just for persons having an obvious disability or functional limitation; it also benefits those using low-end technology with lower modem speeds, persons utilizing wireless Internet connections, and the aging. In addition, the average age of the world population is rapidly increasing (Mynatt et al., 2000) and, as we age our chances of developing a mild or moderate disability increases. By age 65, most persons have lost at least some of their ability to focus, resolve images, distinguish colors, and adapt to changes in light (Lescher, 2000). Measuring web accessibility Currently, most methods (Watchfire, n.d., 2006, A-Prompt, n.d.) of evaluating accessibility provide absolute ratings, either the web site complies with all guidelines or it is considered inaccessible, because the intention of these methods is to assist in the design of an accessible web site. Also, because these tools are typically used during the design or re-design phase of web site development, most include some manual inspection of the web site. For these reasons, current methods do not take into consideration the size and complexity of the web site or provide an accessibility score along a continuum. This makes it difficult to trend accessibility or to compare web sites or groups of web sites. The Web Accessibility Barrier (WAB) score, the formula of which is shown below, was developed with the intentions of overcoming the deficiencies of the current measurements used in web accessibility studies: PP nv N v ðwv Þ WAB Score ¼
p
v
Np
where: p:
total pages of a web site;
v:
total violations of a web site;
nv:
number of violations;
Nv
number of potential violations;
Wv: weight of violations in inverse proportion to WCAG priority level; and Np:
total number of pages checked.
The WAB metric provides a quantitative score along a continuous range of values. It is a proxy indicator of web accessibility that examines 25 checkpoints that can be automatically evaluated, based on WCAG 1.0 and Section 508 guidelines, which are accepted web accessibility standards. The number of violations of the checkpoints is
the basis for the score. The measure utilizes the checkpoint priorities in reverse: priority 1 violations weigh three times more than a priority 3 violation, since priority 1 violations pose more difficulties in accessibility than priority 3 violations. The WAB score for each web site is the summed WAB score of the web pages normalized against the total number of pages. A higher WAB score means more accessibility barriers exist. A lower score means better conformance with WCAG 1.0. A score of zero denotes that the web site does not violate any web accessibility guidelines and should not present any accessibility barriers to persons with disabilities. A previous work (Parmanto and Zeng, 2005) proposes the WAB score as a novel metric for measuring content accessibility of the web for persons with disabilities. This study, which includes reliability and validity testing of the metric, found that the metric provides a good representation of a web site’s accessibility. Further analysis revealed that a threshold exists between accessible and non-accessible web sites. A WAB score of 5.5 or less means a web site has better conformance to the WCAG and contains few barriers to accessibility, whereas scores above 5.5 indicate increasingly more barriers to accessibility. Methodology Materials This study hypothesizes that homepage accessibility is indicative of the accessibility of the entire web site. Because the number and distribution of web sites are undeterminable due to the size and dynamics of the web, many probabilistic sampling methods, such as random or stratified sampling, are not applicable. An alternative sampling method widely adopted by researchers conducting studies on web sites is to utilize the directory services provided by many web search engines. The hypothesis was explored using the first 50 web sites from Alexa.com’s top 500 English web sites on April 10, 2007. Alexaw Internet’s traffic rankings rate how popular a site is with other users (AlexaInternet, n.d.). Since rankings of web sites can change from day to day, depending on the traffic to that site, the most popular web sites can change from day to day. These web sites were chosen to be representative of present day popular web sites. Measurement The WAB metric (Parmanto and Zeng, 2005) was utilized to determine accessibility of each page of the web site. The WAB metric is a proxy indicator of web accessibility and provides a quantitative score along a continuous range of values ranging from perfectly accessible to completely inaccessible, with a lower score indicating better accessibility. Pages from levels zero through three of the web site were evaluated (Figure 1). Each page was counted once: for example, if the homepage and a page at level 2 have links to a search page, the search page was only counted once. Data analysis Descriptive statistics were used to describe the web sites as a whole. Because the real interest is whether the homepage WAB score is correlated to the WAB score of the entire web site, an intra-class correlation (ICC) (Shrout and Fleiss, 1979) statistic was computed comparing the WAB score of the homepage (level 0) to the WAB scores of
Homepage not enough
81
INTR 19,1
82
Figure 1. First four levels of a web site
the other levels of the web site (average of WAB scores of levels 1, 2, and 3). ICC was also computed to determine correlation between the scores of the levels of the web site other than the homepage. The ICC statistic evaluates absolute agreement (within-subjects effects) and consistency (between-subjects effects) depending on how the denominator is designated. The denominator is one of consistency when the variance in the level scores is excluded from the denominator mean square, and it is one of absolute agreement when the variance of the level scores is not excluded from the denominator. Two-way mixed effects model single measure reliability, ICC(3,1), with absolute agreement was used was used in this analysis (McGraw and Wong, 1996). The formula to compute intra-class correlation coefficient for this analysis is provided below: ICC A; 1 ¼
MSrows 2 MSerrors : MSrows þ ðk 2 1ÞMSerrors þ nk ðMScolumns 2 MSerrors Þ
Results Of the top 50 web sites, 33 were evaluated between April 11, 2007 and July 15, 2007. We used an accessibility evaluation tool and crawler which we developed, called Kelvin (Parmanto and Zeng, 2005), to evaluate the pages of each web site for accessibility according to the WAB score. A total of 17 web sites were eliminated from the analysis for the following reasons: . six were unavailable for Kelvin evaluation due to lack of a designated “content type” within the HTML, a Kelvin requirement; . ten were excluded from the study because the site did not generate four levels of data for evaluation: the site was simple and not composed of four levels or the site was composed of multiple unrelated domains (i.e.: geocities.yahoo.com and finance.yahoo.com are not considered to be the same site); and . one web site was predominantly in a language other than English.
The means of each level and the averages included in the analysis are included in Table I. The ICC of level 0 and the average of levels 1, 2, and 3 is 0.250 (p ¼ 0:062) and indicates a low correlation between the homepage WAB score and that of the rest of the web site. An ICC was computed for levels 1, 2, and 3, to see how these levels correlate to each other: 0.784 (p , 0:0001) indicating a high-level of correlation between these levels. ICCs for the homepage and each level were computed (Table II). The homepage accessibility correlates most highly with level 1; however, this correlation is still only moderate. Level 1 is also highly correlated with the average WAB scores of levels 2 and 3. A paired-sample t-test was computed to compare level 0 to the average WAB of levels 1, 2, and 3 and there is a statistically significant difference (df ¼ 32, p ¼ 0:043). The average WAB of all homepages is 8.17 and the average WAB of levels 1, 2, and 3 is 6.93. Finally, a scatterplot of the homepage WABs and the WABs of the average of levels 1, 2, and 3 depicts only a small pattern of correlation (Figure 2).
Homepage not enough
83
Discussion The WAB score of the homepages of the popular web sites evaluated are not strongly correlated to the WAB scores of the rest of the web site, leading to a rejection of the hypothesis that homepage accessibility is indicative of the accessibility of the entire web site. These results demonstrate that evaluating the homepage alone is not sufficient when evaluating web pages for accessibility. Previous studies using only the homepage in the analysis (Davis, 2002; Flowers et al., 1999; Lazar et al., 2003; Lazar and Greenidge, 2006; Loiacono and McCoy, 2006; Klein et al., 2003; Paris, 2006; Spindler, 2002; Yu, 2002) may not have shown the whole picture of accessibility. However, the WAB scores for levels 1, 2, and 3 of these web sites are highly correlated and previous
Level of web site
Total number of pages
Avg. number of pages
Mean WAB score
St. dev.
33 2,217 29,928 264,377 – –
1 67 907 8,011 – –
8.17 6.47 7.06 7.25 6.93 7.16
3.25 2.54 2.28 2.36 2.24 2.29
Level 0 Level 1 Level 2 Level 3 Average of levels 1,2, and 3 Average of levels 2 and 3
Levels
ICC(3, 1)
95% CI
p-value
Level Level Level Level Level Level
0.784 0.25 0.372 0.183 0.134 0.747
0.65-0.88 20.07-0.53 0.05-0.63 20.14-0.48 20.20-0.45 0.55-0.87
, 0.0001 0.062 0.006 0.138 0.218 , 0.0001
1, level 2, and level 3 0 and average of levels 1, 2, 3 0 and level 1 0 and level 2 0 and level 3 1 and average of levels 2 and 3
Table I. Mean WAB for different levels of the web sites
Table II. Intraclass correlation statistics
INTR 19,1
84
Figure 2. Scatterplot of WAB score of level 0 vs average WAB score of levels 1,2, and 3
studies that included more pages than just the homepage (Diaper and Worman, 2003; Sloan et al., 2002; Thompson, 2003), if even a limited number of deeper pages (Jackson-Sanborn et al., 2002; Hackett et al., 2004), may have provided better estimates of the accessibility of the web sites under evaluation. When using popular evaluation tools that include a manual component, like Bobby, using only the homepage for accessibility evaluation is attractive from the practical standpoint. As shown in Table I, the average number of pages that need to be evaluated when evaluating just the homepage is 1, while to evaluate level 1: 67, level 2: 907, and level 3: 8,011. Level 1 is almost two degrees of magnitude more pages than level 0, while level 3 and 2 are one degree of magnitude larger than the shallower level. Manually evaluating all web pages to level 3 becomes prohibitively expensive and impractical. Because the WAB scores of the deeper levels are highly correlated, using the homepage and level 1 of a web site will give more accurate results as to the accessibility of the entire web site. Because level 1 is highly correlated with level 2 and 3, it provides sufficient representation of these levels. The practical implication of the findings is that using only the homepage and level 1 provides accurate results while it still being economical compared to using all pages of a web site. Contributions Because the WAB scores of levels 1, 2, and 3 are highly correlated, it makes sense to limit the evaluations to the homepage and level 1 of the web site for purposes of accuracy and resource management, as it is administratively simpler to obtain data for level 0 and level 1 than it is to collect data for level 2 and level 3. This is particularly
true when using more manual methods of accessibility evaluation. Future studies in web accessibility should also include the homepage and at least one other level of the web site to ensure accurate measurement of accessibility. Limitations A limitation of the study is the small sample size. While 33 popular web sites were used in this study, a future study should be conducted that includes a larger sample size and a wider representation of web sites. Another limitation can be attributed to the variation in sample sizes collected from the popular web sites and the varying number of pages evaluated for each level of the web site. Some of the web sites had thousands of pages included in the analysis, while others had relatively few. This could bias the averages to be more consistent with the scores of the sites that had a larger number of pages evaluated. Future direction In the future, the authors plan to pursue a study that would include a larger sample of web sites and more genres of web sites to determine if the findings apply to various genres. While this study found that the homepage and level 1 provide a representation of site accessibility, it is not clear if this is the optimal sampling method. A future larger study will also examine optimal sampling level to determine if the homepage and level 1 is the optimal sampling method or if an alternative level should be used. References AlexaInternet (n.d.), “Alexa Internet”, available at: www.alexa.com Amtmann, D., Johnson, K. and Cook, D. (2000), “Making web-based tables accessible for users of screen readers”, Library Hi Tech, Vol. 20 No. 2, pp. 221-31. A-Prompt (n.d.), “29U.S.C.§794D (1986) Section 508 of the Rehabilitation Act of 1973”, available at: available at: www.aprompt.ca/overview.html Bucy, E.P., Lang, A., Potter, R.F. and Grabe, M.E. (1999), “Formal features of cyberspace: relationships between web page complexity and site traffic”, Journal of the American Society for Information Science, Vol. 50 No. 13, pp. 1246-56. Burgstahler, S.E. (1992), “Disabled students gain independence through adaptive technology services”, EDUCOM Review, Vol. 27, pp. 45-6. Chisolm, W., Vanderheiden, G. and Jacobs, I. (2001), “Web content accessibility guidelines”, Interactions, Vol. 8 No. 4, p. 34. Coombs, N. (1991), “Window of equal opportunity – online services and the disabled computer user”, Research and Education Networking, Vol. 2 No. 9, pp. 15-19. Davis, J.J. (2002), “Disenfranchising the disabled: the inaccessibility of Internet-based health information”, Journal of Health Communication, Vol. 7 No. 4, pp. 355-67. Diaper, D. and Worman, L. (2003), “Two falls out of three in the automated accessibility assessment of world wide web sites: A-Prompt v. Bobby”, in Johnson, P., Palanque, P. and O’Neill, E. (Eds), People and Computers XVII, Springer-Verlag, Berlin. Flowers, C.P., Bray, M. and Algozzine, R.F. (1999), “Accessibility of special education program home pages”, Journal of Special Education Technology, Vol. 14, pp. 21-6.
Homepage not enough
85
INTR 19,1
86
Godwin-Jones, B. (2001), “Emerging technologies – accessibility and web design why does it matter?”, Language Learning and Technology, Vol. 5 No. 1, pp. 11-19. Hackett, S. (2004), “Accessibility, complexity, and readability of Internet websites through time”, Health Information Management, University of Pittsburgh, Pittsburgh, PA. Hackett, S., Parmanto, B. and Zeng, X. (2004), “Accessibility of Internet websites through time”, ASSETS’04. Atlanta, GA, ACM, New York, NY. Heim, J. (2000), “Locking out the disabled (web site accessibility)”, PC World, Vol. 18, pp. 181-5, (US Edition). Hillan, J. (2003), “PatchWorx: connecting unhealthy and disabled children in an online community”, Clinical Medicine and Research, Vol. 1, pp. 259-60. Jackson-Sanborn, E., Odess-Harnish, K. and Warren, N. (2002), “Web site accessibility: a study of six genres”, Library Hi Tech, Vol. 20 No. 3, pp. 308-17. Klein, D., Myhill, W., Hansen, L., Asby, G., Michaelson, S. and Blanck, P. (2003), “Electronic doors to education: study of high school website accessibility in Iowa”, Behavioral Sciences and the Law, Vol. 21 No. 1, pp. 27-49. Lazar, J. and Greenidge, K.-D. (2006), “One year older, but not necessarily wiser: an evaluation of homepage accessibility problems over time”, Universal Access in the Information Society, Vol. 4 No. 4, pp. 285-91. Lazar, J., Beere, P., Greenidge, K. and Nagappa, Y. (2003), “Web accessibility in the Mid-Atlantic United States: a study of 50 home pages”, Universal Access in the Information Society Journal, Vol. 2 No. 4, pp. 331-41. Lescher, J. (2000), “Designing web sites for the blind”, Econtent, Vol. 23 No. 2, pp. 14-18. Loiacono, E.T. and McCoy, S. (2006), “Website accessibility: a cross-sector comparison”, Universal Access in the Information Society, Vol. 4 No. 1, pp. 393-9. McGraw, K.O. and Wong, S.P. (1996), “Forming inferences about some intraclass correlation coefficients”, Psychological Methods, Vol. 1 No. 1, pp. 30-46. Mynatt, E.D., Essa, I., Rogers, W., Scholtz, J. and Thomas, J. (2000), “Increasing the opportunities for aging in place”, Conference on Universal Usability, CUU 2000, ACM, New York, NY. Nielsen, J. (2000), Designing Web Usability, New Riders Publishing, Indianapolis, IN. Nielsen, J. (2001), “Beyond accessibility: treating users with disabilities as people”, available at: www.useit.com/alertbox/20011111.html Paris, M. (2006), “Website accessibility: a survey of local e-government websites and legislation in Northern Ireland”, Universal Access in the Information Society, Vol. 4 No. 4, pp. 292-9. Parmanto, B. and Zeng, X. (2005), “Metric for web accessibility evaluation”, Journal of the American Society for Information Science and Technology, Vol. 56 No. 13, pp. 1394-404. Ritchie, H. and Blanck, P. (2003), “The promise of the internet for disability: a study of on-line services and web site accessibility at centers for independent living”, Behavioral Sciences and the Law, Vol. 21 No. 1, pp. 5-26. Sanyal, I. (2006), “Empowering the impaired through the appropriate use of information technology and Internet”, Studies in Health Technology and Informatics, Vol. 121, pp. 15-21. Shrout, P.E. and Fleiss, J.L. (1979), “Intraclass correlations: uses in assessing rater reliability”, Psychological Bulletin, Vol. 86 No. 2, pp. 420-8. Sloan, D., Gregor, P., Booth, P. and Gibson, L. (2002), “Auditing accessibility of UK higher education web sites”, Interacting with Computers, Vol. 14 No. 1, pp. 313-25.
Spindler, T. (2002), “The accessibility of web pages for mid-sized college and university libraries”, Reference & User Services Quarterly, Vol. 42 No. 2, pp. 149-54. Thompson, T. (2003), “Research on web accessibility in higher education”, Information Technology and Disabilities, Vol. 9 No. 2. USAccessBoard (2004), “Accessible telecommunications product design technical assistance”, available at: www.access-board.gov/sec508/telecomm-course.htm USDOJ (n.d.), “Federal Agencies’ web pages”, available at: www.usdoj.gov/crt/508/report/web. htm Watchfire (2006), “WebXACT”, available at: http://webxact.watchfire.com/ScanForm.aspx Watchfire (n.d.), “Bobby”, available at: http://bobby.watchfire.com Yu, H. (2002), “Web accessibility and the law: recommendations for implementation”, Library Hi Tech, Vol. 20 No. 4, pp. 408-19. Corresponding author Stephanie Hackett can be contacted at:
[email protected]
To purchase reprints of this article please e-mail:
[email protected] Or visit our web site for further details: www.emeraldinsight.com/reprints
Homepage not enough
87
