Jeffrey P. Bigham University of Rochester Rochester, NY 14617 ...

Accessibility by Demonstration: Enabling End Users to Guide Developers to Web Accessibility Solutions (DRAFT, DO NOT DISTRIBUTE) Jeffrey P. Bigham University of Rochester Rochester, NY 14617 jbighamcs.rochester.edu

Bernie Zhang Pennsylvania State University State College, PA

ABSTRACT

One way to define accessibility for an individual is whether he can, with reasonable effort, accomplish his goal. The personal and subjective nature of accessibility can make evaluation difficult, especially for non-experts. Furthermore, developers often (i) lack easy access to the diversity of assistive technology employed by users, and (ii) are unaware of the different access patterns people might employ using these alternative interfaces. In this paper, we describe a new approach to evaluation called Accessibility by Demonstration (ABD), which enables disabled users to demonstrate accessibility problems, record those interactions as flexible web macros, and easily send these recordings to others for playback on any computer. By simply following a link, developers can use these replays not only to clearly see what problem the user experienced, but also to check that whatever fix they make solves the problem. Since following a link only requires a web browser, no new software needs to be installed. We present an implementation of ABD (WebAnywhere-ABD) targeting blind web users and an evaluation that suggests that WebAnywhere-ABD improves participants’ ability to locate accessibility errors comparable to other less personal evaluation tools. ACM Classification Keywords

H.5.2 Information Interfaces and Presentation: User Interfaces; K.4.2 Social Issues: Assistive technologies for persons with disabilities General Terms

Design, Human Factors, Experimentation Author Keywords

Programming by Demonstration, End User Programming, Web Accessibility, Web Usability, Evaluation, Blind Web Users INTRODUCTION

The final test of web accessibility is whether someone attempting to access web content can do so effectively. Accessibility varies on many dimensions and achieving effective access across dimensions for different users each can be difficult. Furthermore, disabilities can affect individuals in different ways and usability is funda-

Jeremy Brudvik Georgia Institute of Technology Atlanta, GA

mentally tied to skill with appropriate access technology. For example, the same content that is accessible to someone proficient in using the myriad of features (e.g., keyboard shortcuts) available in popular screen a reader, such as JAWS [6] or Window-Eyes [10], might be inaccessible to someone who has only memorized a select few functions. Can content that is only usable by some disabled people using advanced features of specific access technology be said to be accessible? The diverse issues involved in creating accessible web content makes it a frustrating, subjective, and difficult task. Guidelines and standards can serve as a valuable starting point for developers hoping to create accessible web content, but many subjective and practical factors influence the realized accessibility for each visitor. Accessibility and usability are often linked [25] - if a resource is not usable for certain disabled users then it is in effect not accessible. If the access technology tool is not available when users try to access content, then that content is not accessible to them. Effective access can be seen through the following three constituent dimensions: accessibility, availability and usability [12]. Accessibility concerns making content possible to access. For instance, a blind person cannot access the information in an image that does not have an accompanying textual description. In this context, usability means providing access in a way that users will understand how to use effectively. For instance, content that is laid out in a confusing way, or for which no efficient mechanism is provided to find the content the user wants quickly may be accessible but not usable. Finally, availability means having access to the tools required to facilitate accessibility and usability. Access technology is rarely made available on all computers by default, but without it many users are unable to use those computers at all. These dimensions, particularly usability and availability, should also extend to accessibility evaluation as well, but existing tools do not adequately achieve it. As a first step, content creators and developers need access to the access technology used by the groups they are

b a Iterative Improvement

http://statefruits.org/

http://domain.org/XN2320

Demonstrates Accessibility Problem

User

Goto http://statefruits.org/ Ctrl+T (next table) Tab (next focusable) Tab (next focusable) Tab (next focusable) Tab (next focusable) Tab (next focusable) Tab (next focusable)

User Demonstration

http://statefruits.org/

start end

Demonstration of Accessibility Problem

Developer

Figure 1. (a) Using the ABD approach, access technology users can capture problematic interactions and send recordings to developers. (b) Developers can replay these interactions to improve understanding of the problem without installing new software. Developers can replay these recordings after changing their content to test if their changes have helped.

hoping to support to aid in understanding the usability of their web content. We argue that simply having access is not enough, however, as they also need to know how to use these tools and the features the users of these tools that users are likely to know. This paper introduces the concept of Access by Demonstration (ABD) as a method of bringing the experience of access technology users to developers in a low-overhead way. Using this approach, users experienced with their own access technology can demonstrate the problems that they face while using web content remotely and easily share the demonstration by sharing a single URL. Encapsulated in this URL is a demonstration of the problems that they faced, how they used their access technology tool to arrive at the problem, and the access technology itself so developers have immediate access to it. This paper explores ABD in the context of improving web access for blind computer users. Out implementation, WebAnywhere-ABD, is build on top of the WebAnywhere [15], a web-based screen reader that, unlike conventional desktop screen readers, can be used on any computer without installation (even locked-down public terminals). Popular screen readers available for the desktop cost nearly $1000 and required installation, both of which represent a substantial burden to developers who may want to use a screen reader to test their web sites. Developers can use WebAnywhere to understand accessibility problems on their web sites (some developers already are [11]). When combined with the ABD approach, developers do not need to be as skilled at using access technology because the skill is largely pushed off to the expert blind users of WebAnywhere. Example Use Case

ABD can help experienced and inexperienced computer users demonstrate a wide variety of problems that they face in accessing and using content and send those demonstrations to others. To illustrate how the ABD approach can be used in the context of WebAnywhere, consider the following scenarios:

First, consider Betty, an experienced blind computer user who has experienced an accessibility problem on a web site. The homepage of a local restaurant has not used headings on its page, making the process of navigating through the content on the page inefficient. Betty uses WebAnywhere-ABD to demonstrate her path through the web page, first pressing CTRL+H several times to show the response when attempting to jump to the next heading, and then showing the long inefficient process of navigating through the content by using the DOWN ARROW to go through the page element by element. Once she has finished, her recording is capture as a URL that she can send along with a comment to the email address listed on the page, for example “This page does not use headings correctly.” Importantly, she does does not need to visit the webmaster in person to demonstrate the problem. Upon receiving Betty’s email, webmaster John, geographically located far away, simply simply clicks on the link that Betty sent to see a replay of her interactions with his site. Importantly, this replay isn’t just a video but a live interaction with the site. John can interrupt the replay at any point and see how his site responds to keyboard shortcuts of his choosing. He can also update the site and see if that improves the replay, helping him iteratively improve the accessibility of his site and receive feedback during the process. Next, consider James, an inexperienced blind computer user who experienced a problem navigating a web site. James was navigating a web site when his screen reader started reading what sounded like gibberish. James presses CTRL+R and WebAnywhere-ABD records the process he took to get to the point in the page where he was and captures that process in a URL that he can similarly use to share the problem he was having with others. This retroactive recording allows James to easily capture problems that he faces without spending extra time trying to reproduce them and without having to remember how he got into the state where the problem occurred. In this case, the screen reader had

started reading a complicated URL because James had reached an image link that lacked alternative text. In this example, James neither had to understand what problem had occurred or the technical reason for it - he only needed to know that he had reached a point that was inaccessible to him. Summary of Contributions

This work makes the following three contributions: • We introduce the concept of Accessibility by Demonstration, and present an implementation, WebAnywhereABD, for blind users that can be used on any computer without installing new software. • We show how retroactive recording of trails can be used as part of the ABD approach to capture and share recordings of accessibility problems after they have already happened. • We present a study that suggests that WebAnywhereABD can help web developers be more successful at improving some accessibility problems, illustrating the promise and viability of the approach. RELATED WORK

The goal of the ABD approach is to make it easier to leverage the expertise of disabled users in the process of developing and improving the accessibility of web sites. Related work falls generally into the following three categories: (i) work in accessibility assessment and reporting, (ii) systems that involve users in the process of improving web accessibility, and (iii) systems for programming by demonstration and interactive help. Accessibility Assessment

A number of accessibility validators have been developed to help developers evaluate and improve the accessibility of their web pages. Examples include Bobby [3], FAE [4], and WAVE [8]. Developers point these tools to their web sites, and receive back a list of accessibility errors and warnings. Although a valuable first step in ensuring that ones web page is accessible, evaluators have two primary shortcomings. First, validators cannot evaluate accessibility issues that cannot be detected automatically; while evaluation tools can detect if an image lacks alternative text, they cannot judge if that alternative text is appropriate and informative [16]. Subtle usability problems, such as problems with reading order or lack of heading tags or other markup, are even more difficult to detect automatically. These usability problems seem likely to get worse as web pages become more complex and begin to behave more like applications than static document [5]. As a fallback, evaluation tools present a number warnings about content that may have a problem. Accordingly, the second problem with evaluation tools is that they present so many warnings that novice developers might be overwhelmed or be discouraged from fixing

anything. While those who are motivated will investigate all warnings, the persistence of accessibility problems on the web suggests that a typical developers may be unwilling to investigate each issue, especially when many will be revealed to not be problems. As an example, WAVE displays a warning anytime a tabindex attribute is used within a web page. This attribute can be used correctly to enforce a meaningful tab order through a web page, but when used incorrectly may result in a confusing ordering. Because WAVE, and other tools, cannot automatically judge correct usage of tabindex, they present a warning. Figure 2 shows the complexity that can result from all of these warnings. ABD enables users of a web site to demonstrate and share the most troubling accessibility issues for them, helping focus the efforts of web developers and providing an opportunity to improve the understand that developers have of why the issue is a problem. Another approach to evaluation is to provide views of content that simulate what a disabled user might experience. For instance, ADesigner can visually simulate the usability of a web site as either a blind, low-vision, or color-blind person might experience it [27]. WAVE has recently introduced features as well designed to help developers appreciate the problems that disabled users might experience when accessing the web. For instance, one option displays a web page in the linear order exposed by screen readers. Although these tools help developers understand how certain groups might experience the web, they miss out on the personal nature of accessibility and still require developers to understand the problems that these tools highlight. ABD explicitly demonstrates a particular problem experienced by a blind web user and allows a developer to iterate over that specific problem in order to improve it. Screening is the use of access technology to help identify accessibility issues. Henry describes the advantages and limitations of using screening techniques to evaluate accessibility [18]. The advantage is that it may help someone appreciate the experience of someone with different capabilities, but a disadvantage, especially when using a complicated software program like a screen reader, is that an inexperienced user may incorrectly associate their own inability to accomplish a task with the inaccessibility of their site whereas an experienced screen reader user may have no problem. Although valuable, screening also suffers from the fact that screen readers are expensive and complex software programs that developers might be unlikely to install. The ABD approach in contrast to existing methods for screening provides the experience of using a screen reader without requiring developers to install new software. Furthermore, developers do not have to go it alone, but can build from the demonstrations provided by expert blind users. Mankoff et al. compared the results of multiple sighted

developers using screening techniques with evaluation by remote blind users [23]. Developers using screen readers found many more problems than did the blind evaluators, although the problems found by blind users were quite accurate. Commonly, when blind users perform an evaluation they focus on the most problematic accessibility problems and sometimes are not able to fully evaluate a web site because the accessibility problems discovered prevent full access to the site [24]. Takagi et al. describes the problems of users not knowing what is not accessible to them as a challenge for IBM’s Social Accessibility project [29]. ABD seeks to provide the best of both evaluation techniques - disabled users can indicate problems that are problematic for them and send a clear demonstration to developers. From that demonstration, web developers can then try on their own to see if other parts of their site are accessible. Involving Users

Target users should be involved in accessibility evaluation whenever possible. Although Mankoff et al. found that developers discovered the most accessibility problems (had high recall), the blind participants in their study found more subtle problems and were very accurate (had high precision) [23]. Communication between developers and users has primarily been restricted to the descriptions that can be sent over email. Several projects have explored how blind web users might independently improve the accessibility of the content that they access. For instance, by writing Accessmonkey scripts using provided end user tools, blind web users could improve the accessibility of web sites [13]. These scripts could be shared with developers and the changes made by the scripts saved to HTML with the goal of having the developer incorporate the changes into the original page. AxsJAX [1] is a scripting framework that makes web pages into accessible web applications. Key functionality of each web page is exposed using shortcut keys and the semantics of the page are leveraged to make pages easier to use. Although programming is required to create an AxsJAX script, any programmer can write a script that other people can use. AxsJAX requires the user to already have a screen reader installed. The ABD approach seeks to involve non-programmers in the process of improving web pages and provides a clear and easy path to demonstrating problems to developers. The Social Accessibility project seeks to match blind web users experiencing accessibility problems with volunteers who can help them improve these problems [28]. In some sense, the blind users of Social Accessibility are demonstrating accessibility problems and sharing those problems with the volunteers. Currently, however, demonstrations are restricted to selections - users can select an image that lacks alternative text. They can also describe more complex problems using free text

- for example, “This page lacks heading tags.” or “None of the form elements have labels.” Using the ABD extension to WebAnywhere, users can record themselves trying to complete richer tasks and send them to others who can view them without installing new software. Users can also directly improve access and usability for themselves by demonstration, which we discuss next.

Programming by Demonstration & Web Automation

Programming by Demonstration (PBD) and web automation systems have explored how to capture procedural knowledge and express it to users. COACH [26] and Eager [17] are early systems that work with standard desktop applications instead of the web. COACH observes computer users in order to provide targeted help, and Eager learned and executed repetitive tasks by observing users. Tools such as Selenium [7] record and playback scripts through the web and are popular in helping to automate testing for web sites. ABD adds in the ability to share these scripts and experience them using the interface used by the person who recorded the script. Expressing procedural knowledge in order to assist a user who is currently working to complete a task is an important issue for interactive help systems. Stencilbased tutorials demonstrates a variety of useful mechanisms, such as blurring all items except for those which are relevant to the current task and adding useful contextual information within the application with sticky notes [20]. To help convey what a running demonstration is doing, ABD includes a display that indicates what action the user performed to achieve the displayed results (Figure 4). Representing procedural knowledge in a usable way is also a difficult challenge. Keyword commands is one method, which uses simple pseudo-natural language description to refer to interface elements and the operations to be applied to them [22]. This is similar to the sloppy language used by CoScripter to describe webbased activity [21]. The benefit of using pseudo-natural language is that it can help users understand what is happening. ABD records its commands as sequences of user input and displays both the keyboard shortcut and a short description to make these descriptions more understandable. ABD enables a iterative debugging cycle, whereby developers can improve their content and replay the demonstration to test it. TrailBlazer lets blind web users record and replay web tasks [14]. TrailBlazer uses the same scripts as CoScripter [21] but is it targeted at non-visual use. The ABD extension to WebAnywhere makes trail recording and playback similar to those exposed by TrailBlazer easily available without installing new software, and exposes a new use for it - as communication path to developers.

# 1. 2. 3. 4. 5. 6. 7.

Command goto ctrl t tab tab tab tab tab

http://www.nytimes.com/ next table next focusable element next focusable element next focusable element next focusable element next focusable element

Figure 3. An example ABD recording. This recording first visit the New York Times homepage, then skips to the first table in the page, and finally tabs through focusable elements five times. Keyboard shortcuts are accompanied by short descriptions to aid developers who may not be accustomed to access technology. Figure 2. The WAVE accessibility evaluation tool showing output for analysis of a web page.

TrailBlazer scripts are shareable, but a recipient of a TrailBlazer script needs to install TrailBlazer for playback. Because ABD scripts are recorded as user input, users can record actions instead of recording elements. This means that a user can record themselves skipping through a page by headings even if a page contains no headings. If the page is fixed, then these commands will start to work. Importantly, developers don’t need to install anything to get a sense of how their pages appear to a blind person, they can just follow a link. Although the base interface of ABD is similar to tools like TrailBlazer and Selenium, the addition of an easily shareable URL that points directly to access technology that developers can use to further evaluate their sites help lower the overhead to this approach. ACCESSIBILITY BY DEMONSTRATION

The ABD system presented here is implemented as a component of the open source WebAnywhere web application [9]. It adds the ability to record and play back sequences of actions, allowing users to demonstrate the problems that they experience and then share those descriptions with others. WebAnywhere is attractive for use with ABD for the following two reasons: (i) it is an open platform that provides an API for accessing the actions that users perform using it, and (ii) as a web application that requires no software to run, it makes sharing demonstrations recorded using it straightforward. Demonstrating Problems

The interface of ABD allows blind users to record themselves using the web and capture the accessibility problems that they experience as a shareable recording. These recordings are stored on a remote server and accessing a single parameterized URL that loads not only the recording but also the access technology used by the person who recorded the script. Because no software needs to be installed, the ABD approach a low barrier method to get developers to use targeted screening techniques on content identified by disabled users.

For example, the link www..edu/X829DL loads a university homepage and navigates it through it using the CTRL+H shortcut, which skips from heading to heading announcing each as they are visited. Were no headings available on this page, the recording would instead cause WebAnywhere to announce “no heading.” The developer could then try to fix this page, and load it again using the provided URL to test his fixes. The ABD approach always loads the current version of a web page, so recordings reflect how access technology would interact with the current version of the page, allowing developers to iterate over designs. The idea of recording and sharing routes as parameterized URLs was inspired by the USA Track and Field’s “Map it.” On this web site, people can draw a running route on a map, save it to a centralized repository, and then share it with others as a parameterized URL [2]. Capturing all necessary information as a URL makes these recordings quite flexible and easy to share, and enables anyone to play back ABD recordings. Recordings can be sent in an email or instant message, included on social networking sites like Twitter or Facebook, or even printed as part of paper correspondence. Retroactive Trail Recording

Instead of starting out intending to create a demonstration of an accessibility problem, users may also make recordings retroactively. Retroactive recording builds on the idea of smart bookmarks [19], which were designed to let users bookmark dynamic web pages that may not have a static URL associated with them [19]. The idea is to keep a short history of the actions that users complete on the web and record this history as a macro when a user chooses to bookmark the current location. Bookmarks are saved as a sequence of actions that reproduce the browser’s current state, returning users to their current position even if users cannot browse to the page directly. To implement retroactive recording, the ABD component to WebAnywhere keeps a list of all actions that are taken after a new page loads. When the user chooses retroactive recording, the ABD component traverses back-

ward as far as necessary to reach a known state on the page and saves all steps after that point as part of the recording. A known state is any state in which the system knows precisely which element the user is on, such as a WebAnywhere interface component or the first or last node in the page). This is necessary because ABD records only the keyboard shortcuts used and not a direct address to elements, such as an XPATH. Recording shortcut keys, in turn, is critical for recording many common accessibility problems - for instance, the easiest way to demonstrate a lack of headings on a web page is to try to use CTRL+H to navigate between heading tags. Retroactive recording similarly enables ABD users to record an accessibility problem when they experience it by pressing a single shortcut key. Users simply browse the web as they usually would and press the retroactive record shortcut key when they encounter an accessibility problem. This may help novice users who may be unable to reproduce the steps necessary to get into a problematic state from the start, or experienced users who want to quickly record a problem and not bother with starting with a fresh demonstration. Again, the goal is to make creating and sharing recordings of accessibility problems easy and efficient - retroactive recording requires only one extra key stroke to record an entire problematic interaction. User Experience for Developers

The user experience for developers is as follows. First, the developer gains access to an ABD URL, and clicks on it. The URL opens WebAnywhere and the script provided by the user immediately begins to run. Generally, this script will take the developer first to the page on which the problem was demonstrated, and then proceed with a number of actions that the user recorded on that page. Because understanding how access technology works can be confusing, ABD helps the developer by highlighting the content that is being read instead of relying only on the aural speech that a blind person would hear. This may help the developer follow along through the page visually. Secondly, ABD provides a view of the commands that the script is executing (and which the user had previously executed) from which the developer can follow along (Figure 4). This view helps the developer understand not only what commands are being executed, but also easily see the history of commands that have been executed. Because the keyboard shortcuts are displayed, this view may also help the developer independently navigate through the page to check problems later. After making changes to the page to address the problem represented by the recording, the developer can load the same script again and, in most cases, see if their changes have fixed the problem. The main limitation is if they have changed the page so dramatically that the script no longer applies or no longer correctly demonstrates the accessibility problem. In many situ-

Figure 4. A graphical illustration of what WebAnywhere is doing is provided by the ABD component to help users who are not accustomed to using a screen reader understand what is happening, helping to address one of the primary concerns with screening - access technology can be too confusing for a novice to use as part of evaluation.

ations, however, being able to reload new content with a previously-created script allows iterative development of content. EVALUATION

Our evaluation considered whether the addition of task descriptions and the ability to play them back would help web designers improve on the accessibility of their websites. We did not study the ability of blind web users to record these scripts, although a broad study of both the benefit and ease of use of task recording is underway. Instead we focused measuring any benefit WebAnywhere-ABD could provide to designers when evaluating and correcting accessibility problems on websites. For this study, we focused on 4 common types of accessibility problems: • Alternative Text Alternative text is a common means of providing textual meaning for visuals on a web page. This is especially important for blind users as there are no screen readers that can describe visual elements on a page. Designers include “alt” attributes as a part of image tags so that screen readers can read the text in place of the picture. This technique is also useful when web browsers have problems displaying an image. Furthermore, when search engines index a website the alternative text is stored. While alternative text is very useful for these reasons, they are frequently forgotten, thus concealing content from blind users. • Heading Usage Header (i.e. h1, h2, etc) tags are generally used to provide visitors with a sense of structure and hierarchy. Specifically they signify the beginning of a new section and allow users to quickly to find new sections. While for sighted users, these tags provide visual cues as the fonts size and weight vary, blind users can use them to quickly navigate through a page. However, particularly when websites are created using WYSIWYG editors, it may not be clear to the designer if these tags are being used. In some

case a designer will use tags to change the visual style without using a header tag. • Reading Order Screen readers most commonly path through a page linearly by analyzing the HTML source. However, with the addition of advanced CSS techniques, in particular absolute positioning, in some cases how a page is rendered in a web browser may not be how it exists in the source. Again, some WYSIWYG editors use this type of position to provide better flexibility to designers for placing objects. However, if a designer does not pay attention to the order they are inserting content, the rendered view can be vastly different than the reading order in the HTML source. • Tab Order Using the tab key is a common way for users to navigate through a page. The “tabindex” attribute can be used to specify the order page elements are highlighted. This allows designers to create logical order on a website. Problems arise when using some visual tools for creating web pages. Tab order is often determined by the order in which elements are added, not their their textual placement or grouping.

Figure 5. This page uses tables for layout which causes an incorrect reading order when linearized.

This evaluation study was conducted on Amazon’s Mechanical Turk. Participants were paid $2.50 USD for successfully completing each of the 4 tasks for a total of $10. We recruited participants from personal contacts who have no affiliation with this research project. We then controlled for a basic level of skill by asking participants to solve the following problem: Enter the HTML and CSS necessary to produce a paragraph with a 3 pixel red border that contains the text “Hello!!” Tasks

Keeping the 4 common accessibility issues in mind, we designed a set of 4 tasks, 3 of which cover specific issues and 1 that cover all 4 issues. These 4 tasks are described below.

Figure 6. This page uses incorrect heading tags which can make navigation confusing.

would need to locate all the tags and change them to the related header tag. • Tab Order Task -

• Tables Task - The page for this task contains a table with States and their associated fruit (Figure 5). Each picture has adequate alternative text, however because the of the ordering created by the tables, the States are read independent from the fruits, thus creating confusion. This problem can be fixed in a number of ways, all of which involve either moving the text or the picture next to each other in the source code. • Headings Task This task contained a page with a clear outline of headings (Figure 6). Visually each heading looked correct, because of the associated CSS style sheets, however certain ones were actually styled using tags instead of h1, h2, or h3. To fix this, participants

The page for this task included elements that were positioned using absolute positioning that resulted in their visual order not reflecting the order in the HTML source. To correct this, participants must rearrange the sections of the page to match a natural reading order. Some elements were also assigned tabindex values overriding the default tab order, causing tabbing through a web page to skip between different sections of the page without respect to visual semantics. The the ordering of the page is correct, then this can be solved by removing the tabindex values completely. Otherwise, the participant would need to change the values to reflect the visual flow of the page. • Combination Task -

Figure 8. An example of the editing field used by participants. Figure 7. This page contains all 4 accessibility issues.

Accessibility issues rarely exist on their own, so we designed a task that combined all 4 of the common issues we are interested in (Figure 7). All sections of this page were positioned using absolute positioning and randomly arranged in the source code. Many elements on the page had tabindex values that were randomly distributed. Images on the page were may or may not have alternative text. Headings were given a mix of correctly used heading tags and incorrect tags. For each of the 4 tasks, we also introduced 3 condition. For the first condition we asked participants to watch a demonstration of the site’s navigation using WebAnywhere-ABD. We also wanted to measure how WebAnywhere-ABD compared to other tools used by designers, so for another condition participants were asked to use WAVE (web accessibility evaluation tool) [8] for help. As a control condition, we did not provide a tool to assist but instead a link to the WCAG standards website [5]. For all conditions, participants were given a sentence describing the problem from a prospective of a non-technical user. For example, “I have difficulty navigating the page quickly” was provided for the Headings Task.

When they are done, we ask them for a description of what they perceive as the problem on the page. EVALUATION RESULTS

At the conclusion of this study, we had a total of 15 participants who completed 45 tasks. The participants consisted of 11 men and 4 women between the ages of 20 to 38 (Mean=28.6; S.D.=5.68). When asked to rate their own web design skills (1=not skilled and 7=very skilled) the all self-rated very highly (Mean=5.07; S.D.=0.73). Their self-ratings of accessibility experience (1=not experienced and 7=very experienced) was however much lower (Mean=3.93; S.D.=1.98). These demographics fit our ideal population very well. Those who are skilled web designers, but are less experienced with accessibility issues will be the most likely to benefit from this work. We began our analysis by tallying the number of errors fixed by each participant for each task, but found that were almost always able to correct all the similar problems on a page if they found one. For example, if a participant noticed that the headings were not correctly applied, they were able to find all of those errors and correct them. Thus we normalized these results and tallied the error types for each condition. This is presented in Table 1.

Procedure

When the participant accepts the Mechanical Turk HIT they are asked for some demographic information and to complete the qualifying question. Next they are given the sentence description of the problem that exists on the page, and then are presented with one of the 3 conditions outlined in the previous section. They are then asked to evaluate the page using the tool provided. Below this, is a text field that contains the source code of the page and are asked to make changes to that code (Figure 8). The participant can save what they have and view the effects of their changes in another window.

Due to the size of this sample it is difficult to draw conclusive results, however our data shows that there is evidence of a modestly significant improvement of WebAnywhere-ABD as compared to the control condition (t(24)=2.32 at p