GUI accessibility has been the topic of study for many years. ... application and any rendering agent can make direct calls to the AUI engine and to .... IOS Press.
PARALLEL USER INTERFACE RENDERING: ACCESSIBILITY FOR CUSTOM WIDGETS Kris Van Hees1*, Jan Engelen2 1*
Katholieke Universiteit Leuven, Department of Electrical Engineering, ESAT-SCD-DocArch, Kasteelpark Arenberg 10, B-3001 Heverlee, Belgium, 2 Katholieke Universiteit Leuven, Belgium,
ABSTRACT Parallel User Interface Rendering (PUIR) is a novel approach to providing non-visual access to graphical user interfaces, leveraging abstract user interface descriptions. Rather than deriving the alternative rendering as a second generation instance, PUIR provides non-visual renderings as first generation presentations alongside the graphical rendering, thereby preserving the interaction semantics across presentations. Custom widgets are supported through dynamic loading, requiring code to implement the user interaction, and code for each rendering agent to provide the presentation. Absence of an implementation is used to identify unknown widgets, ensuring that users are aware of unknown widgets and that accidental interaction is avoided. KEYWORDS Accessibility, Design-for-All, Universal Access, Multi-Modal Interfaces, HCI INTRODUCTION Graphical user interfaces (GUIs) are emerging in all aspects of life, well beyond the more traditional general purpose computer system. This trend poses a complication for blind users due to the inherent visual interaction model that is associated with these interfaces. Advances in computer science and technology have also led to a higher degree of flexibility and complexity in GUIs, introducing additional challenges as a result of the more limited improvements that have been made to assistive technologies (ATs). Dedicated devices are commonly equipped with a GUI based on a single toolkit, enabling AT providers to fine tune their software for a specific context. General purpose computer system, especially in Unix-based environments, tend to allow a variety of graphical tool kits to be in use at the same time. Such GUIs require a more sophisticated approach, unifying the underlying tool kits to the fullest extend possible. Abstracting the user interface (UI) offers a solution where user interaction semantics can be unified across all tool kits while retaining maximum flexibility in rendering for a multitude of output modalities, thereby providing nonvisual access as an integral component of the UI environment [1, 2, 3, 4].
The use of custom widgets is often a major obstacle for assistive technologies that traditionally depend on deriving their information from the GUI implementation and/or rendering. A common downfall of these techniques is that multiple uses of the same custom widget may each require context-based scripting and/or heuristics. In the PUIR framework, a custom widget must be defined in terms of its user interaction semantics. This ensures that assistive technologies are aware of the custom widget and the interaction is allows. PUIR also ensures that once support for a custom widget is available, any occurrence of the widget will be supported, even across multiple applications. The remainder of this paper first presents related work on GUI accessibility and abstract user interfaces (AUIs). This is followed by a description of PUIR. Section four discusses the handling of widgets, with focus on custom widgets. Section five concludes this paper, offering a direction for future work. RELATED WORK GUI accessibility has been the topic of study for many years. Early research formulated core design issues that are common to the problem area of non-visual access to GUIs. Mynatt and Weber [5] introduced the first four core issues, and Gunzenhäuser and Weber [6] identified a fifth issue, along with a general discussion of common approaches towards GUI accessibility. The HAWK toolkit [7, 8] provides alternative interaction metaphors for non-visual user interfaces, providing interaction objects and techniques that are tailored for non-visual access. The AVANTI project [9] uses this toolkit successfully for its context-based runtime user interface adaptation mechanism. The obstacles that blind users face when trying to operate graphical user interaction models were researches by Barnicle [10], Pontelli et al [11], and Theofanos [12]. Their findings support the observation that application user interfaces are very similar to World Wide Web forms. The UsiXML [13] project at the Belgian Laboratory of Computer-Human Interaction (BCHI) at the Université Catholique de Louvain builds upon these findings. The ability to abstract the user interface lies at the core of the research presented here. The concept of parallel user interface rendering as a way to providing non-visual access to GUIs emerged from doctoral research into the use of abstract user interface definitions as basis for assistive technologies [1, 2, 3]. Kost [14] builds on the value of abstraction as well, using it to provide alternative presentations of the same user interface, albeit not in parallel. This PhD thesis mentions multiple instances of the interface as an area of interest that faces unresolved issues. PARALLEL USER INTERFACE RENDERING The parallel user interface rendering technique is based on the advantages of abstract UI descriptions. Rather than constructing the UI by means of function calls into a specific widget toolkit, application user interface can be defined in abstract form. The abstract UI description can be constructed programmatically using UI design tools that generate the UI in a standardised format like UsiXML [13] or GIML [14].
A proof-of-concept implementation of the PUIR framework as a generic UI toolkit has been developed in Java. It currently provides 25 fairly generic UI elements that are commonly used in applications, such as buttons, text input fields, labels, editable combo-boxes, mutually exclusive toggle lists, ... The rest of this section provides further details on the major components. The AUI engine As shown in Figure 1, this description is interpreted by the AUI engine. This component implements the user interaction semantics for all widgets, and focus management. Interaction with a widget (be it by the user or by the application) results in dispatched events that provide notification to the rendering agents. This ensures consistency between the presentations. Events are also commonly dispatched to the application as a result of user interaction. The application and any rendering agent can make direct calls to the AUI engine and to widgets in the UI. This design ensures that the AUI engine has no knowledge about the implementation details of either the application or the rendering agents.
Figure 1 – Schematic overview of PUIR
GUI rendering agent The proof-of-concept implementation provides a Java Swing based in-process rendering agent. Due to the design of widgets in the Swing toolkit, implementing a rendering agent that does not handle user interaction semantics turned out to be quite a challenge. Various aspects of widget functionality in Swing have been bypassed in order to ensure that the only the AUI engine handles interaction, and that the rendering agent will show the results of the interaction when notification by means of events is received. A side effect of this necessary change is that a few of the widgets now operate slightly differently from the original Java Swing version.
Although user interaction is deferred to the AUI engine, some processing on UI manipulation using a pointer device must be done in the GUI rendering agent. Pointer device operations typically relate to the position of the pointer device, which only makes sense within the context of the graphical presentation. Operations that have no semantic significance (such as most pointer movement operations) are ignored, while the remainder are mapped to user interaction semantics for AUI engine processing. Non-visual UI rendering agent As an alternative to the GUI rendering, the proof-of-concept implementation supports remote rendering agents, i.e. external processes that communicate with the PUIR framework by means of a well defined API across a message bus. Mutual discovery, event passing, and request/response dialogues are supported. A basic speech output presentation has been implemented as an example of non-visual rendering. Use of proxy objects on the side of the rendering agent allows for transparency and caching. Input processing The proof-of-concept implementation supports input from three modalities: • Keyboard: This type of input deals primarily with textual input. Similar to most user interface management systems, PUIR implements various key combinations that perform operations on either the UI level or the widget level. This includes navigation throughout the user interface, text selections and manipulation, short cuts, and accelerator keys. The proof-of-concept implementation processes keyboard input directly at the AUI engine level. This is a simplification in the design that does not actually impact the expected functionality. • Mouse: Mouse input is processed by a graphical rendering agent because mouse operations (like any pointer device operation) relate to the position of the mouse. All events generated due to mouse interaction are interpreted in the context of the position of the mouse, and where appropriate semantic events are dispatched to the AUI engine. • Braille keyboard: This input device combines coded textual input with a limited form of cursor addressable input (positional input). Coded input is translated into its text equivalent and presented for processing as if it was entered on a regular keyboard. Positional input by means of cell selection is translated into its equivalent semantic PUIR event, and dispatched to the AUI engine. In summary, regardless of the input device or the delivery path, input events are presented to the AUI engine for processing. Any results from the user interaction results in events that are dispatched to the registered rendering agents (and often the application as well). PUIR WIDGETS A widget in PUIR is implemented as two separate entities: • The abstract widget. This component implements the user interaction semantics, and it encapsulates the widget data elements. If the component can contain other widgets (i.e. it is a container), it will also implement functionality to instantiate its content based on
•
(part of) the abstract UI description. This abstract widget component operates entirely within the AUI engine. The widget presentation. This component implements the rendering of the widget within the context of a specific rendering agent. Ideally, a widget will have as many presentations as there are rendering agents.
The proof-of-concept implementation of PUIR provides various standard widgets as sets of classes. One class for each abstract widget, and one class for each specific widget presentation. Inheritance is used where possible to model a hierarchy of user interaction elements. The remainder of this section will describe the processing of the user interface in the PUIR framework within the context of an application. Designing a user interface for PUIR During the development of an application, developers and/or designers will typically use a design tool to define the “Look and Feel” of the UI. The tool provides the user with a selection of widgets that can be used to define the UI. Rendering specific information can be encoded along with the design, and the result is an AUI description file in a standardised format. Runtime construction of the user interface When an application is started, all or part of the user interface will be instantiated based on the AUI description. For every widget that is encountered during instantiation, an implementing abstract widget class must be found and loaded. If the widget is not found within the collection of standard widgets in the PUIR framework, a search is performed in alternative locations including the class libraries of the application itself. If the abstract widget class cannot be found, the application UI cannot be instantiated and the application will not be able to execute. This is equivalent with existing tool kits for cases where widgets are used for which no implementing library can be found at runtime. It is important to note that the existence of the abstract widget class is sufficient at this point. It defines all user interaction and it encapsulates all widget data. Rendering the user interface The presentation of the UI is handled by rendering agents. Each widget in the AUI corresponds to a presentation class that is to be instantiated by the rendering agent, and is subject to an equivalent search-and-load procedure as used by the AUI engine. If the presentation class can be found, the widget can be rendered correctly. If the presentation class cannot be found, the rendering agent can take appropriate action to alert the user that a specific widget is not supported. It can then also ensure that the user cannot interact with the widget, thereby avoiding accidental operations. The PUIR framework ensures that all rendering agents operate as first generation presentations, with direct access to the abstract widgets in the AUI engine. Even when a presentation class cannot be found for a widget for a specific rendering agent, important information about the
widget is available. This is important for one special class of custom widgets: cosmetic custom widgets. Cosmetic custom widgets One common use of custom widgets relates to presentation only: cosmetic custom widgets. A good example can be seen in Figure 2.
Figure 2 – Example of cosmetic custom widgets
From left to right, this UI contains: • Four regular buttons. • A VU-meter style widget. This is a cosmetic custom widget that is equivalent to a regular label with a numeric value. No user interaction is supported. • Four knobs. These are cosmetic custom widgets that are equivalent to a slider widget, allowing user interaction to select a value within a certain discrete range. • Four indicator lights. These are cosmetic custom widgets that are equivalent to a regular label. No user interaction is supported. • Four regular buttons. The PUIR framework supports specifying equivalence between custom widgets and core widgets in the AUI description. This allows rendering agents to provide a default presentation based on the equivalent core widget, if no custom presentation class is available. Future enhancements may expand upon this by providing a generic presentation based on other abstract widget information. CONCLUSIONS Parallel user interface rendering is a very powerful technique in support of Design-for-All principles, building upon a solid base of research and development of abstract user interfaces. It provides a framework where alternative renderings of the UI operate at the same level as the native rendering rather than being a derivative. This technique ensures coherence between the presentations across modalities because of a consistent underlying user interaction model, thereby supporting close collaboration between users with varying abilities. Custom widgets in mainstream GUI tool kits have been a common problem with existing assistive technology solutions because of their very nature. The underlying design of PUIR ensures that all widgets are known to rendering agents. Even if no specific alternative presentation is available for a custom widget, assistive technologies will be aware of the existence of the widget. User interaction is often still possible, and the user is at all times
aware of any limitations. Once implementations for alternative presentations are available, the custom widget will operate as any other widget in the PUIR framework. The PUIR framework was developed with the immediate goal of providing non-visual access to GUIs, but its contribution to the field of accessibility goes above and beyond. The generic approach behind PUIR lends itself well to developing alternative rendering agents in support of a variety of disability groups. Accessible remote operation of applications is quite possible as a future enhancement. PUIR can also benefit automated application testing by providing a means to interact with the application without any dependency on a specific UI rendering. ACKNOWLEDGEMENT The research and development presented here is part of work towards a doctoral degree at the Katholieke Universiteit Leuven (Belgium), under the supervision of Prof. Dr. ir. Jan Engelen. REFERENCES [1] Van Hees, K., Engelen, J. (2005). Abstract UIs as a long-term solution for non-visual access to GUIs. In Proceedings of the 3rd International Conference on Universal Access in Human-Computer Interaction. [2] Van Hees, K., Engelen, J. (2005). Abstracting the graphical user interface for non-visual access. In Pruski, A., Knops, H., eds.: Assistive Technology from virtually to reality. 8th European conference for the Advancement of Assistive Technology in Europe, pp. 230-245. IOS Press. [3] Van Hees, K., Engelen, J. (2006). Non-visual access to GUIs: Leveraging abstract user interfaces. In Miesenberger, K., Klaus, J., Zagler, W. L., Karshmer, A. I., eds.: Computers Helping People with Special Needs. 10th International Conference, ICCHP 2006, Linz, Austria, July 11-13, 2006. Proceedings, pp. 1063-1070. Springer. [4] Van Hees, K., Engelen, J. (2010). PUIR: Parallel user interface rendering. In Miesenberger, K., Klaus, J., Zagler, W. L., Karshmer, A. I., eds.: Computers Helping People with Special Needs. 12th International Conference, ICCHP 2010, Vienna, Austria, July 14-16, 2010. Proceedings, Part I., pp. 200-207. Springer. [5] Mynatt, E. D., Weber, G. (1994). Nonvisual presentation of graphical user interfaces: contrasting two approaches. In CHI '94: Proceedings of the SIGCHI conference on Human Factors in computing systems, pp. 166-172. ACM Press. [6] Gunzenhäuser, R., Weber, G. (1994). Graphical user interfaces for blind people. In 13th World Computer Congress, pp. 450-457. [7] Savidis, A., Stephanidis, C. (1995). Building non-visual interaction through the development of the rooms metaphor. In CHI '95: Conference companion on Human Factors in computing systems, pp. 244-245. ACM Press. [8] Savidis, A., Stergiou, A., Stephanidis, C. (1997). Generic containers for Metaphor Fusion in Non-Visual Interaction: the HAWK interface toolkit. In Processings of the Interfaces '97 Conference, pp. 194-196. [9] Stephanidis, C., Savidis, A. (2001). Universal Access in the Information Society: Methods, Tools and Interaction Technologies. In Universal Access in the Information Society, pp. 40-55.
[10] Barnicle, K. (2000). Usability testing with screen reading technology in a Windows environment. In CUU '00: Proceedings on the 2000 conference on Universal Usability, pp. 102-109. ACM Press. [11] Pontelli, E., Gillan, D., Xiong, W., Saad, E., Gupta, G., Karshmer, A. I. (2002). Navigation of HTML tables, frames, and XML fragments. In Assets '02: Proceedings of the fifth international ACM conference on Assistive Technologies, pp. 25-32. ACM Press. [12] Theofanos, M. F., Redish, J. (2003) Bridging the gap: between accessibility and usability. In Interactions, 10(6), pp. 36-51. [13] Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., Florins, M. (2004). UsiXML: A user interface description language supporting multiple levels of independence. In Lauf, M., ed.: Proceedings of Workshop on Device Independent Web Engineering DIWE '04. [14] Kost, S. (2006). Dynamically generated multi-modal application interfaces. PhD thesis, Technische Universität Dresden, Dresden, Germany.
