Multi-devices ''Multiple'' user interfaces: development ... - CiteSeerX

The Journal of Systems and Software 73 (2004) 287–300 www.elsevier.com/locate/jss

Multi-devices ‘‘Multiple’’ user interfaces: development models and research opportunities Ahmed Seffah a

a,*

, Peter Forbrig b, Homa Javahery

a

Human-Centered Software Engineering Group, Department of Computer Science, Concordia University, 1455 de Maisonneuve Blvd. West, Montreal, Que., Canada H3G 1M8 b Software Engineering Group, Department of Computer Science, D-18051 Rostock University, Albert-Einstein-Str 21, Germany Received 12 November 2002; received in revised form 28 August 2003; accepted 14 September 2003 Available online 25 December 2003

Abstract Today, Internet-based appliances can allow a user to interact with the server-side services and information using different kinds of computing platforms including traditional office desktops, palmtops, as well as a large variety of wireless devices including mobile telephones, Personal Digital Assistants, and Pocket Computers. This technological context imposes new challenges in user interface software engineering, as it must run on different computing platforms accommodating the capabilities of various devices and the different contexts of use. Challenges are triggered also because of the universal access requirements for a diversity of users. The existing approaches of designing a single user interface using one computing platform do not adequately address the challenges of diversity, cross-platform consistency, universal accessibility and integration. Therefore, there is an urgent need for a new integrative framework for modeling, designing and evaluating multi-device user interfaces for the emerging generation of interactive systems. This paper begins by describing a set of constraints and characteristics intrinsic to multi-device user interfaces, and then by examining the impacts of these constraints on the specification, design and validation processes. Then, it discusses the research opportunities in important topics relevant to multi-device user interface development, including task and model-based, pattern-driven and device-independent development. We will highlight how research in these topics can contribute to the emergence of an integrative framework for Multiple-User Interface design and validation. 2003 Elsevier Inc. All rights reserved.

1. Introduction Recent years have seen the emergence of a wide variety of computer devices including mobile telephones, Personal Digital Assistants (PDAs) and pocket PCs. Many existing devices are now being introduced as an alternative to traditional computers. Internet-enabled television (WebTV) and electronic whiteboards powered by desktop machines are among the many examples. Other examples include Web applications where users interact with the content, interactive television controlled by hand-held remotes, mobile phones with small * Corresponding author. Tel.: +1-514-848-3024; fax: +1-514-8482830. E-mail addresses: seff[email protected] (A. Seffah), pforbrig@ informatik.uni-rostock.de (P. Forbrig). URLs: http://hci.cs.concordia.ca, http://wwwswt.informatik.unirostock.de.

0164-1212/$ - see front matter 2003 Elsevier Inc. All rights reserved. doi:10.1016/j.jss.2003.09.017

screens and limited keypads, and PDAs with limited screens and pen-based input. Multiple-User Interface (MUI) refers to an interactive system that provides access to information and services using different computing platforms. A computing platform is a combination of computer hardware, an operating system and a UI toolkit. Computing platforms include the large variety of traditional office desktops, laptops, palmtops, mobile telephones, PDAs, as well as the new devices for interactive television. Conceptually speaking, a MUI provides multiple views of the same information on these different platforms and coordinates the services provided to a single user or a group of users. Each view should take into account the specific capabilities and constraints of the device while maintaining cross-platform consistency and universal usability. The following is a scenario that further clarifies the MUI concept and its use:

288

A. Seffah et al. / The Journal of Systems and Software 73 (2004) 287–300

You are riding in a car with your colleague who is driving. Suddenly, your mobile phone comes on, asking if you can take a video conference call from your team in Canada to discuss a project on which you are working. You take the call and as you communicate with them via the integrated car video system, you find out that they need one of the spreadsheets you have saved on your laptop, which is in the trunk of the car. Using your wireless technology, you are able to transfer the file to your PDA, and then send it to your team. A few minutes later your team will receive and open your spreadsheet, using an office desktop and you can start discussing options with them once again via your interactive television from your comfortable home. This scenario is based on (Ghani, 2001). Olsen et al. (2000), Johnson (1998) and Brewster et al. (1998) highlight the design challenges associated with the small screen size of hand-held devices. In comparison to desktop computers, hand-held devices always suffer from a lack of screen real estate, so new metaphors of interaction have to be invented for such devices. Many assumptions that have been held up to now about classical stationary applications are no longer valid for hand-held devices due to the wide range of possibilities currently available. This is because handheld devices have constantly updated capabilities, exploit additional features of novel generations of networks, and are often enabled for mobile users with varying profiles. Even if software tools for developing a large variety of user interfaces on each computing platform are already available or will be in the near future (Myers et al., 2000), the following are the major questions that should be addressed by both academia researchers and industry practitioners: 1. How can the user interface (UI) be adapted to the diversity of computing platforms that exist today? How can the high level of interactivity of the traditional office desktop be maintained or adapted for small devices without a keyboard and mouse? How can we make it possible for users to customize a device they are using in terms of accessibility to information and services? When a single device is customized, how can such customization be reflected on all other devices that the user can use? 2. What kinds of methods are needed for designing for diverse users? Are the design techniques for UI modeling suitable for addressing the problems of diversity, cross-platform consistency, and universal accessibility? 3. Should we strive for uniformity in the services offered, dialogue styles and presentation formats, or should we adapt the interfaces to the constraints

and capabilities of each device and/or each context of use? When designing MUIs, what is the best way to take the constraints related to each type of device into account while assuring the maintainability and cross-platform consistency of this interface mosaic? 4. How can we implement and validate a MUI for d devices without writing p programs, training an army of developers in l languages and UI toolkits, and maintaining l p architectural models for describing the same UI? Are the content mark-up languages adequate for device-independent authoring? The aim of this paper is to clarify these challenging questions. We begin by describing a set of characteristics intrinsic to MUIs, and then examine the relevant background and related work, in particular in the area of adaptable and context-aware user interfaces. This introduction will be followed by a deep analysis of the empirical and user-oriented studies related to MUI. We discuss the development opportunities in important topics relevant to MUI development including modelbased, user interface pattern-driven, and device-independent development models. The discussion about the empirical aspects and the development models is highly speculative and will raise far more fundamental research questions than provide answers. Furthermore, this is a selective list of topics, and not exhaustive. Our goal is to give researchers a glimpse of the most important problems surrounding potential MUI development models.

2. Multiple-User Interface: characteristics and background work Remarkably, although research on multiple views and multi-device or multi-user interaction can be traced back to the early 1980s, there are relatively few examples of successful implementations (Grudin, 1994). Perhaps the main cause of this poor success rate is the difficulty of integrating the overwhelming number of technological, psychological, and sociological factors that affect MUI usability into a single unified design. In the evolution of user interfaces, a multi-user interface has been introduced to support groups of devices and people cooperating through the computer medium (Grudin, 1994). A single user in the context of a MUI is what a group of users is for a multi-user interface. The user is asynchronously collaborating with himself/herself. Even if the user is physically the same person, he/ she can have different characteristics while working with different devices. For example, a mobile user is continuously in a rush, impatient, and unable to wait (Nielsen and Ramsay, 2000). This user needs immediate, quick, short and concise feedback. The same user, while in the office, can afford to wait a few seconds more for further details and explanations.


2.1. Multiple-user interface––a definition We introduced the concept of MUI during the HCIIHM workshop (Seffah and Radhakrishan, 2001). The term ‘‘multiple-user interface’’ is also being used by others (McGrenere et al., 2002; Vanderdonckt and Oger, 2001) with other, although related, meanings. A multiple-user interface can be described as follows: • Allows a single or a group of users to interact with the server-side services and information using different interaction/UI styles. For example, our civil engineer uses pen and gestures on a PDA, function keys or single characters on a mobile phone, and a mouse for a desktop computer. • Allows an individual or a group to achieve a sequence of interrelated tasks using different devices. • Presents features and information that behave the same across platforms, even though each platform/ device has its specific look-and-feel. • Feels like a variation of a single interface, for different devices with the same capabilities. Fig. 1 shows a MUI to the same Internet financial management system. This interface consists of three views: Desktop, PDA with keyboard, and mobile phone. Ideally, from the user perspective, these three different interfaces should be as similar as possible. However, this is not realistic because of the capabilities and constraints imposed by each platform. Therefore, the MUI can be seen as a compromise between customized and platform-dependent UIs. The effort required to keep all interfaces consistent and to maintain them increases linearly with the number of interfaces, as the functionality of the underlying financial system is expanding. The MUI provides both multiple views of the same model and coordinates the user actions gathered from different devices/computers. The model may reside in a single information repository, or may be distributed

289

among independent systems. Each view can be seen as a complete UI for a specific platform (hardware, operating system, and UI toolkit). All these mosaic interfaces form a unique and single MUI. The list of features, the interaction style, and the displayed information/feedback can vary from one platform to another. 2.2. Background work: adaptive and context-aware user interfaces The MUI can benefit from the considerable number of studies done in the area of context-aware (or sensitive) user interfaces. This area is still an active research topic, with many emerging models such as plastic user interfaces (Thevenin and Coutaz, 1999) and the moderator model (Vanderdonckt and Oger, 2001). In a recent essay, Winograd compared different architectures for context of use (Winograd, 2001). As characterized in the previous section, a MUI is a context-sensitive UI. However, this does not mean that a MUI should adapt itself magically at runtime to the context of use (and in particular to the platform capabilities and constraints). The range of strategies for adaptation is delimited by two extremes. Interface adaptation can happen at the factory, that is, developers produce several versions of an application tailored according to different criteria. Tailoring can also be done at the user’s side, for instance, by system administrators or experienced users. In the other extreme, individual users might tailor the interfaces themselves, or the interface could adapt on its own by analyzing the user’s context of use. The consensus from our workshop was that the adaptation of a MUI should be investigated at different steps of the development and deployment lifecycle (Seffah and Radhakrishan, 2001): • User customization after deployment or adaptable user interfaces. Tailoring operations are the entire responsibility of the user. While this ‘‘laissez-faire’’

Fig. 1. An example of a MUI.

290


approach avoids the need for system support, it lacks a central arbitrator to resolve incompatible and inconsistent preferences between devices. The arbitrator should have the ability to make global changes (cross-platform changes) based on local adaptations. This makes MUIs more difficult to write, and the adaptation fails to repay the development cost of support. • Automatic adaptation at runtime or adaptive user interfaces. The idea is to write one UI implementation that magically adapts itself at runtime to any computing platform and context of use. The drawback of this strategy is that there may be situations where adaptation performed by the system is inadequate or even counterproductive. • Just-in-time customization during development or deployment. Developers can use a high-level language to implement an abstract and device-independent UI model. Then, they can, using a rendering tool, generate the code for a specific platform. The User Interface Markup Language, UIML (Abrams and Phanouriou, 1999), and the eXtensible Interface Markup Language, XIML (Eisenstein et al., 2001), aim to support such an approach. They will be discussed in Section 4.3. • Customization during design and specification. This needs an appropriate design methodology and multi-platform terminology to properly build a task model of a MUI. This model may be expressed in one or more notations. Tailoring operations can happen at the abstract interface specification where the dialogue gets modified, for example to shortcut certain steps, to rearrange the order for performing steps, etc. Context-aware UI development refers to the ability to tailor and optimize an interface according to the context in which it is used. Context-aware computing as mentioned by Dey and Abowd (2000) refers to the ‘‘ability of computing devices to detect and sense, interpret and respond to, aspects of a user’s local environment and the computing devices themselves’’. Efforts have already begun to develop frameworks that support the building of context-aware applications. The Context Toolkit (Dey and Abowd, 2000) is an infrastructure that supports the rapid development of context-aware services, assuming the sensing of a perfect context. This framework’s architecture enables the applications to obtain the context they require without knowledge about how the context was sensed. The Context Toolkit consists of context widgets that implicitly sense context, aggregators that collect related context, interpreters that convert between context types and interpret the context, applications that use context and a communication infrastructure that delivers context to these distributed components. It makes it easy to add the use of context or implicit input to existing applications.

3. Usability studies and empirical validation The concept of MUI is relatively new. So far, little research has been done on its cost-effectiveness. Are MUIs really useful, used and usable? This section answers these questions with a special emphasis on MUI usability. McGrenere et al. (2002) studied the use of having multiple interfaces for a new version of an application; for example, MSWord 2000 could include the MSWord 97 interface. Their conclusions were as follows: ‘‘By allowing users to continue to work in the old interface while also accessing the new interface, they would be able to transition at a self-directed pace. Similarly, multiple interfaces might be used to provide a competitor’s interface in the hopes of attracting new customers. For example, MSWord could offer the full interface of a word processor such as Word Perfect (with single button access to switch between the two), in order to support users gradually transitioning to the Microsoft product.’’ Compared to a user interface designed for a specific platform, the two following dimensions complicate the design evaluation of a universal user interface which is the case MUI (Hochheiser and Shneiderman, 2001): • Technological variety. Technological variety implies supporting a broad range of hardware, software, and network access. The first challenge in adaptation is to deal with the pace of change in technology and the variety of equipment that users employ. The stabilizing forces of standard hardware, operating systems, network protocols, file formats, and user interfaces are undermined by the rapid pace of technological change. This variety results also in computing devices, (e.g. mobile phones) which exhibit drastically different capabilities. For example, PDAs use a pen-based input mechanism and have average screen sizes in the range of 3 inches. On the other hand, the typical PC uses a full sized keyboard, a mouse, and has an average screen size of 17 inches. Coping with such drastic variations implies much more than mere layout changes. Pen-based input mechanisms are magnitudes slower than traditional keyboards and are therefore inappropriate for applications such as word-processing that require intensive user input. Similarly, the small screens available on many PDAs only provide coarse graphical capabilities and are not suited for photo editing applications. • Diversity of Users. Accommodating users with different skills, knowledge, age, gender, disabilities, disabling conditions (mobility, sunlight, noise), literacy, culture, income, etc., is the source of a further complication (Stephanidis, 2002). For example, while walking down the street, one user may use a mobile phone’s Internet browser to look up a stock quote. However, it is highly unlikely that this same user re-


views the latest changes made to a document using the same device. Rather, it would seem more logical and definitely more practical to use a full size computer for this task. It would therefore seem that the context of use is determined by a combination of internal and external factors. The internal factors primarily relate to the user’s attention while performing a task. In some cases, users may be entirely focused while at other times they may be distracted by other concurrent tasks. An example of this latter point is that when a user is driving a car, he/she can use a PDA to reference a telephone number. External factors are determined to a large extent by the device’s physical characteristics. It is not possible to make use of a traditional PC as one walks down the street. The same is not true for a mobile telephone. The challenge to the system architect is thus to match the design of a particular device’s UI with the set of constraints imposed by the corresponding context of use. 3.1. Vertical versus horizontal usability Vertical usability deals with all the usability issues related to a specific platform. It can be achieved by applying design guidelines. Many system manufacturers have issued design guidelines to assist designers in developing platform-specific applications. These guidelines can be categorized according to whether they prescribe a design model (i.e. ‘‘do this’’), or whether they discourage a particular implementation (i.e. ‘‘don’t do this’’). For the PalmOS platform [www.palmsource.com], several design guidelines address navigation issues, widget selection, and use of specialized input mechanisms such as handwriting recognition. Microsoft Corporation has also published usability guidelines to assist developers with programming applications targeted at the Pocket PC platform. Giving the user immediate and tangible feedback during interaction with an application is either broadly specified or too simplistic. Adding to this problem is the fact that in many cases, it would appear that the use of different guidelines could create many inconsistencies. Some guidelines can come into conflict, and making a trade-off is not an easy task for MUI developers. In addition to usability requirements specific to each computing platform, a MUI introduces another dimension concerned with the cross-platform usability requirements––we refer to it as horizontal usability. New guidelines that deal specifically with horizontal usability issues have emerged in the last few years. For example, Sun’s guidelines for Java [http://java.sun.com] describe the look-and-feel of a cross-platform user interface. However, this guideline does not take into account the distinctiveness of each device, and in particular, the platform constraints and capabilities. An application’s UI components should not hard-code for a

291

particular look and feel. The Java Pluggable Look and Feel (PL&F) is the portion of a Swing component that deals with its appearance (its look); it is distinguished from its event-handling mechanism (its feel). When you run a Swing program, it can set its own default look by simply calling a UIManager method named setLookAndFeel. 3.2. Usability factors The following are the major usability factors that have to be considered in MUI design: • Interaction Style. Due to mobile device constraints, the UI on a wireless handset must be fundamentally different from the same UI on a PC. In traditional desktop UIs, the keyboard and the mouse are the primary input devices. Changing from a mouse to a stylus on a touch screen requires different interaction techniques. • Information Scalability. Typically, applications that run on low-bandwidth and small-narrow platforms are more restrictive since they cannot replicate the rich information content compared to the UI of a traditional GUI application. The device’s characteristics and constraints also need to be considered with respect to their immediate effect on the task execution time, in order to achieve the same usability for mobile UIs that exist for GUIs. • Task Structure Adaptability. In migration, task structure must often be adapted. In fact, it is likely that tasks performed in a stationary environment are different from the same tasks performed in a mobile environment. For a MUI, migration should maintain support for the same functionality and feedback even if certain features or variations are eliminated in some platforms (Nielsen and Ramsay, 2000). A mobile device can also offer only partial access to a complex service. This restriction of the service can apply to data as well as functions (Karsenty and Denis, 2003). For example, an airline reservation system may make choosing a flight and buying the ticket in two separate steps. This separation should be preserved in all versions instead of having the simplified version, which would unify choosing and buying into a single step. • Visual Meaningfulness. It might be the case that reduced-bandwidth interfaces exhibit more graphical elements with a semantic value, in order to compensate for lack of display space. For example, color might be used to indicate a property that would otherwise (i.e., on a high-bandwidth interface) be indicated in a label. Differences in surface features of the user interface between devices can cause usability problems. Such inconsistencies can interfere with the analogical transfer process by preventing users from transferring

292

•

•

•

•


their initial understanding of the service to the new device. This type of difficulty generally leads to the production of mistakes or at least a sub-optimal use of the service. Questions of interest are the (1) obviousness of these graphical elements themselves and (2) the obviousness of their interpretation. Platform Constraints and Capabilities. Due to device constraints, especially bandwidth capability, the UI on a wireless handset must be fundamentally different from the same UI on a PC. It is more restrictive since it cannot replicate the rich graphics, content, and functionality of the UI of a traditional GUI application. The device’s characteristics and constraints need to be considered also with respect to their immediate effect on the task execution time, in order to achieve the same usability for mobile UIs that exists for GUIs. In traditional desktop UIs, the keyboard and the mouse are the primary input devices. Changing from a mouse to a stylus on a touch screen requires different interaction techniques. Behavior. A MUI can have a different look-and-feel while maintaining the same behavior over different platforms. For example, all user preferences must be preserved across the interfaces. If the end-user has specified a particular access mechanism using one interface, it will be used on all interfaces. User Awareness of Trade-Off. It would be acceptable to have a simplified version of a program that excludes certain additional features (such as specifying a seating preference) that are present in the more advanced version. Missing these features is a trade-off that the user would make in return for the benefits of being able to use the system under various limited circumstances. Service Availability. It is not necessary for all features to be made available on all devices. For example, a PDA interface may eliminate images or it may show them in black-and-white. Similarly, text may be abbreviated on a small display, although it should be possible to retrieve the full text through a standardized command.

These characteristics and constraints are not artifacts of current development technologies, but are intrinsic to the MUI concept. Together, they can be used as guidelines to design or evaluate the usability aspects of a MUI. 3.3. Some findings from user-oriented studies The first difficulty in evaluating a MUI comes from the fundamental differences that exist between stationary versus mobile UIs. Three studies demonstrate such difficulties: First, in the fall of 2000, Nielsen and Norman Group sponsored a field study of mobile phone users (Nielsen and Ramsay (2000)). They gave 20 users a WAP phone

and asked them to use it for a week and record their impressions in a diary. They performed traditional usability tests with users at the beginning and end of the field study. They gave half of the users an Ericsson R320s and the other half a Nokia 7110e. When users were asked whether they were likely to use a WAP phone within one year, a resounding 70% answered no. WAP is not ready for prime time yet, nor do users expect it to be usable any time soon. Remember, this finding comes after respondents had used WAP services for a week, so their conclusions are significantly more valid than answers from focus group participants who are simply asked to speculate about whether they would like WAP. They surveyed people who had suffered through the painful experience of using WAP, and they definitely did not like it. At the same time, when users were asked whether they might get WAP within three years, the ‘‘no’’ responses dropped to 20%. Users obviously see potential in the mobile Internet. It is just not there yet. € Secondly, (Oquist et al., 2003) discuss their practical experiences with stationary versus mobile usability evaluations. In particular, they outline four typical contexts of use that they characterized by monitoring four environmental usability factors. By assessing these factors, it was possible to obtain a profile of how useful a given interface can be in a certain context of use. The usability profiles for several different interfaces for input and output are also presented in this chapter, partly to illustrate how usability can be assessed over multipleuser interfaces, and partly as an illustration of how different interfaces have been adapted to mobile environment attributes. Thirdly, Karsenty and Denis (2003) argue that the usability of individual devices is not sufficient: a multidevice system needs to also be inter-usable. They define inter-usability as the ease with which users transfer what they have learned from previous uses of a service when they access the service on a new device. Based on theoretical considerations and empirical observations gathered from a study with France Telecom, they propose an analysis grid combining two types of continuity, namely knowledge and task, with ergonomic design principles including consistency, transparency, and dialogue adaptability. The concept of inter-usability is very similar to the concept of horizontal usability introduced in Section 3.1. Inter-usability or horizontal usability is a new dimension for studying the usability of MUIs and multi-device user interfaces.

4. Fertile user interface development models We will now discuss promising development models that can facilitate MUIs development while increasing their usability.


4.1. Task model-based development The model-based paradigm uses a central knowledge base to store a description of all aspects of an interface design. This central description is called a model, and typically contains information about the tasks that users are expected to perform using the application, the data of the application, the commands that users can perform, the presentation and behavior of the interface, and the characteristics of users. Model-based approaches for UI development attempt to automatically produce a UI prototype (i.e., a concrete presentation and dialog for a specific platform) from the abstract ‘‘generic’’ model of the UI (i.e., generic task, domain and dialog model) (Puerta, 1997). See also Zhuge (2002) for application of models in other domains. This is done through a mapping process of the abstract model into the concrete user interface or some its elements (Bomsdorf and Szwillus, 1998). For example, given user-task t in domain d, the mapping process will find an appropriate presentation p and dialog D that allows user u to accomplish t. Therefore, the goal of a model-based development approach is to link t, d, and u with an appropriate p and D. Modelbased UI development could be defined as a process of creating mappings between elements in various model components. A model-based approach can foster the emergence of new development approaches for MUIs, or at least, help us to understand more deeply the complexity of MUI development. As a starting point for research in the field of modelbased development for MUIs, we believe that task-based models should be the focus (Patern o, 2001). Such models can foster the emergence of new development approaches for MUIs, or at least, help us to understand more deeply the complexity of MUI development. A task model describes the essential tasks that the user performs while interacting with the UI. A typical task model is a hierarchical tree with different sub-trees of the tree indicating the different tasks that the user can perform. Task models are a very convenient specification of the way problems can be solved. Early investigations show that in the case of a MUI, we should make a distinction between four kinds of task models (M€ uller et al., 2001): 1. The general task model for the problem domain is the result of a very accurate analysis of the problem domain. It describes how a problem can be tackled in general. All relevant activities and their temporal relations are described. Such a model can be considered as the representation of the knowledge of an expert. The state of the art is captured within this model. 2. The general task model for software support contains all activities that have to be executed by a software system or in an interactive way. In some sense this model is the first design decision. It specifies which

293

tasks of the general task model are supported by the software under development. Later on the behavior of the developed software has to be consistent to the specified behavior of this task model. 3. The capabilities of a device are very restrictive in the context of use of a software system. This is the reason why the relation between task models and devices is especially stressed. As we already mentioned there are approaches to transform whole applications from one platform to another one without looking at the tasks that will be supported. But sometimes it is wise to look at the tasks first and to decide which tasks can be supported in an optimal way by which special device. This information is captured in the devicedependent task model. 4. The environment-dependent task model is the most specific one. It is based on several design decisions in previous models and describes computer-supported tasks for a special device. This model describes the behavior of a system based on the available tools, resources and the abilities of the user. It can be interpreted statically (influences are fixed during design time) or dynamically (influences are evaluated during run time). To illustrate the use of these models, let us consider the example of running an electronic shop: The general task model for a shop will be omitted here. Fig. 2 contains a general task model for software support. Since devices are already attached to tasks, it contains some device-dependent task models as well. The task of running an electronic shop consists of the sub-tasks create_shop, e_shopping and ‘‘close_shop’’, which have to be decomposed until the level of atomic tasks is achieved. These three tasks have to be performed in a sequential order. Each atomic task in a leaf (e.g. create_shop) is linked with an operation that causes a change in the task environment. The tasks ‘‘create shop’’ and ‘‘e_shopping’’ have to be performed on a personal computer. ‘‘close_shop’’ can be performed on a mobile phone. The model does not specify that you need a mobile phone, rather, that you need at least such a device. Fig. 4 gives an impression how the example model looks like in an editor. Attached to the tasks are information about the device (D), the artifact which has to be changed (A), the tools that are available (T) and the persons which are expected to perform the task in form of a role (R). This general device-dependent task model allows an extraction of the specific task model for a mobile phone (see Fig. 3). A manger can ‘‘close_shop’’ by phone after it was opened on a PC, a customer can ‘‘choose_offer’’ after looking for one on a PC, and a customer can order or discard a product after having a look (‘‘check_offer’’) at it on a PC.

294


Fig. 2. General device-dependent task model.

Fig. 3. Task model for the electronic shop constraint to a mobile phone.

In the future, it is very important to extract the temporal relations from the task structure as proposed by (Forbrig and Dittmar, 2001). In this way, it is possible to specify device-specific temporal relations separately. These temporal relations can be more restrictive than those of the device-independent task model. Fig. 4 gives an impression of an animated task model with separate specified temporal relations. On the right hand side one can see the temporal relations of the selected task and one can also see the whole set of temporal relations for the task model. One can imagine how this can be changed device-specifically and dynamically for different states in the environment (e.g. user, tools, location).

In conclusion, model-based approaches, in particular the related automatic or semi-automatic UI generation techniques, are of interest to MUI development. UI modeling will be an essential component of any effective long-term approach to developing MUIs. Increased user involvement in the UI development process will produce more usable UI models. Model-based UI systems take an abstract model of the UI and apply design rules and data about the application to generate an instance of the UI. Declarative model-based techniques involve UI modeling techniques for abstractly describing the UI. A formal, declarative modeling language should express the UI model. Current model-based techniques, which are most


295

Fig. 4. Animated device-independent task model for the electronic shop.

frequently using task and domain models, do not generate quality interfaces. UI models for supporting mobility contain not only the visual look-and-feel of the UI, but also semantic information about the interface. New UI models should factor out different aspects of UI design that are relevant to different contexts of use, and should isolate context-generic issues from contextspecific ones. Currently, the task analysis is performed for obtaining a single UI that is adapted for a single context of use. We need to model tasks that can be supported in multiple contexts of use, considering multiple combinations of the contextual conditions. Knowledge bases for domain, presentation, dialog, platform and context of use need to be exploited to produce a usable UI that matches the requirements of each context of use. The model-based techniques proposed for mobile UIs range from relatively low-level implementation solutions, such as the use of abstract and concrete interaction objects, to high-level task-based optimization of the interface’s presentation structure. An important issue for the model-based UI development for MUIs is to incorporate usability guidelines and to provide support for applying these guidelines. 4.2. UI pattern-driven development A UI design pattern encapsulates a proven solution for a usability problem that occurs in different contexts of use. UI design patterns can be used to create a high-level design model, and can therefore facilitate the development and validation of MUIs. Discussion of patterns for software design started with the software engineering community and now the UI design community has enthusiastically taken up discussion of patterns for UI design. Many groups have devoted

themselves to the development of pattern languages for UI design and usability. Among the heterogeneous collections of patterns, those known as: Common Ground, Experience, Brighton, and Amsterdam, play a major role in this field and wield significant influence (Tidwell, 1997; Borchers, 2000). Patterns have the potential to support and drive the whole design process of MUIs by helping developers select proven solutions of the same problem for different platforms. As an illustration, the convenient toolbar pattern (used on web pages) provides direct access to frequently used pages or services. This pattern, also called Top Level navigation (Tidwell, 1997), can include navigation controls for News, Search, Contact Us, Home Page, Site Map, etc. Fig. 5 shows three design solutions for this pattern. Pattern-driven development should not be considered as an alternative approach to model-based and contextaware development. In the context of MUI development, patterns can complement a task model by allowing developers to work at a high level of abstraction (Javahery and Seffah, 2002). Since patterns describe general solutions to problems, there is almost always more than one way to implement patterns. For a Web browser, it is implemented as a toolbar using HTML and JavaScript or a Java applet. For a PDA, it is implemented as a combo box using Wireless Markup Language (WML), whereas for a mobile phone, it is implemented using the selection menu. The implementations should take into account the context of use including platform constraints and capabilities. Due to screen size constraints, the combo box and selection menu implementations are suitable for PDAs and mobile phones, respectively. Pattern authors usually include code samples and maybe even a complete example of at least one solution

296


Fig. 5. The Convenient Toolbar pattern and its implementations.

to aid in the use of a pattern. We take a slightly different approach here by removing implementation details from the pattern itself and codifying them in a strategy. Each pattern may have multiple strategies, each of which defines one implementation of a pattern solution. A pattern is not directly aware of its strategies because we do not wish to limit the number of strategies available and the association of them to a pattern at just creation time. It is conceivable that people other than the original

author may later discover new strategies for applying a pattern. A strategy, then, is assigned the following functions: A strategy may define one of many possible implementations of a pattern solution. To illustrate this issue we will consider strategies for the following HCI patterns for navigation. This list is far from exhaustive, but helps to communicate the flavor and abstraction level of HCI patterns for navigation that we are targeting. Due to space limitations, we can

Table 1 Examples of HCI patterns P.1 P.2 P.3 P.4 P.5

Pattern

Definition or comments

Bread crumbs Temporary horizontal menu bar at top Contextual (temporary) vertical menu at right in content zone Information portal

Navigation trail from home page down to current page; see Amsterdam collection Displayed in a specific context (not permanent). Typically called up by an item in a left-hand vertical menu Called up by a higher-level menu or a link

P.6 P.7 P.8 P.9

Permanent horizontal menu bar at top Permanent vertical menu at left Progressive filtering Shallow menus Sub-sites

P.10

Container navigation

P.11 P.12 P.13 P.14

Deeply embedded menus Alphabetical index Key-word search Intelligent agent

P.15

Drop-down menu

P.16

Hybrid navigation

Broad first and second level on home page. Same principle as the Amsterdam collection’s ‘‘Directory’’ pattern Standard, single-row menu bar Can have one or several levels of embedding See Amsterdam collection 1 or 2 levels in same menu Shallow main menu or broad portal leading to smaller sub-sites with simple navigation architectures Different levels of menu displayed simultaneously in separate zones (e.g. Outlook Express or Netscape Mail) E.g. file manager menu Index contains hyperlinks to pages containing or describing the indexed terms Search engine Human-machine interfaces that aim to improve the efficiency, effectiveness and naturalness of human-machine interaction by representing, reasoning and acting on models of the user, domain, task, discourse and media A menu of commands or options that appears when the user selects an item with a mouse Start with key-word search, then present details of search target in menu format


297

Table 2 Examples of transformation of HCI patterns for different screen sizes HCI pattern in desktop display

Type of strategy

Replacement pattern in small PDA display

P.1

Bread crumbs

Scalable or fundamental

P.2

Temporary horizontal menu

Scalable or fundamental

P.3 P.4 P.5

Permanent vertical menu at left Information portal Permanent horizontal menu at top

Fundamental Scalable Scalable or fundamental

P.6

Temporary vertical menu in content zone

Identical, scalable or fundamental

P.7 P.8 P.9

Progressive filtering Shallow embedded menus Sub-sites

P.10

Container navigation (3 containers)

Identical Identical Identical, scalable or fundamental, depending on context Scalable or fundamental

P.1s––Shorter bread crumb trail; P.15––Drop-down ‘‘History’’ menu P.2s––Shorter menu; P.6––Link to full-page display of menu options ordered vertically P.15––Drop-down menu P.4s––Smaller information portal P.5s––Shorter menu; P.6––Link to full-page display of menu options ordered vertically P.6––Temporary vertical menu in content zone; P.6s––Shorter menu; or P.15––Drop-down menu P.7––Progressive filtering P. 8––Shallow embedded menus –

P.11

Deeply embedded menus

Multiple fundamental

P.12 P.13 P.14 P.15

Alphabetical index Key-word search Intelligent agents Drop-down menu

Identical or scalable Identical Identical Identical or scalable

P.16

Hybrid navigation

Identical or scalable

only provide the title and a brief description, rather than the full description format as described in (Borchers, 2000). The patterns are listed in Table 1. Table 2 describes the types of cross-platform transformations that are recommended for these HCI patterns. These strategies offer the closest and simplest equivalent in the corresponding platform. In the third column, the suffix ‘‘s’’ after a pattern indicates ‘‘scaled (down)’’, and the suffix ‘‘m’’ indicates ‘‘multiple (sequence)’’. 4.3. Device-independent development languages Nowadays, different development languages are available including platform-dependent and independent languages (Fig. 6). Under the umbrella of platform-dependent languages, we classify the wide variety of existing mark-up languages for wireless devices such as the WML or the light HTML version. These languages take into account the platform constraints and capabilities posed by each platform. They also suggest specific HCI patterns for displaying information and interacting with the user in specific ways for each device.

P.10s––Container navigation (2 containers); P.7––Progressive filtering P.8m––Sequence of shallow embedded menus; or P.6––Sequence of single-level menus P.12––Alphabetical index P.13––Key-word search P.14––Intelligent agents P.15––Drop-down menu; or P.15s – Shorter drop-down menu P.16s––Hybrid approach with smaller or less embedded menus

Fig. 6. Evolution of UI Development Languages.

Platform-independent languages are mainly based on UI modeling techniques. Their goal is to allow crossplatform development of UIs while ensuring consistency not only between the interfaces on a variety of platforms, but also in a variety of contexts of use. They provide support for constraints imposed not only by the computing platforms themselves, but also by the type of user and by the physical ambient environment. They should help designers recognize and accommodate each context in which the MUI is being used. Such languages provide basic mechanisms for UI reconfigurations depending on variations of the context of use. In a certain way, they address some of the problems raised by context-aware development (Dey and Abowd, 2000).

298


There are several Extensible Markup Languages available (http://www.xml.org) for a variety of topics: ebXML (commerce), IMS Specifications, SIF Implementation Specification Version 1.0 (education), and so on. In general, these languages do not include display semantics. Therefore, implementations based on them could either be transformed using an XSL Transformations (XSLT) style sheet to one of the vocabularies from the direct approach or natively presented to an XML browser using a CSS style sheet. For user interfaces, there are XML-based languages already available including eXtensible Interface Markup Language (XIML) and User Interface Markup Language (UIML). Some of the advantages of these languages are: • The separation of UI description from content, by providing a way to specify how UI components should interact, and to spell out the rules that define interaction behavior. • The abstraction level that allows the UI to adapt to a particular device or set of user capabilities. • Supports model-based and pattern-driven development. In the following section (4.3.1 and 4.3.2), we discuss the XIML and UIML languages and their applications in conjunction with model-based and UI pattern-driven development approaches. 4.3.1. XIML and task-based development In its most basic sense, XIML is an organized collection of interface elements that are categorized into one or more major interface components. The language does not limit the number and types of components that can be defined. In its first version (1.0), XIML predefines five basic interface components, namely task, domain, user, dialog, and presentation. The first three of these can

be characterized as contextual and abstract, while the last two can be described as implementation-based and concrete. An XIML-based UI specification can lead to both an interpretation at runtime and a code generation at design time. The main features of the language are [www.ximl.org]: • Represents the concrete aspects of a user interface (such as presentation and dialog) and also its abstract aspects (such as context and task models). • Bridges the gap between design and development of user interfaces by providing a single common representational framework for both processes. Using XIML, the design and implementation of a UI will be a series of refinements from an abstract UI representation (of the user context or a task model for example) to a concrete UI representation (of widgets and interaction techniques for example). In the XIML framework, the definition of the interface is the actual XIML specification and the rendering of the interface is left up to the target device to handle. This is an advantage over many model-based interface development systems that do not have this separation established clearly and therefore developers end up mixing up interface logic with interface definition. By simply defining one presentation component per target device, the entire specification can support multiple platforms. Besides the fact that a presentation component is needed for each target device, the framework saves development time and ensures consistency among UIs. The following is a portion of an XIML-based task description and its mapping to concrete UI elements. The tool presented in Fig. 4 was developed on the basis of XIML. It has the potential to evolve the models to final interactive systems.

An XIML-Based Description of Task and UI [Source www.ximl.org] /* Task Description /* UI Specification |


4.3.2. UIML and patterns-oriented development By using XML-compliant implementations, patterns can be easily mapped or rendered into different languages. The languages are Hypertext Markup Language (HTML), WML, and Extensible Hypertext Markup Language (XHTML) 1.0. Documents marked-up using these vocabularies could be styled using a Cascading Style Sheet (CSS). The following portion of code shows how the ‘‘go back to a safe place’’ pattern can be implemented in UIML. This pattern allows users to go back to a safe place, such as a home, when they are lost. It is generally implemented as a button or an icon (strategy): Go! This UIML description can be then rendered, using a UIML code generator, in HTML, Java, and WML using . The feature maps button to Java and HTML: /* Java /* HTML This piece of code is written once for all platforms, like a device driver for an OS.

5. Conclusion In this paper, we discussed the research challenges and the development avenues in cross-platform and multiple-user interfaces. Some challenges are due to the fact a MUI runs on different computing platforms accommodating the capabilities of various devices and the different contexts of use. Challenges are also triggered because of the universal access requirements for a diversity of users. We also demonstrated that current development tools and methods do not adequately address the MUI challenges of diversity, cross-platform consistency, universal accessibility, and integration. Therefore, there is an urgent need for a new integrative framework for modeling, designing, and evaluating MUIs for the emerging generation of interactive systems. As outlined in this paper, an effective MUI development approach should combine different models and approaches. MUI architectures that neglect these models cannot effectively cater to the requirements of the

299

different users. Unfortunately, adoption of a MUI application is contingent on the acceptance of all the stakeholders. Researchers should focus on ways to assist developers in creating effective MUI designs for a large variety of computing platforms. Existing methods work well for regular software development and have thus been adapted for the case of MUIs. However, these methods usually result in tools that do not capture the full complexity of the task. Pattern hierarchies seem to be an exception to this finding. Whereas an individual pattern provides a solution to a specific problem, organized as a hierarchy, patterns guide the developer through the entire architectural design. In this way, they enforce consistency among the various views and breakdown complex decisions into smaller more comprehensible steps.

References Abrams, M., Phanouriou, C., 1999. UIML: an XML Language for Building Device-Independent User Interfaces. In: Proceedings of XML 99, Philadelphia. Bomsdorf, B., Szwillus, G., 1998. From task to dialogue: task-based user interface design. SIGCHI Bulletin 30 (4). Borchers, J.O., 2000. A pattern approach to interaction design. In: Proceedings of the DIS 2000 International Conference on Designing Interactive Systems, New York, 16–19 August 2000. ACM Press, pp. 369–378. Brewster, S., Leplâtre, G., Crease, M., 1998. Using non-speech sounds in mobile computing devices. In: Proceedings of the First Workshop on Human Computer Interaction of Mobile Devices, Glasgow. Dey, A.K., Abowd, G.D., 2000. Towards a Better Understanding of Context and Context-Awareness. In: Proceedings of CHI’2000 Workshop on Context Awareness, The Hague, Netherlands, April 1–6. Eisenstein, J., Vanderdonckt, J., Puerta, A., 2001. Applying modelbased techniques to the development of UIs for mobile computers. In: Proceedings of ACM Conference on Intelligent User Interfaces IUI’2001, Albuquerque, 11–13 January 2001. ACM Press, New York, pp. 69–76. Forbrig, P., Dittmar, A., 2001. Software Development and Open User Communities. HCI 2001, New Orleans, August 2001. Ghani, R., 2001. 3G: 2B or not 2B? The potential for 3G and whether it will be used to its full advantage. IBM developer Works: Wireless Articles, August 2001. Grudin, J., 1994. Groupware and social dynamics: eight challenges for developers. Communications of the ACM 37 (1), 92–105. Hochheiser, H., Shneiderman, B., 2001. Universal usability statements: marking the trail for all users. ACM Interactions 8 (2), 16–18. Javahery, Seffah, A., 2002. A model for usability pattern-oriented design. In: Task Models and Diagrams for User Interface Design (Tamodia’2002), Bucharest, 18–19 July 2002. Johnson, P., 1998. Usability and mobility: interactions on the move. In: Proceedings of the First Workshop on Human Computer Interaction With Mobile Devices, Glasgow. Karsenty, L., Denis, C., 2003. Inter-usability of multi-device systems: a conceptual framework. In: Seffah, Javahery, (Eds.), Multi-Devices, Context-Aware and Multiple User Interfaces. Wiley. McGrenere, J., Baecker, R., Booth, K., 2002. An evaluation of a multiple interface design solution for bloated software. ACM CHI 2002, Minneapolis, 20–24 April 2002.

300


M€ uller, A., Forbrig, P., Cap, C., 2001. Model-Based User Interface Design Using Markup Concepts. DSVIS, Glasgow. Myers, B., Hudson, S., Pausch, R., 2000. Past, present, future of user interface tools. Transactions on Computer-Human Interaction ACM 7 (1), 3–28. Nielsen, J., Ramsay, M., 2000. WAP Usability Deja Vu: All Over Again Report from a Field Study in London, Fall 2000, Nielsen Norman Group. Olsen, D., Jefferies, S., Nielsen, T., Moyes, W., Fredrickson, P., 2000. Cross-modal Interaction using XWeb. In: Proceedings of the UIST’2000, CA, USA. € Oquist, G., Mikael Goldstein, M., Chincholle, D., 2003. Assessing usability over multiple user interfaces. In: Seffah, Javahery, (Eds.), Multi-Devices, Context-Aware and Multiple User Interfaces. Wiley. Patern o, F., 2001. Task models in interactive software systems. In: Chang, S.K. (Ed.), Handbook of Software Engineering Knowledge Engineering. World Scientific Publishing Co. (ISBN: 98102-4973-X). Puerta, A.R., 1997. A model-based interface development environment. IEEE Software 14 (4), 41–47. Seffah, A.T., Radhakrishan, G.C., 2001. Workshop on multiples user interfaces over the Internet: engineering and applications trends.

In: HM-HCI: French/British Conference on Human Computer Interaction, 10–14 September 2001, Lille, France. Stephanidis, C., 2002. User Interfaces for All: Concepts, Methods, and Tools. Lawrence Erlbaum Associates. Thevenin, D., Coutaz, J., 1999. Plasticity of user interfaces: framework and research agenda. In: Sasse, A., Johnson, Ch.. (Eds.), Proceedings of IFIP TC 13 International Conference on Human–Computer Interaction Interact’99, Edinburgh. IOS Press, London, pp. 110– 117. Tidwell, J., 1997. Common Ground: A Pattern Language for Human– Computer Interface Design. Available from http://www.time-tripper.com/uipatterns. Vanderdonckt, J., Oger, F., 2001. Synchronized model-based design of multiple user interfaces. In: Workshop on Multiples User Interfaces over the Internet: Engineering and Applications Trends. HMHCI: French/British Conference on Human Computer Interaction, 10–14 September 2001, Lille, France. Winograd, T., 2001. Architectures for Context. Human–Computer Interaction 16 (2–3). Zhuge, H., 2002. A knowledge grid model and platform for global knowledge sharing. Expert Systems with Applications 22, 313– 320.