ICOs: A model-based user interface description technique dedicated ...

ICOs: A Model-Based User Interface Description Technique dedicated to Interactive Systems Addressing Usability, Reliability and Scalability DAVID NAVARRE, PHILIPPE PALANQUE, JEAN-FRANCOIS LADRY, and ERIC BARBONI IHCS-IRIT University Paul Sabatier

The design of real-life complex systems calls for advanced software engineering models, methods, and tools in order to meet critical requirements such as reliability, dependability, safety, or resilience that will avoid putting the company, the mission, or even human life at stake. When such systems encompass a substantial interactive component, the same level of confidence is required towards the human-computer interface. Conventional empirical or semiformal techniques, although very fruitful, do not provide sufficient insight on the reliability of the human-system cooperation, and offer no easy way to, for example, quantitatively and qualitatively compare two design options with respect to that reliability. The aim of this article is to present a user interface description language (called ICOs) for the engineering and development of usable and reliable user interfaces. The CASE tool supporting the ICOs notation (called Petshop) is a Petri nets-based-tool for the design, specification, prototyping, and validation of interactive software. In that environment models (built with the formal description technique ICOs) of the interactive application can be interactively modified and executed. This is used to support prototyping phases (when the models and the interactive application evolve significantly to meet late user requirements, for instance) as well as the operation phase (after the system is deployed). The use of ICOs and PetShop is presented on several large-scale systems such as a multimodal ground segment application for satellite control, an air traffic control interactive application, and an application for new generation of interactive cockpits in large civil aircraft such as Airbus A380 or Boeing 787. The article emphasizes the demonstration of the expressive power of the notation and how it can support the description of various aspects of user interfaces, namely interaction techniques (both WIMP and post-WIMP), interactive components (such as widgets), and the behavioral part of interactive applications such as the dialog and the functional core. It also demonstrates that PetShop provides dedicated support for prototyping activities of behavioral aspects at the various levels of the architecture of interactive systems. While This work is supported by the EU-funded Network of Excellence ResIST http://www.resist-noe.eu under contract n◦ 026764 and by the CNES (French National Center for Spatial Studies) R&T Tortuga project, R-S08/BS-0003-029. Authors’ addresses: D. Navarre, P. Palanque (contact author), J.-F. Ladry, and E. Barboni, IRIT, University of Toulouse 3, 118 route de Narbonne, 31062, Toulouse Cedex 9, France; email: [email protected]. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or [email protected]. C 2009 ACM 1073-0516/2009/11-ART18 $10.00 DOI 10.1145/1614390.1614393 http://doi.acm.org/10.1145/1614390.1614393 ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18

18:2

•

D. Navarre et al.

the focus is on past work done on various large-scale applications, the article also highlights why and how ICOs and Petshop are able to address challenges raised by next-generation user interfaces. Categories and Subject Descriptors: D.2.2 [Software Engineering]: Design Tools and Techniques—Computer-aided software engineering (CASE); D.2.11 [Software Engineering]: Software Architectures—Languages (e. g., description, interconnection, definition); H.5.0 [Information Interfaces and Presentation (e.g., HCI)]: General General Terms: Languages, Theory, Verification Additional Key Words and Phrases: User interface description language, model-based approaches, formal description techniques, interactive software engineering, interactive prototyping, safetycritical interactive systems ACM Reference Format: Navarre, D., Palanque, P., Ladry, J.-F., and Barboni, E. 2009. ICOs: A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability. ACM Trans. Comput.-Hum. Interact. 16, 4, Article 18 (November 2009), 56 pages. DOI = 10.1145/1614390.1614393 http://doi.acm.org/10.1145/1614390.1614393

1. INTRODUCTION In accord with the recurring desire of increasing the bandwidth between the interactive system and users, more sophisticated interaction techniques are continuously being proposed. Such proposals are usually presented in conferences such as ACM CHI (Human Factors in Computing Systems) or UIST (User Interfaces Software and Technology) with a focus on the innovation and on the usability evaluation of interactive systems proposing such interaction techniques. While a significant effort is currently being undertaken by the Computer-Human Interaction (CHI) community in order to apply and extend current usability evaluation techniques to new kinds of interaction techniques, very little has been done to improve the reliability of software offering these kinds of interaction techniques. As these new interaction techniques are currently increasingly used in the field of command-and-control safety-critical systems, the potential for incidents and accidents has increased. Similarly, the unreliability of interactive software can jeopardize usability evaluation by presenting to the users unexpected or undesired behaviors. Lastly, iterative design processes promote multiple designs through evolvable prototypes in order to accommodate requirement changes and to solve usability problems identified in the usability evaluation phase, thus reducing the reliability of the final system by lack of global and structured design. Researchers and practitioners designing and developing these interaction techniques usually have to build their own user interface tools or twist programming languages [Dragicevic 2004] to be able to implement the interaction technique they want to design and evaluate. For this reason, they also often define prototypes of user interface tools (often also called user interface management systems) to show what language and development environment are needed to develop the proposed interaction technique. Once published, there remains a long process for these innovative interaction techniques to reach the maturity level required for dissemination in industry, as several problems remain to be solved. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

ICOs: A Model-Based User Interface Description Technique

•

18:3

—The first problem is related to the scalability issue where research contributions must be able to go from demonstrational prototypes to real-size applications. —The second problem is related to the reliability issue, that is, to find ways to guarantee the correct functioning of the interactive system. Actions such as verification, validation, and exhaustive testing are possible ways to address this problem. —The third problem is related to the link between user interface tools and the development process of interactive systems. The issue is here to be able to integrate such tools within development processes and other software development tools currently used in industry. In the field of safety-critical systems, additional phases to the more “classical” development processes have to be supported; for instance, the certification phase nowadays, at least in the aircraft industry, goes beyond hardware and software components and involves human factor certification [Riley 2004]. —The last but not least problem addressed in this article is related to the expressive power of the language, that is, how the language is able to cover the various elements that have to be addressed when user interfaces are described. The article presents a formal description technique called ICOs (which stands for Interactive Cooperative Objects) dedicated to the modeling of interactive systems. This formal description technique was first introduced in Palanque [1992] and dealt with form-based user interfaces. Since then the notation has been extended to deal with new user interfaces such as multimodal [Bastide et al. 2004] and 3D interfaces [Navarre et al. 2005] and to address issues raised by various safety-critical-command-and-control systems such as air traffic control workstations [Navarre et al. 2001], ground segments for satellite control [Palanque et al. 2006], or civil [Barboni et al. 2006b] and military aircraft cockpits [Bastide et al. 2004]. The expressive power of the notation makes it eligible for the description of new generation interfaces, as challenges raised by multimodal interfaces are comparable with the ones they raise. The article presents, in one single location, the current version of the notation with its latest extensions and presents by means of several case studies the type of user interfaces addressed. PetShop (which stands for Petri net workShop), the CASE tool supporting the notation (and is available at http://ihcs.irit. fr/petshop), is also presented and its capability to support prototyping activities is introduced. The next section of the article presents an overview of the state-of-the-art in the field of user interface description languages and notations. This related work section aims at structuring the state-of-art according to several criteria applied to each notation/language. The criteria are: type of interaction technique supported, scalability (capability to address real-size interactive systems), availability of tool support, and expressive power. Section 3 presents in detail the ICOs notation and uses the example of the description of a lowlevel interaction technique to present the main characteristics of the notation. Section 4 is dedicated to PetShop and presents on an interactive cockpit ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:4

•

D. Navarre et al. Table I. References and Interaction Coverage for UIDLs Interaction Coverage

Constraints Code-based

Flow-based

State-based

Petri nets

Language Coral Apogee Squeak CSP DMI Subarctic XISL Usixml GWUIMS Tatsukawa Marigold Wizz ed ICON Swingstates Hierarchical 3-State Model GTN HephaisTK IOG Shadow NiMMiT Hinckley OSU MIML ICO

Reference [Szeleky & Myers 1988] [Hudson 1989] [Cardelli et al. 1985] [Smith & Duke 1999] [Anson 1982] [Hudson et al. 2005] [Katsurada et al. 2003] [Limburg et al. 2004] [Silbert et al. 1986] [Tatsukawa 1991] [Willans & Harrison 2001] [Esteban et al. 1995] [Dragicevic & Fekete 2004] [Appert et al. 2006] [Blanch et al. 2006] [Buxton 1990] [Kieras & Polson 1983] [Dumas et al. 2008] [Carr 1994] [Jacob 1986] [Coninx et al. 2007] [Hinckley et al. 1998] [Keh & Lewis 1991] [Latoschik 2002] This Paper

Low Level N N Y N Y N N N N Y Y Y Y N Y Y Y Y Y Y Y Y Y Y Y

Multimodality Y N Y Y Y N Y Y Y N N Y Y N Y N N Y N Y Y Y N Y Y

Tangible N N N N N N N Y N N N N N N N N N Y N Y Y N N Y N

Fusion N N N Code N N N N N N N Code Code N Code N N Code N N Code Y N Code Y

Widget Y Y Y N N N N Y Y N N Y N Y Y N N N Y Y N N N N Y

Rendering Code Y Code Code N Code Code Code Code N N Code Code Code N N N Code N Code Code N Code Code Code

Dialog N N Code N N N Code Code N N N N N N N N N Y N Y Y N Y Y Y

application for civil aircraft how prototyping activities are supported. From excerpts of industrial funded projects, Section 5 presents the expressiveness and the coverage of ICOs. It also presents some limitations of the approach that have arisen from applying it to real-life applications. Section 6 provides an overview of the main advantages and limitations of using ICOs and PetShop, and more generally deals with the issue of using formal description techniques for interactive systems description. That section also explains why ICOs and PetShop feature the necessary constructs for dealing with the challenges raised by next-generation user interfaces. Section 7 concludes the article and presents perspectives to the work presented. 2. RELATED WORK Since the late 1960’s [Parnas 1969] a lot of work has been devoted to notations for user interface description. This section presents an excerpt of such work focusing on 25 specific contributions. These contributions have been selected either because they provide a unique and important contribution or because they played a pioneering role in the area of User Interface Description Languages (UIDLs). This related work is summarized in the three tables presented next. Each table addresses a specific aspect and only Table I contains the bibliographical reference to the work. The other two tables refer to the work by either the name of the UIDL or the underlying description technique on which the work is based, information which is also presented in Table I. The UIDLs considered have been gathered (see rows grouped on the lefthand side ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:5

Fig. 1. The Arch/Slinky model from Bass et al. [1991].

of Figure 1) according to the modeling-intrinsic characteristic of the underlying notation or language: Petri-nets-based, state-based, flow-based, code-based, and constraints-based. This classification is also important as the use of the UIDL requires in most cases a deep understanding of that underlying notation or language.1 We have defined a set of four types of characteristics (interaction coverage, scalability, tool support, and expressiveness) for UIDLs in order to group and compare previous work. The point here is not to provide a ranking of these different contributions but rather to present what was the main focus of the contribution in order to support, for instance, people interested in describing a given User Interface (UI) to select the most appropriate UIDL according to the idiosyncrasies of the target UI. Table I positions the 25 contributions with respect to the first characteristic: interaction coverage. Under that category we represent which components of the architecture of an interactive system are covered by the UIDL. To be more precise, Figure 1 represents an architectural model of interactive systems called Arch. This architectural model has been introduced in Bass et al. [1991] and is detailed in Section 3 where it supports the presentation of the UIDL ICOs. The interaction part of the architecture covers the three righthand-side components: physical interaction, logical interaction, and dialog controller. The other two are related to the noninteractive part of the interactive application and are not considered here, even though how the interactive part relates to them is critical when dealing with the entire interactive systems. Interaction coverage presented in Table I is then divided in five different subelements: low level, multimodality, fusion, widget, rendering, and dialog. The low-level element represents the fact that the research carried out around the UIDL has demonstrated that it is possible to describe low-level interaction. This typically requires from the UIDL the possibility to represent chains of event flows and the possibility to compose low-level events, into higher-level ones (for instance, the sequence of (up, down, up, down) events would be composed in a doubleclick one, provided that the input device has not been moved 1 In

these state-of-the-art tables, we refer to the original paper rather than the most recent one. Indeed, in Jacob [1986] there is no reference to tangible interaction but that early work has been extended in Shaer and Jacob [2005] to such interaction. This is why Jacob [1986] appears in Table III. The same is true for Jacob et al. [1999].

ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:6

•

D. Navarre et al.

and that the total time for that sequence is smaller than a given threshold). For example, from a state move with a button down event, the three-state model of Buxton [1990] activates the state dragging that is disabled with the button up event. The multimodality column in Table I contains the value Y (yes) if the research carried out around the UIDL has demonstrated that it is possible to describe interactions involving more than one input device in the same construction. In HephaisTK [Dumas et al. 2008], a model can represent multiple inputs to allow multimodality. To be able to describe synergistic multimodality (using in a simultaneous way several input devices in which events are fused to trigger specific commands), it is required that the UIDL is able to represent fusion engines. This is represented in the column “Fusion” in Table I. In some cases, the UIDL is able to express at the model level the fusion explicitly (value Y) or if the fusion is made by code embedded in an object of the model as in CSP [Smith and Duke 1999] (in such cases the value in the column is “Code”). While dealing with WIMP (Window, Icon, Menu, Pointing device) interfaces some UIDLs have embedded the possibility to describe interaction components like widgets (“Widgets” column in Table I). For instance, the IOG (Interaction Object Graphs) UIDL has been introduced [Carr 1994] with the specification of a horizontal slider indicator. Tangible interfaces are a rather new interaction paradigm and some UIDLs have been introduced to address their specificities [Jacob 1986]. Dealing with input is important (as presented before) but if the UIDL aims at describing graphical elements of the interaction, then rendering has to be addressed. This can be done by adding rendering in a constraints structure, as in Apogee [Hudson 1989] where application rendering can be specified when input events, are received by graphical objects. One of the architectural components which has received greater attention is the dialog one, as it is typical to interactive systems and requires description of both states (the ones the system can be in) and events (the ones available according to the current state of the system). For instance, XISL [Katsurada et al. 2003] models the dialog part of an interactive Web application in an XML-like format. Table II positions the related work with respect to two characteristics: scalability and tool support. For the scalability aspect we report the type of example used by the researchers to demonstrate the use of the proposed UIDL. There are three different values for scalability according to whether the UIDL has been applied on at the “Toy Example” level, on a medium-size “Case Study,” or on a “Real Size” application. As for the real-size application this can mainly be assessed through technical reports or Web pages of the UIDL as the size is typically too big for a research paper. A toy example is just a little model (or description) for the presentation of the language, such as Hinckley et al. [1998]. Some languages have described a more complex application (called a case study), such as NiMMiT [Coninx et al. 2007] that models a virtual reality application. Few languages have been used in real-size applications for the industrial world; an example is Usixml by Limbourg et al. [2004], who have developed the Web site for Belgian Walloon region ministry. The tool support aspect defines whether web sites or papers contain a Snapshot of the application (like the Shadow [Jacob 1986] snapshot in Shaer and Jacob [2005]), if a “Demo” of the application is available on a Web site, such ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:7

Table II. Scalability and Tool Support Aspects of UIDLs

Coral Apogee Squeak CSP DMI Subarctic XISL Usixml GWUIMS Tatsukawa Marigold Wizz ed ICON Swingstates Hierarchical 3-State Model GTN HephaisTK IOG Shadow NiMMiT Hinckley OSU MIML ICO

Toy Example Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N Y Y Y Y Y Y Y Y

Scalability Case Study N Y N Y Y Y Y Y Y N Y Y Y N Y N N Y Y Y Y N N Y Y

Real Size N N N N N Y N Y N N N N N N Y N N N N N N N N N Y

Snapshot N Y N N Y Y Y Y Y N Y Y Y Y Y N N Y Y Y Y N Y Y Y

Tool Support Demo Downloadable N N N N N N N N N N Y Y N Y Y Y N N N N N N Y N Y Y Y Y N Y N N N N Y N Y N N N Y N N N N N N Y Y Y

as the MIML demo [Latoschik 2002],2 and if the tool is “Downloadable,” like Swingstates [Appert and Beaudouin-Lafon 2006] available on sourceforge.3 Lastly, Table III summarizes the expressiveness of the UIDL. Six different properties of the language are used to characterize this expressiveness. This expressive power is not a goal per se but it clearly defines the type of user interface that can be described using the UIDL and the ones that are beyond their reach. The first three characteristics deal with description of objects and values in the language (this is named “Data Description”), with the description of states (“State Representation”) and the description of events (“Event Representation”). For these three characteristics, there are three possible values. —Y means that that characteristic is explicitly handled by the UIDL; —N means that the characteristic is not explicitly handled; and —Code means that the characteristic is made explicit but only at the code level and is thus not a construct of the UIDL. For instance, data is described in many UIDLs such as ICON [Dragicevic et al. 2004], which allows modeling data emission and reception from an output port 2 http://www.techfak.uni-bielefeld.de/ags/wbski/demos/. 3 http://swingstates.sourceforge.net/.


18:8

•

D. Navarre et al. Table III. Expressive Power of the UIDLs Expressiveness of UIDL

Data State Event Time Descrip- Represen- RepresenConcurrent Dynamic tion tation tation Quantitative Qualitative Behaviour Instantiation Coral Code N N N Y N N Apogee Code N N N N N N Squeak Code N Code N Y Y N CSP Code Code Code N Y Y N DMI Code Code Code N Y N N Subarctic N Code Code N Y N N XISL Code N Code N Y Y N Usixml Code N Code N Y Y N GWUIMS Y N Code N Y Y N Tatsukawa Y N Y N Y Y N Marigold Y Y Y N Y Y N Wizz ed Y N Y N Y Y N ICON Y N Y N Y Y N Swingstates N Code Code N Y N N Hierarchical Code Code Code N Y Y N 3-State Model N Y Y N Y N N GTN N Y Y N Y N N HephaisTK Code Y Y N Y Y N IOG Y Y Y N Y N N Shadow Y Y Y N Y Y N NiMMiT Y Y Y N Y Y N Hinckley N Y Y N Y Y N OSU Y Y Y N Y Y N MIML Y Y Y N Y Y N ICO Y Y Y Y Y Y Y

of a device of the model to the input port of another device. Some UIDLs can also represent states of the system, such as GTN [Kieras and Polson 1983], which represents the states with nodes in the models. Events are also sometimes explicitly represented as in Wizz’ed [Esteban et al. 1995] where connections between bricks represent event flows. Time is also an important characteristic for behavioral description of interactive applications. Two different representations of time are addressed here: quantitative and qualitative. Quantitative time represents behavioral temporal evolutions related to a given amount of time (usually expressed in milliseconds). This is necessary for the modeling of the temporal windows in a fusion engine for multimodal interfaces, where events from several input devices are fused only if they are produced within a same time frame. In ICO (in this article), timed transitions express such constraints. Qualitative time aims at representing ordering of actions such as precedence, succession, and simultaneity. In OSU [Keh and Lewis 1991] a transition between two places represents the fact that the activity represented by the second place will only be active after the first one is achieved. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:9

The last two characteristics of Table III are “Concurrent Behavior” and “Dynamic Instantiation.” Representation of concurrent behavior is necessary when the interactive systems feature multimodal interactions or can be used simultaneously by several users. This can be made explicit in the models like in data flow notations, as in ICon [Dragicevic and Fekete 2004] or Whizz’Ed [Esteban et al. 1995] and in all the notations based on Petri nets (last four rows of Table III). Concurrency representation can also be found in older languages such as Squeak [Cardelli and Pike 1985], where it is possible to represent parallel execution of processes. Dynamic instantiation of interactive objects is a characteristic required for the description of interfaces where objects are not available at the creation of the interface (as in WIMP interfaces where all the graphical objects are predefined and appear in a window) as, for instance, in desktop-like interfaces where new icons are created according to user actions. Supporting explicit representation of dynamic instantiation requires the UIDL to be able to explicitly represent an unbounded number of states, as the newly created objects will by definition represent a new state for the system. Most of the time, this characteristic is handled by means of code and remains outside the UIDLs. Only Petri-nets-based UIDLs can represent explicitly such a characteristic, provided they handle high-level data structures, or objects, as is the case for many dialects [Lakos 1991], [Bastide and Palanque 1990; Genrich [1991]. 3. ICOs: A LANGUAGE FOR USER INTERFACE DESCRIPTION This section is dedicated to the presentation of the ICOs UIDL. To understand the basics of our description language, we have to present the architecture of an interactive application in order to present how each component of that architecture is taken into account with ICOs. For this purpose we use the Arch/Slinky model [Bass et al. 1991] presented in Figure 1. 3.1 Informal Presentation of the Arch Architecture The Arch model was designed to overcome some problems raised by the Seeheim model [Green 1986] which defines the three basic components of interactive applications: the functional core, presentation, and dialog controller. Arch refines the relationship between these three components and introduces two additional ones. —The physical interaction aims at handling interaction at a lexical level. It forwards the inputs from users on interactive objects (by acting on the input devices) and performs the rendering of information provided by the logical interaction component. This component is typically addressed by graphical toolkits. —The logical interaction converts information provided by the physical interaction component into abstract (interface-independent) information which is then transmitted to the dialog controller (and back again). This component corresponds to a syntactic level of interaction and it abstracts the physical interaction, making it independent from the dialog controller. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:10

•

D. Navarre et al.

—The dialog controller ensures the sequencing of the set of events. It describes the set of the authorized events, defines (according to the inner state of the interactive application represented in the dialog controller, too) how the events change the inner state of the application and how the application reacts to these events. —The functional core adaptor translates calls from the dialog controller into functional-core-compliant instructions, and it translates the callbacks from the functional core into dialog-compliant instructions. The functional core adapter is mainly dedicated to adjust the gaps between the functional core and the dialog. —The functional core controls, handles, and executes actions on the domainspecific objects, independently from how they will be rendered to the user (via dialog and presentation components). Components closest to the ground (functional core and physical interaction) are directly linked to the real world while upper components provide a more abstract view of the application. We usually divide the two interaction components (physical and logical) into two communication channels which correspond to the information flow between the user and the application. —The events flow illustrates the transformation of the user actions by the graphical toolkit into higher-level events that are translated into dialogcompliant actions. —The rendering flow illustrates how the system inner state changes that are forwarded by the dialog are translated into logical changes that are, in turn, translated into physical changes. The following sections present our description language and how it handles the various elements of Arch. In order to avoid a formal presentation that might be cumbersome to follow we present the UIDL following two different and complementary paths. —We include a description of an interactive application (Section 3.2) which corresponds to the four lefthand-side components of Arch (from functional core to logical interaction). —We also detail a description of an interaction technique (Section 3.3) which corresponds to the physical interaction component. To illustrate the language constructs of our approach, we use the example of an aircraft application called WXR (for Weather Radar System) which is part of the cockpit set of applications. It allows the crew members to adjust the orientation (the tilt angle) of the weather radar system when needed (the main idea being to use this feature only if necessary, as most of the time the default behavior is the correct one). There are three possible states for tilt angle selection: auto adjustment, auto stabilization, and setting up manually the tilt angle (as shown on lower part of Figure 2, the upper part allowing changing the mode of the weather radar independently from the tilt selection). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:11

Fig. 2. Snapshot of the WXR application in civil commercial aircrafts.

3.2 Description of an Interactive Application Using ICOs ICO formalism is a formal description technique dedicated to the specification of interactive systems and finds its roots in Bastide and Palanque [1990]. It uses concepts borrowed from the object-oriented approach (dynamic instantiation, classification, encapsulation, inheritance, client/server relationship) to describe the structural or static aspects of interactive systems, and uses high-level Petri nets [Genrich 1991] to describe their dynamic or behavioral aspects. As the ICO UIDL has evolved since its first version presented in 1992 (the Ph.D. thesis of Philippe Palanque [1992] in French), we present here the latest version. To describe it we first introduce the underlying principles that are defined in a notation called CO (Cooperative Objects) [Bastide et al. 1999], which requires the presentation of high-level Petri nets on which it is based. We the present how CO notation has been extended to be able to describe event-driven interactive systems. The following paragraphs (Section 3.2.1) briefly recall basics principles of Petri nets and the reader familiar with this notation may go directly to Section 3.2.2 that presents the cooperative objects description technique. Section 3.2.3 presents the main features of the ICO UIDL. 3.2.1 Petri Nets (PN). Petri nets were initially introduced by C. A. Petri [1962] and have been since then extensively used for the modeling of discrete event systems. Petri nets are a formalism that features a complete equivalence between a graphical and an algebraic representation thus making it possible to perform mathematical analysis of models. In few words, a Petri net is an oriented graph composed of a set of places (usually represented by ellipses) which symbolize state variables holding tokens which symbolize values and a set of transitions (usually represented by rectangles) which symbolize actions and state changes. A set of arcs connect places to transitions (called input arcs) or transitions to places (called output arcs), and represent the flow of tokens through the Petri net model. The state of the modeled system is represented by the distribution of tokens across the set of places (called marking). The dynamic behavior of a marked Petri net is expressed by means of two rules: the enabling rule and the firing rule. A transition is enabled if each input place (places linked to the transition with input arcs) contains at least as many tokens as the weight of the input arcs it is linked to. When an enabled transition is fired, tokens are removed from the input places of the transition and new tokens are deposited into the output places (the number of tokens is defined by the weight on the output arcs). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:12

•

D. Navarre et al.

Fig. 3. Metamodel of the cooperative object formalism.

These rules illustrate one of the main properties of Petri nets, called locality of enabling and firing of transitions, which makes, for instance, Petri nets enabled to model true concurrent systems. This is because the firing of a transition only impacts its input and output places, leaving the rest of the model unchanged. 3.2.2 Cooperative Objects Notation (CO). A cooperative object states how the object reacts to external stimuli according to its inner state (these stimuli are either method calls (from other objects) or event notifications). As classic Petri nets do not easily allow describing data, we use object Petri nets to describe the behavior of classes, called the Object Control Structure (ObCS). The introduction of Object Petri Nets (OPN) [Lakos 1991; Valk 1998] provides means to handle a more complex data structure using the object-oriented paradigm. Therefore, an ObCS is still a Petri net in terms of structure and behavior, but it holds a much more complex set of information. For instance, in this notation, tokens in the places are comprised of a typed set of values that can be references to other objects (allowing, for instance, use in a transition of methods calls) or preconditions about the actual value of some attributes of the tokens. The metamodel in Figure 3 presents the static structure of the cooperative object formalism. The CO formal description technique is thus able to model the entire behavior of an object, including state description, state changes, and preand post-conditions. For instance, Figure 4 shows a subpart of the functional core part of the WXR application, which has two roles: —checking the validity of the tilt angle set; and —setting the tilt angle back to its initial value after a predefined time frame. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:13

Fig. 4. Subpart of the functional core behavior liked to the setting of the tilt angle.

The cooperative object shown by Figure 4 implements the method setTiltAngle with the following signature. void setTiltAngle (float anAngle) throws IllegalArgumentException This signature is mapped into a set of three places called SIP setTiltAngle, SOP setTiltAngle, and SEP setTiltAngle, and the behavior of this object as follows. —When an object invokes this method, a token is set into place Sip setTiltAngle; when such state change occurs, two possible transitions are enabled and may be fired: checkAngleIsValid if the angle value is between the correct bounds (between −15 and +15 degrees) or checkAngleIsInvalid if not (this is represented by the precondition “anAngle ≤ −15 anAngle >15”). —Firing checkAngleIsInvalid leads to setting a token into the special place SEP setTiltAngle, which means the method setTiltAngle raised an exception. A possible alternative modeling for the behavior of this object could have been to automatically correct the angle (in order to keep it between the correct bounds) instead of raising an exception. —Firing checkAngleIsValid leads to: —setting a token in the special place SOP setTiltAngle, which means the method execution was correct; —calling the method newAngleNotification (which belongs to the dialog part) to notify the fact that a new angle has been set; and —changing the value of the token in the place CurrentAngle (to contain the new value). —Changing the value of the token hold by the place CurrentAngle leads to activating the timer contained in transition tiltAngleModificationTimeout (which is set to 30 000 milliseconds represented as [30000] in the transition), ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:14

•

D. Navarre et al.

which means if no modification is made on the tilt angle before 30 seconds the tilt angle will be automatically set back to 0, and a notification will be sent to the dialog part. Table IV presents more exhaustively the features of CO, dividing them into two aspects; structural aspects (lefthand column) and behavioral aspects (righthand column). The two lines in the table present what part of the description is specific to Object Petri Nets and which one is specific to the CO notation. 3.2.3 Interactive Cooperative Objects UIDL (ICO). ICOs are used to provide a formal description of the dynamic behavior of an interactive application. An ICO specification fully describes the potential interactions that users may have with the application. The specification encompasses both the “input” aspects of the interaction (i.e., how user actions impact on the inner state of the application, and which actions are enabled at any given time) and its “output” aspects (i.e., when and how the application displays information relevant to the user). In the ICO UIDL, an object is an entity featuring four components: a cooperative object which describes the behavior of the object, a presentation part, and two functions (the activation function and the rendering function) which make the link between the cooperative object and the presentation part. Figure 5 presents the metamodel of the ICO UIDL and makes prominent these four parts. The CO part is an excerpt of Figure 3 and contains only three items (marking, substitution, and instance) directly connected to the ICO metamodel. Indeed, the entire ICO metamodel is the sum of Figure 3 and Figure 4. As stated earlier, we present how ICOs are used for describing an interactive application using the WXR application presented in the introduction of Section 3. Interactive applications in the area of interactive cockpits correspond to classical WIMP interfaces and this example is used here just to illustrate basic concepts involved in ICO, and how WIMP interfaces are handled. We thus successively present the four ICO parts for that application. Later in the article, other examples will show the expression capabilities for post-WIMP interfaces. Presentation part. The presentation of an object states its external appearance. In the case of a WIMP interface, this presentation is comprised of a structured set of widgets organized in a set of windows. Each widget is a means for the user to interact with the interactive system (user → system interaction) and/or a means for the system to present information to the user (system → user interaction). The presentation part involved in an ICO specification corresponds to the logical presentation part of the Arch architecture (Figure 1). This means that the way used to render information (either in the ICOs description and/or code) is hidden behind a set of rendering methods (in order to render state changes and availability of event handlers) and a set of user events, embedded in a software interface, in the same language as the one used for the COs interface description. For WIMP interfaces, the physical presentation part of Arch does not need to be explicitly represented with an ICO description, as it is linked to implementation aspects such as the graphical toolkit used, and the embedded input ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:15

Table IV. Key Features of the Cooperative Objects Description Technique OPN specific

CO specific

Structural Aspects Typed Place and tokens: Each place is a typed tuple that may hold a set of tokens which are tuples of values of the corresponding type. Transition with Actions and Preconditions: Transitions may be constituted of a precondition (expressed as a predicate depending of the values of tokens in the input places) and actions on the values of the tokens in the input places. Variable names on arcs: Each arc is labeled with a tuple of variable names, in order to make actions and preconditions of a transition able to handle tokens values. Inhibitor and Test arcs: Inhibitor arcs are used as zero tests allowing the enabling of transitions if the input places are empty; test arcs are used for testing presence and values of tokens without removing them from the input place while firing the transition. A software interface describes the set of public methods offered by the object. The syntax used to express this set is the Java syntax as it is precise enough for describing method signature and basic types. Several communication protocols: • Unicast synchronous communication (e.g., method calls): Cooperative objects may communicate using method calls, where these methods are the ones from the associated software interface. A binding mechanism provides a translation of one signature of one method into a set of special places in the Petri net itself (called Service Input Port (SIP), Service Output Port (SOP), and Service Exception Port (SEP)), which corresponds to input or output or exception parameters of the method. The semantic of this way of communication is based on a rebuild mechanism, which describes the communication as a Petri net. • Multicast asynchronous communication (e.g., event communication): Cooperative objects may provide a set of events to which other cooperative objects may listen. The CO formalism defines ways to add/remove listeners, to trigger events and catch events using event handlers: • Any cooperative objects proposes two methods to allow any object which implements the software interface EventSink to be notified when the corresponding event is triggered: • void addEventListener(String eventName, EventSink aListener); void removeEventListener(String eventName, EventSink aListener); • In any transition of a cooperative object, an event may be triggered using the following special instruction: trigger(String eventName, EventObject anObject); • Any cooperative object which listens to the event triggered by another cooperative object may receive this event by the firing of a special set of transitions called an event handler, whose name is linked to the name of the event triggered. Each event handler transition name is the result of the concatenation of the event name, the character “ ”, and some other text (e.g., eventName someText).

Behavioural Aspects Enabling of transition: The rule remains the same as for classic Petri nets (e.g., enough tokens in each input places) but, in addition to the presence of tokens, the precondition (if any) is evaluated using the values of tokens (called substitutions). Firing of transitions: The firing is similar as in classic Petri nets, but the value of the produced tokens may be the result of actions inside the transitions. Additionally, the action inside the transitions may result in a method call to another CO or in the raising of an event (that is caught by registered OPN). Instantiation: Every ObCS can be instantiated and allows multiple executions of objects from the same class as with object-oriented approaches and can be parameterized by constructor arguments that associate a marking to the instantiation. For example, in a case of a multiple mouse interaction each mouse driver has a distinct instance of an ObCS class with different class parameters and so each driver has its own coordinates that can be found in the marking of the instance. Observability: The execution of the ObCS is fully observable, based on an event mechanism providing extra registration methods dedicated to the observation of the transition firing and the place marking evolution: • Observation of the firing of transition is allowed by the methods void addFiringListener(String eventName, EventSink aListener); where eventName is “transition fired.” void removeFiringListener(String eventName, EventSink aListener); where eventName is one of “to“token removed” or ken entered”, “marking reset”. • Any cooperative object interested in receiving these particular notifications must be registered and must offer the corresponding event handlers. This feature exploits the design pattern called Observer [Gamma et al. 1994], e.g., it provides means for the notification of state changes (e.g., marking changes for places), transitions, and event handlers availability and firing.


18:16

•

D. Navarre et al.

Fig. 5. Metamodel of the interactive cooperative object UIDL.

and output device. The abstraction of the presentation part proposed by the ICO notation allows the use of ICO for the modeling of every kind of interactive application (from classical WIMP interfaces to complex multimedia rendering), as the description philosophy is to encapsulate the presentation part into a software component offering a dedicated fac¸ade (such as the example presented by Figure 6, where the software interface is presented making explicit the set of events and the two sets of rendering methods). The interesting aspect of this description philosophy is that ICO descriptions can be easily connected to a dedicated description technique such as UsiXML [Limbourg et al. 2005] for classical WIMP interfaces, or SVG [2003], or classical program code, and hidden behind a fac¸ade corresponding to the logical presentation part in the Arch architecture. Cooperative object. Using the cooperative object description technique, ICO adds the following features. —links between user events from the presentation part and event handlers from the cooperative object description; —links between user events availability and event handlers availability; and —links between state in the cooperative object changes and rendering. As stated in Section 3.2.2, a CO description is made up of a software interface and its behavior is expressed using Petri nets. The WXR page does not offer public methods (except the default ones for allowing the event mechanism), ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:17

Fig. 6. Software interface of the page WXR from the user application MPIA.

Fig. 7. Behavior of the page WXR.5

and this is why there is no software interface here. Figure 7 shows the entire behavior of page WXR, which is made of two unconnected parts. The Petri net in the upper part handles events received from the five CheckButtons (see Figure 2 for the presentation part). Even though there are CheckButtons4 the actual behavior of there application makes it only possible 4 This

terminology for widgets of interactive applications in cockpits is defined in the ARINC 661 standard [ARINC 661, 2005]. 5 While the main part of the figures representing Petri nets are snapshots of the editing using the PetShop CASE tool, this figure and some additional ones have been edited using a classical graph editor to improve its readability.


18:18

•

D. Navarre et al.

Fig. 8. Activation function of the page WXR.

to select one of them at a time. The current selection (an integer value from 1 to 5) is carried by the token stored in MODE SELECTION place and corresponds to one of the possible CheckButtons (OFF, STDBY, TST, WXON, WXA). The token is modified by the transitions (new ms = 3, for instance) using variables on the incoming and outgoing arcs as formal parameters of the transitions. The Petri net in the lower part handles events from the 2 PicturePushButton and the EditBoxNumeric. Interacting with these buttons will change the state of the application. In the current state, this part of the application is in the state manual state (in which a token is in NOT AUTO place, and a token is in the STABILIZATION OFF place). This configuration of tokens is required in order for the edit box to be available to the user (visible on the model as transition change Angle T1 is in a darker color). Activation function. For WIMP interfaces, user → system interaction (inputs) only takes place through widgets. Each user action on a widget may trigger one of the CO event handlers. The relationship between user services and widgets is fully stated by the activation function that associates each event from the presentation part to the event handler to be triggered and to the corresponding rendering method for representing the activation or the deactivation. When a user event is triggered, the activation function is notified (via an event mechanism) and requires the CO to fire the corresponding event handler providing the value from the user event. When the state of an event handler changes (i.e., becomes available or unavailable), the activation function is notified (via the observer and event mechanism presented before) and calls the corresponding activation rendering method from the presentation part with values coming from the event handler. The activation function is fully expressed through a mapping to a CO behavior element. Figure 8 shows the activation function for page WXR. Each line in this table describes the three objects taking part in the activation process. The first line, for instance, describes the relationship between the user event ask off (produced by clicking on the CheckButton OFF), the event handler off (from the behavior), and the activation rendering method setWXRModeSelectEnabled from the presentation part. More precisely, when the event handler off becomes enabled, the activation function calls the activation rendering method setWXRModeSelectEnabled providing it with data about the enabling of the event handler. On the physical interaction side, this method call leads to the activation of the corresponding widget (i.e., presenting the CheckButton OFF as available). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:19

Fig. 9. Rendering function of the page WXR.

When the button OFF of the presentation part is pressed, the presentation part raises the event called asked off. This event is received by the activation function which requires the behavior part to fire the event handler off (i.e., the transition off T1 in the Petri net of Figure 7). Rendering function. System → user interaction (outputs) present to the user the state changes that occur in the system. The rendering function maintains the consistency between the internal state of the system and its external appearance by reflecting system states changes on the user interface. Indeed, when the state of the cooperative object changes (e.g., marking changes for a given place), the rendering function is notified (via the observer and event mechanism) and calls the corresponding rendering method from the presentation part with tokens or firing values as parameters. In a similar way as for the activation function, the rendering function is fully expressed as a CO class. The rendering function of the WXR application is presented in Figure 9. In this table one line describes the three objects taking part in the rendering process. The first line, for instance, describes the relationship between the place MODE SELECTION, the event linked to this place (and in which we are interested in token enter), and the rendering method showModeSelection from the presentation part component. The significance of this line is that when a token enters the place MODE SELECTION, the rendering function is notified and calls the rendering method showModeSelection, providing it with data concerning the new marking of the place that is used as parameters of the rendering method. 3.3 ICOs for the Description of Interaction Techniques This section presents how to use the ICO description technique to describe what contains the physical interaction component of the Arch model. The key point is to describe how raw events coming from the input devices are translated into higher-level events used by the logical interaction component. This is a very important aspect for next-generation interfaces, as these interfaces typically involve new input or output devices whose events have to be processed and managed within the interactive system. A good example of such an issue can be found with the makers of the Wii game console regularly proposing new devices [Schlomer et al. 2008]. The various layers represented in Figure 10 handle events at a different level of abstraction from lower-level events (directly produced by input devices) to dialog-level (or task-level) events. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:20

•

D. Navarre et al.

Fig. 10. Decomposition of the Arch physical interaction component.

Fig. 11. Graphical component to edit the tilt angle.

—The first level (physical interactions) handles raw input events from the input devices (where the model reads on a regular the state of the device and produces corresponding events). —The second level contains descriptions which manage higher-level events (such as clicks, drag, etc.) by combining raw events. —The third level manages the connection between the higher-level events and the graphical components (higher events are used to find the graphical components they are related to, and then the graphical components handle these events according to their inner states to finally produce widget-level events). —The fourth level handles the events produced by the graphical components and dispatches them to event handlers (such as the ones generated by user interface management systems). In our approach, ICO models are used at each level. This architecture has been used for the formal description of many interaction techniques, including gesture, voice, and multimodal interactions (two-handed, voice and speech, etc.). As an illustration, we introduce a graphical component for the editing of the tilt angle to replace the edit box in the WXR application (Figure 11 shows the graphical representation of this component). This component allows the selection of an angle by dragging the arrow that represents the current angle. As presented, the current angle must remain between the bounds. 3.3.1 Raw Events. Figure 12 presents a model handling raw events from an input device (here the input device is a Keyboard Cursor Control Unit (KCCU) which is available in interactive cockpits and behaves like a mouse) and producing upper-level events such as mouseMove, mousePressed, ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:21

Fig. 12. Model of the raw events handling for both configurations.

mouseReleased, etc. This model can be read as follows: First the value of a variable “changed” defined in transition CheckMouseState (upper transition in Figure 12) is tested every 100 milliseconds ([100] at the bottom of the transition) in order to verify if the state of the input device has changed since the previous check. According to the value of the variable, transition hasNotChanged or hasChanged will fire. Then, two further tests are performed in order to identify the movement of the KCCU and the state of the button. The movement test is represented by transition axisChanged, (lefthand side of the model) according to (x, y) coordinates (mouseMove). Transition buttonChanged (righthand side of the model) checks to see if there has been a change in the state of the KCCU button6 which produces mousePressed or mouseReleased events (code “trigger” in the respective transitions). After the axisChanged and buttonChanged tests, transition done is fired, placing a token in place MouseNumber to be ready to again start the process. 3.3.2 Higher-Level Events. Figure 13 represents a typical interaction technique that produces simple or double clicks and events corresponding to the dragging activity. The model receives events mousePressed, mouseReleased, 6 In

the current model the KCCU only features one button.


18:22

•

D. Navarre et al.

Fig. 13. Example of model producing higher level events.

and mouseMoved from the raw events model presented in Figure 12 which, then processed to raise click, doubleClick, beginDrag, drag, and endDrag events. The cooperative object presented in Figure 13 behaves as follows. —The doubleClick event occurs when the mouse button has been pressed and released twice within a predefined temporal window and without any mouseMove event in between (firing of transition mouseReleased T2 and code trigger raising doubleClick). —If a mouseMove (firing of transition mouseMove T3) or a timeout (timer T1) occurs after the first sequence of mousePressed, mouseReleased (firing of transition mousePressed T1 and mouseReleased T1), a simple click event is produced. —If a mouseMove occurs in between the first sequence of mousePressed and mouseReleased, a beginDrag event is triggered (firing of transition mouseMove T1), and then the model is able to produce drag events while receiving mouseMove events, or to produce an endDrag if the mouse button is released. —If a mouseMove occurs in between the second sequence of mousePressed and mouseReleased, a simple click is triggered at the same time as a beginDrag event (firing of transition mouseMove T4). Afterwords, the model is able to produce drag events each time a mouseMove event is received, or to produce an endDrag event in the event that mouseReleased is received. —If a timeout occurs in between the second sequence of mousePressed and mouseReleased (corresponding to the fact that the user has been too slow for performing the second click), a simple click is triggered and the model then behaves as if only the first mousePressed has occurred (firing of transition timer T2). 3.3.3 Dispatching Events to Picked Graphical Components. Dispatching events to graphical components (widgets) requires being able to get these components from the set of graphical components on the user interface. This ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:23

Fig. 14. Example of a picking manager.

Fig. 15. Software interface of presentation part of the angle selector component.

mechanism is called a “picking mechanism.” An example of such mechanism modeled with ICOs is presented in Figure 14. The modeling principle here is to identify (when an event mouseMove is received) which graphical component is under the mouse cursor. If a component is identified, every event (among mousePressed, mouseReleased, click, doubleClick, beginDrag, drag, and endDrag) is then forwarded to it (possibly resulting in an adaptation of the coordinates). In the current state of Figure 14 one object has been picked and that object (of which the reference is stored in place MouseOnObject) is ready to receive the set of events (mouseMove, mouseClick, mousePressed, mouseReleased, and beginDrag) as the corresponding transitions are enabled (graphically represented in dark gray). The graphical component able to receive these events must handle them. This handling might have an impact on the inner state of the component as well as on its graphical appearance. To describe such graphical components, we use part of the ICO notation: a cooperative object, a logical presentation part, and a rendering function. There is no need for an activation function as there is no deactivation of the events production). Figure 15 shows the set of methods used to render the state changes of the angle selector component (showDraggingArrow which is used to render the dragging of the arrow showing the selected angle; showArrow which renders the current value of the angle by drawing the arrow; showEnabled which renders the component is enabled or not depending on the value of its parameter). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:24

•

D. Navarre et al.

Fig. 16. Behavior of the angle selector component.

Fig. 17. Rendering function of the angle selector component.

Figure 16 presents the behavior of the angle selector. —The upper part defines whether the component is enabled or not according to the service call of setEnabled. A token is placed in the place Enabled or NotEnabled depending on the value of the parameter of the method call (that can be TRUE or FALSE). —The lower part handles the dragging of the arrow. The token in place CurrentAngle holds the current value of the angle. If the component is enabled and the event beginDrag is received, the token is moved to place DraggingArrow which is updated for every occurrence of a drag event. The code inside the transitions beginDrag and drag corresponds to retrieving the mouse cursor coordinate from the event received, and computing through a dedicated method the value of the corresponding angle. When the endDrag event is received, the component triggers the event called A661 SELECTION7 that is managed by the rest of the application. Figure 17 presents the rendering function which relates state changes to rendering. For instance, the first line describes that when a token enters the place CurrentAngle, the method showArrow is called with the value of the angle (held by the token) as a parameter (see Figure 15). With these three elements the angle selector component is fully described, and may be used within a window as described in the next section. 7 This

name in defined in ARINC 661 specification standard [ARINC 661, 2005].



•

18:25

Fig. 18. Integration of the angle selector within the WXR application.

3.3.4 Dispatching Component-Level Events to Event Handlers. Figure 18 shows the graphical integration of the angle selector component within the WXR application. Beyond the graphical integration, the integration has to deal with: —handling the event produced by the component (called A661 SELECTION); and —linking the event handling the event production at the logical interaction level (production of the event asked changeAngle from Figure 6). This integration depends on the technology used to implement the physical interaction part. For instance, using Java language, the event handler will be represented by a dedicated listener (angleSelectorListener) which will receive the notification of the A661 SELECTION and then throw the call of a dedicated method (void A661 SELECTION(A661SelectionEvent anEvent)). Then, in this method, some code would be dedicated sending a notification to the logical interaction level with a line such as getPhysicalPart().notifyEvent(“asked changeAngle”, someValue). Another possibility would be to model a cooperative object with a similar behavior. This section has presented how the ICO UIDL can be used not only for modeling interactive applications but also for modeling an interaction technique. These two elements have been used as running examples for presenting the basic characteristics of the ICO UIDL. More complex examples are presented in Section 5, which is dedicated to case studies. Before that, the next section presents PetShop, a CASE tool supporting the ICOs notation. 4. PETSHOP: A CASE TOOL SUPPORTING THE ICOS NOTATION This section presents how the activities for describing an interactive system (using ICOs) are supported by a tool suite. Thanks to Petri nets, an ICO specification is fully executable, which makes possible to prototype and to test an application before it is fully implemented [Navarre et al. 2000]. The models produced can also be validated using analysis and proof tools developed within the Petri nets community and extended in order to take into account the specificities of the Petri net dialect used in the ICO formal description technique. Section 4.1 presents the main tools involved in ICO models descriptions, while Section 4.2 details in particular (through an example) how the tools may be involved in a rapid system prototyping activity. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:26

•

D. Navarre et al.

Fig. 19. Abstract behavior of PetShop (in gray user-triggered activities, in white autonomous activities).

4.1 Modeling Activities of an ICO Description Editing an ICO specification requires the definition of the four parts that constitute it, namely, the behavior described using a cooperative object, the logical interaction part, and the activation and rendering functions. We discuss in the following sections the editing of these four parts. 4.1.1 Editing the Behavior Part. A complex part of the editing of an ICO specification is its behavior. Indeed, building or modifying a Petri net model is highly demanding. To support such complex activity, in PetShop, usability has been carefully considered for the three main tasks involved in modeling activities: editing, simulation, and analysis [Barboni et al. 2003]. Figure 19 represents the high-level structure of PetShop showing that the three tasks identified previously are supported. In PetShop, it is possible to continuously edit, execute, and analyze the instances of ObCSs with a complete interleaving. —While editing an instance of an ObCS (modifying structural aspects such as adding/removing place, transitions, etc.), the ObCS (which is the class) is updated, and this leads to updating all the instances of this ObCS. —The execution of any instance of the same ObCS is independent from the execution of the other instances of the same ObCS, and this execution may be fully controlled by the user of PetShop (in terms of starting, pausing, stopping and executing the instance in a step-by-step way). This execution is not stopped by the modification of the model of the instance. —While editing and running an instance, the corresponding ObCS is continuously analyzed (place invariants, transition invariant, etc.). The CASE tool PetShop (illustrated in Figure 20) allows classical manipulations on the Petri nets add/remove/modify (Petri net items, marking, code within transitions, etc.) and offers classical editing services (copy/cut/paste, undo/redo, navigation amongst the models, etc.). In PetShop a toolbar (upper righthand side of Figure 20) allows the user to start, stop, or pause an instance of the ObCS. There are two kinds of execution of instances: ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:27

Fig. 20. Snapshot of PetShop CASE tool.

—a normal execution in which the user is a spectator of execution and visually controls the execution of an instance (transitions are fired using random enabling substitutions); and —a step-by-step execution in which the user can select a substitution to fire the transition. While running, the execution of instances gives the following feedback to the user: —the marking is shown by the number of tokens in a place; —the fireability of transitions is shown by color changes: purple for fireable or gray for not fireable; —the firing of a transition and the updating of the marking; and —analysis of models: The results of the algorithms of analysis are continuously updated and displayed on a dedicated tab called “Analysis” on Figure 20. External services. PetShop also provides observability and controllability services via an API for externals programs. Observability services send events to subscribers when markings change, substitutions change, and events are fired in transitions. Controllability services receive events from external sources and fire the related transition of a user service. A logging tool has been added (log tab in Figure 20) to log traces of execution into an external file, allowing further analysis (mainly for performance and usability evaluation [Bernhaupt et al. 2007]). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:28

•

D. Navarre et al.

4.1.2 Editing the Other Parts. Logical interaction. As stated in Section 3.2.3, the physical presentation part of the Arch architecture is not directly addressed by an ICO specification, as it is linked to implementation aspects such as the graphical toolkit to use or the input and output devices, and so on. This implies that the use of dedicated tools that support these specificities and a packaging activity (e.g., hiding implementation behind a software interface such as described in Section 3.2.3) must be compliant with the description of the logical interaction part needed for the ICO specification. The editing of the physical part is thus possible with almost any UIMS or specific tool, the only constraint being to be able to produce a Java object with an ICO-compatible software interface. For instance, previous work on such connection with development tools involved: —a third-party interface server used on a testbench for interactive cockpits for civil aircraft [Barboni et al. 2006b]; and —SVG-compatible technologies to represent the graphical part in Barboni et al. [2006a]. Activation and rendering functions. Activation and rendering methods are expressed in a declarative way using a tabular expression. The inner structure of these two functions is really simple so that it could be expressed with any plain text description (such as XML). This simplicity comes from the fact the two functions simply link the observability and controllability mechanisms of PetShop tool to the methods defined in the logical interaction part. Usually, we propose a dedicated Java API based on Java reflection mechanism as a default tool to implement the two functions; however, we have also described them by means of Petri nets (ObCSs) in Barboni et al. [2006a]. 4.2 Illustration: Rapid System Prototyping By providing a way of performing both model execution and model editing at the same time, PetShop allows for rapid prototyping, iterative, and modeless construction of interactive applications [Navarre et al. 2001]. Figure 21 presents a snapshot of PetShop at runtime; the small window on bottom left part of the picture corresponds to the execution of the presentation part of WXR application driven by the execution of the Petri net in the bigger window beneath. As shown on Figure 21, PetShop allows (at the same time) the editing (it is possible to add/remove places, transitions, and arcs, change marking, etc.) and simulation (while editing it is possible to see the enabled transitions (shown in darker color in Figure 21)), and analysis (the set of P and T invariants are computed while editing and simulating (not visible in the figure)). Thus, at the same time, the user can interact with the application (use the widgets on the WXR application) and see the impact of his/her action on the Petri net model. Another possibility is to modify the model and see the impact on the application behavior. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:29

Fig. 21. Integrated program editing and execution within PetShop.

Within PetShop, prototyping from specification is performed in an interactive way. At any time during the design process it is possible to introduce modifications, either in order to make the specification more precise or to change it. The advantage of model-based prototyping is that it allows designers to immediately evaluate the impact of a modification while building a complete and unambiguous description of the behavior of the application. To be more concrete, we present how lexical and syntactic modifications are performed using PetShop on the WXR interactive application. 4.2.1 Lexical Modifications. Lexical part of the user interface gathers elementary elements of the presentation (for instance, the drawing of a button) and all the elementary actions offered to the user (such as clicking on a button). Lexical modifications concern the addition, removal, or modification of these kinds of elements. Changing the rendering for the current mode of WXR. When the first button named CTRL is pressed, the WXR application switches from automatic mode (presented as AUTO to the user) to manual mode (presented as to the user). As a lexical modification, we propose to change the graphical rendering of the automatic mode. The AUTO text will appear on a blue background (instead of the standard gray one). —At the Modeling Level. Nothing changes in the models. —At the Code Level. The content of the method showAuto must be modified and this must be done using the IDE used to implement the physical interaction part. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:30

•

D. Navarre et al.

Fig. 22. Modified behavior of the picking manager to handle control click events.

Changing the event triggering the action of buttons. As demonstrated in Section 3.3, ICOs are able to describe interaction techniques. In the current set of models, when a mouse click is detected, the mouse clicked event is sent to the picking manager and the picked widget is asked to react to that event. As a modification, we propose to change this behavior and to use a combination of control and click instead of a simple click. In order to encompass such change we need to modify the following. —At the Modeling Level. The mouse clicked event is produced by the model that handles raw mouse events (presented in Figure 13) and sent to the model responsible in picking widgets and providing it with higher-level events (presented in Figure 14). A modified model of the picking manager needs to receive keyboard events too, so as to be able to store the state of the control key (e.g., store whether the control key is pressed or not), and when receiving the mouse clicked event, the picking manager asks the picked widget to act if and only if the control key is pressed too (framed part of Figure 22). —At the Code Level. No modification is necessary. 4.2.2 Syntactic Modifications. The syntactic part of the user interface describes the links and relationships between the lexical elements (for instance, clicking the first CTRL button allows clicking on the second one, which then in turn enables the edit box). Modification of the mode selection. In the model presented in Figure 7, changing the mode (in the mode selection part of the window) is always available (whatever tilt selection is selected). Another potential behavior could be to allow the mode selection only if the weather radar TILT angle selection is in the automatic mode. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:31

Fig. 23. Modified behavior for the WXR application.

—At the Specification Level. Adding test arcs from the place AUTO (from the model presented by Figure 7) to the five transitions off T1, wxa T1, wxon T1, test T1, and stdby T1 will make these transitions available only if there is a token in the place AUTO. In other words, the five corresponding event handlers will be available only if the WXR application is in auto mode (as shown on Figure 23). —At the Code Level. No modification is necessary.

Modification of the visualization of possible tilt angles. In the current version, the tilt angle is edited through a classical edit box. As proposed in Section 3.3, the editing technique could be different, using the angle selector component. —At the Modeling Level. The modification is different if the ICO description has been used (or not) for describing the behavior of the widgets. If so, the modification of the angle selector component should be done in the same way as presented in Section 3.3. If the component is not described using ICO, no modification to the models is required. —At the Code Level. If the component is fully described using ICO, only the physical interaction part of the graphical angle selector must be coded. On the other hand, if the component is not described using ICO, handling of events and its behavior must be coded too (and should be behaviorally equivalent to the one described in Section 3.3). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:32

•

D. Navarre et al.

5. CASE STUDIES/PROJECTS DEVELOPED WITH ICOs AND PETSHOP The ICO UIDL has already been applied to different application domains such as air traffic control interactive applications [Navarre et al. 2002], space command-and-control ground systems [Palanque et al. 2006], interactive military cockpits [Bastide et al. 2005], or civil cockpits [Barboni et al. 2006b]. We present in this section four of these projects as an illustration for the expression capabilities of the notation, highlighting three particular aspects: description of multimodality, description of dynamic space of interaction, and in-depth description of WIMP applications. The underlying aim of this section is to provide the reader with detailed enough information to assess the positioning of ICOs with respect to the state-of-the-art tables in Section 2. The formal analysis aspects of the case studies are presented in Section 6. 5.1 Ground Segment of Satellite Control: Multimodality The case study presented here addresses the problem of the specification of a space ground system application studied by the project called IMAGES, funded by the CNES (French space agency) [Palanque et al. 2006]. The application exploits multimodal interaction techniques for the manipulation of a 3D representation of a DEMETER satellite, which stands for Detection of Electro-Magnetic Emissions Transmitted from Earthquake Regions.8 One of the key features of this work was to be able to fully describe a multimodal interaction technique in an interactive application and to assess the performance of trained operators with this kind of technique. This application features multimodal interaction techniques to an operator in charge of moving the point of view in a 3D model of a satellite. This navigation can be done either by rotating the 3D model of the satellite directly, using the mouse on the 3D image, or using in a multimodal way the two control panels presented Figure 24. The control panel (left part of Figure 24) entitled “point de vue” allows the operator to manipulate the current position of the point of view of the 3D image using the set of buttons in the top righthand side of the figure. The set of buttons in the “Orientation” subsection allows one to rotate the satellite image in any direction. The set of button pairs in the “Position” subsection allows one to move the satellite image in any direction (up/down, left/right, and backward/forward). We do not present the other parts of the user interfaces as they are related to functions beyond the scope of this article, and we only focus here on the multimodal interaction on the button pairs. Figure 25 shows an example of multimodal interaction for this application. On this figure, the user is concurrently using three input devices: two graphical input devices (a mouse and a trackball9 ) and a speech recognition system. The speech recognition system only reacts to two different words: fast 8 More

information about this satellite’s functions and missions can be found on http://smsc.cnes. fr/DEMETER/index.htm. 9 From now on, we refer to these input devices as right mouse and left mouse. Indeed, the type of input device considered (trackball, graphic tablet, . . .) has no impact on the modeling, provided it triggers the set of expected events. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:33

Fig. 24. The 3D representation of the satellite and its control panel.

Fig. 25. One of the multimodal interactions in the application.

and slow. The interaction takes place in the following way: At any time, the user can use any mouse to press the graphical buttons that change the point of view. In Figure 25, the button that moves the satellite image backwards has been pressed using the right mouse. Simultaneously, the left mouse is positioned on the button moving the satellite image to the left. At that time, the 3D image has already started to move backwards, and as soon as the other button is pressed, the image moves both backwards and to the left. The user is also able to increase or decrease the movement speed by uttering the words fast and slow. In Figure 25, the word “fast” has been pronounced and recognized by the speech recognition system. This action reduces the amount of time between two movements of the image. Indeed, the image is not moved according to the number of clicks on the buttons, but according to the amount of time the buttons are kept pressed by the user. In this multimodal application, there is no fusion engine, per se; the two mice are handled independently, and the speech interaction affects movement speed regardless of the user’s input performed with the mice. Another multimodal ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:34

•

D. Navarre et al.

Fig. 26. Software architecture of the multimodal interactive application.

interaction technique involving a fusion mechanism has already been described using ICOs and the interested reader can look at Bastide et al. [2004]. The modeling is structured according to the software architecture presented in Figure 26. This architecture is fully compliant with the one presented in Figure 10. The righthand side of the figure presents the three input devices. To handle this set of input devices, we use ICO models to translate raw input events into higher-level events which are used by three other ICO models: a picking manager responsible in providing a picked button pair with mouse events, a set of button pairs behavior models (here there are six instances of the model, one for each pair) and a “continuous move” which combines events from button pairs and from speech recognition to produce requests of view point changes to the functional core. The ICO model in Figure 27 represents the complete and unambiguous temporal behavior of the speech-based interaction technique, as well as how speech commands impact the temporal evolution of the graphical representation of the 3D image of the satellite. Taking into account the current marking of the model of Figure 27 (one token in places delay, idle, and core), only the three transitions startMove 1, faster , and slower are available. These transitions describe the multimodal interaction technique available (i.e., how each input device can be used to trigger actions on the system). Transitions faster and slower are triggered when the user utters one of the two speech commands “fast” and “slow.” In the initial state, these commands are available (visible on the model as transitions are painted in dark gray) and will remain available until the upper limit or the lower limit is reached (delay> 1000 for transition slower and delay< 100 for transition faster, respectively, as shown in the code of the transitions). Figure 28 presents the model of the second ICO class of the application. This class is responsible for describing the behavior of each button pair. By button pair, we mean the buttons that are performing opposite actions like (up, down), (left, right) and (backward, forward). These three button pairs are represented on the righthand side of Figure 25. To model these incompatible behaviors, the ICO description represents the fact that the user can press either the positive or negative button (e. g., up being the positive and down being the negative). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:35

Fig. 27. ContinuousMove: Model of the temporal evolution of movements driven by speech.

Fig. 28. ButtonPair: Mutual exclusion of the pair of buttons for changing the point of view.


18:36

•

D. Navarre et al.

Fig. 29. Snapshots of the radar image.

Connection between the button pairs and the input devices (the mice) is performed using the set of ICO models presented earlier in article, that is, in Figure 12. (for raw events handling), in Figure 13 (for higher-level events production), and in Figure 14 (for the picking manager). 5.2 Air Traffic Control Workstation: Dynamic Instantiation and Management of Interactive Components The work presented here is part of a case study that has been considered in the European Project MEFISTO10 which was a long-term research project dedicated to the design of safety-critical interactive systems, with a particular focus on the air traffic control application domain. This case study belongs to the en-route air traffic control domain and, more specifically, addresses design issues for applications involving data-link technologies for ground-air and air-ground communication. Using such applications air traffic controllers can communicate with pilots in a sector (a portion of the airspace) through digital commands. For instance, for asking the pilot to change the flight level of the aircraft, the controller interacts directly on the label of the aircraft represented on the radar screen and the corresponding request is directly sent to the aircraft. An example of such an application is given by Figure 29. Such communication mechanisms avoid bandwidth overload, which is critical with radio communication where all the communication between aircraft and ATC is shared by all the actors in the sector. In particular, we focus on the activities related to when an aircraft changes air sector (called transfer). One key feature of such an application is its dynamic space of interaction; the user must interact with the graphical representation of several planes, where both the number of planes and their appearance (in terms of date and position) are not predictable. Each time a new plane enters the sector, it is graphically represented on the radar image. It is possible for the user to select a plane by clicking on 10 http://giove.isti.cnr.it/mefisto.html.



•

18:37

Fig. 30. ObCS of the PlaneManager (left side), the Plane (top right), and the menu (bottom right).

its graphical representation. Clicking on the flight representation will change its state to the assume state, meaning that the air traffic controller is now in charge of this plane. The action of assuming the plane changes its graphical representation can be seen in Figure 29 (planes 5802, 6144, and 2858 are assumed while plane 3643 is not assumed yet). Once a plane is assumed, the controller can send clearances to this plane. In this case study we only consider the change of frequency clearance corresponding to the controller’s activity of transferring a plane to an adjacent sector. When the plane has been assumed, the button “FREQ” is enabled. Clicking on this button opens a menu allowing the controller selecting the new value for the frequency of the next sector (see plane 2858 on the righthand side of Figure 29). We only consider here a subset of the set of classes and objects of the case study specification, which contains three cooperating classes for which the ObCS is represented on Figure 30. —PlaneManager is in charge of handling the set of planes in a sector. —Plane handles the possible states of a plane (not assumed, assumed, or terminated) and the related possible actions. —Menu represents the behavior of the menu for selecting a frequency. The key point in this set of models is the way dynamicity of the interaction space is handled. Using places that hold references to the different planes, the ObCS of PlaneManager is able to store all the planes (without any constraints on the ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:38

•

D. Navarre et al.

number of these planes) in the places called Planes, AssumedPlanes, OpenedPlanes, or TerminatedPlanes, depending on their state. Such modeling is critical for describing direct manipulation interactions where widgets are dynamically created according to the opertaor’s actions. In addition, the ICO activation function (which is based on marking evolution and not marking descriptions) can handle such dynamic interaction space by linking the availability of an event handler (for instance, userAssume) and the availability of the corresponding user events (here, clicking the graphical representation to assume the plane). This linking takes full advantage of the fact that in Petri nets the state space is described “in intention” and not “in extension,” thus making it possible to describe in a simple model behaviors potentially exhibiting an infinite number of states. 5.3 Interactive Cockpits of Large Civil Aircrafts: WIMP Interfaces Compliant with ARINC 661 Specification The WXR application used as an illustration for the presentation of both the notation and the associated tool (Sections 3 and 4) is part of a project funded by DPAC (French Civil Aviation Research Programme), which aims at studying the specification of cockpit interactive applications based on the ARINC 661 specification. The Airlines Electronic Engineering Committee (AEEC) (an international body of airline representatives leading the development of avionics architectures) formed the ARINC 661 Working Group to define the software interfaces to the Cockpit Display System (CDS) used in all types of aircraft installations. The standard is called ARINC 661, Cockpit Display System Interfaces to User Systems [ARINC 661, 2005]. In ARINC 661, a user application is defined as a system that has two-way communication with the CDS (Cockpit Display System): —transmission of data to the CDS, possibly displayed to the flight deck crew; and —reception of input from interactive items managed by the CDS. According to the Arch architecture (see Figure 1), the CDS part (presented in Figure 31) corresponds to the presentation part of the entire system (made available to the crew members), and the set of UAs may be seen as the merge of both the dialog and the functional core. ARINC 661 then puts on one side the input and output devices (provided by avionics equipment manufacturers) and on the other side the user applications (designed by aircraft manufacturers). The consistency between these two parts is maintained through the communication protocol defined by ARINC 661. The architecture presented in Figure 32 proposes a detailed view on the different bricks involved in the specification both the user application and the CDS. Applications are executed in a Cockpit Display System (CDS) that aim to provide the flight crew with all the necessary information to ensure a safe flight. ARINC 661 is (currently) not used for primary cockpit applications (involved while flying the aircraft) such as PFD (Primary Flight Display) and ND ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:39

Fig. 31. Abstract architecture and communication protocol between cockpit display system and a user application.

Fig. 32. Detailed proposed architecture compliant with ARINC 661.

(Navigation Displays) and only deals with secondary applications (involved in the management of the flight) such as the ones allocated to the MCDU (Multiple Control Display Unit). For previous CDSs (such as the glass cockpit of the A320) these applications were not interactive (they only displayed information to the crew) and inputs were made available through independent physical buttons located next to the display unit. The location of such ARINC 661 applications (in the cockpit) in between the pilot and the first officer make it possible for both of them to interact with them. ICO is used to model the behavior of all the components of an interactive application compliant with the ARINC 661 specification. This includes each interactive component (called widgets), the user applications (UAs), the entire window manager responsible for the handling of input and output devices, and the dispatching of events (both those triggered by the UAs and by the pilots) to the recipients (the widgets or the UAs). The two main advantages of the architecture presented in Figure 32 are as follows. —Each component that has an inner behavior (server, widgets, UA, and the connection between UA and widgets, e. g., the rendering and activation functions) is fully modeled using the ICO formal description technique, thus making it possible to analyze and verify the correct functioning of the entire system. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:40

•

D. Navarre et al.

—The rendering part is delegated to a dedicated language and tool (both SVG (Scalable Vector Graphics) and Java raw code have been used in the project), thus making the external look of the user interface independent from the rest of the application. In addition, this provides a framework for easy adaptation of the graphical aspect of cockpit applications. The formal description of both MPIA application (that contains the WXR application) and the corresponding subpart of the CDS involve (at design time) a set of 30 models. At execution time, many models of the same class are instantiated and the running application involves more than 100 models. The work presented here has been extended to take into account potential malfunctions in the input (output, respectively) devices that allow the operators to provide (perceive, respectively) information or trigger commands (perceive command results, respectively) to the system. The introduction of a reconfiguration mechanism for both input and output devices has been presented in Navarre et al. [2008a, 2008b]. 5.4 Virtual Chess: Multimodal and Virtual Reality Application While previous case studies have presented mouse-centered interaction techniques, that is, user interfaces where the main interaction is performed using a graphical input device similar to a mouse, the current section shows that ICOs are able to address more unusual ones. This case study is extracted from Navarre et al. [2005] and presents more precisely how multimodal interfaces can be fruitfully exploited for manipulating objects in a virtual reality application. Virtual Reality (VR) applications exhibit specificities compared to classical WIMP (Window, Icon, Menu, and Pointers) interactive systems. Typically, VR applications are based on 3D representations, are usually multimodal, with complex interaction techniques where inputs and outputs can be very complex due to many different devices (data gloves, eye trackers, 3D mouse or trackball, force-feedback devices, stereovision, etc.). Using or implementing VR applications requires addressing issues like immersion, 3D visualization, handling of multiple input and output devices, or complex dialog design, etc. These issues make modeling and implementation of VR systems very complex, mainly because it is difficult to describe and model how such different input events are connect to the application [Smith et al. 1999]. Even though the interaction and visualization can be seen, at a higher level of abstraction, as continuous, when it comes to low-level modeling, events produced and processed are always dealt with in a discrete manner. This explains why we do not address the issue of continuity in our modeling framework. 5.4.1 Informal Description of the Virtual Chess Application. The virtual chess application is inspired from the traditional chess game. It was originally developed as a testing ground application to support user testing of the selection of 3D objects in VR environments [Nedel et al. 2003]. The interaction includes the manipulation (selecting, moving, releasing) of chess pieces and the selection of the view mode (plan view or perspective view). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:41

Fig. 33. Some of the devices employed: (a) motion captor attached to a 5DT data glove; (b) 3D stereoscopic glasses; (c) scenario of user testing.

The manipulation of pieces can be done either by using a classic mouse or a combination of data glove and motion capture device. When using a mouse the selection is done by first clicking on the piece and then clicking on the target position. When using the data glove 5DT11 and a motion captor12 (such as the ones presented in Figure Figure 33 (a)), closing the hand is recognized as a selection, and opening the hand after a successful selection is recognized as a release. In this case, the motion captor is used to give the pointer position (x, y, z) while the fingers flexion is used to recognize the user gesture. The selection of the view mode is done by pressing the key 0 (for the top view) or key 1 (for the perspective view) on a classic keyboard. In addition to these input devices, a user can wear stereoscopic glasses (see Figure 33(b)) in order to have a stereo experience. Figure 33(c) provides the general scenario for the user physical interaction with devices. The users can move one piece at a time (horizontally, vertically, and/or in diagonal) and the virtual chess application does not take into account the game rules. All that the users can do is to pick up a piece, move it to a new position, and drop it. If a piece is dropped in the middle of two squares it is automatically moved to the closest square. Users cannot move pieces outside the chessboard, but they can move pieces to a square occupied by another piece (in such a case the piece comes back to its original position). In a real chess game, the movement of the pieces over the game board is performed with the hand. This has led to the implementation of the virtual hand interaction technique, which represents the pointer position by a virtual hand, as shown in Figure 34. Visual feedback is given by automatically suspending the selected piece over the chessboard and changing its color (from gray or white to red). 11 5DT

from Fifth Dimension Technologies (http://www.5dt.com/). of Birds from Ascension Technology (http://www.ascension-tech. com/).

12 Flocks


18:42

•

D. Navarre et al.

Fig. 34. User interaction using direct manipulation (virtual hand technique) with visual feedback. From left to right: picking, moving and dropping a piece.

Fig. 35. View modes: (a) perspective view; (b) top view.

Figure 34 and Figure 35(a) show the chessboard in the perspective view mode while Figure 35(b) shows it in the top view mode (2D view). 5.4.2 Modeling the Virtual Chess with the ICO Formalism. Based on the Arch architecture (as presented on Figure 1), modeling of VR applications leads to describing the behavior of input and output devices, the general dialog between the user and the application, and the logical interaction provided by the interaction technique. Input devices modeling. The behavior of the application is based on three main logical events: pick(p), move(p), and drop(p), where p represents the piece being manipulated. Figure 36 (upper part) presents the ICO models describing the behavior of the mouse (corresponding to an interface in which the entire interaction is performed using a mouse). Figure 36 (lower part) models similar behavior when a motion captor, a data glove, and a keyboard are used. These models represent how physical inputs are converted into logical events. When using a mouse (top part of Figure 36), these so-called logical events are represented as a composition of the low-level events mouseMoved(x,y) and mouseClicked(x,y), which are produced by the mouse driver. Each time the user moves the mouse, a mouseMoved(x,y) event is triggered and captured in the ICO model. The logical events pick(p) and drop(p) are associated to the lowlevel event click(x,y) that is produced by the mouse. The events pick(p) and drop(p) are determined by the sequence of low-level events (the first click(x,y) is interpreted as a pick(p) while the following click(x,y) is interpreted as a drop(p)). When using the data glove and the motion captor (bottom part of Figure 36) the logical events are produced after polling the state of the input devices. Every time an event idle() is triggered (the event idle is produced by the internal loop implemented by graphic libraries, such as OpenGL), it fires the transition init which captures the current fingers’ flexion from the data glove (g = fdGetGesture(FD)) and the spatial hand’s position from the motion captor (p = fobGetPosition(FOB)) storing that information in variables g and p, ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:43

Fig. 36. Two models capturing low-level events from input devices (upper part a mouse; lower part a combination of a motion capture and of a data glove) and producing logical-level events (pick, move, and drop).

respectively. The transitions pick, notPick, drop, and notDrop compare the current and previous position (which are stored in the token of place last). If the current position is different from the previous one and the hand is opened, the model triggers an event drop(p) in the current hand position. If the hand is closed and its position is different from the previous one, then the model triggers an event move(p). Dialogue modeling. In order to be compliant with the Arch software architecture, the dialog component must remain independent from the UI part, that is, from the input devices used to perform the tasks. This is the case with the model presented in Figure 37 as it receives and processes three logical events (pick(p), drop(p), and move(p)) whatever input device is used. Indeed, when receiving event move (one of the transition move 1, move 2, or move 3 will ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:44

•

D. Navarre et al.

Fig. 37. Model of the dialog controller.

be available) the current coordinates are updated (in place Idle, notPicked, or Picked, depending on the transition fired). After the reception of movement, when an event pick occurs (transition pick 1), the fusion of the events is possible (i.e., both the position and the related command pick are known). In order to perform the fusion, the fusion engine needs to know whether there is a piece on the chessboard at position p. This is performed in the code of the transition pick 1 (c=getCase(p)). The variable c is used to store the value of the piece at position p. From that state two options are possible: either c contains a reference to a piece or c is empty. The first case is handled by the righthand side of Figure 37 while the other one is handled by the lefthand side. If an event pick occurs and the place pieces contain a reference to square c, then the user can move the corresponding piece p (using the transition move 2) or drop it (using the transition drop 1). Otherwise, the user can only move the hand over the chessboard for a while (transition move 3) and then the system returns to the initial state. Rendering and interaction technique modeling. In this section we do not present the rendering functions related to the aforesaid models, but we discuss the underlying principle (more details may be found in Navarre et al. [2005]). The challenge we have to face with this application is to render the scene in a 3D view or in a 2D top view. To be able to render this scene, the global rendering process requires information to render from two independent sources: one related to the view mode (controlled by the keyboard) and the other related to the ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:45

dialog state (including the pieces and their location on the chessboard). More precisely, rendering can only be performed by fusing multimodal information available in two rendering functions. To overcome this challenge, we introduce extensions to the ICO formalism by adding rendering events. Such rendering events are added to the models each time a change in the state of the system modifies some information that has to be rendered in the graphical display. As in classical ICOs, this is done by the rendering function but in this case classical rendering methods are replaced by rendering events to trigger. In this case study, the physical rendering is done by means of a single 3D visualization display, but it can easily be extended to work with several output devices just by replacing the rendering method calls by other rendering methods (in such a case there will be modifications in the model to handle fission of events, that is, to define which information is presented on which input device) or other events captured by another ICO model describing output devices (in this case, one ICO model is required for each device). 6. BENEFITS OF THE APPROACH AND PERSPECTIVES FOR NEXT-GENERATION USER INTERFACES In previous sections we have shown how ICOs notation is able to address the challenges raised by the description of user interfaces and how it also deals with the description of a large-scale application. Indeed, the interactive cockpit application presented in Section 3.2.3 is made of 97 models communicating together while the other case studies have presented various elements demonstrating the expressive power of the notation. These elements correspond to the items of the classification presented in Table III. The reason for focusing on the use and deployment of formal description techniques lies in the fact that they are the only means to provide modeling both in a precise and unambiguous way all of the components of an interactive application (presentation, dialog and functional core) and to propose techniques for reasoning about (and also verifying) the models. Applying formal description techniques can be beneficial during the various phases of the development process from the early phases (requirements analysis and elicitation, user interface prototyping, etc.) to the later ones including validation (usability testing, functional testing, etc.). We present in the following three sections the benefits of the approach in terms of expressive power and the capabilities of dealing with next-generation interfaces, in terms of formal analysis of models and in terms of exploitation of models for other purposes than reliability. 6.1 Addressing “Next User Interfaces” Challenges with ICOs The aim of the various case studies presented before was to demonstrate that ICOs are able to address many issues raised by user interfaces and interactive systems (see classification in Table III). This section aims at projecting these capabilities with respect to the requirements for next-generation interfaces. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:46

•

D. Navarre et al.

While it is by definition difficult to identify in exhaustive way what the nextgeneration interfaces will require from description techniques, it is possible to envision the needs from their evolution over past years. We have demonstrated in the various case studies that ICOs is a notation that is able to deal in a formal way with data representation (by means of object-based tokens in the Petri nets). States modeling is achieved both by the distribution of the tokens in the places of the Petri nets and by their value. Events triggering is represented in the code of the transitions by using the keyword “trigger” while events catching is done by means of a special type of transitions called “synchronized.” Time can be easily represented qualitatively by the structure of the Petri nets while quantitative time is represented by a delay associated to transitions. Concurrent behavior modeling is one of the strengths of Petri nets, which feature “true concurrency” semantics [Genrich 1991]. Lastly, dynamic instantiations are easily handled by storing references to instantiated objects as tokens in the places of a Petri net. This is particularly useful when interactive objects are dynamically created either via user action (e.g., creating new objects by means of a palette in a drawing tool) or by the system (e.g., new icons representing aircrafts in a sector on an air traffic control workstation). While these aspects are currently required for describing most current user interfaces and will still be required for the next generation, other ones are likely to become prominent in the near future. 6.1.1 Dealing with New Input Devices. New-generation interfaces will offer to users more and more exotic input devices. A good example of such evolution comes from game industry, such as the makers of the Wii game console [Schlomer et al. 2008] proposing a large variety of input devices based on the Wii-mote. As shown in the virtual chess case study, ICOs are able to manage input devices such as the Floc Of Birds by poling at a regular time interval the input device in order to capture motions performed by the users. Another integration has been performed using input device triggering events as, for instance, in the interactive cockpit application where the events are produced by the KCCU (Keyboard Cursor Control Unit). At a higher level of abstraction these events have to be handled in drivers not always provided in adequate ways by the hardware manufacturers. It is thus critical for the description technique to be able to represent such low-level time-based behaviors. We have presented in Section 3.3 how drivers can be modeled using ICOs in addition to an interaction technique. This connection between hardware and software elements is critical and is likely to become even more prominent with next-generation interfaces. 6.1.2 Dealing with Visualization Beyond Standard 2D Desktops. In a similar way as for input devices, output devices are likely to evolve and be more demanding for the description technique than standard 2D screens. Manipulating languages dedicated to these new output devices is thus required. As shown with the chess case study, ICOs are able to handle exotic output devices such as stereoscopic glasses. The satellite case study has shown how ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:47

Java 3D code can be integrated to the rendering function of ICOs. In other case studies not presented here [Bastide et al. 2004] we have used ICOs for handling 3D sound and head-up display outputs. Application domains such as gaming, (military) simulation systems, or advanced visualization tools will require more and more involvement of 3D rendering in the presentation part of interactive systems. This requires from user interface languages the ability to address in a simple and decoupled way such rendering techniques. 6.1.3 Dealing with Widgets. Nowadays, most interaction between users and systems takes place by means of predefined standard interactive objects called widgets. Even though introduced more than 20 years ago in the IBM CUA standard [IBM 1989] they are still largely used as they provide significant benefits both to users (making easy to predict how they have to be manipulated and how they behave) and to developers, and they are usually available in the programming environment and thus require little effort to be integrated. User interface description languages will be required to be able to describe current widgets (as they are defined by standards) but also future widgets that will be made available by next-generation programming environments. This is indeed of critical importance, as the sets of widgets will remain otherwise outside the scope of the user interface description language and thus will not be encompassed in the testing and verification of the interactive applications. In the case study of ARINC 661 specification we have described using ICOs 21 widgets from the standard (e.g., the angle selector in Figure 16). However, these widgets are very similar to the ones of the IBM CUA standard and do not encompass more advanced interaction techniques such as gesture or speech. 6.1.4 Dealing with Large Scale Applications. Most user interface description languages have so far remained at the prototype level, being tested only on case studies as a way to experiment with their capabilities to deal with one or several aspects of the interactive systems presented in Table III. Being able to deal with large case studies and real-world applications will become a requirement for these languages if their authors want to see them going beyond the research laboratories. In the interactive cockpit application we have been partly addressing such a problem. To deal with the large number of models executed simultaneously, the PetShop tool had to be restructured. Similarly, the fact that some models were of large size meant that the interaction and visualization techniques in PetShop had to be enhanced. However, even though the application was a real-life one, its complexity was limited with respect to other cockpit applications, as, for instance, the autopilot of the FMS (Flight Management System). Dealing with such applications surely will impose new requirements in terms of simulation time and management of models. 6.1.5 Dealing with the True Concurrency Requirement. Mass-market products such as the Wii game console [Schlomer et al. 2008] or the iPhone [Jobs et al. 2008] feature a native multimodal interaction either by means of several input devices (two or more Wii-motes, for instance) or by means of a multimodal device such as the multitouch tactile interaction on the iPhone. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:48

•

D. Navarre et al.

Being able to describe user interfaces for such new products will, certainly, become a necessity in the near future for next-generation user interface description languages. While, thanks to their “true concurrency” semantics, Petri nets will remain (with other underlying formalisms such as Pi-Calculus) a good candidate to handle such behaviors, these new products will be more demanding in terms of the number of concurrent threads, synchronization between threads, and response time. ICOs have been used in several domains for describing multimodal interfaces [Ladry et al. 2009] dealing with several different interaction techniques and input devices (speech, gesture, bimanual, etc.) at a time. However, the number of connected input devices was always limited and defined in a static way (not possible to plug-in and remove them). New devices such as multitouch screens will require the possibility to handle a large number of “input devices” (the fingers used for interacting on the surface) but also to handle them dynamically (as they can be added or removed while interacting). 6.2 Formal Analysis of Models The use of formal description techniques can be greatly enhanced if they are supported by formal analysis techniques. In the area of Petri nets, such analysis techniques can be used for the detection of deadlocks (when no transitions are available/fireable in the Petri net model), the presence or absence of a terminating state, the boundedness of the model, the liveness of the model, the reversibility and home state, or to verify requirements, that is, to verify properties of the system model such that a certain state can never be reached, for example, or that a given interface component is always enabled. Furthermore, certain analysis techniques can be used to extract scenarios of events leading to a particular state; this is useful for tracing history and for connecting interactive system behavior to incidents or accidents [Basnyat et al. 2006]. The analysis we consider is performed on the underlying Petri net (a basic place/transition Petri net simplification of the original Petri net). The advantage of the underlying Petri net is that the classical properties listed earlier are possible to prove with dedicated algorithms, but an important drawback is that properties verified by the underlying Petri net are not necessarily true on the original one. Thus, we usually use the result of classical property analysis as an indicator that highlights potential problems in the Petri net. Verifying these properties directly on the ICOs model is still a research challenge for the Petri nets community, that is still focusing on Petri nets with reduced expressive power such as colored Petri nets [Jensen et al. 2007]. 6.2.1 Structural Analysis. Petri nets provide a number of benefits for designers who wish to reason about the behavior of an interactive system. Numerous formal analysis techniques exist which can be used to analyze and verify the correctness of the models. Methods of analysis include checking for service availability, which can be used to identify deadlocks and analyze the model liveness (for every widget there will always be at least one sequence of events available on the user interface that will make that widget enabled). When modeling a system, it may ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:49

be important to ensure that the initial state is always reachable (through a sequence of actions). This is particularly relevant for safety-critical interactive systems that may be required to reset/restart/shutdown quickly in an emergency situation. Such properties allow evaluation of the quality of the model before system implementation. An example of structural analysis can easily be performed on the case study presented by Section 5.1. Figure 28 presents the behavior of a button pair for which the expected behavior is to have a mutual exclusion of both buttons of the pair (i.e., when pressing on of the two buttons the other one becomes disabled). To obtain this behavior, we model the button pair as two distinct processes sharing a same resource (a classical mutual exclusion) represented by a token put in place Idle. Each process is represented by a set of two transitions (XXXPress and XXXReleased , where XXX stands for both Positive and Negative) and a place (XXXOn) which represents the activity of the corresponding process (i.e., when marked the process XXX is working and thus “reserves” the shared resource). To ensure the mutual exclusion of the two processes we then have to prove that the three places are in mutual exclusion. The structure of the Petri net presented in Figure 28 ensures this property, thanks to the following structural properties. —The invariant analysis results in places PositiveOn, Idle, and NegativeOn and belongs to the same place invariant, that is, the sum of the marking of these three places is constant and equals to the initial marking of these places (M(PositiveOn) + M(Idle) + M(NegativeOn) = M0 (PositiveOn) + M0 (Idle) + M0 (NegativeOn), where M(X) is the function that provides the current marking of a place X and M0 (X) provides the initial marking of a place X). —The value of the initial marking M0 (PositiveOn) + M0 (Idle) + M0 (NegativeOn) is 1; it implies that M(PositiveOn) + M(Idle) + M(NegativeOn) = 1, which could be translated as each place contains at most one token, and if one of the places contains one token, the two others will be empty. The three places are then in mutual exclusion. A more detailed description on how to perform formal analysis of Petri nets models representing user interface behaviors can be found in Palanque and Bastide [1995]. 6.2.2 Reachability Analysis. Since our interest is in human-computer interaction, tracing scenarios of activities (both user and system) is particularly useful. A formal analysis technique based on marking graphs (called reachability analysis) may be used to identify the set of reachable states for a given model. The analysis starts from a known initial marking and identifies all the markings that a model can possibly reach, as well as the necessary conditions and event sequences that lead to each marking. However, from a safety perspective, forward reachability is not as effective as backward reachability for two main reasons. —It is often impractical to generate and analyze the entire reachable state spaces (this is particularly true for user interfaces offering sophisticated interaction techniques). ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:50

•

D. Navarre et al.

—Claiming that unsafe states are not reachable is more effective than stating all reachable states are safe. This backward reachability approach is useful, for example, after an incident or accident has occurred. Indeed, the model of the system can be used in order to simulate events leading to the unsafe state, as shown in Basnyat et al. [2006]. Such analysis techniques are less meaningful if the unsafe states are unknown, typically during early design phases. 6.3 Exploiting Models In Section 4.2 we have already presented the use of our model-based approach as a way of supporting in an easy way rapid system prototyping. Executing the specifications (as when models are run within PetShop) has the immediate benefits of providing a prototype available for usability evaluation. In Bernhaupt et al. [2007] we improve the idea of using model-based approaches as a support for usability evaluation using a marking graph (briefly introduced in Section 6.2.2) to describe low-level interaction scenarios. This approach improves classical usability evaluation methods in two directions. —Supporting Usability Evaluation Using Model-Based Scenarios and ModelBased Prototyping. While usability evaluation methods are quite efficient for tracking structural usability problems (based on ergonomic criteria or navigation problems in an application), multimodal applications often present finegrain interaction techniques difficult to assess due to their intrinsic complex nature involving concurrent and time-constrained behaviors. For instance, the description of the temporal evolution presented in the case study in Section 5.1 shows how complex low-level multimodal interaction can be. When it comes to testing the usability of such behavior it is required first to provide a detailed description of the behavior to the evaluators but also to make it possible to modify such behavior if the results of the usability testing require doing so. —Modifying Models to Accommodate Changes Due to Usability Evaluation Results. The aim of usability evaluation is not only to identify usability problems in a user interface, but also to provide information to designers to modify the system and user interface to reduce or remove the identified problems. Here again, model-based approaches can be of great interest. Indeed, as scenarios have been extracted from models, it is easier to point out which part of which model is involved in the identified usability problems. The main contribution of these interaction scenarios is to provide an explicit and complete description of the set of interactions available for the user. This makes it possible to test the usability of an interactive application not only at a high level like tasks or goals, but also at a lower level of detail. This makes it possible to test: —the interaction technique per se, that is, how difficult it is to manipulate the input devices for low-level tasks like pointing and selecting objects. In the ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:51

case of multimodal interactions it is also possible to evaluate the difficulties for users to combine input devices. —the link between interaction technique and task execution by making explicit what kind of low-level actions have to be executed in order to perform higherlevel tasks and to reach goals. —the complexity of the interaction technique in order to be able to predict users’ difficulties in interacting with the application. In order to get more figures about this complexity we need to apply our approach to several interaction techniques and to analyze correlations with the results from actual usability tests. We already partially addressed these issues in the domain of air traffic control [Palanque et al. 1997] and we are now carrying on with more complex interaction techniques and application domains with stronger real-time constraints, such as aircraft cockpits. Other complex behaviors relative to qualitative temporal behaviors (number of milliseconds for the delays) can also be represented and thus exploited during usability tests. Indeed, the interaction technique can embed temporal behaviors like the one presented in the case study of Section 5.1 and this can have an impact on user performance. The explicit representation of these temporal evolutions in the interaction technique models makes it possible to incorporate such intrinsic values while usability experts analyze the results of the usability tests. Lastly, this explicit representation of low-level scenarios is useful for selecting which scenarios will be evaluated with users. Indeed, in the field of safety-critical systems scenario identification is critical, as some tasks can be performed very rarely (such as, for instance, setting a satellite to a survival mode or involving very low-probability events like failures). Being able, from analysis of the interaction models, to detect potential difficulties for users can fruitfully influence the way usability tests will be performed. 7. CONCLUSION This article has presented the ICOs user interface description language. This UIDL is based on a formal notation (a dialect of object Petri nets) and promotes a model-based approach for the description of interactive applications. It covers the various components of an interactive application from the management of low-level events of input devices to the functional core of the application. The advantage of using models instead of a programming language is that it provides designers with abstract constructs, making possible to address user interface design issues at different levels of abstraction (keeping away from implementation concerns when needed). In addition to model-based level, ICOs provide a clear connection to the computer language used for the coding of the graphical appearance of objects. As stated in Section 3.3, the promoted language is Java (as some Java code can be added to transitions and interpreted at runtime) even though connection to other languages such as UsiXML or SVG is available too. This connection to the underlying programming language allows to ICOs build complete interactive applications. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:52

•

D. Navarre et al.

Another advantage of using models with a formal semantics13 is that it makes it possible to assess the models with respect to identified properties. This aspect of verification has been presented in Section 6 and is of primary importance when safety-critical systems are under consideration. The article has shown (Section 5) that the ICOs UIDL has been used for the construction of various command-and-control interactive systems in safety-critical domains (including military multimodal cockpits, interactive cockpits of civil large aircrafts such as the A380 or Boeing 787, satellite control ground segments, and air traffic control workstations). The article has also introduced PetShop, the tool suite dedicated to the engineering of interactive systems. Fully supporting the ICOs UIDL, it allows in a modeless way the editing, simulation, and execution of models supporting design processes based on iterative prototyping phases. The tool is publicly available and can be downloaded at http://ihcs.irit.fr/petshop/. It is important to note that the contribution around ICOs does not address issues related to the support for the design and implementation of multiplatform interface as in Meskens et al. [2008] or Berti and Paterno [2005] as, in the case of safety-critical systems multiple designs are favored. As stated in the Introduction, the fact that ICOs are based on an object Petri net dialect induces quite a high learning curve for engineers working on the projects. Indeed, skills in object-oriented programming, Java, user interface design, and implementation and Petri net-based modeling are required. Such high requirements might limit the economical applicability of ICOs to the field of safety-critical interactive systems. However, in the article we tried to demonstrate that the solution offers substantial benefits, both in terms of prototypability of interaction techniques and reliability of the resulting systems that otherwise would have been impossible to achieve. This work belongs to a more ambitious project dedicated to the engineering of interactive systems for safety-critical applications addressing, on an equal basis, safety, reliability, usability, and evolvabilty of interactive systems. This concern comes from the observation that a significant effort is currently being undertaken by the CHI community in order to design and evaluate new kinds of interaction techniques while very little has been done to improve the reliability of the software offering these interaction techniques. As these new interaction techniques are currently increasingly deployed in the field of command- andcontrol safety-critical systems, the potential of incidents or accidents increases. Similarly, the unreliability of interactive software can jeopardize usability by showing unexpected or undesired behaviors. Lastly, iterative design processes promote multiple designs through evolvable prototypes in order to accommodate requirement changes and results from usability evaluations, thus reducing reliability of the final system by lack of global and structured design. The aim of future work around the ICOs UIDL is on one hand to provide notations and tools for building interactive systems featuring new kinds of interaction techniques, as has been done for multimodal interfaces. On the other hand, we also want to support verification and validation activities in order to 13 The

formal semantics of ICOs is available at http://ihcs.irit.fr/palanque/Ps/ICOFormalDef.pdf.



•

18:53

support certification phases that represent an understudied and critical phase in the development process of interactive safety-critical applications. REFERENCES ANSON, E. 1982. The device model of interaction. In Proceedings of the 9th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’82). ACM, New York, 107–114. APPERT, C. AND BEAUDOUIN-LAFON, M. 2006. SwingStates: Adding state machines to the swing toolkit. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST ’06). ACM, New York, 319–322. ARINC 661-2. 2005. ARINC Specification 661-2. Prepared by Airlines Electronic Engineering Committee. Cockpit Display System Interfaces to User Systems. BARBONI, E., BASTIDE, R., LACAZE, X., NAVARRE, D., AND PALANQUE, P. Petri net centered versus user centered Petri nets tools. In Proceedings of the 10th Workshop on Algorithms and Tools for Petri Nets (AWPN’03). BARBONI, E., CONVERSY, S., NAVARRE, D., AND PALANQUE, P. 2006. Model-Based engineering of widgets, user applications and servers compliant with ARINC 661 specification. In Proceedings of the 13th Conference on Design Specification and Verification of Interactive Systems (DSVIS’06). Lecture Notes in Computer Science, Springer. BARBONI, E., NAVARRE, D., PALANQUE, P., AND BASNYAT, S. 2006. Exploitation of formal specification techniques for ARINC 661 interactive cockpit applications. In Proceedings of the HCI Aero Conference (HCI Aero’06). BASNYAT, S., CHOZOS, N., AND PALANQUE, P. 2006. Multidisciplinary perspective on accident investigation. Reliab. Engin. Syst. Safety 91, 12, 1502–1520. BASS, L., LITTLE, R., PELLEGRINO, R., REED, S., SEACORD, R., SHEPPARD, S., AND SZEZUR, M. R. 1991. The Arch model: Seeheim revisited. In User Interface Developers’ Workshop, Version 1.0. BASTIDE, R. AND PALANQUE, P. 1990. Petri nets with objects for specification, design and validation of userdriven interfaces. In Proceedings of the 3rd IFIP Conference on Human-Computer Interaction (Interact’90). BASTIDE, R., PALANQUE, P., OUSMANE, S., LE, D.-H., AND NAVARRE, D. 1999. Petri net based behavioral specification of CORBA systems. In Proceedings of the International Conference on Application and Theory of Petri nets (ATPN’99). Lecture Notes in Computer Science. Springer. BASTIDE, R., NAVARRE, D., PALANQUE, P., SCHYN, A., AND DRAGICEVIC, P. 2004. A model-based approach for real-time embedded multimodal systems in aircrafts. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI’04). BASTIDE, R., BARBONI, E., LACAZE, X., NAVARRE, D., PALANQUE, P., SCHYN, A., AND BAZALGETTE, D. 2005. Supporting INTUITION through formal specification of the user interface for military aircraft cockpit. In Proceedings of the HCI International Conference. BERNHAUPT, R., NAVARRE, D., PALANQUE, P., AND WINCKLER, M. 2007. Model-Based evaluation: A new way to support usability evaluation of multimodal interactive applications. In Maturing Usability: Quality in Software, Interaction and Quality, E. Law, E. Thora Hvannberg, G. Cockton, and J. Vanderdonckt, Eds., HCI Series, Springer. BERTI, S. AND PATERNO` , F. 2005. Migratory multimodal interfaces in multidevice environments. In Proceedings of the 7th International Conference on Multimodal Interfaces. ACM Press, 92–99. BLANCH, R. AND BEAUDOUIN-LAFON, M. 2006. Programming rich interactions using the hierarchical state machine toolkit. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI’06). ACM, New York, 51–58. BUXTON, W. 1990. A three-state model of graphical input. In Proceedings of the IFIP Tc13 3rd International Conference on Human-Computer Interaction, D. Diaper, D. J. Gilmore, G. Cockton, and B. Shackel, Eds. North-Holland Publishing, Amsterdam, The Netherlands, 449–456. CARDELLI, L. AND PIKE, R. 1985. Squeak: A language for communicating with mice. SIGGRAPH Comput. Graph. 19, 3, 199–204. CARR, D. A. 1994. Specification of interface interaction objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence, B. Adelson, S. Dumais, and J. Olson, Eds. ACM, New York, 372–378. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:54

•

D. Navarre et al.

CONINX, K., CUPPENS, E., DE BOECK, J., AND RAYMAEKERS, C. 2007. Integrating support for usability evaluation into high level interaction descriptions with nimmit. In Interactive Systems: Design, Specification, and Verification. Lecture Notes in Computer Science. Springer. DRAGICEVIC, P. 2004. Combining crossing-based and paper-based interaction paradigms for dragging and dropping between overlapping windows. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST’04). ACM Press, 193–196. DRAGICEVIC, P. AND FEKETE, J. 2004. Support for input adaptability in the ICON toolkit. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI’04). ACM, New York, 212–219. DUMAS, B., LALANNE, D., GUINARD, D., KOENIG, R., AND INGOLD, R. 2008. Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces. In Proceedings of the 2nd International Conference on Tangible and Embedded Interaction (TEI’08). ACM, New York, 47–54. ESTEBAN, O., CHATTY, S., AND PALANQUE, P. 1995. Whizz’Ed: A visual environment for building highly interactive interfaces. In Proceedings of the Interact’95 Conference. 121–126. GAMMA, E., HELM, R., JOHNSON, R., AND VLISSIDES, J. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Addison Wesley. GENRICH, H. J. 1991. Predicate/Transitions nets. In High-Levels Petri Nets: Theory and Application, K. Jensen and G. Rozenberg, Eds. Springer, 3–43. GREEN, M. 1986. A survey of three dialog models. ACM Trans. Graph. 5, 3, 244–275. HINCKLEY, K., CZERWINSKI, M., AND SINCLAIR, M. 1998. Interaction and modeling techniques for desktop twohanded input. In Proceedings of the 11th Annual ACM Symposium on User Interface Software and Technology (UIST’98). ACM, New York, 49–58. HUDSON, S. E. 1989. Graphical specification of flexible user interface displays. In Proceedings of the 2nd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology (UIST’89). ACM, New York, 105–114. HUDSON, S. E., MANKOFF, J., AND SMITH, I. 2005. Extensible input handling in the subArctic toolkit. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’05). ACM, New York, 381–390. IBM. 1989. Common user access: Advanced interface design guide. IBM, SC26-4582-0. JACOB, R. J. 1986. A specification language for direct-manipulation user interfaces. ACM Trans. Graph. 5, 4, 283–317. JACOB, R. J., DELIGIANNIDIS, L., AND MORRISON, S. 1999. A software model and specification language for non-WIMP user interfaces. ACM Trans. Comput.-Hum. Interact. 6, 1, 1–46. JENSEN, K., KRISTENSEN, L., AND WELLS, L. 2007. Coloured Petri nets and CPN tools for modeling and validation of concurrent systems. Int. J. Softw.Tools Technol. Transfer 9, 3, 213–254. JOBS, S. P., FORSTALL, S., CHRISTIE, G., LEMAY, S. O., HERZ, S., ET AL. 2008. Touch screen device, method, and graphical user interface for determining commands by applying heuristics. United States Patent Application 20080122796. Kind Code A. KATSURADA, K., NAKAMURA, Y., YAMADA, H., AND NITTA, T. 2003. XISL: A language for describing multimodal interaction scenarios. In Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI’03). ACM, New York, 281–284. KEH, H. C. AND LEWIS, T. G. 1991. Direct-Manipulation user interface modeling with high-level Petri nets. In Proceedings of the 19th Annual Conference on Computer Science (CSC’91). ACM, New York, 487–495. KIERAS, D. AND POLSON, P. G. 1983. A generalized transition network representation for interactive systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’83), A. Janda, Ed. ACM, New York, 103–106. LADRY. J-F., NAVARRE, D., AND PALANQUE, P. 2009. Formal description techniques to support the design, construction and evaluation of fusion engines for sure (safe, usable, reliable and evolvable) multimodal interfaces. In Proceedings of the 10th International Conference on Multimodal Interfaces (ICMI’09). LAKOS, C. 1991. Language for object-oriented Petri nets. #91-1. Department of Computer Science, University of Tasmania. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.


•

18:55

LATOSCHIK, M. E. 2002. Designing transition networks for multimodal VR-interactions using a markup language. In Proceedings of the 4th IEEE International Conference on Multimodal Interfaces. 411–416. LIMBOURG, Q., VANDERDONCK, J., MICHOTTER, M., BOUILLON, L., AND LOPEZ-JAQUERO, V. 2005. USIXML: A language supporting multi-path development of user interfaces. In Proceedings of EHCI-DSVIS’04 Conference. Lecture Notes in Computer Science, vol. 3425. Springer, 200–220. MESKENS, J., VERMEULEN, J., LUYTEN, K., AND CONINX, K. 2008. Gummy for multi-platform user interface designs: Shape me, multiply me, fix me, use me. In Proceedings of the Working Conference on Advanced Virtual Interfaces (AVI’08). 233–240. NAVARRE, D., PALANQUE, P., BASTIDE, R., AND SY, O. 2000. Structuring interactive systems specifications for executability and prototypability. In Proceedings of the 7th Eurographics Workshop on Design, Specification and Verification of Interactive Systems (DSV-IS’00). Lecture Notes in Computer Science. Springer. NAVARRE, D., PALANQUE, P., BASTIDE, R., AND SY, O. 2001. A model-based tool for interactive prototyping of highly interactive applications. In Proceedings of the 12th IEEE International Workshop on Rapid System Prototyping. IEEE Press. NAVARRE, D., PALANQUE, P., AND BASTIDE, R. 2002. Reconciling safety and usability concerns through formal specification-based development process. In Proceedings of the HCI-Aero’02 Conference. NAVARRE, D., PALANQUE, P., BASTIDE, R., SCHYN, A., WINCKLER, M., NEDEL, L., AND FREITAS, C. 2005. A formal description of multimodal interaction techniques for immersive virtual reality applications. In Proceedings of the INTERACT’05 Conference. Lecture Notes in Computer Science, Springer. NAVARRE, D., PALANQUE, P., AND BASNYAT, S. 2008. Usability service continuation through reconfiguration of input and output devices in safety critical interactive systems. In Proceedings of the 27th International Conference on Computer Safety, Reliability and Security (SAFECOMP’08), M. D. Harrison and M.-A. Sujan, Eds. Lecture Notes in Computer Science, vol. 5219. Springer, 373–386. NAVARRE, D., PALANQUE, P., LADRY, J. F., AND BASNYAT, S. 2008. An architecture and a formal description technique for user interaction reconfiguration of safety critical interactive systems. In Proceedings of the XVth International Workshop on the Design, Verification and Specification of Interactive Systems (DSVIS’08). NEDEL, L. P., FREITAS, C. M. D. S., JACOB, L. J., AND PIMENTA, M. S. 2003. Testing the use of egocentric interactive techniques in immersive virtual environments. In Proceedings of the INTERACT IFIP TC 13 Conference on Human Computer Interaction. IOS Press, 471–478. PALANQUE, P. 1992. User-Driven user interfaces modeling using interactive cooperative objects. (in french). Ph.D. University Toulouse I, France. PALANQUE, P. AND BASTIDE, R. 1995. Verification of an interactive software by analysis of its formal specification. In Proceedings of the IFIP Human-Computer Interaction Conference (Interact’95). 181–197. PALANQUE, P., BASTIDE, R., AND PATERNO, F. 1997. Formal specification as a tool for objective assessment of safety-critical interactive systems. In Proceedings of the Interact’97 Conference. Chapman and Hall. PALANQUE, P., BERNHAUPT, R., NAVARRE, D., OULD, M., AND WINCKLER, M. 2006. Supporting usability evaluation of multimodal man-machine interfaces for space ground segment applications using Petri net based formal specification. In Proceedings of the 9th International Conference on Space Operations. PARNAS, D. L. 1969. On the use of transition diagrams in the design of a user interface for an interactive computer system. In Proceedings of the 24th National ACM Conference. 379–385. PETRI, C. A. 1962. Kommunikation mit Automaten. Rheinisch-Westfaliches Institut fur Intrumentelle Mathematik an der Universitat Bonn, Schrift Nr 2. RILEY, V. 2004. Human factors in avionics certification. Avionics Mag. 28, 9, 42–46. SCHLOMER, T., POPPINGA, B., HENZE, N., AND BOLL, S. 2008. Gesture recognition with a Wii controller. In Proceedings of the 2nd International Conference on Tangible and Embedded Interaction. ACM Press. ACM Transactions on Computer-Human Interaction, Vol. 16, No. 4, Article 18, Publication date: November 2009.

18:56

•

D. Navarre et al.

SHAER, O. AND JACOB, R. J. K. 2005. Toward a software model and a specification language for next-generation user interfaces. In Proceedings of the ACM CHI Workshop on The Future of User Interface Software Tools. SIBERT, J. L., HURLEY, W. D., AND BLESER, T. W. 1986. An object-oriented user interface management system. In Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’86), D. C. Evans and R. J. Athay, Eds. ACM, New York, 259–268. SMITH, S. AND DUKE, D. 1999. Using CSP to specify interaction in virtual environments. Tech. rep. YCS 321. University of York. SVG W3C. 2003. Scalable vector graphics (SVG) 1.1 specification. http://www.w3.org/TR/SVG11/. SZEKELY, P. AND MYERS, B. 1988. A user interface toolkit based on graphical objects and constraints. In Proceedings of the Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA’88), N. Meyrowitz, Ed. ACM, New York, 36–45. TATSUKAWA, K. 1991. Graphical toolkit approach to user interaction description. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’91), S. P. Robertson, G. M. Olson, and J. S. Olson, Eds. ACM, New York, 323–328. VALK, R. 1998. Petri nets as token objects: An introduction to elementary object nets. In Proceedings of the 19th International Conference on Application and Theory of Petri Nets (ICATPN’98). Springer. WILLANS, J. S. AND HARRISON, M. D. 2001. Prototyping pre-implementation designs of virtual environment behavior. In Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction, M. R. Little and L. Nigay, Eds. Lecture Notes In Computer Science, vol. 2254. Springer, 91–108. Received January 2009; revised July 2009; accepted August 2009


ICOs: A model-based user interface description technique dedicated ...

ICOs: A model-based user interface description technique dedicated ...

Suggest Documents

StateWebCharts: a Formal Description Technique Dedicated to ...

User Interface Description Languages for Next Generation User ...

User Interface Description Languages for Next Generation User ...

USIXML: A User Interface Description Language for Context-Sensitive ...

USIXML: a User Interface Description Language for Specifying ...

A Review of XML-compliant User Interface Description Languages

A Review of User Interface Description Languages for Mobile ...

USIXML: A User Interface Description Language for Context-Sensitive ...

Harmless, a Hardware Architecture Description Language Dedicated ...

Generating User Interface Code in a Model Based User Interface ...

User Interface

Charger Interface Description Charging interface ... - Banggood

Charger Interface Description Charging interface ... - Banggood

A modelbased PID controller for Hammerstein

Multi-channel OLSR with a Dedicated Control Interface - CiteSeerX

Ergonomics user Interface Standards

Search User Interface Design

User Interface Design 2

ST950 User Interface Handbook

User Interface Overview

Modelbased analysis of skin conductance

SAP User Interface

Flow - User Interface Engineering

GRAPHICAL USER INTERFACE (GUI)