Model-based Design, Generation, and Evaluation of Virtual User Interfaces Jean Vanderdonckt, Chow Kwok Chieu, Laurent Bouillon, Daniela Trevisan* Université catholique de Louvain, Belgian Laboratory of Computer-Human Interaction (BCHI) Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium For instance, it is rather complicated to evaluate the usability of a virtual UI that is currently being developed, even without end users. In his keynote address at Web3D’2000, Neil Trevett raised the question of UI transition: in order to make a 3D UI useful for mainstream users, there is a need to transform flat 2D Graphical User Interfaces (GUIs) coming from the traditional desktop into 3D. If an application running a GUI is launched in a 3D desktop, everything returns to a flat 2D representation as soon as the application is active: the true 3D aspects are lost in the transition. Therefore a 3D version of such UIs could change the situation and encourages a larger adoption and use of 3D desktops.
Abstract A transformational approach is presented that models a virtual reality scene based on Abstract Interaction Objects (AIOs) from which virtual user interfaces can be designed (by progressive AIO assembling), generated (by automated code generation in VRML/X3D), and evaluated (by static analysis of the ongoing model). The underlying model is shared with user interfaces descriptions in other worlds like Windows applications, web pages, etc. It is therefore possible to recuperate existing interfaces in these worlds to automatically create their counterpart in VRML/ X3D. This process is called virtualization of flat user interfaces. This approach allows the progressive and flexible retargeting of non-virtual interfaces in the virtual world to achieve a better integration with other applications.
To address the above requirements, we developed Envir3D, a 3D modeling tool adopting a model-based approach: rather than having only the virtual UI being designed and drawn manually, an underlying model of the virtual UI is maintained at design time that entirely captures the layout (hereafter called presentation) and the behavior (hereafter called dialog) of the virtual UI so that it remains independent of any computing platform, any particular Look & Feel. When a model of the virtual UI is maintained, model-based design can be achieved. Once the model has been created and finalized, Envir3D automatically generates the VRML97/X3D code for the developer. The process of this automated generation brings several benefits: each time a modification occurs, it can be applied on the previously built model and a new code can be re-generated. At any time, the currently being designed model can be submitted to the checking against usability guidelines that can provide the developer with a very first feedback about the potential usability quality of the current virtual UI. Finally, parts of whole of any existing model can be reused for modification, expansion, and reincorporation in Envir3D.
CR Categories: D.2.2 [Software Engineering]: Design Tools and Techniques – User interfaces. I.3.6 [Computer Graphics]: Methodology and Techniques – Interaction techniques. I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism – Interaction techniques. H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems – Artificial, augmented, and virtual realities. H.5.2 [Information Interfaces and Presentation]: User Interfaces – Graphical user interfaces, input devices and strategies, interaction styles, user-centered design. General terms: Design, Human Factors, Algorithms. Keywords: Abstract Interaction Object (AIO), Automated code generation, Concrete Interaction Object (CIO), Interface evaluation, Retargeting, Static analysis, Transformational approach, Usability Guidelines, Virtualization.
The remainder of this paper is structured as follows: section 2 will report on related work conducted in the domain and will show the sources that inspired the development of Envir3D. Section 3 will explain how the process of model-based design of virtual UI creation is supported by Envir3D and will highlight the original contribution of this tool. A complete example of virtual UI creation is given for a control room. Section 4 will then exemplify how it is possible to quickly evaluate the virtual UI against usability guidelines by inspection of the recently created model. Thanks to the underlying model, it will then become possible to address Trevett’s question of UI transition: a web page will be reverse engineered to recover its underlying model that will be in turn converted into a corresponding model for a virtual UI. At this time, the step of automated code generation takes place again to generate the code corresponding to this new model. Section 6 will show another example in the domain of car dashboard prototyping to illustrate rapid prototyping. Finally, Section 7 will conclude the paper by summarizing the main points of the paper and outline some future work we want to conduct. Also, the integration of Envir3D with other tools supporting the same model-based approach will be discussed.
1 Introduction Many 3D modeling tools exist on the market that allows developers to quickly draw a virtual world or a virtual user interface (UI). While they become more and more sophisticated and more supportive in the design task, every operation is done manually in a graphical way, which is consistent with the visual nature of the task. Consequently, each change that may be required by the end user or by any change in the requirements should be performed manually within the tool, thus increasing the work load of maintaining the virtual UI up-to-date. In addition, such tools offer little or no guidance on how to produce a virtual UI that is usable for the end user or on how to progressively guide the developer in the development process [Jacob et al., 1999]. *
e-mail: {vanderdonckt, chow, bouillon, trevisan}@isys.ucl.ac.be
Copyright © 2004 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
[email protected]. © 2004 ACM 1-58113-845-8/04/0004 $5.00
The discussion that is given in the rest of this paper will focus on illustration based on VRML97 virtual UIs. It remains pretty similar for X3D UIs as conversion tools exist that establish correspondence between both formats.
51
a HTML page. Each HTML tag is mapped onto a virtual object so as to create a landscape representing the structure of tags used and the contents of the web page (Fig. 2).
2 Related Work The development of Envir3D has been mainly driven by a combination of different sources of inspiration: model-based design, model-based evaluation, and automated code generation. Only the combination of these three aspects will ultimately allow us to address Trevett’s UI transition problem. Here, we review some work conducted in these respective domains and other related work done in the domain of 3D user interfaces. One important facet of model-based design, as implemented in Envir3D, is the relying on a knowledge base of objects that can be manipulated logically. For example, IBIS (Intent-Based Illustration System) [Seligmann & Feiner, 1991] generates illustrations automatically, guided by communicative goals which specify that particular properties of objects, such as their size, color, or location are to be conveyed in the illustration. Instances of these objects can then be laid out to represent the interface of an electronic device front panel whose behavior has to be explained (Fig. 1). Similarly to IBIS, Envir3D has access to a knowledge base that contains a collection of objects, including information about their geometric properties, material and location. These objects will serve for creating the virtual UI.
Figure 2. Virtual objects used for HTML tags. 3D UIs are not always superior to 2D UIs just because they add another dimension. For example, a usability study [Sebrechts et al., 1999] conducted on NIRVE, a tool that supports visualization of search results, revealed that the 3D visualization UI required a higher work load to the users than for 2D, that is progressively decreasing when experience is gained. Only when the user’s expectations were clearly addressed, the 3D UI was assessed to be comparable to its 2D counterpart. Even in other domains, the superiority is not proved, like in surface editing [Waterhouse, 1996].
Figure 1. An IBIS-generated virtual UI. A second aspect of model-based design that is widely studied by researchers is the independence of the produced output with respect to various computing platforms, languages, or formats. Rather than having a tools that is bound to a particular environment, the model-based design allows expressing a model in a way that high-level characteristics are captured, irrespectively of the different ways to implement it. For example, IMPROVISE [Zhou & Feiner, 1997] (http://www.cs.columbia.edu/~zhou/project/project. html) focuses on automatically generating coherent visual discourse which refers to a series of connected visual displays. Developers of IMPROVISE developed a visual language that is independent of any graphics platform or package. Similarly, Envir3D stores a definition of a virtual UI in a language that is independent of any rendering system. For this purpose, XIML [Puerta & Einsenstein, 2002 ; 2003] has been selected and used.
3 Process of Virtual User Interface Creation After having introduced some related work, the process of creating a virtual UI as supported by Envir3D is explained as graphically represented in Fig. 3. First of all, widgets of interest are considered to be subject to abstraction with respect to some physical properties so as to be able to manipulate a same concept throughout various programming languages and mark-up languages. This process is explained in the first sub-section.
A third aspect resulting from the model-based design is that it enables a first evaluation of the model as in NGOMS [Byrne et al., 1994], where a task model is used to compare the usage of a real UI with respect to what the UI is expected to support as represented in the model. Another possibility of automated evaluation consists of submitting the virtual UI of interest to the examination of its model with respect to established usability guidelines.
Then, these abstractions are incorporated into a library of objects that can be used for establishing mappings between the abstract objects and its counterparts in the different cases considered. An abstraction may have one or several concretizations as explained in the second sub-section. Based on this mechanism, the third section shows how to create virtual worlds thanks to Envir3D by manipulating the basic abstractions. With this, a virtual world can be created quickly, while maintaining an internal representation of the virtual world. This representation can be exploited for different purposes, like verification of properties of interest related to the objects that compose the virtual world.
A fourth aspect can be considered as a consequence of the modelbased approach: when the activity of developing a UI is replaced by its modeling, it is expected that the tool is powerful enough to automatically generate the corresponding code. Automated generation of VRML code has been successfully used in various domains, such as guided tours from physical models [Chittaro et al., 2003], city scenes from city models [Schilling & Zipf, 2003], and graphical art. For example, initiated by the San Francisco Museum of Modern Art, Eden Garden (www.Entropy8Zuper.org) automatically generates a VRML world from the tags contained in
The fourth sub-section explains the process of automatically generating the VRML97 code for the previously specified virtual world, according to a model-based approach.
52
1993]. First of all, the abstraction that is used here is with respect to physical attributes like position or dimension. For instance, it does not matter how a particular list box may look like in a particular computing platform, for a particular UI. Therefore, the physical position (e.g., in terms of absolute coordinates) and the physical dimensions (e.g., in terms of pixels) can be left out if needed.
VRML Object
Abstraction of objects
Connected VRML Object
AIO definition file
Each AIO is identified by an unique generic name (e.g., check box), general and particular abstract attributes (e.g., height, width, possible choices, state), abstract events (e.g., check or uncheck), primitive functions (e.g., PrActivateCheckBox, PrDeactivateCheckBox). By definition, each AIO does not have any graphical appearance, but each AIO is connected to 0, 1 or many CIOs in different environments with different names, presentations, or behaviors.
Incorporation into AIO library AIO library
Instantiation of AIO into CIO
Envir3D: virtual world creation
Definition of CIO attributes
XIML model
Definition of Relationships between CIOs
In Envir3D, each AIO should have at least one corresponding CIO to properly work: if a AIO has no corresponding CIO, this means that no resulting UI can be produced ; if a AIO is mapped to only one CIO, the concretization is straightforward ; if a AIO can be mapped onto several CIOs, the designer is able to choose which CIO will concretize the AIO or leave the tool do it automatically thanks to pre-encoded rules. All AIOs share a series of common abstract attributes:
Envir3D: automated code generation VRML97/X3D file
VRML97-enabled browser
x
Figure 3. The complete process for creating a virtual UI. 3.1 Abstraction of Objects As soon as a model-based approach is adopted, a need arises to obtain a model of objects of interest. The first step of the virtual UI creation therefore consists of abstracting existing VRML object into a definition of that object that can be manipulated in the tool. To represent these objects, the concept of Concrete Interaction Object (CIO) is defined as any entity of the UI that users can perceive (e.g., text, image, animation) and/or manipulate (e.g., a push button, a list box, a check box). It could be an object from a library, a widget provided by a toolkit. When one looks at different widgets, objects, controls that are available in various computing platforms, it can be observed that, except for low-level details of style, these objects basically remain similar. Fig. 4 graphically depicts different forms of CIOs: on the first line, check boxes with different “Look” but same “Feel” (respectively from Java Swing, Microsoft Windows, OSF/Motif), on the second line, check boxes with different “Look” and different “Feel” (respectively, a check box in Microsoft Windows, a XmToggleButton in OSF/Motif, and a BoxArray in Garnet).
x x x
x
x x
AT_LEFT_ABSCISSA, AT_LEFT_ORDINATE, AT_LEFT_ DEPTH: attributes defining the position of the object in the 3D space if needed. AT_LENGTH, AT_HEIGHT, AT_DEPTH: attributes defining the size of the object. AT_ORIENT: attribute defining the orientation of the object in the 3D space. AT_SIDE: attributes defining the side where other objects are attached without taking into account the orientation of the object. This attribute is used to form new objects from existing objects. AT_TYPE: attribute defining the type, physical or virtual, of the object. Physical objects depict those objects that only exist in the physical world, outside the working UI. For instance, a cupboard is considered physical as it cannot hold any interaction. Conversely, a dial can be virtual if it is part of the UI or physical if it is defined externally without any support. AT_COLOR: attribute defining the color of the object. AT_DIM: attribute specifying whether the object is 2D or 3D. For example, a dial can be rendered in 2D or 3D depending on the concretization choosen.
Note that the list of abstract attributes of each object can be extended on demand as some 3D objects introduce some attributes that 2D objects do not possess. For instance, AT_TEXTURE is considered typical for a 3D objects, while irrelevant in a 2D world. Each AIO that can be used by Envir3D is defined by a definition file in XIML format. We will briefly introduce XIML later in this section. A definition file for an AIO declares the abstract attributes and their types, and the different alternative generic graphical representations for CIOs derived from this AIO. For each alternative graphical representation a VRML file must be provided. This VRML description will be later used by Envir3D to insert the graphical representation of CIOs in the final virtual UI.
Figure 3. Some variations of Concrete Interaction Objects. While different CIOs may exhibit different “Look & Feel”, it can be observed that basically they share the same behavior. For this purpose, the concept of Abstract Interaction Object (AIO) is introduced and defined as an abstraction of a set of CIOs with respect to a set of common properties [Vanderdonckt and Bodart,
53
any time, the developer can move objects around to form the desired virtual world. The upper right window of Fig. 6 denotes the project window showing the instantiated CIOs forming the world. In this example, four CIOs have been created: a computer screen, a keyboard, and a mouse on top of a table. The hierarchy mirrors the composition of CIOs, here a workstation for example. Therefore, if the table is moved, all objects that are attached are moved accordingly. 2. Definition of CIO attributes: as soon as a CIO is created, the property window (lower right window in Fig. 6) displays the values of attributes of this CIO. Attributes that can be governed graphically are specified by direct manipulation of objects, when the developer is moving, sizing, composing objects. All other attributes that cannot be specified graphically are defined in the property window. 3. Definition of relationships between CIOs: the relationships between CIOs are automatically inferred from the direct manipulation activities of the developer. If, for instance, the developer drops a computer on a table, this is interpreted as a computer being attached on top of the table. Also the relative positioning of objects one with respect to the other is analyzed to establish logical relationships between CIOs, such as grouping, aligning, balancing, centering.
Based on the above concepts of CIO and AIO, each VRML objects of interest (e.g., a .vrml file) is considered as a CIO and abstracted in Envir3D as an AIO, stored in an AIO definition file containing the relations between abstract attributes and concrete attributes, for example. This step is performed only once. The initial VRML object is then connected to this definition to become ready for being incorporated in the AIO library. 3.2 Incorporation into AIO Library Each newly created AIO can be incorporated in the AIO library by simply copying the connected VRML object along with its definition file in the directory of supported AIOs. At run-time, Envir3D detects the incorporation of a new object and prompts the user for specifying the location of its entry in the hierarchy of AIOs contained in the Objects browser (Fig. 4). For example, there are several graphical representations for the AIO dial, based on the type (physical or virtual), the shape (circle or square) and the range covered by the dial (90° to 360°). The AIO library typically contains AIOs that are control objects (e.g., button, dial, gauge, list box, drop down list box, radio button, check box, joystick), static and decoration objects (e.g., separator, group box, line), containers (e.g., window, dialog, tabbed dialog).
As output, the virtual UI is saved in the XIML (eXtensible userInterface Markup Language) format (www.ximl.org) [Puerta & Eisenstein, 2002, 2003]. XIML is an XML-based mechanism designed to manage the complex interactions among users, applications, devices, and user interfaces. XIML offers a common representation for interaction data. XIML is a descriptive language of UI without worry about how user interfaces will be effectively implemented.
Figure 5. Several entries of AIO_Dial in the AIO library. While most AIO exist in both in physical 2D and virtual 3D formats, some of them are unique to the 3D world, like cupboard, table, chair, control panel (Fig. 5). When a sub-type object needs to be incorporated into the AIO library, it can be inserted as a subentry in the hierarchy denoting that some values of some abstract properties (e.g., attributes, events, or primitives) are fixed. In Fig. 4 for example, the 90° Physical Dial is a sub-type of the general AIO Dial type. Other sub-types of the AIO Dial are also illustrated depending on the minimum and maximum values of the range.
Figure 6. Structure of the XIML concepts (from [Puerta & Eisenstein, 2003]). In its most basic sense, XIML is an organized collection of interface elements that are categorized into one or more major interface components. The language does not limit the number and types of components that can be defined. Neither there is a theoretical limit on the number and types of elements under each component.
3.3 Envir3D: Virtual World Creation Creating a virtual world with Envir3D is pretty much similar to any traditional 3D modeler. The model that is underlying is completely transparent to the eyes of the developer. The process of creation is decomposed into three steps (Fig. 6):
In a more practical sense, however, it is to be expected that an XIML specification would support a relatively small number of components with one major type of element defined per component. XIML predefines five basic interface components, namely task, domain, user, dialog, and presentation. The first three of these can be characterized as contextual and abstract while the last two can be described as implementational and concrete. We now examine two of these five components as they are particularly used in expressing, describing, and daving the virtual UI (Fig. 7).
1. Instantiation of an AIO into a CIO: to progressively build the virtual world, the developer can instantiate any existing AIO into a CIO by selecting a AIO in the objects browser (left window in Fig. 5), by dragging it and dropping it onto one of the two surface areas (middle windows in Fig. 6) representing the X-Y and Y-Z views. Whatever the surface area chosen, the corresponding view is generated in the other coordinated window: any change in one window is propagated in the other. At
54
Graphical representation along X-Y
Project window containing CIOs forming the virtual scene Property window containing values of CIOs attributes
Objects browser allowing access to the AIO library Graphical representation along Y-Z
Figure 6. Modeling a virtual world with Envir3D. Presentation. The presentation component [Puerta & Eisenstein, 2003] defines a hierarchy of interaction elements that comprise the concrete interaction objects that communicate with users in an interface. Examples of these are a dialog box, a dial, a slider, or a complex widget such as an ActiveX control to visualize custom data. It is generally intended that the granularity of the elements in the presentation component will be relatively high so that the logic and operation of an interaction element are separated from its definition. In this manner, the rendering of a specific interaction element can be left entirely to the corresponding target display system. For example, if a joystick element is specified, this specification will not contain the rendering specific attributes, but rather the abstract attributes that are assumed to be rendering independent.
manner, XIML creates a body of knowledge that can support design, operation, and evaluation functions for user interfaces. Attributes. In XIML, attributes [Puerta & Eisenstein, 2003] are features or properties of elements that can be assigned a value. The value of an attribute can be one of a basic set of data types or it can be an instance of another existing element. Multiple values are allowed as well as enumerations and ranges. The basic mechanism in XIML to define the properties of an element is to create a number of attribute-value pairs for that element. In addition, relations among elements can be expressed at the attribute level or at the element level. As in the case of relations, XIML supports definitions and statements for attributes, and also predefines some typical attributes of interest for UIs. In our context, each abstract attribute of a AIO is expressed by a XIML attribute. AIO are then defined as a set of abstract attributes.
Dialog. The dialog component [Puerta & Eisenstein, 2003] defines a structured collection of elements that determine the interaction actions that are available to the users of an interface. For example, a “Click”, a “Voice response”, and a “Gesture” are all types of interaction actions. The dialog component also specifies the flow among the interaction actions that constitute the allowable navigation of the UI. In our context, the dialog is not mandatory, but can be added when the behavior of the virtual world needs to be expressed. For example, a joystick may be left specified without any behavior attached to the specification of the direction (no dialog part) or may be connected to semantic functions, operations (with dialog part).
3.4 Envir3D: Automated Code Generation On basis of this XIML file, the 3D scene can be automatically generated in VRML97 (Virtual Reality Modeling Language) format. As defined in the VRML97 standard, it is a file format for the description of 3D objects and scenes. VRML was created to be used on the Internet, Intranet or autonomous clients. The objective of VRML is to be an universal interchange format for 3D graphics and multimedia. Currently, only the common attributes to all CIOs are taking into account during the translation between the scene described in XIML format and the final 3D scene in VRML format. A VRML file is composed of nodes. There are three important nodes for our purposes, namely Transform, Inline and Viewpoint. The node Inline is used to include in a scene a other scene or object whose the description is in an external VRML file listed in the field url of the Inline node. The node Transform allows to apply transformations, such rotation, scale or size, on an object of the scene. In the XIML description, each presentation element is associated with the VRML file that represent the generic graphical representation of this CIO. Basically the conversion tool of Envir3D includes the generic graphical representation of each CIO of the virtual UI, thanks to the node Inline. Each generic graphical representation has predefined size (one
Relations. The interaction data elements captured by the various XIML components constitute a body of explicit knowledge about a user interface that can support organization and knowledgemanagement functions for UIs [Puerta & Eisenstein, 2003]. There is, however, a more extensive body of knowledge that is made up of the relations among the various elements in an XIML specification. A relation in XIML is a definition or a statement that links any two or more XIML elements either within one component or across components. For example, “Data type A is displayed with Presentation Element B or Presentation Element C” (relation in italics) is a link between a domain-component element and a presentation-component element. By capturing relations in an explicit
55
top of a desktop and if its relative position with respect to the table is important, then the abstract attributes denoting the computer’s position are captured and locked. This process prevents the tool from creating an instance of a compound AIO that is not consistent with its initial definition.
unit for each side) and is added by the node Inline at the origin of the axis of the VRML scene. Transformations are then applied to resize, orient and move the included object at the desired position in the final 3D scene. Lastly, when the file is opened, the view is centered at the center of the scene thanks to the node Viewpoint. Fig. 8 reproduces the rendering of the previously generated workstation as generated by Envir3D. It can be rendered in any VRML-enabled browser (here, Cortona).
Unlocking a compound CIO. When an instance of a compound AIO is dragged from the AIO hierarchy and dropped onto the working surface area, a locked instance is created. However, when the need arises, it is possible to unlock the instance to relax constraints imposed on the values of some attributes, some primitives, etc. Any value can then be changed. The compound CIO that is unlocked no longer represents a true instance of the compound AIO. To foster consistency and to avoid adding extraneous AIO, such a unlocked compound CIO is stored as an instance of the initial compound AIO anyway, but with overriding. Concretization of a AIO. As any AIO can be associated with different CIOs depending on the target computing platform, we can distinguish basically different forms of concretization. When an AIO does not exist in a particular computing platform, a default CIO belonging to this environment is selected. When an AIO does have a unique concretization, it can be either by simulation (if the object does not exist natively but is simulated by a special implementation) or by direct selection (an existing CIO is used). The direct selection can map a 2D or 3D objects depending on the choice imposed by the designer. For example, a “Dial” AIO can be mapped onto - An edit box for a computing platform that does not support the dial. - A flat 2D dial for 2D or 3D UIs - A 3D dial when a corresponding objects exists in the library.
Figure 8. 3D VRML scene of the workstation. Creation of a compound AIO. An interesting feature of Envir3D is the possibility to add a new AIO to the AIO library based on CIOs of the current world. Once CIOs, concrete instantiations of AIOs, are created that constitute a new object per se, it is possible to abstract it into a compound AIO. For a compound CIO to be abstracted into a compound AIO, the CIOs that compose the group are analyzed along with their relationships, such as position, grouping, aligning. A new VRML object is created in the AIO library with all its sub-components. A new compound AIO is created by composition of the individual AIOs playing a role in the compound AIO. The newly created compound AIO therefore inherits all abstract attributes, events, and primitives from its children. The name of the compound AIO is specified (Fig. 9). An entry is inserted in the AIO hierarchy at an appropriate place (Fig. 10). Envir3D automatically generates the XIML definition file and the generic graphical representation for the new AIO. The corresponding XIML file therefore reflects the decomposition of the objects along with their AIO-CIO mappings [Nakatani et al., 83].
In the last two cases, the “Dial” object can be connected through a dialog part to the rest of an interactive application so that the true value of the dial is constantly associated with a behavior. A more complex example. To illustrate how the above operations are useful, the control room of a Japanese PWR Mitsubishi nuclear plant will be produced with Envir3D (Fig. 11). This control room can be decomposed into five identical workstations sharing same presentations and dialogs (behavior). Therefore, the first step to do in Envir3D consists of designing an individual workstation by creating the CIOs of the controls [Degani et al., 1992] appearing on the control board of the workstation (Fig. 12).
Figure 9. Creation of a new compound AIO.
Figure 10. After its creation, an new AIO is available as any other AIO. Locking a compound AIO. When a compound AIO is created, some abstract attributes that were initially left unspecified become frozen to preserve the structure of the composition and the assignment of values of attributes of interest. This process is called locking a compound AIO. For example, if a compuer is placed on
Figure 11. The control room of a Japanese nuclear plant.
56
Figure 12. Modeling of the control board.
Figure 14. Modeling of the complete control room.
Once a complete workstation is obtained, it can be declared as a compound AIO to be inserted in the AIO library. Fig. 13 shows the new AIO AIO_Case_study_Workstation that is created based on this model. In Fig. 13, a control panel has also been created to future usage. Once this AIO is available, five locked instances of can be assembled side by side to create the final setup (Fig. 14).
Figure 15. The final control room of a Japanese nuclear plant.
Figure 13. The two new compound AIOs. When the final setup is obtained, the virtual world is saved and the corresponding XIML file is automatically produced. The XIML here contains all the CIOs with their attributes, events, and primitives (if any used), the relationships between the CIOs, and the mappings between the CIO and the connected VRML objects. Simultaneously, a VRML file is automatically generated and can be manipulated in a VRML-enabled browser (Figs. 15 and 16).
4 Virtual User Interface Evaluation
Figure 16. Visualization from multiple points of view.
Since a XIML model of the virtual world is maintained internally to Envir3D at any time, this model can then be exploited for other activities that are traditional to model-based design, such as, model checking, consistency checking, automatic verification of properties of interest, both on the presentation and the dialog, automatic evaluation of the model quality. In particular, we are interested by testing as automatically as possible whether the current virtual world is compliant with usability guidelines that may be recommended [Smith & Walker, 2003; Wood, 99]. Properties to be assessed can be either static (when defined before use of the virtual UI – e.g., a knob should be large enough) or dynamic (when checked during the usage of the virtual UI – e.g., any knob should be turned in some consistent way throughout the UI).
In the US, the NUREG-0700 [O’Hara, 94 ; O’Hara et al., 94] represents an official source for governing the quality of interfaces of control rooms. Every control room should be to some extent compatible with guidelines expressed in this standard. These guidelines typically govern the location, the size, the format of objects, their functions to ensure that they are accessible enough. To support this activity, some usability guidelines from this standard have been incorporated in the tool to perform automated evaluation on-demand of these guidelines. For this to happen, the actual values of CIO attributes and their relationships are submitted to a static analysis to detect deviations between their values and those recommended by the standard. In this way, proper sizes and locations of objects can be assessed.
57
5 Virtualization of Non-Virtual User Interfaces To address the UI transition, the model-based approach can be exploited to migrate existing non-virtual UIs to the virtual world. This process is called virtualization, as it transforms an existing 2D UI into its 3D equivalent. This process is depicted in Fig. 17. HTML Page
Vaquita: reverse engineering XIML model
Envir3D: virtualisation of non-virtual user interface
VRML97/X3D file
VRML97-enabled browser
Figure 19. Part of the registration form.
Figure 17. Virtualization process.
The translation of the attributes of Vaquita to those of Envir3D was more difficult. As Envir3D, Vaquita sees the size of the elements of web pages as absolute size. This aspect was not difficult to resolve. As for the names of AIOs a list was written with the size attributes of Vaquita in one side and the size attributes of Envir3D on the other side. But Vaquita considers the position of a element in relation to others elements of the web page. On the contrary, Envir3D sees the position of the objects as absolute position. So a translation from relative positions to absolute positions was needed. Moreover, for each presentation attributes, AT_ORIENT, AT_SIDE and AT_TYPE were added with "front" as value for the two first attributes and "virtual" as value for the latter attribute. These three attributes are specifically to Envir3D. Finally all the other attributes are not modified as Envir3D do not take them in consideration when translating a XIML presentation model to a VRML file. After modification of the XIML file produced by Vaquita Envir3D can re-engineer the registration form web page to a 3D VRML scene.
Envir3D creates a VRML file based on a presentation model expressed in XIML. Vaquita [Bouillon & Vanderdonckt, 2002] reverse engineers any HTML page into a XIML presentation model. As Envir3D, Vaquita uses the AIO concept to express the elements of a web page. Starting from a XIML file produced by Vaquita, Envir3D can create a 3D VRML scene of the HTML page based on mapping tables (Fig. 18).
Figure 18. Mapping table between the physical and the virtual worlds. We illustrate this re-engineering process on a typical registration form. First Vaquita reverse engineers the HTML page and produces a presentation model expressed in XIML. This XIML description can not be used immediately by Envir3D. Indeed, the attributes of AIO's are not the same between Vaquita and Envir3D. Moreover some objects considered by Vaquita are not available in Envir3D and the names of AIOs are not the same between the two tools. We created new AIOs in Envir3D in order to have all the AIOs necessary to the re-engineering process of the web page. To supply the lack of coherence between the names of AIOs between Vaquita and Envir3D, we wrote a list where all the AIOs in Vaquita are listed with their corresponding AIO in Envir3D.
Figure 20. The original interface regenerated in VRML. Obtaining the same form in VRML was also a goal to incorporate 2D interfaces into a 3D world more easily while reducing the UI
58
transition effort. For example, a VRML UI can be incorporated more easily in a 3D desktop environment like SUN Looking Glass: http://wwws.sun.com/software/looking_glass/
7 Conclusion and Future Work
6 Rapid Prototyping of Virtual User Interface
In this paper, we have presented Envir3D, a tool that enables developers to graphically specify virtual worlds, while transparently capturing an underlying model that can be used for evaluation and re-engineering. The model was stored in XIML, a XMLcompliant language that is shared across many tools, whether they are helping the development of 2D or 3D UIs. To bridge the gap between the physical and the virtual world, a mapping table has been introduced to allow Envir3D to find out the AIO in the virtual world that is the closest possible to the AIO in the non-virtual world. One the main possible evolution of Envir3D is the taking into account a larger part of interaction. Currently all the 3D worlds are generated by Envir3D with their native behavior. For example, a list box displays the possible values and allows the end user to pick one value. But if the behaviour would impose that the selection of one of these possible values would affect an other part of the UI, this should be modelled in the dialog model and generated accordingly. This is not done currently. In the reality the quality of the interactivity between the users and the control room environment is highly important. The models (task, domain, user, dialog and presentation) included in XIML allows us to add this interactivity in the description of the scene. VRML allows also the addition of interactivity in the 3D scenes.
The first purpose of Envir3D was the modeling of virtual UIs. For instance, control room environment of nuclear plants in order to help engineers during the conception and the evaluation of control room environments, especially from a point of view of the ergonomics. But the scope of Envir3D can be easily extended to other domains of application. One of them is the conception and the evaluation of control boards of cars, trains,… We have modeled the control board of a Volkswagen Touran car (Fig. 21). Some AIOs have been manually added in order to have a realistic scene. We have not modeled all the control environment. Indeed, rearview mirror, wing mirrors and glasses are also part of the control environment. There are necessary to the driver to keep control of the car. That is the main difference with the domain of the control room environment. We cannot modeled all the control environment, only a part of it (Fig. 22).
Acknowledgements We gratefully acknowledge the support of the CAMELEON research project (http://giove.cnuce.cnr.it/cameleon.html), the CAMELEON partners for fruitful exchanges and discussions, and the SIMILAR network of excellence (http://www.similar.cc), the European research task force creating human-machine interfaces similar to human-human communication of the European Sixth Framework Programme. We also acknowledge the support from the Région Wallonne under contract WALEO 21/5129, as part of the MERCATOR research project (http://www.tele.ucl.ac.be/PROJ/Mercator_ Multi_e.html)
References BOUILLON, L., AND VANDERDONCKT, J. 2002. Retargeting Web Pages to other Computing Platforms. In Proceedings of 9th IEEE Working Conference on Reverse Engineering WCRE'2002 (Richmond, October 29November 1, 2002). IEEE Computer Society Press, 339–348. BYRNE , M.D., WOOD, S.D., FOLEY, J.D., KIERAS, D.E., AND SUKAVIRIYA, P.N. 1994. Automating interface evaluation. In Proceedings of conference on Human factors in computing systems CHI’94 (Boston, April 1994). ACM Press, New York, 232–237.
Figure 21. The control board of a Volkswagen Touran car.
CHITTARO, L., RANON, R., AND IERONUTTI, L. 2003. Guiding Visitors of Web3D Worlds through Automatically Generated Tours. In Proceedings of 8th International Conference on 3D Web Technology Web3D’2003 (Saint-Malo, March 9-12, 2003). ACM Press, New York, 27–38. DEGANI, A., PALMER, E.A., AND BAUERSFELD, K.G. 1992. «Soft» Controls for Hard Displays: Still a Challenge. In Proceedings of the 36th Annual Meeting Conference of the Human Factors and Ergonomics Society HFES’92 (Atlanta, October 12-16, 1992). Human Factors Society, Santa Monica, 52–56. JACOB, R.J.K., DELIGIANNIDIS, L., AND MORRISON, S. March 1999. A Software Model and Specification Language for Non-WIMP User Interfaces. ACM Transactions on Computer-Human Interaction, vol. 6, no. 1, 1–46.
Figure 22. The control board of a Volkswagen Touran car rendered in VRML.
59
MOLINA, J.P., GONZÁLEZ, P., LOZANO, M.D., MONTERO, F., AND LÓPEZJAQUERO, V. 2003. Dridging the Gap: Developing 2D and 3D User Interfaces with the IDEAS Methodology. In Proceedings of 10th Int. Conf. on Design, Specifications, and Verification of Interactive Systems DSVIS’2003 (Funchal, June 2003), LNCS Vol. 2844. Springer-Verlag, Berlin, 2003, pp. 303-315.
VANDERDONCKT, J. AND BODART, F. 1993. Encapsulating Knowledge for Intelligent Automatic Interaction Objects Selection. In Proceedings of ACM Conference on Human Aspects in Computing Systems InterCHI’93 (Amsterdam, April 24-28, 1993). ACM Press, New York, 424– 429. WATERHOUSE, J.F. 1996. A Comparison of 2D and 3D Interfaces for Editing Surfaces Reconstructed from Contours, Ph.D. thesis, University of Waterloo.
NAKATANI, L.H., AND ROHRLICJ, J.A. 1983. Soft Machines: a Philosophy of User Interface Design. In Proceedings of the 1st Conference on Human Factors in Computing Systems CHI'83 (Boston, December 12-15, 1983). ACM Press, New York, 19–23.
WOOD, J. 1999. Why do we need an international standard for control rooms?, Egham. Accessible at http://www.ccd.org.uk/communications/ control.html
O'HARA, J.M. July 1994. Advanced Human-System Interface Design Review Guideline, General Evaluation Model, Technical Development, and Guideline Description. Document NUREG/CR-5908 - BNLNUREG-52333, Vol. 1, U.S. Nuclear Regulatory Commission, Brookhaven National Laboratory.
ZHOU, M.X., AND FEINER, S.K. 1997. Top-down Hierarchical Planning of Coherent Visual Discourse. In Proceedings of 2nd International ACM Conference of Intelligent User Interfaces IUI'97 (Orlando, January 6-9, 1997). ACM Press, New York, 129–136.
O'HARA, J.M., BROWN, W.S., BAKER, C.C., WELCH, D.L., GRANDA, T.M., AND VINGELIS, P.J. July 1994. Advanced Human-System Interface Design Review Guideline, Evaluation Procedures and Guidelines for Human Factors Engineering Reviews. Document NUREG/CR-5908 - BNL-NUREG-52333, Vol. 2, U.S. Nuclear Regulatory Commission, Brookhaven National Laboratory/Carlow International Inc. PUERTA, A., AND EISENTEIN, J. 2002. XIML: a common representation for interaction data . In Proceedings of 7th ACM International Conference on Intelligent User Interfaces IUI’2002 (San Francisco, January 13-16, 2002). ACM Press, New York, 214–215. PUERTA, A., AND EISENSTEIN, J. 2003. XIML: A Multiple User Interface Representation Framework for Industry. In Multiple User Interfaces Cross-Platform Applications and Context-Aware Interfaces, A. Seffah and H. Javahery, Eds., Chapter 7, John Wiley & Sons, 2003, 119–148. SEBRECHTS, M.M., VASILAKIS, J., MILLER, M.S., CUGINI, J.V. AND LASKOWSKI, S. 1999. Visualization of Search Results: A Comparative Evaluation of Text, 2D and 3D Interfaces. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR’99. M.A. Hearst et al., Eds., ACM Press, New York, 3–10. SELIGMANN, D.D., AND FEINER, S. 1991. Automated Generation of IntentBased 3D Illustrations. In Proceedings of 18th Annual Conference on Computer Graphics and Interactive Techniques SIGGRAPH’91 (Las Vegas, July 28-August 2, 1991), Computer Graphics, vol. 25, no. 4, 123–132. SELIGMANN, D.D., AND FEINER, S. 1993. Supporting Interactivity in Automated 3D Illustrations. In Proceedings of the 1st International ACM Conference on Intelligent User Interfaces IUI’93 (Orlando, January 1993). ACM Press, New York, 27–44. SMITH, K. AND WALKER, B.A. 2003. Design of the control room environment, Denver. Accessible at http://www.isa.org/Content/Content Groups/Divisions1/Technical_Papers3/DesignoftheControlRoomEnviro nment.pdf SCHILLING, A., AND ZIPF, A., 2003. Generation of VRML City Models for Focus Based Tour Animation - Integration, Modeling and Presentation of Heterogeneous Geo-Data Sources. In Proceedings of 8th International Conference on 3D Web Technology Web3D’2003 (Saint-Malo, March 9-12, 2003). ACM Press, New York, 39–47. VANDERDONCKT, J. AND BERQUIN, P. 1999. Towards a Very Large Model-Based Approach for User Interface Development. In Proceedings of 1st IEEE International Workshop on User Interfaces to Date Intensive Systems UIDIS’99 (Edinburgh, September 1999). IEEE Computer Society Press, 76–85.
60
