SUMIT: A Virtual Reality Embedded User Interface ...

Proceedings of Virtual Concept 2005 Biarritz, France, November 8th – November 10th, 2005

SUMIT: A Virtual Reality Embedded User Interface Prototyping Toolkit Michael Quinn 1, Joshua Moline 2, Aaron Ziegler 3, Paul Pukite 4, Daniel Challou 5 (2) :United Defense, L.P. Armament Systems (1) :United Defense, L.P. Armament Systems Division M250 4800 East River Road Division M250 4800 East River Road Minneapolis, Minneapolis, MN 55421, USA MN 55421, USA Phone763-572-7202/Fax 763-572-7077 Phone 763-572-7202/Fax 763-572-7077 E-mail : [email protected] E-mail : [email protected] (4) : United Defense, L.P. Armament Systems (3) :United Defense, L.P. Armament Systems Division M250 4800 East River Road Division M250 4800 East River Road Minneapolis, Minneapolis, MN 55421, USA MN 55421, USA Phone763-572-7202/Fax 763-572-7077 Phone763-572-7202/Fax 763-572-7077 E-mail : [email protected] E-mail : [email protected] (5) : United Defense, L.P. Armament Systems Division M250 4800 East River Road Minneapolis, MN 55421, USA Phone763-572-7202/Fax 763-572-7077 E-mail : [email protected]

Abstract: United Defense L. P. Armament Systems Division has created the Simulation User Machine Interface Toolkit (SUMIT). SUMIT is a web-enabled rapid development environment for the fast synthesis of operational user requirements. SUMIT enabled us to generate an operational conceptual user interface and integrate the interface with virtual vehicles (i.e., user interface, simulation software middleware, virtual hardware, and synthetic environment) in under one month. We then used the resulting virtual vehicle prototypes to support immersive design reviews. SUMIT provides a three dimensional interactive context in which the proposed UI design can be exercised. SUMIT-generated crew interfaces are embedded in a virtual vehicle that is High Level Architecture (HLA) compliant, so they can be interfaced to, and run real-time in distributed collaborative exercises. SUMIT's built-in trace functionality produces Extensible Markup Language (XML) data in real-time. The XML data is used to automatically generate Unified Modeling Language (UML) sequence diagrams and automated system test cases for verifying key performance-based requirements. SUMIT is completely constructed out of open-systems software components including OpenGL, Python, and XML. Key words: Virtual Prototype; User Interface; Immersive Design Environment; UML; Immersive Design Toolkit;

environment. A complete prototyping environment provides the capability to model both the hardware and software aspects of a product's user interface in the context of the product's operational environment. Virtual reality techniques employed by SUMIT allow interaction with a product model before any hardware is built and even before any on-board software is written. In a modeling and simulation development environment, change is a paramount concern. Models are used early in a product's development cycle to elicit requirements from the customer and users [1]. As development proceeds, models are used to quickly try design changes and assess their impact on the system's usability and fitness for its purpose. User interfaces in particular are well suited to an agile model based development approach. The complexities of human cognition are difficult to design for. This leads us to use an iterative approach when designing user interfaces. A prototype UI is developed, it is tested, changes are made, and it is tested again [2]. When the UI is embedded in a complex hardware and software implemented product, the level of design difficulty increases again. Not only must the UI change, the entire system must change to meet constantly changing customer and user requirements.

We employ an Integrated Virtual Prototype (IVP) to model the entire system in context. An IVP includes models of United Defense is a vehicle integrator developing products for Human Computer Interfaces, mechanical designs, software the U.S. Government. We have developed SUMIT to answer prototypes, physical simulations, external stimuli, and our need for a complete user interface prototyping environmental conditions. The IVP can be executed in real1- Introduction

Paper Number 122

-1-

Copyright Virtual Concept

Virtual Concept 2005

SUMIT: A Virtual Reality Embedded UI Toolkit

time to exercise the design in its operational context. The IVP take advantage of the Graphics Processing Unit (GPU). constantly integrates the product throughout its development Apple has accelerated their Quartz rendering engine using the GPU [10]. Cairo’s underlying glitz library [11] uses the life cycle. same approach, promising GPU accelerated windows icons, SUMIT is designed to meet our need for a flexible rapid menus and pointing interfaces. Sun’s 3D desktop experiment prototyping tool providing UI integration with IVPs of “Looking Glass”[12] uses Java 3D to “hang” windows in a complex real-time systems. These systems typically require 3D dimensional room-like environment. Apple’s desktop multiple users interacting through multiple interfaces to switcher maps the 2D UI graphics to a cube and rotates the accomplish concurrent time critical tasks. Moreover, the users 3D geometry to emphasize the switching metaphor. Most of are responsible for safely managing automated equipment and these experiments take the two dimensional desktop and map processes while maintaining a high level of situational it to a simple 3D object such as a plane or cube. awareness. Pad++ [13] takes a 21/2-D approach, applying multiple levels of detail to a very large virtual 2D desktop. Pad++ allows the user to pan and zoom in an out of varying levels 2- User Interface Toolkits of information detail, providing a spatially consistent model We divide existing user interface toolkits roughly into two of the data. dimensional, two and a half-dimensional and threedimensional categories for the purposes of this discussion. Two Much of the research in the area of 3D user interfaces dimensional (2D) GUI toolkits provide a usually fixed set of focuses on improving the methods of construction of virtual widgets that enable the familiar “desktop” interaction spaces, interaction with large scientific datasets and metaphors. These toolkits comprise text boxes, buttons, menus, navigation within them. Conner’s 3D widgets [14] have been radio button groups, sliders, and scroll bars among others. built upon for years and put to good use in 3D authoring and Many two dimensional toolkits extend into the two and a half data visualization environments. dimensional category by allowing the overlapping of windows, menus and other widgets emulating a desktop with stacked, Several virtual prototyping methods include the integration overlapping paper documents. The vast majority of current of CAD data to provide interactive virtual products GUI toolkits fall into these two categories. representing hand-held consumer devices [15][16]. The user 2D GUI builder products are ubiquitous [3][4][5]. Many commercial packages focus on evolutionary prototyping for multiple deployment targets. The GUI prototype is developed in a 2D space and then tied to the system software via functional callbacks. These tools typically provide a proprietary graphics layer that insulates their layout and design environment from the rendering libraries. These packages usually try to remove any code development from the process, relying instead on state charts, flow diagrams and “channel” or “slot” type data connections. The emphasis with these packages is on “drag and drop” ease of use for the nondeveloper.

is allowed to rotate the virtual device to view it from different angles and sometimes is allowed to interact in a scripted fashion with the user interface pages. These virtual product prototypes most closely approximate SUMIT’s approach and motivation. 3-Motivation

To fully understand the operational concept of the design, the customer, user and engineer must be allowed to interact with the model. SUMIT is designed to enable this interactivity. 2D GUI toolkits limit understanding of embedded user interface product integration. A disconnect is immediately The layout tools in these packages re-implement (with varying introduced between the CAD represented hardware and the degrees of success) many of the graphical authoring techniques 2D GUI prototype when they are physically separated on that have been developed and refined in tools that are well different computers. known to graphic artists. This drives up the cost of the packages and results in expensive proprietary licensing. The A fully integrated virtual prototype enables a user to eventual auto-generated code is typically heavy, cryptic and approach and interact with the physical representation of the difficult to maintain and adapt. product and its user interface in the same continuous virtual environment. SUMIT enables this capability. Further, the X Window System [6], Windows GDI [7], Swing [8], and operational concept of a complex product is affected by the Cocoa [9] based toolkits provide a standard set of 2D and product’s environment (both physical and operational). 21/2D widgets that give the developer little opportunity to change the “look and feel” of the interface. They provide some SUMIT User interface prototypes are embedded into a larger customization capability, but the process is usually quite product model that can execute the logical and physical involved and requires extensive programming. operations of the proposed product [17]. Virtual Reality (VR) techniques and tools allow the visualization of, and Open Source GUI development packages typically focus on 2D interaction with this model in its proposed operational interfaces and even those that use OpenGL do so to implement environment. 2D widgets that mimic their X-type counterparts. The run time product model is used as a reference model to Some Open Source and proprietary toolkits are beginning to guide engineers as they analyze the system and its proposed

Paper Number 122

-2-

Copyright Virtual Concept

Virtual Concept 2005

SUMIT: A Virtual Reality Embedded UI Toolkit

design. SUMIT enables the model to generate analysis and and handles the translation of mouse, touch screen and design artifacts that encode its execution in the form of swim- joystick events into messages bound for the SUMIT Screen lane diagrams. Multiple user interfaces may be needed to logic layer. accomplish concurrent tasks in a complicated system. SUMIT generated user interface requirements provide concurrent swim-lane diagrams showing the overlapping user interface tasks and their interactions. Additionally, all simulation data needed to drive the UI and to accept commands from the user can be logged during execution. These data, in the form of eXtensible Markup Language (XML) messages, can be inspected in their raw form and/or post-processed via eXtensible Style Sheet Language Translator (XSLT) translation into html pages showing data flow. This data capture capability is particularly useful in conjunction with User Interface Testing. Users are guided through a series of tasks executed upon the Integrated Virtual Prototype. The user’s reactions to the system’s UI are captured via video, audio and notation. The resulting user test data can then be correlated with the SUMIT data logs to give Human Factors Figure 2: SUMIT input sequence. and Systems Engineers a clear picture of user/system interaction. 4- Architecture

SUMIT is built around a three-tier architecture concept that follows the “model view controller” pattern. This pattern is commonly used to de-couple the system's domain model from its controller logic and the display logic presenting the user interface output graphics and input devices. Loose coupling of these layers gives the developer a great degree of agility when changes are required. SUMIT fits into an overall system model that encodes the important hardware, software and user events needed to accomplish a mission scenario. The components of the overall model interfacing to SUMIT are illustrated below.

Figure 1: SUMIT Architecture.

The System Simulation Object Model (SSOM), or simulation model, represents the business or plant logic of the system. It provides an HTML engineering interface which exposes its internal processes and data. Some of the objects in the SSOM represent user task state machines. They are simply state transition networks that wait for generic user events to enter via a command pattern web stimulus. The SUMIT Screen Logic layer contains logical abstractions of the user interface tasks and displays. The tasks are grouped into Contexts which are roughly equivalent to "screens", "frames" or "gadgets" in other user interface paradigms. The contexts provide stimuli to the SSOM's user task state machines in response to Presentation layer inputs. The Screen logic layer holds "screen state" and is aware of such attributes as the number of users, their roles, their current presentation state and their possible input options. The SUMIT Presentation layer displays graphics

Paper Number 122

-3-

Figure 3: SUMIT output sequence.

The SUMIT Screen Logic Layer contains Contexts and Artifacts. A SUMIT Artifact represents the logical half of a live graphical "widget". It may represent a label, button or indicator such as a speedometer. The Artifact is a logical unit that may be aware of some of its Presentation Layer counterpart's geometrical attributes. The deflection of a needle on the speedometer is an example. The Artifact may be an active object, in that it may periodically request a value from the simulation model layer representing a real-world value such as the vehicle's speed. The Artifact contains any data transformations that may be needed to translate its model data to geometrical screen display values. A Screen Logic Layer Context, on the other hand, represents a grouping of graphical artifacts and/or a logical grouping of commands and responses. The Context is analogous to a "gadget" or a "frame". It contains a collection of Artifacts and is aware of their logical interdependencies. The Context may accept a button push event from the Presentation Layer and then update the positions or states of several Artifacts to reflect the results of the input. The Context may also send commands via HTTP requests to the simulation model layer. Typically, an Artifact only reflects the model's internal state, whereas the Context may change the model's internal state.

Copyright Virtual Concept

Virtual Concept 2005

SUMIT: A Virtual Reality Embedded UI Toolkit

The model's user task state machines respond to high level logical operations. The Screen Logic Layer's Context state machines respond to screen level user operations. The Screen Logic Layer's Artifact state machines respond to "widget" level changes. 5- Operation

updates to the Presentation layer. The user now presses the "Displace" button and the Screen Logic Layer forwards the message to the simulation model. The model responds by changing its user task state to "displacing" and then initiating the crane and spade retractions. The Screen Logic Layer reflects the model's internal state by periodically querying the engineering interface for the spade state and the crane state. It then re-computes the required graphical state of its screen Artifacts and sends messages reflecting those changes to the Presentation Layer. After the re-configuration of the crane and spade hardware, the simulation model changes its crew task state to "Moving". The Screen Logic Layer is made aware of this change via its system state Artifact which is polling for system state changes. This artifact detects the change and then forwards a new string to its corresponding Presentation Layer graphic which then indicates "MOVING".

Let's look at a particular user task to illustrate a SUMIT UI in operation. Our example is a notional hazardous waste handling robot. The robot is remotely tele-operated. One common user task is to prepare the robot for driving. Our initial state is "locked down"; the system is parked with its spades down supporting crane operations. The driver wishes to transition to a mobile state and move to a new location. The user presses the "Move" button and the system responds by retracting and locking down the crane, retracting the spades and awaiting driving commands. The user places the system in gear and then accelerates and steers using the joystick. The speedometer, gps location and attitude indicator update in response to vehicle 6- UI Content in Immersive Environments movement. The on-board cameras update their views on the UI The content in the user interface is displayed in SUMIT's screen. Presentation Layer using OpenGL. Individual UI elements such as push buttons are drawn as simple rectangular primitives with texture mapped images to supply details such as icons and drop shadows. SUMIT renders text glyphs as individually textured polygons. SUMIT also provides a dynamic texture object that can be used to display video or map data. These dynamic textures can be applied to any graphical element making resizing of video view ports as simple as scaling a rectangle. "Custom" widgets such as compass dials, attitude direction indicators and animated icons can be constructed directly from geometrical primitives or using texture mapped transparent layers. All SUMIT graphics are treated as OpenGL polygons, triangles and lines, giving us great flexibility in the look of the UI. The texture maps representing a button, for example can be swapped out, (even at run time) changing the look of the element without affecting its behavior.

Figure 4: A notional tele-operated robot. rendered in the SUMIT virtual environment

This image was

Initially, the simulation model user task state machine is in the "lift" state. The user presses the "move" hardware button to look at the move screen. The "move" button geometry is picked via an OpenGL picking traversal. The "move" geometry pick triggers the sending of an XML message to the SUMIT screen logic layer indicating the selection of the "move" button. The "Lift" display Context in the Screen Logic Layer receives the input and responds by unloading itself and loading the "Move" display Context. The "Move" display Context loads its constituent display Artifacts which in turn load their geometry descriptions into the Presentation Layer. The display Artifacts then drop into their steady state tasks and periodically grab data from the simulation model and send

Paper Number 122

-4-

Figure 5: An operator’s station in the SUMIT virtual environment

The use of OpenGL for the screen artifacts also opens up the possibility of embedding the screens in the system CAD

Copyright Virtual Concept

Virtual Concept 2005

SUMIT: A Virtual Reality Embedded UI Toolkit

model. The user can be immersed in the overall system model and interactively affect it. SUMIT's graphical components are easily embedded in a larger three-dimensional virtual environment. In fact, we have used the SUMIT renderer to display 3D maps, terrains and vehicles. SUMIT may be used interchangeably with other 3D scene graphs in our simulations. By holding a well defined yet flexible XML interface to our scene graph tools, we are able to send vehicle level simulation data back into the SUMIT renderer. This allows full UI interaction with the virtual vehicle.

presentation and portability to a wide range of display devices. Our SUMIT Screen Logic Layer is currently implemented in the Python scripting language. This language provides an interpreted object oriented development environment. Python also easily supports http-based interfaces and XML processing. We built a core collection of Python architectural constructs which support the manipulation of display contexts and artifacts, event handling, message passing and multi-threading. We then extended this core toolset with canonical UI constructs such as push buttons, radio button groups, scroll widgets, range widgets, etc. When applying the toolset to a particular system model, we inherit from the base artifacts and contexts to provide specific logical UI functionality. Since SUMIT is built around a three tier layered architecture, network communications are an important enabler. We chose to adopt the open XML standard for data and graphics representation. The Presentation layer is able to accept inputs via XML and then transform that data into geometrical screen and CAD artifacts. It then accepts device events and translates those into XML representations for shipment to the Screen Logic Layer. The Screen logic layer also transforms XML into actions and data that are then used to stimulate the system model layer.

Figure 6: An embedded maintenance interface in the SUMIT virtual environment

Physical devices such as variable function programmable (VFP) buttons, joysticks and control panels can be imported into SUMIT from CAD data. CAD models of these devices may be the only existing representation. SUMIT allows the continuous and discrete inputs on these devices to be interactively tested in their operational context even before they actually exist. These virtual input devices can be connected to the simulation model via XML web requests. These requests stimulate the model to accomplish such tasks as driving the vehicle, picking graphical display objects or even moving the axes of robotic arms. CAD models of proposed display hardware can be integrated into a SUMIT virtual environment. Proposed arrangements of displays in a workstation can be tested interactively and even modified on the fly.

Portability is provided by the use of these open standards technologies. Web based communications media are inherently cross-platform. OpenGL is supported on almost all computing platforms which makes our Presentation Layer mobile via a thin windowing system specific layer. Python is supported on Linux, MS Windows, Sun, and MacOS. Since it is an interpreted language, its code is automatically portable. 8- Authoring Tools

SUMIT's approach to content creation follows the same flexibility rule as its implementation. We avoid the creation of custom authoring tools by re-using the many sophisticated open source and commercial tools already available.

7- Open Standards Technologies

Human Factors Engineers work in a variety of formats such as html, Flash and even Power Point. We import these graphics as simple image data and then segment them into SUMIT graphical artifacts through image manipulation tools and 3D modelers.

Recent advances in graphics cards have made the rendering of detailed 2 and 3 dimensional graphics faster and more easily affordable. Tools developers are struggling keep pace and fully exploit the new hardware features. Our philosophy with SUMIT has always been to keep the graphics separated from the logic so that the graphics can be easily updated and changed. OpenGL provides an open graphics standard which is capable of representing a large range of graphical presentations while still enabling exploitation of graphics card advances. In SUMIT, we treat screen objects as OpenGL floating point graphical entities that can be arbitrarily transformed and textured. This allows flexibility in

Authoring for SUMIT is accomplished in a variety of commercial and open source tools. We have used Adobe's Photoshop, Alias Wavefront's Maya, Centric's Designer's Work Bench and other commercial 2 and 3D tools to manipulate screen images, textures and CAD data. Wings 3D, the Gnu Image Manipulation Program (the GIMP), Inkscape and Art of Illusion (AOI) are among the open source tools we have used to author SUMIT data. SUMIT's toolset includes a converter that takes output files from these tools and produces files in SUMIT's XML data format. This simple converter abstracts the authoring from the runtime

Paper Number 122

-5-

Copyright Virtual Concept

Virtual Concept 2005

SUMIT: A Virtual Reality Embedded UI Toolkit

renderer, saving us much development time that might defined a typical user task to include an end to end system otherwise have been devoted to the development of custom prototype of a single use case such as “drive a pre-planned route”. Such a task might include on the order of 3 to 5 authoring tools. screens or screen states and 5 to 10 widgets per screen integrated with the underlying software models. Using SUMIT, we are able to accomplish the same amount of work 9- Automatically Generated Engineering Artifacts in 3 to 4 days. We recently turned out a complete Integrated Since all the events, data and graphics are encoded in XML, Virtual Prototype of a vehicle including five user tasks in one they can easily be logged by the Screen Logic Layer and later month. played back to illustrate the user's interactions with the system and the data required to enable those interactions. We use XSLT to transform the XML data into html pages illustrating 12- Conclusion the look and temporal flow of the user interface. SUMIT contains a data logger that provides a stream of XML encoded Years of hard experience with User Interface prototyping events, graphical data and communications data. The events have led us to develop SUMIT: a flexible toolset that correspond to user inputs such as button presses, releases and separates the simulation logic from the graphics and enables mouse drags. The graphical data is in the form of snapshots the communication and analysis of system engineering data and also XML that represents the screen graphics displayed at for embedded user interfaces. SUMIT is a step toward a the time of the event. The communications data represents all future system prototype that is fully interactive and the UI interactions with the simulation model. immersive for the user and the designer. Selected events can be pulled from the log data and played back via our test tools. This playback functionality allows a pass/fail determination of UI behavior. The simulation passes the test if the UI completes the pre-recorded task by presenting the correct input options at the right time. Confirming the correct graphical layout of the UI is still a manual task, but may be aided in the future through use of the graphical log data. Visual UML swim lane diagrams are produced from SUMIT XML log data via XSLT scripts. An engineer can view the swim lane diagrams in an html browser. Thumbnail images of each screen change are embedded in the diagrams next to the simulation time of each event. Each display device may occupy a swim lane in the diagram. This way, concurrent changes on different displays can be visualized. This postprocessed data has been used as input to software requirement documents. When these documents are combined with the simulation reference model, engineers have a clear specification of what is to be built. SUMIT display contexts encode their interconnections by referring to each other. A static map of the UI's connectivity is obtainable from this data. 10- HLA Connectivity

Some UDLP simulations are HLA enabled. They can be connected to an HLA exercise and interact with other simulations in a virtual world space. SUMIT is easily connected to these simulation models and has been used to test virtual vehicles in large-scale HLA exercises. We have also interfaced SUMIT implemented models with United Defense’s GVIS HLA viewer. The use of a network of HLA enabled simulations provides a higher fidelity operational environment and improves our predictions of overall system performance. 11- Results

Before the development of SUMIT, we used several commercial GUI builder toolkits. Typical development time using these tools was two to three weeks per user task. We

Paper Number 122

-6-

Copyright Virtual Concept

Virtual Concept 2005

SUMIT: A Virtual Reality Embedded UI Toolkit

10- References

[1] Sommerville I, Sawyer P, Requirements Engineering, 203207, Wiley, 1999. [2] Nielson J, Usability Engineering, 93-98, Morgan Kaufman, 1993. [3] VAPS Human Machine Interface Prototyping and Development Tool. www.engenuitytech.com/products/VAPS. [4] Altia, Altia, Inc. www.altia.com. [5 ] GLStudio, Distributed Simulation Technology, Inc. http://simulation.com/products/glstudio. [6] X Window System, www.x.org. [7] Windows OS, www.msdn.microsoft.com [8] Java Swing, java.sun.com [9] Cocoa, www.apple.com/developer/cocoa [10] Apple Quartz Extreme, www.apple.com/graphicsimaging/quartz [11] Cairo, http://cairographics.org/introduction [12] Project Looking Glass www.sun.com/software/looking_glass. [13] Pad++: Zoomable User Interfaces (ZUIs), www.cs.umd.edu/hcil/pad++/ [14] Brookshire D. Conner, Scott S. Snibbe, Kenneth P. Herndon, Daniel C. Robbins, Robert C. Zelesnik, Andries van Dam, Three-Dimensional Widgets, in Proceedings of the 1992 Symposium on Interactive 3D Graphics, 1992. [15]Viewpoint Media Player, www.viewpoint.com/pub/products. [16] RCID virtual www.rcid.ncl.ac.uk/virtual/virtual.html.

prototyping

[17] Pukite P, Ph.D., Sallman W, Challou D, Ph.D., Prototyping Ada Component Environment (PACE) – Software simulation middleware for the rapid development of Virtual Prototypes in Proceedings of the 23rd Annual Army Science Conference, December, 2002.

Paper Number 122

-7-

Copyright Virtual Concept