AN INTERFACE MARK-UP LANGUAGE FOR WEB3D John D. Roberts1, Ilmi Yoon1, Sanghyuk Yoon2, Edward H. Lank1 San Francisco State University 1600 Holloway Ave., San Francisco, CA 94132 USA
[email protected],
[email protected],
[email protected]
Rocky Mountain Biological Laboratory 800 County Road 317, Crested Butte, CO 81224 USA
[email protected]
ABSTRACT Due to the lack of standards for 3D graphics on the WWW and for embedded systems, developers suffer as they are required to choose between different technologies or to support multiple technologies depending on the goals of an application. These decisions affect the final product by introducing restrictions based on the technology, or requiring additional overhead in order to support multiple technologies. Even though 3D graphics data itself can be relatively easily described in XML format, this still requires overhead for the developer to build a new graphical user interface (GUI) for each technology. In this paper we describe the design of a GUI mark-up language for use in generating and porting GUIs in multiple environments for the web or embedded systems. We describe the current contents of the mark-up language, called GUIML, along with the translation tools we have developed to create and render GUIML into the final technology.
which may be addressed by ongoing standardization efforts such as U3D or X3D. Aside from distribution problems, developers also are forced to attempt to address deficiencies in the presentation technologies themselves. Of special interest to the authors is the lack of a common specification for user interface generation for web 3D. Though individual technologies (OpenGL, X3D, Java3D, Shockwave) may include technology-specific widgets (i.e. lingo for Shockwave) or language-specific widgets (i.e. Glut for OpenGL) to attempt to alleviate this problem, no common libraries of user interface widgets exist that can provide a seamless user experience regardless of presentation format. Furthermore, many web 3D formats entirely omit user interface widgets, focusing instead on scene navigation and object examination within a 3D scene. In this paper, we describe on-going work to address the problems encountered while attempting to create uniform user interfaces for various 3D web technologies. In order to create a generic basis for our graphical user interfaces, we opted to create an XML specification called the Graphical User Interface Mark-up Language (GUIML). In creating this language, we have attempted to define both standard components of a graphical user interface, as well as the standard behaviours required of the interface components. Furthermore, we have attempted to create a specification that is easily extensible, so that additional components can be added quickly, and so that existing components can be modified easily. To this end, we have defined a mark-up language that is capable of abstracting all of the basic user interface functionality, as well as being supportable by the presentation formats (for example, position and color information is usable in most 3D presentation formats). The information encapsulated in the GUIML instance file therefore can specify exactly how each item is to be rendered in the multiple presentation formats by indicating such things as position, height, width and colour. In this paper, we first describe some background work in the design of internet delivered user interfaces. We then go on to describe our specific approach to UI delivery for web-based multimedia applications. Finally,
KEY WORDS Mark-up language, XML, User Interface mark-up language
1. Introduction In the Webs on the Web (WOW) project [1], the authors are creating dynamically generated three-dimensional visualizations of ecological data using a 3D web technology selected by the end user. A number of technologies are available, including VRML or X3D, Shockwave, and a Java Web Start based web applications. The difficulty encountered while generating visualizations using these various technologies is that there does not currently exist a method for generating a uniform user interface across the multiple visualization technologies, because of both a lack of standards for interface components in the various 3D web technologies , and a the lack of powerful client-side tools for generating complex user interfaces. The lack of a common standard still poses problems for web 3D content distribution on the internet, many of
we present the results of this work, specifically a realization of User Interface components delivered using a variety of 3D presentation formats. We conclude with a description of our on-going work.
2. Background
Internet applications are typically developed intensively on the server side to address a number of factors, including performance, improved service to clients, scalability, liability, and load balancing. Many efforts have been made to address these server-centric issues through a diverse assortment of technologies such as transaction managers, application servers, Active Server Pages, server-side scripting languages such as PHP, Java Server Pages (JSP), and other Java related technologies like the JSP Tag Libraries, and Enterprise Java Beans. In an attempt to address some of these same issues on the client side, clever combinations of HTML tags and primarily client-side scripting languages such as Javascript have been employed. The goal, here, is to introduce a client-side rendering mechanism that provides an appropriate user interface regardless of the technology being used. The specification of a variety of user interfaces based on technologies or devices is an active area of research. Approaches to this problem can be segmented into two broad categories. First, there are those technologies that specify user interfaces for one platform and then seek to provide mappings to other platforms. Second, there are a set of techniques that seek to separate the specification of the behaviour of a user interface from the representation of that interface. In the area of mapping a pre-defined user interfaces for one technology to other platforms, we point to a variety of research projects. The most general of these are plastic interfaces which allow user interfaces to transform themselves into forms appropriate for a variety of devices [2]. Other researchers have explored the transformation of User Interfaces for specific devices, for example from a standard desktop interface to personal digital assistants [3] or from a desktop interface to voice interfaces [4]. Our approach, GUIML, more closely follows the creation of a model of an interface which can then be rendered into an appropriate form for various devices. Several systems attempt to model abstractly and then generate a specific device or technology specific User Interface. These systems include the iCrafter system in the Stanford iRoom project, which analyzes a hardware device’s characteristics and generates the best UI for that device on the server. If no specific UI is available for that device, it will generate a UI at the highest possible resolution for the specified device [5] As well, the SpeakEasy project allows devices to broadcast capabilities, and then returns an interface based on the characteristics of that device [6]. Our approach is based on an XML-specification of a User Interface. A number of XML based applications exist that are capable of rendering a complete user interface based on the specifications of the XML instance
file. These applications generally provide both a visual environment for developing the user interface and generating the corresponding XML file, as well as the capability to parse and render the user interface from the generated XML instance document. The commercial applications such as Jeasy and OOP-Research’s Java Swing GUI Builder allow for the generation of user interfaces using the Swing libraries of the Java programming language. The Views Project performs in a similar manner, using a C# based approach to the same problem of generating user interfaces. Yet another project that has made significant progress in the area of XML descriptions of user interfaces is the User Interface Markup Language (UIML) [7]. The UIML project has been extensively developed, and appears to deliver user interfaces in C#, but still fails to provide a method of consistent rendering of user interfaces across multiple platforms. The drawback with many of these existing XMLbased approaches is that they are constrained to a specific environment. At this time, we are not aware of any existing widget libraries that translate a generic model of a user interface into appropriate web-based 3D graphics technologies. Furthermore, existing applications for the generation of user interfaces tend to be oriented towards standalone applications, and are incapable of being integrated with multiple application types and formats. Due to our focus on Web 3D, we contend that a language that specifically describes user interface components is necessary, especially in order to satisfy our goal of delivering uniform user interfaces across the plethora of Web 3D technologies. Furthermore, we require a language capable of tight integration with internet applications in order to effectively deliver both user interface and content to the client.
3. Conclusion
In order to create a generic basis for our graphical user interfaces, we opted to create an XML specification called the Graphical User Interface Markup Language (GUIML). In creating this language, we have attempted to define both standard components of a graphical user interface, as well as the standard behaviours required of the interface components. Furthermore, we have attempted to create a specification that is easily extensible, so that additional components can be added quickly, and so that existing components can be modified easily. Figure 1 is an example GUIML file. GUIML currently specifies the following user interface components in its Schema: panel, button, listmenu, slider, textfield, label, imagefield, and canvas. In addition to these standard graphical user interface components, we have also included a specification for component behaviour and user interaction. This element, called the callback element, allows the GUIML to specify an object which will interact with the instance document’s object model to retrieve user interface information, and relay it to the 3D presentation technology.
4. Application Integration
Figure 1: A GUIML instance file which generates three sliders. Finally, elements and attributes were created that allowed specification of the placement of the individual interface components. In order to allow the mark-up language to be flexibly translated to multiple technologies while still preserving the appearance of the user interface, we opted to position elements using an HTML table-like layout. To this end, attributes such as row and column were added to allow for this placement, as well as column and row span elements to allow individual interface elements to span multiple table cells. Additionally, other standard interface descriptors were added, like foreground and background colour elements, to allow flexible control over the appearance and presentation of the user interface. Figure 2 is the realization of the GUIML file shown in Figure 1 on the client side.
Figure 2: Using the preceding XML file to specify sliders controlling illumination in a 3D scene.
After the development of a standard method of representing the graphical user interface that will be made available to the client, the next step was to create a means of integrating the user interface specified by the GUIML into the existing web application. In the existing instance of the 3D web application we were working with (WOW), users selected a data set and a presentation format (Shockwave, VRML, X3D, etc.), and the server generated the visualization in the appropriate format and sent the generated information to the client. As noted previously, this visualization often included format-specific user interface components, resulting in an uneven distribution of application functionality across the various format types, as well as relying heavily on server-side processing to complete the request. Also, in order to update visualization information, costly requests were required between the client and the server, often requiring a large data exchange in order to re-retrieve datasets and perform additional calculations. Figure 3 depicts the typical system architecture.
Figure 3: The standard client-server request model for distribution of our visualization data. 4.1 Distribution of GUIML In order to facilitate the distribution of the GUIML instance file with the 3D visualization data, we added a transparent level of indirection to the web application. Where previously users interacted with processes on the server in order to select dataset information as well as the presentation format in which the dataset would be visualized, users now select only presentation format. It should be noted that this was a design decision unique to our web application; dynamic selection of the appropriate format could be made on the server side depending on the capability of the access point to our application (i.e. the client’s web browser), but this method allows for greater control by the end user. Figure 4 shows the new architecture of GUIML. Once the determination of presentation format has been made, a process is launched on the client side to receive and interpret the GUIML instance file into the appropriate format. Our instance files are designed in order to allow the user to select visualization datasets from within the user interface, thereby providing the seamless transition from the old web application to the
new web application with the standardized user interface. In addition to this, datasets as well as modules to allow for client-side calculations on the visualization dataset can now be cached to remove a great deal of processing burden from the server to the client.
general in order to allow for the leveraging of formatspecific event handling. The callback element specifies the name of an object or method which implements the desired behaviour. This allows, for example, the instantiation of a Java object in our Java Web Start application that can interact with the document object model, and appropriately update the visualization. Similarly, the VRML version simply uses the object name in a script node, allowing event processing to be handled by a more powerful programming language, like Java.
5. Implementation and Ongoing Results
Figure 4: The new client-server model including the GUIML translation mechanism. 4.2 Language Mappings The transition process from the old style of serving the visualization to the new style of providing the graphical user interface as the access point to the visualization glosses over the inner workings of the translation process. In order to make GUIML useful as a means of providing a standardized graphical user interface across the multiple presentation formats, a mapping was required from the mark-up language to each of the presentation formats supported by our web application. To this end, we created a mark-up language that was capable of abstracting all of the basic user interface functionality, as well as being supportable by the presentation formats (for example, position and colour information is usable in most 3D presentation formats). The information encapsulated in the GUIML instance file can specify exactly how each item is to be rendered in the multiple presentation formats by indicating such things as position, height, width and colour. In addition to the question of basic layout, the definition of the elements in the presentation format needed to be defined. As examples, in the case of the Java version of our application, the translation mechanism could simply build the graphical user interface using elements from the Swing library; in the VRML/X3D format will have a library of widgets defined as external proto objects, and these object can then be transformed to the specification of the instance file. 4.3 Supporting User Interaction Once the basic user interface has been rendered, a means of supporting interaction with the interface must be provided, along with preserving the possibility of making use of the functionality of the underlying presentation format. As mentioned previously, a special element, called the callback element, was declared in the mark-up language that allowed for the specification of a means of interaction between the user interface and the presentation format. As with the other component of the GUIML, the specification of the callback element was intentionally
The current implementation uses Java Web Start technology, combined with the Java OpenGL bindings (JOGL) and Swing, to produce a usable and flexible means of specifying a user interface co-existing with 3D content. One simple application that was designed to test the implementation consists of a simple GUIML instance document that specifies three sliders, along with a canvas and callback elements for each object. The mark-up in Figure 1 and the interface in Figure 2 depict this simple example. The callback objects allow the use of the sliders to control the orientation of the lighting in the rendered image.
Figure 5: An application window containing examples of many of the GUIML widgets. Another more complicated example is shown in Figure 5. In Figure 5, we see a number of the GUI components that we can render using various 3D presentation technologies with our architecture. Figure 6 is a depiction of the GUIML file used to generate the window in Figure 5.
6. Conclusion and Future Work
This paper has discussed the development of a graphical user markup language, called GUIML, in order to provide a standardized method of user interface generation across multiple web 3D technologies. The development of the graphical user interface markup language seeks to abstract general user interface component and functionality, while
simultaneously providing a rigid definition of appearance in order to preserve the similarity of user interface components across multiple web 3D technologies. Our results have been promising as we continue to develop libraries for additional Web 3D technologies.
3D formats converge to a universal format. One such attempt at standardization in the form of X3D appears to be a promising area for the implementation of GUIML as a standard method of specifying user interfaces.
7. Acknowledgements
This project is being funded by National Science Foundation Div. Of Biological Infrastructure, Biological Databases and Information, No. DBI-0234980.
References:
Figure 6: The GUIML instance file that displays examples of many of the implemented widgets. Our work continues in various directions. First, additional language mappings need to be defined in order to make the GUIML and its associated translation service usable across additional multimedia formats. We are currently experimenting with Lingo for Shockwave, as well as with C# in order to provide the same functionality on multiple platforms. C# looks especially promising due to the ease with which applications can be developed for handheld systems supporting the Windows .NET compact framework. In addition to this, we are looking at ways that the callback functionality can be defined more generally, to allow both increased functionality within the user interface, as well as increased ability to interact with the presentation format chosen by the user. A means of tighter integration with the presentation format is needed, especially in the area of allowing for client-side processing of user-generated events, without the need to send events to a format-aware remote processing object or script, which may still be required by some web 3D technologies. We anticipate that as standardization efforts progress, many of these problems will be solved as the various web
[1] I. Yoon, R. Williams, E. Levine, S. Yoon, J. Dunne and N. Martinez, Webs on the Web (WOW): 3D Visualization of Ecological Networks on the WWW for Collaborative Research and Education, Visualization and Data Analysis 2004, IS&T/SPIE Symposium on Electronic Imaging, January 2004. [2] G. Calvary, J. Coutaz and D. Thevnin, A Unifying Reference Framework for the Development of Plastic User Interfaces, Proceedings EHCI 2001, Toronto, May 2001, pp. 11 - 13 [3] O. Buyukkokten, H. Garcia-Molina, A. Paepcke, and T. Winograd, Power Browser: Efficient Web Browsing for PDAs, CHI 2000, pp. 430 - 437. [4] E. Mynatt and K. Edwards, Mapping GUIs to Auditory Interfaces, UIST 1992, pp. 61 – 80. [5] B. Johanson, A. Fox, T. Winograd, The Interactive Workspaces Project, IEEE Pervasive Computing Magazine 1:2, April-June 2002, pp. 71 – 78. [6] M. Newman, S. Izadi, K. Edwards, J. Sedivy and T. Smith, User Interfaces When and Where They are Needed, UIST 2002, pp. 27 – 30. [7] M. Abrams, C. Phanouriou, A. Batongbacal, S. Williams, J. Shuster, UIML: An ApplianceIndependent XML User Interface Language; http://www8.org/w8-papers/5b-hypertextmedia/uiml/uiml.html.
