Tangible User Interface for Chemistry Education: Portability, Database, and Visualization
Joakim Almgren, Richard Carlsson, Henrik Erkkonen, Jonas Fredriksson, Sanne Møller, Henrik Rydgård, Mattias Österberg, Morten Fjeld Chalmers University of Technology, Gothenburg, Sweden {vilgon, cri, erkkonen, jonafred, traco, tronic, osterber}@dtek.chalmers.se,
[email protected] Abstract Augmented Chemistry (AC) is a tangible user interface (TUI) for organic chemistry education. Based on the outcome of an extensive evaluation we are in the process of improving the AC system. Firstly, it has been ported to different operating system and now runs on Linux-, Windows-, and Mac OS X based platforms. This enables the use of a wider range of hardware; USB and Firewire (IEEE1394) cameras are now supported. Secondly, system capacity to visualize molecules compatible with XML-based public-domain databases (DB) has been implemented. This gives users the ability to download and interact with any molecules. Thirdly, to enable keyboard-free system configuration and offering DB access, a GUI has been carefully implemented into the TUI. 3D rendering is also being improved using shadows and related effects, thereby enhancing depth perception and graphical overlay realism.
1. Introduction With scientific input from the Swiss Federal Institute of Technology (ETH) 1, Augmented Chemistry (AC) was realized at HyperWerk FHBB2 [1][2]. It was further developed and evaluated in a joint project which involved ETH, HyperWerk FHBB, and aprentas3 school of chemistry. As part of the project, an extensive evaluation of AC was conducted by Kristina Bötschi [3][4] [5], who studied how AC compares to the traditional ball-and-stick method of learning organic chemistry. On the basis of the first version of the AC system and drawing on Bötschi’s findings, we have carried out a broad re-design addressing three major issues:
1
Eidgenössische Technische Hochschule Zürich: http://www.eth.ch/ HyperWerk FHBB: http://www.hyperwerk.ch/ 3 aprentas: http://www.aprentas.com/ 2
• extended portability • molecule database (DB) • improved visualization All three issues called for further implementation in the AC system. Extended portability was needed to increase the potential software and hardware solutions. Improved database (DB) functionality was needed to enable an expanded library of molecules for viewing and manipulation. For keyboard-free system configuration and DB access a GUI was implemented into the TUI. Improved visualization is work in progress, aiming to facilitate users’ understanding of complex 3D information and models.
2. Tangible user interface AC is a tangible user interface (TUI) enabling its user to compose and directly interact with three dimensional (3D) molecular models [4]. The system was designed to teach organic chemistry concepts such as tetrahedrons, octet rule, and bonding. Following the conventional implementation of the AR Toolkit [6], physical tools carry one or more fiducial markers, connecting each tool to an animated 3D model so that both tool and model can be seen in a composite image.
Figure 1. Physical tools: booklet and gripper. The system consists of booklet, gripper, cube, platform, camera, and software. The booklet offers elements by printed picture and name (Fig. 1). One hand browses the booklet, which offers one element per page.
By using the gripper, users can pick up elements from the booklet and add them to the molecule being constructed at the platform (Fig. 2).
to an increased number of SW and HW platforms, there is also a wider range of options in terms of cameras (Fig. 4). AC now works with USB and Firewire (IEEE1394).
Figure 2. The AC system in use: Booklet, gripper, cube and platform (left to right). User rotation of the cube affects molecule rotation, hence determining how and where the new element shall connect. A set of specialized functions are realized as cards and activated when drawn onto the platform. The functions are browser, tag-toggle, cleaner, benzenetemplate, and dipole (Fig. 3) [5]. Further use of cards to realize system configuration and DB access was not seen as feasible, as it would overuse hardware real estate. Instead, limited physical space triggered an idea of integrating a graphical user interfaces (GUI) into the TUI.
Figure 4. The AC system running on a standard workstation with a Firewire camera.
3.2 Multilingual support To further make the AC system accessible to more users, internationalization has been prepared for. In the first version of the AC system, language information was exclusively presented in German. The AC system has now been prepared for translation into further languages. This required that molecule names, structure information, and educational text/audio are stored in a DB in such a way that it is enabled for multiple languages.
4. Molecule database
Figure 3. The function cards.
3. Portability The AC system has been extended and now runs on a wider range of hardware and software platforms. New options and an easy-to-install CD-ROM enable installation on multiple software and hardware platforms, including laptops. Also enabled for multilingual support, the AC system is now accessible to a potentially larger user community.
3.1. Operating systems and cameras To reach more potential users and to simplify the use of AC, it should run on different operating systems. While AC was originally developed for Linux and cameras requiring a frame grabber. We have recently ported it for use on Windows and Mac OS X. In addition
By enabling users to construct and examine organic molecules in 3D through the combination of different atoms, improved understanding of organic chemistry is expected. However, interviews revealed that there was a need not only to visualize and interact with user-defined molecules, but with any predefined molecule [3].
4.1. Advantages of an external database Besides creating and interacting with organic molecules, a new system feature is to visualize any existing molecules. While manual assembly of a molecule, atom by atom, is an effective way of learning structural details, it can also be cumbersome for bigger molecules. That is, the pedagogical benefits of the TUI may be lost when user attention is drawn more to interaction than learning. Avoiding such unwanted effects and at the same time aiming for a more versatile TUI, a DB of predefined molecules was viewed as advantageous. While the first version of the AC system used its own proprietary format for storing molecule
definitions and information, we have newly enabled the system for standard molecule file formats.
4.2. Database format A variety of molecule DBs is available, for instance Beilstein4, ISIS5, and PubChem [7]. Used by professionals for many different purposes, these DBs differ in many ways; most importantly in file format. After extensive research into alternative molecule file formats and after interviewing subject-matter experts, we chose an XML based format. This DB was chosen since XML is a standard easily convertible to non-standard formats. An XML enabled DB containing vast amounts of molecule and substance information is the PubChem, which was realized by the National Center for Biotechnology Information [7]. By having access to PubChem, users can easily choose almost any molecule of interest, visualize it in 3D, and interact with it through AC.
a side effect, keyboard-free operation gives more effective tabletop real estate. To assure portability, the GUI was realized using OpenGL. To avoid a one-to-one reclaim of screen real estate, we could not design a GUI that was permanently displayed on the screen. Instead, a permanent corner button activates a pop-up menu, each alternative activating a graphic overlay dialogue box (Fig. 5). Apart from offering system settings, one graphical overlay dialogue is a browser for the molecule DB (Fig. 5). All the molecules that have been loaded into the system, predefined or imported from external DBs, are indexed with their chemical or substance name. This allows users to quickly iterate through the molecules or to select one by clicking its name. When using the browser, molecules are displayed in 2D structure representation (Fig. 6).
5. GUI into TUI, dual mode, 3D rendering In the last part of this paper, we present realizations where particular care is paid to the integration into an existing TUI. First, we present how a graphical user interface (GUI) was seen as beneficial to support system settings and DB access, second we show how dual mode presentation of learning content was realized. Finally, we present 3D display and rendering issues.
5.1. GUI into TUI: design issues User studies showed that a challenge in using the AC system – as compared to a tradition ball-and-stick method of chemistry education – was that controlling system settings often obstructed the learning process [3]. Typical configuration settings more suited to GUI are molecule size, element labeling, and system parameters. Such settings were initially mapped onto keyboard function keys. However, many users turned out to be challenged in using function key while keeping a complex chemical model in mind. Adding support for importing molecules from external DBs also made it necessary to visualize some kind of overview of a large molecule DB. Hence, to simplify interaction with system configuration settings and the new DB functionality, a purely mouse controlled GUI was integrated into to the primary TUI. The GUI is beneficial in offering more usable system settings, replacing function keys from the first system version. As
4 5
MIMAS; http://www.mimas.ac.uk/crossfire/ MDL; http://www.mdl.com/
Figure 5. GUI overview with menu button and main menu (right) and molecule importer (left).
Figure 6. GUI focus showing molecule browser realized as a transparent dialogue.
5.2. Dual mode: text and sound information
6. Conclusion
Evaluations showed that there was a need to improve the method of presenting molecule information [3]. In the first versions of the AC system, when a predefined molecule was loaded or when a constructed molecule was recognized by the AC system, users were given information about the particular molecule or substance by spoken audio text. A problem was the lack of (re-)wind audio information. To enhance control of the information flow, a dual mode system is being developed. A graphical overlay with text containing information about the current molecule is work in progress and shall enable users to choose among text, audio, and both.
Based on the first version of the AC system and combined with outcome of a comparative evaluation of the AC system [3], we have and are still realizing a set of new functionality and features in to the system. Functions already realized are: portability for Windows and Mac OS X, enabling the use of different camera types, compatibility with an external XML database, and the realization of a GUI into the TUI. We are still in the process of translating chemical information into other languages than German, of supporting dual mode for learning text and spoken learning content, and realizing shadows rendering. We expect to finish this work within short6 and to be able to report substantial results in a final version of this paper.
5.3 Improved 3D visualization and rendering Visualization of molecules is iteratively being improved and evaluated. The user study [3] showed that manipulating the structure of the molecules using the TUI was relatively difficult. Many small mistakes, e.g. misplacement and accidental removal of atoms, disrupted the learning process. To reduce the amount of effort needed to learn and to operate the system, we believe that the interface needs to be more natural and feel more like the conventional ball-and-stick method but with the beneficial aspects of a TUI. To enhance the appearance of the computer-generated molecules, the rendering of shadows is added to the graphics engine. The use of shadows and information projection on different planes (Fig. 7), allows to order and structure complex information. Shadows play a big part in visualizing the appearance of 3D models by adding greater depth perception [8]. The added depth perception helps users to manipulate the molecules with greater precision, thus enhancing the TUI.
References [1] Voegtli, B., “Augmented Collaboration”, Diploma thesis of Benedikt Voegtli, HyperWerk FHBB, 2002. [2] Fjeld, M. and Voegtli, B.,“Augmented Chemistry: An Interactive Educational Workbench”. Video program and proc. of International Symposium on Mixed and Augmented Reality (ISMAR), 2002, pp. 259-260. [3] K. Bötschi (in press): Evaluation von Augmented Chemistry als computergestütztes Lernsystem für den Chemieunterricht. Lizentiatsarbeit am Psychologischen Institut der Universität Zürich, Fachrichtung Psychologische Methodenlehre. [4] Fjeld, M.; Hobi, D.; Winterthaler, L.; Voegtli, B.; Juchli, P., “Teaching electronegativity and dipole moment in a TUI”, Advanced Learning Technologies, IEEE, 2004, pp. 792-794. [5] Voegtli B., Fjeld M., and Juchli P.: Tangible User Interfaces, Learning, and Design. In G. Boorman (ed.): Total Interaction. Basel/Berlin/Boston: Birkhäuser Publishers, 2004. [6] Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K. and Tachibana, K., “Virtual Object Manipulation on a Table-Top AR Environment”. Proc. Int. Symposium on Augmented Reality (ISAR), 2000, pp. 111-119. [7] National Library of Medicine, “The PubChem Project”, http://pubchem.ncbi.nlm.nih.gov/. [8] Youngung Shon; McMains, S., “Evaluation of drawing on 3D surfaces with haptics”, Computer Graphics and Applications, IEEE, 2004, pp. 40-50.
Figure 7. The concept of rendering molecules with shadows and projecting information on different planes. 6
This work should be finished by May 2005, when the project this work is part of will be terminated.
