Augmented Reality and Rigid Body Simulation for Edutainment The Interesting Mechanism - an AR puzzle to teach Newton physics Philip Buchanan, Hartmut Seichter, Mark Billinghurst, Raphaël Grasset HIT Lab NZ, University of Canterbury Private Bag 4800 Christchurch, New Zealand philip.buchanan,hartmut.seichter,mark.billinghurst,
[email protected]
ABSTRACT Physics simulation is becoming more common in computing. We have developed a comprehensive toolkit to connect the physical and virtual world within Augmented Reality (AR) using rigid body simulation. Unlike existing techniques of embedding physics simulations into 3D environments, the use of rigid body simulations within AR requires a different approach. To demonstrate our approach we developed an edutainment game based on the concept of chain reactions and physical contraptions. In this paper we elaborate on the constraints introduced by mixing AR and rigid body simulation, and how it subsequently effects the visual richness and perceptual appearance of an AR simulation. We describe our implementation approach and provide an analysis of additional scenarios which would be enriched by physical simulation.
Categories and Subject Descriptors J.5 [Computer Applications]: Arts and Humanities; D.0 [Software Engineering]: General
General Terms Augmented Reality, Edutainment, Physics Simulation
Keywords Augmented Reality, Rube Goldberg Machine, Tangible User Interface, Edutainment
1.
INTRODUCTION
Physics teaching, for a lot of people, should bring back memories of a Rube Goldberg Machine 1 stretching across the whole classroom. It is a quintessential tool to teach the nature of Newton physics as it connects various representations of forces, friction and their transformations and conversions. 1
Rube Goldberg Website: http://www.rubegoldberg.com/
! " # $#% &' ()(* ++ (,&,( ) -+
17
Figure 1: Default view of a part of the contraption with guiding lines In this paper we set out to extend the notion of the Rube Goldberg machine to add a new dimension: augmenting the process with new meaning and simulating environments where gravity or other physical principles do not follow known rules. Supporting grasp across media through directly manipulated tangible user interfaces is paramount for understanding complex processes [11, 4]. Thus, we hope to add a new dimension to teach physics in a playful way. Recently, hardware manufacturers have adding physics simulation capabilities to common computing components to enrich applications and games with realistic physics simulation. Conceptually, the physical simulations enrich visualizations by adding properties to virtual objects that are visually perceivable. The term physics simulation encloses a variety of modes of visually representing processes in the real world. This paper and the project it describes involves only rigid body simulation, also know as rigid body dynamics, which is one of the least complex forms of physics simulation. The impact of rigid body simulation on visual perception in simulated environments was discussed early on in [2]. However, the complexity of supporting collision detection
can be mixed and don’t interfere with optimization strategies in the scene-graph. The Interesting Mechanism is based upon the physical interaction possibilities provided by a Rube Goldberg Machine. To leverage the possibilities of Augmented Reality, our simulation is driven by the placement of physical AR markers, each of which represent virtual physical objects. To facilitate easy setup and allow for future use in other situations, we used XML files to set up the simulation. The XML files outlined a correspondence between: visible virtual objects, which were 3D models designed in a modeling program; the virtual physical objects, which were used by ODE in the simulation; and the AR markers, which were used by the camera tracking.
Figure 2: A contraption with various objects used to solve the puzzle. On the lower edge a HUD shows the markers to be used on convex and concave meshes made it unfeasible for realtime simulation until recently. Today’s computing power has reached the level where realistic newtonian physics is within the capabilities of mainstream computer hardware. To test our physics integration implementation we choose to implement a virtual contraption building challenge based on the Rube Goldberg machine. The basic idea is to transport an object from one point to the other using a variety of objects to bridge gaps, introduce impulses or other means of convoluted forms of acceleration. This computer game concept was introduced by Kevin Ryan and Jeff Tunnell in the early nineties with a series of games called The Incredible Machine, and our name choice tries to honor this. The early work on ARToolKit [3] hinted at physical simulation as a logical way to enrich AR applications. Later Geiger et al. [6] demonstrated a similar approach to ours but without the use of a scene graph. DART [9] showed the integration of physics in conjunction with a graphical authoring tool. Imura et al. [10] demonstrated an AR application using Fluid Dynamics simulation.
2.
IMPLEMENTATION
This section outlines the implementation details of the physics in conjunction with AR and the actual game play.
2.1
Physics in AR
The core of the application consists of osgODE, which is a nodekit developed by the HITLabNZ around the Open Dynamics Environment (ODE 2 ) for the OpenSceneGraph (OSG 3 ). Our nodekit supports, unlike other attempts to implement physics in OpenSceneGraph, a component based approach without inheriting the actual nodes. Events from the simulation environment are introduced early in the traversal process so that Augmented Reality and simulation events 2 3
Open Dynamics Environment http://www.ode.org OpenSceneGraph http://www.openscenegraph.org
These XML files also allowed complex object setups, with nested objects corresponding to parent and child nodes within the scene graph. This not only allowed for many ODE objects to be attached to one AR marker, but also helped facilitate optimization by the scene graph. In current AR implementations, scene graph node position is driven by the AR marker location [8]. However, scene graph node position is also driven by the ODE simulation, and our system was required to reconcile this conflict. Our XML files allowed for the owner of a scene graph node to be specified, allowing both AR markers and ODE to drive the node position in different circumstances. When AR marker locations are used to set the position of virtual objects within the physics simulation, tracking errors and motion jitter introduce unrealistic forces. The velocity of objects attached to markers can change rapidly, producing huge accelerations which cause problems for ODE. Heavy filtering of AR tracking data introduces a lag between the real and virtual objects. A tradeoff between these two situations was required. Furthermore the update rate for transformations from the tangible user interface can be estimated at approximately 15-30fps on conventional systems. Transformations in AR through tangible user interfaces are unpredictable, but rigid body simulations like ODE rely on predictable behaviour and use a time based approach to make transformations mathematical stable for any point in time. These differences in the approaches introduce a race condition between AR and ODE. In an earlier application [7] we used a event mechanism to compensate for this. In The Interesting Mechanism however we designed the user interaction around it so that this problem is avoided for now. Many of the scenarios in our game required objects to sit on a flat surface. It was found that a small paper marker was not sufficient to track the orientation of the surface exactly, and mis-alignments between virtual objects and the perceived ground plane would occur. Our solution was to use the data from all visible markers to create a best-fit ground plane. Furthermore the use of various special effects, which usually enrich 3D game applications, need specific treatment to take occlusion from elements of the TUI into account.
2.2
18
Game Play
The game play revolves around the idea of the Rube Goldberg machine to create increasingly convoluted and complex machines. To best explore the possibilities provided by this mix of Physical Simulation and Augmented Reality, we developed a game based upon an existing and proven mechanic. The Interesting Mechanism is a type of transport game, where the player must move an object from a start location to the finish. The movement of this object can only be influenced indirectly, by placing further objects in its path to divert, redirect, impede or assist progress. In our game, the player manipulates AR markers which correspond to objects in the simulation. Not all of the objects in the simulation can be moved, and the challenge lies in creating a device using only the provided objects. Each AR marker corresponds to one virtual object, and different markers are used if different objects are required for each scenario. This correspondence allows for easier visual recognition of objects when moving the AR markers. A number of different movement options are provided by ODE, and while all were implemented in our system, not all are present in the final set of challenges. Hinges, motors, conveyor belts and different surface textures allow for a wide range of gameplay options, and increase the length and complexity of the machines that can be built. While our implementation encourages players to solve challenges, enjoyment and exploration of this system is possible through free-form play. No penalty is incurred for building creative machines based around the objects provided in each situation. This also allows for many different solutions to each challenge. Additionally, various parameters of the simulation environment can be changed to let the user experience the same puzzle in a martian or lunar conditions with applying gravity and atmospheric settings.
3.
FUTURE DIRECTIONS
An important improvement we are working on is the introduction of shadows to guide the users perception in the AR view. The objects are purposefully simple and only lightly shaded, making their functions immediately obvious, but impeding depth perception for the player. This is a crucial aspect of the game play, and at times proved difficult for the player to see. However, the use of shadows in AR is not as straight forward as in a fully simulated game environment, as it is required to consider properties of the real world such as the actual direction of light sources in the scene. To implement shadows, the introduction of a illumination extraction algorithm is needed [1, 5]. At the moment, the introduction of forces into the simulation is only possible for small movements due to the constraints of the ODE implementation. In future we want to enhance our prediction mechanism to compensate for penetration errors introduced through the uncertain transformation in AR. One solution we pursuing is the dynamic adjustment for the
19
Figure 3: Building a catapult to kick off the small box on top of the big box. The interaction with the tangible user interface controls the trajectory. The guiding line show the intended movement penetration testing in ODE. The application can log previous motions to dynamically adjust the simulation parameters. Additionally a prediction mechanism for the trajectory is needed. Furthermore we have plans to extend the game play beyond a single desktop machine into a collaborative game where objects can wander from one The Interesting Mechanism machine to the next one. As part of our efforts in AR authoring, the rigid body simulation will also be integrated with an in-house content development tool called ComposAR 4 to allow users to develop their own TIM contraptions. Another challenge we are facing is the use of a detachable webcam. It introduces a mirroring problem which the user needs to get used to.
4.
CONCLUSION
We presented a novel way of using AR and a rigid body simulation for a simple and effective tool for edutainment. Players can learn about a variety of forces in a game play fashion. By simulating the dynamic parts of rigid body dynamics we are able to visualize various environments which potentially can teach about abstraction of the forces. We hope to use our toolkit for user studies to evaluate the perceptual issues of rigid body physics in AR.
5.
ACKNOWLEDGMENTS
We want to thank Andre Schollmeyer who implemented the initial version of osgODE as part of an internship at the HIT Lab NZ. This research project is funded by the New Zealand Government FRST grant UOCX0608.
6. 4
REFERENCES
http://www.hitlabnz.org/wiki/ComposAR
2006. [11] B. Shneiderman. The future of interactive systems and the emergence of direct manipulation. College Park, Md.: University of Maryland, 1982.
Figure 4: Challenge completed! landed properly on the big box.
The small box
[1] K. Agusanto, L. Li, Z. Chuangui, and N. W. Sing. Photorealistic rendering for augmented reality using environment illumination. Mixed and Augmented Reality, 2003. Proceedings. The Second IEEE and ACM International Symposium on, pages 208–216, Oct. 2003. [2] D. Baraff. Analytical methods for dynamic simulation of non-penetrating rigid bodies. Computer Graphics, 23(3):223–232, 1989. [3] M. Billinghurst and H. Kato. Real World Teleconferencing. In ACM CHI’99, pages 194–195, Pittsburgh, PA, 1999. [4] W. Bruns. Grasping, communicating, understanding: connecting reality and virtuality. AI Soc., 10(1):6–14, 1996. [5] G. Drettakis, L. Robert, and S. Bougnoux. Interactive common illumination for computer augmented reality. In J. Dorsey and P. Slusallek, editors, Rendering Techniques ’97 (Proceedings of the Eighth Eurographics Workshop on Rendering), pages 45–56, New York, NY, 1997. Springer Wien. [6] C. Geiger, T. Schmidt, and J. Stocklein. Rapid development of expressive ar applications. Mixed and Augmented Reality, 2004. ISMAR 2004. Third IEEE and ACM International Symposium on, pages 292–293, Nov. 2004. [7] S. Gilroy, M. Cavazza, R. Chaignon, S. M¨ akel¨ a, M. Niiranen, E. Andr´e, T. Vogt, M. Billinghurst, H. Seichter, and M. Benayoun. An emotionally responsive ar art installation. In ISMAR 2007, 2007. [8] J. Looser, R. Grasset, H. Seichter, and M. Billinghurst. Osgart - a pragmatic approach to mr. In ISMAR 2006, 2006. [9] B. MacIntyre, M. Gandy, S. Dow, and J. D. Bolter. Dart: A toolkit for rapid design exploration of augmented reality experiences. In User Interface Software and Technology (UIST’04), 2004. [10] Y. M. Masataka Imura, Yoshihiro Yasumuro. Fountain designer: Control virtual water as you like. In ISMAR
20
