Multitouch Interaction for Tangible User Interfaces Hartmut Seichter∗
¨ Grasset† Raphael
Julian Looser‡
Mark Billinghurst§
Human Interface Technology Laboratory New Zealand
A BSTRACT We introduce a novel touch-based interaction technique for Tangible User Interfaces (TUIs) in Augmented Reality (AR) applications. The technique allows for direct access and manipulation of virtual content on a registered tracking target, is robust and lightweight, and can be applied in numerous tracking and interaction scenarios. Index Terms: H.5.1 [Information Systems]: Information Interfaces and Presentation—Artificial, augmented, and virtual realities; I.4.6 [Computing Methodologies]: Image Processing and Computer Vision—Edge and feature detection 1
I NTRODUCTION AND R ELATED W ORK
Real world interaction is typically ambidextrous, fosters the use of many fingers, and is supported by the haptic sensation of touch, allowing people to grasp, explore and understand objects. TUIs aim to couple such interaction with digital UIs. For AR interfaces the link between physical and virtual objects is crucial. However, continuous interaction with virtual objects has typically relied on general purpose input devices (which are often unavailable for mobile users) or on the physical relationships between tracking targets which limits access to information in the augmented environment. Multi-touch enabled devices like the iPhone utilize multiple finger inputs to reduce modes in the user interface. However, even without multi-touch, auxiliary input devices can divert attention and disrupt interaction in an AR interface. Our new interaction method enhances standard AR target interaction with computer vision based multi-touch input in the vicinity of the tracked target. TUI research is highly relevant to AR. DataTiles [9] integrated multiple facets of how TUIs can be utilized to access and manipulate digital content and can thus be seen as an implementation of paradigms introduced in Tangible Bits [5]. Merging both TUIs and finger tracking has also been investigated. An early AR approach was FingerTracker [2], which used a set of radial markers. However, these markers also make the system less portable and require calibration. A marker-less finger tracking system was introduced in [1], but the system relies on a rigid configuration of the camera towards the tracking area. The work of [8] describes a finger tracking system on top of fiducial markers, giving both a pose in relation to a non-rigid camera as well as the additional input. In many cases a tracked target is occluded by the virtual content registered to it, thus using the target interior for tracking fingers constrains the overall UI design. An extensive account of interactions with hand gestures and augmented content can be found in [11]. A more general approach of using touch and gestures in an augmented setting is been introduced in [4]. Another approach is to use a constraint for an advantage as in [6]. The occlusion based method supports various degrees of freedom ∗ e-mail:
[email protected] [email protected] ‡ e-mail:
[email protected] § e-mail:
[email protected] † e-mail:
IEEE International Symposium on Mixed and Augmented Reality 2009 Science and Technology Proceedings 19 -22 October, Orlando, Florida, USA 978-1-4244-5419-8/09/$25.00 ©2009 IEEE
Figure 1: Accessing various stages of a Lego model.
but compromises on fidelity and precision, requiring a large number of targets for complex interfaces. 2
I MPLEMENTATION
Our aim was to provide multiple finger touch input in the vicinity of augmented tiles (similar to those of DataTiles [9]), thus supporting spatial interactions with 3D content on the desktop and mobile devices. Although finger tracking within the target interior is possible [8], it is computationally expensive and better suited to direct interaction with a registered 3D object rather than interaction with TUI elements. Instead, our approach allows for various degrees of freedom based on defined touch tracking areas, in which touches are independently detected and simultaneously tracked. Our AR touch input algorithm is optimized for speed with relatively high robustness. It was implemented with the cross-platform SSTT1 (Simplified Spatial Target Tracker) library. On a standard PC, tracking touches along a target’s four borders adds less than 1ms per target (12ms on a Samsung Omnia). The following steps outline the tracking implementation. The first step of the algorithm uses the perspective transformation computed for the target pose to create a perspectively unwarped image. This intermediate image is then sub-sampled and thresholded. Multiple line samplings extract contrast changes in the regions of interest, that is, the surrounding border of the tracking target. We tested various configurations and settled for an implementation using three scales for the line sampling area. This allows us to track an approaching finger and allows for better robustness. The sampling uses an accumulation buffer in order to exclude small 1 http://www.technotecture.com
213
contrast changes in the image. Positive responses are filled into an n-dimensional vector (n being the number of detected touch points) of normalized touch point positions on the respective axis. Each touch point is registered with the normalized touch size. This information is then filtered to provide a coarse rejection for jitter and outliers. This implementation allows us to use printed material for the touch area as long as the hue component is below the dynamic threshold calculated for region of interest. The algorithm can be used also for either dark on light contrasts or vice versa. A histogram comparison of the whole image against the region can detect adverse lighting conditions, potentially triggering some in-system user guidance. The algorithm is limited by the length of the touch area retrieved from the video image. Naturally, the precision of the touch area deteriorates with distance from the camera and slant angle of the target. However, the interface can potentially identify and manage these situations by providing user guidance. 3
I NTERACTION
The multiple touch input streams can be mapped in various ways to output parameters. Such mappings are similar to using physical sliders, although due to augmentation, accessibility can be improved with spatially registered on-screen guidance (such as AR ToolTips [10]) which can guide a dynamic interaction mode [3] or provide hints about the physical interactions the user must perform. Furthermore, because the touch tracking occurs in close proximity to the tracked target, we can utilize this spatial relationship for designing novel interactions. The standard interaction mode involves a fixed relationship between the touch areas and the tracked target. This allows, for example, touch controlled sliders along the edges of the target to manipulate parameters of the registered 3D content. Another interaction possibility is to map the orientation of the target to the interface mode, so that the same touch input can be used to adjust multiple parameters. Visual guidance would be necessary in this case to inform the user of the current mapping. The input from touch tracking can be processed in different ways. A direct (or absolute) mapping is useful for direct access to specific stages in a process. For example, a touch halfway along the target’s edge could jump to the middle of an animated 3D sequence. Direct mappings, however, can be cumbersome when the target value range is non-linear, or high precision is required. In such cases, a relative mapping is more sensible. The touch sliders can provide continuous values, or the values can be discretized into a set of output bins, allowing in the simplest case, buttons. Touch enabled TUIs are useful for directly manipulating the corresponding virtual entities. The spatial proximity affords a direct mapping between content and interaction device. Hence, the correct spatial and dynamic mapping between the touch input and the content is essential for successful touch based interaction with the 3D content. Beyond the spatial relationship, the touch input we introduce can be used in the same manner as traditional sliders in electrical appliances and computational devices. Thus, direct, iterative and parameterized input are possible, allowing high fidelity interaction with registered content. Another aspect of our spatial touch approach is the physical design and resulting haptic and visual properties. Unlike other TUIs based on markers, a target surrounded by touch areas may need to be smaller than standard targets in order for the user to operate multiple touch areas with one hand (e.g. a pinch motion along neighbouring edges). Also, the common “magic-mirror” metaphor for desktop AR means a mirrored touch layout and tracking areas may be required. Haptic parameters such as groves can considerably enhance the usability of a touch interface [9], as evidenced by notebook trackpad designs. Such approaches allow an explorative
214
and also purely tactile usage. Targets are trimmed to fit the touch layout to allow the finger to slide along the edge. A small grove near the target border prevents the fingers from sliding over the interior of the tracking target. 4 C ONCLUSION AND F UTURE W ORK We have introduced a pragmatic method for tracking multiple neartarget touches. The spatial proximity lends itself to a number of interaction mappings. Using interactive touch targets enable a number of applications that require explorative and interactive access to spatial data. We see our touch tracking as a starting point from which a number of enhancements can be explored. An important next step is to evaluate the initial designs and mappings. Furthermore, work is required to fine-tune the tracking. One avenue is to combine touch tracking with natural feature tracking capable of reliable tracking with a limited number of features. In terms of interaction, we are certain that a combination of touch with a Magic Lens [7] interface can create a comprehensive explorative tool. ACKNOWLEDGEMENTS Funding for this project was provided by the FRST grant Accessible AR (UOCX-0704). We also wish to thank LDRAW enthusiast Jim DeVona for the scout model. R EFERENCES [1] J. Crowley, F. Berard, and J. Coutaz. Finger tracking as an input device for augmented reality. pages 195–200, 1995. [2] K. Dorfm¨uller-Ulhaas and D. Schmalstieg. Finger tracking for interaction in augmented environments. In ISAR, pages 55–. IEEE Computer Society, 2001. [3] G. W. Fitzmaurice and W. Buxton. An empirical evaluation of graspable user interfaces: towards specialized, space-multiplexed input. In CHI ’97: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 43–50, New York, NY, USA, 1997. ACM. [4] D. Holman, R. Vertegaal, M. Altosaar, N. Troje, and D. Johns. Paper windows: interaction techniques for digital paper. In CHI ’05: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 591–599, New York, NY, USA, 2005. ACM. [5] H. Ishii and B. Ullmer. Tangible bits: Towards seamless interfaces between people, bits and atoms. In CHI, pages 234–241, 1997. [6] G. A. Lee, C. Nelles, M. Billinghurst, and G. J. Kim. Immersive authoring of tangible augmented reality applications. In ISMAR ’04: Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality, pages 172–181, Washington, DC, USA, 2004. IEEE Computer Society. [7] J. Looser, R. Grasset, and M. Billinghurst. A 3d flexible and tangible magic lens in augmented reality. In ISMAR ’07: Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, pages 1–4, Washington, DC, USA, 2007. IEEE Computer Society. [8] S. Malik, C. McDonald, and G. Roth. Hand tracking for interactive pattern-based augmented reality. In ISMAR, pages 117–126, 2002. [9] J. Rekimoto, B. Ullmer, and H. Oba. Datatiles: a modular platform for mixed physical and graphical interactions. In CHI ’01: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 269–276, New York, NY, USA, 2001. ACM. [10] S. White, L. Lister, and S. Feiner. Visual hints for tangible gestures in augmented reality. In ISMAR ’07: Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, pages 1–4, Washington, DC, USA, 2007. IEEE Computer Society. [11] M. Wu and R. Balakrishnan. Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In UIST ’03: Proceedings of the 16th annual ACM symposium on User interface software and technology, pages 193–202, New York, NY, USA, 2003. ACM.
