Mobile Augmented Reality Interaction Techniques for ... - CiteSeerX

Proc. IEEE and ACM ISMAR 2006, Santa Barbara, CA, Oct 22-25, 2006, 235-236

Mobile Augmented Reality Interaction Techniques for Authoring Situated Media On-Site Sinem Güven

Steven Feiner

Ohan Oda

Department of Computer Science, Columbia University {sinem,feiner,oo2116}@cs.columbia.edu

(a)

(b)

(c)

Figure 1. (a) User takes a snapshot of a set of shelves and (b) brings the tablet down for comfortable authoring. (c) Freeze-Frame. Using the frozen frame, the user annotates the shelves.

ABSTRACT We present a set of mobile augmented reality interaction techniques for authoring situated media: multimedia and hypermedia that are embedded within the physical environment. Our techniques are designed for use with a tracked hand-held tablet display with an attached camera, and rely on “freezing” the frame for later editing. 1. INTRODUCTION We use the term situated media [5] to refer to multimedia and hypermedia that are embedded in the user’s surroundings. In previous work, we described an approach for authoring situated media using our desktop MARS Authoring Tool [4]; here, we introduce our mobile authoring techniques. 2. MOBILE AUTHORING TECHNIQUES The mobile version of the MARS Authoring Tool, shown in Figures 1 through 3, runs on a Tablet PC, equipped with a camera and a microphone, and is position- and orientation-tracked. It provides an AR authoring platform by mixing registered 3D graphics and a live camera feed. Interaction with the system is performed using a stylus and a rotating input device. We support three different techniques for authoring 3D situated media: freezeframe, freeze-n-link, and free-n-move. 2.1 Authoring Situated Multimedia: Freeze-Frame Freezing refers to capturing and freezing a frame in an AR or VR system by pausing tracking and camera input. Researchers have explored freezing in several applications. Amire [1] uses freezing for calibration to make up for the jitter experienced with AR Toolkit. AR PDA [3] uses freezing to provide augmented views of an object, and to compensate for large system latencies. In contrast, our system uses freezing to support authoring. Schmalstieg et al. [7] propose transparent pen and pad props for table-top VR. They provide a snapshot tool, in which different snapshots (frozen frames) of the VR scene are captured to help users browse the scene from different view points. Instead, we use freezing to author situated media through mobile AR. Boom Chameleon [8] is another system that uses snapshots of the 3D environment for VR authoring. Although most of the Boom Chameleon techniques could also be applied to an AR setting,

they do not provide the manipulation and linking capabilities we need to author situated hypermedia. Our Freeze-Frame technique allows the user to capture a snapshot (frame) of the environment to work on it, and later maps the results back to the physical world. For example, suppose that the user is interested in organizing their lab. There are lots of books, manuals, games, and software scattered around the room, and the user wants to provide virtual annotations for these real objects to allow other users to locate them more easily. The user starts by annotating objects in their original locations. Each time, the user aligns the tablet to view the physical objects of interest (Figure 1a), and freezes the frame, pausing camera input and tracking. The user can then bring the tablet to a comfortable position to start authoring (Figure 1b). The user freezes a frame that depicts a set of shelves, and starts annotating some of the items on them. For example, the user creates the label “Maya Books” and situates it next to several manuals. Upon resuming tracking and camera input, the labels are interpolated to their world positions. Figure 1(c) shows the tablet after the user has frozen and annotated the current frame. Positioning virtual objects in this manner allows the user to specify the (x,y) coordinates of the objects in the viewplane; however, further manipulation is needed if the objects are also to be moved along the z-axis. To achieve this, the user can rely on two different approaches. If a registered 3D model of the environment is available, the newly created virtual object is assigned the z-coordinate at which the ray from the camera through the object intersects the model. The virtual object is then interpolated to the target z-coordinate and the user can further manipulate its location for more precision. If a 3D world model is not available or there is no object at the desired z-coordinate, the user can use an additional rotating input device after picking the object to “reel” [2] the object in and out along the ray. Next, the user sees a bunch of games scattered on a desk. Instead of labeling the games one by one, the user creates an audio snippet that contains all their names. The user then uses Freeze-Frame to associate the audio snippet with the desk on which the games reside. 2.2 Editing and Organizing Situated Media: Freeze-n-Move Users are sometimes interested less in precision, than in being able to easily move things around the world (in essence,

(a)

(b)

(c)

(d) Figure 3. Freeze-n-Link. User can select previously captured frames to link their associated media.

(e)

(f)

Figure 2. Freeze-n-Move. (a–b) Two snippets falling in the user’s frustum are selected. (c–d) User deselects the image snippet and moves only the audio snippet. (e–f) Finally, the audio snippet is released to its new location.

reorganizing the “world desktop”). The Freeze-n-Move technique makes it possible to move a group of objects to another location while maintaining their organizational integrity, using an approach inspired by Fix and Float [6]. In our application scenario, having annotated all objects of interest, the user now determines the new locations for the objects. First, the games on the desk will be moved to a drawer under the TV. Before the user physically moves the games to the drawer, they capture an image of the games and associate it with the desk to mark the original location of the games (Figure 2a). Next, the user physically moves the games to the drawer, and wants to select and move their associated audio snippet, which was created earlier. The user then switches to the Freeze-n-Move mode, in which the system determines which objects currently intersect the view frustum and automatically selects them. (If desired, hither and yon clipping planes can be enabled and adjusted to further narrow the selection.) Selecting the snippets causes them to highlight and appear stuck to the screen upon resuming tracking and camera input (Figure 2b). The user then deselects the image snippet (Figure 2c) and moves the audio snippet next to the drawer that now contains the games (Figure 2d–e). Finally, the user releases the audio snippet to its new location (Figure 2f), as in Freeze-Frame. 2.3 Authoring Situated Hypermedia: Freeze-n-Link Freeze-Frame can be used to link two or more situated multimedia objects that can be captured within the same frame. If this is not possible, Freeze-n-Link can be used to create links between objects or locations in different frames. Suppose the user now wants to create a link between the original and the current location of the games, so that other users can locate the games easily. First, the user captures a frame of the desk associated with the image of the relocated games. Next, they capture a frame of the drawer along with the audio snippet. Each

time the user captures a frame, its associated 3D content is stored in our system database. Now that the user has both frames, they activate the linking functionality to draw a link connecting the image snippet to the audio snippet (Figure 3). After the user deactivates linking, they can see the new 3D link, guiding the user from the original location of the games to their new location. If the user wishes to link frames other than the last two, the browsing interface can be used to select them. 3. CONCLUSIONS & FUTURE WORK We have described a set of techniques that make it possible to annotate the real world without requiring a detailed 3D representation of it. We are currently working on extending our system to support authoring outdoors and to provide more precise 3D object manipulation. This research was funded in part by DAFAFRL Contract FA8650-05-2-6647, Office of Naval Research Contract N0001404-1-0005, NSF Grants ANI-00-99184, IIS-00-82961 and IIS-0121239, and gifts from Microsoft and Autodesk. REFERENCES 1. D. F. Abawi, R. Dörner, M. Haller, and J. Zauner, "Efficient Mixed Reality Application Development," Proc. CVMP '04 (European Conf. on Visual Media Production), London, UK, March 15–16, 2004, 289–294. 2. D. A. Bowman and L. F. Hodges, "An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in Immersive Virtual Environments," In Proc. I3D '97 (Symposium on Interactive 3D Graphics), Providence, RI, April 27–30, 1997, 35–38. 3. Geiger, C., Kleinnjohann, B., Reimann, C., and Stichling, D. Mobile AR4ALL. Proc. IEEE and ACM ISAR ’01 (Int. Symp. on Augmented Reality), New York, NY, 2001, 181–182. 4. S. Güven and S. Feiner, "Authoring 3D Hypermedia for Wearable Augmented and Virtual Reality," Proc. IEEE ISWC '03 (Int. Symp. on Wearable Computers), White Plains, NY, October 21–23, 2003, 118–126. 5. S. Güven and S. Feiner, "Interaction Techniques for Exploring Historic Sites Through Situated Media," Proc. 3DUI '06 (Symposium on 3D User Interfaces), Alexandria, VA, March 25–26, 2006, 1–10. 6. G. Robertson and S. Card, “Fix and float: objects movement by egoentric navigation,” Proc. UIST ’97 (Symp. on User Interface Software and Technology), Alberta, Canada, October 14–17, 1997, 149–150. 7. D. Schmalstieg, L. Encarnação, and Z. Szalavári, "Using Transparent Props for Interaction With the Virtual Table," Proc. I3D '99 (Symp. on Interactive 3D Graphics), Atlanta, GA, April 26–29, 1999, 147–153. 8. M. Tsang, G. Fitzmaurice, G. Kurtenbach, A. Khan, and B. Buxton, "Boom Chameleon," Proc. ACM UIST '02 (Symp. on User Interface Software and Technology), Paris, France, October 27–30, 2002, 111–120.