Photoshop) to support rapid prototyping of AR experiences by experienced designers, who will be able to build and test parts of their experience as they see fit,.
Prototyping Applications with DART, The Designer’s Augmented Reality Toolkit Blair MacIntyre and Maribeth Gandy GVU Center, Georgia Institute of Technology, Atlanta, GA (blair, maribeth)@cc.gatech.edu Abstract In this position statement, we describe our approach to prototyping AR applications embodied in the Designer’s Augmented Reality Toolkit (DART), a system we are creating to allow non-technologists to easily create augmented reality (AR) experiences. Over the past year our research has been focused on the creation of this toolkit that can be used by technologists, designers, and students alike to rapidly prototype AR applications. Current approaches to AR development involve extensive programming and content creation as well as knowledge of technical topics involving cameras, trackers, and 3D geometry. The result is that it is very difficult even for technologists to create AR experiences. Our goal was to eliminate these obstacles that prevent such users from being able to experiment with AR. The DART system is based on the Macromedia Director multimedia-programming environment, the defacto standard for multimedia content creation. DART uses the familiar Director paradigms of a score, sprites and behaviors to allow a user to visually create complex AR applications. DART also provides low-level support for the management of trackers, sensors, and camera via a Director plug-in Xtra.
1. Introduction Over the past few decades, AR researchers have explored a wide variety of domains; most recently, we have been focusing on the use of dramatic AR experiences in education and entertainment settings. However, the primitive state of AR technology, and the lack of tools to support traditional design activities (such as rapid prototyping and incremental experience testing) has made this shift in focus from task-focused to experiential domains difficult. When working in task-focused domains, such as equipment maintenance and repair, it is feasible for a team of technology experts to work closely with domain experts to understand a particular problem, and build a solution to the problem using widely accepted HCI techniques such as iterative design. Unfortunately, in experiential domains, this separation between designers1
and technologists does not work; designers are most effective when working directly with a medium, and working through an intermediary seriously hinders (or even destroys) the creative process. Most instances of successful design using cutting-edge technologies have been slow, painstaking endeavors carried out by close-knit teams or technically sophisticated individual designers. By making AR more accessible to a wider range of designers, we can begin to achieve its potential to create powerful dramatic and educational experiences.
2. Motivation During our work through the past years, the immaturity of AR technology (both hardware and software) has limited the exploration of ourselves, our collaborators and our students. We have found that even when working technology (accurate trackers, calibrated head-worn displays etc.) is available in a powerful prototyping environment such as Director, creating AR experiences is still prohibitively difficult, even for experienced AR developers and Director programmers. The extensive time required to create content and to write programs to handle all the components of an AR experience often means that there is a large amount of latency between the inception of an idea and its realization. The difficulty in creating these applications results in less experimentation with different approaches to an experience. Often the end result is not what was intended, and the resources are not available for further iteration. For non-technologists it is almost impossible to create AR experiences. For example, we have found in our “AR Design” class that students (especially the non-computer science students) resort to tediously creating video mockups of possible AR experiences instead of implementing prototypes. We believe this is due both to their lack of expertise in areas such as tracking, and to the complicated software development that would be required.
3. Overview The primary purpose of our research is to advance the state of the art in AR by enabling designers to work directly
1
For simplicity, we use the term designer to refer to those people primarily interested in creating experiences or artifacts of some kind; this could include graphics designers, artists, architects, game designers, media theorists, museum exhibit designers and so on. Similarly, we use the term
technologist to refer to those people primarily interested i n advancing the state of the art of the enabling hardware and software technology; this could include computer scientists, HCI researchers, engineers, and so on.
with AR as a new medium for dramatic experiences. The three main goals of our research are: 1 ) identifying and supporting appropriate design activities for AR experiences, 2) creating open, extensible design tools that support and encourage these activities, and 3) solving fundamental technical research problems to enable the creation of these tools. These three goals are intimately related, and cannot effectively be pursued independently. Design tools must support and encourage an appropriate set of activities if they are going to be useful to designers, and must be open and extensible to allow them to be appropriated and used in ways unanticipated by their creators. Without understanding the evolving requirements of the design tools, we cannot know what technical problems are important to solve. Conversely, as we solve technical problems, we enhance the capabilities of the design tools, which in turn opens up new possibilities for design explorations. Our system, DART ( the Designers Augmented Reality Toolkit), is the tangible manifestation of our research activities. DART is being built as a collection of extensions to the Macromedia Director multimediaprogramming environment, the de-facto standard for multimedia content creation. Our aim is to enable designers to rapidly develop and test their AR experiences, in the same environment that will be used to deploy the final experience. This last point is critical; while we are focused on supporting early design activities, designers can gradually evolve their prototypes as they see fit. Polished content can be mixed with crude content, elaborate narratives and complex behaviors can be tested as desired, and changes to “complete” experiences can be rapidly prototyped. A guiding principle of our work is that we should build on the existing design practices and tools used by experienced designers, rather than expect these designers to adopt radically new practices and tools. Our focus on these particular principles comes from our own experiences, and from practical and theoretical lessons from the HCI, AR, Media Studies and Experience Design communities. We are adapting existing design practices (e.g., storyboarding, informal prototyping, animatics2) and tools (e.g., from physical models and pencil sketches, to computer programs such as Director, 3D Studio and Photoshop) to support rapid prototyping of AR experiences by experienced designers, who will be able to build and test parts of their experience as they see fit, without having to “finish” the experience first. The first major focus of our work on DART is on the use of informal content as the basis for supporting rapid design exploration and reflection [3][10]. Our goal is to 2
An animatic is a sequence of still frames with a narration, used to tell the story of a film.
integrate the use of storyboards and other design documents into early AR experience design. We are developing computational abstractions that combine informal content with an underlying engine supporting simple narrative structures and content behaviors, so that designers can rapidly explore non-trivial experiences in the space between storyboards and Figure 1. A mock-up working experiences. The of a ghost in a importance of informal content graveyard. goes beyond simply allowing designers to quickly prototype complex content, however. Goel found that the design space is more effectively explored when informal, “sketchy” content is used, and that typical drawing programs focus effort on a single design idea [5]. Other research bears this out [7][13]. The second element of our work will be support for early experience testing using Wizard-of-Oz (WoZ) techniques. WoZ-controlled experience testing is important in AR environments not just because it gives the designer the means to control the internals of their prototype, but also because it allows designers to simulate sensing, complex character behaviors, partial narratives and other parts of the system that have not yet been implemented [6]. Based on our experience, and the experiences of those building other interactive physical environments such as MR theme park rides [11], we believe that the combination of early experience testing with rapid content creation in real physical spaces is the key to rapid design of experiential AR systems, and that dealing with the interactions between the physical and virtual worlds is what distinguishes AR design from other interactive system design. By supporting the creation of informal content, we also allow the designer to focus on the specification and testing of the user’s interaction with the combined physical/virtual experience and with the interactions between the physical and virtual content elements. These relationships are not simple to deal with; they may evolve over time, may involve multiple physical and virtual elements, and may be based on the output of a heterogeneous collection of sensors. Consider the sketch of a ghost standing among the graves in a historic cemetery, shown in Figure 1. It would be impossible to know how the ghost would interact visually with the monuments and headstones, and with the likely movements of the user, if the designer could not explore this experience in situ. We have chosen Director as the basis for our work because it is a flexible multimedia-authoring tool with a rich feature set, programming environment and plug-in SDK (allowing us to extend it as necessary), and because most designers, including the design students with whom
we work, have significant experience using it. As with other non-AR researchers who have extended Director to support their collaborations with artists and designers, we do not assert that it is the ideal environment for our work (i.e., for authoring AR experiences), but it is a very good system in many ways, and is sufficiently powerful to deploy fully working AR prototypes. Furthermore, while it is possible to do simple things without programming in Director, virtually any interesting Shockwave content requires at least simple programming. Therefore, experienced Director users are often surprisingly good programmers.
integrate with existing Director behaviors and concepts. This allows experienced designers to adapt the behaviors to their own needs.
4. DART Architectural Overview The DART system consists of behaviors (extensions to the Director environment written in LINGO) and Xtras (plug-ins for Director written in C++). The DART Xtras provide AR services such as video capture, tracking (via VRPN), and fiducial registration (currently, via the ARToolkit) to Director applications. The user can place behaviors onto the Director score (an arrangement of channels that organize, display and control the application over time). These behaviors allow for the graphical creation of an AR application. Behaviors also represent the basic building blocks of any AR application: virtual objects to be placed in the 3D environment, tracker information, video input (for video mixed AR), and triggers that control the logic (state machine) of the application. The complex services provided by the Xtras are encapsulated by “low level” behaviors that present the user with simple abstractions for services such as VRPN trackers and camera parameters and configurations. The user can add objects (3D models, 2D storyboards, video, audio) to her AR environment by placing “Actor” behaviors on the score. These actor behaviors present the user with simple input fields where she can configure such properties as the position and appearance of these objects. These “actors” can have their position linked to tracking information provided by the tracking and fiducial registration behaviors. DART provides a library of triggers that can be used to create the state machine logic that will control the behavior of the “actors” in the application. These “trigger” behaviors allow the user to specify “when X happens, do Y.” For example, the user could place a trigger behavior on an “actor” that would move the position of the actor when a certain sensor value was received, or would start the video on a textured object when a certain time was reached. These simple triggers can be used to build up complex behaviors for an AR application. The key feature of the DART behaviors is that they are not intended to provide comprehensive support for AR experiences: indeed, we believe an attempt to create such a library would be misguided and doomed to failure. Rather, they are designed to provide a simple yet complete illustration of how to create AR experiences using the low-level AR services provided by the Xtras, and to
Figure 2. Screen shot of DART being used to create an
AR application using video-based “actors”.
5. Conclusion DART is not yet widely distributed, so we cannot yet draw any conclusions from our design. However, based on our own experiences using Director (and initial versions of DART), we are optimistic that we will meet our design goal, namely to support the creation of AR experiences, and the exploration of AR as a new medium, by non-technologists. If we truly wish to see AR develop as a medium, the importance of putting AR directly into the hands of nontechnologists cannot be stressed too heavily. The history of media teaches us that a medium is never just the technology itself [9]. While the pure technology provides a set of features that can be exploited (what we in HCI would call “affordances” [8]), the useful forms for a new medium emerge through use, and that these media forms always develop relative to the media culture in which they are created; both the designers and the consumers of a new medium approach it through the lens of those media they currently understand (a process known as remediation [1]). Media forms can be thought of as sets of conventions that can be used by developers to create meaningful information experiences for these users, and the tools used to work with a medium reflect the common media forms. Consider the evolution of film. The film camera, projection, and distribution system (invented and perfected from about 1890 to about 1930) constituted a new technology of representation but did not define what we think of as “film” [2]. This technology developed into a large number of different film forms over time: the Hollywood narrative film, the documentary, the propaganda film, and so on. The common structure of modern film forms in turn define the necessary features of
digital video editing software, such as Apple’s iMovie. Programs like iMovie define the capabilities and experience of most people working with video. The medium of AR will evolve similarly; we are taking the first steps toward defining the tools and processes that will eventually define how vast numbers of people create and experience AR and other mixed physical/virtual experiences. Similarly, research has shown that understanding the act of composing content is the key to developing support for new media, since the most critical aspect of a medium will be about composition, from experts down to everyday people [4][12]. Unfortunately, the history of media has also shown that any medium (not just AR) will not reach its potential until it is put into the hands of designers. While our goal is to enable designers to work independently, we also recognize that while technology is in its infancy, multidisciplinary exploration can lead to revolutionary advances in understanding that would be impossible by either community in isolation: designers are provided with the tools they need to explore the design possibilities afforded by this new medium, and technologists are immersed in an environment where novel technical challenges need to be solved.
6. References [1] Bolter, Jay David and Grusin, Richard. Remediation: Understanding New Media. MIT Press, 1999. [2] Bordwell, D. and Thompson, K. (1997) “Film Art: An Introduction.” University of Wisconsin, Madison. [3] Buchenau, Marion and Suri, Jane Fulton. “Experience prototyping.” In proceedings of Designing Interactive Systems, August 2000, pages 424–433.! [4] diSessa, A. (1985). “A principled design for an integrated computational environment.” HumanComputer Interaction, 1(1): 1-47.
[5] Goel, V. Sketches of Thought. Cambridge, MA. The MIT Press, 1995. [6] Klemmer, Scott R., Anoop K. Sinha, Jack Chen, James A. Landay, Nadeem Aboobaker, Annie Wang. "SUEDE: A Wizard of Oz Prototyping Tool for Speech User Interfaces." CHI Letters: ACM Symposium on User Interface Software and Technology: UIST '00, 2000. 2(2): p. 1-10. [7] Landay, James A., and Brad A. Myers, "Sketching Interfaces: Toward More Human Interface Design." IEEE Computer , vol. 34, no. 3, March 2001, pp. 56-64. [8] Norman, D. (1990) “The design of everyday things.” Currency/Doubleday. [9] Oren, T. (1990). “Designing a new medium.” In B. Laurel (Ed.), The Art of Human-Computer Interface Design (pp. 467-479). Reading, Mass.: AddisonWesley. [10]Schön, D. A. “The Reflective Practitioner: How Professionals Think in Action.” Basic Books, NY. 1983. [11] Schell, Jesse and Shochet, Joe. "Designing Interactive Theme Park Rides: Lessons Learned Creating Disney's Pirates of the Caribbean -- Battle for the Buccaneer Gold." In 2001 Game Developers Conference Proceedings. CMP Media: San Francisco, CA. 2001. [12] Soloway, E., Guzdial, M., and Hay, K. E. (1993). “Reading and writing in the 21st century.” In EDUCOM Review, 28(1), 26-28. [13] Strothotte, T., Preim, B., Raab, A., Schumann, J., and Forsey, D.R. (1994) “How to Render Frames and Influence People.” In Computer Graphics Forum (13)3, Proceedings of euroGraphics 1994, pp. 455466.
