Work-in-Progress
CHI 2015, Crossings, Seoul, Korea
Augmented Miniature Prototyping Toolkit for UX in Interactive Space Han-Jong Kim CIDR Lab, Department of Industrial Design, KAIST 291 Daehak-ro, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected] Tek-Jin Nam CIDR Lab, Department of Industrial Design, KAIST 291 Daehak-ro, Yuseong-gu, Daejeon 305-701, Republic of Korea
[email protected]
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). CHI'15 Extended Abstracts, Apr 18-23, 2015, Seoul, Republic of Korea ACM 978-1-4503-3146-3/15/04. http://dx.doi.org/10.1145/2702613.2732744
Abstract It is challenging to envision the user experience (UX) within an interactive and ubiquitous computing space in the early phases of a design process due to the large scale and complex interactivity among multiple people and agents. To address the limitations of existing tools, we present miniStudio, a prototyping toolkit for interactive and ubiquitous computing environments larger than human scale. It consists of software and hardware kits for constructing miniature physical spaces and testing dynamic interactions in them through a combination of paper models and Lego. It also supports prototyping for interactivity using projection-based augmented reality with invisible markers and a pro-cam system in instantly created miniature spaces. We applied Adobe Photoshop as an interactivity authoring environment since it is one of the most popular design tools. From the preliminary evaluation with junior designers, miniStudio showed the potential to support the iterative exploration of interactive spaces in the early phases of the design process.
Author Keywords Augmented miniature; interactive space; prototyping tool; spatial user experience
ACM Classification Keywords H.5.2 User Interfaces – Prototyping
2229
Work-in-Progress
CHI 2015, Crossings, Seoul, Korea
Introduction The subject matters of design have expanded from single products to integrated product ecologies. Due to the development of ubiquitous computing and the Internet of Things [16], it has become necessary for designers to consider the dynamic user experience in large, interactive, and ubiquitous computing spaces. In such spaces, the scale of the system is larger than humans, and the hardware and software components are interconnected to support user experiences within them. These spatial experiences used to be designed by architects or interior designers. Recently, designing spatial user experience has become an important topic for interaction designers specializing in ubiquitous and pervasive computing [12]. However, there have been few studies on developing design support tools for spatial user experiences [10]. Prototyping tools for spatial user experiences should support rapid construction and modification for designers’ reflection. They need to help promote effective communication with various stakeholders during the design process. However, conventional prototyping methods, such as sketching and 3D modeling, have limitations in designing for the new interactive spaces. It is challenging to test dynamic and complex interactions in the spaces, and the methods also cannot effectively prototype spaces larger than human scale. Some previous studies [8] have introduced ways of building real-scale prototypes. However, they require expensive resources and are thus not appropriate for the early phases of the design process. In this paper, we present an augmented miniature prototyping toolkit for effective exploration of user
experiences in interactive and ubiquitous computing spaces. Designers can construct physical spaces faster with miniatures. Moreover, by adding interactivity to miniatures, it is possible to test dynamic interactions in the spaces. We applied projection-based augmented reality technology to do so. We also considered the toolkit to be effective for use by designers without high-level implementation skills.
Related Works Designers use storyboards, informance [1], and experience prototyping [2] to explore system-level user experiences in large environments. Tangible design tools [11] and lo-fi, small-scale prototypes [13] have been proposed to support scenario-based methods for large-scale service design. However, such methods and tools deal with static ideas, making it hard to build and reflect on complex and dynamic interactions in a space larger than human scale. Meanwhile, several miniature-based prototyping methods have been proposed for complex and largescale prototypes in the HCI field. For example, miniature-based simulations, such as Urp [14] and Hybrid Prototyping [9], were applications for urban planning and large public displays that were hard to be implemented in actual prototypes. Miniatures have also been used to explore interiors [6] and theater stage designs [5]. However, most previous studies focused on manipulating virtual contents tangibly. Few studies have explored interactive miniatures as a design prototyping toolkit. Although there have been attempts to use static miniatures for design, there is a lack of studies on the direct application of dynamic representations in scaled miniatures.
2230
Work-in-Progress
CHI 2015, Crossings, Seoul, Korea
miniStudio To address the existing tools’ limitations, we present miniStudio, a prototyping toolkit for interactive and ubiquitous computing environments larger than human scale. The key features of miniStudio, as a design support tool, can be explained in three parts. First, it supports the rapid construction and modification of miniatures through a combination of paper and prefabricated Lego bricks. Custom-made 3D paper models can resolve the limitations of freeform shape construction. Projection-based augmented reality [3, 7, 17] is provided for augmented prototyping [15] with the miniatures. miniStudio is equipped with invisible AR(Augmented Reality) markers and a pro-cam module. Second, it supports prototyping for interaction with the miniatures. The surfaces of the miniatures can be changed through defined events using projection mapping. Users can set intended states with a series of images. Dynamic changes of surfaces are made according to user-defined events such as the location of the miniature, proximity, and time.
Figure 1. Components of miniStudio: (a) Studio Generator, (b) Configurator, and (c) the software and pro-cam of Studio Projector.
Lastly, it uses Adobe Photoshop as an interactivity authoring environment. Photoshop is one of the most popular 2D graphic-editing software programs available, and it also supports effective work of designers by allowing existing image sources to be edited. Fogarty et al. noticed that layers could specify both appearance and semantic meaning for designers [4]. miniStudio applies this notion to represent the interactive and dynamic surfaces of miniatures without programming, especially as a designer-friendly toolkit. We used the layers of Photoshop to define states and events, as well as the graphical images being projected.
Components of miniStudio miniStudio consists of three components. The first component is Studio Generator, a software tool for generating miniature 3D paper models (Figure 1a). A user can set the dimensions of the paper model to be combined with Lego bricks. The tool provides several model templates for quick modeling. The dimensions are set to Lego brick units (8x9.6x8mm). The paper models are exported as unfolded figures so that they can be constructed and assembled with Legos by punching. When an invisible AR marker sticker is attached to the center of the paper, the paper model becomes a projected surface for interactive digital contents. The second component is Studio Configurator, a Photoshop plug-in application used for defining events and state changes (Figure 1b). It imports unfolded drawings of 3D paper models saved in Studio Generator. Users can assign multiple graphic images over the planar figure as states of the surface. By clicking the panel buttons in Studio Configurator, a user can add layer groups to define event blocks. Images of the event group are projected sequentially as an animation onto the paper model when an associated event is triggered. The final component is Studio Projector, an integrated hardware and software pro-cam kit for projection mapping (Figure 1c). The camera of the pro-cam module detects and tracks a white invisible marker on the miniature model. Then, the projector part of the pro-cam projects relevant surface images made in Photoshop onto the marker. This component has a software sub-component for initial calibration between the camera and projector.
2231
Work-in-Progress
Figure 3. Tangible event icon of (a) scene and (b) position.
CHI 2015, Crossings, Seoul, Korea
Example Events to Support Interactivity Currently, a user can define four types of events to support the interactivity of miniatures: scene, move, proximity, and time (Figure 2). A scene event is triggered when the tangible icon of a scene (Figure 3a) is detected in the miniature’s space. It can be used to manually change the state mode, just like key pressing. Move and proximity events are triggered according to the distance to the position icon (Figure 3b) or other model. Time events are called when a certain amount of time has passed, and the timer can be reset in Studio Projector.
Figure 4. Examples of model templates in Studio Generator.
Figure 2. Four types of events supported in miniStudio.
Prototyping Workflow with miniStudio
Figure 5. Progress of assembling a miniature model.
Figure 6. Dialog box for adding event group on Photoshop.
Step 1: Generate Miniature Model The first step is to build a basic form of the model using Studio Generator. Templates can be used for a quick start (Figure 4). The user defines the model’s dimensions. Sliders are provided for setting the model’s width, height, and depth. A 3D model is instantly shown by the software. The user saves the model by clicking the save button. Then, the user makes an unfolded paper model on Studio Configurator. A planar figure of the saved model is loaded when the user clicks the import button on the panel in Photoshop. Holes for assembly with Legos and square areas for attaching the marker sticker are automatically generated on the figure. The printed model can be assembled with Legos, and marker stickers are attached to the surface of the paper model (Figure 5).
Step 2: Make Surface Images and Define Events The second step is to define the virtual images and events to interactively project onto the miniature models. Layered images on Photoshop are used to map surface images. The layer group named default is generated by default. When users click the add event button on the panel of Studio Configurator, a dialog box (Figure 6) appears to input the parameters. After that, a new layer group for the event is generated. Sublayers with images inside an event group are used to create frame animations (Figure 7). Each layer group can be associated with a relevantly defined event. When graphic editing and event defining are completed, the user saves the authored results by clicking the export button for projection mapping. Step 3: Set Up the Pro-cam Module and Test The testing is done with the constructed miniature model and Studio Projector. The pro-cam module projects defined digital contents at a distance of about 500mm from the miniature’s space. Surface images that are generated in Photoshop are interactively projected onto a miniature model when the Studio Projector software runs. Users can adjust the thresholds of proximity events or reset the timer for time events while testing. When the surface images are modified in Photoshop, clicking the load button is used to update the contents.
Implementation
The Studio Generator and Projector software programs were developed in Processing. Studio Configurator was developed using Photoshop Script. We built Photoshop functions to automate specific tasks of importing, exporting, and defining events. Each script was arranged on a customized panel with Adobe Configurator.
2232
Work-in-Progress
Figure 7. Layer groups for events and sub-layers for animations
CHI 2015, Crossings, Seoul, Korea
One of the key characteristics of Studio Projector is the invisible AR marker sticker used for projection mapping (Figure 8). Willis et al. used IR-absorbing ink to hide markers [17]. We used reflective stickers to reduce the cost of making invisible markers in the aforementioned technique and to enable a simple attaching task. The sticker is produced by silk-screen printing white paint onto a white reflective sticker. The pro-cam module contains 850nm IR luminosity and an IR pass filter of over 780nm in front of the camera module. It can filter out distractions caused by projected image. The NyARToolkit Processing library was used for multiple marker detection. Inverted markers were printed to secure enough reflecting light for detection. Figure 9 shows the process of recognizing an AR marker.
Figure 8. Invisible AR marker with white printing on white reflective material.
Figure 9. Process of recognizing reflective invisible markers.
Preliminary Evaluation
Figure 10. Results of the simple tasks: (a) interactive lighting, (b) interactive kiosk, (c) media artwork
We conducted a preliminary user study to investigate how potential users adopt miniStudio and determine its value as a prototyping toolkit for interactive space. The study was conducted in two parts. In the first part, to examine how individual designers used the system, three student designers were asked to complete simple design tasks involving interactive lighting, interactive kiosk, and media artwork. Figure 10 shows the results of the experiment. The participants completed the interactive space prototyping quickly. The average completion time was 13 minutes 39 seconds (from 5m 34s to 25m 3s) to make one or two miniature models
with a single event. The participants made results with diverse complexity and fidelity in the surface images. The second part of the user study was conducted as a design workshop in which a group of three student designers participated. The theme of the workshop was to design an exhibition space consisting of ubiquitous computing and systems. The example interactive technologies were interactive banners and responsive advertisements. The two-hour workshop included an introduction (10 min), idea generation (20 min), prototype building (60 min), and a group interview (30 min). The interactive miniature space created from the workshop is shown in Figure 11. The individual usage and the design workshop showed several potential uses of miniStudio. Firstly, miniStudio could be used to concretize ideas that are hard to represent. The participants mentioned that using both physical models and dynamic contents was effective for identifying and concretizing vague ideas of interactivity. Secondly, the participants were positive about miniStudio as a specialized tool for designing interactivity in larger spaces. They stated that the most distinctive characteristics were the dynamic representation and real-time testing of a space’s interactivity, which other static prototyping tools such as Legos and storyboards cannot support. Lastly, miniStudio assisted in effective iterative prototyping with its wide level of fidelity. The participants said that it could be applied to build interactivity quickly, with a rough model at the beginning, as well as to detailed testing, with elaborate models used to progress ideas. Meanwhile, aspects for improvement of miniStudio were found. Some of the participants felt that the usage workflow was complex. They did not understand
2233
Work-in-Progress
CHI 2015, Crossings, Seoul, Korea
the usage sequence in the beginning due to the separated components. They also stated that the current version of the tool had limitations in supporting complex events. The participants reported that it was difficult to identify the overall relations between events and define the same events for multiple models. Thus, it is necessary to improve miniStudio with more intuitive and effective ways of defining events.
Conclusion and Future Work
Figure 11. Results of the group design workshop for the exhibition venue.
In this paper, we presented an augmented miniature prototyping toolkit, miniStudio, for exploring user experiences in large-scale interactive spaces. Based on the development of the toolkit and preliminary user evaluations, we identified several issues to consider further. The prototyping system will be improved for seamless and intuitive usage. Advanced techniques for projection-based augmented reality will be tested to increase accuracy and responsiveness. It is necessary to conduct a structured evaluation to examine the impact on real design practice and the most beneficial design application cases of interactive spaces.
users in planning theatrical productions. In Proc. IUI 2012, ACM (2012), 109–118. [6] Hosokawa, T., Takeda, Y., Shioiri, N., Hirano, M., and Tanaka, K. Tangible design support system using RFID technology. In Proc. TEI 2008, ACM (2008), 75–78. [7] Kakehi, Y., Yamaoka, J., Akatsuka, D., and Naemura, T. Tablescape animation. SIGGRAPH 2009: Talks on, ACM Press (2009), 1–1. [8] Marquardt, N., Diaz-Marino, R., Boring, S., and Greenberg, S. The proximity toolkit: prototyping proxemic interactions in ubiquitous computing ecologies. In Proc. UIST 2011, ACM (2011), 315–326. [9] Nakanishi, Y. Virtual prototyping using miniature model and visualization for interactive public displays. In Proc. DIS 2012, ACM (2012), 458–467. [10] Saarinen, P., Partala, T., and Väänänen-Vainio-Mattila, K. Visualize your spatial experience (VYSE): a method and a case study in an exhibition center. In Proc. NordiCHI 2012, ACM Press (2012), 486-495. [11] Sanders, E.B.-N. and Dandavate, U. Design for Experiencing: New Tools. In First International Conference on Design and Emotion, TU Delft (1999), 87–91. [12] Spohrer, J. and Stein, M. User experience in the pervasive computing age. MultiMedia, IEEE 7, 1 (2000), 12–17.
References
[13] Svanaes, D. and Seland, G. Putting the users center stage: role playing and low-fi prototyping enable end users to design mobile systems. In Proc. CHI 2004, ACM Press (2004), 479–486.
[2] Buchenau, M. and Suri, J.F. Experience prototyping. In Proc. DIS 2000, ACM Press (2000), 424–433.
[14] Underkoffler, J. and Ishii, H. Urp: a luminous-tangible workbench for urban planning and design. In Proc. CHI 1999, ACM Press (1999), 386–393.
[1] Burns, C., Dishman, E., Verplank, W., and Lassiter, B. Actors, hairdos & videotape - informance design. In Proc. CHI 1994, ACM Press (1994), 119–120.
[3] Dalsgaard, P. and Halskov, K. Tangible 3D tabletops: combining tangible tabletop interaction and 3D projection. In Proc. NordiCHI 2012, ACM Press (2012), 109-118. [4] Fogarty, J., Forlizzi, J., and Hudson, S.E. Specifying behavior and semantic meaning in an unmodified layered drawing package In Proc. UIST 2002, ACM Press (2002), 61-70. [5] Horiuchi, Y., Inoue, T., and Okada, K. Virtual stage linked with a physical miniature stage to support multiple
[15] Verlinden, J.C., De Smit, A., Peeters, A.W.J., and van Gelderen, M.H. Development of a flexible augmented prototyping system. Journal of WSCG 11, 3 (2013), 496503. [16] Weiser, M. The Computer for the 21st Century. Scientific American 265, 3 (1991), 94–104. [17] Willis, K.D.D., Shiratori, T., and Mahler, M. HideOut: mobile projector interaction with tangible objects and surfaces. In Proc. TEI 2013, ACM Press (2013), 331-338.
2234
