Tangible Interfaces in Design Computing - Cumincad

Tangible Interfaces in Design Computing Hartmut Seichter, Thomas Kvan Department of Architecture, The University of Hong Kong, Hong Kong, China [email protected], [email protected]

Sensorial richness is essential in the design process, yet digital design tools do not respond to this need. Tangible interfaces offer an opportunity for interaction with design computing systems to explore means of supporting a wider range of experiences. In this paper we look at implementations of tangible interfaces through a framework based on the concept of affordance. We conclude with a brief introduction to experimental tangible interfaces that have been developed in order to carry out collaborative user evaluations in a design studio setting and evaluate these in the same framework. Keywords: Augmented Reality; Tangible Interfaces; HCI; Design Computing

Introduction Architectural design is a collaborative effort involving different professions in a process of postulation, negotiation and resolution. As they negotiate, language plays a major role, not just natural but design language as represented by a common vocabulary of design symbolism. Designers rely on communication through the multiplicity of representation. By its nature, architecture relies on a variety of media and modes of communication. If we take a look at design computing tools available today, we observe that the majority of tools make weak allusions to traditional modes of working, failing to take into account new means of communication. A collaborative system needs to facilitate acts of negotiation which are usually not achieved with the adoption of the WIMP (Window, Icon, Menu and Pointer) interface (Apple Computer Inc., 1992), which masks access to shared data. Digital design produces a virtualization by removing real world physical properties, such as weight, fric-

tion etc., rather than using existing tools to create a sensual experience of space. More commonly, these design systems strive to represent visual appearance like shape and colour, failing to simulate other sensations like the coldness of stones, the velvet touch of polished bare wood, not to speak of the smell of mown grass. While it may be argued that many of these sensations are difficult to simulate, this does not diminish their importance or impact in design. Simulation of a wider range of sensorial interfaces should not be ignored. There are sensations that are easier to simulate, like 3D aural and dynamic physical properties, which are not considered in current tools. A fixation with minimally interactive visual media is justified by assumptions of the tools architects ‘traditionally’ used to work on their design. Representation and product, distant in earlier forms of architectural representation (Evans, 1997), drift further apart.

Virtual Environment 2

159

Design Beyond Reality What do the tools and media do better than the existing ones (McLuhan, 1994)? The problems of interfaces pose challenges that unveil opportunities, in particular to enable actions impossible in real-world representations. Frequently stated but often overlooked, digital interfaces should not seek to replicate analogue processes. An interface need not to replace reality but can augment it. If we take a look at research in the field of medicine or geography (Kaufman et al., 1997, Shelton and Hedley, 2002) it becomes very clear that a simulation through augmentation has its advantages especially in accessibility, decision-making, learning and primarily sharing (Arias et al., 2000). In this paper we will explore the boundaries of what can be achieved with tangible interfaces in architectural design in order to facilitate the transition from intent, bringing process and product closer together. In particular, we will briefly examine the nature of interfaces in Virtual Reality (VR) systems. VR demands different interfaces. Immersion of a user into a virtual environment requires changes both in the interfaces and also the way we design (Henry and Furness, 1993, Davidson and Campbell, 1996, Gross and Do, 2000, Regenbrecht et al., 2000, Schnabel, 2003). One approach to this is to combine both virtual and real; Augmented Reality (AR) enhanced VR by stepping back and integrating the simulation into a physical environment. Initially appearing as tools for physical interactions of digital models (Frazer et al., 1980), the idea of tangible interfaces has emerged again as ‘props’ in VR (Stoakley et al., 1995, Poupyrev et al., 1998). VR and AR do share necessities in interface design (Piekarski and Thomas, 2002) that are grounded in the possibility of decoupling the physical and virtual environment. What is missing here is a real extension of productive interactivity by augmented reality media in order to provide the designer with a more satisfactory tool to communicate design without cluttering the process with an extensive range of tools. These need

160 Virtual Environment 2

to be easily perceived as offering a range of actions, not singular operations.

Augmented Affordance Gibson (1979) coined the concept of affordances, a term subsequently reinterpreted by many people. He articulated the term to identify the significance between the world and an actor, either a person or animal. In this paper, we follow the common interpretation by Norman (1988) that introduces a distinction between perceived and real affordances, which is important in design. An affordance is not pre-determined and fixed but will change as new actors or situations appear. Physical, psychological and cultural concepts influence the perception and usage of an object. Actual affordance is partly dependent on those external influences and part upon the nature of the object itself. Just as real and perceived affordances can not be easily discerned in all cases, nor can we draw simple distinctions between real and virtual representations. We can instead conceive of digital design taking place in a virtuality-reality continuum (Milgram et al., 1994, Azuma, 1997) in which we place the techniques of AR. In this continuum, we find a reflection of the distinction between real and perceived affordances, and in the realm of AR we observe that interaction is different in nature to that we know in other domains. Rekimoto has described this as Augmented Interaction (1995). Tangible interfaces can be seen as offering a conduit between the real or perceived affordances implied by the physical properties of the interface tool and the affordances created by the digital behaviours in the virtualized interface. To elucidate this idea, we propose the term augmented affordance. Let us illustrate the idea with an example. A simple AR interface augments a real object visually, which in turn reacts to a physical action to change the displayed image. It could be argued that this is the similar to Donald A. Norman’s „door“. The door’s physical affordance of the real object is limited; the augmented

affordance can easily be altered using software. It is also feasible to implement an augmented affordance in contradiction to the affordances of the real object. For ease of operation, however, real and perceived affordances should be complementary, otherwise a mapping problem occurs. Nevertheless, the ideal tangible interface does enhance the interface beyond physical boundaries. The relation between the physical object, which is the handle for the interface and its behaviour as a virtual object needs to be understood. The programmed behaviour between them creates the interface. Unlike in current desktop environments, the relationship is different as real and perceived objects do not map with each other. Some collaborative design requires participants to work together around physical objects. It has been observed that having a shared visual space helps collaborative activities in certain contexts (Clark and Brennan, 1991). Collaborators can better understand the current state of their task and can communicate and ground their conversations more efficiently, leading to faster and better task performance. This is particularly the case in tasks that are visually complex or when participants have no simple vocabulary for describing their shared world (Kraut et al., 2002). Direct influence on the design process is imposed by interfaces as they can influence on the way we resolve a problem. Schön (1983) describes the design process through the concept of ‘reflection-in-action’. Reflection is a self-communication helping the designer to understand and perhaps reframe a problem. From the discussion of the Shannon-Weaver Model (1949) and other transmission models, we learn that noise introduced by a variety of sources, especially by access, can disturb communication and in consequence affect understanding of the communication significantly. That means that interpretational varieties can render an interface useless as the encoded message can be read but not understood. Introducing lower level symbols (Knauer, 2002) through tangible interfaces can partly bypass interpretational noise.

Review In order to understand existing augmented interfaces (limited to audio and visual augmentation) and their methodologies, we can identify the advantages that they offer designers. To reach this understanding, we established a simple taxonomy to identify important elements for designing: • Shareability – The physical appearance, tangibility or real affordance of an interface by means of transparency to the user in a collocated distal collaborative setting. • Affordance – How does the interface twist the usage to gain benefit from prior knowledge for the actual programmed behaviour? Physical property and virtual behaviour are important factors to render the beneficial aspect for a creative tool. It needs to be identified what can be achieved beyond conventional interfaces. • Adoptability – The initial encounter for an technology, including interfaces, is a major hurdle (Rogers, 2003). Thus, the reduction to real world entities does help to overcome this, but it could also confuse the user. What are the compromises done here? Overview of Tangible Interfaces Tools have long combined real interfaces with extended functionality. The Mechatronic Design Modelling (Frazer et al., 1980), did not yet overlay real and virtual objects in one space, but is considered one of the earliest tangible design tools. The actual affordance of spatial arrangement, for example stacking and connecting, triggers a programmed interface to generate a virtual space. The 1:1 mapping of the real and virtual interface provides a clear advantage for novice users. Evidence that digitally enhanced real objects lower the mental workload has been presented by Fitzmaurice et al. (1995) and later Tang et al. (2003). Both experimented with augmented assembly based on LEGO-like systems, much like Mechatronic Design Modelling.


161

Another non-immersive example by Klemmer (2001) has demonstrated a PostIt-Notes interface with an intelligent mapping technique of real and virtual affordances to enhance the layout process of websites. Its interaction based on notes written on pieces of paper, which in turn get their meaning through virtual outlining by sketching connecting arrows. The simplicity of the haptic interface combined with the augmented complexity of outlining makes it easier for users to get familiar with the system. Arias et al. (2000) demonstrated with The Envisionment and Discovery Collaboratory the advantages of combining real and virtual objects on a shared table. In consideration of the concepts discussed by Schön regarding reflection, it sports a separate reflective space. Nevertheless, they admittedly had interaction problems as the mapping of real and virtual behaviours were limited, hence the meaning of the separate space was not perceived properly. This could be due to the interaction technique used, which resembled dragging and clicking similar to desktop systems. Based on these findings, newer research like Luminus Table (Ishii et al., 2002) or MausHaus Table (Huang et al., 2003) developed the idea of bench-like systems further. The interfaces combined the affordances through their physical appearance. MausHaus went further by using arbitrary objects that could be combined on the table. Findings for the Augmented Workbench highlighted an interesting issue with the lack of physical appearance (e.g. mass, solidness). Thus, showing the actual affordance of an object can be similarly disturbed through the perceived one. The addition of immersion by personal displays opened new opportunities for tangible media. Information rendered in place allows interfaces to interact directly with their real world triggers. The original Magic Book (Billinghurst et al., 2000) as one of the common applications of the Augmented Reality Toolkit (Billinghurst and Kato, 1999) shows how a real object extends its physical appearance to the user. Resembling a book it links pages with fiducial markers to virtual objects. The perception of


a storytelling book does not change while interacting with it. Subsequently, the perceived affordance of viewing a 3D object in the book and inside the environment does not change and is easy to learn and remember for a user. Tiles (Poupyrev et al., 2001), a design tool for dashboard instruments, introduced interactive content to the usual marker concept. But here a priori knowledge is needed to recognize that the Tiles patterns afford connections through spatial proximity. The augmented interface acts here through its physical affordance with a complex background task of rewiring in and outputs. New opportunities for tangible interfaces open up with automatically registering AR applications using projective or holographic methods. The general tendency is to blur the boundary between real and virtual in the search of a ubiquitous interface. Summary The interfaces discussed above show a common strategy, to support collaboration by sharing physical interfaces. The abstract action of accessing a data model is made more comprehensible by the immediacy of the interface. The downside of those approaches is clear; there are broad assumptions about the mapping of the interface. This inflexibility is the price paid in order to reduce confusion. Tangible interfaces create new opportunities through their physical appearance and break down the functional complexity which exists in current desktop tools. Yet there are all new opportunities that have to be explored (Buxton, 1997).

Experiments with Tangible Interfaces This brief observation indicates a series of interfaces, some of which have been tested in a design studio setting. What yet has to be evaluated is a measurement in terms of user performance and satisfaction in a collaborative setting with AR. The basic idea is to map familiar objects with new meaning. Avoiding

a clash between established knowledge of the object and the new interaction technique is essential to successfully create those devices. Tangible interfaces hide the complexity of the data structures by providing modeless access to them. In comparison WIMP, by its nature, distinguishes between data and access, usually requiring the user to first decide the mode of access, and then provide the data. Sketchand+ This AR prototype utilizes a scribbling interface through the metaphor of a digitizer tablet (Seichter, 2003). Computer-literate designers immediately perceive the affordance of drawing. Within the scope of digital design it has the advantage that the system can be used instantaneously, yet it does distort normal behavior. The virtual response is a 3D sketch rather than two-dimensional. Thus, the interface visually affords simple drawing but extends that into the third dimension. It also changes the internal behaviour; after the initial stroke has been made all subsequent strokes will be interpreted as editing. The directness in the interfaces does not require the user to set modes. By comparison, the interface for SketchBook Pro (Alias, 2004) demands the user to establish the mode by selecting icons from a toolbar or selecting menu items. The tangibility of the slot system makes it easy to integrate it into a collaborative setting.

BenchWorks Use of sketchand+ identified limitations of the maximum model size, so a new system was developed called BenchWorks. Here a 1.5x1.0m2 digital whiteboard was mounted horizontally as working surface (Seichter, 2004), providing a much larger interactive shared space allowing new interaction techniques. The traditional pen, paper and eraser metaphor creates the link between the traditional and virtual tools (Figure 1). The interactive collaborative space is defined by the sensitive area, which is shared by collocated and distant designers. They can use both real and virtual models in an arbitrary mixture. Real models are distributed and shared through 3Dscanned representations. Sandbox Experiments with the BenchWorks prototype raised an interesting question. If the physicality of interfaces influences the performance, to what extend is a dynamic physical simulation beneficial inside an AR setting? As described in the introduction, designing utilises our senses. But despite using real objects within BenchWorks, perceived properties such as bounciness, friction and collision offer new opportunities. These behaviours do not only help the immersion of the user they are also used to introduce new design methodologies. To engage users to interact with these new proper-

Figure 1 Left: Scribbling on the table. Right: Occlusion-corrected rendering mixing real and virtual models.


163

ties, new tools have been added to the BenchWorks prototype: the bucket and a pistol (Figure 2). The interaction is similar to what we know from the real world, with the difference being that they can affect the real objects as well as the virtual and they are not only locally effective. The bucket is used to pour mass models inside the augmented scenery or scrape them off the surface. The interaction follows the rules we know from our environment. In order to apply forces the pistol can throw objects, which in turn bounce and collide with others, affecting their position and shape. Using these interaction techniques, backed by a dynamic physical simulation through Open Dynamics Environment (ode.org: Dec 2003), we hope to explore the boundaries between real and perceived affordances of interfaces. When virtual and real objects behave similarly, the

perceived affordance is blurred, but with the twist that the parameters of the virtual part are adjustable. This is essential to check assumptions designers make about the behaviour of interfaces and, respectively, their affordances. From this, we can find new solutions for tangible interfaces.

Conclusion This paper highlighted the possibilities of a new generation of interaction devices. Tangibility and in turn transparent access across users in remote or local settings seem to have the largest impact on effectiveness for collaborative design computing. Furthermore a brief introduction of current research regarding design interaction based on the libTAP (libtap.sf.net: Jun 2004) framework and AR Toolkit

Figure 2 Left column: Objects bounce into the model as they are poured in. Right column: The user “shoots” into the scenery - velocity and distance will determine the effect on the bench.


has been brought forward. This ongoing effort will lead to an evaluation of the effectiveness of tangible interfaces in a design studio setting. In addition to evaluation users will be encouraged to design their own tools within the framework, in response to flaws they identified. In the end we hope to get valuable data about user behaviour in a design scenario regarding augmented interaction and tangible interfaces.

Acknowledgement Thanks to the members of the Research Group at the Department of Architecture, The University of Hong Kong for comments and critique and the technical staff for support.

References SketchBook Pro, Toronto, Canada 2004 Apple Computer Inc.: 1992, Macintosh Human Interface Guidelines, Addison-Wesley. Arias, E., Eden, H., Fischer, G., Gorman, A. and Scharff, E.: 2000, Transcending the Individual Human Mind - Create Shared Understanding through Collaborative Design, ACM Transactions on Computer Human Interaction, 7 (1), 84-113. Azuma, R. T.: 1997, Survey on Augmented Reality, Presence: Teleoperators and Virtual Environments, 6 (4), 355-385. Billinghurst, M., Campbell, S., Hendrickson, D. and Chinthammit, W.: 2000, Magic Book: Exploring Transitions in Collaborative AR Interfaces, SIGGRAPH 2000. Billinghurst, M. and Kato, H.: 1999, Real World Teleconferencing, ACM CHI’99, Pittsburgh, PA, pp. 194-195. Buxton, W. A. S.: 1997, Living in Augmented Reality: Ubiquitous Media and Reactive Environments in K. E. Finn, A. J. Sellen and S. Wilbur (eds.) Video Mediated Communication, Erlbaum, Hillsdale, N.J, pp. 363-384. Clark, H. H. and Brennan, S. E.: 1991, Grounding

in communication in L. B. Resnick, J. M. Levine and S. D. Teasley (eds.) Perspectives on socially shared cognition, American Psychological Association, Washington DC, pp. 127-149. Davidson, J. and Campbell, D. A.: 1996, Collaborative design in virtual space - Greenspace II: A shared environment for architectural design review, in P. McIntosh and F. Ozel (eds) Design computation: collaboration, reasoning, pedagogy, Acadia, University of Arizona, pp. 165-179. Evans, R.: 1997, Translations from drawing to building, MIT Press, Cambridge, Mass. Fitzmaurice, G. W., Ishii, H. and Buxton, W. A. S.: 1995, Bricks: Laying the Foundations for Graspable User Interfaces, SIGCHI conference on Human factors in computing systems, ACM Press, Denver, Colorado, United States, pp. 442-449. Frazer, J. H., Frazer, J. M. and Frazer, P. A.: 1980, Intelligent Physical Three-Dimensional Modelling Systems, Computer Graphics 80 Conference, pp. 359-370. Gibson, J. J.: 1979, The Ecological Approach to Visual Perception, Houghton Mifflin, Boston. Gross, M. D. and Do, E. Y.-L.: 2000, Demonstrating the Electronic Cocktail Napkin, ACM Human Factors in Computing - CHI’96, Addison Wesley, Vancouver, pp. 5-6. Henry, D. and Furness, T.: 1993, Spatial perception in virtual environments: Evaluating an architectural application, Virtual Reality Annual International Symposium, ACM Press, pp. 33-40. Huang, C.-J., Do, E. Y.-L. and Gross, M. D.: 2003, MouseHaus Table, a Physical Interface for Urban Design, User Interfaces Software and Technology, UIST 2003, ACM Press, Vancouver, B.C., Canada. Ishii, H., Underkoffler, J., Chak, D., Piper, B., BenJoseph, E., Yeung, L. and Zahra, K.: 2002, Augmented Urban Planning Workbench: Overlaying Drawings, Physical Models and Digital Simulation, IEEE and ACM International Symposium on Mixed and Augmented Reality, ACM Press, Darmstadt, Germany.


165

Kaufman, S., Poupyrev, I., Miller, E., Billinghurst, M., Oppenheimer, P. and Weghorst, S.: 1997, New Interface Metaphors for Complex Information Space Visualization: an ECG Monitor Object Prototype, in K. Morgan (ed) Medicine Meets Virtual Reality, IOS Press, pp. 131-140. Klemmer, S. R., Newman, M. W., Farrell, R., Bilezikjian, M. and Landay, J. A.: 2001, The designers’ outpost: a tangible interface for collaborative web site, UIST. Knauer, R.: 2002, Transformation : basic principles and methodology of design, Birkhauser. Kraut, R. E., Gergle, D. and Fussell, S. R.: 2002, The use of visual information in shared visual spaces: informing the development of virtual co-presence, ACM conference on Computer supported cooperative work, ACM Press, New Orleans, Louisiana, USA, pp. 31-40. McLuhan, M.: 1994, Understanding Media: The Extension of Man, The MIT Press, Cambridge. Milgram, P., Takemura, H., Utsumi, A. and Kishino, F.: 1994, AUGMENTED REALITY: A CLASS OF DISPLAYS ON THE REALITY-VIRTUALITY CONTINUUM, in Telemanipulator and Telepresence Technologies, Vol. 2351. Norman, D.: 1988, The Psychology of Everyday Things, Basic Books, New York. Piekarski, W. and Thomas, B. H.: 2002, Unifying Augmented Reality and Virtual Reality User Interfaces, in University of South Australia, Wearable Computer Laboratory, School of Computer and Information Science, Mawson Lakes, SA, 5095, Australia, pp. 15. Poupyrev, I., Tan, D., Billinghurst, M., Kato, H., Regenbrecht, H. and Tetsutani, N.: 2001, Tiles: A Mixed Reality Authoring Interface, INTERACT 2001 Conference on Human Computer Interaction, Tokyo, Japan. Poupyrev, I., Tomokazu, N. and Weghorst, S.: 1998, Virtual Notepad: Handwriting in Immersive VR, VRAIS, IEEE, Atlanta, Georgia. Regenbrecht, H., Donath, D., Kruijff, E., Seichter, H. and Beetz, J.: 2000, VRAM - a virtual reality aided


modeller, in D. Donath (ed) 18th Conference on Education in Computer Aided Architectural Design in Europe (eCAADe 2000), VDG Weimar, Weimar, pp. 235-237. Rekimoto, J. and Nagao, K.: 1995, The World through the Computer: Computer Augmented Interaction with Real World Environments, UIST, ACM Press, pp. 29-36. Rogers, E. M.: 2003, Diffusion of innovations, Free Press, New York. Schnabel, M. A.: 2003, 3D Maze: Creation and Translation of Abstract Virtual Architectural Environment, in K. T. Lee and K. Mitchell (eds) International Conference on Computers in Education (ICCE), Hong Kong, pp. 1201-1209. Schön, D. A.: 1983, The reflective practitioner: how professionals think in action, Basic Books, New York. Seichter, H.: 2003, sketchand+, in A. Choutgrajank, E. Charoensilp, K. Keatruangkamala and W. Nakapan (eds) Computer-Aided Architectural Design Research in Asia (CAADRIA), Bangkok Thailand, pp. 209-219. Seichter, H.: 2004, Benchworks: Augmented Reality Urban Design, in H. S. Lee and J. W. Choi (eds) Computer-Aided Architectural Design Research in Asia (CAADRIA), Yonsei University Press, Seoul, Korea, pp. 937-946. Shannon, C. E. and Weaver, W.: 1949, The mathematical theory of communication, University of Illinois Press, Urbana. Shelton, B. E. and Hedley, N. R.: 2002, Using Augmented Reality for Teaching Earth-Sun Relationships to Undergraduate Students, Augmented Reality Workshop, ACM Press, Darmstadt, Germany. Stoakley, R., Conway, M. J. and Pausch, R.: 1995, Virtual Reality on a WIM, Interactive Worlds in Minature, SIGCHI’95, ACM Press, Denver, CO, pp. 265-272. Tang, A., Owen, C., Biocca, F. and Mou, W.: 2003, Comparative Effectiveness of Augmented Reality in Object Assembly, CHI Letter, 5 (1), 73-80.