The Impact of Tangible User Interfaces on Designers' Spatial Cognition

The Impact of Tangible User Interfaces on Designers’ Spatial Cognition Mi Jeong Kim Key Centre of Design Computing and Cognition, University of Sydney Mary Lou Maher Key Centre of Design Computing and Cognition, University of Sydney RUNNING HEAD: TANGIBLE USER INTERFACES AND SPATIAL COGNITION Corresponding Author’s Contact Information: Mi Jeong Kim, PhD student Key Centre of Design Computing and Cognition, University of Sydney Wilkinson Building G04, Sydney, NSW2006, Australia, Phone +61 2 9351 2053 Email: [email protected]

Brief Authors’ Biographies: Mary Lou Maher is a design researcher with an interest in novel user interfaces to support designing and computer support for collaborative design; she is the Professor of Design Computing at the University of Sydney. Mi Jeong Kim is a design researcher with an interest in designers’ cognition while using tangible user interfaces; she is a PhD student in the Key Centre of Design Computing and Cognition at the University of Sydney.

-1-

ABSTRACT Most studies on tangible user interfaces for the tabletop design systems are being undertaken from a technology viewpoint. While there have been studies that focus on the development of new interactive environments employing tangible user interfaces for designers, there is a lack of evaluation with respect to designers’ spatial cognition. In this research we study the effects of tangible user interfaces on designers’ spatial cognition to provide empirical evidence for the anecdotal views of the effect of tangible user interfaces. In order to highlight the expected changes in spatial cognition while using tangible user interfaces, we compared designers using a tangible user interface on a tabletop system with 3D blocks to designers using a graphical user interface on a desktop computer with a mouse and keyboard. The ways in which designers use the two different interfaces for 3D design were examined using a protocol analysis method. The result reveals that designers using 3D blocks perceived more spatial relationships among multiple objects and spaces, and discovered new visuo-spatial features when revisiting their design configurations. The designers using the tangible interfaces spent more time in relocating objects to different locations to test the moves, and interacted with the external representation through large body movements implying an immersion in the design model. These two physical actions assist in designers’ spatial cognition by reducing cognitive load in mental visual reasoning. Further, designers using the tangible interfaces spent more time in restructuring the design problem by introducing new functional issues as design requirements, and produced more discontinuities to the design processes, which provides opportunity for reflection and modification of the design. Therefore this research shows that tangible user interfaces changes designers’ spatial cognition, and the changes of the spatial cognition are associated with creative design processes.

-2-

CONTENTS

1. INTRODUCTION 2. SPAIAL COGNITION IN DESIGNING 2.1. Epistemic action vs. Pragmatic action 2.2. Spatial Cognition 2.3. Creative Design Process 2.4. Hypotheses 3. COMPARING GUI TO TUI 3.1. Experiment Design Interfaces: 3D block vs. Mouse and Keyboard Systems: Tabletop vs. Desktop Applications: ARToolKit vs. ArchiCAD Design Tasks: Home office and Design office Participants 3.2. Experiment Set-ups TUI session GUI session 3.3. Experiment Procedure Training Experiment 4. METHOD: PROTOCOL ANALYSIS 4.1. Protocol Analysis in Design Research 4.2. Coding Scheme 4.3. Protocol Coding Segmentation Coding Process 5. ANALYSIS OF DESIGNERS’ SPATIAL COGNITION 5.1. Overall Observations 5.2. Analysis of the Three Levels of Designers’ Spatial Cognition Action Level: 3D modeling and Gesture Actions Perception Level: Perceptual and Functional Activities Process Level: Set-up Goal Activities and Co-evolution 5.3. Correlation between Physical Actions and Perceptual Activities 3D modeling Actions and Perceptual Activities Gesture Actions and Perceptual Activities 6. DISCUSSION AND CONCLUSION 6.1. Hypotheses Validation and Conclusion 6.2. Implications and Future Direction

-3-

1. INTRODUCTION A current paradigm in the study of Human-Computer Interaction (HCI) is to develop novel user interfaces which afford a natural interaction that take advantage of both human and computer perceptual capabilities (Turk 1998). People are developing tangible user interfaces (TUIs) as alternatives to traditional graphical user interfaces (GUIs) to meet a need for a more natural and direct interaction with computers. The term ‘TUIs’ was introduced by Ulmer and Ishii (Ullmer and Ishii 1997) as an extension of the ideas of ‘graspable user interfaces1’; they argued that TUIs allow users to ‘grasp & manipulate’ bits by coupling digital information with physical objects and architectural surfaces. Numerous tabletop systems have been customized for design applications and demonstrate many potential uses for TUIs (Coquillart and Wessche 1999; Fjeld et al. 1998; Obeysekare et al. 1996; Ullmer and Ishii 1997). They restore some of the tangibility by providing various physical interfaces through which designers create and interact with digital models. We are particularly interested in TUIs employed in tabletop systems for design applications since the tangible interaction afforded by the TUIs has potential to offer significant benefit to designers for 3D design. Most studies on TUIs for tabletop systems are being undertaken from a technology viewpoint (Fitzmaurice et al. 1995; Regenbrecht et al. 2002; Underkoffler and Ishii 1999). They described the fundamental ideas behind the systems and implemented prototypes for possible applications. Some initial user studies were conducted for the implementation of the prototypes, but the focus has been on the functionality of the prototypes, and the prototypes have not been evaluated from a cognitive perspective. Further, many researchers have argued that TUIs improve designers’ spatial cognition, but there has been no empirical evidence to support this (Fjeld et al. 1998; Lee et al. 2003; Ma et al. 2003). Although some researchers have reported on the users’ perception of TUIs using survey questionnaires or designer comments, the subjective nature of selfreports questions their validity as measures of cognitive ability (Vega et al. 1996). Our research starts from this gap in the existing research on TUIs, technology-oriented study, anecdotal views and subjective measurement of cognition. In this research we study the effects of TUIs on designers’ spatial cognition using protocol analysis. In the context of this research, spatial cognition is defined as perceiving and reasoning about visuo-spatial information in an external representation in architectural design. TUIs can be easily and rapidly manipulated because of the natural interaction afforded by the physical artifacts. However, this brings a question whether such a physical interaction improves designers’ spatial cognition in a real design task. We believe that a more in depth understanding of the effects of TUIs on designers’ spatial cognition would provide a perspective other than usability and is essential for the 1

Fitzmaurice Fitzmaurice, G (1996). Graspable User Interfaces. PhD Thesis, University of Toronto, Fitzmaurice, GW, Ishii, H and Buxton, W (1995). Bricks: Laying the Foundations for Graspable User Interfaces. In I. Katz R. Mack, L. Marks (Ed.) Proceedings of the CHI'95 Conference on Human Factors in Computing Systems, ACM Press, New York, 442-449. defines and explores graspable user interfaces, presenting five basic defining properties: space-multiplex both input and output; concurrent access and manipulation of interface components; strong specific devices; spatially-aware computational devices; and spatial re-configurability of devices.

-4-

development of tabletop systems. Based on a literature study we propose that TUIs on a tabletop system will change some aspects of designers’ spatial cognition for 3D design. Furthermore the changes in spatial cognition may be associated with creative design processes. Through the comparison of design collaboration using a GUI vs. a TUI in a pilot study, we found improved spatial understanding of object relationships in the TUI collaboration (Maher and Kim 2005). In this paper we report the results of an experiment using a protocol analysis, in which designers’ cognitive activities are collected using the think aloud method. The significance of this research is to empirically examine the ways in which designers perform spatial design using TUIs, in terms of spatial cognition.

2. SPATIAL COGNITION IN DESIGNING This research is concerned with designers’ spatial cognition while carrying out 3D spatial configuration using user interfaces in a digital environment. Cognitive design studies have put much emphasis on the localized information processing at the individual designer level. However, we approach the study of ‘spatial cognition in designing’ from three different perspectives: action, perception and process. This distributed cognition approach emphasizes the interaction of a person with tools and artifacts (Halverson 1994; Rogers and Ellis 1994). It is important to understand how user interfaces to digital models affect designers’ actions, how spatial cognition2 is defined in designing, and then what aspects of the design process are associated with designers’ spatial cognition. A consideration of these three perspectives formed the basis for the development of our hypotheses and coding scheme.

2.1. Epistemic action vs. Pragmatic action Fitzmaurice (Fitzmaurice 1996) discussed the notion of epistemic and pragmatic actions to provide the underlying theoretical support for graspable user interfaces. Epistemic actions refer to ‘exploratory’ motor activity to uncover information that is hard to compute mentally. One example of an epistemic action is a novice players’ movement in chess, which offloads some internal cognitive resources into the external world using physical actions. Many players move pieces around to candidate positions to assess the moves and possible counter-moves by an opponent. In contrast, pragmatic actions refer to ‘performatory’ motor activity that directs the user closer to the final goal. For example, the user expecting only pragmatic actions would set a goal first, and perform the minimal motor action to reach the goal (Fitzmaurice 1996; Gibson 1962; Kirsh and Maglio 1994). Our interest is in the argument that interfaces with physical objects may offer more opportunities for epistemic actions (Fitzmaurice 1996). The potential affordances of the TUIs such as rapid manipulability and physical arrangements may reduce the designers’ cognitive loads, thus resulting in changes in designers’ spatial cognition. Grey et al. (Gray et al. 2000) demonstrated that small features of an interface constrain interactive behaviors, thereby having effects on cognitive behaviors. The coincidence of action and

2

In this research the term ‘designing’ refers to a design activity and the term ‘design’ refers to the result of the design activity.

-5-

perception spaces inherent in the design of TUIs enables epistemic actions, which provides direct-interpreted 3D design platforms (Fjeld et al. 2001; Lee et al. 2003). In addition, we consider designers’ hand movements along with their design activity as possibly beneficial for cognitive processing because such movements characterize the spatial relationships among entities, thus promoting spatial reasoning (Goldin-Meadow 2003; Lavergne and Kimura 1987). In the ‘coin-counting’ experiment, Kirsh (Kirsh 1995; Kirsh and Maglio 1994) demonstrated that organizing activities such as positioning and arranging the position of nearby objects reduce cognitive loads as an complementary strategy for the task performance. Thus, we explored the role of hand movements or gestures in terms of a complementary strategy for designing, which would serve a similar function to the 3D modeling actions that are integral to cognition.

2.2. Spatial Cognition ‘Spatial’, or ‘visuo-spatial’, cognition is a broad field of enquiry emerging from a range of disciplines (Foreman and Gillett 1997; Knauff et al. 2002). According to De Vega (Vega et al. 1996), people process visuo-spatial information at least two different ways. The first way is to pick up information through the visual perception about the visuo-spatial features of objects and spatial relations among them. Such visual perception deals with the transition from sensation to perception, in which perceptual images of spatial scenes are constructed in a bottom-up fashion. The second way is to process visuo-spatial information without sensory support, derived from the top-down retrieval, or generation of virtual images that are used in the context of explicit or implicit task demands. Through the construction of mental representations, people can combine visuospatial elements in new ways, perform and simulate mental transformations on them, and engage in reasoning and problem solving. While designing, designers are involved in spatial cognition through constructing either the external or the internal representations (Bruner 1973; Tversky 2005), in which they do abstract reasoning from the representations, more specifically functional inferences related to the behaviors of entities in problem-solving tasks (Carroll et al. 1980). Each level of representation leads designers to evolve their interpretations and ideas for solutions through the execution of action and reflection (Goldschmidt and Porter 2004; Norman 1993; Schön 1992). For this research we define designers’ spatial cognition as reflective interaction between the external representation and the internal representation of the problem-solution processed by the perception and reasoning about visuo-spatial information. By 'perceiving' we mean the process of receiving and interpreting information from the representations and by ‘reasoning’, the thinking and problem-solving activity which goes beyond the information given, and which is closely related to functions of a physical artifact and space.

2.3. Creative Design Process Cognitive psychology associates ‘creative’ with certain processes that have the potential to produce ‘creative’ artifacts in designing (Gero 1992; Visser 2004). In this research, we adopt the notions of ‘S-invention’ and ‘co-evolution’ for the ‘problem-

-6-

finding’ behaviors associated with creative design process. First, Suwa et al. (Suwa et al. 2000) proposes situated invention of new design requirements (S-invention) as a key to obtaining a creative outcome. ‘S-invention’ refers to the set-up goal activities of introducing new functions as design requirements for the first time in the current task in a situated way. The introduction of new constraints captures important aspects of the given problem, going beyond a synthesis of solutions that satisfy the initially given requirements. In a similar context, Cross and Dorst (Cross and Dorst 1999) proposes that creative design can be modeled in terms of the co-evolution of problem and solution spaces. Co-evolutionary design is an approach to problem-solving in which the design requirements and solutions evolve separately, but affect each other (Maher et al. 1996). The restructuring of a problem reflects a change in the designer’s perception of a problem situation. With regards to the designers’ perception of a problem situation, Suwa et al. (Suwa et al. 2000) propose ‘unexpected discoveries of attending to implicit visuospatial features in an unexpected way’ as a key to gaining a creative outcome. Suwa and Tversky (Suwa and Tversky 2001, 2002) propose the co-generation of new conceptual thought and perceptual discoveries’ as ‘constructive perception’. Such ‘constructive perception’ allows the designer to perceive in another way, which may evoke the ‘reinterpretation’ that provides the opportunity for the designer to be more creative (Gero and Damski 1997). For the generation of ‘re-interpretations’ in external representations, Gero et al. (Gero and Damski 1997; Gero and Kelly 2005; Gero and Yan 1993; Jun and Gero 1997) emphasize the process of ‘re-representation’ producing multiple representations since it allows emergence to occur, thereby introducing new variables for the revision of design ideas and, as a consequence, leading to creative design.

2.4. Hypotheses The background study on TUIs has argued that interfaces employing manipulable physical objects have potential affordance of epistemic actions reducing cognitive loads. We may argue that TUIs support designers’ spatial cognition if the 3D modeling actions produced by TUIs can be characterized as epistemic actions while designing. In a similar way, designers’ gestures are also considered as organizing activities which reduce cognitive load. Therefore, at the Action level, we hypothesized about designers’ physical actions while using TUIs as follows; Hypothesis 1: The use of TUIs can change designers’ 3D modeling actions in designing - 3D modeling actions may be dominated by epistemic actions. Hypothesis 2: The use of TUIs can change designers’ gesture actions in designing – more gesture actions may serve as complementary functions to 3D modeling actions in assisting in designers’ cognition. The unstructured forms of pictorial representation in sketches can potentially be perceived in different ways (Purcell and Gero 1998). However, 3D spatial configuration, being dealt with in this research, does not present such ambiguous representations. Rather, the functions of objects and spaces associated with the external representations are ambiguous. We expected that the tactile interaction afforded by TUIs may stimulate

-7-

designers to attend to the dynamic spatial relationships among elements rather than single elements. Further the multiple representations produced by the TUIs may encourage designers to create new visuo-spatial features. The perception on the spatial relationships is especially functional, and this abstract relationship can be linked to more conceptual information. Therefore, at the Perception level, we hypothesized on designers’ perceptual activities while using TUIs as follows; Hypothesis 3: The use of TUIs can change certain types of designers’ perceptual activities - designers may perceive more spatial relationships between elements, and create more and attend to new visuo-spatial features through the production of multiple representations. Our research is based on the assumption that the changes in designers’ spatial cognition may affect the design process. If the ‘problem-finding’ behaviors and the process of ‘re-representation’ increase while using TUIs, we may argue that the design process is affected by the changes of designers’ spatial cognition, ultimately leading to creative design. Therefore, at the Process level, we hypothesized the effect of the changes of spatial cognition on the design processes while using TUIs as follows; Hypothesis 4: The use of TUIs can change the design process – the changes in designers’ spatial cognition may increase problem-finding behaviors and the process of ‘re-representation’, which are associated with creative designing.

3. COMPARING GUI TO TUI In order to highlight the expected changes in spatial cognition while using TUIs, we compare designers in the following two settings: A tabletop design environment with TUIs and a desktop design environment with GUIs. The use of two interfaces is the major variable in the study, while the remaining variables are set in order to facilitate the experiments but not influence the results.

3.1. Experiment Design Interfaces: 3D blocks vs. Mouse & Keyboard Based on the literature study, we decided to use 3D blocks as tangible input devices for a TUI and a mouse and keyboard as input devices for a GUI in the experiments. Among various types of 3D blocks, we adopted the same method used by Daruwala et al. (Daruwala 2004; Maher et al. 2004) because of its simplicity and relevance to our study. Multiple 3D blocks allow direct control of virtual objects as space-multiplexed input devices, each specific to a function and independently accessible. The binary patterns attached to the 3D blocks were made in ARToolKit for the display of the 3D virtual models (McCarthy and Monk 1994). In terms of GUIs, a mouse and keyboard are highly generalized time-multiplexed input devices which control different functions at different times. These two general input devices are as a baseline against which to compare the 3D blocks of a TUI. They are used to manipulate a set of GUI elements such as windows, icons and menus that reside in a virtual form (Fitzmaurice et al. 1995; Turk 1998).

-8-

(a)

(b)

(c)

Figure 1 (a) a 3D block with a pattern; (b) a ‘shelf panel’ virtual model; (c) multiple 3D blocks

Systems: Tabletop vs. Desktop An existing tabletop system was chosen as the design environment for using a TUI as opposed to a conventional desktop computer system for a GUI. The tabletop system constructed by Maher et al. (Daruwala 2004; Maher et al. 2004) is the medium in which the tangible input and output devices reside and tangible interaction takes places. Figure 2 shows the tabletop system with its horizontal and a LCD vertical display and input devices (Daruwala 2004). The vertical screen was to extend the visualization of the spatial system shown in plan view on the horizontal display surface. The desktop system is a typical desktop PC comprising a LCD screen, a mouse and keyboard. The physical control space with the mouse and keyboard are separated from the virtual output space by a vertical screen.

(a)

b)

(c)

Figure 2. Tabletop system; (a) Horizontal table (b) Vertical screen (c) 3D blocks (Daruwala 2004)

Applications: ARToolKit vs. ArchiCAD Since AR has the closest relation with TUIs, it was used as the framework for the TUI experiments. ARToolKit3 was chosen for its suitability for allowing the objects to retain their physical forms and augmenting the 3D visual outputs on the vertical display. For a GUI, we used ArchiCAD because designers are already familiar with CAD software, and ArchiCAD is a popular CAD system with typical GUI features. ARToolKit (Billinghurst et al. 2003) determines the virtual cameras’ viewpoint to detect the tracking fiducial markers using vision based tracking methods as shown in Figure 3. In order to create the database for the design tasks, 30 furniture models were selected from the library in ArchiCAD and made into the VRML models. This was done to allow the same furniture models to be used for the two different design environments.

3 ARToolKit is free AR software including tracking libraries and source codes for the libraries.

-9-

(a)

(b)

(c)

Figure 3. Diagram showing the image processing used in ARToolKit; (a) a live video image (b) a binary image; (c) a virtual overlay from http://www.fhbb.ch/hgk/af/livingroom/livingroom1/sources/ARToolKit2.33doc.pdf

ArchiCAD enables the designer to create a ‘virtual building’ with 3D structural elements like walls, doors and furniture, and provides pre-designed objects in a library. ArchiCAD can create both 2D and 3D drawings, but the designers were required to manipulate furniture only in a 3D view, thus the ability to interact with the objects in ArchiCAD was similar to that in ARToolkit. The 3D forms of virtual models of ArchiCAD allowed designers to pick them up and drag them using a mouse, and thus provided designers with a method of manipulation similar to that of the 3D blocks. The same 30 virtual furniture models used for the 3D blocks were selected.

(b)

(a)

(c)

Figure 4. ArchiCAD; (a) 2D view (b) 3D view (c) Library from http://www.graphisoft.com

Design Tasks: Home office and Design office An appropriate design problem for the experiment has to be devised carefully in order to have a manageable size of protocols (Akin 1986; Cross et al. 1996). We considered our design problem to be easily understood by architecture students and then chose a smallscale space-planning problem using furniture because this framework seemed to be the most relevant for both the 3D blocks in ARToolKit and ArchiCAD. Each 3D block represents a piece of furniture, and pre-designed furniture can be imported from the library in ArchiCAD using the mouse and keyboard. The two design tasks were developed to be similar in complexity and type as shown in Figure 5. In order to stimulate designers’ perceptual activities on 3D objects, their relationships to each other, and their location within a 3D space, we made the design tasks renovation tasks for redesigning existing studios. The goal of the home office design task was to define four required areas, sleeping, kitchen & dining, working, and living & meeting areas for renovating a residential studio into a home office for a - 10 -

computer programmer. The goal of the design office task was to define four required areas, that of the designer, the secretary, the reception and utility areas for renovating a designer’s private studio into a commercial design office for the designer and a secretary. Windows

Bathroom entrance

Bathroom

Entrance

Windows

(a)

No Glass wall

Entrance Glass wall

Figure 5. (a) 3D Home office plan and (b) 3D Design office plan

Participants This research explores how different HCI affordances may change designers’ spatial cognition using protocol analysis, so the decision on the number of designers is different from those in HCI research that generalizes basic human performance capability from a large number of designers. Designing is a high level of cognitive activity, so many empirical studies on designers’ cognition include a relatively small number of designers to seek an understanding of specific cognitive processes (Akin and Moustapha 2003; Ball 2003; McNeill 1999). Each segment of the design protocols is a data item, so our protocols contain a large number of data elements. We use fewer designers, but still have a significant amount of data to validate a quantitative analysis. The designers are 2nd or 3rd year-architecture students competent in ArchiCAD, so they have almost the same range of experience and educational backgrounds.

3.2. Experiment Set-ups The two experiment set-ups simulating TUI and GUI design environments were constructed in the same room. Each designer participated in a complete experiment, consisting of a design task in a TUI session and a second design task in a GUI session. It was anticipated that the comparison of the same designers in two different interface environments would provide a better indication of the impact of the user interfaces than using different designers and the same design task in each environment. TUI session The tabletop environment includes a horizontal table and a vertical screen to facilitate multiple views of the 3D model. 3D blocks with tracking markers are placed on the tabletop. Figure 6 shows the equipment set-up of the TUI session. A DVR system (digital video recording) was set to record two different views on one monitor. A camera is used to monitor a designer’s behavior and the other view is a video stream directly from the LCD screen. This enabled the experimenter to simultaneously observe designers’ physical actions and the corresponding changes in the external representation. One

- 11 -

(b)

microphone is fed into the DVR system and the camera is located far enough from the table to observe a designer’s gestures as well as the 3D modeling actions. DVR

LCD screen

3D blocks Microphone Camera

Figure 6. Experiment Set-up for the TUI session

GUI session Figure 7 shows the set-up for the GUI session. Instead of the horizontal table, a typical computer configuration with a vertical screen, keyboard and mouse are used. The overall experimental set-up was similar to that of the TUI session. However the GUI setting reduced the camera’s view compared to the camera angle in the TUIs session, and made it hard to include the external representation and designers’ behaviors in one shot. DVR Desktop Microphone LCD Screen Mouse & keyboard Camera

Figure 7. Experiment Set-up for the GUI session

3.3. Experiment Procedure Two pilot studies of individual designers were carried out and then with the lessons learned from the pilot studies, nine more structured experiments were conducted. Two experiments were later dropped from the study because of insufficient information from the protocols, so the final results are based on seven designers. Each designer performed one design session in one day, and went back to the other session on another day. This is to eliminate the learning effect and designers’ tiredness, which may affect the result of the second session. Each design session was completed in 30 minutes. Training In the training sessions designers were engaged in manipulating the input devices in order to review their skills in using specific features of the applications. They did a warm-up task involving thinking aloud to make sure of their capability for verbalizing their thoughts during the design session. They were instructed that the entire black markers on the 3D blocks should be in the field of view of the web camera to obtain the

- 12 -

visualization of digital models in the TUI session. For the GUI session, they were asked to work in the 3D view in ArchiCAD and instructed on how to access to the furniture library constructed for the experiments. Experiment Designers were given time to read through the design briefs prior to the beginning of the design sessions. They were asked to report as continuously as possible what they are thinking as they carry out the design tasks for about 20 minutes. They did not need to produce a final design by the end of the session because the focus of the experiments was on the design process, not the final design output. An experimenter stayed in front of the DVR system to observe the experiment process, not interfering with the designers. However the experimenter reminded designers of verbalizing their thoughts when the designers did not think aloud for over 1 minute, and answered their questions during the design sessions. Table 1 shows the outline of the experiment sessions. Table 1. Outline of the experiment sessions Interface/Application Hardware Training/ Design session Designer Design Tasks

TUI session GUI session 3D blocks/ARToolKit Mouse and keyboard/ArchiCAD Tabletop and webcam/LCD screen Desktop/LCD screen 5-10 mins/20 mins 5-10 mins/20 mins Individual 2nd or 3rd architecture student Home office or Design office renovation

There were some concerns about the validity of the settings with regards to three different conditions: application, work space and design task. First, in order to eliminate the effect of the manipulability caused by the applications, we recruited competent designers in ArchiCAD, and restricted the required functions of ArchiCAD for the tasks to simple ones. It was consequently observed that there was no significant difference in designers’ capabilities regarding the two different applications. Secondly, in order to compensate for the different work spaces between two environments, we adopt a LCD screen for the TUI session instead of HMDs. This was intended for providing a same visual modality for the designers. Thirdly, the two design tasks were carefully developed to be similar in complexity and type and to stimulate designers’ spatial cognition for 3D design. Designers had to reason about 3D objects and the spatial relationships between these objects despite the fact that they developed a 2D layout to work on the design tasks.

4. METHOD: PROTOCOL ANALYSIS Protocol analysis is a widely accepted research method that makes inferences about the cognitive processes underlying the task performance (Foreman and Gillett 1997; Gero and Mc Neill 1997; Henwood 1996). We collected concurrent protocols using the thinkaloud method. No questionnaire was used because our focus is on capturing the contents of what designers do, attend to, and say while designing, looking for their perception of discovering new spatial information and activities that create new functions in the design.

- 13 -

4.1. Protocol Analysis in Design Research A protocol is the recorded behavior of the problem solver which is usually represented in the form of sketches, notes, video or audio recordings (Akin 1986).Recent design protocol studies employ analysis of actions which provide a comprehensive picture of physical actions involved during design in addition to the verbal accounts given by subjects (Brave et al. 1999; Cross et al. 1996). In design research, two kinds of protocols are used: concurrent protocols and retrospective protocols. Generally, concurrent protocols are collected during the task and utilized when focusing on the process-oriented aspect of designing, being based on the information processing view proposed by Simon (Simon 1992). The ‘think-aloud’ technique is typically used, in which subjects are requested to verbalize their thoughts as they work on a given task (Ericsson and Simon 1993; Lloyd et al. 1995). On the other hand, retrospective protocols are collected after task and utilized when focusing on the content-oriented or cognitive aspects of design, being concerned with the notion of reflection in action proposed by Schön (Dorst and Dijkhuis 1995; Foreman and Gillett 1997; Schön 1983). Subjects are asked to remember and report their past thoughts after the task, where the videotape of their sketching activities is provided for alleviating the selective retrieval due to decay of memory (Suwa and Tversky 1997). A number of protocol studies investigated designers’ cognitive activities (Goldschmidt 1991; Kavakli and Gero 2002; Suwa et al. 1998).

4.2. Coding Scheme Our coding scheme comprises five categories at three levels of spatial cognition: 3D modeling and gesture actions at the Action level, perceptual activities at the Perception level, and set-up goal activities and co-evolution at the Process level. We selectively borrow sub-categories from the Suwa et al. (Suwa et al. 2000; Suwa et al. 1998). The Action level represents motor activities produced in using the interface. 3D modeling actions represent operations on external representation, which largely describe the ‘movement’ of 3D objects and the ‘inspection’ of representations and design briefs. Gesture actions represent designers’ movements other than 3D modeling actions. The ‘Design gesture’ code is applied to segments when designers develop their ideas via large hand-movements over the plan, and the ‘General gesture’ code is applied to segments when designers simply move their hands without a specific design intention. ‘Touch gesture’ is applied to segments when designers touch multiple 3D blocks using their hands, or digital images using the mouse, which do not accompany any change in the design. The Perception level represents how designers perceive visuo-spatial features from the external representation. Attention to an existing visuo-spatial feature, creation of and attention to a new visuo-spatial feature, and unexpected discoveries of a new visuospatial feature were investigated as a measure of designers’ perceptive abilities for spatial knowledge. For example, if a designer says “there’s a desk near the entrance”, this can be coded as attention to an existing spatial relationship, but if the designer says “I’m moving this desk to go near the window”, this can be coded as creation of a new spatial relationship. Designers sometimes discover a space unexpectedly. For example, “a little - 14 -

bit, the layout is not...you end up with empty space..!” This example suggests that during inspection the designer has noticed the unexpected appearance of an empty space. The Process level represents ‘problem-finding’ behaviors associated with creative design. Set-up goal activities refer to activities of introducing new design functions as design requirements, which restructure the design problem. In terms of the semantic mode, set-up goal activities basically belong to functional activities. If a designer considers the view from a glass wall to outside, it is still a functional activity. However, if s/he says “let’s put a display place in front of the glass wall for the view”, then this becomes an instance of set-up goal activity. The co-evolution category refers to design activity that explores cognitive movement between design problem and solution spaces. Table 2. Spatial Cognition Coding Scheme Action Level 3D modeling actions

PlaceNew PlaceExisting ReplaceExisting Rotate Remove Library InspectBrief InspectScreen InspectTable

Place a new object from the library Change the location of a initially given object for the first time Change the location of an existing object Change only the orientation of an existing object Delete/remove an existing object Check library for objects through screen or virtual library Inspect the design brief Inspect layout on the screen Inspect layout on the table

Gesture actions

Design gesture General gesture Touch gesture Modeling action Perception Level

Large hand movements above the 3D plan General speech-accompanying hand gestures Touch a 3D blocks with hands or a mouse No gesture because of the modeling actions

Perceptual activities

E-visual feature E-relation E-space E-object N-relation N-space D-visual feature D-relation D-space Process Level

Attention to an existing visual feature of an element Attention to an existing relation among elements or orientation of an element Attention to an existing location of a space Attention to an existing location of an object Creation of a new relation among elements Creation of a new space among elements Discovery of a visual feature of an element Discovery of a relation among elements Discovery of an implicit space between elements

Set-up goal activities

G-knowledge G-previous G-implicit G-brief G-repeat

Goals to introduce new functions derived from explicit knowledge or experience Goals to introduce new functions extended from a previous goal Goals to introduce new functions in a way that is implicit Goals to introduce new functions based on the given list of initial requirements Repeated goals from a previous segment

Co-evolution

P-space S-space

The features and constraints that specify required aspects of a design solution The features and behaviours of a range of design solution

- 15 -

Combined codes We combined some codes of 3D modeling action, perceptual activity and set-up goal activity into generic activity components in order to highlight observed patterns of design behaviors in the two design environments as shown in Table 3. Table 3. Combined Codes Combined Codes New Revisited Inspection Existing Creating Discovery Object Space Spatial relation S-invention Others

Individual Codes PlaceNew, PlaceExisting ReplaceExisting, Rotate InspectScreen, InspectTable E-visual feature, E-relation, E-space, E-object N-relation, N-space D-visual feature, D-relation, D-space E-visual feature, E-object, D-visual feature E-space, N-space, D-space E-relation, N-relation, D-relation G-knowledge, G-previous, G-implicit G-brief, G-repeat

Coding Categories 3D modeling actions

Perceptual activities

Set-up goal activities

New_Revisited_Inspection. ‘New’ activities refer to 3D modeling actions of importing an object from the furniture library or changing the location of a given object for the first time. When an object is re-arranged later, it is coded as ‘revisited’ activity. ‘Inspection’ activity refers to the actions of inspecting external representations. Existing_Creating_Discovery. The perceptual activity codes are combined into three generic activities: perceiving an existing visuo-spatial feature, creating a new visuospatial feature, and discovering a new visuo-spatial feature unexpectedly. The ‘Existing’ sub-category takes place in the problem space, and Creating and Discovery subcategories belong to the solution space.. Object_Space_Spatial relation. The perceptual activity codes are again combined in another way according to the focus of designers’ attention; perceiving individual objects, perceiving space, and perceiving spatial relationships among 3D objects. S-invention_Others. The codes ‘G-knowledge’, ‘G-previous’ and ‘G-implicit’ are instances of the S-invention, which refers to especially the emergence of new design issues for the first time during the design process.

4.3. Protocol Coding Segmentation A protocol study involves protocol collection, segmentation, coding and analysis. Segmentation is dividing the protocols into small units, which are to be assigned to relevant codes according to a coding scheme. The recorded data were transcribed, and then segmented along the lines of designer’s intentions or the changes of their actions. Not only the contents of the verbal protocols but video recording of the 3D modeling activity were looked at to decide the start and end of a segment. Table 4 shows the segmentations of the protocols excerpt from a TUI session, where a single segment sometimes comprises a short paragraph and sometimes several sentences.

- 16 -

Table 4. Segmentation: Intention based technique Segment Segment 21 Segment 22 Segment 23 Segment 24 Segment 25

Time 04:34-04:43 04:43-04:50 04:50-04:53 04:53-05:00 05:00-05:06

Transcripts This thing is quite tall. May be it should be moved to the corner or something. This desk fits nicely with this Just looking at the alternative desk. This is a corner desk. So move it to here…..ok….

3D modeling actions InspectScreen ReplaceExisting ReplaceExisting PlaceNew ReplaceExisting

Coding Process Transcriptions were done by native English speakers and then the segmentation was done by one of the coders. The protocol coding was done concurrently by two coders, and a final protocol coding was achieved using a process of arbitration. The coders read the transcripts and watched the video. By using INTERACT and FileMaker, they coded each segment according to the coding scheme. Each segment has a single code in 3D modeling and gesture actions, and multi-codes in perceptual, functional and set-up goal activities. After each coder finished the coding, they combined their results in a joint arbitration process in which the coders consulted the transcript, referring to the video when it was necessary to clarify the subject’s actions. When there was a disagreement each coder explained reasons for their results and by a consensus approach, an arbitrated result was achieved. Figure 8 shows an arbitrated data of Designer 1.

Figure 8. Arbitrated data of the Designer 1 in the TUI session

The reliability of the coding process was measured by calculating the Kappa values between two coders through the three coding phases (1st coder run, 2nd coder run, and arbitration run). Table 5 shows the average kappa values for each session. The Kappa values are bigger than 0.75, which means the reliability of the coding is quite high. In general the reliability of coding the Action level was higher than other levels, because physical actions are coded by inspecting what happens on the screen. Table 5. Kappa values for the three coding phases Kappa values between F&S F&A S&A TUI session 0.77 0.86 0.78 F & S: First coder’s coding and second coder’s coding F & A: First coder’s coding and arbitrated coding S & A: Second coder’s coding and arbitrated coding

Kappa values between GUI session

- 17 -

F&S 0.79

F&A 0.85

S&A 0.81

5. ANALYSIS OF DESIGNERS’ SPATIAL COGNITION Prior to going into the protocol analysis we observed how designers cognitively interacted with external representations and investigated how often they changed their intention in order to get a sense of the overall tendency of the results.

5.1. Overall Observations There were some differences between two design sessions, which are largely categorized into three aspects of designing: initial approach to tasks, development of design ideas and gestures. First, designers in the TUI sessions initially characterized the four required areas, considering design issues at a more abstract level. However, designers in the GUI sessions went into the library directly to search for the furniture. This finding suggests that designers in the TUI sessions start with the formulation of the design problem whereas designers in the GUI sessions start in the solution space. Second, designers in the TUI sessions randomly placed pieces of furniture on the horizontal display, and then decided on the locations of them while moving around the furniture. Designers of the TUI sessions seemed to develop design ideas using the information derived from perceptual activities being stimulated by modeling actions. On the other hand, designers of the GUI sessions seemed to develop design ideas based on the information initially given by the design briefs. For example, they often said that “the designer might need a big work desk” or “the programmer might need more seats”. Third, it was interesting to note that designers of the TUI sessions often kept touching the 3D blocks, and designers of the GUI sessions showed similar touching actions using the mouse while inspecting. We questioned the role of the ‘touching’ actions in assisting in cognition because ‘touching’ actions did not accompany any change in the design objects, but seemed to be involved in the cognitive processes. Table 6 shows the segment durations of design sessions, which give us an idea about how frequently the designers’ intentions changed over the timeline of the activity due to the segmentation technique employed for this research. The average segment duration of the TUI sessions (10.6 sec) is shorter than that of the GUI sessions (17.9 sec), which suggests that designers in the TUI sessions started new actions quicker and generally had more actions in the same amount of time. The total time of each GUI session was cut at a same time point when the corresponding TUI session was completed. Each designer engaged in two sessions in varied order, in each of which the designer was provided with different design tasks in order to eliminate the learning effects. Table 6. Intention shifts in designers’ behaviors: duration of segments Session Design task Task completion

Total time Segment no Mean (sec) Std. Deviation

Designer 1

Designer 2

Designer 3

Designer 4

Designer 5

Designer 6

Designer 7

TUI1 GUI2 B A Yes Yes 19 min 133 80 8.54 14.22 5.52 9.87

TUI2 GUI1 B A Yes No 15 min 89 66 10.21 13.14 9.20 8.37

TUI1 GUI2 A B Yes No 20 min 120 81 9.88 14.59 7.25 16.43

TUI2 GUI1 A B Yes No 18 min 93 55 11.39 18.78 8.16 11.62

TUI2 GUI1 B A Yes No 17 min 83 39 12.01 24.52 8.44 14.60

TUI1 GUI2 B A Yes No 19 min 99 61 11.38 18.41 7.30 15.09

TUI2 GUI1 A B Yes No 11 min 62 31 11.00 21.93 6.33 11.81

Session: 1– first session; 2 – second session / Design task: A - Home office; B – Design office

- 18 -

5.2. Analysis of the Three Levels of Designers’ Spatial Cognition Considering the findings in the observation and initial investigation we analyzed the coded protocols using both statistical and graphical approaches. For the statistical analysis, we performed a Mann-Whitney U test on each category of encoded protocols to examine for significant differences in occurrence of or time spent on cognitive activities. To further measure the differences affected by the interfaces, we explored the structures of designers’ behavior visually through the graphs. Similar patterns were found for all designers, so designer 1’s behavior patterns are demonstrated as an example. This section presents the results of the protocol analysis at each level, and discusses the implications for designers’ spatial cognition. Action Level 3D Modeling Actions Table 7 shows the average occurrence of the combined codes of 3D modeling actions in TUI versus GUI sessions. There are significant differences in the occurrence of ‘New’ (Z=-2.888, N=7, p