Expressive interaction for conceptual design of shape An empirical study on methods and their effectiveness. T. Wiegers, G. Kuczogi and I. Horváth Delft University of Technology Subfaculty of Industrial Design Engineering Jaffalaan 9, NL 2628 BX Delft, The Netherlands
[email protected]
Keywords: HCI (Human-Computer Interaction), Shape expression, CACD (Computer-Aided Conceptual Design), Interaction methods, Empirical study.
Abstract This paper describes a study of human interaction methods to communicate shape ideas. The aim of the study is to get insight in the way people exteriorize shape if they are not bound to the limitations of certain CAD systems. The study is started because current CAD systems generally are not able to support conceptual design in a convenient way. Instead of forcing the designer to speak the language of CAD systems, our goal is to build a system for Computer Aided Conceptual Design (CACD) that is able to interpret a way of interaction that connects closer to the methods designers normally use to express shape. We launched an empirical study to find out how people exteriorize shape, and how effective they do this. The goals of the study are (1) to explore the knowledge that is needed to implement a more effective user interface; (2) to offer opportunities for the designers to express shape concepts in a more natural way; and (3) to find a link between the thinking units of the designer and the modeling units of the CACD system. The gained insights will be used in a later phase to improve humancomputer interaction for shape conceptualization.
Introduction Many computer tools exist for the design of shape, however for the conceptual phase of design they are not very suited (Hennessey, 1993). To offer effective support for shape conceptualization, the tools should satisfy the needs of the designer. To meet this, a research project is started to the working methods of the designer during shape conceptualization (Wiegers, 1999). It focuses on the way a shape of a product is brought outside (exteriorized) when ideated by a designer. Which activities does the conceptualizing designer perform, which support is required and which support is possible. How can we take care of a smooth interaction between the designer and the computer, how
do we know what de designer wants and how can we stimulate his or her creative activities. This is a number of questions that this research wants to address. The results will be used to design an improved method of human-computer interaction (HCI) for a system for computer-aided conceptual design (CACD). The research at hand aims to capture how a designer brings an imagined shape to the external world. In other words, exteriorization of shape images. We will do this by observing designers who express shape in their traditional way. The assumption is that during the interaction the computer should adapt to the working methods of the designer as much as possible. The designer should be prevented from having to learn a special language and having to use special methods by which his attention is diverted from his main task, namely the ideation of a shape. It is an advantage if the designer can, for HCI, use the interaction skills and habits he already masters. Then the HCI need not hinder the shape conceptualization. Of course, also new interaction methods can be developed that offer a number of advantages that pay the effort to learn new methods. The first experiments, however, direct to the methods designers already apply. The study should provide insight in effective methods that designers apply in shape exteriorization. The insight will be used in a future phase of the research to introduce more effective HCI methods for shape conceptualization. The aim of the observations is not just to copy the observed interaction methods to a computer environment. The situation in which a designer communicates with a colleague designer is different from the situation in which he communicates with a CACD system. A CACD system has different possibilities and limitations. That will be reflected in the method of interaction.
With the gained insight the interaction method will be adapted to the skills of the designer, so that he has to pay less attention to the way he brings the message and spend more attention to the contents of the message. The new interaction method should enable the designer to work more comfortable. He will, for example, feel less hampered than with the use of a nowadays CAD system and with a traditional input device like a mouse.
1
2
3
4
5
6
Method In the experiment two subjects, A en B, play a role. They jointly do four tests. during each test A has to explain a shape to B. The difference between the tests is in the methods A is allowed to use. B has to make a sketch of the shape, as understood by him. The sketches of B are evaluated by a panel. The experiment is performed 12 times, in 12 sessions. Each session is performed with a different couple of subjects. In total, 12 couples of subject performed 48 test, see Table 1.
Figure 1. The shapes A had to exteriorize Test 1 - All methods are allowed. Test 2 – Sketching and speech. Test 3 - Gesturing and speech. Test 4 - Physical objects and speech (Figure 2).
Table 1. Schema of subjects, sessions and tests. Session Subjects Type Allowed methods Test number of test Speech Sketch Gesture Physical number object 1 A1, B1 1 1 • 2 2 • • 3 3 • • 4 4 • • 2 A2, B2 1 5 • 2 6 • • 3 7 • • 4 8 • • 3 A3, B3 1 9 • : : : : : : : :
: 12
: A12, B12
: 1 2 3 4
: • • • •
: •
:
•
:
•
: 45 46 47 48
In the experiment we can distinghuis three parts: the exteriorization by A, de interpretation by B, and the evaluation by the panel. The exteriorization by A A shape is shown to A (Figure 1). He has to express this shape to B. B did not see the shape. He is in another room and can hear and see A through a one-way video connection. When A did see the shape well, the shape is hidden again. A must then express the shape from his memory. The time A needs for the exteriorization is recorded. We distinguish four methods of exteriorization: speech, gestures, sketching and the use of physical objects. The methods A is allowed to use are, per test, the following.
Figure 2. Physical objects available to A. The interpretation by B After A finishes the exteriorization, B starts making a sketch of the shape as understood by him. (Figure 3). After B finished his sketch, he got a sheet with photographs of four shapes. One of those shapes is the one expressed by A. B has to indicate which of the four shapes he thinks it is. The sketch of B and his choice from the list are used as a measure for the effectivity of the shape communication. We should realize that the effectivity of the shape communication depends both on the quality of the exteriorization of A and of the interpretation by B. The evaluation by a panel The sketches made by B were judged by a panel of 5 independent evaluators. The evaluation took two rounds. In the first round the evaluators compared the sketches to the shapes that were used in the tests. Each evaluator was shown all sketches of all subjects B.
Results First this section gives a general overview of the test data and of the scores of B’s sketches. Next the results are analyzed in relation to the allowed methods, to the shown shapes and to the subjects. Then we take a look to the activities A performed and to the expression units A used. Table 2. Allowed methods, shown shapes and times used for exteriorization.
Figure 3. Some examples of sketches made by B. The evaluators had to say which shape they thought the sketch depicts. This round was to evaluate whether the sketched shapes could be identified. In the second round the sketches were shown again, but now with an indication which shape the sketch aimed to express. The evaluators judged in how far the sketch gave a good impression of the intended shape. The possible scores range from 0 to 10 (0 = no similarity, 10 = outstanding). Of course, A was shown a different shape each test. Six different shapes were used in total. These shapes were spread among the different tests of the experiment in such a way that not the same shape was used for the same exteriorization method each time. During shape conceptualization a designer can exteriorize a shape that exists only in his mind. Because of that, in the experiment A had to exteriorize the shape from his mind, without having the shape in front of him. This is still not exactly the same as exteriorizing a shape that the designer is ideating. However, when we let A ideate a shape, the individual shapes can be very different. It would be difficult to determine in how far the results depend on the differences between the individual shapes. We have shown existing shapes to A. We selected shapes that are realistic for industrial design processes (Figure 1).
Test Subject Allowed team methods *) 1 1 All 2 1 Sketch 3 1 Gesture 4 1 Phys Obj 5 2 All 6 2 Sketch 7 2 Gesture 8 2 Phys Obj 9 3 All 10 3 Sketch 11 3 Gesture 12 3 Phys Obj 13 4 All 14 4 Sketch 15 4 Gesture 16 4 Phys Obj 17 5 All 18 5 Sketch 19 5 Gesture 20 5 Phys Obj 21 6 All 22 6 Sketch 23 6 Gesture 24 6 Phys Obj 25 7 All 26 7 Sketch 27 7 Gesture 28 7 Phys Obj 29 8 All 30 8 Sketch 31 8 Gesture 32 8 Phys Obj 33 9 All 34 9 Sketch 35 9 Gesture 36 9 Phys Obj 37 10 All 38 10 Phys Obj 39 10 Sketch 40 10 Gesture 41 11 All 42 11 Sketch 43 11 Gesture 44 11 Phys Obj 45 12 All 46 12 Sketch 47 12 Gesture 48 12 Phys Obj
Shape Right shown guess 4 1 2 1 6 1 1 1 2 1 4 1 5 0 6 0 5 0 6 0 1 1 3 1 1 1 3 1 2 1 5 1 1 1 2 1 3 1 4 1 2 1 4 1 3 1 1 1 4 0 6 0 5 0 1 0 1 1 4 1 2 1 3 1 4 1 6 1 2 1 5 1 2 1 4 0 5 1 6 1 3 0 5 1 1 1 2 1 6 1 1 1 3 1 4 1
*) Speech was allowed in all tests.
Duration (h:mm:ss) 0:01:21 0:00:50 0:00:35 0:00:44 0:00:48 0:01:05 0:00:12 0:00:26 0:00:39 0:00:40 0:00:25 0:00:49 0:00:47 0:01:33 0:01:02 0:00:33 0:00:28 0:00:18 0:00:13 0:00:29 0:00:30 0:00:45 0:00:48 0:00:37 0:00:23 0:00:27 0:00:15 0:00:22 0:00:30 0:00:58 0:00:53 0:01:03 0:00:42 0:00:47 0:00:55 0:00:51 0:00:33 0:00:12 0:00:16 0:00:12 0:00:40 0:00:36 0:00:29 0:00:53 0:00:29 0:00:45 0:00:42 0:00:25
• Overview of the test data Table 2 gives an overview of the results of the 48 tests. The table shows per test the allowed exteriorization methods, the shown shapes, the time A spend on exteriorization, and whether B guessed the right shape. A number of sketches of the subjects B are shown in Figure 3. The scores, given by the five evaluators, are normalized and listed in Table 3. The normalized values can better be compared to each other. The values range from –4.2 to +2.2. The time A spent for the exteriorization varied from 12 to 93 seconds. In 38 out of the 48 tests, B selected the right shape from the list. The evaluators identified the intended shape in 37 out of the 48 cases. This is an average value. The individual values ranged from 33 to 38. • Analyses of the results First, we take a look at the scores from Table 3. The average scores per exteriorization method are calculated and plotted in. Figure 4.
0,6 0,4 0,24 0,2 0,0 -0,2
0,00
-0,14
-0,09
-0,4 -0,6
Figure 4. The average sores of the different methods. The figure shows also the standard deviation of each method. The averages plus and minus the standard are depicted as vertical lines. There is an area from –0.04 to +0.13 in which they all overlap. Because of this, we will also look to the data in different ways.
Table 3. Normalized scores of the five evaluators given for the sketches made by subject B. Evaluator Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 Min Max
HE
HA
JA
JO
ER
-1.42 0.74 0.74 -0.13 1.17 -0.13 -2.71 -0.56 -1.42 0.74 -0.13 -0.13 0.31 -0.13 1.17 -0.13 -0.13 -0.13 -0.13 -0.56 0.74 -0.13 -1.85 -0.13 -0.13 0.31 0.31 -0.13 1.17 0.74 0.74 0.74 1.17 -0.13 -0.56 -2.28 0.31 0.74 -0.13 -0.13 1.60 1.60 -2.71 0.31 0.74 0.31 -1.42 1.17 -2.71 1.60
0.51 1.61 0.88 0.88 0.88 -0.59 0.14 -0.59 -2.05 0.88 1.61 0.14 0.51 -0.59 0.14 -0.22 0.14 -0.59 -1.32 -0.59 -4.24 0.88 -0.59 -0.59 0.14 -0.59 -0.59 0.14 1.24 1.24 0.88 0.51 -0.59 -0.59 0.14 0.14 0.88 -0.59 -0.59 -1.32 -0.59 0.51 -0.59 0.88 0.14 0.14 0.88 0.88 -4.24 1.61
-0.14 0.65 1.91 1.28 0.96 -0.77 -0.61 -0.92 -1.24 0.02 -0.30 1.91 1.28 0.96 1.91 0.33 0.02 -0.30 -1.24 -0.61 -1.24 -0.92 -0.92 0.65 -0.92 0.02 0.65 -0.92 -0.30 -0.92 0.02 1.91 0.02 -1.24 -0.92 -0.61 -1.24 -1.24 -0.92 -0.92 0.02 -0.92 1.28 0.02 0.65 1.59 0.96 1.28 -1.24 1.91
-0.78 1.98 0.87 -0.23 0.87 0.32 -1.33 -1.33 -0.78 -0.78 -0.23 1.43 0.32 0.87 0.32 -0.23 -0.78 0.32 -0.78 -0.78 1.98 -0.78 -1.33 1.43 -1.33 -0.23 -1.33 -0.78 1.43 -0.23 1.43 1.43 -0.78 -0.23 0.32 -1.33 1.43 0.32 -0.23 0.32 -0.78 1.43 -1.89 -0.23 -0.23 1.43 -0.23 -0.23 -1.89 1.98
-1.44 1.25 1.25 -0.99 0.80 0.36 -0.09 -0.99 -0.99 1.25 -0.09 -0.09 1.70 1.70 1.70 1.25 -0.54 0.36 -0.54 -1.44 1.25 -0.09 -0.99 -1.89 -1.44 -0.09 -1.89 -0.99 -0.54 0.36 0.80 -0.54 -0.54 0.36 -0.09 -1.44 -0.09 -0.09 0.80 -0.09 -0.09 0.36 -0.99 -0.09 2.15 0.80 -0.09 0.80 -1.89 2.15
Average
-0.65 1.24 1.13 0.16 0.94 -0.16 -0.92 -0.88 -1.30 0.42 0.17 0.65 0.82 0.57 1.05 0.20 -0.26 -0.07 -0.80 -0.80 -0.30 -0.21 -1.14 -0.11 -0.74 -0.12 -0.57 -0.54 0.60 0.24 0.77 0.81 -0.14 -0.37 -0.22 -1.10 0.25 -0.17 -0.21 -0.43 0.03 0.59 -0.98 0.18 0.69 0.85 0.02 0.78 -4.24 0.80
A second approach is to look how the different methods behave per subject. If the scores of a single session are compared, the results are not influenced by the fact that one subject is better skilled than another. For each session, we compared the tests in which not all methods were allowed. Table 4 shows which methods score best and which ones scored worst. The table shows that sketching & speech was the most effective combination in 7 out of 12 cases. Gesturing & speech was the least effective combination in 7 out of 12 cases (if session 5 is included). Table 4. Most least effective method in each session. S = Sketching G = Gesturing & speech P = Physical object & speech G/P = G and P have equal scores Session
Most effective
Least effective
1 2 3 4 5 6 7 8 9 10 11 12
S S P G S P S P G S S S
P G G P G/P G G S P P G G
• Activities of subject A Some tests were analyzed in more detail, to investigate which activities A performed. The steps within a typical test are described below. 1. Mention a known shape A starts from a shape that is well-known by her or him, and, as A assumes, also by B. This is a quick way to explain the global shape and to reduce the complexity of the explanation. A expresses the shape verbally, often including terms with an emotional value, e.g. a “clumsy shoe” or “a dumb puppy”. Sometimes, A applies more abstract terms, like “the extrusion of an ellipse”. Sometimes A shows the known shape by taking a physical object in hand. 2. Explain the differences As soon as a known shape is communicated, A starts to explain in which it is different from the intended shape. Typically, this explanation starts verbally and later some gestures are added. The verbal start is sometimes an introduction to and an explanation of a sketch that is made, or of a physical object that is used. Often, the spoken story is sufficient to understand the intended
shape. Sometimes a sketch is sufficient. However, even then, the sketch is accompanied with spoken explanation. Gestures were never sufficient. They described locations and contours, but the relations with the previous expressions were verbally explained. For the expression of differences between shapes, relative terms were used like thicker, smoother, the opposite end. These terms were often expressed by words, but sometimes they were expressed by making gestures or by sketching contours. Terms that express a location were also frequently used, e.g. the ends, edges, center or front side. 3. Describe details The next step of A is to give a detailed description of a specific shape feature. The location of the feature is denoted and its shape is expressed. 4. Summarize Then the description of the shape is summarized. The summary lets the different pieces fell into their places. At the same time, the summary is a check whether all essential shape characteristics are expressed. 5. Finish Finally, after the summary, necessary additions, corrections and clarifications are added. • Expression units During the above activities of A, we identified the following types of shape expression units. Speech In most test speech dominated as a method to express shape. Often speech and another way of expression complemented each others. Metaphors were used for shapes that were associated with existing objects. Some examples are: egg, potato, slipper, little round car, the head of a dumb puppy, globefish, nose, bread, roll, hat of a bishop, brick , clumsy snow boot, shoveling disk, hand grenade, soap, shoe, almond, boulder and rugby ball. Several adjectives were used that have an emotional value, like clumsy and awkward. Verbs were also used to indicate shape, e.g. swung over, shifted into each other, removed with a big ice spoon, take a bit of, extrude, cut off, and scooped out. Verbal descriptions were often accompanied by gestures. Finally, manipulations were referred to for indicating roundings, edges and holes. Also geometric terms were used to indicate curves, surfaces and volumes. Gesture Gestures often accompanied speech. At the beginning, when the global shape was expressed, gestures generally started before the verbal expression emerged. When later a geometric description was found, in several
cases gestures were used at the same time or after the verbal description. Gestures can give much freedom to a person who expresses shape, because they shift the problem of solving ambiguities to the receiver, whether this is a human or a computer. Instantaneous (static) gestures were used as well as dynamic ones (sweeps). In many cases, the shape expressed by a gesture, was also verbally explained. It seems often the gestures are more important for the person who is exteriorizing the shape, than for the person who has to interpret the exteriorization This corresponds with the observations of Kauffmann and Breebaart (2000). Some gestures were used as a kind of sketching, whether in the air, on the desk top or on the surface of an object in hand. These type of sketches were almost always two dimensional. Sketches Sketches were generally used in combination with speech. The different quality of the sketches that were used for the shape exteriorization was not reflected in the results of the observing subject. Physical objects These were used as a reference base to be able to express the locations of local shape elements, and to solve ambiguities in spoken shape expressions. For these purposes, the physical object and the intended shape need not be very similar. Physical objects were also used to specify specific shape details. Similarity was important in these cases.
Discussion Speech was used in all tests. The dominance of speech was also found by van der Vorst and Veldhuizen (2000). Speech and manual activities were often performed simultaneously. This reduces the need of switching devices or input modes. The designer can just continue to express the intended shape. Sketches were always combined with verbal explanation. This can explain why the sketch of B, who did not see the shape, several times were better than the sketch of A. There were gestures which help the designer to prepare the exteriorization, without being necessary for the receiver to understand the shape expression. Apparently, gestures can help the exteriorization process. Other gestures did really explain shape. To this category belong deictic gestures. Gestures which point to a position, to make clear to which location a specific description applies. The specific description may be a spoken shape characterization, a detail sketch, a physical object as a shape example, or another gesture. Shape was also expressed by gestures which sketch a
shape in the air or on a surface. Sketching in the air is used for global shape description. Since these sketches were nearly always two dimensional, several technologies to capture them are available today. Not the technology is the bottle neck, but the degree of integration into the conceptualization process of the designer. The effectivity of HCI can be improved if the designer gets more freedom to sketch; not only on a sheet of paper or a tablet on his desktop, but also on an object he has in his hand, or with one or two fingers in the air. The designer's skill and the type of shape influence the effectivity of gestures. Though gestures which express shape can be effective, in many cases the accompanying speech was sufficient to understand the communicated shape. However, gesturing without speech did not occur. According to Weimar and Ganapathy (1989) gesturing alone is not adequate. For HCI, in these cases, it is more important to allow this type of gestures than to understand them. This means that HCI systems should enable the designer to use both hands for gesturing, whether these gesturing is captured, or not. In the mean time, shape details can be specified by speech. About physical objects something similar can be said. The most important is to enable the designer to use physical objects. To grab them, take them in their hands and speak about them. They can also be used as a reference, e.g. on which a finger sketch is made to point out a shape detail. Making a precise scan of a physical object may be a time consuming task. However, in conceptual shape design this is often not necessary. Subjects, for example, used objects to explain on which side a certain shape feature was located.
Conclusion and future research This section gives a summary of the observations, then it lists recommendations for HCI, and finally, it discusses directions for future research. Summary of observations 1. Speech was used in all tests. It was the most frequently used method of shape exteriorization. 2. Designers can effectively exteriorize shape by using speech and sketching. 3. Gesturing often stimulates the process of exteriorization. 4. Gestures were often accompanied with words containing the same information. 5. When gestures provided extra information, it was about the location of a shape feature, or about the course of a curve or surface. 6. Physical objects functioned as a reference to indicate specific locations or shape features.
• Recommendations for HCI Interaction methods can be made more effective when they are adapted to the skills of the designer. Then the designer does not need to pay much attention to the method of interaction, but he can focus on the contents of the interaction. Speech recognition If speech recognition is supported, the hands are free for other activities Sketch input Sketching should not be limited to the desktop surface. Designers must be able to sketch in a plane they prefer, or in the plane that can best present a surface or crosssection of the artifact. Gesturing Whether a CAD system is able to recognize gestures or not, the designer should be free to gesture. An input device should not hinder the designer to use both hands freely. Gestures that indicate a specific location should be recognized. Physical objects. Designers should be free to take a physical object in hand and to indicate a specific location or contour. • Directions for future research More research is necessary to investigate which terms should be recognized to effectively support the designer by speech recognition. A device must be developed that does not hinder the designer in making gestures and grabbing physical objects. A possible implementation could be a small device that can be held in one hand while the other hand holds a physical object, or a pen for digital sketching. The device has flat, sensitive surfaces that enable sketching and pointing. The blockshaped device can function to represent an ideated shape, as if the user has the shape in hand. A regular monitor screen and a tracker are sufficient to display the ideated shape. The block could be enhanced to facilitate multiple types of use. If it is deformable, the designer can use it to bend the object and press or squeeze it. Devices with similar functionalities and based on similar ideas are already proposed by e.g. Sachs e.a. (1991) and Tovey (2000).
References Hennessey JM, "Computerized conceptualizing in industrial design - The IDEATE research projects", Interface '93, 1993, pp. 112-117. Kauffmann C, Breebaart L, "Designer's experiences of the use of computer tools for shape
conceptualization", June 2000, Technical DE123, Delft University of Technology.
report
Sachs E, Roberts A, Stoops D, “3-Draw: A Tool for Designing Shapes,” IEEE Computer Graphics & Applications, Nov. 1991, pp. 18-26. Tovey M, "Concept design CAD for the automotive industry", Proc. TMCE'2000, Horváth I, Medland AJ, Vergeest JSM (Eds), April 18-21, 2000, Delft, The Netherlands, DelftUniversity Press, Delft, pp. 9-18. van der Vorst R, Veldhuizen C, "Shape complexity and the method of expression", May 2000, Technical report DE112, Delft University of Technology. Weimer D,. Ganapathy SK, “A synthetic visual environment with hand gesturing and voice input”, Proc. ACM CHI’89 Conference on Human Factors in Computing systems, 1989, pp. 235-240. Wiegers T, Horváth I, Vergeest JSM, Opiyo EZ and Kuczogi G, "Requirements for highly interactive system interfaces to support conceptual design". Proc. CIRP'1999, University of Twente, Enschede, The Netherlands, Kals H, van Houten F (Eds.), Kluwer Academic Publishers, Dordrecht, 1999, p. 69-78.
Correspondence: T. Wiegers, Jaffalaan 9, NL 2628 BX Delft, The Netherlands. E-mail
[email protected] Fax: +31 15 278 1839
