~ CHI’92
May3-7, 1992
AN INTERFACE FOR INTERACTIVE SPATIAL REASONING AND VISUALIZATION James R. Osborn and Alice M. Agogino
Mechanical Engineering Department University of California at Berkeley Berkeley, California 94720 (510) 548-8464 (510) 642-6450
[email protected] .edu
[email protected] ●
●
students to improve their spatial reasoning abilities in a nattmd, intuitive, and self-motivating way.
ABSTRACT An interface for software that creates a natural environment for engineering graphics students to improve their spatial reasoning and 3D visualization skills is described. The skills of interest involve spatial transformations and rotations, specifically those skills that engineers use to reason about 3D objects based on 2D representations. The software uses an intuitive and interactive interface allowing direet manipulation of objects. Animation capability is provided to demonstrate the relationship between arbitrary positions of an object and standard orthographic views. A second skill of interest requires visualization of a cuttingplane intersection of an object. An interface is developed which allows intuitive positioning of the cutting-plane utilizing the metaphor of a “pool of water” in which the object is partially submerged. The surface of the water represents the cutting plane. Adjustment of the pool depth combined with direct manipulation of the object provides for arbitrary positioning of the cutting-plane. Subjective evaluation of the software thus far indicates that students enjoy using it and find it helpful. A formal testing plan to objectively evaluate the software and interface design is underway.
SPATIAL REASONINGTASKS AND SKILLS Why is spatial reasoning necessary within many fields of engineering including mechanical and civil? The obvious answer is that these fields involve the design of 3D structures which are often interrelated in complicated ways. A more relevant answer from the perspective of a student in an introductory course in engineering graphics is that engineers often must abstract 3D information and relationships from 2D representations of objects. Until recently, perhaps, almost all engineering design information was conveyed in the form of descriptions and 2D drawings on paper. The chosen 2D views are usually a standard set of orthographic views, which taken together, should unambiguously specify the 3D objects being represented. With the growing use of computers to store engineering design information, it is now common for engineers to work directly with 3D representations of objects. Nevertheless, these representations are still typically displayed on a 2D medium, the computer screen. Therefore, even if engineers use computers exclusively for representing design information, they must still be able to form a mental connection between the 2D representations that they are working with and the true 3D nature and relationships of the objects being represented.
KEYWORDS: spatial reasoning, three dimensional visualization, direct manipulation, engineering graphics INTRODUCTION Spatial reasoning is a mental process that involves thinking about relationships between three-dimensional (3D) objects. In the context of mechanical engineering, spatial reasoning frequently involves the skill of internally representing 3D objects deseribed by two-dimensional (2D) representations displayed on paper or on a computer screen. However, many individuals drop out of engineering programs because of a lack of confidence in their abilities when confronted with the traditional presentation of introductory engineering graphics material such as descriptive geometry. The purpose of this development effort is to create interactive software that incorporates a user interface allowing these
Cognitive Science Literature Some basic strategies have been identified that individuals typically use when solving spatial problems [3]. Consider a problem in which an individual must determine if several orthographic views are consistent with an isometric view. This type of problem is a task which involves the mapping between a 3D representation and several 2D representations of art object (Figure 1). One possible strategy to solve this type of problem is to form a mental image of the object based on the given 3D representation of the object, then mentally rotate that internal representation into the corresponding orthographic positions, and finally test to see if it matches the given 2D views [8]. Another strategy could be to mentally compare
Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. Q
1992
ACM
0-89791-513-5/92/0005-0075
1.50
75
May3-7, 1992
~ [HI ’92
feedback. Computers are able to display moving images that capture the user’s imagination and thereby create a “fun” environment for self-motivated improvement in spatial reasoning skills. The modern personal computer offers an effective platform for the creation of interactive graphics software that can be used to tap into a user’s curiosity. Usage of this type of software could naturally enhance the connection between 2D representations of an object and its true 3D nature.
D & ❑ cdl TOP
FRONT
SIDE
Several computer based approaches have been suggested to improve spatial reasoning abilities. One of these approaches utilizes solid modelling software as an environment to teach engineering graphics in a context which simultaneously enhances spatial reasoning and visualization skills. However, this approach provides less immediate feedback since considerable training is required before a student can begin to work with objects.
ISOMETRIC
Figure 1: 3D to 2D Mapping
edges and edge intersections to procedurally test if the gi{en isometric view is consistent with the given 2D views. As one might guess, the mental rotation strategy produces faster responses for individuals who are capable of using it. The procedural technique tends to be more accurate but is less efficient particularly for complicated objects. Evidence suggests that high aptitude spatial reasoners switch between both strategies flexibly to adapt to changes in processing demands and structural characteristics of individual problems [3].
This development effort is primarily concerned with creating an interactive environment that students of all ability levels can use immediately, without training, to improve spatial reasoning and visualization skills. Usage of this software does not preclude any other approaches to engineering graphics instruction and should be used in parallel with whatever methods are taught including the more traditional approaches. In order to make the software immediately interactive, students should work with preconstructed relatively simple objects. By keeping the chosen objects simple and using a simple internal software scheme, such as polygonal facets, to render the objects, this type of interactive nature with direct manipulation can be achieved.
Problems involving the opposite mapping are also important. In this case, the task involves the interpretation of a set of 2D orthographic representations of an object to determine if they are consistent with a 3D representation of the object. This skill is important to engineers who must understand 3D information represented with 2D drawings. In any case, visualization skills help individuals solve spatial reasoning problems by allowing them to create accurate internal spatial representations of objects.
DEVELOPMENT GOALS AND SPECIFICATIONS The goal of the software is to improve the user’s 3D to 2D mapping and 3D visualization skills. These skills are impofint in many aspects of engineering, but the primary focus of this software should be to improve the student’s abilities at interpreting 2D representations such as engineering drawings. The standard orthographic top, front, and side views are fundamental to engineering drawing representations. Therefore, the software should give the user experience with these views and their relationship to the object being represented.
Improving Spatial Abilities The renovation of the engineering graphics curriculum is the focus of many current research efforts. Much of the motivation for this focus is based on the assumption that 3D solid modelling tools or other computer-aided tools will dominate engineering design in the future [1, 7]. Recent studies have shown a broad diversity in spatial abilities and learning styles [5]. Thus, regardless of the mode of instruction of engineering graphics, there is a need to improve general spatial reasoning and visualization skills. The premise of this software development effort is that spatial reasoning and visualization abilities can be improved by means of interactive courseware to build intuition and successful reasoning strategies.
The view of an object from any arbitrary position is essentially a process of projection. The second fundamental goal of the software is to improve the user’s understanding and experience with the concept of cutting-plane intersections. This type of experience can be very difficult to demonstrate dynamically with physical models. Since there are few everyday experiences which directly correspond to cutting-plane intersections, special attention must be made to ensure that the cutting-plane interface is not confusing.
Interactive Spatial Reasoning Software The foregoing discussion indicates that it is desirable to, enhance spatial reasoning abilities by increasing an individual’s ability to form internal 3D representations of objects given 2D representations. Why use computers to accomplish this? Computers offer capabilities that are difficult to implement with paper exercises or static physical models. These capabilities include the possibility of a structured or guided interactive nature with dynamic
One final goal that should be stated explicitly is that the software should be as intuitive and interactive as possible. This will ensure a positive and self-motivating result.
76
~ (HI ’92
I
May3-7, 1992
Ideally, the software should satisfy all of these gords and require no training, no manuals, and no keyboard input.
For both of these two controllers, the user can manipulate the object about any axis in the plane of the screen (XY control or rolling) by clicking inside of the circle and holding the mouse button down. The axis about which the object is rotated lies in the plane of the screen and is Peqmdicular to the path that the mouse is dragged. The user can manipulate the object about an axis peqxmdicular to the plane of the screen (Z control or spinning) by clicking outside of the circle and holding the mouse button down. The difference between the two controllers is that the “Virtual Sphere” allows some spin control within the circle as well as outside of it.
~ Based on these goals, the following software specifications emerged 1.
Provide an environment which allows the user to interactively explore any arbitrary position of a given object using direct manipulation.
2.
Provide the capability to demonstrate standard orthographic and isometric views with animation.
3.
Provide a cutting-plane mode which allows interactive arbh.rary positioning of the cutting-plane relative to the object and displays the resulting cutting-plane intersection.
4.
Provide all functionality of the software with a singlebutton mouse as the only required input device.
Though the subjects thought these two controllers were the most intuitive of the five controllers they tried, they felt the action was still somewhat non-intuitive since it required the artificial boundary created by the circle to discriminate the two dominate modes of operation for each controller. Brainstorming yielded the suggestion that the boundary of the object itself could be used instead of the artificial circle (Figure 3). The final control strategy is essentially equivalent to the “Continuous XY with Additional Z“ controller but uses the object boundary to discriminate the two modes of operation instead of an added circle, The resulting action is very intuitive and a user requires very little practice to understand how to directly manipulate an object.
SOFTWARE DEVELOPMENT AND RESULTS Direct Object Manipulation The design of a user interface control strategy for direct manipulation of 3D images was of primary importance early in the software development. Fortunately, research which explores various methods for manipulating 3D images with 2D controls was available. In Chen et al. [2], several controllers are presented for manipulation of 3D images with a mouse. A software demonstration of five different types of controllers described in this paper was used to subjectively evaluate the “intuitiveness” of each.
d
OUTSIDE
INSIDE
Ten subjects who participated in this evaluation favored the two controllers called the “Continuous XY with Additioml Z“ controller and the “Virtual Sphere” controller. Both of these controllers define two regions within which mouse clicks are interpreted differently. The two regions are delimited by a circle which is circumscribed about the displayed object (Figure 2). The type of manipulation that occurs is defined by which region that the user clicks in. The other three controllers use more abstract metaphors for manipulation and were not considered further.
I
Figure 3: New Controller Boundary
Mouse Pointer Feedback The capability of changing mouse pointer shape can be used to give added visual cues about the active regions of the interface and the behavior of the software (Figure 4). The normal “arrow” pointer (used for selecting menus, dragging windows, and clicking in buttons) is changed to a “hand” shape to indicate when direct manipulation is possible. When direct manipulation is initiated, the pointer changes to a “crossing-arrows” shape for rolling manipulation and to a “spinning-arrows” shape for spin manipulation. When the cutting-plane depth is being adjusted (described below), the pointer changes to a “grabbing-hand” shape.
OUTSIDE
INSIDE
ef9+@cJ
@
Figure 4: Arrow, Hand, Crossing-Arrows, Spinning-Arrows, and Grabbing-Hand Pointers
Figure 2: Chen et al. Controller Boundary
77
May3-7, 1992 ‘“
CHI’92 Display Options Color was used as one display option to create a more realistic “solid-model” representation of objects. The usage of color is optional and secondary, however, and thus should not present any comprehension problems to colorblind users [6]. Hidden-line and wire-frame display modes in black and white are also provided. The availability of these three display modes allows the user to choose the particular representation most helpful and revealing for the particular object being viewed.
can choose to see the object animated into positions which correspond to the standard orthographic top, front, and side views. The object can also be returned to the isometric reference position or back into the last position that the object was in prior to directly manipulating or animating it. Finally, the object can be placed in an arbitrary position This is provided so that the (mix up) without animation. user can test his or her skills at manipulating the object into the standard orthographic positions. The process of rolling the object is depicted in Figures 6 and 7. In Figure 6, the user has moved the mouse pointer into the main display box to prepare to manipulate the object. In Figure 7, the mouse has been clicked, held down, and dragged to roll the object in anew position.
Display Object Interfaee The primary element in the Display Object interface is the large square box in the upper-right portion of the interface window (Figure 5). In this box, the currently open object is displayed. As the mouse pointer enters the box, it is changed to the hand shape to indicate that direct manipulation of the object is possible. To perform rolling of the object, the user clicks on the object, and while holding the mouse button down, drags in the desired direction. The object is rotated about an axis in the plane of the screen which is perpendicular to the dragged path. To give the user feedback that rolling is taking place, the mouse pointer is changed to the crossing-arrows shape. To perform spinning of the object, the user clicks within the box, but not on the object, and while holding the mouse button down, drags in the desired direction. The object is rotated about an axis located at the center of the box and perpendicular to the screen. The mouse pointer is changed to the spinning-arrows shape to give feedback that spinning is taking place.
Figure 6: Initiation of Object Manipulation
Immediately below the main display box is a series of pushbuttons (Figure 5). By clicking on these buttons, the user
Isometric tlieux
~~ .————— —-. .. —. —.—.. — ....
+
● Click - On 1?Drag - the object on the right to manipulate it about eny rmis in the plane of the screen.
cIa@mmiiEEKl
Click - Outside 0’ Orag - the object on the right to rotate it ebo.t an axis
●
/@j
@Solid
normal to the plane of the screen.
Oisplay Options:
Figure 5: Display Object Interface
78
Model 0 Hidden Line OILlire Frame
[HI ’92
Mav3-7,199 plane intersection. If the user can directly manipulate the cutting-plane while the object remains stationary, then that plane could be flipped or rotated relative to its original position and will rarely be in a position parallel to the screen. When the corresponding cutting-plane intersection is displayed in a second area on the screen, the user might become confused by the relationship between the intersection and the cutting-plane itself.
I
I
Figure 7: New Object Position
An inversion of roles in the above concept solves the Instead of leaving the object associated problems. stationary, the cutting-plane is held stationary while the object is directly manipulated. With this type of control strategy, there is a one-to-one correspondence between the cutting-plane (now always parallel to the screen) and the separately displayed cutting-plane intersection. All that is needed to give the user complete control of the cuttingplane position is a front to back depth adjustment. This control strategy is a direct extension of the direct manipulation controller introduced with the Display Object interface. Therefore, it should be easier for the user to become accustomed to its action.
During
Rolling
To the left of the main disulav box are two small static images (Figure 5). One ok tkese images is the current object displayed at one half its size and in an isometric position. This image is always in the isometric position. Next to this object image is an isometric cube which shows the position of the x, y, and z axes for the standard isometric position. The three visible faces of the cube are shown with the corresponding top, front, and side labels. These two images, taken together, are intended to provide a reference to the user which defines the standard views for the current object.
A “pool of water” metaphor can be associated with this control strategy. The surface of the water represents the cutting-plane. The depth of the water pool corresponds to the depth of the cutting-plane. The pool is represented by the color gray. The parts of the object behind the cuttingplane are dimmed to indicate that they are submerged.
Immediately below the two reference images is a brief description of two direct manipulation techniques deseribed above (Figure 5). The shapes into which the mouse pointer changes during object manipulation are visually reinforced by a pair of icons next to the descriptions.
A difficulty associated with this cutting-plane control method is that when the object is displayed in color, some users misinterpret the dimmed colors as shadows. Once the user understands the water pool metaphor, this confusion usually ceases. However, in an effort to provide a more intuitive cutting-plane user interface, the choice of a surface representation picture was added. The picture is drawn between the portion of the object behind the cutting-plane and the portion in front. A user can select one of several pictures or none at all.
In the lower-right portion of the window, three radio-style buttons are provided to select the object display option (Figure 5). This style of button indicates a set of mutually exclusive choices. Solid Model displays the object in color, Hidden Line displays the object in black and white with hidden edges removed, and Wire Frame displays all edges of the object in black and white.
Cut Object Interface The main element of the Cut Object user interface for cutting-plane manipulation is the huge square box in the upper-right portion of the interface window (Figure 8). The main display box is placed in the same position as it is in the Display Object interface, because once again, this box is used for direct manipulation of the object. As before, the mouse pointer is changed to the hand shape when it passes into this box, and is changed to the crossing-arrows or spinning-arrows shape when the object is manipulated. The display box itself now represents a portion of the cuttingplane. The background color within the box is gray to represent the pool of water. The parts of the current object behind the cutting-plane are dimmed to simulate submersion into the pool. Below the main display box, the same animation buttons and display option radio-style buttons are provided as in the Display Object interface.
Cutting-Plane Manipulation The original idea for manipulation of a cutting-plane, and perhaps the most intuitive, involved always displaying the object in an isometric position and then providing a rectangular “plate” that would slice this image. By direct manipulation of the plate, the user could position the cutting-plane. However, to give complete control of the cutting-plane position, a lateral or front to back positioning of the plate would also be necessary. Thus, a cutting-plane control strategy based on this concept would need much more than the object manipulation control discussed previously since that control provides only for rolling and spinning. A second problem with the above concept for cutting-plane control has to do with the display of the resulting cutting-
79
May3-7, 1992
~ [HI ’92
I CuttlmgPlane ●
+
Click
Control of the
- On t? Drag
- the
bar to adjust cutting plane.
UP raises down
I Intersection
the
Iowars
level tha
czz13cmE3miiiEIm m@i!@mcziK3
Depth
the Ieuel Drogglng and
@ Solid
dragging
Oisplay
Options:
level.
0
Model
Hidden
O Wire
Line Frame
Figure 8: Cut Objeet Interface
Immediately to the left of the main display box is the water pool depth control (Figure 8). The mouse pointer is changed to the hand shape when it passes into the depth control region. By clicking in the depth control, holding the mouse button down, and dragging up and down, the user can adjust the pool depth And ‘hence the cutting-plane
Cutting
●
t ~
Click
Control ID( the
Plane
- On & Orag
- the
bar to adjust cutting plane.
UP raises down
Intersection
the
lowers
Ieual tha
position relative to the object. The mouse pointer is changed to the grabbing-hand shape to give the user feedback that depth adjustment is taking place. The partially submerged object is redrawn as necessary while the user continues to adjust the depth (Figure 9).
‘epth Depth
m
m
tha level Dragging and
~
m@z!El~ @l Solid
dragging
Display
Ieuel.
Options:
0
O Wire
Figure 9: Cut Object Interface
80
During
)-todel
Hidden
Line Frame
Depth Adjustment
V CHI’92
M(w3-7.1992
The large square box in the upper-left portion of the window displays the cutting-plane intersection (Figure 8 or 9). The intersection is drawn with the same scale as the object itself so that the one-to-one correspondence between the two images is emphasized. The intersection image is updated as necessary while the object is directly manipulated in the main display box and while the water pool depth is adjusted.
rest of the Macintosh special attention.
and hence did not require
The software was written in C using the THINK C development environment and makes full use of the Macintosh operating system graphics routines. The code which performs all transformations and other internal object representation algorithms were written from scratch in an effort to make the software as fast, and therefore as interactive, as possible. The objects are represented using polygonal facets. Back-face determination is used to sort the facets for simple convex objects. Optionally, a depth sort or a binary space partitioning tree can be used to sort the facets on more complicated objects [4]. The drawing of the objects is double-buffered@ enhance the smoothness of the display.
Below the cutting-plane intersection, a brief description is provided for water pool depth adjustment (Figure 8 or 9), This description is accompanied by a grabbing-hand icon to visually reinforce the mouse pointer feedback which occurs during depth adjustment. A surface menu is added to the standard menu bar (at the top of the screen) to provide the user with a selection of pictures to serve as surface representations. The selected surface picture, if any, is displayed between the submerged and unsubmerged portions of the object. Several different types of pictures are possible including grids, wavy lines (to represent the water surface), or even lily pads! Figure 10 shows the Cut Object main display box with a surface picture reminiscent of a cartoon representation of glass using several groups of diagonal lines;
Figure 10: Cut Object Main Display “Glass” Surface Picture
interface
Software Evaluation Evaluation is currently in the formative stages. During the course of development, over 60 subjects have had the opportunity to use the software. Subjects ranged from junior-high school students to mechanical engineering undergraduate and graduate students. Over half of the subjects were given paper exercises before and after using The exercises involved sketching the software. orthographic views given an isometric view and vice versa. For the cutting-plane mode, subjects were given isometric views of objects and asked to predict if certain intersection depictions were consistent’ with those views. In another exercise, subjects were given the cutting-plane intersection (such as that shown in the left box in Figme 9) and asked to draw the cutting-plane onto a given isometric picture. Subjects universally had the subjective opinion that using the software was intuitive, helpful, and fun. ”Some subjects also reported improved self-confidence and increased motivation on spatial reasoning tasks. An interesting observation drawn from some subject’s descriptions is that the dynam’ic nature of the software makes the orthographic views more understandable. For these subjects, the capability to move the object slightly off of an orthographic position was enough of a hint to clarify their intuitive understanding of the object. Other subjects reported that color was extremely helpful in understanding the objects.
Box With
The Apple Macintosh personal computer was selected as the development platform because of its graphics capabilities, its pre-existing intuitive interface, and its availability to a large number of users. The necessary Macintosh hardware configumtion requires a minimum of 8 bit color to enhance the realistic representation of objects and a floating-point coprocessor to maintain the interactive nature of the software.
To obtain an objective evaluation of the software in terms of its usefulness in improving spatial abilities and visualization, a similar series of exercises will be administered to a larger group of subjects who are all students in the introductory engineering graphics course at U. C. Berkeley. The course typically has an enrollment of over 100 individuals. A strategy utilizing tests before and after software use with a control group that does not use the software is planned for the spring semester of 1992.
The selection of the Macintosh as the development platform allowed development efforts to focus primarily on the object manipulation aspect of the software. The usage of menus and windows in the software is consistent with the
CONCLUSIONS Based on subjective evaluation, the goal of producing an intuitive interactive software environment for manipulation of objects has been satisfied. Students do find the resulting
81
May3-7, 1992
[Hi ’92
REFERENCES 1. Barr, Ronald E., and Davor Juricic. The Engineering Design Graphics (EDG) Curriculum Modernization Projecc A White Paper Summary. In Proceedings of the NSF Symposium on Modernization of the Engineering Design Graphics Curriculum (Austin, Texas, August 5- 7). University of Texas Printing Department, Austin, Texas, 1990
environment intuitive, fun, and motivating. The design of an intuitive cutting-plane user interface was successful as a result of creating a novel “pool of water” metaphor which simplified the required user interaction and built on user interaction skills learned from direct object manipulation. However, the success of the software in improving spatial reasoning and visualization abilities has yet to be objectively verifkd. Plans are underway to accomplish this in the Spring of 1992. Future development efforts include addressing the reverse spatial reasoning process involving mapping from 2D to 3D representations. Current concepts include a user interface which allows users to directly manipulate multiple orthographic representations of an object to see the relationship to the resulting 3D representation. Efforts will also be directed at the creation of a variety of exercises for use with the software and software integration with a hyperscripting program to allow the development of intelligent tutoring interactive courseware. ACKNOWLEDGEMENTS The project discussed here was supported by the National Science Foundation as part of “Synthesis: A National Engineering Education Coalition,” which is focused on the renovation of engineering education. The Spatial Reasoning Group at U. C. Berkeley (in alphabetical order) consists ofi Alice M. Agogino, John E. Bell, Fred Beshears, Tom Knudsen, Howie Lan, Debbie Lee, Dennis K. Lieu, Marcia C. Linn, James R. Osborn, Alice Wong, and William Wood. The authors would also like to gratefully acknowledge the contributions of the ~ollaborators at the Syn~esis Coalition institutions: Rollie Jenison at Iowa State”University and Adebisi O. Oladipupo The authors would also like to at Hampton University. thank Michael Chen for making controller demonstration software available. The software described here was developed using equipment donated to the University of California at Berkeley by Apple Computer, Inc.
82
2.
Chen, Michael, S. Joy Mountford, and Abigail Sellen. A Study in Interactive 3-D Rotation Using 2-D Control In Proceedings of ACM SIGGRAPH Devices. (August). Vol. 22, no. 4, ACM, New York, 1988
3.
Cooper, Lynn A. Strategic Factors in Complex Spatial Problem Solving. Invited paper presented at the 56th annual meeting of the Midwestern Psychological Association (May 4). Chicago, Illinois, 1984.
4.
Foley, James D., et al. Computer Graphics Principles and Practice. 2nd Ed., Addison-Wesley, 1990
5.
Linn, Marcia C. and Anne C. Peterson. Emergence and Characterization of Sex Differences in Spatial Ability: A Mets-Analysis. Child Development, 56 (1985), 1479-1498.
6.
Monk, Andrew. Fundamentals of Human-Computer Interaction. Academic Press, London, 1985
7.
Oladipupo, Adebisi O. Solids Modeling in Freshmen Engineering Graphics Using Silverscreen. In Proceedings of ASEE Annual Conference, 1991
8.
Shepard, R. N., and J. Metzler. Mental Rotation of Three-Dimensional Objects. Science, Vol. 171, 1971
