Developing and Evaluating Collaborative ... - Semantic Scholar

Developing and Evaluating Collaborative Engineering Studios Jonathan Sevy Vera Zaychik Thomas T. Hewett William C. Regli Geometric and Intelligent Computing Laboratory Department of Mathematics and Computer Science Drexel University 3141 Chestnut Street Philadelphia, PA 19104 http://gicl.mcs.drexel.edu

Abstract We discuss a vision of and work in progress on a collaborative engineering environment, the Collaborative Design Studio, being developed at the Geometric and Intelligent Computing Laboratory at Drexel University. This works seeks to integrate Computer-Aided Design (CAD) and communication tools through automatic extraction and embedding of design context into communications, as well as provide archiving and retrieval of informal communications. While preliminary development of the studio has proceeded using relatively informal input from professional design engineers, direct evaluation by intended users of the system will be sought to guide further development of the system to ensure that it accurately meets their needs and fits into their work environment. Such formative user input will be obtained through surveys and observation of subjects as they complete specified tasks. Future summative evaluation will attempt to measure changes in user productivity and efficiency. Discussion focuses on issues and questions relating to the effective design and evaluation of such systems, and describe proposed studies for evolving the current environment.

1. Introduction We describe our vision of a collaborative engineering environment, the Collaborative Design Studio, consider issues and questions relating to the effective design and evaluation of such systems, and describe proposed studies for evaluating and evolving the current environment. This work is being undertaken at the Geometric and Intelligent Computing Laboratory at Drexel University. While preliminary development of the studio has proceeded using rel-

atively informal input from professional design engineers, we feel it is essential to incorporate direct evaluation by intended users of the system to ensure that it accurately meets their needs and fits into their work environment. Such information will be obtained through surveys and observation of subjects as they complete specified tasks, analyzed, and used to guide further development of the system.

2. Current Vision of a Collaborative Design Studio The fundamental goal of the Collaborative Design Studio project is to enhance the design engineering process through the integration of computer-aided design and engineering tools, communication tools, and archiving functions. Design context is automatically extracted from CAD tools for inclusion in communications, and all design-related communications are automatically archived in a searchable database. We are attempting to achieve this while incorporating commercial off-the-shelf (COTS) tools, with underlying integration functions being transparent to users. The proposed environment should provide a number of enhancements to the design engineering process. A central advantage will be the enabling of efficient and effective distributed design efforts, with geographically separated engineers able to work together effectively. The capabilities of the studio are being specifically targeted to this increasingly important work mode; however, they should prove advantageous in traditional settings as well. A significant benefit should be a reduction in design development time, facilitated by the automatic inclusion of design context into communications. Even frequent and informal forms of communication (such as e-mail) will include precise and unambiguous reference to parts and features in the current de-

sign, making it easier for engineers to have detailed discussions without requiring face-to-face meetings. The integration of communication tools and design tools allow these contextual references to initiate the automatic display of the relevant design model or diagram, freeing the receiver of a communication from the task of locating and loading the appropriate file. In addition, the automatic inclusion of specific, unambiguous design references should reduce the number of errors or misunderstandings generated by misinterpretation of designer-produced verbal descriptions of the design feature and context to which the communication applies. Finally, the automatic archiving of all communications, together with embedded design context information, should greatly aid many design activities. Design reuse and adaptation will be facilitated by designers’ having access to all communications related to previous design efforts, which can provide valuable insight into tradeoffs and compromises made which might not be reflected in formal documentation. During protracted design efforts, such an archive may provide the only access to rationales used by engineers who have retired or left the project. Manufacturing and maintenance activities should similarly benefit from being able to identify hidden considerations relating to specific part functions. A side benefit is the opportunity that such a transparent integrated environment presents to researchers wishing to study workgroup interactions. The automatic and transparent archiving of communications provides access to all but the most informal verbal communications exchanged during the design process, in a completely non-intrusive fashion, permitting a researcher to study interactions without perturbing the participants’ normal work environment.

3. Studio Architecture The user-accessible components of the Collaborative Design Studio consists of three groups of tools: CAD/CAE tools, collaborative work tools, and archiving tools. These are integrated through a number of software modules. The design of the environment has been guided by two requirements: standard commercial tools should be incorporated whenever possible, so as to avoid expecting designers to use proprietary software in place of effective and familiar alternatives; and the integration of tools should be as nearly transparent as possible, placing few additional burdens on users. The current prototype utilizes the Structural Dynamics Research Corporation I-DEAS suite as its CAD/CAE package, Netscape Communicator as the email client, Collaborative Virtual Workspace 1 (CVW, developed by the Mitre Corporation) as a video-conferencing tool, and Oracle as the database archive (the search interface to this archive being web-based, allowing the use of any web 1 http://cvw.mitre.org

browser). However, the integrating modules have been designed with an emphasis on “package independence,” to allow substitution of alternate packages with relatively minor effort (as long as the substitutes provide a sufficiently rich feature set — for example, CAD/CAE software with an application programming interface permitting the extraction of design information). Design context is extracted from the current design for embedding into a communication by the client communication module. This module communicates with the CAD/CAE software (using the Open I-DEAS API in the case of SDRC I-DEAS) and creates an XML document incorporating the extracted design data organized according to a design-communication specific message model. This message model is central to the integration of the components in the studio: all design and communication context is captured in these messages, which provide the fundamental level of organization of information. A designer initiates a communication through a script which first calls the client module to extract the design context and create the XML document and then starts the appropriate communication tool, with options that associate the generated XML document (as an attachment in the case of email) and automatically include the archiving server as a participant (or recipient) in the communication. The user will observe little of this background processing, being presented with the selected communication tool as in the normal (unintegrated) setting. The client communication module also functions on the receiver side, using the attached XML design context document to display the appropriate design part in the CAD package. The archiving server extracts design context from the XML document associated with a received communication and uses it to drive storage of the communication in the archive. The server also permits web-based search access to archived communications, returning retrieved information in the form of XML documents which are used by the communication module to display the associated design files. While we feel the Collaborative Design Studio will provide many obvious benefits for designers, there are still a number of questions to be addressed. One particular issue is the amount and level of design context information to include into the XML documents. The message model has been designed to be extensible, permitting varying types of context to be included; however, this leaves open the question of exactly which information to include, which cannot be adequately answered without some form of input from the intended users of the system. In addition, it will be important to understand the role of the various tools used by designers when they employ the integrated suite in the Collaborative Design Studio. Some questions to be addressed

are: to what extent does integration enhance the design process, and how can this be measured; how is the pattern of use of the tools changed when integration is incorporated; and how does access to archived communications influence design efforts. The issue of how we hope to obtain this information is addressed below.

4. Formulating Evaluation Questions and Procedures There are two levels of criteria that are important in thinking about ways to facilitate use of an engineering design environment. One is the interface or tactical level; the second is the interaction or strategic level. Interface level criteria address issues such as how interface components and widgets should be built or “shaped” – for example, aircraft cockpit instruments need to show certain necessary information clearly. In contrast, interaction level criteria address the goals of the user and focus on the way the user thinks about the task. Certainly, a concern with interface features is important; however, optimizing these features without attending closely to the overall flow of interaction can lead to unworkable or unusable systems. Adopting the goal of understanding the designers’ tasks and how they proceed in solving their problems immediately commits one to developing explicit answers to several questions. Exactly who are the target users? What knowledge are they assumed to have, and what are their skills? What specific tasks will such users want to complete and are there generic types of tasks or task characteristics? How will successful user performance be measured or assessed? As a number of sources on Task Analysis [10, 14] have pointed out, the user’s task can have a significant impact on how tools are used or not used. Consider the use of CAD systems in architecture. It is possible to divide architectural design into two broad categories of activity. One type of activity is associated with preparation of a model or design representation to illustrate the design to a customer. Another type of activity is associated with sketching out and developing a novel design concept. Even casual observation of usage patterns of architectural CAD systems reveals task differences. While it is not unusual to find someone trained in the use of an architectural CAD system putting it to use in polishing and preparing drawings for presentation, it is quite a challenge to find an architect using such a CAD system while creating an original design concept.

4.1 A taxonomy of design activities As pointed out by Brown and Chandrasekaran [2] and Dym [5], there are three identifiable types of engineering design activity: routine, variant, and creative. Routine

design involves taking an existing design and making minor tweaks in the artifact being designed to develop an improved artifact. Variant design involves using existing components or subassemblies to create a new version of an existing artifact. Creative design involves coming up with a new design concept (though existing components or subassemblies may become components of the new artifact). It is expected that the tasks and activities of designers will vary depending on the type of design being performed. It will thus be necessary to account for this when studying the designers’ activities and evaluating a collaborative system.

4.2 A role of analogical thinking in design One place to begin identifying task-critical functionalities for a human centered engineering design environment is with a realistic understanding of current procedures, practices and activities that the environment is meant to augment or replace, as discussed in Hewett and DePaul [7]. For example, there are a variety of reasons for believing analogical thinking is a key process involved in creative advances in science and engineering [9]. Of particular relevance to thinking about the creation of design environments is work on the use of analogy, both in individual problem solving [3] and in generating new insights during the social process of science and engineering [4]. Clement [3] found that the steps used by scientists in developing a problem solution are not necessarily derivational or formal, but often proceed by analogy. He gave a mechanics problem to 10 participants sophisticated in physics but with no extensive experience solving problems from the class, and asked participants to “think aloud” while solving the problem. Examination of videotapes from these sessions revealed that the typical solution procedure involved memory search for an analogous situation with similar characteristics, conducted through association or by examining dimensions of similarity. The problem solver then developed a possible solution on an analog which had been judged to be appropriate; once the solution had been worked out for the analog, the individual then proceeded to test the solution on the original problem. Dunbar [4] reported on a year-long study conducted in four molecular biology laboratories, where he followed several projects. The processes studied included “planning the research, executing the experiments, evaluating the experimental results, attending laboratory staff meetings and public talks, planning further experiments, and writing journal articles (p. 365).” As Dunbar reports, “analogies were an important source of knowledge and conceptual change (p. 381).” The researchers Dunbar studied used three different classes of analogy. Local analogies were those within the

same domain as the research being conducted and were drawn from a previous experiment to the current one. This type of analogical reasoning typically occurred when a scientist would attempt to map a currently unsuccessful experiment onto a similar experiment that had been successful. Local analogies seemed to be one of the main mechanisms for driving the research program forward. Regional analogies involved mapping a system of relationships from a similar domain onto the problem domain being studied. This type of analogical reasoning was typically “employed when the scientists were working on elaborating their theory and planning a new set of experiments (p. 383).” These regional analogies seemed to be one of the main mechanisms for allowing the scientists to fill in gaps in their own knowledge and to suggest new questions to ask about the entities being studied. Long-distance analogies involved mapping a concept from a very different domain onto the problem domain being studied. Typically, “long-distance analogies were used to highlight features of the research that were salient,” and were used to, “bring home a point or to educate new staff members of a laboratory (p. 383).” This suggests investigating the role of analogical thinking in both individual and collaborative engineering design, exploring the apparent parallel which exists between Dym’s categorization of engineering design activities (routine, variant, and creative) and Dunbar’s comments about the types of analogy that were used in the research laboratories which he studied — i.e., are routine, variant and creative design characterized or driven by the use of local, regional, and distant analogies, respectively. Our preliminary hypothesis is that the role of analogy in collaborative engineering design is analogous to its role in scientific research. At an individual level it is a way to generate alternatives and solve the problem of selection of alternatives. At a social level it is a way to extend knowledge, to ask new questions about a design, and to communicate with design team members in working out novel solutions to design problems. Consequently, one issue in developing measures and metrics will be finding ways of assessing when and to what degree analogical thinking takes place. In the event that data support the hypotheses that analogical thinking plays a critical role in design and that different types of design problems (routine, variant, and creative) involve different types of analogies (local, regional, and long-distance), it would be valuable to determine how analogical thinking can be supported in a collaborative design studio. One approach would be to provide a library giving access to local and regional design analogs (somewhat along the lines of the design repository described by Regli, [13]). Such design examples could be stored as modifiable templates, adaptable for use in solving analogous design problems. The design repository should contain ex2 dvirc.org

amples drawn from more than one related design problem domain, to support and facilitate regional and long-distance analogical thinking. In addition, these inquiries into characteristics of the engineering design process will guide the development of the message model used to archive design decisions and provide design context during communication. It is expected that the type of design context information most useful to designers will vary somewhat depending on whether design is routine, variant or creative. It is apparent that a collaborative engineering design environment presents a wide array of questions to be addressed. There are questions concerned with interface level details, i.e., are the tools usable and useful given their intended purpose, as well as those about facilitating interaction, both a single designer’s interaction with the tool set and the social interaction among members of a team. In addition, a collaborative studio simultaneously creates an environment in which questions about the nature of design and collaborative design can be formulated and answered. Answers to these questions, such as the role of analogical thinking in design and its relation to the basic types of design, can in turn be used to evolve the studio to better support design functions.

5. Proposed Approaches to Evaluation of the Collaborative Design Studio 5.1 Survey of working engineering designers In order to develop baseline data about how design engineers currently think they collaborate, we have under development a survey which we propose to circulate to working engineering designers. In this survey we explore both the types of knowledge which engineering designers feel they use and the nature of how they store, retrieve, and/or exchange that information to other designers (e.g., use of analogical thinking). This survey is being structured based upon both our own experience and some of the work of others (e.g., [11, 12, 15]). Once the survey document has been pilot tested to improve its wording, etc., we intend to disseminate the questionnaire to a large number of design engineers in New Jersey, Pennsylvania, Delaware and Maryland in conjunction with our collaborators with the Delaware Valley Industrial Resource Center (DVIRC) 2 . Not only will this survey provide us with normative data about what designers currently think about what they do, it will also be used to seek volunteers to participate in further studies in which experienced engineers are the most appropriate participants. In short, the survey provides us with a “map of the terrain,” an indication of tasks that need to be studied and questions to be answered.

5.2 Thinking out loud studies In order to capture snapshots of the actual working and collaborative practices of engineers, we currently anticipate the need for one or more studies in which engineers and/or student engineers participate, working on real or simulated design problems. Again, these studies will allow us to establish a baseline for existing collaborative procedures and then to learn more about the weaknesses of the tools and environment at the interface level when infelicities or outright errors are identified for us by people engaged in a design task — e.g., is there an unusable tool or feature? In addition, we anticipate the need for one or more studies focusing on the interaction level of analysis so that we can improve the cognitive engineering of the environment and tools — e.g., have we structured the environment in such a way that the designer is forced to hold more information in memory than a human being can maintain in an active state? Protocol analysis, or thinking out loud, is a technique to obtain subjects’ verbal reports about a task being performed concurrent with the process of completing that task. It has been shown that this technique can produce valuable data while not altering subjects’ mental processes [6]. Subjects are instructed to verbalize their thoughts while performing the task being studied — i.e., to think out loud. It should be noted that this is somewhat different from having participants explain their actions while they are engaged in the task; in particular, it is relatively unobtrusive to the problem-solving process (except in certain exceptional cases, such as time-critical activities). The technique gives insight into subjects’ mental processes, course of thought and problem-solving approach in addressing the specified task. This real-time verbal report will also be supplemented with retrospective reports and interviews of the type reported by Hewett and Scott [8]. The retrospective reports provide an indication of what participants “think they were doing” during the activity; the former gives evidence of what they “actually were doing.” It is generally desirable that subjects employed in such a study have prior familiarity with the process being studied to gain any reliable insight into that process. Thus if the process of designing a mechanical artifact is being studied, the participants of the study should be people familiar with mechanical design, perhaps industrial designers or design engineers. If in addition what is being evaluated is not the CAD software itself, but the integration of that software into an environment including other applications, then the participants should be familiar with the use of a CAD package (though not necessarily the specific package used in the study, since most CAD applications are similar in behavior, at least in those components likely to be relevant for the study). Considering that each test group should consist

of at least 10 teams of subjects to guarantee an acceptable level of confidence in the results, it becomes clear why such studies are seldom done: it is very hard to get appropriate subjects. For our particular problem, the main task assigned to users would be to solve a design problem as a group, collaboratively, while being spatially separated. The chosen problem needs to be relatively simple, able to be solved in a matter of hours, while still requiring active collaboration of members of the design team. The activity to be observed is how CSCW tools are used in the process of design, and how their integration with design authoring tools influences the activity — in particular, is such integration beneficial, and what kinds of information about the design should be incorporated into communications. A control group of designers will work on a design problem under “normal” conditions, in which design authoring and CSCW tools are both present but not bound together in any way; another group will be given access to the Collaborative Studio, with communication tools integrated with CAD software. Experimenters will observe subjects, recording what kind of information is being exchanged, how, in which cases, and for what purposes participants use the communication tools at their disposal, and other pertinent data. The observers will record in addition how long it takes to complete the task, the frequency and amount of information exchanged, and the classification of that information. The types of measures and metrics which are relevant here include such things as quality of the design and suitability of the design, as assessed by judges who are unaware of which type of groups did the design. Also relevant will be measures of participant satisfaction with the end result and the work progress. For our particular problem, the main task assigned to users would be to solve a design problem as a group, collaboratively, while being spatially separated. The chosen problem needs to be relatively simple, able to be solved in a matter of hours, while still requiring active collaboration of members of the design team. The activity to be observed is how CSCW tools are used in the process of design, and how their integration with design authoring tools influences the activity — in particular, is such integration beneficial, and what kinds of information about the design should be incorporated into communications. A control group of designers will work on a design problem under “normal” conditions, in which design authoring and CSCW tools are both present but not bound together in any way; another group will be given access to the Collaborative Studio, with communication tools integrated with CAD software. Experimenters will observe subjects, recording what kind of information is being exchanged, how, in which cases, and for what purposes participants use the communication tools at their disposal, and other pertinent data. The observers will record in addition how long it takes to complete the task, the fre-

quency and amount of information exchanged, and the classification of that information.

5.3 Interaction Process Analysis Another aspect of the Collaborative Studio design that we intend to study is the categorization of exchanged communications. We have developed a preliminary set of categories for the types of design communication that could be exchanged: request for change, change notification, information request, information reply, etc. However, the study will help to improve and augment the classification. The communication tools require the user to select the type of communication being sent, with the ability to supply a userdefined type if none of the options are appropriate. However, there is often a difference between what a subject says he or she is doing, and what is actually being done. Thus protocol analysis will be used to try to determine the intention which lies behind the sending of each communication, which should help identify possible classification schemas. Linguistic analysis will also be performed on the subjects’ verbal reports to determine patterns, significant and preferred words (such as action verbs [1]), and the like.

6. Conclusion The Collaborative Design Studio is being designed to provide features and functions likely to prove valuable in any engineering environment, whether traditional or distributed. However, until user studies are undertaken, the value of the system as a whole and the relevance of specific features remain unproven. To guide further development, we have proposed several evaluation strategies, which will serve as a primary source of information for the evolution of the design environment.

Acknowledgements. This work was supported in part by National Science Foundation (NSF), Knowledge and Distributed Intelligence in the Information Age (KDI) Initiative Grant CISE/IIS-9873005 and Grants ENG/DMI9713718 and CISE/CDA-9729827. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or the other supporting government and corporate organizations.

References [1] J. L. Austin. How To Do Things With Words. Harvard University Press, Cambridge, Massachusetts, 1962. [2] D. C. Brown and B. Chandrasekaran. Design Problem Solving : Knowledge Structures and Control Strategies. M. Kaufmann Publishers, Loas Altos, CA, 1989. [3] J. Clement. Observed methods for generating analogies in scientific problem solving. Cognitive Science, 12:563–586, 1988. [4] K. Dunbar. How scientists really reason: Scientific reasoning in real-world laboratories. In R. J. Sternberg and J. E. Davidson, editors, The Nature of Insight, pages 365–395. The MIT Press, Cambridge, MA, 1995. [5] C. L. Dym. Engineering Design: A Synthesis of Views. Cambridge University Press, New York, NY, 1994. [6] K. A. Ericsson and H. A. Simon. Protocol Analysis: Verbal Reports as Data. The MIT Press, Cambridge, Massachusetts, revised, third printing edition, 1999. [7] T. T. Hewett and J. L. DePaul. Towards a human-centered scientific problem solving environment. In E. Houstis, S. Gallipolous, J. R. Rice, and R. Bramley, editors, Enabling Technologies for Computational Science. Kluwer Scientific Publishers, Boston, MA, 2000. [8] T. T. Hewett and S. Scott. The use of thinking-outloud and protocol analysis in development of a process model of interactive database searching. In H.-J. Bullinger and B. Shackel, editors, Human Computer Interaction– INTERACT ’87, pages 51–56. Amsterdam: North-Holland, September 1987. [9] K. J. Holyoak and P. Thagard. Mental Leaps: Analogy in Creative Thought. The MIT Press, Cambridge, MA, 1995. [10] B. Kirwan and L.-K. Ainsworth, editors. A Guide to Task Analysis. Taylor and Francis, London, England UK, 1992. [11] M. L. Maher, S. J. Simoff, and A. Cicognani. Understanding Virtual Design Studios. Springer Verlag, 2000. [12] S. E. Poltrock and G. Engelbeck. Requirements for a virtual collocation environment. Information and Software Technology, 41(6):331–340, April 1999. [13] W. C. Regli and V. Cicirello. Managing digital libraries for computer-aided design. International Journal of Computer Aided Design, 32(2):119–132, Februrary 2000. Special Issue on CAD After 2000. Mohsen Rezayat, Guest Editor. [14] T. L. Seamster, R. E. Redding, J. R. Cannon, and J. M. Ryder. Cognitive task analysis of expertise in air traffic control. International Journal of Aviation Psychology, 3(4):257–283, 1993. [15] J. C. Tang and L. J. Leifer. A framework for understanding the workspace activity of design teams. In Proceedings of the Conference of Computer-Supported Cooperative Work, 1988.