14design team performance: metrics and the impact of ... - CiteSeerX

14 DESIGN TEAM PERFORMANCE: METRICS AND THE IMPACT OF TECHNOLOGY Larry Leifer

Introduction For over fifteen years the Stanford University Center for Design Research has been concerned with the formal study of engineering product development teams at work in academic and corporate settings. This chapter describes one such effort, the implementation of team-based, distance learnining techniques in a Stanford University School of Engineering course, “Team-Based DesignDevelopment with Corporate Partners”. The course is distributed nationally by the Stanford Instructional Television Network for on-campus full time students and off-campus industry based part time students. Applied ethnography methods like video interaction analysis have been used to reveal the detail and pattern of activity of design teams. Computational support systems have been developed to facilitate their work. Throughout these studies we have seen that their activity closely resembles the most attractive aspects of self-paced education described by constructivist learning theorists. The design environment has been instrumented to see if technical and behavioral interventions did, in fact, improve performance. It is this learning validation phase of our work, and its dependence on technology, both for instructional delivery and for assessing performance outcomes, that we wish to share with you in this chapter. To put the findings in context, we must first introduce the pedagogic framework that guides our value judgments. Hence, we introduce the concept

2

Design Team Performance: Metrics and the Impact of Technology

and practice of "Product-Based-Learning" (PBL) and the feedback instrumentation model we use for assessing PBL performance. Within this framework, the discussion will focus on technology specific issues, the rationale for technical intervention and a sampling of performance assessment results.

The Context of Engineering Design Education Economic and technologic forces are changing the landscape of engineering. There is an increasing need for organizations to form joint design development teams that collaborate for the life of a project and then disperse. These teams need to quickly locate, evaluate and make effective use of the best resources available (tools, facilities, people), wherever they may be found. We will concentrate on a subset of this problem that is critical in the early stages of product development when new ideas, processes, components and materials are being explored and prototype solutions are developed. We seek to enhance the ability of teams to exploit their own shared knowledge, novel tools and manufacturing processes for improved product performance, reduced cost, and documentation of their exploration for those who will inevitably follow, the next generation re-design team. In this way we believe that the slow cycle of exploration, maturation and widespread adoption of advances in information handling, materials and processing capabilities can be compressed from decades to years. Our work is ongoing in the form of an annual series of experiments involving 12 to 15 design teams in a product design-development laboratory at Stanford University. Ten years of careful observation and analysis of teams at work in their graduate studies and in industry, are beginning to yield objective predictors of design quality outcome and management strategies for assuring uniformly high team performance. The pedagogic framework of this test-bed course and associated research program are informed by work in the social and cognitive sciences.

Kolb's Learning Cycle Students have been observed to learn in four different ways (see Figure 14.1). Kolb (1984) proposed that repeated cycles of experiences moving through these learning modalities improve understanding. The cycle is best started with concrete experience, proceeding to abstraction (Harb, Durrant & Terry, 1993). Beginning with a need for learning stimulated by immediate experience, the learner should reflect and question his perceptions of the learning environment

Evaluating Corporate Training: Models and Issues

3

before proceeding to abstraction and experimentation. The complete cycle brings understanding in depth. Prompt feedback of the intention and meaning of that experience plays an important role in the preservation of learning for future reference.

Concrete Experience (reverse engineering)

Reflective Observation (notebook thinking)

product based learning Active Experimentation (design synthesis)

Abstract Conceptualization (modeling & analysis)

Figure 14.1 Four phase loop of experiential learning

Those who seek to support distant learning, asynchronous team work and interdisciplinary collaboration must be vigilant about preserving the critical features of an experiential learning framework while exploring the power of emerging technology to bring these methods to more people, in more places and at lower cost than has previously been possible. Leifer (1995), after Kolb (1984), models student experiential learning as a four phase loop. Kolb observed that a cycle of experiences improves understanding and builds bridges between theory and practice. This is a qualitatively satisfying view of product-based-learning as well. It remains a challenge to objectively and quantitatively demonstrate performance improvements when this pattern is experienced.

Constructivist Learning Educational theorists, including Jean Piaget, have developed a model of learning by which students develop knowledge structures based on previous experience. A theory of how to teach science known as the "scientific learning cycle" is a direct outgrowth of Piaget's ideas and constructivism (Harb, Durrant and Terry, 1993). Twenty years in engineering design education supports the view that direct experience is the learning medium of choice in our domain of higher

4

Design Team Performance: Metrics and the Impact of Technology

education and parallels the fact that professional experience is the measure by which most engineers, scientists and other professionals are rated. One may summarize the lessons of constructivist research as declaring that learning is best done by creating something, a product, that embodies our knowledge. This is product-based-learning.

Vygotsky's Model Vygotsky, as represented by Moll (1990), argues that knowledge is social before it is personal, which suggests that it must be interactively and socially constructed. This is usually observed through language usage though it can also be seen in the use of diagrams and sketches. In this model, learning must be external and shared before it can be internalized and made personal. Much of the current work on group-based learning derives from his thinking. Once again, this model reflects our experience that engineering design is a social activity (Leifer, Koseff and Lenshow, 1995) and design team failure is usually due to failed team dynamics. However, learning failure is usually blamed on the individual. It is increasingly clear that design is a learning activity and that the social nature of learning is promoted in PBL.

Product-Based-Learning and ME210 It is hypothesized that Product-Based-Learning (PBL) methodology and technology have evolved in a manner that will make widespread PBL adoption and assessment financially feasible for undergraduate, graduate and continuing education. PBL is defined as problem oriented, project organized learning activity that produces a product for an outsider. Unlike other versions of PBL that focus only on the project or problem, the product focuses our attention on delivering something of value beyond a "training exercise", something suitable for evaluation by outsiders. The objectives of PBL curricula may be stated briefly as follows (after Bridges & Hallinger, 1995): 1. 2. 3. 4. 5. 6. 7. 8.

Familiarize students with problems inherent in their future profession; Assure content and process knowledge relevant to these problems; Assure competence in applying this knowledge; Develop problem formulation and problem solving skills; Develop implementation (how to) skills; Develop skills to lead and facilitate collaborative problem solving; Develop skills to manage emotional aspects of leadership; Develop and demonstrate proficiency in self-directed learning skills.

Evaluating Corporate Training: Models and Issues

5

The Stanford University School of Engineering course, "Team-Based DesignDevelopment with Corporate Partners." (ME 210) has served as a model for exploring this approach. The seeds of our understanding of PBL have emerged from this design-development laboratory where eighteen years of cumulative evidence is making soft factors like "team personality" ring true statistically as well as intuitively. Objective metrics are being derived from detailed examination of the formal written records produced by design teams in the natural course of doing their job.

Technical focus on Mechatronics in ME210 Our curriculum technology focus is Mechatronic systems design-development (smart products). Mechatronics is frequently defined as the integration of realtime software, electronics, and mechanical elements for imbedded systems. In our definition, we include human-computer-interaction and materials selection. Typical products in this domain include: disk drives, camcorders, flexible automation-systems, process-controllers, avionics, engine controls, appliances and smart weapons. Mechatronics is a particularly good medium for introducing PBL because it is dependent on interdisciplinary collaboration. Implementation of this curriculum builds, in part, on recently developed internet tools and services for distributed product-development teams (virtual design teams). Such teams are increasingly common in the mechatronics field. Using the World-Wide-Web (WWW) as an informal, work-in-progress document archive and retrieval environment, we electronically instrument design-team activity and knowledge sharing for project management and learning assessment purposes. Examples include: •

Share: the MadeFast experiment (URL http://www.madefast.org/), 6 universities, 6 corporations, built a virtual company, delivered a product in 6 months and documented it all on the WWW (Toye et al. 1994);

•

ME210: a graduate curriculum in cross-functional-team mechatronic systems design at Stanford (UTL http://me210.stanford.edu), 12 to 15 companies, 45 students, 15 of them remote, build and document Mechatronic products each year;

•

NSF Synthesis Mechatronics : a multi-university (8) codevelopment project focused on undergraduate mechatronics (NSF Synthesis Coalition web (URL http://www.synthesis.org).

6

Design Team Performance: Metrics and the Impact of Technology

ME210 as a Product Based Learning Model Corporate clients of the Stanford Center for Professional Development (SCPD) who sponsor ME210 projects are increasingly adamant about the need to give their employees a continuing, life-long education opportunity without losing them to full time study on campus. Encouraged by this demand, ME210 has been offered since 1994 to SCPD students across the country and since Autumn 1995 to students and corporations around the world. The course is intensely experiential, hands-on. Distributed design teams composed of 2 to 4 members work on different industry funded projects. They are supported by a four person teaching-team, a practicing design-team coach and a corporate liaison. The 19941995 academic year included Ford, GM, FMC, Lockheed, Pfizer, 3M, Raychem, NASA-JPL, Redwood MicroSystems, Quantum, Toshiba, Seiko and HP Medical Products. The deliverable is a functional product prototype and detailed electronic documentation of the product and development process. These teams won 11 of 12 awards in the Lincoln Foundation Graduate Design Competition in 1995. The competition is based on formal external peer review of design documentation.

Instrumented Team Learning ME210 is supported by a World-Wide-Web based information system created to meet the dual challenges of design knowledge capture and interactive distance education. ME210 events are distributed by video and Internet channels. The focus is on capturing and re-using both informal and formal design knowledge in support of "design for re-design." Video (preferably a digital video server) is used for high-bandwidth, real-time transmission of lectures, design reviews and demonstrations. The internet is used for low-bandwidth data transmission, email, knowledge capture and retrieval. Electronic mail is used extensively for communication between the teaching staff, students, coaches and industry liaisons. Communication is automatically archived and organized by a HyperMail utility on the web. The ME210-Web also includes the class schedule, syllabus, assignments, 5 years of past student work, design-team background information and a HyperMail archive of working-notes. The webarchive functions as a cumulative team and curriculum memory. It facilitates informal knowledge sharing within the class and shares with subsequent generations the legacy knowledge of prior experience. In addition to distance education benefits, the dynamic nature of the ME210Web fundamentally changes the model of interaction between student teams, teaching staff, coaches and industry liaisons. Before the ME210-Web, product

Evaluating Corporate Training: Models and Issues

7

development team progress was only observable at formal meetings and quarterly reviews. With the ME210-Web, work-in-progress is available for review by all members of the community, any time, any place. This option augments and strengthens traditional briefings. It facilitates feedback from the teaching team, coaches and corporate partners to the design team. Importantly, teams can share their "lessons-learned" in real-time. All of this gives our corporate partners a basis for judging the validity of our curriculum and the value of their financial investment in engineering education. The course and the web become a model for high-tech "instrumented" learning. This extensive use of high tech delivery systems has led to formulation of the following PBL assessment instrumentation model. Using the model, performance studies demonstrate that diverse, distributed teams that make effective use of electronic communication and documentation services can, and do, out-perform co-located design teams. To achieve this, several lines of innovation have been introduced systematically in the ME210 curriculum. Major changes since 1990 include the following: : Behavior-based restructuring: 1. 2. 3. 4.

The conception of engineering as a social activity has been emphasized. An open-loft community replaced a closed work room environment. The first 5 weeks of a 30 week development cycle are now devoted to team building exercises rather than product development. Personal Preference inventory scores have been added to the list of diversity factors used to design peak performance teams.

Technology-based restructuring: 1.

Knowledge capture and sharing (WWW access) has been promoted as a deliverable equal in importance to hardware and software. 2. The use of desktop computers for numerical engineering has been deemphasized. 3. The use of laptop computers (mandatory) for design-knowledge capture and sharing and re-use has been strongly encouraged. 4. Conceptual prototyping (physical and qualitative) is advocated and detailed design deferred. Knowledge Capture for Collaborative Learning It is the custom in industry to conserve knowledge through formal documentation. Though this benefits re-design across generations of engineering teams, individual engineers often consider immediate benefits of documentation too small to justify the required effort. Consequently, documentation is treated

8

Design Team Performance: Metrics and the Impact of Technology

as a necessary evil to be done after the fact, after the finer points of decisions and rationale have eroded from memory. While some engineers have such details written down informally in meeting notes and notebook entries, these records tend to be personal accounts which are difficult to share with colleagues during the course of design. Our informal survey of existing computer-supported collaboration tools showed that most information exchange mechanisms are built on a discourse metaphor. Designers in a virtual co-located meeting scenario may use shared white boards, IRC (Internet Relay Chat), e-mail or usenet newsgroups to actively enter into synchronous and asynchronous conversation threads. These tools allow them to extend meetings over space, time and offer opportunities to record on-going discourse. However, as engineers know, meetings are only one part of teamwork. In fact, a significant portion of teamwork occurs in parallel, outside of meeting rooms. Indeed, the strategy of concurrent engineering is to gain lead time by minimizing sequential dependencies and maximizing parallel task flow. To be effective in an engineering team, it is equally important to be aware of parallel tasks without engaging in prolonged discussions. This mode of informal sharing is exhibited when teammates drop into each other's offices, look over each other's shoulder at work in progress, or ask one another, "What are you working on?" While it is possible to record conversations, the inherent purpose of discourse is time-sensitive information exchange, transmit and forget, rather than information archiving. Conversely, thought-based information is primarily intended for one's personal archive and re-use. For effective collaboration, we observe that it is valuable to capture and integrate collective discourse knowledge and knowledge generated for personal use (see Table 14.1). Table 14.1 Comparison of discourse-based and thought-based high tech tools Knowledge Work

PRIMARY AUDIENCE SHARE TENDENCY RECORD TENDENCY ANALOG EXAMPLE DIGITAL EXAMPLE ASSESSMENT VALUE

Discourse Based

Thought Based

others shared NOT RECORDED conversations Email potential

self NOT SHARED recorded personal notes Enotes demonstrated

Evaluating Corporate Training: Models and Issues

9

We have found that Email is the discourse medium of choice for asynchronous dialogue. Notebooks, broadly used as a metaphor, are the thought medium of choice for most knowledge workers. As no single paper notebook serves all application domains, a variety of electronic notebooks have been created. Our particular notebook, PENS (Personal Electronic Notebook with Sharing (Hong, Toye & Leifer, 1995)) fills the need for an ultra-light application that supports and implements www-mediated selective sharing of one's working notes. In review, we have identified three sources of learning performance data that can be used for "work in progress" team assessment. Subsequent sections of this chapter will explore team performance using these materials. Each is available in electronic form and is WWW accessible. Hence, assessment can be performed by anyone, anywhere, anytime. 1. formal reports; 2. discourse-based email; and 3. electronic thinking-notes.

Product Based Learning Assessment Product-based-learning integrates five key pedagogic themes, each central to assessment: 1) externally sponsored projects motivate learning; 2) theory and practice are synthesized in hands-on development; 3) real-world projects demand multi-disciplinary experience; 4) project management requires creativity, problem formulation, teamwork, negotiation, oral communication, and written documentation; 5) naturally occurring by-products of project work (proposals, presentations, lab-notes, products and reports) directly support formative, summative and validative assessment. Each aspect of PBL benefits from computer and internet technology in three ways. First, computer mediated work is more easily shared than paper based work. The need to share information is driven by cooperative learning and by the real-world context of PBL activity. Second, formal presentations and documentation are more easily created and disseminated electronically. Third, electronic documents facilitate learning quality assessment through emerging content analysis (Mabogunje, 1996) and organization communication pattern analysis (Mabogunje, Leifer, Levitt & Baudin, 1995). PBL pedagogy themes map closely to the activity and issues of real product development. Accordingly, the framework for our approach to learning assessment is derived from observational methodology in the design research

10

Design Team Performance: Metrics and the Impact of Technology

community, especially the work of Suchman (1987), Tang (1989) and Minneman (1991) Their findings told us what to look for while the authors' experience in flight simulation research suggested the use of an instrumentation framework. However, in contrast to flight cockpit-team emphasis on precision communication, our design-team observation studies have clearly shown that the preservation of ambiguity through negotiation is a necessary component of productive design environments (learning environments). It is also enabled by computer and WWW mediated communication technology.

Instrumentation Feedback Model for Assessment The guiding metaphor for our assessment and evaluation activity is one of instrumentation. We use this term in the sense of observing both independent and dependent variables in an automatic feedback control environment similar to that found in aircraft flight simulators. The model, illustrated in Figure 14.2, is an adaptation of the visualization first introduced by Atman, Leifer, Olds & Miller (1995). noise student

industry

4

educator

input pedagogy 1

external project content

2

course design

teaching activity 3

learning external outcome project outcome

course outcome

learning activity 5

6

7

formative assessment summative assessment validative assessment

Figure 14.2. Design-Team performance assessment model

The model includes seven instrumentation "Nodes" required to observe most critical input-output relationships. A comprehensive assessment program must be able to observe activity at all Nodes concurrently, a requirement that has only become achievable through advanced technology.

Evaluating Corporate Training: Models and Issues

11

Design-Team performance (learning) is assessed and the assessment itself validated (authenticated) in Product-Based-Learning through three feedback paths. In the validation loop, curriculum outcomes at Node-7 can be compared with industry performance standards across courses and campuses to assure that the curriculum targets relevant professional skills and content. In the summative assessment loop at Node-6, individual projects can be compared with each other to assure that the course is on target. For the inner most, formative assessment loop at Node-5, work in progress can be compared with peer standards to assure that the team is on target. Results obtained from assessment along these three feedback paths supports triangulation across data sources, methods and assessors. In each assessment situation we identify the locus for instrument insertion, the variables of interest and observation methods of choice. The nature of the activity and information structure in the process both influence these choices. Historically, the emphasis has been on output assessment, factors that can be observed at Node-6 (formative-assessment), Node-7 (summative-assessment), and Node-8 (validative-assessment). However, from a flight simulation point of view, it makes little sense to observe outputs without at least partial knowledge of related inputs. Hence, we have focused our effort on instrumenting the input Nodes 1 through 4: Node-1: explicit definition of pedagogic objectives and professional performance requirements, including the description of scenarios and situations in which performance is to be judged; Node-2: course and instructor specific selection of pedagogic methods and continuous comparison of course, team and individual outcomes with peer standards; Node-3: student and team specific assessment of incoming content knowledge and process management experience for continuous comparison with goals and work-in-progress indicators of performance achievement; Node-4: recognition of, and action taken to reduce the impact of uncontrolled, unobservable and simply unknown factors that contribute noise or uncertainty that affects the overall performance and assessment process. By convention, noise is shown as a lumped parameter input to the core teaching-learning process. In practice, noise may enter the system at any Node. Subsequent sections of this paper review something of the impact of technology on our ability to instrument, understand and manage team productivity and learning through the study of input-output relationships in this model.

12

Design Team Performance: Metrics and the Impact of Technology

Metrics enabled by Technology Through longitudinal analysis of noise entering the teaching situation in the form of students' prior experience and personal learning style preferences we have seen that the structure, membership and self management of product development teams can be addressed in new and productive ways. The union of an explicit feedback assessment model and advanced technology for instrumenting and facilitating team activity have begun to yield performance metrics that can be used by a team to monitor and accelerate its own productivity. This information is also available to aid project supervisors, including instructors, in the assessment and prediction of team outcome performance. We report two examples of our experience with metrics that have proven valuable in both understanding and facilitating Product-Based Learning within a collaborative, team environment. The first is a subjective, behavior-based, index for controlling the "noise" of student learning preferences as an input to team performance (Node 4). The other is an objective, content-based, measurement of work in progress to predict the quality of a team's final product. Data from work in progress assessment is used by teams for self-assessment (Node 3) and by project supervisors as an in-course corrective mechanism (Node 2). Both types of metrics depend on advanced technology in the workplace for their implementation and acceptance.

Team-Preference Profile Management The idea of using questionnaires to guide the formation of design teams had its origins in 1989 in a Workshop on Creativity for design professors organized at Stanford by Bernard Roth, Rolf Faste and Douglas Wilde. To determine the effectiveness of the workshop, participants were surveyed both before and after the two-week intensive experience to determine their "Gough Creativity Index" (GCI), a measure of creativity devised by the psychologist Harold Gough. The GCI is an empirically determined linear transformation on the four scores generated by the Myers-Briggs Type Indicator (MBTI), a questionnaire used widely for psychological, educational and vocational counseling (Briggs-Myers & McCaulley, 1985). As reported by Wilde (1993), the workshop enhanced the creativity index of the participants and promised to be a useful guide for the design of effective teams. In 1990 this idea was extended to ME210. After responding to the MBTI, the students were asked to voluntarily form teams composed of individuals whose MBTI scores differed in at least two of the four variables. Subsequent

Evaluating Corporate Training: Models and Issues

13

performance confirmed that teams having a high GCI member in balance with other preference profiles tended to win more and better design competition awards. Initial findings were refined in subsequent years and applied to teams of two, three and four members. Mathematical transforms were developed to use MBTI numerical scores and vector analysis to decompose the score vector into the 2, 3 or 4 components required to guide team formation (Wilde, 1993, September). It was also possible to compute Jungian cognitive modes, one of which corresponds closely to the GCI (Wilde & Barberet, 1995). The MBTI has since been replaced by a much briefer Preference Questionnaire. This saves time and de-emphasizes the counseling aspect of the survey. Students wishing to investigate the counseling side further are encouraged to do the MBTI under the supervision of a licensed counselor. The Preference Questionnaire is viewed simply as an expression of personal preferences rather than an indicator of personality type. Indeed, the data are only used to place students in broad preference groups rather than personality type categories. Teams are then formed in a peer-to-peer protocol that rewards five team diversity factors: 1. 2. 3. 4. 5.

include one person from each personal preference group; balance gender representation; maximize ethnic diversity; maximize technical background diversity; and maximize the members' geographic distribution.

Since 1977, ME210 final reports have routinely been submitted for consideration in the Lincoln Foundation Graduate Design Competition. The reports are read and evaluated by designers and design professors, who award twelve prizes to those they consider best. Judges change from year to year and are blind to the authors and their university affiliations. Since 1991, balanced high GCI teams, have clearly been more successful at winning Lincoln awards, not only more of them, but also of higher quality (Wilde, 1997). Table 14.2 summarizes the awards won during the five-year period of our experiment and the preceding thirteen (control group) years. The table shows that the percentage of prize-winning teams from ME210 doubled, from 29% to 60%, during the experimental period. High GCI teams had a higher winning frequency (63%) than did the other teams, whose performance (50%) had already been improved from 29% simply by diversification. The study shows that the quality of prizes won was also considerably better for high GCI teams, which won a high proportion of Silver, Gold and Best of Program medals (42%) triple that for the other teams (14%). In the best year to date, 1995, student teams won eleven out of the twelve prizes

14

Design Team Performance: Metrics and the Impact of Technology

awarded nationally. Team preference diversification thus corresponds to the quantity of prizes won, an indication that distributing human talent diversity amongst the project teams raised overall quality especially at the top. Table 14.2 Summary of Lincoln Prize Awards High-GCI Team Period 1977-90 1991-95

Yrs 13 5

N 49

Aw 31

% 63

Other Teams N 14

Aw 7

% 50

All Teams N 138 63

Aw 40 38

Yrs = number of years studied; N = number of teams; Aw = number of awards won; = percentage of teams winning awards.

% 29 60 %

The personal preference questionnaire and group transform algorithm have been implemented as World-Wide-Web forms [http://me210.stanford.edu]. Given InterNet access, anyone, anywhere, can complete the questionnaire to identify their preference profile. Another web form is used to characterize the teampreference diversity map. As documented in Hong, Toye and Leifer (1995), WWW mediated peer team formation, using voluntary personal preference data, is one of the most unique and useful applications of technology to team performance management in ME210.

Design Documentation Content Analysis Engineers will, of course, be more at ease with objective, technical content factors that predict team performance. Accordingly, and in response to the increasing number of Engineering classes that require open-ended problem solving, for example, design assignments, we have been studying product design documentation in the interest of supporting design knowledge re-use and grading. Traditional tools for measuring student performance such as written examinations and multiple choice questions are not appropriate assessment instruments in project courses. Grading product documentation is uncomfortably subjective. As the instructional situation moves from one in which there is one right answer and method to one in which there are several alternative answers and methods, there is a need for assessment tools that can accommodate multiple points of view, be context dependent and content specific. Studies of redesign behavior by Baya et al. (1990) demonstrated that designers ask for information about prior designs in terms of noun-phrase clauses. It was

Evaluating Corporate Training: Models and Issues

15

also observed that an automatic content search engine that could use nounphrases in simple associative relationships was far superior to key-word searches in terms of the designer's appreciation of search results. Hence, noun phrases have become the object of attention in recent documentation content analysis studies. As the commentary on technical language usage below demonstrates, these noun phrases may not be good grammar. But as engineering instructors and practitioners will recognize, they are common in practice and may be more important than the commentary by Conner (1968) in "A Grammar of Standard English" would lead us to believe. In the newer technologies - notably in engineering - the (nomenclature) conventions are not systematic or clear; the (engineers) themselves are either unaware of the lack of clarity and system, or do not choose to make the effort to repair it. Therefore anyone who undertakes to read technical documents must make his way through agglomerations like these: • • • • • • •

the highest previously available intrinsic coercive force single side band transmission high frequency stability high-energy particle accelerator internal transducer excitation supply the segmented multiple ablative chamber concept combustion chamber crossover manifold coolant passages

.... This situation will stay with us until the (engineers) establish some firm conventions and hold to them as chemists and mathematicians hold to theirs. As correctly pointed out by the grammarian, noun phrases of this type break several grammatical rules. Viewed differently, they point to the fact that technical language is itself inventive and potentially a rich source of information about design-team performance quality. The raw data for our study of noun phrases (Mabogunje, 1997) comes from a set of three reports, named "design-requirement-documents," submitted by students in ME210. The projects analyzed were taken from the 1992-93 academic year and dealt with a wide range of products including a catheter for gene therapy in the human body, a control mechanism for maintaining the focus of an infrared optical system, and a power locking device for an automobile door. In Figure 14.3, team grades have been superimposed on a graph of the number of distinct noun phrases found in each of their quarterly design-requirementdocuments.

16

Design Team Performance: Metrics and the Impact of Technology

To calculate the level of association between the incidence of unique nounphrases and peer assessment of design quality (the grade), a gamma measure of association was calculated. For the ten project teams studied, gamma was equal to 1.0, a perfect ordinal prediction, for the winter quarter. For the spring quarter, gamma was 0.71 (strong but less than perfect), following intervention by the teaching team to stimulate the performance of slack teams. We found that the number of distinct noun phrases in these documents was already very high in the Autumn report, written 5 weeks into a 25 week develop cycle.

Number of Distinct Noun Phrases in the Quarterly Design Requirement Document 1600

A+

1400

A A+ A A+

.

1200 1000 800 600 400

A A A A A A AAB

A+ AB+ B+ B+

Grade

B 200 0 Autumn

Winter

Spring

Academic Quarter

Evaluating Corporate Training: Models and Issues

17

Figure 14.3 Association between distinct noun phrases and academic grades

Projects assigned the letter grade B or B+ corresponded to reports with the lowest noun phrase counts while teams assigned the letter grade A or A+ produced reports with the highest noun phrase counts. It was not possible to make this sort of differentiation based on the readability of the documents for which the gamma association with the class grade was low (0.36 for the winter quarter and 0.31 for the spring quarter). The gamma association between the grade and the number of words in the document was higher than that for readability but lower than that for noun phrases (0.93 in winter and 0.49 in spring). Hence the number of distinctive noun phrases in formal project documentation appears to be an objective predictor of product quality ranking. Grading is often perceived as highly subjective. It is based on several factors and different events. It did not include objective document content analysis. It may in the future. While detailed examination of the learning implications of these results must be deferred, it is important to point out that the results are, in part, validated by the performance of these same teams in Lincoln Graduate Engineering Design competition where the top awards were given to projects with highest number of distinctive noun phrases.

Summary, Implications and Invitations This chapter has described our experience in the design, implementation and assessment of the Stanford University School of Engineering course, ME 210, “Team-Based Design-Development with Corporate Partners”. A theme that pervades our work is the role of technology both in implementing learning environments and in assessing performance outcomes. Utilization of internet capabilities to facilitate team-based, distance learning has changed the character of the learning environment in profound ways. Learning in PBL environments is applied within the context of the course. Virtual design teams, separated geographically but linked by a variety of technology-based communication tools create actual product protypes. Work in progress data, again technology instrumented, is used for self-assessment as the work of the teams evolves. The metrics produced by this data have been shown to predict quality of the final products. The overall quality of designs produced by these diverse, collaborative teams stands up well against industry standards as reflected in the Lincoln Design Competition.

18

Design Team Performance: Metrics and the Impact of Technology

The data for these studies was available because the community is rewarded for electronic knowledge-capture and knowledge-sharing. Early work on design knowledge re-use had revealed that designers engaged in redesign scenarios are persistent and insightful questioners. Documents that are rich in noun-phrases had proven most useful during redesign and were most easily indexed for automated knowledge retrieval. Successful automation of design knowledge retrieval has, in turn, given us a metric for assessment of performance as revealed in project documentation. Teams working on real job assignments (PBL) naturally produce documents that can be used to assess performance and learning. We have observed that objective measures of these documents, such as the frequency of unique noun phrases can predict the final quality of the team's work as judged by peer review. The key to this kind of assessment is an atmosphere that values, promotes and technically supports "real-time" project documentation. Our experience, tools and metric validation studies only hold for engineering product development teams working in the ME210 community and supported by our knowledge capture and sharing environment. Any extrapolation to other situations, domains and communities requires careful examination. However, we are encouraged to explore the utility of these methods more widely because, in part, the design-team application domain is amongst the most ambiguous and challenging of assessment scenarios . The role of technology in this endeavor is critical. We could not have attempted, let alone performed the objective studies without careful technology insertion. The application of advanced technology was itself based on the advanced technology of video interaction analysis used for direct observation of development teams at work in industry and in our own academic situation. Importantly, the validation of these measurements was made possible by the framework of external peer review in the Lincoln Design Competition. Inevitably, these results pose more questions than they answer. How robust is the noun-phrase metric? Can it be manipulated by individuals and teams without corresponding quality outcomes? Can it or related metrics be used on a day-to-day basis with email and electronic notebook records to provide real-time feedback on work in progress? How do other factors like ethnicity, gender, expertise, geographic-distribution and organizational factors affect team performance? In short, we are encouraged. Our curriculum continues to evolve. New technology is being evaluated. A culture of technology assisted self-assessment

Evaluating Corporate Training: Models and Issues

19

in higher education is in place. Corporate training and life long-learning are now part of our education model and methodology.

Acknowledgments Few, if any, of these ideas are entirely free from the influence of friends, colleagues and students. I would first like to thank the generations of ME210 students who have guided and surprised me through the years as I attempt to understand how they defy entropy, create order and fine machines. Regarding the assessment of their performance, Ade Mabogunje, David M. Cannon and Vinod Baya have been especially valued contributors. Regarding pedagogy and the assessment of engineering education, I am particularly indebted to Sheri Sheppard and Margot Brereton. John Tang and Scott Minneman opened the doors to ethnographic methodology in design research and established the observe-analyze-intervene research framework that has guided many subsequent studies. Perceiving and filling a need, George Toye and Jack Hong created the PENS notebook environment and helped formulate our high-tech assessment strategy. Doug Wilde's diligent exploration of the correspondence between team dynamics and design performance has demonstrated something of the truth in an old assertion that "it's the soft stuff that's hard, the hard stuff is easy."

References Atman, C., Leifer, L., Olds, B., & Miller, R. (1995). Innovative assessment opportunities [video tape]. (Available from the National Technical University satellite broadcast, Fort Collins, CO) Baya, V., Givins, J., Baudin, C., Mabogunje, A., Toye, G., & Leifer, L. (1992). An experimental study of design information reuse. In Proceedings of the 4th International Conference on Design Theory and Methodology (pp. 141-147). Scottsdale, AZ: ASME. Brereton, M.F., Sheppard, S.D. & Leifer, L.J. (1995). How students connect engineering fundamentals to hardware design. In Proceedings of the 10th International Conference on Engineering Design, Prague (WDK 23 Vol 1, pp. 336-342). Zurich, Switzerland: Heurista. Bridges, E.M., Hallinger, P. (1995). Problem Based Learning: in leadership development. (ERIC Clearinghouse on Education Management) Eugene, OR.

20

Design Team Performance: Metrics and the Impact of Technology

Briggs-Myers, I., & McCaulley, M. H. (1985). Manual: A guide to the development and use o f the Myers-Briggs Type Indicator, 2nd Edition. Palo Alto, CA: Consulting Psychologists' Press. Conner, J.E. (1968). A grammar of standard English . (pp. 133-134). Houghton Mifflin.

Boston:

Cutting, D., et al. (1992). A practical part-of-speech tagger. In Proceedings of the Applied Natural Language Processing Conference, Trento, Italy. Harb, J., Durrant, S., & Terry, R. (1993). Use of the Kolb Learning Cycle and the 4MAT System in engineering education. ASEE Journal of Engineering Education.82, (3),. 70-77. Hong, J., & Leifer, L. (1995). Using the WWW to support project-team formation. In Proceedings of the FIE'95 25th Annual Frontiers in Education Conference on Engineering Education for the 21st Century, Atlanta, GA. Hong, J., Toye, G., & Leifer, L. (1995). Personal electronic notebook with sharing. In Proceedings of the IEEE Fourth Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE), Berkeley Springs, WV. Kolb, D. A. (1984). Experiential learning . Englewood Cliffs, N.J: Prentice-Hall. Leifer, L. (1995). Evaluating product-based-learning education. In Proceedings of the International Workshop on the Evaluation of Engineering Education , Osaka, Japan. Leifer, L., Koseff, J., & Lenshow, R. (1995, August). PBL White paper. Report from the International Workshop on Project-Based-Learning, Stanford, CA. Mabogunje, A. (1997). Measuring Mechanical Design Process Performance: A Question Based Approach. Unpublished doctoral dissertation, Stanford University. Mabogunje, A., Leifer, L., Levitt, R.E., & Baudin, C. (1995, November 1-5). ME210VDT: A managerial framework for measuring and improving design process performance. In Proceedings of the FIE'95, 25th Annual Frontiers in Education Conference on Engineering Education for the 21st Century, Atlanta, GA. Minneman , S. (1991). The Social Construction of a Technical Reality: empirical studies of group engineering design practice. (Xerox PARC Technical Report SSL91-22) Doctoral dissertation, Stanford University.

Evaluating Corporate Training: Models and Issues

21

Moll, L. (Ed.) (1990). Vygotsky and Education . Cambridge, UK: Cambridge University Press. Suchman, L. (1987). Plans and situated actions: The problem of human machine communication. Cambridge, UK: Cambridge University Press. Tang, J.C. (1989). Listing, Drawing and Gesturing in Design: a study of the use of shared workspaces by design teams. (Xerox PARC Technical Report SSL-89-3) Doctoral dissertation, Stanford University. Toye, G., Cutkosky, M., Leifer, L., Tenenbaum, M., & Glicksman, J. (1994). SHARE: A methodology and environment for collaborative product development. International Journal of Intelligent and Cooperative Information Systems, 3 (2), pp.129-153. Wilde, D. J. (1993). Changes among ASEE creativity workshop participants. Journal of Engineering Education, 82 (3), pp.57-63 Wilde, D. J. (1993, September). Mathematical resolution of MBTI data into personality type components . Paper presented at the American Society of Mechanical Engineers 1993 Conference on Design Theory and Methodology, Albuquerque, NM. Wilde, D. J. (1997). Using student preferences to guide design team composition. In Proceedings of the American Society of Mechanical Engineers 1997 Conference on Design Theory and Methodology , Albuquerque, NM Wilde, D. J., & J. Barberet (1995). A Jungian theory for constructing creative design teams. Proceedings of the 1995 Design Engineering Technical Conferences, 2, (DE-Vol. 83), pp.525-30.

About the Author Larry Leifer holds a Bachelor of Science degree in Mechanical Engineering (Stanford ‘62), a Master of Science in Product Design (Stanford ‘63), and a Ph.D. in Biomedical Engineering (Stanford ‘69). While at the Swiss Federal Institute of Technology in Zurich his work dealt with neuromuscular control of posture; functional electrical stimulation of hearing; and electro-physiological measures of human information processing On the faculty at Stanford since 1976, he presently teaches global-team based product design-development methodology and develops computational supports services in collaboration with corporate partners. As director of the Stanford Center for Design Research since 1982 he works with anthropologists, educators and computer-scientists to understand through applied ethnography, facilitate through advanced computer-

22

Design Team Performance: Metrics and the Impact of Technology

communication technology, and measure the performance of creative design teams.