teamwork skills in engineering students, a design team ... design projects such as a bridge or a stool. ..... 7) Patton, M.Q., Qualitative Evaluation and Research.
Session 13b2 An Evaluation of a Design Team Facilitator Training Program For Engineering Upperclassmen Daniel Knight, Elaine Seat, William Poppen, Roger Parsons, Gary Klukken and Mark Hector The University of Tennessee College of Engineering/College of Education Knoxville, TN 37996 Abstract - Due to increasing demands for improved teamwork skills in engineering students, a design team facilitator training program was created for engineering upperclassmen. This program is an integral part of the new engage freshman curriculum at the University of Tennessee. The present study is an evaluation of the structure and impact of the program using four criteria: (1) facilitator reactions, (2) facilitator behavior, (3) participant personality, and (4) design team dynamics. Significant results were found for a measure of each criterion. The implications of these results are discussed and summarized.
Introduction In modern industrial settings, the team structure is increasingly being used as a basic unit of work organization [1]. In the field of engineering education, there has been increasing concern that undergraduates are not being prepared properly for working in teams [2]. In response, ABET has revised its requirements to include a teamwork component [3]. To help meet these requirements at the University of Tennessee, members of the College of Engineering have formed a partnership with members of the College of Education. Through this partnership a design team facilitator training program has been developed for engineering upperclassmen. This program is an integral part of the new engage freshman curriculum at the University of Tennessee. The present study is an evaluation of this program using both qualitative and quantitative data. Program Design
meetings, facilitators either run a structured group exercise designed to improve team performance or respond to team problems as they emerge. In the third phase of the class, the facilitators are broken up into small groups of six. These small groups meet with one of the instructors of the class to process the events that have taken place in their freshman team meetings. The goal of these facilitator meetings is to develop solutions tailored to the facilitators' specific teams. There are two objectives to the program. The first is to provide training in teamwork skills for the engineering upperclassmen. The second objective is to provide support for the freshmen students as they work on their first design team and make the transition from high school to college. Program Evaluation The program is in its second year of development. During the fall semester of the first year, the program ran with 18 facilitators and 60 freshman broken up into twelve groups. An assessment plan was developed for the program. Data from this assessment were used to evaluate the program and changes were made to the program based on this evaluation [4]. During the fall semester of the second year of the program, 21 facilitators worked with 150 freshman that were divided into 30 teams. The evaluation plan for the 1999-2000 program is both formative and summative. In being formative, the plan seeks to provide information for the improvement of the program. In being summative, the plan seeks to provide information on the effectiveness of the program in completing its objectives. The evaluation plan includes a mixed-method evaluation strategy relying on both qualitative and quantitative methods of data assessment [5]. The evaluation plan is based on the assessment of four criteria: the reaction of the facilitators to the program, the behavior of the facilitators, the personality characteristics of the participants, and the group dynamics of the freshman teams [6]. The first criterion, participant reaction, is important because of the need to understand the subjective experience of the facilitator trainees. Based on information from the previous year's evaluation effort, facilitator reactions provide a wealth of useful data for evaluating
The facilitator training program is implemented as a class during the fall and spring semesters. Training takes place each week with three phases of training. During the first phase, the facilitators meet in a classroom environment in which basic skills of group facilitation are taught. Various instructional methods are used including lecture, video, and structured role play exercises. The second phase of the program is an applied component where the facilitators are each placed with two design teams and instructed to use their skills to facilitate the teamwork. The teams are composed of freshmen in engineering who are enrolled in a design class that requires design projects such as a bridge or a stool. During the team 0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b2-6
Session 13b2 specific components of the training program as well as assessing the impact of the program on those involved. Some of the more effective methods for obtaining an assessment of subjective experience are qualitative [7], and one such technique was employed for this study. Existential-phenomenological interview methods were used to gain the perspectives of the facilitators. Phenomenological interview methods focus on the investigation of experience [8]. Each facilitator was interviewed individually at the end of the semester and responded to the following question, "What has been your experience of the facilitation class?" The second criterion, the behavioral characteristics of the facilitator, is necessary to provide an objective, quantitative assessment of the facilitation to complement the subjective, experiential viewpoint provided by the qualitative data. To assess facilitator behavior, a behavioral rating form has been developed. The form is designed to measure the facilitation skills that have been taught in the initial phase of the program and implemented with the freshman teams. The form was given to the freshman team members at the end of the semester. It is expected that high scores on the facilitator behavior inventory will be correlated with healthy group dynamics in the freshman teams. The third criterion, the personality characteristics of the participants, is necessary to determine the effect of personality preferences on both facilitator behavior and the dynamics of the groups. Facilitators were assessed at the beginning of the semester on the Myers Briggs Type Indicator (MBTI) [9]. The MBTI generates scores on four dimensions of personality. The first dimension is extroversion/introversion. Extroverts are more sociable and dominant while introverts are more reserved and selfcontrolled. The second dimension is sensing/intuition. Sensing types pay more attention to details and facts while intuitive types pay more attention to abstract ideas and possibilities. The third dimension is thinking/feeling. Thinking types prefer to make decisions based on rational thought while feeling types prefer to make decisions based on emotions. The final dimension is judging/perceiving. Judging types prefer to have things settled and closed while perceiving types prefer to continue to keep matters openended. In the first year of the program, the facilitators had much higher scores on the extroversion scale than the freshman team members. Based on the results of last year's study, it is expected that extroversion/introversion will be the dimension most related to facilitator behavior as well as healthy group dynamics in the freshman team. The fourth criterion, the group dynamics of the freshman team, is necessary to determine the impact of facilitation on the freshman team process. The freshman team members were administered a pre-, mid-, and post-test using two scales from the SYMLOG instrument [10]. SYMLOG is an acronym which stands for Systematic Multi-level Observation of Groups. The instrument
contains a number of scales designed to measure group process. The two group process scales chosen for this study are entitled task orientation and team orientation. Task orientation measures the acceptance of the specifics of the task as presented by authority and the desire to organize the task into the specified product. Team orientation measures the level of cooperation and positive affect toward the teamwork component of the group. Research has demonstrated that more effective teams have higher task and team orientation scores. In previous studies unfacilitated design teams composed of seniors in engineering demonstrated deterioration in team orientation scores across the semester [11]. Last year, facilitated freshman design teams maintained their team orientation scores, but demonstrated deterioration in task orientation scores as the difficulty of the projects increased. More structured exercises targeted toward task orientation were added to this year's program. Based on previous results and modifications in this year's program, it was expected that the freshman teams would maintain consistent task and team orientation scores across the semester.
Method
Twenty-one facilitators and 150 freshmen team members participated in the fall 1998 program. Five of the 21 facilitators were female. All were Caucasian. Forty of the 150 freshman team members were female. Eighteen freshmen were African-American. Five were AsianAmerican. Two were Latin-American. Qualitative and quantitative data were collected in the study. The qualitative data were collected at the end of the semester after all of the training was complete. The quantitative data were collected three separate times, just prior to the presentation of the first, fifth, and sixth freshman design projects during the third, tenth, and sixteenth week of the semester. The MBTI was administered at the pre-test. The Facilitator Behavior Inventory was administered at the post-test. The task and team orientation scales were administered at the pre-, mid-, and post-test. As described, the qualitative assessment relied on phenomenological interviewing. Following the initial question, "What has been your experience of the facilitation class?", the facilitators were instructed to talk about whatever stood out to them about their experience. The interviewer asked follow up questions to clarify and broaden the description of the experience [12]. When the facilitator felt as if he/she had covered everything, the interview was over. Interviews with each facilitator lasted approximately 30 minutes. The interviews were audio-taped, transcribed, and analyzed for themes in the experience. A theme is a pattern of experience present across situations in the transcripts. These themes were organized into a structure representing a verbal description of the experience of the 0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b2-7
Session 13b2 facilitation class. In the write-up of the experience the description of the themes were supported by quotes from the transcripts [13]. The quantitative assessment plan relied on the use of three instruments, the Myers-Briggs Type Indicator (MBTI), the Facilitator Behavior Inventory, and two scales from the SYMLOG group observation system. The Myers Briggs Type Indicator Form G is a 126 item measure of personality. The instrument is a forced choice instrument that asks to choose from opposing responses on an item. The instrument yields scores on four scales: extroversion/introversion, sensing/intuition, thinking/feeling, and judging/perceiving. The possible range of scores is from -60 to +60. All of the scales have demonstrated acceptable reliability estimates and evidence of validity. The Facilitator Behavior Inventory is a 33 item behavior rating form designed to measure the application of facilitation skills in a work team. The instrument has been developed for this study. Team members are asked to rate facilitator behaviors such as "The facilitator is able to help the team to communicate better." Each item is followed by a series of five responses ranging from strongly disagree to strongly agree. The rating form yields scores ranging from 33 to 165. Internal consistency reliability estimates using Cronbach's alpha = .92. Validity work on the scale is ongoing. Two group process scales entitled task orientation and team orientation were used from the SYMLOG instrument where an eighteen item adjective checklist measures both scales and respondents are asked to rate themselves according to how they see themselves in their team. Each of the eighteen items consists of an adjective followed by a series of five responses ranging from "never" to "always". The adjective checklist yields scores with a possible range of -18 to +18. Both the task and team orientation scales have demonstrated acceptable internal consistency and interrater reliability estimates. In addition, evidence for the concurrent and construct validity of the scales has been established [14].
Results Phenomenological Interviews
an evaluation of the program in terms of its differences from the typical engineering class, and (3) an evaluation of the impact of the program on the facilitator's professional and personal development. When evaluating the program as a whole, the facilitators generally reported a positive experience. Facilitators used terms to describe the program such as "good experience", "really enjoyable", "fun", "worthwhile", "useful", but "not too difficult". Another subtheme in the evaluation was an assessment of the class in terms of its differences from the typical engineering class. "It was fun reading, you know as engineers sometimes we don't read words as much as it is formulas. So that was kind of a nice change of pace." Another stated, "I have some friends in other colleges of engineering and I talk to them about what they are doing and...basically in all the classes you just do problems...Wow, I'm so much more prepared than they will be by the time they graduate." However, other students were distressed by the difference in the class and the standard engineering format. One reported, "Sometimes it was really redundant....I don't know if it's just because I'm used to a fast-paced, really hard class, but I'm used to going into a class, getting lectured for fifty minutes and you're done and you go home and do the homework.” Other facilitators spoke of the difficulties in adjusting to the facilitator role. "I've never understood how psychology could be a science because, for me, science is two plus two. I guess it’s a totally different kind of science, that way. It was really hard for me to grasp how they (the instructors) had like forms for this, but you could go into a situation (with the freshman team) and you would have to throw that out the door. You'd have to come up with something totally different...It's just weird to see how people (team members) don't stay in the roles they might be in for the week ahead." Another commented, "In engineering classes they teach us how to problem solve, they want us to look at situations. But they do it with all the same equations. Everything is done the same way. All we have to do is take something that looks different and figure out how it works the exact same way as something else we have already done. So, we don't actually do things all the time different in the same class....I mean I would walk into the (freshman team portion of the) class and I would have this idea in my head of how they (the freshmen) would do it and they would do it completely different." A third subtheme in the evaluation is an assessment of the impact of the program on the facilitators' ability to work in teams. In evaluating the professional impact, facilitators often talked about how the program has affected their ability to work in teams. One stated, "I'd say learning to listen to people's ideas...We sit there and listen a lot of times when they're doing strategy or brainstorming sessions. So, it's taught me that when I'm in teams that other people do have ideas and you can't just shut them out, or close your head to their ideas. You need to listen to them, weigh, and listen
The results of the qualitative analysis of the phenomenological interviews produced five themes that characterize the experience of the facilitator training program. The five themes are as follows: evaluation, groups, learning, doing, and expectations. A full description of each theme, its sub-themes, and supporting quotes is an entire study of its own. In line with the goals of the present study, these results will be focused on one theme, the facilitator's evaluation of the program. There were three subthemes in the theme of evaluation: (1) an evaluation of the program as a whole, (2) 0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b2-8
Session 13b2 to what other people have to say. Not be passive, but not be aggressive either...So, its just the kind of thing that has built up over time, where all of the sudden it's like, you know, its good to listen." Facilitators also talked about how the class material had affected them personally. One reported, "We learned about the Myers-Briggs. You learned what kind of person you are, and now I kind of look at myself in certain situations, and I say, 'I did that because I'm an extrovert,' or 'Of course I did that; I'm intuitive.'...That was fun....just seeing how you interact with people socially." Facilitator Behavior Inventory The range of scores on the Facilitator Behavior Inventory varied from a minimum of 55 to a maximum of 165 with the mean at 127. A mean rating for each facilitator was calculated from the ratings of each of their team members. These mean scores were correlated with the team and task orientation scores for the freshman teams. These mean scores significantly correlated with the pre-test r=.44, midtest r=.40, post-test r=.43, p's < .05. for the team orientation scale. For the task orientation scale no significant relationships were found. Thus, the expectations for the relationship between facilitator behavior and group dynamics were partially supported. Higher facilitator ratings were associated with higher team orientation scores, but not higher task orientation scores. Myers-Briggs Type Indicator MBTI scores were converted to a 120 point continuous scale with extroversion, sensing, thinking, and judging scores as the high points on the scale. Mean scores were calculated for both the facilitators and the freshman team members. A multi-variate analysis of variance procedure found significant differences between the means for the facilitators and freshmen on the extroversion/introversion scale F (1, 159) = 5.35 p < .05. As expected, the facilitators scored significantly higher M = 56.84 than freshmen M = 42.21 on the extroversion/introversion scale. There was no significant difference between facilitators and freshmen on any of other MBTI scales. Table 1 shows the relationship between facilitator MBTI scores, behavior ratings, team orientation scores, and task orientation scores.
Table 1. Correlation Between the Facilitator MBTI Scales and Facilitator Behavior Ratings, Team Orientation Scale, and Task Orientation Scale
Behavior Team Pre Team Mi Team Po Task Pre Task Mi Task Po
Extrovert .65* .32* .29* .30* .08 .13 .09
Sensing .36* .02 .08 .11 .01 .13 .04
Thinking -.17 -.05 .10 .07 .04 .16 .05
Judging .30* .19 .22 .09 .03 .14 .11 * p
