Session S2A ADMINISTERING, SCORING AND DEBRIEFING TEAM DEVELOPER Elaine Seat1 and T. Paul McAnear 2 Abstract Team Developer is a computer-based peer rating instrument that allows team members to provide each other with feedback about their team behaviors. Team Developer has been used in the College of Engineering at the University of Tennessee since 1999 as a team building tool with courses having as few as two teams up to courses having 450+ students on 96 teams. This paper discusses the practical use of Team Developer from set up, administration, scoring, and debriefing. Assessment applications and lessons learned will also be discussed. Index Terms Peer-Review, Team Developer, Teams, team evaluation.
INTRODUCTION Evaluation and assessment are all part of a day’s work for educators. Engineering educators are familiar with assessment of individual performance. However, with the advent of teams in the classroom and teamwork as a part of the final evaluation procedure, the nature of assessment changes. Assessment of team performance not only needs to evaluate the tangible end product of the team, but also the performance of individuals in the group in achieving the end product. Evaluation and assessment suggest to the instructor how students are learning, and also provides feedback to the student as to their individual learning. Exams and project evaluations do not provide information and feedback about an individual’s performance in the team, and thus, new methods of providing this feedback are necessary. Courses with a team component for engineering students suggest that not only are students learning a technical skill, they are also learning how to work in a team. The skill of working in a team is not learned by simply being placed on a team, but is one that is learned through instruction and feedback. Teamwork is a learned skill. Teamwork skills are behaviors, not the usual cognitive processes of the typical engineering course. Instructors know what skills their students bring to a course by the learning objectives of the pre-requisites for that course. Students have a minimum common skill set. However, psychologists suggest that teamwork skills, or team interaction behaviors, are learned from the family dynamic of that student. Thus, there is no common skill set of team behaviors that students bring to their team-bases courses.
1 2
EVALUATING TEAMWORK SKILLS Teamwork skills can be evaluated by observation of the instructor. However, the persons in the best position to evaluate those skills are fellow team members [1]. For example, if a team member claims that they are not included in outside team meetings, the instructor may not be privy to information such as that student has such a busy outside calendar that they are not available for outside meetings, or that they simply have not shown up for outside meetings. The important factor in team skills is whether or not the team members can work together and with each other’s style, not the instructor’s evaluation of performances. Peer evalutation in group situations allows team members to each provide feedback to the other members about how that member is perceived. Peer evaluation provides feedback to a student about how fellow team members assess their performance and has been shown to improve team behavior [2]. Feedback has been shown to be most effective when it specifies particular behaviors for possible change[3]. The Team Developer [4] is a peer review data collection and feedback instrument for use in assessing team behaviors. Team Developer itself is a software program that allows team members to rate themselves and their other team members on team behaviors. Developed by psychologist Jack McGourty, the instrument is simple, and uses descriptors for behaviors in terms that are common to engineers and engineering students. The behaviors that are described are in the context of a task team and are described in terms such as: Listens attentively to others without interrupting Applies logic in solving problems rather than personality trait descriptors used in some peer evaluation instruments [5] such as: Circle the best choice describing (person): extroverted, outgoing, positive affectionate, likeable, fun to be with From the position of the evaluator, students (and instructors) are typically more comfortable evaluating other students observable behaviors than evaluating their feelings or personality characteristics. From the position of the evaluatee, our experience has been that engineers and engineering students respond on the more concrete level of behaviors that can be observed and described. For instance,
Elaine Seat, University of Tennessee, Engineering Fundamentals, 101 Estabrook Hall, Knoxville, TN 37996
[email protected] T. Paul McAnear, University of Tennessee, Engineering Fundamentals, 103 Estabrook Hall, Knoxville, TN 37996
[email protected]
0-7803-6669-7/01/$10.00 © 2001 IEEE October 10 - 13, 2001 Reno, NV 31st ASEE/IEEE Frontiers in Education Conference S2A-6
Session S2A students in engineering classes have trouble figuring out how to be more affectionate, likeable and fun to be with, but are better able to create an action plan to improve their ability to listen attentively without interrupting.
THE TEAM DEVELOPER The Team Developer peer rating instrument provides evaluation in four categories: Communication, Decision Making, Collaboration, and Self Management. The Team Developer Administrator version provides for setting up, scoring, and reporting results. The Student version simply runs the survey and gathers the peer rating inputs for each question. The administrator (usually the course instructor) can choose which questions should be included from a predefined list, allowing the survey to be as short or long as deemed appropriate. The wording of the questions may also be edited to better fit the audience. This ability to choose and to edit questions is important for two reasons. First, the length of the survey must be altered depending on the teams and the context to match available time. Secondly, the audience may not understand some terminology and editing is necessary. For instance, we used the original question Encourages frequent polling among team members with freshman teams. During the debrief, students asked us what the question meant because they did not understand the use of the word polling. In subsequent administrations, we had the choice to either not use the question or to reword it. Once Team Developer has been prepared, each team member takes the survey. It is completely computer based and upon starting, the student finds their name in an alphabetical list of all persons in this administration, enters a password, and the survey starts. After a screen that tells the student about what will be surveyed, each question appears with the student and all team members listed below it as shown in Figure 1.
answered for all persons, the student saves and exits. The data is saved for later scoring. Each team member is provided a feedback sheet after scoring that provides them with their self score for each question, and the average of the score their teammate gave them. The scores for the average in each category are also given so that there is an average self and team score for Communication, Decision Making, Collaboration and SelfManagement. Figure 2 provides a sample of the feedback sheet returned to team members.
FIGURE 2 TEAM DEVELOPER FEEDBACK REPORT
SETTING UP TEAM DEVELOPER
FIGURE 1 TEAM DEVELOPER SAMPLE SCREEN
The person taking the survey rates their personal performance and each of their team members and then proceeds to the next question. When the last question has been answered, the student is prompted to redo any question skipped or person not rated on any question before they are allowed to exit the program. Once all questions are
The administrator (usually the course instructor) sets up Team Developer by choosing the questions to be used in each category and inputting the teams and their corresponding team members. To set up Team Developer, the administrator starts the Administrator version, selects either an existing group of teams or creates a new group. When working with our freshman teams, each semester we simply copy the Freshman master team to a new team named Course Name Semester Year Administration (ie EF102 Spr 01 01) without any team members. This action simply starts a new peer evaluation set with our standard questions. To perform a second administration to the same set of teams, we copy an administration that has the questions and teams and simply rename it. For example the second administration
0-7803-6669-7/01/$10.00 © 2001 IEEE October 10 - 13, 2001 Reno, NV 31st ASEE/IEEE Frontiers in Education Conference S2A-7
Session S2A would copy the previous example and be named EF102 Spr 01 02. Once the question set is prepared, the administrator then inputs the teams and the team members. This input is accomplished by manually entering a team member’s first name, last name, and team name. The team name is used by the program to choose which person’s names to group together for display and scoring. Figure 3 shows the team entry screen used by the administrator to input teams and team members.
access to the freshman program computer labs, we make floppy disks for distribution to each person and ask that they return them the following class period. This procedure has difficulty in that we have disk problems in approximately 20% of the survey takers. Some students find their disk unreadable, while others take the survey and then we are unable to read their disk upon return. When we administer Team Developer to our freshmen, we have between 350-450 students on 65-96 teams (5 to 6 students per team). We make a list of the student’s Section, Team Number and Name, and then create a directory for each student on the freshman engineering server (ie, Mark Smith of Section OE, Team 5, would be OA_5_Smith_Mark). The image files are copied into each directory and students simply go to their personal directory and run the WTD.EXE file. The freshmen students take Team Developer at the conclusion of a class period in the computer lab. Team Developer requires little introduction. We direct survey taking as follows: • The student is directed to the WTD.EXE file and asked to run it • The student is asked to find their name in the alphabetical list of all team members • The student is asked to log in using the password and the last four digits of their Social Security Number. • Students begin taking the survey. • They are allowed to leave class when finished.
SCORING TEAM DEVELOPER FIGURE 3 TEAM DEVELOPER TEAM & TEAM MEMBER ENTRY SCREEN
Once the names and teams are input, the administrator makes a disk image. The disk image is a group of files that are to be either copied on to a floppy disk or into a unique directory for each student or team member. The survey taker then just uses their floppy disk or finds their unique directory, and runs the WTD.EXE file to start Team Developer.
Team Developer is scored when all team members have completed the survey. One of the caveats of peer evaluation is that all team members must have taken the survey to have their scores input for averaging. To score Team Developer, the administrator selects Tally Results and selects the student directory (or floppy disk). Team Developer uses its database of teams and team members to find the scores of the other team members for use in providing the average team rating for each item.
PROVIDING STUDENT REPORTS ADMINISTERING TEAM DEVELOPER Administering Team Developer requires that each team member run the WTD.EXE file in the directory of files created when the administrator created the disk image of that team or group of teams. Then the administrator must get that directory back for scoring. At the University of Tennessee, we use with senior capstone project teams throughout the college and with the freshman in our Engineering Fundamentals program. We use two methods of administration with these groups. When we have a small number of teams and team members (
