Session S2G PORTFOLIO USE IN SOFTWARE ENGINEERING EDUCATION: AN EXPERIENCE REPORT Richard L. Upchurch1 and Judith E. Sims-Knight 2 Abstract—This paper discusses the use of an electronic portfolio in a software engineering course at the University of Massachusetts Dartmouth. The Learning Portal provides the standard function of such systems for students to post their work. It also provides a syllabus function so instructors can post all information and assignments to the Portal. It goes beyond these basic functions, however, to facilitate reflective practice. It allows both students and faculty to give feedback to student work and it collects various types of student work, including survey forms that require students to reflect upon their work. It also provides functions for team interaction. In this paper we will describe how the electronic portfolio was used in this course, including what artifacts were captured and how students used the system. We conducted an interview study of students after they finished the course to ascertain how they felt the portfolio changed the way they learned, the issues they encountered in working within such an environment, and their perspectives on how such a support system might influence their behavior in the future. Index Terms—Assessment, Portfolios, Software Engineering
I. INTRODUCTION Software engineering, like the other engineering disciplines, faces increasing pressure to provide students with those skills required both to succeed in their first professional job and to engage in lifelong learning. This is a curriculum design challenge that requires considerable deliberation. The curriculum ultimately designed must meet the litmus test of accreditation provided by the accrediting agency. ABET, in Engineering Criteria 2000 [1], requires that all engineering programs demonstrate their effectiveness by assessing eleven student learning outcomes. These outcomes include the expected engineering related goals such as “an ability to design a system, component, or process to meet desired needs,” and more ambitious goals such as “a recognition of the need for and an ability to engage in lifelong learning.” This transition to accreditation based on how well students can demonstrate both knowledge and skills is a reflection of the movement in higher education from teacher-centered to student-centered learning. The teachercentered tradition has been the cornerstone of higher education, with engineering education merely adhering to the dominant doctrine, for what seems an eternity. The teacher-centered model characterizes students as products 1 2
[2]. Three assumptions are associated with the teachercentered model [3]: 1. An(y) educational process is considered culturally neutral as well as linear and rational. 2. Language serves as a conduit for the transmission of information. 3. The teacher becomes the “manager” of the classroom with the learning process heavily dependent upon the pronouncement and enforcement of rules. This theory of learning is that instructors fill students, the vessels, through knowledge-telling activities. Students demonstrate that they have learned by relating the contents of the vessel to the instructor. The shift to a learner-centered approach is fueled by a number of instructional research projects [4][5][6] demonstrating that students learn better in active learning environments. Constructivism is the theory of learning that informed these studies. In constructivism, the learning process is conceived as an activity in which new information is linked to prior knowledge Thus, the things we “know” are tentative, and are refined and changed through assimilating or constructing knowledge through activities such as self-explaining, writing, interacting with others and with other ideas. In addition, knowledge is more than the contents of a vessel; it includes declarative knowledge, interrelated structure of constructs and how-to skill. The implications for teaching practices are enormous. Students connect what they are learning to their own experiences and current knowledge, thus making their learning conceptually coherent and personally meaningful. The key teaching practices include opportunities to reformulate and articulate newly found meanings. This activity is critical to successful learning. What seems apparent is that the transition from teachercentered to learner-centered education requires 1) new ways of engaging students, 2) new considerations of what engagement means, and 3) new methods for determining if this engagement is effective. Results from research studies provide some guidance in this regard: (a) students who reflect on what they are learning learn better both on declarative and procedural tasks [7],
Richard L. Upchurch, Computer and Information Science, University of Massachusetts Dartmouth, N. Darmouth, MA 02747,
[email protected] Judith E. Sims-Knight, Psychology Department, University of Massachusetts Dartmouth, N. Darmouth, MA 02747,
[email protected]
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S2G-1
Session S2G (b) most students do not naturally do this [89], and (c) inducing students to reflect upon learning material is effective [10-11]. Reflective practice has also been identified in research on the acquisition of expertise. Those practitioners who develop into the most skilled experts engage in deliberate [12] and reflective practice. In the teacher-centered model, instructors review student work products with little regard for the underlying mental processes. A typical grading scenario is to check the answer, if correct mark then go to the next question else check work. The implicit assumption is that if the product is correct then the process that produced it is sound and if the product is incorrect then informing the student should be enough to correct the process. This assumption is wrong in two ways. First, correcting the product does not ensure that the students' understanding is corrected. Second, getting correct answers does not ensure that students are developing processes that will be effective in life-long learning. Research has shown that students who focus on their performance (getting answers correct) tend to demonstrate less perseverance and lower achievement than students who focus on their learning (process rather than product) [13-14]. Students with learning orientation tend to focus on their metacognitive processing and therefore are more likely to develop lifelong learning skills [15]. With this is mind we seek a strategy that helps expose students' mental activity and processes required in their learning, to make these processes visible to them (and to the faculty, so that the faculty can promote its development). To do so we must change from the typical adversarial relationship between teacher and student imposed by the teacher-centered model to a collaborative model where the student and the teacher have a shared vision for learning.
II.
PORTFOLIOS
Many have suggested portfolios as a strategy for increasing the visibility of student learning. According to [16], the advantages of portfolios are that they are longitudinal and collaborative, and have diverse content. A portfolio is a collection of student work that tells the story of achievement or growth [17] across a variety of content and skill areas. Students who construct portfolios in different classes throughout their undergraduate careers will have a collection of work akin to professional portfolios, which tend to be collections of diverse content representative of skill and competence. Portfolios, and their electronic counterparts, have been recognized as a viable tools to support learning and assessment in higher education [18-22]. Portfolios have much greater potential, however, than just helping with career development. Properly constructed, portfolios can help (force?) students to focus on their learning processes. For example, process documents such as work plans, records of the effort associated with
producing products, process postmortems, and improvement plans (see our work in software process education [23-25]) can be included. A learning portfolio's purpose is to provide visibility to ongoing cognitive processes and development. Since our perspective here is developmental, we adopt the cognitive apprenticeship model [26] within a continuous process improvement model. Under such a model the mentor, perhaps the instructor, coaches students by providing feedback and critiques to help students assess their learning, and to develop understandings about the nature of their knowledge and skill. Students need to reflect not only on their work but also on the process by which they worked and the assessments provided both by their own assessments of the process (process postmortems) [27-28]. The acquisition of factual knowledge can be assessed and evaluated at a single point in time, but tracking the conceptual change or development, as is required for procedural and metacognitive skill, requires a longitudinal approach. We can't, and shouldn't, expect more than incremental changes in short periods of time. Our experiences with this pedagogical paradigm led to a realization that an electronic portfolio is a critical component in the type of learning environment we need for our students. It provides the longitudinal record of work, process postmortems, and feedback from peers and instructor that serve as the data from which students can improve.
III. SOFTWARE ENGINEERING The software engineering course is a required course in the Computer and Information Science Department. The course is part of the requirements for both computer science and computer engineering majors, and is taken during their senior year. The content of the course is typical of such courses in the range of topics [29]. The intent is to cover the software development lifecycle including those practices associated with large-scale development, e. g., configuration management and software quality assurance. The course has both a lecture and laboratory component. The course requires a project. The project is a semesterlong team project. The lab for the course supports the project portion of the course, and provides a common time for project team members to meet to plan and organize, as well as meet with the instructor to review progress. Groups of 4-6 students work together to specify and build a product during the semester. Each project had a real customer with a real need. Student teams worked with the customer to determine the requirements. The instructor chose one student from each team to be project manager. The project leaders met with the instructor and received a draft problem statement typically described as a statement of work. The project managers reviewed the resumes of those enrolled in the course and specify which individuals they wished to have on their team. The team selection was a round-robin
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S2G-2
Session S2G process. During the selection activity project managers were asked to specify why particular people were chosen. Thus, the project manager has to defend a choice based on project criteria and the contents of the resume. Each person on the team was assigned a process role, e. g., quality assurance, for the duration of the project. The process roles (quality assurance, configuration management, project planning, project tracking) were distributed among team members through negotiations between the team members and the project manager as were the development activities, e. g., design and coding.
IV. THE LEARNING PORTAL IN SOFTWARE ENGINEERING The Learning Portal [30], an electronic portfolio system developed locally, supports two major classes of users: students and teachers. The Learning Portal provides functions for students to post their work. Student access the Portal to complete assignments either through a web-form or uploading documents. All material submitted during the course, other than examinations, are available in the Learning Portal. Material in a student's Portal has a collection of properties. One important property is visibility. Using the visibility property students can choose to make their work available to other students (or just to themselves or to themselves and the instructor). The visibility function provides a handy way for students to archive versions of their work. The Learning Portal environment allows students to capture a longitudinal record of the development of ideas through the artifacts produced. The visibility feature allows a document to be personal until it reaches a stage when the student is prepared for it to be viewed by others. Alternately, students may record ideas and ask a peer to provide a critique early in development. This visibility feature facilitated the institution of reviews as a standard part of the course for it facilitated the sharing of work. In addition student reviewers post their reviews of the work of others, which in turn is seen by the authoring student as feedback. Hence, the review becomes part of the developmental record for the student. Thus, at the end of the process the portal displays the original document, review of that item, and the reformulation based on the review. The feedback functionality is available to both students and instructors. The feedback function provides the user with a web-form, either structured or narrative. This form can be designed specifically for an assignment (such as producing a rubric for analyzing particular assignments) and associated with the particular assignment in the system. Instructor functionality includes managing the course and viewing student work. One particular feature is assignment management. The instructor creates assignments, provides information related to assignment, establishes the due date, and can specify the form for
students to complete in response to the as signment as well as the form used for feedback. All instructor feedback is accomplished using the feedback feature. Just as in the student review, all instructor remarks, given as feedback, are captured as part of the student portfolio. The instructor can access student work either by viewing a particular assignment, or by viewing the work submitted by a particular student. Recall that the only material visible to the instructor is the material that has been explicitly marked as such by the student. The Learning Portal is designed to make team activity (see [31] for a more thorough description of the teaming portion of this course) easier. We created a project page for each team, so there was a common area for work-inprogress and a mechanism for feedback from peers. The intent here exceeds standard configuration management functionality. We were interested in providing functionality that would support teams as they engage in the activities of product development rather than merely providing document management. Making project work available from a web application means that students can access work from any web browser at any time. Furthermore, the system provides a handy archive so the group's work is available at the end of the semester for the project postmortem.
V. INTERVIEW STUDY We were interested in whether students see a value in using a tool such as this, particularly in its support of learning. Our conjecture was that students would see how a longitudinal record could benefit their planning and progress. Most of the computer science majors had used some variant of the Learning Portal in other courses during their undergraduate years. The software engineering course provided the opportunity to contrast students using the Portal for the first time, the computer engineering students, with the computer science students. We used an open-ended interview format so as not to bias students' responses (see Appendix I). The software engineering course enrolled 43 students of which one dropped during the semester. The 42 remaining students consisted of 12 computer engineering majors and 30 computer science majors. We selected 10 students at random from each group to participate in the interview. We collected protocols from 9 students, 3 computer engineering and 6 computer science students. A research assistant from the Psychology Department conducted the interviews and recorded the responses. The authors reviewed transcriptions of the interviews. Four benefits were mentioned by at least 4 students. First, they appreciated the organization provided by the Portal, in its syllabus function as well as in archiving assignments (mentioned by 8 of the 9 students). Second, they found it facilitated their team functioning (6 students). Third, they liked being able to review their own work (4 students). Fourth, they liked being able to provide feedback
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S2G-3
Session S2G and get feedback easily (4 students). Fifth, they appreciated being able to access the Portal from any browser (4 students). The students' biggest frustrations all related to accessibility (the Portal was unavailable for two periods in the semester, one caused by the University networking system and one caused by a hacking) and to certain features of the interface. None of the students were frustrated by the system as a whole nor by the requirement to use it. Although the interviewees were generally positive about using the Portal, they said little that would indicate they appreciated the system as a reflective tool. Even the computer science majors who had used a portfolio system in prior courses showed little evidence of internalizing reflective practice, although four of them appreciated the power of reviewing one’s work to assess change. There was no mention in the computer engineering protocols of the issue of conceptual change or progress, and how the portal might affect such learning outcomes. This is a somewhat disappointing finding, but perhaps not surprising considering that both the Portal and the instructor's use of it are still in early stages of development. A second issue that we examined carefully was whether students adapted the Portal to other learning situations. Of the nine students in the interview study three of them had used the Portal to organize and archive work in other courses. All of these were computer science students. Though there was little discretionary use, all students reported their support for using a system such as this in other courses. The remarks indicated that the system helped them organize their work, and made a number of arduous tasks easier to manage, including support for group process required for the development project.
VI. CONCLUSIONS The Learning Portal with its longitudinal record of work and attitudes was designed as a course-based portfolio to creating a learning environment that promotes reflective practice. It not only provides a place for students to store their assignments, but it structures their work by assignment type to make it easy for them to review their progress over a number of assignments. These assignments can be products (programs, plans, etc.), self-assessments or reflective essays [28]. The review can include feedback from faculty and from other students. In addition, the documents and feedback can be made visible to one's team. Our interviews indicated that the majority of students appreciated the archiving and team facilitation aspect and that many also valued the feedback and reviewing capability. These are the Portal tools that make reflective practice possible. That there was little evidence in the interviews that students had adopted reflective practice is not surprising., to have discussed their reflective practice in this open-ended interview, students would have to have both an explicit knowledge of their reflective practice and a vocabulary to
discuss ti. This course required peer reviews of project documents and initial and final reflective essays [see 28], but it did not explicitly teach students how to engage in reflective practice. The students may have actually improved in reflective practice, but th is was not assessed by the interview. The interviews and our analysis of the student-system interaction during the teaching of this course indicate a number of areas requiring increased attention. The functionality to support team projects was woefully inadequate. The strategy we employed, which worked reasonably well for individuals, did not accommodate groups. The primary problem is shared-ownership. The initial design of the Learning Portal identified the artifact space and its management to be the responsibility of the student, hence privileges belong to the individual posting the artifact. In a collaborative context, such as the team project, the issues of ownership and responsibility are not as clear. An artifact that a student provides as part of the group should be reflected within the individual’s portfolio, yet available as part of the group’s collective. Neither the instructor nor the student should be required to navigate around the multiple artifact spaces to find an individual's collection of material. A second, and related, issue is the handling of reviews and/or feedback. Currently a feedback/review is only posted within the reviewee's artifact space as feedback. Students engaging in the review activity as a reviewer should have the review they provide as part of their artifact collection automatically. It would be an important activity for a student to reflect on the caliber of reviews they provide. Students need experience in critiquing their products to recognize how their process and products change over time. The review process is a critical component of their learning, thus requires more explicit attention. In summary, the interview study of the Learning Portal indicates that students found it valuable, particularly for helping them to organize their work schedule and their documents and in coordinating team functions. A sizeable number also appreciated the feedback function and the ease of access. Although the interviewees for the most part failed to see the value of the Portal to close the continuous improvement loop, it seems likely that as the design and functionality improves, the Learning Portal can become a major tool by which students can develop reflective practice.
VII. REFERENCES [1]
[2] [3]
Accreditation Board for Engineering and Technology, Inc. "Engineering Criteria 2000: Criteria for Accrediting Engineering Programs." 1999 Bowers, C. A., & Flinders, D. J. Responsive Teaching, Teachers College Press, New York, New York, 1990. Catalano, G. D & Catalano, K. C., “Transformation: From TeacherCentered to Student-Centered Engineering Education,” Frontiers in Education Conference, Pittsburgh, PA, 1997.
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S2G-4
Session S2G [4]
[5]
[6]
[7] [8]
[9]
[10]
[11]
[12]
[13] [14]
[15]
[16]
[17] [18]
[19]
[20] [21] [22] [23]
[24]
[25]
Scardamalia, M., Bereiter, C., and Steinbach, R. “Teachability of reflexive processes in written composition,” Cognitive Science, 8, 1984, 173-190. Schoenfeld, A. H. Problem solving in the mathematics curriculum: A report, recommendations, and an annotated bibliography, (MAA Notes #1). Washington, DC: Mathematical Association of America, 1983. Brown, A. L., & Palincsar, A. S. “Guided cooperative learning and individual knowledge acquisition,” In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser. Hillsdale, NJ: Erlbaum. 1989, p. 393-451. Pirolli, P. & Recker, M. “Learning Strategies and Transfer in the Domain of Programming,” Cognition and Instruction, 12, 1994. Chi, M., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. “Selfexplanations: How students study and use examples in learning to solve problems, ” Cognitive Science, 13, 1989, p. 145-182. Berardi-Colletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. “Metacognition and problem solving: A process-oriented approach,” Jounral of Experimental Psychology: Learning, Memory, and Cognition, 21, 1995, pp. 205-221. Chi, M., de Leeuw, N., Chiu, M., & LaVancher, C. “Eliciting selfexplanations improves understanding,” Cognitive Science, 18, 1994, pp. 439-477. Bielaczyc, K., Pirolli, P., & Brown, A. L. “Training in selfexplanation and self-regulation strategies: Investigating the effects of knowledge acquisition activities on problem-solving,” Cognition and Instruction, 13, 1995, 221-252. Ericsson, K. A., Krampe, R., & Tesch-Römer, C. “The Role of Deliberate Practice in the Acquisition of Expert Performance,” Psychological Review, 3, 1993, pp. 363-406. Dweck, C. S. "Motivational processes affecting learning, " American Psychologist, 41, 1985, 1040-1048. Button, Scott B. & Mathieu, John E. “Goal orientation in organizational research: A conceptual and empirical foundation,” Organizational Behavior and Human Decision Processes, 67(1), 1996, pp. 26-48. Ford, J. Keven, Smith, Eleanor, M., Weissbein, Daniel A., Gully, Stanley M., & Salas, Eduardo "Relationships of goal orientation, metacognitive activity, and practice strategies with learning outcomes and transfer." Journal of Applied Psychology, 83(2), 1998, 218-233. Yancey, K. B., “Portfolios in the Writing Classroom: A Final Reflection,” in Portfolios in the Writing Classroom, Ed. K.B. Yancey, Urbana, Ill:, National Council of Teachers of English, 1992. Sharp, J. “Using Portfolios in the Classroom,” Frontiers in Education Conference, Pittsburgh, PA, 1997. Rogers, G. Williams, J. “Electronic Portfolios as a Tool for Assessing Student Outcomes,” Frontiers in Education Conference, San Juan, Puerto Rico, November 10-13, 1999. Rogers, G. & Williams, J. “Asynchronous Assessment: Using Electronic Portfolios to Assess Student Outcomes,” ASEE Annual Conference, Charlotte, NC, June 20-23, 1999. Olds, Barbara M. "The Use of Portfolios for Outcomes Assessment," Frontiers in Education Conference, Pittsburgh, PA, 1997. Mourtros, N. "Portfolio Assessment in Aerodynamics," Frontiers in Education Conference, Pittsburgh, PA, 1997. Stanford Learning Laboratory “Learning Careers,” http://sll.stanford.edu/projects/hlc/index.html. Upchurch, R., & Sims-Knight, J. E. “Integrating Software Process in Computer Science Curriculum,” Frontiers in Education Conference, Pittsburgh, PA, November 5-8, 1997. Upchurch, R., & Sims-Knight, J. “The Acquisition of Expertise in Software Engineering Education,” Frontiers in Education Conference, Tempe, AZ, November 4-7, 1998. Upchurch, R., & Sims-Knight, J. E. “Designing Process-Based Software Curriculum,” Proceedings of the Tenth Conference on
[26]
[27]
[28]
[29] [30] [31]
Software Education and Training, Virginia Beach, VA, April 13-16, 1997. Collins, A., Brown, J. & Newman, S. “Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics,” in Knowing, learning, and instruction: Essays in honor of Robert Glaser, L. B. Resnick, (ed.), Erlbaum :Hillsdale, NJ, 1989, pp. 453494. Upchurch, R., & Sims-Knight, J. E. “In Support of Student Process Improvement,” Proceedings of CSEE&T'98, Atlanta, Georgia, February 22-25, 1998. Upchurch, R., & Sims-Knight, J. “Reflective Essays in Software Engineering,” Frontiers in Education Conference, San Juan, Puerto Rico, November 10-13, 1999. Upchurch, R. "CIS 480 Software Engineering,' http://www2.umassd.edu/cis3/coursepages/pages/cis480/outline.html. Upchurch, R. & Sims-Knight, J. E. "The Learning Portal," ASEE, Albuquerque, NM, June 2001. Sims-Knight, J., Powers, T. & Upchurch, R. "Teams in Software Engineering Education," Frontiers in Education Conference, Boston, MA.
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S2G-5
Session S2G
Appendix I: Interview form for Learning Portal follow-up Format • All sessions audiotaped. • Proceed from general to specific questions. • Take notes as participant talks. Interviewer has to decide whether follow-up questions have been fully answered. • Ask your own follow-up questions to clarify, follow a line of thought, etc. Question Number 1 1. "Tell me how you use the learning portal. How have you incorporated it into your learning?" Follow-up questions. Separate this discussion into two parts--one about problems they had and one about how they used the Portal. If they harp on the problems, do that first and then say something like "Let's now talk about other aspects of the Portal--when it wasn't giving you trouble." Follow-up to how they use it. . "How was it helpful?" "How was it different from the way you previously worked?" • "Has your opinion changed over the course of the semester?" • "Have you used it in ways other than its required uses?" Follow-up to problems . . . or, if they haven't raised issue, "What problems have you had using the system?" NOTE: there were two major "attacks" on the system, one in September when network services made a change that cut out the Portal and one recently when a hacker attacked the server over Thanksgiving. We want to know what other problems they had and whether they overcame those problems. • "How do you deal with the fact that sometimes the Portal has been unavailable? What do you do?" • "How does that affect your feelings about the Portal?" Questions 2-5 (refer back to earlier answers, if appropriate). 2. "What was the best thing about using the Learning Portal?" 3. "What was your biggest frustration in using the Learning Portal?" 4. "Would you want to use it in other classes?" "Would you be upset if you had to use it in other classes?" 5. "Would you use the Learning Portal on your own?" "How would you use it?" If not included in answer, "would you be interested in using the Learning Portal to prepare a portfolio for job applications?" "Think for a moment about the way you have learned in this class using the Portal. Would you say that you have learned more or in a different way from having the portal assignments?" (Their Portal assignments were concept maps). "Think about next semester's courses. Will you go about studying differently than you did last semester?" If they say yes, explore how they will change. If they say no, explore why they don't feel it would be worthwhile to change. There would be two categories of responses, at least--because it would be too much trouble without the Portal or something about the assignments per se.
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference S2G-6
