Basser Department of Computer Science ... Educational theory suggests that student learning is en- hanced when students pay attention to their own learn- ing.
Supporting Reflection in Introductory Computer Science Alan Fekete and Judy Kay and Jeff Kingston and Kapila Wimalaratne Basser Department of Computer Science University of Sydney 2006 Australia {fekete, judy, jeff, kap}@cs.usyd.edu.au
Abstract Educational theory suggests that student learning is enhanced when students pay attention to their own learning. In this paper, we describe a range of innovative techniques that we use to encourage students to reflect on the state of their knowledge, and on the process by which they acquire it. Examples include providing web-based arrangements for students to practice assessing material based on the criteria we use in marking, and allocating marks in assessment for reflective writing.
1 Introduction This paper describes a range of techniques that we have used in our teaching, in order to encourage students to reflect, that is, to think about their own state of knowledge, method of learning, or process of coding. Our goal is to produce students who are aware of which topics they understand and which they don't, who can identify what errors they make most often, and who can describe the way they tackle learning a new language feature. Some instructors, when they hear this, say "I have enough trouble getting students to write a loop that terminates correctly; any time available should be spent on improving their knowledge of programming, not on their introspective musing on their own learning process." However, evidence suggests that reflection actually enhances the technical mastery acheived. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not m a d e or distributed for profit or commercial advant -age and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGCSE 2000 3/00 Austin, TX, USA © 2000 ACM 1-58113-213-1100/0003...$5.00
The important role of reflection follows from its role in assisting students to develop metacognitive skills: these are the skills involved in thinking about learning and learning how to go about learning successfully. This includes the knowledge and skills required to assess when one knows something. The importance of metacognitive skills is well recognised, as indicated for example by [3, 6, 9]. Additional evidence for the efficacy of reflection is in the Personal Software Process ("PSP") [1], which involves professional software developers measuring their productivity, defect rate, etc, and using this self-awareness to modify their programming practice. There are solid grounds for the belief that PSP leads to a significant benefit in the technical skills of programmers. Thus we consider that reflection is worth encouraging, for its indirect effect on the technical skills and knowledge which are our ultimate purpose in teaching Computer Science. Growing out of this, we have introduced a focus on reflection into several subjects which we teach (to very large classes). This paper describes how we have done this; we hope the techniques will prove useful to other instructors. In section 2, we describe the university structure and the subjects from which our examples are taken. The rest of the paper describes our innovations, divided into three groups. In section 3, we present some techniques used during the normal semester activities, such as programming assignments. In section 4, we show how we use assessment to stimulate and reward reflection. I n section 5, we discuss support for reflection through selfassessment.
2 EducationalContext We teach in a large state university. We typically have 750 or more students in the first semester programming subject, which takes an objects-first approach using a simple language called Blue [2] which is especially designed for novices. The class includes about 250 intending CS majors, 200 engineers, and a wide range of oth144
er students with science and business interests. About 600 students continue to a second semester subject covering inheritance, recursion, parsing and similar topics. Both the first two semesters are taught using a variant of Problem-Based Learning [8, 7], where students work in groups on large, realistic ill-defined tasks, and use these to drive them to consult appropriate learning materials as needed. The third semester subject covers data structures in Java, and is taken by about 550 students (including all E E majors as well as CS ones). It is taught conventionally, with lectures, programming assignments, and closed labs.
3
Reflection in Learning
In this section we show how we can encourage students to think about their own activities as the semester proceeds. We give students detailed statements of the skills and concepts they need to master, so that they can continually check their achievement against our expectation. Also, we expect students to make written plans for their study and programming, and afterwards they check their progress against their own intentions.
3.1
Explicit goals
Within the department we have a detailed sets of objectives for each subject. This is intended so the instructor knows what must be covered, and also to allow teachers of later subjects to know what they may safely build on. For students to reflect on their learning, and also to aid students in separating key requirements from enrichment material, we believe it's important that these goals be told to them explicitly (rather than merely being implicit in assessment weighting, etc). One effective way which allows students to compare their learning with t h a t intended, is to provide an inventory of concepts [5]. In the first and second semester subjects, we do this as a list, where each entry has place for a score indicating how well the student feels they know this material (the scores range from "I've never heard of it", through "I know this well enough to try to apply it" all the way up to "I know this well enough to explain to a friend"). For example, for inheritance, the list of concepts is •
•
Inheritance and creation
• Inheritance of contracts • Calls to super routines
3.2
Reflective diary
Planning one's work is a vital aspect of following a reliable process for development of software. In the introductory subject, we require students to plan by insisting that each group, and also each student as an individual, must produce and submit weekly planning sheets, identifying the activities they will do in the next week. The design of these sheets is based on those used in Wood's pioneering work using P B L in Chemical Engineering studies [8, 7]. In order for the students to "close the loop" and reflect on the planning they did, we require them to also produce a weekly diary entry in which they review their previous plans, and suggest ways to improve future planning. Extensive use of reflective diaries for assessment (in a non-computing field) is described in [4]. In our subject, the diary also tracks the students' confidence in the key concepts being covered. Both planning sheets and reflective diary entries axe submitted through a protoype Web-based interface, which has places for predefined aspects (such as "Things to learn"). The basic requirements of this system were that: The actual creation of planning sheets and reflective diaries on a regular basis should not consume an unreasonable amount of a student's time. Planning sheets and reflective diaries should be filed so that they can be easily managed and reviewed by the student. For feedback and assessment purposes, it should be possible for tutors to obtain a summary of a student's reflection. Here is an example diary entry (which we provide to students as a model for their work). 1. New things learned : Learned how to write a driver class and I can now read and write files using Blue. I can use the book command. 2. Reading done : Read Kay P. 26 - 34 and Kingston Ch. 7 3. Review of last week's plan: I did fairly well. I'm now in a group for Problem 2 and we've made a good start - wrote a structural prototype. I've also wrote a small driver class t h a t read in my name from the terminal and created an Account object and showed it to my tutor.
Why inheritance is needed
• How to get inheritance • Deferred and redefined routines • Polymorphism • Static and dynamic type • Assignment a t t e m p t 145
4. Ideas to improve next plan: I stuck to all my goals ahead of schedule. Maybe I should have added "decide on group roles" to my plan. 5. Group progress : Doing well so far. We've swapped contact details and arranged a meeting time. We've completed a structural prototype and divided our workload between us in such a way that we can all satisfy the marking guidelines 6. Improving group interaction: No problems with interaction so far (still early days). We're looking at the possibility of setting up an email account on GeoCities so we can have a common place to store our group project mail. 7. My contribution to project this week: Showed the others what a driver class was. Helped write the structural prototype with rest of group. 8. My confidence ( from 1 - 10 ) about: • Blue language : 8 • Correctness (pre/post/invar) : 6 • Blue d e b u g g e r : 9 • Unix: 5 • Testing: 4 • Design by contract : 6
culture where most subjects are assessed by an accumulation of points (partial credit) from a sequence of small tasks. Students seem to lack the time management to start on work early enough to be able to revise it, when other subjects have more immediate deadlines. Thus we need to also offer regular deadlines every few weeks, with each task done to some level, and then the students move on. Still, we have found that we can introduce some explicit criteria, announced in advance, for each small task, corresponding to different numbers of points awarded. To demonstrate how this works, here are some fragments from the assessment as announced for a programming assignment (a B-tree) in the third semester subject. Similar criteria are given for other aspects of the work, such as functionality, or asymptotic analysis of the routines. There are three marks for style of code. • 0 if the program is generally hard to understand (eg inadequate header comments, insufficient data hiding, not using the idioms/conventions of the language) • 1 if the intention is clear, but routines are not well-chosen, or there are significant flaws in style (such as inadequate internal comments or redundant code), or the code does not correctly maintain the data structure. • 2 if the routines are well-chosen, and there are only minor flaws in coding style • 3 if the program is well-written throughout
Online tools provide tutors with a summary of the student's efforts, in order to simplify the assessment process. This shows, for each student and group, which weekly submissions have been received. 4
Assessment and Reflection
There are 3 marks for extensions beyond the functionality specified. The mark will reflect both the significance of the extension, and the quality with which it is carried out. An example of a 3 mark extension (if done well) is to design a class with an interface that is accessed by Iterators. A 1 mark extension might be an operation to count the number of keys represented in the mapping.
As all educators know, assessment provides powerful messages to students about what we value. We use assessment in several ways to enhance reflection. We make the criteria used to award marks explicit, so students can examine their own work in advance and discover what more could be done to improve their grade. Also, we award points explicitly for the reflection students do. 4.1
Explicit assessment criteria
Pedagogic theory advocates "criterion-referenced assessment", where each student's grade is determined by their level of achievement compared to explicitly announced standards (without comparison to the rest of the class); that is, it is opposed to "grading on a curve" and to "impression grading". In most implementations, criterion-referenced assessment means setting requirements for each overall grade level. Usually there are ways for the student to demonstrate, for each requirement, whether or not it has been met; there are repeated opportunities to revise unsatisfactory work till it reaches the required level to show a criterion has been met. In our experience, it's difficult to introduce this in a
4.2
Assessment of reflection
We award students points based on reflective activities, in assignments a n d / o r exams. In the first semester subject, the grading includes a component for the planning sheets and reflective diaries. For example, students are told that to get half the individual marks for this component, their submitted individual plans and diaries must have all the following • Most weeks the submission gives some indication of progress (the demonstrated achievements of the past week, such as about a page of code written in the past week) 146
• Most weeks the submission shows a serious attempt at planning, with concrete things to be achieved, each with a demonstrable outcome, like a working routine.
For example, a question to design a class might have assessment instructions as follows. • Assess the choice of publicly visible routines: Here -2 means "Completely unable to solve specified problem", 0 means "Some holes, but on the right track" and 2 means "Fully able to solve the problem" • Assess the presentation of routine headers (including Pre/ Post Conditions, Parameters and Comments for routines): Here -2 means "Terrible (eg, bad names, no comments or pre- or post- conditions, bad choice of parameters)", -1 means "Major problems with most bits", 1 means "Mostly OK, some problems with some bits", 2 means "Everything OK". • Assess the choice of instance variables and there types: Here -2 means "Is missing essential information" 0 means "Has essential information, but also unnecessary information (eg instance variables that do not store information between routine calls, but are only needed locally within some routines)" 1 means "Has variables to store appropriate information, but some have unsuitable type (eg variable for a collection of students whose type is defined as Student)" and 2 means "Has everything needed and nothing else, with correct types"
• Most weeks there is a serious attempt at reflection on past planning, such as a review of the past week's plan, noting how well goals were achieved, and, where they were not, assessment of why and how to improve. For a grade of 7 out of 10, extra conditions must be met, such as "consistent progress", "regular evaluation and testing of code", and a "working (if incomplete) version by week 9". In the second semester programming subject, 10% of the final exam is devoted to a question which asks students to explain how they learned a specific topic. For example "An important topic in this course is inheritance. In about half a page, describe how you learned about this concept during the semester (you should explain what specific difficulties you were faced with, and what resource material you used to overcome these)." The grading of this is based on the extent to which the student provides details that convince the reader that they are describing real events, and the extent to which the student connects events with relevant concepts. 5
Self-assessment
Besides the summative and formative effects of assessment by the instructor, students can reflect on their state of knowledge if they understand the assessment criteria, and apply these criteria to their own work. To encourage students to think about the criteria, we use a substantial web-based system where students can assess sample code and compare the scores they award to our expert judgement. We also use student's exam preparation to explain why questions are asked, and how they are graded. 5.1
Web-based self-assessment
In the first programming subject, we provide some practice questions, each linked to a particular stage of the students development. The questions are typical of exam and assignment work: they require the student to design classes or routines for a particular situation, or to implement a described class or routine.
After the student has evaluated the sample code in terms of the announced criteria, the system displays a side-by-side matching of the scores awarded by the student, with those awarded by an expert (instructor). The student can also view a copy of the sample code with annotations, in the form of hypertext links from points in the code to remarks explaining why these aspects are good or bad. For example, the sample solution for a design might have annotations like the following. • Class Comment: "A Class of books" does not make very clear that this is a collection of (perhaps many) books, and that the emphasis is on changes to the membership of the collection rather than on changes to the individual books. • Creation Routine Comment: Inappropriate internal details included (eg that elements are stored in an array). Instead the comment should speak about the aspects that matter to the client (eg there are no books in the collection) • Instance variables: the variable finishedLoop does not store any useful facts between routine
First, the student is asked to answer the question themselves. Next they can improve their understanding of the assessment criteria, by applying them to several provided answers (each clearly marked as a sample, rather than an ideal!) After seeing a sample answer, the student is able to assess it against stated criteria, by filling in a Web form. 147
calls; it should instead be a local variable in routines that contain a loop • Adding New Book Info: The header does not explain what should happen when there is not enough room to add this new book. Finally, the student is ready to examine their own solution, and reflect on its quality in terms of the criteria they have mastered on the sample codes. 5.2
Annotated sample exams
Handing out sample exam questions is common practice. We have gone further, by giving students an indication with each sample question of why it is asked, and how it would be marked. For example, a question on code reading can be prefaced by the following remarks: "This is your chance to show that you can read and understand small segments of code. You may need to trace the code, working through one line at a time. You may also or instead be asked to state what the output is, or write a comment that describes the purpose of the code, or improve the names of identifiers. A question asking for tracing will get full marks if you give the right value; an answer which is based on going round the loop once too many or few times would get 75%. A question asking for a routine comment requires that you really explain the overall effect, and also describe what each parameter and variable represents." Importantly, the annotations are also linked to the course goals, which are issued in the same set of notes a s the sample exam. Essentially, we make it clear to students that the idea of the exam is to enable us to determine how well they have met the goals. So the code reading exam question includes an explicit annotation referring to the goal that students be able to read small programs.
ments are influenced by many other aspects of our courses. We are happy with the way the classes are going, and find that there is little added work for staff in applying these ideas. Students do not seem happy with the requirement to fill in planning sheets, so we are trying to improve this aspect. On the other hand they appreciate the explicit detail in the assessment criteria. In student evaluations of the courses that use these methods, over 60% of the class answer the question "How clear were the objectives of this subject" with a score of 4 or 5 out of 5. References
[1] Humphrey, W. Introduction to the Personal Software Process. Addison Wesley, 1997. [2] Koelling, M. The Blue language. Journal of ObjectOriented Programming 12, 1 (March 1999), 10-17.
[31
Lawson, M. Being executive about meta-cognition.
Cognitive Strategies and Educational Performance (1984). (E. Kirby (ed)). Lovie-Kitchen, J. Use of reflective journals for assessment in problem-based learning. Research and Development in Problem-Based Learning 4 (1997), 350-355.
[5]
Tamir, P. What do learning theories and research have to say to practitioners in science education?
Research and Development in Higher Education 7 (1984), 162-166. (J. Lublin (ed)). Volet, S., and Lawrence, J. Goals in the adaptive learning of university students. Learning and Instruction (1990), 497-516. (H. Mandl, E. De Corte, N. Bennett, and H. Friedrich (eds)).
[7]
Woods, D. Problem-based Learning: How to Gain the most from PBL. McMaster University Books, 1994.
IS]
Woods, D., Wright, J., Hoffman, T., Swartman, R., and Doig, I. Teaching problem solving skills. Engineering Education 66, 3 (1975), 238-243.
66 Conclusions
We have shown a range of techniques which we use in our classes, in order to encourage students to think about what they know, how they learned it, and how well it matches the announced goals of the subject. These are applied in classes of hundreds of students, and can be introduced without great effort. We do this because educational theory indicates the benefits that follow if we can encourage our students to reflect on their learning processes, achievements and their progress towards the set goals of the course.
Yussen, S. The role of meta-cognition in contem: porary theories of cognitive development. Metacognition, Cognition and Human Performance (1985). (D. Forrest-Pressley (ed)).
It is hard to demonstrate the effect of the methods described in this paper, as any observations and measure148
