Using Lab Exams to Ensure Programming Practice in an ... - CiteSeerX

Using Lab Exams to Ensure Programming Practice in an Introductory Programming Course Joao ˜ Paulo Barros, Lu´ıs Estevens, Rui Dias, Rui Pais, and Elisabete Soeiro Instituto Politecnico de Beja, ´ Escola Superior de Tecnologia e Gestao ˜ Rua Afonso III, n. 1, 7800-050 Beja, Portugal {jpb,

lfve, rui.dias, rui.pais, elisabete.soeiro}@estig.ipbeja.pt

ABSTRACT The high variability of students’ backgrounds typically found in introductory programming courses creates additional difficulties to foster motivation and ensure the desired competency and retention rates. This paper presents and discusses the results of applying an assessment methodology strongly based upon lab exams (or on-computer exams) with the objective of ensuring programming practice competency while maintaining students’ motivation. It discusses the difficult balances of several assessment methods, details the used assessment methodology and finally presents students results and views. These show that lab exams, although more demanding then group assignments, can be made popular among students if they perceived them as fairer.

Categories and Subject Descriptors K.3.2 [Computers and Education]: Computer and Information Science Education—Computer science education, Curriculum

General Terms Human Factors, Management

Keywords Introductory programming course, CS1, lab exam, on-computer exam, student retention, student motivation, object-oriented programming

1.

INTRODUCTION

Introductory programming courses for computer science students are well known for their student attrition rates [7]. Some studies [1, 8] have pointed out students’ lack of fundamental problem solving skills as the main reason for attrition and weak programming competency. We agree that weak problem solving skills is one of the main reasons for

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ITiCSE’03, June 30–July 2, 2003, Thessaloniki, Greece. Copyright 2003 ACM 1-58113-672-2/03/0006 ...$5.00.

student attrition. Yet we also believe that it belongs in the same group as weak mathematical thinking capabilities [3] and, in general, to weak prior preparation for a computer science course. From our experience we believe that two additional and complementary causes for student low achievement and attrition are: an excessive dependency on group work and insufficient assessment opportunities. These can even lead students to fraudulent behavior. Based on these experience and on students’ opinions, we decided to deeply modify the assessment and grading system with three main objectives: increase programming practice; decrease fraud and dependency on other colleagues and decrease students drop out by increasing recoverability opportunities. Next section presents the course background and details the motivations that led us to deeply modify the course assessment and grading system. In Section 3 we discuss the pros and cons of several pondered assessment techniques and Section 4 presents the chosen ones. Section 5 presents the course results and the students’ views based on classroom experience and a students’ survey. Section 6 summarizes the benefits we found in the new assessment rules, but also some drawbacks that we were able to identify. From these we present some conclusions emphasizing the learned lessons and pointing out future enhancements to the proposed assessment rules.

2.

BACKGROUND

Our institution offers one bachelor course in computer science. The course is offered in two parallel editions: a daylight one and a night one. The former is attended, almost exclusively, by full-time students coming from high school; in the latter enrolled students are mostly part-time students with different kinds of jobs. The respective introductory programming course had been using C++ since their inception in 1997. In reality it consisted of two courses, one for each semester: in the first (C1), C++ language was used to teach structured programming; in the second (C2), students learned to build programs consisting of a few classes, eventually using dynamic memory allocation. The assignments and assessment followed the tradition in similar courses in the curricula: three to five open lab assignments to be completed by groups of two to three students (students had to submit the results after one week or more); and a final written exam based almost exclusively on code writing. During the several course editions we noted that many

students were practicing much less than they were supposed to as they excessively relied on their group mates. There were also numerous students’ complains about the alleged course difficulty, especially in C2 where only 2 to 3 open lab assignments were proposed. No formal surveys were conducted at the time, but the several teachers concluded, based on students view and other anecdotal evidences, that the higher student retention and success in C1 compared to C2 were due to three factors: 1. The first course had more assignments which facilitated student recovery; 2. Students’ background was of some help in C1 but not in C2; 3. There was an excessive dependency on group work in C1 and even more in C2. In 2001 the school conducted a major restructuring of the entire curricula. This provided the right context for some radical changes in C1 and C2. The following section discusses the several options that were considered at the time.

3.

THE COURSE

From the beginning it was clear that we could increase programming practice and hopefully increase success rates by redefining course content and the assessment and grading rules. Course content was modified mainly due to the existent gap between C1 and C2. This resulted, to a large extent, from the procedural to object-oriented programming transition. This motivated the change to an objects-first approach. After some survey in literature, BlueJ [5] came out as the best environment for our purposes and Java came together. Java has also the added advantage of increasing student motivation, as it is a well-known language that they typically believe they must know. As such we now have C1 and C2 following an objects-first approach using Java and BlueJ.

3.1

Assignments

When deciding about which assignment methods to use we considered written exams and three additional on-computer options: Written exams - each student has to complete, on paper, exercises related to code reading and writing; Open lab assignments - each student completes the assignment on their own time, and has to turn it in after a week period or longer; Close lab assignments - each student completes the assignment in the lab and turns it in at the end of the class; Lab exam - each student completes the assignment in an exam like environment. This means special time and space allocation, as well as rigid time and fraud prevention control. We evaluated the relative value of these four assessment methods when a single instance is used and when several

Table 1: Assessment methods evaluation: w - written exam; *w - multiple written exams; o - paired open lab assignment; *o - multiple paired open lab assignments; c - paired closed lab assignment; *c multiple paired closed lab assignments; l - lab exam; *l - multiple lab exams w *w o *o c *c l *l Fraud control 5 5 1 1 3 3 5 5 Assessment breath 1 3 5 5 3 5 3 5 Student perspective 1 1 5 5 3 3 2 2 Worplace resem- 1 1 5 5 4 4 2 2 blance

are considered. We based the evaluation solely in classroom experience and our own. The evaluation was expressed between 1 and 5 using a relative ordering where 5 is better (Table 1). Note that the grading from 1 to 5 only intends to specify a relative ordering among assessment methods adequacy regarding each of the 4 evaluated items. These were fraud control (plagiarism and cheating) where we also include the possibility of an excessive dependency on group work; assessment breath which somehow measures the simplicity to assess all course contents; student perspective based on our own perception of students’ opinion, resulting from classroom experience and some informal conversations with students; and a comparison with workplace scenarios. We believe students’ perception of assessment methods value is the result of a comparison with workplace scenarios. This justifies the similar scores. Although we believe group assignments are of the up most importance for computer science students, we also think an exception should be made in introductory programming courses, as individual assignments force students to effectively test their own abilities and motivation. This is particularly true in our bachelor courses where every year we find students who do not really know what computer science is. Some are even shocked to find they have to study math. This wide gap among students’ motivations and reality is particularly acute in freshman students and easily leads the less prepared ones to illusory easy ways to complete the course. Our previous anecdotal evidence had clearly showed that numerous students managed to complete C1 by excessive dependency on other colleagues’ work. This was made possible by group assignments. We also found that presentations with individual questions to each group members were of little use as most students managed to understand quite well the delivered assignments yet failed on the written exam. This was a consequence of the difference between understanding solutions and getting there by themselves. We decided to abandon group open lab assignments in the introductory programming courses, as they could not guarantee the student would be the only responsible for the assignment completion. Paired closed labs were rejected due to the same reason. This convinced us to replace assignments by lab exams. While one lab exam has some known advantages [2], we thought that only multiple lab exams could provide a rigorous and progressive students’ self-assessment along the semester. So as to avoid increased class hours, we decided to use half of the classes in a week for the lab exams. As

students have two hours labs, two each week, we managed to use one of the labs by splitting the class in two and giving forty-five minutes to one hour for each lab exam. This was done in six weeks (part of the 15 weeks for classes in each semester) for the six lab exams. The retakes were conducted outside classes.

3.2

Grading

Based on the fact that, regarding fraud control, the exam labs are as rigorous as the written exams, we felt comfortable to increase their weight in the final grade. This brought the advantage of answering conspicuous students claims regarding labs low weight in the final grade. We maintained the final term written exam now with a weight of 40% against the older 50% to 60%. The labs are now worth 60%. Yet, the biggest difference is not in the relative weights of labs exams and the written exam but in the way the labs grade is obtained. This is presented in the next section.

4.

THE LAB EXAMS

We based our lab exams on the proposal by Jacobson [4] but we made several modifications, namely:

1 - the program does not produce the correct output but it follows the style rules and is near the correct solution; 2 - the program produces the correct output but either does not follow the style rules or is significantly more complex than necessary 3 - the program produces the correct output, fully respects the style rules and is simple enough. Students are not forced to achieve a specific grade but are allowed to retake any or all of the lab exams to improve a grade of 0 or 1. This was decided on the grounds that students should be allowed to choose the exams where they would invest more, based on the workload of other courses in the semester. We also leave out the highest and the lowest lab exam grade. The objective is twofold: to favor students’ regularity, and to forgive a poor lab exam score or a missing lab exam eventually due to a sporadic high workload. Overall the multiple exams fulfill the objectives stated in Section 1 by forcing individual practice, increasing student motivation and self-assessment opportunities, as well as teachers’ feedback along the semester.

1. We maintain coding in the final written exam; 2. Some assignments imply the writing of an entire program, although necessarily small; 3. Students are offered a total of six lab exams instead of five; 4. We grade the lab exams between 0 and 3, instead of 0 and 2, and we not require a minimal grade in any lab exam; 5. Students are allowed, but not forced, to retake any lab exam; 6. The maximum grade in a retake lab exam is 2 and it always replaces the corresponding lab exam grade; 7. The lower and the higher lab exam grades are not used and the labs grade is the total of the remaining four; 8. A zero grade is attributed to each non-turned in exam. The rationale behind maintaining code writing in the final exam was to evaluate students in an environment where trial and error is simply not possible. Writing of an entire program in some of the first C1 assignments gives students some context to the more complex assignments in which they only write parts of a program due to timing constraints. We believe this helps student motivation as it allows students to feel they know how to make programs and not only how to modify them. Yet this is only possible for the first simpler assignments. The lab exams were always open book. We decided for six lab exams as a way to cover all course contents in a more balanced way. The lab exam grade was expanded, relative to [4], to distinguish a fully working program with deficiencies from a (near) perfect one. The rationale for each grade was the following: 0 - the program does not produce the correct output and either is not fully compliant with the style rules or the tentative solution seems to be very far from the correct one;

5.

RESULTS

To evaluate the modification in the assessment and grading rules, we collected data from all editions of courses C1 and C2 in the first and the second semester of the academic year 2001/2002. This totaled three C1 courses and two C2 courses with 205 students. Two of the C1 courses and one C2 course were given in the night course with far less students. Although these were older part-time students we found no significant differences in the success rates. Next we present the average retention and exam grades along the semester, as well as some students’ opinions based on classroom experience and a survey questionnaire.

5.1

Students Results

Figure 1 illustrates the retention rate along the semester as the percentage of students completing the exams relative to the number of students who completed at least one assignment. We use the number of students who completed at least one lab exam as a reference so as to screen out a large number of students (typically 30% to 50%) who are enrolled but cannot be counted as students as they miss all classes, assignments and exams. We must also stress that students’ high school grade was very low, almost identical to the minimum acceptable to enter the bachelor course as this accounts for these students low grades. Due to the already stated rule, the higher and the lower scores were not counted. This meant that some students, having completed four lab exams could miss the last two, with little consequences, or the last with no consequences in their final lab grade. For example, a student with grade 2 in all labs could afford to have a zero in the last exam, as it would be the lower score to be left out. From our perception this justifies the higher number of students in the written exam compared to exam lab 6; and also the average grades decrease in the last labs, followed by an increase in the written exam average grade (Figure 2). Unfortunately, this fact can have induced a less significant preparation to the final written exam. It was not possible to quantify this assumption but we think it might have contributed to the final failure rate (Table 2).

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 2 lab exams

3 lab exams

4 lab exams

5 lab exams

6 lab exams

written exam

Figure 1: Retention as the percentage of students who completed the referred number of lab exams and the percentage who completed the final mandatory written exam. The percentage is relative to the number of students who completed at least one lab exam. 3,00

2,50

2,00

1,50

1,00

0,50

Table 3: Students’ opinion. The sentences were classified in a Likert scale: 1 - totally disagree; 2 - disagree; 3 - no opinion; 4 - agree; 5 - totally agree. Average Median Mode Written exam should be 4.5 5 5 open book Lab assignments should be 3.5 4 4 individual Lab assignments should 4.0 4 5 weight more than written exam in the final grade On average, the course was 4.0 4 5 more demanding than others in the same semester Table 4: Students’ opinion. The sentences were classified in a Likert scale: 1 - very significantly contributed to a worse final grade; 2 - significantly contributed to a worse final grade; 3 - it did not significantly affected the final grade; 4 - significantly contributed to a better final grade; 5 - very significantly contributed to a better final grade. Average Median Mode Lab exams 3.4 4 4 Written exam 3.5 4 4

0,00 Lab 1

Lab 2

Lab 3

Lab 4

Lab 5

Lab 6

Written

Figure 2: Average final grades in each lab exam and on the final written exam. Compared to the open lab based assessment, there was no significant improvement in the attrition rate as measured by the number of students present in the final written exam. Yet we believe this is still an improvement for now we have a similar retention rate but with the advantage of ensuring additional programming practice. The success rates after attrition (the only ones we have enough data to compare) were only marginally better (7% on average).

5.2

Students View

To get additional feedback from students, we also distributed a small questionnaire. 64 students responded in a total of 205. This section presents the students’ view based on classroom experience and questionnaire results. From classroom experience of the four teachers involved, it was clear that students preferred the lab exams to the previously used open lab group assignments. They particularly praised the perceived increased fairness resulting from the numerous assignment opportunities. One student said the lab exams gave them more work along the semester but in the end they were not forced to study hard before the final written exam. In the questionnaire, we asked students about their agree-

Table 2: Total of students, average lab grade, average written grade (from 0 to 3) and success rate. Quantity Labs Written Passed 205 1.38 1.22 42%

ment with several sentences related to the course (Table 3). As we expected a large majority of students prefer open book exams and favor lab assignments over written exams. Students think assignments should be individual. This seems to indicate that the ”student perspective” score on lab exams, in Table 1, should be 4 instead of 2, as they favor them over group assignments. Students perceived the course as more demanding then other courses. As discussed in the next section, we attribute this to the retakes popularity. We also asked about lab exams and final written exam perceived contributions to the obtained final score (Table 4). From this we draw the conclusion that students favor lab exams. Even students who failed the course give a 3.0 average score to lab exams.

6.

DISCUSSION

The preference for individual lab exams was clearly supported by students opinions expressed in the questionnaire and on classroom experience. We also found that only students that perceived the course as too demanding showed preference for non-individual lab assignments. We believe these students would be the real victims of the group assignments as they would be too dependent on other colleagues work and would be more easily tempted to some kind of fraudulent behavior. The massive students’ preference for open book written exams and the consistently weaker results in the written exam than on the lab exams seem to indicate that the lab exams trial and error inherent possibility is favoring students’ results. This should be fight by a more careful design of the lab exams leaving fewer opportunities for trial and error solutions. This would allow the written exam to evaluate other fundamental skills as code reading, code understand-

ing and code modification. Testing these skills can have the bonus advantage of decreasing ”too little time” complains. We also reached the following conclusions: • Students liked individual tests, the fact they did not have to worry with the worse score, and the opportunity to retake exams. There were considerable less complains regarding the lab assignments than on previous years where open lab group assignments had been used. • The written exam should continue to be open book just like the lab exams and should assess different capabilities, namely code reading and code understanding. • The increased number of lab exams and the retake possibility helped students’ continuous self-assessment along the semester. It also effectively assisted teachers in tracking students’ progress, allowing corrective measures to be taken. • Weights should be introduced as a way to fight the low popularity of the last assignments. In fact, only these can evaluate more advanced concepts and their neglecting worsens the preparation for the final written exam and the overall acquired competency. • Although fundamental, retake opportunities should be reduced. To our surprise students decided to complete, on average, an additional 2.5 lab exams as retakes. This meant they had to complete 8.5 lab exams in 15 weeks of course classes. This was clearly more than expected and turned out as a bit unpopular among colleagues teaching other courses as they though students were spending too much time with C1 and C2. Besides, this fact significantly contributed to the perception of a more demanding then average course and probably contributed to worse scores as students saw the first take as a way to see what they should expect and then were happy with the maximum retake score of 2.

7.

CONCLUSION

We found the use of lab exams useful for ensuring student competency in the assessed contents. This was especially relevant, as we had consistently observed students misleading themselves by the excessive dependency on group assignments. Especially significant was the fact that lab exams clearly improved student motivation. This was testified not only by their stated popularity when compared to group assignments, but also by students’ view that lab exams based assessment helped them achieve a better grade. Additionally, the multiple lab exams, including the retake possibility, were able to avoid an attrition increase while slightly increasing the success rates.

In future course editions, and besides the already pointed enhancements, we intend to allow extra credit through an optional part of each lab exam, as already proposed by Eric Roberts [6]. With this we hope to fight a less numerically significant but more insidious kind of discouragement: the one affecting better prepared students who find the course too undemanding. We also intend to evaluate the use of computer based assessment systems for the lab exams scoring. That can allow teachers additional time for student attendance, which would compensate for the time spent on exams instead of lab classes.

8.

ACKNOWLEDGMENTS

The authors thank their colleague Isabel Sofia Brito for a really useful comment on an draft of this paper. The authors also express their thanks to all the students who provided feedback on the assessment methodology.

9.

REFERENCES

[1] T. Beaubouef, R. Lucas, and J. Howatt. The UNLOCK System: Enhancing Problem Solving Skills in CS-1 Students. ACM SIGCSE Bulletin, 33(2):43–46, 2001. [2] Mary Elaine Califf and Mary Goodwin. Testing Skills and Knowledge: Introducing a Laboratory Exam in CS1. In Proceedings of the 33rd SIGCSE Technical Symposium on Computer Science Education, pages 217–221. ACM Press, 2002. [3] P. B. Henderson, D. Baldwin, V. Dasigi, M. Dupras, J. Fritz, D. Ginat, D. Goelman, J. Hamer, L. Hitchner, W. Lloyd, J. Bill Marion, C. Riedesel, and H. Walker. Striving for Mathematical Thinking. ACM SIGCSE Bulletin, 33(4):114–124, 2001. [4] Norman Jacobson. Using On-Computer Exams to Ensure Beginning Students’ Programming Competency. ACM SIGCSE Bulletin, 32(4):53–56, 2000. [5] Michael K¨ olling and John Rosenberg. Guidelines for Teaching Object Orientation with Java. In Proceedings of the 6th Annual Conference on Innovation and Technology in Computer Science Education, pages 33–36. ACM Press, 2001. [6] Eric Roberts. Strategies for Encouraging Individual Achievement in Introductory Computer Science Courses. In Proceedings of the Thirty-First SIGCSE Technical Symposium on Computer Science Education, pages 295–299. ACM Press, 2000. [7] Matt Roddan. The Determinants of Student Failure and Attrition in First Year Computing Science. http://www.dcs.gla.ac.uk/~roddanmc/Report.pdf. [8] Mack Thweatt. CS1 Closed Lab vs. Open Lab Experiment. In Selected Papers of the Twenty-Fifth Annual SIGCSE Symposium on Computer Science Education, pages 80–82. ACM Press, 1994.