Self-Predicted and Actual Performance in an Introductory ... - UCSD CSE

Self-Predicted and Actual Performance in an Introductory Programming Course Paul Denny Andrew Luxton-Reilly John Hamer Department of Computer Science The University of Auckland Private Bag 92019 Auckland, New Zealand

Dana Dahlstrom

Helen Purchase

Department of Computer Science and Engineering University of California, San Diego 9500 Gilman Drive La Jolla, CA, USA

Department of Computing Science University of Glasgow Glasgow, Scotland

ABSTRACT Students in a large introductory programming course were asked twice to predict their scores on the final exam: once at the beginning of a six-week module, and once at the end. In between, students in only one of the two lecture streams recorded subjective confidence in their answers to individual questions on weekly quizzes. Students’ predictions were moderately correlated with their scores. Students who attended more quizzes had not only higher exam scores, but improved their predictions more than those who attended fewer quizzes. Practice recording confidence on individual quiz questions did not yield significantly more improvement in exam predictions. Several findings from previous work are confirmed, including that women were significantly more underconfident than men.

Categories and Subject Descriptors K.3.2 [Computers and Education]: Computer and Information Science Education—Computer science education,Self-assessment

General Terms Experimentation, Human Factors, Measurement, Performance

made more accurate and that it would be worthwhile to develop techniques for doing so [16]. Investigators in metacognition have distinguished micropredictions, in which subjects judge how likely they are to be correct on each question, from macropredictions of overall performance [20]. The main question motivating the work we report here is whether, in the context of an introductory programming course, students who regularly recorded micropredictions during quizzes would improve their ability to make macropredictions of their final-exam performance or to answer the quiz or exam questions correctly. Ancillary to the main thrust, we also investigated other basic questions: how informative and well-calibrated students’ microand macropredictions were, and how quiz attendance (likely to reflect lecture attendance in general) affected exam scores and macropredictions. Previous work showed perceived self-efficacy at the end of a programming course was better correlated to performance than was self-efficacy at the beginning [22]; we also generally expected predictions to be better at the end. Previous work also suggests women are less confident (and less overconfident) than men in computerscience courses specifically [1, 9], and in general, particularly for tasks perceived as congruent to male sex roles [17, 13]. Accordingly we expected similar findings regarding gender. Our research questions can be summarised as follows:

Keywords

1. Quiz micropredictions: When students are more confident on an item, are they more likely to be correct? How informative and well calibrated are micropredictions, and does this vary by gender or quartile?

learning, metacognition, confidence, gender

1.

INTRODUCTION

2. Exam macropredictions: How did students’ late exam predictions differ from their early predictions, and how well did each predict actual exam scores?

The ability to self-assess what one does and does not know is a metacognitive skill that plays a key role in learning through, for example, allocation of study time [15, 2]. Research has shown less-skilled learners not only make more errors, but are less able to distinguish when their answers are and are not correct [11]. Earlier work has suggested a learner’s “feeling of knowing” can be

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ITiCSE 2010 Bilkent, Ankara, Turkey Copyright 2010 ACM X-XXXXX-XX-X/XX/XX ...$10.00.

3. Attendance and performance: What is the relationship between the number of quizzes attempted and exam scores? Between quiz scores and exam scores? 4. Effects of micropredictions: Does recording item-specific confidence estimates on weekly quiz questions improve performance, or macropredictions of exam performance?

2.

METHOD

The Engineering Computation and Software Development course (ENGGEN 131) is compulsory for all first-year engineering students at the University of Auckland. ENGGEN 131 consists of a

Table 1: Quiz-response data by stream (n is students). 8 a.m. stream 10 a.m. stream quiz

n

µ correct

1 2 3 4 5 6

169 158 130 123 127 133

38.3% 51.9% 44.2% 27.5% 33.2% 62.4%

total

840

43.3%

µ correct

µ confidence

309 295 253 249 227 167

40.8% 54.9% 49.9% 28.3% 43.1% 67.5%

63.6% 60.2% 55.7% 49.3% 49.8% 60.9%

1,500

46.4%

56.8%

n

6-week MATLAB-programming module followed by a 6-week Cprogramming module. Students attend weekly lab sessions, work on individual projects, and sit a test (worth 5%) at the end of the MATLAB module, for which they receive marks before the start of the C module. Lectures are three days per week in two separate, nominally interchangeable streams (at 8 a.m. and 10 a.m.). Lecture attendance is not compulsory and students may attend either stream, although most gravitate to one or the other. We collected data during the C module in 2009. During the module, at the beginning of each Friday lecture, quizzes were held consisting of 5 multiple-choice questions on concepts covered that week. Questions were projected overhead so the time for each question was the same for all students. Students’ responses were scanned immediately after the lecture, and the same day, each student received e-mail showing each quiz question and whether their answer was correct. Correct answers were not published; students who had answered incorrectly were encouraged to attempt again to determine correct answers for themselves. In addition to answering quiz questions, students at the 10 a.m. lectures (but not at 8 a.m.) recorded their confidence in each answer using the following scale: (A) I’m very (90–100%) confident in my answer (B) I’m moderately (70–90%) confident in my answer (C) I’m somewhat (50–70%) confident in my answer (D) I’m slightly (30–50%) confident in my answer (E) I’m hardly (