Lessons from On-line Programming
Examinations
Denise M Woit
David V Mason
School of Computer Science, Ryerson Polytechnic University Toronto, Ontario, Canada M5B 2K3
School of Computer Science, Ryerson Polytechnic University Toronto, Ontario, Canada M5B 2K3
[email protected]
dwoit @ scs.ryerson.ca 1. ABSTRACT We have been using a combination of conventional and on-line programming examinations for the past 3 years in our first year programming courses. We plan to continue using on-line components because of the positive feedback from both students and faculty. We describe the operational and pedagogical lessons we have learned, illustrating problems encountered and our solutions, in the hope that others considering on-line examinations may also benefit from our experiences. 1.l Keywords Automated marking, on-line examinations
2. INTRODUCTION In addition to conventional methods,we administer on-line tests and examinations to approximately 150 first year computer science students in order to evaluate their practical skills. The on-line exams have been successful from the point of view of both students and faculty, as reported in [4]. When the on-line testing experiment began 3 years ago, we hoped it would provide data to assistus in answering the following operatiorial and pedagogical questions. ODerational
what we have learned, in the hope that it may benefit others
considering on-line examinations. Although others have used on-line componentsin course-work and examinations, we could not find any reports of on-line computer programming examination experiments with which to compareours [l, 2, $6, 71.
1. Could we create a secure testing environment (no interstudentcommunication)? 2. Would such an environment be flexible enough for examinations’? 3. Would students attempt covert communications during examinations?
3. OPERATIONAL
LESSONS
Below, we present the operational lessons learned, as they relate to the questions of Section 2.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
3.1 Security and Flexibility [Questions 1,2,3.] Originally, testing was done within a Unix environment, using the chroot system call to provide airtight security. A detailed description of the setup and
ITiCSE ‘98 Dublin, Ireland 0 1998 ACM l-581 13-000-7/98/0008...
4. Could good automatedmarking programsbe created? 5. Should the on-line component be administeredusing no paper (question sheet,etc.)? Pedagonical Would students feel more motivated to learn a course’s “practical skills” component when such skills are to be testedon-line? Could on-line examinations help us distinguish between students who achieved good marks through their own diligent work, and students who achieved good marks by copying assignmentsand memorising code fragments for the examinations? How would the marking time for on-line tests compare to that of conventional tests? Would an automatic marking program assign marks in an equitable, reasonablemanner? lO.Could on-line testing be as useful as conventional testing? We have collected data over a 3-year period to help us answer these questions. Data was collected by surveying students, faculty involved in the on-line testing, and faculty teachingthe studentsin subsequentsemesters.Details of the early surveys (questions, responses,and conclusions) have been presented in [4]. Through administering the on-line tests and analysing the data collected, we have learned several lessons, both operational and pedagogical, from both studentsand faculty. Below, we present a summaryof
$5.00
257
usage of the testing environment is available in [3]. Our data showed that 1. the environment allowed no breachesof security, and 2. that only 5% of students even attempted to exploit covert paths, with commandssuch as telnet and email, because they were putting all of their energies into answering the test questions. Data to support our findings was taken from the shell commandhistory files that were maintained throughout the on-line test, for each student. Because of these positive results, we expect to be able to relax security somewhatthis term, which will allow us to provide a graphic test environment. We will continue to collect data in order to evaluatethe successof this new environment.
3.2 Automatic
4. PEDAGOGICAL
LESSONS
Below, we present the pedagogical lessonslearned, as they relate to the questions of Section 2. Our early lessons are presentedin Section 4.1; the most recent lessonsin Section 4.2.
4.1 Motivation
of Students
[Questions 6,7.] As reported fully in [4], both student and faculty surveys showed that on-line tests did motivate students to learn practical skills, largely because the students believed that those who simply copied and memorisedwould achieve lower marks with on-line testing.
4.2 Better Evaluation
of Students
(*
[Question 10.1 Originally, we had expected on-line examinations would be better than conventional at evaluating some student skills. Upon analysis of our data we concluded that, overall, on-line testing is no better: however, for someskill sets, on-line testing makesit easier to create good test questions, and to evaluate student responses. The skill and knowledge sets that we identified were: design, problem solving, testing, debugging, manual program execution, and language knowledge. We recognisedthat in conventional tests,we frequently conflate these skills into fairly course-grained questions such as “write a program to do ...‘I or “what does the following program do?“. We originally believed on-line testing to be superior becausewe immediately saw mechanismsto test these skill sets individually. Only after further reflection and data analysis did we realise that conventional tests could also be designedto test the skills in this way.
Marking
[Questions 4,8,9.] The original automatic marking programswe createdwere not successfulin me first year for two reasons: 1. We assumedstudentswould follow directions. 2. The marks tended to be polarised. Changeswere made in subsequentyears to mitigate these problems. We solved problem 1 by supplying a “checker” program which reported any failure to follow directions (misnaming files or functions; inconsistent interfaces, etc.) All students chose to use the checker because it helped them catch careless mistakes. Problem 2 was solved by scaling the marks and designing test questions differently. The marking program now attemptsseveral discrete tests at various levels, assigning a variety of marks. Questions are now designedto make student solutions more modular. For example, if students are asked to write a linked-list ADT, we now require each operation in a separatefile. We test a student’s “delete”, for example, by linking it with our own insert and print. Thus, the marking program no longer assigns zero for a student’s “delete” just because her/his “insert” is incorrect. Once the marking programs and question design were modified, our data showed that overall marking time was reducedby 30%, on average.
4.2.1 Program Testing Our initial expectation was that on-line examinations would be better at evaluating students’ “testing” skills. However, we have since concluded that for our first-year courses,the main skill of “testing” is the students’ability to create good test cases from a given specification. We found that questions to evaluate testing skills are as easily created for on-line or conventional examinations. 4.2.2 Debugging Initially we expected that on-line examinations would be better at evaluating students’“debugging” skills. However, we have since concluded that the analytical skills necessary for debugging can be as adequately tested for with conventional tests. However, we have found it easier to creategood debugging questions for on-line examsbecause studentscan solve them with an actual debugger.
3.3 Usability of Environment [Questions 2,5.] Our secure environment (see Section 3.1) forced us to use a rather restricted interface: Studentscould create several windows, and cycle through them; however, the screendisplayed only one window at a time. Originally we supplied the test questionsin a file on-line, and expected students to do rough-work on-line. Our survey results showed all students were unhappy with these in an exam context, becausethey could view only one file at a time. In subsequenton-line tests these problems were resolved by handing out a paper question sheet and providing the studentswith paper for rough-work. In the future, we hope to provide a graphic environment which will allow viewing of more man one file at once, as mentioned in Section 3.1.
4.2.3 Manual Execution Upon starting our on-line testing experiment, we had not even considered the skill of manual code execution. However, our data analysis shows that students’ ability to manually trace, walk-through and inspect their code plays a more significant role in determining their marks with
258
We have resolved the major original operational problems to the satisfaction of both faculty and students. Operational “annoyances”,such as the inability to display more than one window at a time in the test environment, are being addressedin the most recent round of on-line testing. We are continuing to collect data in order to determine the success of these changes, as well as to corroborate our previous findings. Our most recent lesson is that neither faculty nor studentsjudge on-line testing worse or better at testing major skill-sets. However, in our situation at least, on-line tests have the advantagethat marking time can be substantially reduced, and they seem to better motivate studentsto learn the practical skills we expect of them.
conventional testing than with on-line testing. In fact, we have found that on-line testing is very poor at evaluating the skill of manual-execution, though it is unclear how important this skill is in a modern programming environment. 4.2.4 Language Knowledge Our initial expectation was that on-line examinations would be better at evaluating students’“language knowledge”. We have since concluded that although good “language knowledge” questions can be derived for both on-line and conventional examinations, it is easierto mark the questions to a high standard with on-line examinations. Our data shows that our marking time for “language knowledge” questionshas decreasedby 50%, on average. 4.2.5 Design and Problem Solving We had not expectedto find that on-line examinations were of any benefit in evaluating students’“design” and “problem solving” skills; analysis of our data did not cause us to change our opinion. In fact, our evidence suggeststhat online examinations may be poor at evaluating such skills, precisely because the students have an “oracle” (the computer) with which they can explore their solutions. The availability of the oracle was advantageous for those students who used it to explore several different strategies and identify the best solution. However, most students did not use the oracle this way. Some studentsused the oracle to create “quick-and-dirty” solutions without much thought. Yet other studentsbecamefixated on one potential solution; as has sometimes been observed in similar programming environments, such as the ACM Programming Contests, studentswere often unwilling to declare a line of attack as unproductive and re-think their strategy. This is in counterpoint to conventional tests where it is not uncommon to see one or even two solutions that have been completely erasedand restarted. Becausewe have no clear evidence that on-line testing is as good as conventional in evaluating design and problem solving skills, and because we have found it difficult to create superior on-line test questions in these areas, we prefer to use conventional meansto evaluate theseskills.
5. CONCLUSIONS Administration of on-line testing for the past 3 years has taught us many lessons,both operational, and pedagogical.
259
6. REFERENCES Ul Ager, T., Online placement testing in mathematicsand
chemistry, Journal of Computer-Based Instruction. Volume 20, Number 2, 1993,pp. 52-57. VI Bennett, R.E. and Wadkins, J.R., Interactive performance assessment in computer science: the Advanced Placement Computer Science (APCS) Practice System, Journal of Educational Computing Research. Volume 12, Number 4, 1995,pp. 363-78. 131 Mason, D.V. and Woit, D., Creating a Secure Environment for On-Line Examinations, Technical Report 1998-001.Work in progress. [41 Mason, D.V. and Woit, D., Integrating technology into computer scienceexaminations,Proc. 29th Annual SIGCSE Technical Symposium. Atlanta, Georgia, February 26-March 1,1998. [51 Sanford, R.S. and Nagsue, P., SELl!TEST, a versatile menu-driven {PC] tutorial simulates test-taking, Computers in Education Journal. Volume 2, Number 1, 1992, pp. 58-69. [f51Walworth, A. and Herrick, R.J., The use of computers for educational and testing purposes,Proc. Frontiers in Annual Conference. Education. Twenty-first Engineering Education in a New World Order., IEEE. 1991.pp. 510-14. [71 Wresch, W., The imminence of grading essays by computer - 25 years later, Computersand Composition. Volume 10, Number 2, 1993. pp. 45-58.
