Preface to the Special Issue on Automated Assessment of Programming Assignments PETER BRUSILOVSKY, University of Pittsburgh AND COLIN HIGGINS, University of Nottingham Programming computers is becoming an increasingly popular activity, not only for computer science students but across a large number of other disciplines. But funding and other resources for teaching programming have not, for the most part, kept pace with the continuous growth in the number of students studying programming. Academic institutions face the challenge of providing their students with better quality teaching while minimizing the amount of additional work for staff. While traditional teaching methods can be enhanced by audio/visual means and advances in online learning, in the area of assessment the problems continue to persist. Computer-based assessment (CBA), which over the years has become an increasingly important teaching tool, can help educators solve these problems. For the past twenty years, educators have reported on the practical and pedagogic benefits of using automated assessment tools to assess student coursework in programming. The purpose of this issue of JERIC is to explain cutting-edge research on the automated assessment of student programs, and other kinds of programming assignments, to a wider audience. It is devoted to both fully automated assessment and to partial student program assessment by machine. We do not provide an overview of the field in this introductory statement. A recent survey by Ala-Mutka [Ala-Mutka 2005], the review of Douce et al. in this issue [Douce et al. 2006], and several other articles in this issue provide a comprehensive overview and offer various ways to classify the multitude of known work. However, to introduce the articles in this issue, the editors found it useful to distinguish three categories of systems: (1) to assess program-tracing skills; (2) to assess program-writing skills; and (3) assess intelligent programming tutors. The first group of systems attempts to assess students’ knowledge of programming language semantics by presenting students with a program and asking them to trace it. The answer to this type of problem results from its execution: What was printed? What was the final state of the variables in the data structures? The ability to automatically evaluate student answers stems from the system’s ability to execute the program or the algorithm with the same data and compare that result with the one entered by the student. The QuizPACK system reported by Brusilovsky and Sosnovsky [2006] in this issue _________________________________________________________________________________________ P. Brusilovsky, Department of Information Science and Telecommunications, School of Information Sciences, The University of Pittsburgh, PA; 15260; email:
[email protected]. C. Higgins, Learning Technology Research Group (LTRG), School of Computer Science and Information Technology, The University of Nottingham, UK; email:
[email protected] Permission to make digital/hard copy of part of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date of appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Permission may be requested from the Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036, USA, fax: +1 (212) 869-0481,
[email protected] © 2006 ACM 1531-4278/06/0900-ART01 $5.00
ACM Journal of Educational Resources in Computing, Vol. 5, No. 3, September 2005. Article 1.
2
●
P. Brusilovsky and C. Higgins
provides a good example of a system that assesses program evaluation skills. QuizPACK generates parameterized self-assessment questions that allow the students to identify the strong and weak sides of their C program tracing skills. Their article provides an overview of QuizPACK and reports the results of an extensive classroom study of the system. Other systems that belong to this class are the problettes developed by [Dancik and Kumar 2003; Kostadinov and Kumar 2003; Kumar 2002; 2005]. A group of conceptually similar systems use evaluation-based approaches to assess student knowledge of algorithms and data structures. These systems ask a student to mentally execute an algorithm from a specific starting state and submit a trace of actions or the final results of the algorithm’s execution. The systems TRAKLA and TRAKLA2 presented by Malmi et al. [2006] in this issue are among the best known in this class. These tools were specifically designed to assess algorithm simulation exercises and allow students to resubmit their solutions after obtaining feedback. In particular, the tools check whether the required algorithm has not only been used, but also implemented appropriately. By checking the solution against specific requirements, they test the student’s conceptual understanding; other examples of systems in this class are PILOT [Bridgeman et al. 2000] and MA&DA [Krebs et al. 2005]. The second group of systems focuses on evaluating the student’s ability to write programs. Since solving programming problems and producing working code are fundamental skills for computer science graduates, these systems form the majority of work on automated assessment of programming assignments. To automatically assess student solutions most of the reported systems (especially early ones) used different versions of the testing approach, which execute student programs against a set of tests and compare the results provided by the teacher or model program. The review article by Douce et al. in this issue provides an excellent overview of systems that use this testing approach, from early ones to modern Web-based systems. Modern advanced program evaluation systems combine several approaches for program evaluation. This special issue includes articles about the two bestl-known examples of modern automated program evaluation systems: BOSS and CourseMarker. The article by Higgins et al. [2006] presents the CourseMarker system, which is a redesigned and vastly extended Java implementation of the original Ceilidh system first used in 1989. Students solve programming (and other) exercises and submit their solutions to CourseMarker, which in turn returns immediate results and feedback to the students. The marking process is detailed, based on regular expression-matching of source code and test output, and returns a detailed feedback breakdown of the success or failure of the student’s work. There is also provision for course management and tracking of student progress. The article by Joy et al. [2006] introduces BOSS, an online submission system that provides features for course management and automated testing for correctness and quality. It can also check for plagiarism and provides an interface for marking and delivering feedback. It was originally developed in 1993 by the Computer Science Department at the University of Warwick as a text-based tool, running under Solaris. After several revisions, the current version of BOSS has a client-server architecture and uses separate client programs for students and academic staf; Web-based and Java-based clients have been made publicly available. The last system, intelligent programming tutors, is not defined by the specific programming-related knowledge that it is trying to assess, but by its approach to assessment. While the systems of the first two groups rely mostly on their ability to execute a program or a fragment of code to evaluate student answers, intelligent tutors ACM Journal of Educational Resources in Computing, Vol. 5, No. 3, September 2005.
Preface
●
3
attempt to analyze the answer by using a range of knowledge-based approaches. The knowledge-based approaches use different kinds of representations to solve problems in the domain and the domain itself. For example, ELM-ART [Weber and Brusilovsky 2001] analyzes student LISP programs by using a library of correct, suboptimal, and incorrect plans. SQL-Tutor [Mitrovic 2003] diagnoses student errors in solving SQL programming problems by using a set of domain constraints. A knowledge-based analysis typically allows intelligent tutors to achieve a significantly better “understanding” of student answers and to provide extensive feedback for incorrect or suboptimal answers. This group of systems is represented in this special issue by the Scope Tutor in the article by Kumar [2006]. The Scope Tutor is a full-fledged intelligent tutor that can generate a problem for the student, assess the student’s knowledge, provide elaborate feedback, and proceed with the next problem that is most relevant to the demonstrated level of knowledge. The tutor is able to “understand” and diagnose student answers due to the embedded knowledge about the domain. Altogether the articles in this issue form a versatile comprehensive picture that should help the reader to understand the state-of-the-art in automated assessment of programming assignments. We hope that this issue will be useful for researchers and practitioners who teach programming-related subjects and beyond. REFERENCES: ALA-MUTKA, K.M. 2005 A survey of automatic assessment approaches for programming assignments. Comput. Sci. Edu. 15, 2, 83-102. BRIDGEMAN, S., GOODRICH, M.T., KOBOUROV, S.G., AND TAMASSIA, R. 2000. PILOT: An interactive tool for learning and grading. In Proceedings of the 31st ACM SIGCSE Technical Symposium on Computer Science Education (Austin, Tex., 2000), ACM, New York,139-143. BRUSILOVSKY, P. AND SOSNOVSKY, S. 2006. Individualized exercises for self-assessment of programming knowledge: An evaluation of QuizPACK. ACM J. Edu. Resources Comput. DANCIK, G. AND KUMAR, A.N. 2003. A tutor for counter-controlled loop concepts and its evaluation. In Proceedings of the 2003 Frontiers in Education Conference (Boulder, Colo.) Session T3C. DOUCE, C., LIVINGSTONE, D., AND ORWELL, J. 2006. Automatic test-based assessment of programming: A review. ACM J. Edu. Resources Comput. This issue. HIGGINS, C., GRAY, G., SYMENOIDIS, P., AND TSINTSIFAS, A. 2006. Experiences on automatically assessed algorithmic exercises. ACM J. Edu. Resources Comput. This issue. JOY, M., GRIFFITHS, N., AND BOYATT, R. 2006. The BOSS online submission and assessment system. ACM J. Edu. Resources Comput. This issue. KOSTADINOV, R. AND KUMAR, A.N. 2003. A tutor for learning encapsulation in C++ classes. In Proceedings of the ED-MEDIA 2003, World Conference on Educational Multimedia, Hypermedia and Telecommunications (Honolulu), 1311-1314. KREBS, M., LAUER, T., OTTMANN, T., AND TRAHASCH, S. 2005. Student-built algorithm visualizations for assessment: Flexible generation, feedback, and grading. In Proceedings of the 10th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE'2005, Monte de Caparica, Portuga), 281-285. KUMAR, A. 2006. Generation of problems, answers, grade and feedback - Case study of a fully automated tutor. ACM J. Edu. Resources Comput.This issue. KUMAR, A.N. 2002. A tutor for using dynamic memory in C++. In Proceedings of the 2002 Frontiers in Education Conference (FIE 2002, Boston, Mass.). KUMAR, A.N. 2005. Results from the evaluation of the effectiveness of an online tutor on expression evaluation. In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (St. Louis, Mo.), 216-220. MALMI, L., KARAVIRT, V., KORHONEN, A., AND NIKANDER, J. 2006. Experiences on automatically assessed algorithmic exercises. ACM J. Edu. Resources Comput. This issue. MITROVIC, A. 2003. An intellignet SQL tutor on the Web. Int. J. AI Edu. 13, 2-4, 173-197. WEBER, G. AND BRUSILOVSKY, P. 2001. ELM-ART: An adaptive versatile system for Web-based instruction. Int. J. AI Edu. 12, 4, 351-384. Available at http://cbl.leeds.ac.uk/ijaied/abstracts/Vol_12/weber.html.
ACM Journal of Educational Resources in Computing, Vol. 5, No. 3, September 2005.
