automatic plagiarism detection in programming laboratory courses

ISBN: 978-989-8704-04-7 © 2014 IADIS

AUTOMATIC PLAGIARISM DETECTION IN PROGRAMMING LABORATORY COURSES Nikolaos D. Tselikas1, Maria Samarakou2, Dimitrios Karolidis2, Pantelis Prentakis2 and Spyros Athineos3 1

Department of Informatics and Telecommunications, University of Peloponnese End of Karaiskaki Street, Tripoli 22100, Greece 2 Department of Energy Technology Engineering, T.E.I. of Athens Ag. Spyridonos, Aegaleo, 12210, Greece 3 Department of Electronic Engineering, T.E.I. of Athens, Ag. Spyridonos, Aegaleo, 12210, Greece

ABSTRACT In programming language courses, programming exercises and assignments are typical measurable tasks used to determine the skills and knowledge of students on the respective programming language. On the other hand, widespread access to the Internet has served as something of a double-edged sword with respect to plagiarism appearance in exercises, assignments, etc. We present a viable architecture implementing plagiarism detection and decision in a programming laboratory environment. The proposed architecture could be realized as an extension of an already implemented diagnosing and evaluation system for laboratory based academic exercises, which exploits the legacy system as well as the already existing plagiarism detection tools, such as MOSS or JPlag. We combine the input of plagiarism detection tools with the corresponding logging information of the existing monitoring and tracing subsystem of the legacy system, in order to conclude in a safer manner whether a case of potential plagiarism indeed is plagiarism or not. KEYWORDS Computer Aided Education; Automatic Programming Exercises Assessment; Plagiarism detection; Logging information.

1. INTRODUCTION In programming courses, programming exercises and assignments are typical measurable tasks used to determine the skills and knowledge of students on the respective programming language. The implemented source code – sometimes accompanied with documentation – is usually the touchstone for students’ final grades. In other words, the delivered source code should solve the problem described in the exercise or assignment and the corresponding assessment is mainly based on the success of the source code to provide the expected results, examining, sometimes, its efficiency, too (Carter, J. et al., 2003). Moreover, assessment provides feedback to the educator showing whether the programming learning goals are being met or not, while it also ensures for an outside observer that students achieve the respective learning goals. On the other hand, assessment also provides feedback to the student, rendering him/her able to realize his/her skills and capabilities in programming. Educators examine and grade the students' exercises either manually or by using automatic tools to reduce the effort required for such a task in large classes. Manual assessment requires a significant amount of time in order to evaluate a student’s level of understanding on concepts covered in class or in laboratory. Assessment results are usually used to identify a student’s weaknesses and strengths. An educator can use this information to make and/or modify lesson plans to reinforce certain subjects (Yoo, J. et al., 2006). The results can also be used by the educator to help establish a personal learning plan for each student. However, the assessment and evaluation task is often a time consuming and tedious task, which is the main reason that a lot of educators use automatic assessment tools to evaluate programming exercises. Actually, the idea of assessing programming assignments automatically started alongside with the teaching of programming, since

232

7th IADIS International Conference Information Systems 2014

one of the first systems developed to assess the student’s programs was written in Assembly language and appeared in the early 60’s (Hollingsworth J., 1960). The most common technique for the assessment of programming assignments, either manually or automatically, is to test the correctness of the source code which the students supply, to execute it on predefined data and inspect the output compared with the expected results (Rashkovits, R. and Lavy, I., 2013). On the other hand, the growing amount of information on the Internet and the ease with which students have access to this information has produced an increasing temptation for students to download Web material, possibly modify it and hand it in as their own unaided work (Culwin, F., Lancaster, T., 2001). Plagiarism could be defined as the stealing of words or ideas of another person and use them without proper acknowledgement (Batane T., 2010). Student plagiarism could be further defined as plagiarism with the intent of gaining academic credit; and students are not always so innocent. It is common knowledge among the higher education community that a number of students are engaged in some forms of academic dishonesty (Mei, Z., DongSheng, L., 2010). There are several forms and kinds of student plagiarism and cheating. For example, in homework assignments, students may take material from the web, copy from books, even pay for assignment solutions or simply work too closely with their friends trying to practice the “CTRL-C/CTRL-V technique” (Zobel J. and Hamilton M., 2002). In laboratory exams of a programming course, one could say that plagiarism is very difficult to be met. Unfortunately, this is not true, since students discover various ways to cheat, especially if the computers in the lab provide Internet access without surveillance/restrictions, or by using their smartphones. In this case, they can also copy material from web, even the whole solution of the exercise, they can use messenger applications or email to exchange and copy the solution or they may even copy over shoulders. In this paper we propose an extension of an already implemented diagnosing and evaluation system for laboratory based academic exercises, enabling it to detect potential student plagiarism. The special conditions of the laboratory environment (e.g., students have restricted time to submit the exercise, the physical presence of educator in the laboratory, etc.) may reduce students’ “sophisticated plagiarism” on one hand (e.g., time limitations discourage them to change the software logic probably copied from another student or from web), but it can lead to “crude plagiarism” (i.e., direct copy and paste either from another student or from web). We propose that a plagiarism detection system could be built by combining plagiarism detection methodologies, as they have been developed by educators, with the logging information of a laboratory tracing tool. The rest of the paper is organized as follows. Section 2 describes the functional requirements of such a system based on the current approaches and tools, enabling the automatic assessment of programming assignments, as well as on the dominant techniques and methodologies used in plagiarism detection. Section 3 presents the proposed architecture while the paper summarizes in Section 4.

2. SYSTEM REQUIREMENTS This Section describes the proposed system’s foundations, i.e., the current approaches and tools enabling the automatic assessment of programming assignments, as well as the dominant techniques and methodologies used in plagiarism detection.

2.1 Automatic Assessment and Logging in Programming Courses The first prerequisite for our proposed system’s realization is to be able to assess and log programming assignments. There are several surveys of automatic assessment regarding programming exercises. The most recent and characteristic ones are presented below. Ihantola et al. presented a systematic literature review regarding the development of automatic assessment tools for programming exercises, describing the major features that the tools support and the different approaches they are using both from the pedagogical and the technical point of view (Ihantola P. et al, 2010). The same review lists also numerous open source automatic assessment tools available for testing, which can be the basis of our system too. Ala-Mutka presents methods and techniques used by automated assessment tools and shows how they are generally used (Ala-Mutka, K., 2005). In addition to correctness, Ala-Mutka

233

ISBN: 978-989-8704-04-7 © 2014 IADIS

cites that some assessment tools analyze program efficiency, coding style and the existence of inline documentation. The author concludes that dynamic analysis, i.e., assessment based on executing the program, should be used in order to evaluate functionality, efficiency and testing skills. Shamsi and Elnagar presented eGrader, a suitable tool for the problem of dynamic analysis for students’ Java exercises, based on the JUnit framework (Shamsi, F., and Elnagar, A., 2012). Its analysis process is based on graph representation of the program, while the graph representation is based on the Control Dependence Graphs and Method Call Dependencies, which are constructed from the abstract syntax tree of the source code. Yoo et al. presented an extensible visualization tool that can be used in a web-based closed laboratory system, which helps both educators and students to trace deficiencies in a student’s understanding back to individual concepts (Yoo, J. et al., 2006). Even if the aforementioned approaches and tools enable automatic assessment of programming exercises and assignments, most of them do not come along with logging or tracing tools. On the other hand Samarakou et al. presented a diagnosing and evaluating system used in laboratory-based exercises (Samarakou, M., et al., 2013). Part of this system is the Monitoring and Logging Subsystem (MLS), which is the functional entity responsible to collect student logs. MLS logs a student’s actions during the exercise, such as, button clicks, key presses, combination of pressed keys (such as CTRL-C or CTRL-V), changes in variables, idle time intervals, etc, thus it could be a good basis for our proposed architecture.

2.2 Plagiarism Detection Widespread and broadband access to the Internet has served as something of a double-edged sword with respect to plagiarism (Youmans, R. J., 2011). This indicates the second prerequisite for the realization of our proposed system, i.e., the efficient plagiarism detection tools, algorithms and methodologies (Konstantinidis, A., et al., 2013). Numerous techniques for detecting partial copies make use of the fingerprinting concept (Hoad, T. C. and Zobel, J., 2003). A document, which can be either text or source code, is transformed in a nonblank string, by removing any space, whitespace, tab, new line, etc characters. This string forms the initial string, which is the initial input of the algorithm. The initial string is divided into k-grams, where k is a parameter chosen by the plagiarism investigator. A k-gram is a contiguous substring of initial string of length k. Note that there are almost as many k-grams as there are characters in the document, as every position in the document (except for the last k − 1 positions) marks the beginning of a k-gram. The next step is to hash each k-gram and select some subset of these hashes to be the document’s fingerprints. In all practical approaches, the set of fingerprints is a small subset of the set of all k-gram hashes. If the hash function is chosen so that the probability of collisions is very small, then whenever two documents share one or more fingerprints, it is extremely likely that they share a k-gram as well. Further details about fingerprinting and its varieties can be found in (Schleimer, S. et al., 2003). There are also several clone detection techniques that are based on pure string-based methods (Roy, C. K. and Cordy, J. R., 2007). In this approach, the target source program is considered as sequence of lines/strings. Two code fragments are compared with each other to find sequences of same text/strings. Once two or more code fragments are found to be similar in their maximum possible extent are returned as clone pair or clone class by the detection technique. The survey describes also tokenbased methods. Tokens are strings after discarding whitespaces, comments and identifier names; most academic plagiarism detection systems work at this level, using different algorithms to measure the similarity between token sequences. Parsing trees is another technique allowing higher-level similarities to be detected, while tree comparison can normalize conditional statements and detect equivalent constructs as similar to each other (Son, J.-W. et al. 2006). Another technique is based on mining Program Dependency Graphs (PDGs), which are the graphic representation of the data and control dependencies within a procedure (Liu C. et al., 2006). This technique is both effective and efficient, detecting plagiarism that easily slips over existing tools, and it usually takes a few seconds to find plagiarism in programs having thousands of lines of code. Another methodology followed in plagiarism detection in programming exercises is based on “metrics”, which capture “scores” of code segments according to certain criteria, e.g., “the number of loops and conditionals” or “the number of different variables used” (El Tahir Ali, A. M. et al, 2011). Even if metrics are simple to calculate, they can also lead to false positives, e.g., two fragments with the same scores on a set of metrics may do entirely different things. A detailed analysis and presentation of many other plagiarism detections methods, tools and techniques can be found in (Roy, C. K. and Cordy, J. R., 2007).

234

7th IADIS International Conference Information Systems 2014

The most popular, free (not for commercial use) and web-based available tools for plagiarism detection based on the afore-mentioned techniques are MOSS (MOSS, 2013), provided by the Stanford University and JPlag, provided by Karlsruhe Institute of Technology, respectively (Prechelt, L. et al., 2002). On the other hand, our target is the monitoring of laboratory programming exercises and the corresponding solutions are usually no more than a hundred of source lines of code, because of the time limitation. Furthermore, a programming language has very few characteristic elements compared to a natural language (Rosales, F. et al., 2008). From the point of view of plagiarism, this smaller size has some implications. A first observation is that to develop two identical, or at least very similar, program fragments is much easier than to write two identical or even very similar paragraphs when using natural language. At the same time, whether the cause of the similarity is plagiarism or whether such similarity is simply derived from an analogous working method based on the same theoretical knowledge is difficult to distinguish, especially in short programming solutions. Rosales et al. presented their solution to this issue in (Rosales, F. et al., 2008), while we believe that a simpler and probably more accurate answer on the question “Are you sure about the detected potential plagiarism?” could be given by the proposed architecture, analyzed in the next Section.

3. THE PROPOSED ARCHITECTURE The proposed high-level architecture of the diagnosing and evaluation system for laboratory based academic exercises, with enhanced student plagiarism detection, is depicted in Figure 1.

Diagnosing Subsystem

Students

Monitoring and Logging Subsystem

Modeling Subsystem

Database Evaluation Subsystem

Plagiarism Decision Subsystem

Internet

Existing Plagiarism Detection Tool (e.g., MOSS or JPlag)

Figure 1. The proposed high-level architecture

235

ISBN: 978-989-8704-04-7 © 2014 IADIS

The functional entities of the legacy system which have already been developed are the white ones, while the required new functional entities - either internal or external - that have to be added are colored in grey. The legacy system consists of the Monitoring and Logging Subsystem, which monitors and records a student’s actions on the laboratory computer during the exercise, the Diagnosing Subsystem, which determines and evaluates the initial knowledge of the student, the Modeling Subsystem, which creates the student model based on individual knowledge profiles and the Evaluation Subsystem, which is used to evaluate the performance of students based not only on the final result but on several other factors. All subsystems are interconnected and have access to the system’s database. Further information regarding legacy system’s functional entities can be found in (Samarakou, M., et al., 2013) as well as in (Papadakis, A., et al., 2013). Plagiarism Decision Subsystem is the new required functional entity and the heart of the new added functionality. It requires access to the system’s database, specifically on the data of the Monitoring and Logging subsystem, acquiring the data recorded for each student. Once all students have submitted their programming laboratory exercises, the Plagiarism Decision Subsystem submits the source code of all exercises to an existing plagiarism detection tool (e.g., MOSS or JPlag). The next step is to receive the corresponding results concerning similarities between the exercises which the students submitted; this is a first indication of potential plagiarism. Submitted exercises and solutions which present high average similarity have to be further investigated, as they are suspected of plagiarism. On the other hand, as explained in the previous section, if the submitted data consist of only a few source lines of code, the possibility of high average similarity increases and the indication of the plagiarism detection tool alone is not safe enough. For that reason, the Plagiarism Decision Subsystem is able to double check all cases by tracing back to the database’s student logs. Suspicious logs, such as students which remain idle for a long period during the laboratory but their solution is perfect, messenger and/or email client and/or browser usage and CTRLC/CTRL-V detection, are used to reinforce the plagiarism detection tool indications. Thus, based on the double input, i.e., the indications of a plagiarism detection tool and the student logs, traced by the Monitoring and Logging Subsystem data stored in the database, the Plagiarism Decision Subsystem may decide with greater certainty whether a plagiarism indication is valid or not. What is critical and has to be further investigated, in order to achieve as accurate results as possible, is the determination of which plagiarism indication factors should be used to check the student log files with and their effect on the plagiarism detection tool results, which could be fine-tuned after trials and stress tests.

4. CONCLUSION We presented a viable architecture regarding potential student plagiarism detection and decision. The proposed architecture could be realized as an extension of an already developed diagnosing and evaluation system for laboratory based academic exercises. The proposed architecture exploits the legacy system as well as already existing plagiarism detection tools, such as MOSS or JPlag. We combine the input of plagiarism detection web tools with the corresponding information logged by an existing monitoring and tracing subsystem, in order to conclude in a more safe manner whether a case of potential plagiarism is indeed an actual case of plagiarism or not. Future work in our work include to add extra features into the proposed system such as real time alerting the instructor or giving a remote display of students who are copy pasting code, are sitting idle or surfing internet during the laboratory sessions. We believe that such a system will not only help the instructor or the educator in a programming laboratory course but will ultimately benefit the students who will be forced to practice and develop their own codes and hence avoid taking the plagiarism shortcut to learning.

236

7th IADIS International Conference Information Systems 2014

ACKNOWLEDGEMENT This research has been co-funded by the European Union (European Social Fund) and Greek national resources under the framework of the “Archimedes III: Funding of Research Groups in TEI of Athens” project of the “Education & Lifelong Learning” Operational Programme.

REFERENCES Ala-Mutka, K., 2005. A survey of automated assessment approaches for programming assignments. Computer Science Education, Vol. 15, No 2, pp.83–102. Batane, T., 2010. Turning to Turnitin to Fight Plagiarism among University Students. Educational Technology & Society, Vol. 13, No 2, pp. 1–12. Carter, J. et al., 2003. How shall we assess this? Proceeding of ACM ITiCSE-WGR '03 Working group reports from ITiCSE on Innovation and technology in computer science education. New York, NY, USA, pp. 107-123. Culwin, F., Lancaster, T., 2001. Plagiarism issues for higher education, VINE, Vol. 31, Is. 2, pp.36 – 41. El Tahir Ali, A. M. et al, 2011. Overview and Comparison of Plagiarism Detection Tools. Proceedings of the Dateso 2011, Annual International Workshop on DAtabases, TExts, Specifications and Objects, Pisek, Czech Republic, pp. 161-172. Hermann M. et al., 2006. Plagiarism - A Survey. Journal of Universal Computer Science, Vol. 12, Is. 8, pp. 1050 – 1084. Hoad, T. C. and Zobel, J., 2003. Methods for identifying versioned and plagiarized documents. Journal of the American Society for Information Science and Technology, Vol. 54, pp. 203–215. Hollingsworth J., 1960, Automatic Graders for Programming Classes, Communications of the ACM, Vol. 3, No. 10, pp. 528-529. Ihantola P. et al, 2010. Review of recent systems for automatic assessment of programming assignments. Proceedings of the 10th Koli Calling ACM International Conference on Computing Education Research (Koli Calling '10). New York, NY, USA, pp. 86-93. Konstantinidis A. et al., 2013, Plagiarism: Examination of Conceptual Issues and Evaluation of Research Findings on Using Detection Services, Contemporary Educational Technology, Vol. 4, Is. 3, pp. 212-221. Liu C. et al., 2006. GPLAG: detection of software plagiarism by program dependence graph analysis. Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD '06). New York, NY, USA, 872-881. Mei, Z., DongSheng, L., 2010. An XML plagiarism detection algorithm for Procedural Programming Languages. Proceedings of International Conference on Educational and Information Technology. Chongqing, China, pp.V3-427 - V3-431. MOSS: A system for detecting software plagiarism, 2013. MOSS, available online: http://www.cs.berkeley.edu/aiken/moss.html Papadakis, A. et al., 2013. Enhancing Laboratory Experience and Assessment through Monitoring and Processing of Students’ Activities. Proceedings of the IEEE Global Engineering Education Conference (IEEE EDUCON 2013), Berlin, Germany, pp. 1332-1337. Prechelt, L. et al., 2002. Finding plagiarisms among a set of programs with JPlag, Journal of Universal Computer Science, Vol 8, No 11 (JPlag url: https://wwwipd.ira.uka.de/). Rashkovits, R. and Lavy, I., 2013. FACT: A Formative Assessment Criteria Tool for the Assessment of Students' Programming Tasks. Proceedings of the international conference on information engineering, World Congress on Engineering. London, England, pp. 384-389. Rosales, F. et al., 2008. Detection of Plagiarism in Programming Assignments, IEEE Transactions on Education, Vol. 51, No. 2, pp. 174-183. Roy, C. K. and Cordy, J. R., 2007. A Survey on Software Clone Detection Research, Technical Report No. 2007-541. School of Computing. Queen's University at Kingston. Ontario, Canada. Samarakou, M., et al., 2013. A student diagnosing and evaluation system for laboratory-based academic exercises. Proceedings of 24th annual conference of the Society for Information Technology and Teacher Education. New Orleans, USA, pp. 4038-4043.

237

ISBN: 978-989-8704-04-7 © 2014 IADIS

Schleimer, S. et al., 2003. Winnowing: local algorithms for document fingerprinting. Proceedings of the ACM SIGMOD international conference on Management of data (SIGMOD '03). New York, NY, USA, pp. 76-85. Shamsi, F., and Elnagar, A., 2012. An Intelligent Assessment Tool for Students’ Java Submissions in Introductory Programming Courses, Journal of Intelligent Learning Systems and Applications, Vol. 4 No. 1, pp. 59-69. Son, J.-W. et al. 2006. Program Plagiarism Detection Using Parse Tree Kernels, PRICAI 2006: Trends in Artificial Intelligence Lecture Notes in Computer Science. Vol. 4099, pp. 1000-1004. Yoo, J. et al., 2006. Student progress monitoring tool using treeview. Proceedings of the 37th ACM SIGCSE technical symposium on Computer science education. New York, NY, USA, pp. 373-377. Youmans, R. J., 2011. Does the adoption of plagiarism-detection software in higher education reduce plagiarism? Studies in Higher Education, Vol. 36, Is. 7, pp. 749-761. Zobel J. and Hamilton M., 2002. Managing student plagiarism in large academic departments. Australian Universities Review, V.45, No.2, pp. 23-30.

238