Innovative Instruction, Assessment & Evaluation in Programming ...

Innovative Instruction, Assessment & Evaluation in Programming Language Concepts David M W Powers

Abstract— This paper describes the development of a course in Programming Concepts over a period of a decade. Initially we present educational principles that underlie the approach, as well as specific and generic aims for this course. We then proceed to show how an integrated program of assessment and instruction have allowed teaching and comparison of multiple programming paradigms and concepts within a common framework. The specifiation of specific and generic skills and the extents to which they are expected to be covered has been closely validated with surveys framed around them. We also address some general issues in modern education..

C

I. INTRODUCTION

Computer Science/Information Technology education in many universities runs the risk of becoming a one language shop as we turn out generations of IT and CS graduates who know little more than Java and their favourite IDE. A typical IT graduate may be scared of recursion, unwilling to tackle optional courses involving programming in C, C++ or Python, and even uncertain as to how to actually write and compile a Java program without using the hand-holding of the tools they were taught with. Whilst many universities introduce additional programming languages in a range of core and option courses, in obvious question is whether our teaching is effective in teaching the minority languages and paradigms and in broadening the capabilities of our graduates. Whilst many universities use the ubiquitous Java as their main teaching language, it was not designed as a teaching language, and a weakly object-oriented teaching approach may end up teaching neither effective imperative programming or a theoretical understanding of Object-Oriented Programming (OOP or OO). The ACM curriculum seeks to ensure we develop a balanced CS graduate through exposure to multiple paradigms and a deeper theoretical understanding of both programming languages, compilation and execution, and program writing, verification and debugging. This paper URRENTLY

Manuscript received November 1, 2011. David M W Powers is Professor of Computer Science in the School of Computer Science, Engineering & Mathematics, Flinders University, and Professorial Fellow in the Beijing Municipal Key Laboratory for Multimedia and Intelligent Software Technology, Beijing University of Technology (phone: +86-150-1254-7923 in China or +61-414-8204307 in Australia; fax: +61-8-8201-3663; e-mail: David.Powers @ flinders.edu.au). This work was supported in part by the Chinese Natural Science Foundation under Grant No. 61070117 and Grant No. 609730057.

discusses the evolution of a series of topics related to program languages over the course of the last 15 years, in a context which varied over time: at the start of this period, the normal graduate studied Computer Science and a comprehensive variety of paradigms were introduced in first year, with Imperative Programming being the primary language focus and paradigm. As Pascal gave way to Ada and then Java, as Computer Science was ousted by Information Technology, as basic skills in using brandname word processors and spreadsheets vied with foundational skills in computer science for time, the alternate programming paradigms can be relegated to optional second year and/or third year courses at a significant number of universities. This paper focuses on the design and evaluation of an optional third year course that is for some students the first and only exposure they have to these alternatives to basic Java/C++ programming: a course that is compulsory for our Computer Science students, but optional and not in general taken by our programming-phobic Information Technology and Software Engineering students (versions are available for both undergraduate and graduate students). Even if Java (or C++) is the only language (paradigm) a student will use throughout their professional career, there is much to gain from considerations of efficiency of programming versus execution, from exposure to declarative languages and specification verification, and to formal benchmarking and testing of programs, as well as to a formal definition of and introduction to Object-Oriented Programming (OOP). There is anecdotal and survey evidence that students become better programmers for exposure to these programming language concepts, and our Programming Language Concepts course has been designed from the ground up to comparatively explore multiple programming paradigms in the context of a single task for which typically 18 to 20 different implementations will be compared! Two further specific issues are worthy of note at this point, and have been addressed in the design of the topic. The course is available at graduate and undergraduate levels, and the graduate students are expected to be more mature, will typically have a degree in a different discipline and less actual programming experience, and have a much higher proportion of international students with different language and cultural backgrounds. For both graduate and undergraduate students, for both Australian and International students, issues of academic integrity have needed careful attention, although the

precise nature of the issue seems to vary with the demographic. Often the issue seems to be that students will Google first, ask questions later, and avoid thinking completely! Any assessable work set anywhere in the world is likely to have a solution on the web somewhere. If not, it is possible to hire someone to write it for you, and requests and offers relating to assignments can often be found! Also, the Asian students in particular have a strong cultural propensity to work together and may not appreciate the Western norms of individual assessment. A further factor that represents a feature of our times and derives from our technological orientation: the tendency of students not to attend lectures. The lectures, seminars and workshops are videoed and some students do indeed watch them – indeed many of the international students watch them multiple times and rewind to go over bits they didn’t understand. However, there are advantages to being there in the lecture theatre, and interestingly providing optional seminars and workshops seems to have actually increased attendance, although they are still videoed – probably this is due to two factors, their more interactive style and the desire to get the hints as soon as possible! A final general principle is that assessment should be designed to be able to accurately assess and characterize the capability of students of all capabilities and backgrounds. This obvious requirement for all assessment in every discipline and course, is all too seldom met in university courses, in my experience, and the training of academics in the principles we illustrate is even rarer! This paper proceeds to discuss the design of the topic and its assessment as an integrated whole that meets these desiderata, then continues to discuss the formal criteria which it was built around and the evaluation tool and results that were developed and undertaken since 2008. II. REVIEW OF COMPUTER SCIENCE EDUCATION Valentine [1] in a brief meta-review of CSE publications focused on first year teaching distinguishes: Marco Polo (went there, did that), experimental (formally tried and tested), tools (programs/systems), nifty (innovative puzzles, assignments, games, etc.), philosophical (debate, soapbox), John Henry (exceptionally, if not unnecessarily, challenging) – here listed in descending order of prevalence over the preceding two decades. Marco Polo papers are characterized by proposing and giving anecdotal evidence of the utility of particular approaches or courses, while experimental papers involve set up of formal paradigms for testing new approaches against controls, and performing formal quantitative analysis of student performance and/or survey results. Few papers, if any, achieve the degree of multifactorial control that is desirable in quantitative science, so there is something of a spectrum between these two extremes, with many papers falling in between

somewhere. Generally, a properly controlled experiment is beyond the capability of a single researcher at a single institution, and it also faces the issue that some students may be taught in ways that are believed to be substandard, raising equity concerns. Even a multi-researcher multi-institution team faces issues relating to the level and perceived reputation of the institution and degree, as well as the quality of both the staff and the students involved in the experimental teaching. A meta-study of actual teaching practice and results across a large number of institutions, can address these issues but then faces issues relating to lack of balance of the many factors. In addition, university, faculty or accreditation level mandates about courses or survey techniques may disrupt even the best laid plans for experimental validation. Lister [2] uses a Keynote soapbox to advocate treating our teaching and our students as proper subjects for CSE research. He also addresses common boom and bust patterns that apply to CSE in common with other disciplines and domains, whilst exploring the idea of folk pedagogues or pedagogies in analogy to folk healers and medicine, with factors driving curriculum change revolving around ‘influential or outspoken individuals’ [3] rather than formal processes of evidence, research or even philosophical debate. In Lister’s words, ‘A Marco Polo paper is folk pedagogy committed to paper’ and appears to correspond to an alternate classification in terms of ‘anecdotal evidence’. However, the cited authors [1],[2] agree that Marco Polo papers have their place in terms of hypothesis generation, or coming up with new ideas about how to teach. This is not a unique problem to CSE, but applies to Education in general, and Educational Theory is in danger of being an entrenchment of folk pedagogy or pseudo-psychology in view of the difficulty of running properly balanced studies, and the appeal to ‘well known facts’ to justify the cut corners and omissions. There is a tendency to appeal to constructivist theories, cognitive load theory and psychological insights about working memory and chunking, and each proponent of a competing theory can point to ‘experiments’ that support their approach. Moreover, generalist approaches to education tend to face an additional problem, that they try to sell a one fits all model that applies equally well in all academic disciplines, and make the assumption that success in one discipline implies utility in the remainder [2]. This paper is ‘Marco Polo’ to the extent that it proposes new and innovative approaches to teaching a specific subject, but aims to be ‘Experimental’ in that surveys specific to the modified course are analyzed in conjunction with an analysis of Student Evaluation of Teaching (SET) results over a period of a decade, for a pool of related topics, including precursor and parallel topics. The factor analysis of SET results also takes into account the marks achieved by the students, and exposes

factors that do and do not load or correlate with student performance. However, there are confusing factors that are still hidden, including time-related factors such as maturing of the teacher and the corresponding unknown changes to student intake. These can to an extent be distinguished by reference to performance of a broader pool of teachers in CSE. III. SPECIFIED/TARGETED SKILLS Nationally, student outcomes are assessed across degrees and universities in terms of a set of Generic Skills (first 6 in Table I). In order to assure that degrees indeed train across this range of skills, in our faculty course coordinators are asked to provide weightings as to to what extent it is aimed to develop these skills in a topic, and additional Generic Skills are included at this level (7 and 8). Rather than arbitarily answering this survey for the topic and forgetting about it, the Programming Language Concepts (PLC) was designed from the ground up to support development of specific skills, based around the Sebesta textbook of the same name (various editions over the course of the decade). TABLE 1: SKILLS MATRIX AND STUDENT QUESTIONS Generic Skills 1. Problem-solving skills 2. Analytical skills 3. Team-work skills 4. Ability to tackle unfamiliar problems 5. Written communication skills 6. Ability to plan work independently 7. Numerical skills 8. Scientific report writing skills Scientific and Technical Skills 1. Determine appropriate paradigm for task 2. Adapting programming style to the task 3. Writing reusable understandable code 4. Writing provably correct efficient code 5. Fast bug-free coding style 6. Program evaluation and benchmarking 7. Efficient tool use for presenting results Other Skills 1. Sort quickly and efficiently 2. Adapt to new environments quickly 3. Reduce problems logic/recursively Recursion easier than Iteration? ObjOriented easier than ad hoc coding? Logic Programming easier than Imperative? Functional Programming easier than Imperative?

Table 1 shows the standard Generic Skills across all degrees and programs. In particular, students were expected to benchmark their programs and provide a written report. Table 1 also shows the key Scientific and Technical Skills from the Degree program that are in focus in PLC. It also includes three Other Skills that are specific to the PLC topic as developed, and in particular the fact that all

programming assignments revolve around sorting as a single simple task that can be performed in many ways. These were all provided to the faculty in advance as part of the proposal document and an addition to the separate Generic Skills survey. In addition four questions were added for survey purposes to assess whether student attitudes to the different paradigms had changed during the topic. As for most students this was the first exposure to most of the paradigms, the main value of these questions may be to expose misconceptions, and of course the recursion question reflects a brief prior exposure to recursion which for many students seems to leave the impression a difficult and abstruse alternative to iteration. The first assignment was designed specifically to address this issue. IV. ASSIGNMENTS AND WORKSHOPS As PLC is based on ACM/ACS curriculum and goes through the whole of the set Sebesta textbook systematically, the focus of our discussion will be on the major Programming Assignments given to the students, and additional Workshops that supported them in a mix of lecture, laboratory and tutorial style. However, Sebesta was supplemented with specific material teaching Scheme and Prolog as representative of current standardizations of classical FP/LP programming languages, and Ada, Java and C++ as modern languages representing different approaches to abstraction and OOP – along with brief exposure to other appropriate languages such as Smalltalk, Haskell, Curry, F#, etc. A. Theory & Practice of Language Processing As Prolog was designed for parsing and includes Definite Clause Grammars (DCG) as part of the standard syntactic sugaring, the Syntax and Semantics components of Sebesta (originally Chapter 3) were postponed to the very end of the course allowing a. earlier treatment of the Programming Paradigms needed for the Programming Assignments, and b. use of DCGs for exploring grammars. Originally the course occupied a 25% student load for the semester, but the current version has been expanded to a 33% load by incorporating Theory of Computation material, which also has a focus on Grammar, the Chomsky hierarchy, formal machines, computability and complexity (including definition of the concept of order as used in evaluating and benchmarking algorithms). This also fits well with the late treatment of Sebesta's language related material. The unique combination of Programming Language Concepts and Theory of Computation, and the use of our common thread of major assignments, helps to show that the theory is actually relevant, and to put it into use immediately. B. Sorting as an ongoing Programming task Sorting was chosen as the primary programming task, indeed the sole programming task other than what was needed as part of the total framework, which also

included a benchmark harness students had to develop. The advantages of this choice will be illustrated as we go through the assignments in turn. Note that QuickSort was the primary algorithm in focus, and where the most instruction and support was given in relation to Sorting. MergeSort was given as a second algorithm which was compulsory for students in the graduate version of the topic, with their briefer exposure to programming (in general their Bachelors degree was in another area). However, undergraduates were permitted to attempt the additional work and received the better of the mark awarded and their exam mark in this case – viz. they could go into the exam knowing they’d passed, or even choose to skip it! Other sorting algorithms were also introduced and explained in specific contexts, including in terms of parallel sorting and the difficulty of achieving linear speedup, however no parallel programming assignments were given. Currently assignments are provided with two deadlines – an early-bird deadline that attracts a bonus mark (10% providing they achieve at least 40% - viz. they have passed) and a final deadline that allows improvement or completion of their code after feedback. In addition optional workshops and laboratory sessions are provided between the deadlines to give additional support and hints. The 10% bonus compensates students for working this all out for themselves, and also provides and incentive to get started early rather than leaving it to the last minute, or later! In the case of Assignment 1, the deadlines are 8 weeks apart, but less for later ones. The effect of this change is clearly evidenced from the progress students are making in the weekly lab sessions. Where as previously most of the class would not make significant progress on the assignment (or in some cases even come to grips with the specification) until a couple of weeks before the deadline, they are now almost all tackling it a couple of prior to the initial deadline. A relatively small proportion of the class actually hands in a substantially complete assignment, and gets the bonus, and they have still tended to hand in the final version to seek to improve marks further. The bonus was expected to discourage this, and clearly does so for those that go from 90% to 100%. The combination of starting early and having students complete the assignments early were expected (and proved) to allow more time for tutorial assistance of the students who were struggling. Generally the feedback given on the draft assignment was sufficient to allow the more advanced students (in time if not ability) to improve them adequately to achieve (near) full marks on resubmission. 1)

Asst 1: Imperative In-place Object-free In Assignment 1 we stack the deck against iteration by requiring recursive and iterative implementation of an in-place sort with at most O(log n) additional memory (stack) for Quick Sort, and O(1) additional memory for Merge Sort. Note that most of the algorithms and code students can find on the web do not satisfy this

requirement – students did find iterative algorithms, and even comparisons of multiple algorithms, but they did not tend to meet these memory requirements (or the students did not correctly recognize those that did). In the case of QuickSort, it is essential to deal with the smaller partition first to avoid the stack growing to O(n) in the worst case., and correct iterative pseudo code was provided In the case of MergeSort, it is quite tricky to program for arbitrary N if one thinks top down. It is quite easy if one works bottom up, but the pseudo code provided was the traditional top down recursive form. Here students found, usually for the first time, that the recursive version was dead easy, and iterative hard. Note that the algorithms were provided in the first week of the course, and the two (QS and MS) algorithms hand-simulated in an optional workshop in the first two weeks, and students were constrained to use only C or C-like Java (use of Objects prohibited). This meant students were able to get onto it early, and appreciate its difficulty earlier, and there is clear evidence that the entire class do indeed start on it earlier, and the optional workshops have excellent attendance, even though only about a third hand in a complete early-bird assignment. 2) Asst 2: Object-Oriented with strict Interface One of the issues with Java’s brand of OO is that so-called Classes have optional components to their interfaces, violating the expectation that anything you write for the Interface will work with any implementation. The List Abstract Interface is no exception, including optional features that are not implemented in all implementations. The Array List and Linked List implementations each provide extensions that are optimized toward their implementation, but not supported in the other. This assignment required implementing the Sorting algorithms using only the mandated features of the List interfaces, and ensuring that the same program with just ArrayList substituted by Linked List throughout, worked in the correct order (n log n) irrespective of which implementation was used. Very few succeed. This illustrates some very important points in relation to the nature of the OO contract and the difficulty of specifying an interface that works well independently of implementation. It is also rather salutary for students who see themselves as good OO programmers, and often have extensive commercial experience too! 3) Asst 3: Logic vs. Functional – plus extras The very short Prolog and Scheme programs are well known, especially for QuickSort in the case of Prolog. However, there is an issue with these naïve QuickSorts that is better known for naïve Reverse. They depend on Append to rescan a list. Thus students are asked to avoid using Append in their Prolog implementation. Whilst multiple passes are common in Functional Style, optimization of QuickSort can lead to extraordinary improvements for Prolog – indeed an optimized Parallel

version can achieve O(log n) time on a n processor PRAM [5]. As these solutions are so trivial, students are also asked to implement Permutation sort and a Quadratic sort (bubble, insertion or selection) in Prolog. By popular request, the Scheme implementation may be replaced by Haskel, Curry or F#. For example QuickSort using list comprehensions is a two liner in Curry or Haskell, and one of those just deals with the empty list, qs qs

[] = [] xs@(p:_)=

qs qs

[x|x