Embedded questions in ebooks on programming: useful for a

Embedded Questions in Ebooks on Programming — Useful for a) Summative Assessment, b) Formative Assessment, or c) Something Else? Juha Sorva

Teemu Sirkiä

Department of Computer Science Aalto University, School of Science Espoo, Finland

Department of Computer Science Aalto University, School of Science Espoo, Finland

[email protected]

[email protected]

ABSTRACT

further ideas and research on these topics. Although not all ECQs are multiple-choice, some are, and it is instructive to consider the literature on multiple-choice questions. We do that in Section 2, next, which also presents some related work on ebooks. Section 3 illustrates the uses of ECQs in a particular ebook for CS1. In Section 4, we discuss some challenges in using ECQs, before Section 5 concludes the article by pointing to some open questions.

We discuss uses of embedded computing questions (ECQs) in interactive electronic textbooks on programming, identifying a non-exhaustive list of three main categories of ECQs and nine subcategories. The main categories are: ECQs that introduce content, ECQs that reinforce learning, and ECQs that highlight content. We provide examples from an existing ebook, discuss how student perceptions may pose challenges to the use of ECQs, and invite the research community to debate ECQs and investigate them empirically.

2.

RELATED WORK

CCS Concepts

2.1

•Applied computing → Interactive learning environments; E-learning; •Social and professional topics → Student assessment; CS1;

Multiple-choice questions (MCQs) are commonly used in summative assessment. They are easy to grade and amenable to automatic assessment, making them commonplace in computer-based tutorials and interactive electronic textbooks. However, MCQs have also received widespread criticism. Critics have argued that, at least in its typical forms, MCQ-based assessment: evaluates mere memorization; only requires selecting rather than constructing responses; promotes a transmission model of learning; allows students to reach the correct answer without understanding why; cannot probe students’ conceptions; provides only predetermined feedback; does not involve students in goal-setting; encourages surface learning strategies and “gaming,” and is unvalued by many students (see, e.g., [29, 22, 33, 14, 3]). On the other hand, it has been argued that if — if — MCQs are carefully crafted, they can be used to gauge students’ skills at application and analysis. Such work has also been undertaken within computing education (e.g., [33, 16, 28]). Some critics, too, acknowledge that MCQs can be useful as a supplement to other forms of assessment (e.g., [3]). Much of the discussion around MCQs in the literature has focused on the use of batteries of MCQs for testing, especially MCQ examinations in which students have a single attempt at finding a solution. There are other uses, too, including midcourse tests with formative feedback [5, 25], peer discussions around MCQs [18, 23], and student-generated MCQs [6].

Keywords interactive ebooks on programming, embedded questions

1.

INTRODUCTION

Electronic books (ebooks) are increasingly popular, also within computing education [13]. One advantage that ebooks have over traditional textbooks is that they can contain interactive elements such as practice problems, simulations, discussion threads, etc. Students’ answers to some embedded tasks can be automatically assessed. Questions that the user answers by selecting from given options or typing in a short answer are a simple form of interactive, automatically assessable ebook content. In this paper, we use the term embedded computing question (ECQ) for questions of this type which have been integrated into an electronic textbook on computing. This article is a discussion paper that examines the roles that ECQs can play, suggests uses for ECQs, illustrates those uses with examples from an ebook on introductory programming (CS1), and invites Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

2.2

Multiple-Choice Questions

Ebooks in Computing Education

A growing number of interactive ebooks have been created for purposes of teaching programming and data structures (see, e.g., [1, 9, 10, 17, 20, 24, 27, 30, 31]). In addition to more generic ebook features, these books have interactive content that is specific to computing, such as executable and editable code fragments, Parsons problems [7, 9], and

Koli Calling 2015, November 19 - 22, 2015, Koli, Finland c 2015 Copyright held by the owner/author(s). Publication rights licensed to ACM.

ISBN 978-1-4503-4020-5/15/11. . . $15.00 DOI: http://dx.doi.org/10.1145/2828959.2828961

152

• The ebook is not self-contained. Some ECQs require students to work outside the ebook (in an IDE).

dynamic visualizations of algorithms and programs [26, 31]. Many of these ebooks feature ECQs, primarily in the multiple-choice format. Typically, the stated purpose of the ECQs is to provide students with opportunities to review what they have read and to check their understanding (e.g., [10]). Sometimes, ECQs are used in combination with visualizations (e.g., [31]); the interaction required by ECQs may help students overcome the “deceptive clarity” of the visualizations and promote sense-making and self-monitoring [15, 19, 21]. The use of ECQs in ebooks on computing is consistent with the recommendations of Hazzan et al. [12], who have argued that teachers can illuminate different aspects of computing and provide students with richer cognitive practice by asking questions of various types in addition to posing programming problems.

3.

Below, we discuss several roles that ECQs may fulfill. Our chief concern, here, is not cognitive levels, question formats, or specific topics; it is how the ECQs relate to the rest of the ebook and the students’ learning environment. In these terms, we identify four main roles for ECQs in our ebook. Introduce Content: The ECQ introduces new content that has not been previously covered in the ebook. We identify three subcategories: Teach a New Topic, Create Puzzlement, and Ease into a Project. Reinforce Learning: The ECQ encourages the student to use their understanding of previously covered content. We identify four subcategories: Interpret Content, Coding Drill, Discover a Result, and Scaffold for Coding. Highlight Content: The ECQ draws attention to something important or easy to miss. Two subcategories are: Check Outcome and Request Confirmation.

ROLES OF EMBEDDED QUESTIONS

In this section, we consider several distinct roles that ECQs may play in an ebook. Some of these roles are specific to computing and some are more generic. We approach the topic by considering how ECQs have been used in an online ebook that we have personally contributed to. The broader pedagogical approach used in the ebook has been discussed by [32]. Our ebook has been tailor-made for a large-class CS1 at Aalto University. The most important learning activities in the ebook are programming problems in which the students program in an offline IDE, but which are introduced and submitted through the ebook, often extending skeleton code that has been discussed in the text. These programming tasks are also pivotal in determining the students’ grades. More central to the present article, however, are the over 200 ECQs of the ebook, which play supporting roles in learning and assessment. Overall, our use of ECQs in the ebook and CS1 has the following characteristics.

The roles are not mutually exclusive. The subsections below discuss each role in turn and provide examples, which have been abbreviated and adapted from the Finnish originals in the ebook; we only present the gist of each example question.

3.1

ECQs that Introduce Content

ECQs in this category aim, in one way or another, to introduce new computing topics that have not otherwise been covered by the ebook. Students are expected to learn about the new topic as they discover the answer to the ECQ. The goal of this MCQ is to Teach a New Topic: You’ve now seen something of how to work with integers, but what about division, which we need for our application? Experiment in the REPL: Which of the following best describes what the operator / does? [Several options, elided.]

• We use multiple formats: typical MCQs with one correct option, MCQs with multiple correct answers, and short answers into text fields (e.g., a number or a line of code). • The ECQs are automatically assessed so as to provide instant feedback and to make the ebook suitable for independent study by large numbers of students. • Most ECQs are worth a minuscule number of points towards the course grade. • The MCQs, and to some extent the short-answer questions as well, produce written feedback. The feedback often does not explicitly reveal the correct answer but guides the student towards it with a hint or with a reference to a visualization or other section within an ebook (cf. [5]). Most correct answers also produce written feedback that may additionally clarify why other answers were incorrect. • Students can submit multiple times (which provides opportunities to close the gap between their present conceptions and the intended ones [22]). To discourage superficial guesses, the number of attempts to gain points for a question is limited; the limit depends on the question but is always intended to be high enough that each student can reach the correct answer with the help of the feedback. • The questions target various levels of Bloom’s taxonomy [2], most commonly Apply or Understand. • The questions match various patterns of computing questions identified in the literature [12, 28], such as code tracing and completion tasks.

The answer to this question has not been previously presented in the ebook, and the student is unlikely to know or guess it unless they have prior experience. They are expected to find out the answer themselves, primarily by working in the REPL (i.e., an interactive interpreter). It is expected that after they have done so, answered the question, and read the feedback, they will have an understanding of (truncating) integer division that is sufficient for present purposes. There are several other noteworthy things about the above question, which also apply to many other questions in the ebook (both within and outside this category): • The question seeks to encourage a habit of experimenting with language constructs and API methods in the REPL. • It somewhat blurs the line between MCQs and programming assignments. • It covers content that might have been covered by a static text in a textbook or less interactive ebook. An ECQ may also serve to Create Puzzlement: Consider this statement: println(2147483647 + 1) What does it print out? Try it in the REPL. Did the output match your expectations? Please also enter the output here: [Text Field]

153

This is similar to the previous question in that it introduces something not previously covered, but this question does not so much aim to provide an understanding of a topic but to raise questions and motivate the following text or tasks. It is interesting to note here that the answer to a question can be an integral part of a book despite the fact that it is never explicitly presented. For instance, the text in our ebook that follows the above question is written so that it makes reference to “the output,” making clear the need to answer the ECQ before proceeding with reading. In an ebook, it would also be possible to block access to certain sections until particular ECQs are answered, or to warn of such missing prerequisites dynamically. ECQs may help a student Ease into a Project. For instance, in our ebook, an assignment that requires students to extend a “large” program (dozens of classes, several of which the students need to understand) is preceded by ECQs about the given code. The ECQs are not intended to teach computing concepts as such but to provide structure to the students’ attempts to make sense of a relatively complex task.

3.2

Use the operators + and * to create a pastiche of the theme from Jaws. That is, write an expression of type String which consists of the following parts, in order: 1.