Developing a Common Format for Sharing ... - Semantic Scholar

Developing a Common Format for Sharing Programming Assignments Stephen H. Edwards (chair)

Jürgen Börstler

Lillian N. Cassel

Virginia Tech 2050 Torgersen Hall (0106) Blacksburg, VA 24061

Umeå University

Villanova University Dept CSC Villanova PA 19085

[email protected]

[email protected]

[email protected]

Mark S. Hall

Joseph Hollingsworth

Univ. of Wisconsin—Marathon Co. Wausau, WI 54401

Indiana University Southeast New Albany, IN 47150

[email protected]

[email protected] assignment instructions, preparing supporting libraries or scaffolding code, and setting up feedback mechanisms to improve student performance. While many instructors are willing to share the results of their work, most do not have any extra time or effort they can devote to publishing or documenting so that their assignments can be used or adapted by other educators.

ABSTRACT Computer science educators spend a lot of effort designing programming assignments, and many are willing to share the results of this investment. However, sharing of programming assignments occurs primarily in an ad hoc manner through informal channels. There are no widely used mechanisms that support instructors in finding and sharing such resources. Often, the additional work required to prepare and self-publish assignment resources in a way that others can then adapt or reuse is a significant inhibitor. Also, other instructors may have to spend an inordinate amount of time and effort to reshape a potential assignment into something that can be used in their own courses. This working group report proposes a common format for packaging assignments for sharing. This format is easy for instructors to create (requiring no specialized tools), is extensible and flexible enough to handle assignments written for any programming language at any level of proficiency, supports appropriate metadata, and is easily manipulated by software tools. As more and more instructors use automated grading tools to process student submissions, it is our hope that such an interchange format can lead to a community practice of sharing resources in a way that overcomes existing barriers to such reuse.

In addition, there are no well-known community services where instructors can go to share assignment resources easily, or where instructors can go to look for new ideas. Some ad hoc collections do exist, such as the Stanford “nifty assignments” site [18]. The nifty assignments site provides electronic resources for assignments showcased in the “nifty assignments” special sessions held at SIGCSE. While the assignments at this site demonstrate high quality and are easy for many to reuse, the bulk of assignments produced in our community remain hidden and difficult to access. Other collections like CITIDEL [5], the National Science Digital Library collection for computing educators, serve primarily as collections of web links with associated metadata. For programming assignments, such sites offer little value beyond Google. The purpose of this working group is to develop a standard interchange format that will make it easy for CS educators to share programming assignments with each other. In this context, we are focused on programming assignments, rather than the more general problem of sharing any educational resources used in a course. The aim is three-fold: to make it easier for instructors to package their work so that others can access it, to make it easier for instructors to find appropriate assignments to reuse or adapt, and to make it easier for tools to support the processing and grading of submissions.

Categories and Subject Descriptors K.3.2 [Computers and Education]: Computer and Information Science Education—Computer science education

General Terms Standardization

Keywords

2. THE PROBLEM

Interchange, reuse, programming assignment, automated grading.

When sharing assignment resources comes up, most instructors currently think of searching via an engine such as Google. If a standardized interchange mechanism were available, many would then think of developing some central repository service to which instructors could submit their work. Unfortunately, such repositories have not worked well within the CS education community in the past [16], and focusing on repositories would

1. INTRODUCTION Developing effective programming assignments for students is hard. Instructors invest a great deal of energy in coming up with engaging ideas that also illustrate the desired concepts, writing up

inroads — SIGCSE Bulletin

- 167 -

Volume 40, Number 4 — 2008 December

lead one to miss the true impact that an open standard for interchange could have.

3. RELATED WORK Of existing repositories targeting CS educators, CITIDEL [5] is most prominent. As the CS education community’s collection within the National Science Digital Library (NSDL), it provides umbrella access to a number of other indexed collections, including the Computer Science Teaching Center and the Computer Science Teachers Association’s K-12 repository. Also of note is the SIGCSE Education Links collection [22]. All of these provide the ability to browse or search a body of annotated web links. Some also provide the ability for contributors to attach files. All are poorly populated with programming assignments, and most educators prefer to use Google rather than such targeted link collections [16]. This may be because Google casts a much wider net, while providing essentially the same value. In contrast, by focusing on how to package all of the resources associated with an assignment in a self-contained, reusable form, this paper attempts to address this problem by working toward the infrastructure necessary to provide significant value beyond curated link collections.

First, a standardized interchange format would be a valuable tool for authors looking to add value to their textbooks. While instructor’s guides have their place, formulating some or all of a text’s programming assignments so they can be distributed in prepackaged, downloadable form could boost the intellectual value that went into writing those assignments in the first place. Similarly, authors of laboratory guides and manuals could package their efforts in a way that makes them more readily available to their audience—and hence, more valuable. At the same time, many institutions currently use automated tools to process programming assignment submissions from students, although the majority of these are homegrown. Nearly all the major publishing houses have produced some form of on-line assignment grading support in the past several years. More recently, open-source alternatives are beginning to appear. WebCAT (the Web-based Center for Automated Testing) [7][8], for example, currently is used by 30 separate institutions, with more adopters each semester.

3.1 ITiCSE Working Group Reports

Instructors using such grading tools often look for extensibility and the ability to add their own assignments. However, most existing systems present a kind of “black hole” effect—once you configure and set up your assignment, there is no effective way to export or share your work with others or perform incremental updates easily.

At the first ITiCSE conference in Barcelona in 1996, a working group published guidelines on the formatting of computer science assignments for submission to a digital library [14]. The guidelines suggested an assignment cover sheet with the following elements: title, summary, type of interaction, goal, objectives, time, feedback location, resources and materials required, and hints. In addition, this working group report discussed the potential problems that arise with respect to assignment solutions available via a repository. In general, students should not be allowed to access such solutions.

As these services begin maturing and as more instructors adopt them, the creation of an open standard for packaging and distributing assignments fills an important role and may help shape the path of grading tools. In particular, if instructors have an easy way to describe their assignments, this forms a natural medium of exchange for importing and exporting. At the same time, once tools begin using common formats, instructors who use such tools no longer have to manually create or package up their assignments—they can simply export the assignments they already use with such grading tools in ready-to-share form.

A 1997 ITiCSE working group report identified desirable qualities for lab assignments destined for inclusion in any repository [13]. The following categories of features for evaluating laboratory activities were identified: completeness, portability, flexibility, and creativity. This evaluation process would ensure that the benefits of submitting assignments would increase the quality and the longevity of the assignment.

With these considerations in mind, the ways in which an interchange format can affect repository features, usage, and value become more apparent. After all, a packaged assignment can include more than just an assignment writeup. It may also contain sample solutions, necessary support classes, skeleton files or scaffolding, reference test cases, input data files, or any other resources a contributor wishes to include. As a result, an assignment-sharing repository can offer its users much more in a single package than they would likely find with more generalpurpose web searches. Further, we fully expect that those reusing an assignment will want to customize, adapt, and tailor it to their own setting. By providing a clear explanation of the rights available to reusers, an interchange format can help address an important concern for instructors [16]. A repository could also provide mechanisms for instructors to share their experiences with assignments, rate assignments they have used, and then share any new variations or enhancements. Together, these capabilities go beyond the traditional library-based model of a searchable index combined with asset retrieval, and are important for providing the value necessary for building a self-sustaining community.


A working group led by Grissom et al. at ITiCSE 1998 described recommendations for implementing a web-based digital library to support computer science education [10]. The implementation proposed an edited repository that uses a peer review system that “formally reviews” all submissions to ensure quality. In addition, the group proposed classifying resources submitted to the repository using the ACM classification system [1], which identifies eleven categories that represent the spectrum of the CS discipline. The report also listed existing repositories of Computer Science education materials [10]. 3.2 Other Publications A study by Mitchell and Lutters supports the premise that “computer science course material repositories are under utilized, even though the demand for these materials is high” [16]. They conclude that the primary cause is the low visibility of CS repositories. In addition, the study noted that most CS faculty are not actively searching for repositories, and instructors that do search find existing repositories to be of limited value: “Nonetheless, faculty indicated concerns regarding the trustworthiness of materials and the difficulty in finding them”

- 168 -


[16]. However, CS instructors reported a high success rate when using textbooks and internet search engines.

enough to allow future extensions in unknown directions without major reworking.

A paper by Carbone et al. encourages academics to start thinking about the assignments they use in the classroom [2]. The paper explores various characteristics of programming tasks that can affect student learning and understanding: “Teachers were not able to recognize the internal difficulty of the exercises which prevented unconfident students from solving or submitting them and subsequently, led them to give up the course” [2]. In addition, the paper recognizes that teachers can struggle to find good programming assignments.

Support for self-contained assignments. Ideally, assignments should be self-contained to be portable. This means the format should support the inclusion of all sorts of assignment-related artifacts, including supporting class libraries, provided skeleton code, and test data or automated test cases. At the same time, provisions for these features should not get in the way of authors who do not use them. Support for licensing and access restrictions. While many instructors are willing to give their assignments away, others are concerned with intellectual property restrictions and licensing. Publishers are concerned with controlling access to their intellectual property as well. The format must provide a mechanism for imposing license restrictions, when desired.

Virginia Tech and Villanova are collaborating on an approach to create a repository that houses a collection of computer science course syllabi [24]. Tungare et al. discuss several tools that help support the creation, modification, and use of the knowledge captured in the syllabus repository. The tools “allow us to directly manipulate the knowledge embedded in course syllabi and to use and reuse them as a point of collaboration among the Computer Science faculty and other constituents” [24]. They also acknowledge that the “availability of a syllabus repository opens up possibilities for many innovative applications.”

Platform neutrality. system-specific.

Programming language neutrality. Instructors use many different programming languages to teach, so the format should not presume assignments are targeted at a specific programming language.

A General Purpose Registry/Repository Information Model was drafted in 2000 [9]. The draft “proposes a UML model and an XML services interface for a general-purpose registry/repository.” The draft defines a repository as “a facility that stores relevant descriptive information about registered objects, and a registered object is something important that an author or producer wants to have visible to the world so that it can be discovered and used by a client or customer.” Any implementation should support a services interface that can be used “to register new objects, provide appropriate metadata for those objects, browse or query registry content, filter out irrelevant references, and retrieve the contents stored in the repository.”

Ease of tool processing. In addition to ease of human access and creation, it is important that the format be easy for tools to process as well. This is necessary to promote import/export features as well as searching, indexing, and other capabilities.

5. PROPOSED FORMAT Given the requirements presented in Section 4, the authors met as an ITiCSE 2008 working group to develop an interchange format meeting the requirements. This article describes the result. The intent of the packaging format described here is to provide a self-contained bundle that combines all of the resources associated with the assignment—a package that stands on its own. An entire assignment is packaged as a single zip file with its own internal subdirectory structure. At a minimum, such a file contains an assignment writeup, preferably in HTML (assignment.html) and a simple metadata file (either assignment.properties or assignment.xml) with information about the assignment. Assignments that are more complex may contain many additional supporting files. Naming conventions guide the placement of commonly occurring supplementary files, to promote consistency and make it easier for others to navigate assignments and find what they are looking for. Authors also can provide any additional content they wish within the zip file, using whatever organizational scheme is most relevant to a given assignment. All text files within an assignment should be encoded using UTF-8. A smart repository could perform platform-specific conversion or normalization of line endings while uploading and downloading assignment zip files.

4. REQUIREMENTS FOR A SOLUTION To best capitalize on the potential for impact, a common assignment format should carefully balance these critical goals. Ease of creation. Instructors have little time to invest in new techniques, so any format must be simple and easy for instructors to create, preferably requiring no special tools or software. Ease of transportation. To maximize utility, a single-file assignment bundle would provide the best support for simple uploading, downloading, e-mailing, and sharing of assignments. Support for metadata. In addition to containing raw artifacts, such as assignment write-ups, test data, or example solutions, an assignment bundle must also support the storage of rich metadata. In addition to basic information such as author and institution, other information including CC2001 (or CC2005) knowledge unit information, course level, external library requirements, and even personalized information tags would be helpful. An extensible metadata format is best.

While such a zip file is directly usable by someone downloading it from a repository, we do not expect that most instructors would find it appealing to create one from scratch, especially considering the potential detail in the metadata file. Instead, we imagine that a repository could provide an easy-to-use form-based web interface for incrementally assembling various resources into a single assignment bundle and filling in the metadata values. Such an interface potentially could allow changes or additions to be made over time as assignments evolve. Such an interface may

Extensibility and flexibility. Ideally, instructors with simple, straightforward assignments should be able to package them up simply and quickly. At the same time, more elaborate assignments with multiple supporting documents, alternative implementation samples, supporting class libraries, and automated test data should also be supported. The format should be flexible


The format should not be operating-

- 169 -


subdirectories) use short (usually two-letter) names that conform to RFC 4646 [20]. Thus, the top-level of a multi-lingual upload might look like this: /assignment.properties /assignment.html /... other files ... /de/assignment.html -- folder in German /de/... other files ... /en/assignment.html -- folder in English /en/... other files ... /fr/assignment.html -- folder in French /fr/... other files ... /se/assignment.html -- folder in Swedish /se/... other files ...

use a wizard-like model or a direct-manipulation style of interaction, but its design is outside the scope of this paper. Instead, here we focus only on the structure of the interchange format itself.

5.1 Zip File Structure We propose the following subdirectory structure for an assignment’s zip file package (except for the first two files, all other content is optional): /assignment.properties (or .xml) /assignment.html (or .txt, .doc, .docx, ...) /include/… /lib/… /solution/… /software-tests/public/… /software-tests/reference/… /software-tests/private/… /start/… /frame/…

Authors who prefer to maintain symmetry in their multi-language assignment’s subdirectory structure are free to place all resources other than the metadata file, including those in the assignment’s primary language, within such language-specific subdirectories. For simplicity, however, when only one language is used, resources that are not contained within a language-specific subdirectory in the zip file are presumed to be in the assignment’s primary language, as defined by the metadata file. As a result, we presume any content within the zip file that is not contained within a language-specific subdirectory is written in the assignment’s primary language, as defined by the metadata file.

5.1.1 Metadata file The assignment’s metadata is stored in a single text file stored in the root of the zip file. This file may be in either Java properties file format [23] or XML. The metadata file contains the set of metadata properties defined in Section 5.2.

5.1.2 Assignment writeup

5.1.4 Optional contents

When most people think of an assignment, they think first of the assignment’s writeup, which might also be called an assignment specification, an assignment handout, or instructions for the student. Here, we will use the term writeup.

In addition to the assignment writeup itself, many assignments include a variety of additional supplementary files used in an assignment. These should be added to the zip file in appropriate subdirectories based on the conventions described below. Each subdirectory should also contain a readme.txt file with auxiliary information whenever necessary. Secondary language translations of supplementary files are placed in an identical subdirectory structure within the corresponding language’s directory in the zip file.

An assignment contains a writeup as a single file, preferably in HTML. Simple vanilla HTML is preferred since it maximizes ease of reuse by others; fancy layout or HTML generated by Microsoft Word or similar tools should be avoided. External style sheets and javascript resources should also be avoided—the intent is for each packaged assignment to stand on its own. HTML pages using extended characters outside the US-ASCII 7-bit set should have an appropriate content type specification in their header (UTF-8 encoding is the preferred choice). Images should be placed in an /images/ folder within the root of the zip file and referred to using relative paths. Relative links to other files within the zip file structure are also fine, including links to code files within the package.

If any source code files or binary files in any of these subdirectories are provided under a different license than the assignment itself (see Section 5.2.1), an appropriate license.txt file should be provided in the respective subdirectory describing the terms. Note that the license of any included resource must grant redistribution rights in order for the files to be included in an assignment upload that will be shared with others.

The writeup can be provided in multiple formats, if available. Although we do not encourage proprietary document formats (e.g., Microsoft Word), such formats are still acceptable. Potential submitters should not be discouraged from submitting by too strict requirements on the writeup format.

When necessary source files or binaries cannot be included as part of the assignment due to licensing restrictions, the readme.txt file should specify what source files or binaries are needed and where they can be obtained by someone using the assignment.

5.1.3 Multi-language support

INCLUDE FILES

Each assignment has a primary language (such as English, Spanish, German, etc.) defined in its metdata file. All of the assignment’s resources should be provided in this primary language. In addition, one can provide translated versions of any or all of the assignment’s files—the writeup, any sample solutions, any reference tests, etc.—in one or more secondary languages, which are also listed in its metadata file.

/include/ This directory should be provided if there are any program source files that must be available to the student for importation by their own solution code when it is being written. Examples include C or C++ header files, Perl modules, or library modules for interpreted languages that are typically imported in source form (e.g., PHP, Python, Ruby, Scheme, Prolog, etc.).

All assignment files in secondary languages should be placed in a subdirectory within the zip file named after the language in which they are provided. Languages (and their corresponding


- 170 -


All necessary source files should be placed in a subdirectory structure appropriate for the target programming language within the include/ subdirectory.

may be provided as feedback or “hints” to the student by some instructors, but students do not handle execution of these tests themselves.

A readme.txt file should describe where one can find the latest version(s) of these source files on the Internet, as well as links to any associated on-line documentation, if available.

Sometimes, authors may also wish to provide an additional set of public tests that he or she does intend to provide to students. Alternatively, some authors may wish to use (completely secret) private tests that are intended for instructor use only, rather than included in any form of (even limited) student feedback. Authors who wish to make such distinctions can do so by creating appropriately named subdirectories: /software-tests/public/... /software-tests/reference/... /software-tests/private/...

LIBRARY FILES /lib/ This directory should be provided if there are any compiled program files or libraries that must be available to the student for linking by their solution code. Examples include Java jar files, C or C++ libraries, etc. If the assignment is designed for a programming language that uses platform-independent binaries, then binary library files can be included in this directory. For languages that use platform-specific binaries (i.e., natively executable object code), the source code used to generate the necessary library files should be provided in a lib/src/ subdirectory along with an appropriate build script (such as an ANT build.xml file or a makefile).

Often, using separate files and/or classes for each group of software tests is sufficient. However, in cases where that does not offer enough control, the contributor can group tests into appropriate subdirectories corresponding to the subdirectory structure in /solution: /software-tests/public/1/... /software-tests/public/2/... /software-tests/public/3/... /software-tests/reference/1/... /software-tests/reference/2/... /software-tests/reference/3/... /software-tests/private/1/... /software-tests/private/2/... /software-tests/private/3/...

The readme.txt file should describe where the latest version of these binary and/or source files can be found on the Internet, as well as links to any associated on-line documentation. If only platform-specific binary files are included in your assignment upload, please be sure to provide appropriate platform dependency information in the metadata file. If possible, provide links in the readme.txt file indicating where alternative binaries (or source) for other platforms can be obtained on-line.

STARTING CONTENTS FOR STUDENTS /start/

SAMPLE SOLUTIONS

This directory should be used to provide the starting project students should download and work from, if desired, e.g., a model for a zip file that students can download to begin working on an assignment. The starting project may contain any files the author wishes to supply, including but not limited to any mixture of the following:

/solution/ This directory should be provided if a sample solution for the assignment is provided (strongly recommended!). Selfcommenting source code in the assignment’s target programming language should be used. Files should be arranged within this subdirectory the way a student’s solution would be organized. If only one sample solution is provided, it can be placed directly in /solution. If multiple sample solutions are available, they should be separated into numbered subdirectories as follows: /solution/1/... /solution/2/... /solution/3/...

SOFTWARE TESTS

1.

Complete source files provided for student use, requiring no student modifications.

2.

Source files that the student is directed to modify or extend, including non-working partial files or templates that require the user to fill in the blanks.

3.

Empty placeholders that stand in for files the student is to create entirely from scratch.

4.

Complete data files or other resources intended for loading or manipulation by student programs.

5.

Project management or build files (makefiles, ANT build files, etc.) that will help the student set up or build a project.

/software-tests/ This directory should be used to contain any tests or test data files for the solution that the student is to develop. These should be in a format appropriate to the style of the assignment and the intended target programming language. Examples include sample input text files (with or without matching expected output captures), sample input data files, JUnit tests, CxxTest tests, etc. The manner in which tests are to be executed is specified in the assignment’s metadata file.

Authors can use any appropriate subdirectory structure to organize or group resources with the /start folder.

Normally, the contents of the /software-tests folder is presumed to contain reference tests—that is, tests intended to be used by instructors or course staff for grading or sanity checks, but that are not designed for public release to students. Often, some limited indication of success or failure of reference tests

While the /start folder describes resources the student is supposed to have available as a starting point, the /frame contains resources that are only used for assessment, and that are never intended for student use or visibility.


FRAME /frame/

- 171 -


showing how the property would be defined in Java properties format, as well as how the same property would be encoded in XML.

Sometimes, an assignment or exercise may require a student to write a code fragment instead of a complete program. This code fragment may be a single function, a single method, a single block of code, or even a single statement. As a concrete example, consider JavaBat [19], which gives students practice coding questions. Each JavaBat question is about writing a single Java method.

5.2.1 Required metadata properties TITLE title = Dice Rolling Experiment

In this case, the student’s submission or answer is not compilable on its own. Continuing with the JavaBat example, the student’s answer is just a single Java method. While this can be considered a complete answer to the question that was asked, at the same time it is not a complete compilation unit or program that can be independently assessed on its own. Assessment of correctness may then require that the student’s work be placed inside a frame, which provides the environment in which the student’s submission is intended to be examined.

Dice Rolling Experiment The title property contains a title for this assignment, provided by the contributor. Like a paper title, it should be as descriptive as possible. However, titles are not presumed to be unique.

ABSTRACT abstract = Build a class that simulates \ rolling two pair of dice ... Build a class that simulates rolling two pair of dice ...

The /frame directory should be used to contain the frame used to assess the student’s submission, if necessary. Normally, it consists of one or more program source files with clearly identified placeholders where the student’s submission (or specific pieces of the student’s submission) must be substituted in order to produce a self-contained compilation unit or module for the target programming language.

The abstract is a free-form text property that provides space for a one- or two-paragraph abstract of what this assignment is about and why you would want to use it. Contributors should be encouraged to make the first sentence (or first 30 words or so) as descriptive as possible. Repository services can then use the first sentence (up to 30 words) as an extremely concise mini-abstract when presenting search summaries.

The /frame is intended purely for assessment use, and is not intended for student visibility.

PROGRAMMING LANGUAGE

5.2 Metadata File Content

programming.language = java

The assignment’s metadata is stored in a single text file stored in the root of the zip file. The metadata file can be either a Java properties file [23] called assignment.properties or an XML file called assignment.xml. The choice of the Java properties file format supports easy creation and maximal readability, especially for new users. A repository system need not use this format internally, however. An isomorphic XML representation of the same metadata may be more appropriate for tool processing or asset harvesting. As a result, the packaging format supports either choice for the metadata file. A smart repository could take uploads in either format and provide for downloads in either format, depending on the user’s preference.

This property describes the target programming language for this assignment (case-insensitive), using the common language name of the language (e.g., “java”, “c++”, “python”, etc.). Values such as “any” (or “any-oo”, “any-procedural”, “any-functional”, “anylogic”, etc.) or a comma-separated list of specific language names can be used to indicate assignments that do not require a specific programming language. Dependencies on the language version can be expressed in optional properties presented in Section 5.2.3.

AUTHOR(S) author1.name.first = Stephen H. author1.name.last = Edwards author1.email = [email protected] author1.url = http\://... author1.affiliation = Virginia Tech author1.phone = +1 540 231 5723

The metadata properties or fields are divided into four groups. Required properties should always be provided for all assignments, all the time. Since this set of properties imposes a de facto lower bar on the effort required of anyone creating a new assignment in this format, this set is intentionally small. Recommended properties are expected for all assignments, but some contributors may still fail to provide them (an appropriate upload interface can nag them without forcing them). Optional properties are not required. Contributors should be given a chance to provide them, but should not be nagged. We expect that many assignments will not provide some or all of the optional properties. Finally, repository properties are provided automatically by a third party, typically when an assignment is uploaded to some service. These properties are not intended for direct editing by end users or contributors, but are provided in downloaded assignments for informational purposes, and to aid in tracking the relationships between modified or adapted versions of assignments and their original ancestors.

Stephen H. Edwards [email protected] http\://... Virginia Tech +1 540 231 5723 The author*.* fields identify the author(s) of the assignment. At a minimum, one author’s name and email address should be provided as a point of contact. The remaining author fields are optional. Multiple authors can be specified by using different numeric suffixes to cover the number of authors/contributors

Sections 5.2.1-5.2.4 describe each category of metadata properties in turn. Each property is illustrated using a simple example


- 172 -


involved. For a repository, this information can be automatically supplied using data from the contributor’s account profile.

The license property provides the name of the license under which rights to use this assignment are granted. If omitted, the assumption is “all rights are reserved by the copyright holder.” When a new assignment is uploaded to a repository, a default could be suggested, such as the Creative Commons license in this example. It is also possible to provide a drop-down list of the most common license choices, including the creative commons choices and approved OSI open source licenses [17]. The license.url property provides the URL under which the full text of the license is available on-line. In the XML format, these two properties can be embedded alongside the copyright owner in the tag using the rel-license microformat [21].

The XML structure shown in the example above is provided for simplicity of manual creation by authors when necessary. However, repository systems may alternatively embed vCard records directly within an tag using the hCard microformat [12]. This alternative provides a richer, more standardized way to represent individual contact information for XML metadata that is not hand-edited.

DATE date.created = 2008-12-31 14:05:27 UTC date.modified = 2008-12-31 14:05:27 UTC

5.2.2 Recommended metadata properties ASSIGNMENT TYPE

type = program

The date.* fields contain a pair of timestamps that indicate when this assignment was created and when it (or its metadata) was last updated. These attributes could presumably be tracked and automatically inserted by any assignment repository.

program The type property gives one of the following values, characterizing the type of deliverable the student will produce: program, class, method, function, procedure, block, statement, declaration, or other. While all of these values are phrased in the singular, each may also be used for assignments where multiple deliverables of the same type are produced (e.g., an assignment where a student writes two cooperating classes, or implements two methods, etc.).

LANGUAGES languages = en en The languages tag provides a comma-separated list of one or more languages in which this assignment’s files are written. Values should conform to RFC 4646 [20]. If no value is given, then “en” is assumed.

ASSIGNMENT CONTENT keywords.topics = recursion, methods, ...

Individual files within the assignment should use one language only, but multiple translations within the assignment are fine. Note that the first entry in this list is considered the primary language for this assignment, and it is expected that all supporting files will be available in that language. Other languages in the list would then imply that translations for some or all of the files would also be provided in the given language as part of this assignment.

recursion, methods, ... The keywords.topics property provides an optional commaseparated list of keywords or tags intended to identify the knowledge content that is the focus of the assignment, where appropriate. Contributors are encouraged to be specific. If possible, CC2001 (or newer) knowledge units or terms from the Computing Ontology [3] are recommended. Contributors also may use more generic terms, including any other appropriate labels they feel will be recognizable by others.

COPYRIGHT copyright = (c) 2008 Virginia Tech (c) 2008 Virginia Tech

PREREQUISITE KNOWLEDGE keywords.prerequisites = variables, ... variables,...

The copyright property indicates the copyright holder and date. If the author is interactively creating a new packaged assignment through a repository site, the copyright information could be automatically filled in using the contributor’s account information and past preferences.

The keywords.prerequisites property contains a commaseparated list of keywords or tags provided by the contributor to indicate what skills students are expected to have in order to be able to tackle this assignment. Unlike the “topics,” these skills are not the focus of the assignment. However, students who have not mastered these skills may not be able to undertake the assignment or produce an effective solution. The same contents comments requiring vocabulary usage as in “topics” apply here as well. The XML tag is placed within , alongside .

LICENSE license = Creative Commons Attribution-\ Noncommercial 3.0 United States License license.url = http\://creativecommons.org/\ licenses/by-nc/3.0/us/ Creative Commons Attribution- Noncommercial 3.0 United States License


OTHER KEYWORDS OR TAGS keywords.other = GUI, JUnit, TDD, ... GUI, JUnit, TDD, ...

- 173 -


The keywords.other property contains a comma-separated list of keywords or tags provided by the contributor. This catchall keyword list is available for including any subject-related terms that do not fit in either the keywords.topics or keywords.prerequisites category. Examples would include keywords related to the context, problem area, or novelty of the assignment, rather than specific programming skills or concepts. There is no predefined vocabulary, although authors are encouraged to use common terms recognizable by others.

TEST COMMAND command.test = java org.junit.runner.\ JUnitCore ReferenceTests java org.junit.runner.JUnitCore ReferenceTests Indicates what action to take to run or test a student submission (which could be completely empty if it is inapplicable). Common values might be:

LEARNING OBJECTIVES

•

main.exe arg1 results.txt

•

ant run-tests

•

make testall

This property provides a free-form text field for the contributor(s) to explain the educational goals or learning objectives of the assignment in prose form, as a supplement to the “topics” property and the abstract. Even when the abstract already discusses learning objectives, they should be repeated here in full. Contributors are encouraged to be precise, using active verbs and relevant levels from Bloom’s taxonomy where possible.

•

java MyClass abc results.txt

•

java org.junit.runner.JUnitCore MyTests

•

A single sh-style command line (a simple in-line script)

PREREQUISITE ASSIGNMENTS

5.2.3 Optional metadata properties PROGRAMMING LANGUAGE SPECIFICS

learning.objectives = This assignment is \ designed to allow students to practice... This assignment is designed to ...

Ideally, the run/test command should be something that any instructor who wants to reuse the assignment could presumably do.

prerequisite.assignments = ... ...

programming.language.vendor = GNU g++ programming.language.version = 3.5+

This property contains a (possibly empty) comma-separated list of UUIDs (repository.uuid values, see Section 5.2.4) identifying any other assignments students should have completed before they will be ready to undertake this specific assignment. For example, if students develop a linked list in a “Program 1” assignment, and then in “Program 2” they reuse their list implementation in building a larger solution, “Program 2” should list the UUID for “Program 1” in its assignment.prerequisites.

If this assignment requires a specific implementation of the target programming language, specify it in the optional programming.language.vendor attribute. Similarly, a specific version of the target programming language can be specified in the optional programming.language.version attribute. Use a suffix of “*” to indicate any version that begins with the given prefix (e.g., “3.5*” matches 3.5, 3.5.1, and 3.5.7, but not 3.6 or 4.0), or “+” to indicate any higher version (e.g., “3.5+” matches 3.5, 3.5.1, e.5.7, 3.6, and 4.0, but not 3.4.9).

BUILD COMMAND command.build = javac *.java javac *.java

INTENDED COURSE keywords.course = CS1, CS1705: Introduction\ to Object-oriented Development I

This property indicates what action to take to compile or build a student submission, if applicable. This value could be omitted or left empty for interpreted languages, for example. We can imagine that some common values might be things like: •

make

•

ant

•

javac *.java

•

g++ *.cpp -o main.exe

•

A single sh-style command line (a simple in-line script)

CS1, CS1705: Introduction to Object-oriented Development I The keywords.course property identifies the academic course(s) for which this assignment is targeted. It can be a comma-separated list. Where possible, complete course titles should be provided, although contributors can also use generic terms (like “CS1”) or tags (like “introductory”) or any other appropriate label they feel will be recognizable by others. Specific course numbers can also be provided if desired, which may help those searching for related assignments, or searching for related resources on the internet at large.

Ideally, the build command should be something that any instructor who wants to reuse the assignment could presumably do. For assignments that include a “frame” (see Section 5.2.1), the build command should use the frame correctly.


- 174 -


EXTERNAL URLS

ESTIMATED TIME

info.url1 = http\://courses.cs.vt.edu/\ ~cs1705/assignments/tetris.html info.url2 = http\://catspace.web-cat.org/\ assignment/1234 info.url3 = http\://www.citidel.org/\ assignment/1234

estimated.time = 2 weeks The estimated time is an estimate of how many hours (or days, or weeks) students should be given to work on a solution—normally, the difference between when an assignment is handed out and when it is due. A laboratory assignment might have an estimated time of one or two hours, while a longer programming assignment might have a value of one or two weeks.

http\://courses.cs.vt.edu/~cs1705/ass ignments/tetris.html http\://catspace.web-cat.org/assignme nt/1234 http\://www.citidel.org/assignment/12 34

The format for this value is an integer followed by a unit name. The allowed unit names are hour, hours, day, days, week, and weeks.

ESTIMATED SIZE

The info.url* properties provide space for a set of publicly accessible URLs where people can view information about this assignment. For example, the contributor might provide a URL for a site he or she maintains with more information about this assignment. This is another way for contributors to publicize their own work and efforts related to this and other assignments. It is purely optional. Alternatively, any repository to which the assignment has been uploaded may also provide a URL for information from that repository. That may be a way for others reusing an assignment to post comments, read about the experiences of others, provide their own experiences, and so on.

estimated.size = 250 This property indicates how many lines of code (not including comments or blank lines) students are expected to write to produce a 100% correct solution.

5.2.4 Repository metadata properties UNIVERSALLY UNIQUE IDENTIFIER repository.uuid = \ 611B46BB-0DA6-40DB-8988-932AEB6A79A4

CITATIONS

info.cite1 = \ http://doi.acm.org/10.1145/1384271.1384303 info.cite2 = P.V. Gestwicki, Computer \ games as motivation for design ...

The repository.uuid is a universally unique identifier [15] for this assignment. In many cases, this would be generated by the repository or service to which the assignment is uploaded.

http://doi.acm.org/10.1145/1384271.1 384303 P.V. Gestwicki, Computer games as motivation for design ...

ASSIGNMENT VERSION repository.version = 1.2 The repository.version is a generated version number for this assignment. Repositories may wish to use an appropriate versioning scheme to allow contributors to revise assignments, or even to allow others to create “branched” versions of assignments that meet alternative needs. Each new version would presumably have its own UUID to distinguish it from other versions of the same assignment.

Occasionally, assignments are taken from textbooks, or may be associated with a conference paper or a research article. The info.cite* properties provide for references to one or more external citations. For maximum flexibility, each individual citation is a free-form text field that can be as simple as a document URL, or as complex as a full, human-readable citation. It is also possible to embed a more structured citation record within this field using an XML-encoded BibTex reference or an OpenURL-encoded citation [4].

ANCESTRY repository.parent.uuid = \ 2932F022-53E2-4288-8CA6-871FB93033BE repository.root.uuid = \ A306987F-E867-4C6E-A3CB-39C0FAB71973

DIFFCULTY estimated.difficulty = 3

The estimated difficulty is an intuitive indication of the level of difficulty of this assignment, on a scale of 1 to 10, where 10 is the most difficult. Difficulty should be rated by the contributor assuming that students have mastered the prerequisites listed in the keywords.prerequisites and that students have received the necessary classroom preparation to write solutions using the keywords.topics. In other words, for students who are prepared for this assignment, how difficult or tricky is this assignment?


These properties provide a way of recording the “ancestry” of a given assignment by referring back to the “original” (or root) assignment from which this particular variation was derived, as well is the immediate “parent” assignment. These fields need only be provided when this assignment is descended from (or a variant

- 175 -


of) another. The root field contains the repository.uuid for the original assignment this one was derived from—i.e., the “root” of the variant/derivation tree to which this assignment belongs. The parent field contains this assignment’s immediate parent in its ancestry tree.

programming.language.vendor, estimated.difficulty, estimated.time, estimated.size, repository.root.uuid, info.*, repository.history, repository.acl, and repository.version. The Dublin Core field DC.Publisher will receive a value from the repository to which the file is initially loaded. The Dublin Core field DC.Contributor will receive a value from the system used to create the assignment object. The Dublin Core field DC.Format will receive the value “application/zip”.

CHANGE HISTORY repository.history = ... ...

6. IMPLICATIONS FOR A REPOSITORY 6.1 Copyright and Licensing

The repository history is a human-readable summary of the change history for this (version of the) assignment. If the repository or service hosting this assignment allows contributors to describe changes when they upload modified or customized assignments, then a consolidated presentation of the change history associated with this particular version can be generated here for informational purposes.

Every repository exists to provide access to materials of interest to a particular set of users. Intellectual property rights govern what can legitimately be presented. Any repository accepting submissions in the format described here expects that the submitter has the right to present and share the material, just as a repository would for any other type of material. This paper describes in detail a format for specifying a programming assignment. Particular issues related to intellectual property come from the source of the programming assignments described. In some cases, the submission is the original work of the submitter and is not derived from work of any other source. The format requires that the submitter specify the copyright terms for such work. This is a required field in the metadata specification. Some repositories may have expectations about the copyright attached to a submitted work. The metadata field provides the information necessary for a repository to accept or reject the submission, based on its criteria for copyright.

ACCESS CONTROL repository.acl = SomeGroupName SomeGroupName The repository may wish to provide a comma-separated list of identifiers used to enforce access control restrictions for this assignment at its home repository, provided in the metadata purely for informational purposes. Here, “acl” stands for “access control list.” Changes to this field should be made directly through the repository interface’s access control features.

Many programming assignments may have a root in a published and copyrighted work. The required metadata described here provides fields in which a submitter can describe the source of the candidate work. A given repository will have policies on how derived work will be addressed. One option is to seek permission from publishers to present assignments that are closely related to material in published works, with the expectation that the wide exposure of a small amount of the text will provide publicity that will draw attention to the rest of the book. Another option is to require that submitters obtain permission to reuse the source material. In some cases, the work may be derived from very old sources and may have been modified to the point of bearing little resemblance to the original. It is up to the submitter and the repository manager to determine what permissions are needed and to obtain them. This standard provides a mechanism to convey the information needed to make these decisions. It does not endorse the publication of works without permission of the copyright holder.

CUSTOM PROPERTIES custom.property = some value One strength of the Java properties format is its extensibility. Since a properties file is just a set of name=value pairs, it is easy to add new properties to the metadata, including new properties meaningful to assignment writers, reusers, repositories, or tools. This same kind of extensibility can also be supported in the XML format by providing a generic way for custom properties to be added within the context of the XML schema for the metadata file. As many new custom properties can be provided as desired, in either format.

5.3 Mapping to Dublin Core

6.2 Coping with Posted Solutions

Table 1 shows the mapping between the assignment metadata fields and the standard Dublin Core Element Set [6]. A synthetic author.name field for an assignment can be produced by concatenating the author.name.first and author.name.last fields and will map to the DC.Creator field. The concatenated author.name field is constructed by the assignment entry site from the author information provided by the submitter. The contents of the type field are drawn from a controlled vocabulary that is specific to this application.

Solutions to problems form a valuable resource for instructors. An option in the submission format described here allows a problem submitter to provide one or more sample solutions for reference. Students are very adept at finding posted solutions and so repositories have a challenge to maintain the integrity of solutions provided with a submission. This represents a problem for the submitter, who wants the solution restricted to instructors, and to potential users of the assignment who do not want to use an assignment for which solutions are available. A repository manager has some possible responses to this problem that may improve confidence for the submitter and for the potential user.

The following assignment metadata fields do not map to Dublin Core Elements: learning.objectives, command.build,


- 176 -


•

•

A repository may register users and identify them as instructors or students, then limit access to some sections of the repository. There is a potential for republication of resources that are intended only for instructors, say by instructors who want to make them available for their own students. Such experiences have been observed and serve as a warning to be wary about the security of any resource that is posted, even with restricted access.

Table 1: Mapping of metadata attributes to the Dublin Core.

A repository might do a crawl of the Web seeking solutions to a particular problem and collect those solutions. The repository could then offer a plagiarism detection service for instructors. The idea would be to find the solutions that the students are likely to find and make them available for comparing against student responses.

Assignment Metadata Field

From the perspective of the potential user of a posted assignment, the safest way to assure the integrity of the assignment is to use the posted assignment and its solution as a starting point and to create a derivative that addresses the same or similar learning objectives, uses similar components, perhaps produces similar output, but has noticeable differences that make it unique so that a copied solution will be identifiable and will not satisfy the current assignment. If the assignment comes from a repository that offers a plagiarism detection service, the modification of the assignment may not be necessary. However, if the project is to be used in consecutive terms, the possibility of archived solutions cannot be avoided and using a modified version is still a safer prospect.

6.3 Quality Assurance Every repository faces challenges in assuring the quality of its content, and a programming assignment repository is no exception. On the one hand, there is the basic issue of correct and consistent content in a submission. Is the descriptive material sufficient and appropriate for the submission? Is the assignment appropriate for the audience indicated? Is the prerequisite list of topics suitable for this assignment? If there is a dependency among several assignments, is that sufficiently and clearly indicated? These are important questions at the time of a submission, but they must also be asked again later. Are the links to related materials still valid? Have curriculum practices changed in ways that affect the stated dependencies? The standard programming project format described in this report is intended to provide sufficient explicit information to allow these factors to be checked when the submission is considered for use.

title

DC.Title

abstract

DC.Description

programming.language

DC.Description

keywords.topic

DC.Subject

keywords.prerequisites

DC.Description

keywords.course

DC.Subject

keywords.other

DC.Subject

author.name (derived)

DC.Creator

date.created

DC.Date

date.modified

DC.Date

language

DC.Language

copyright

DC.Rights

license

DC.Rights

license.url

DC.Relation

type

DC.Type

prerequisite.assignments

DC.Relation

repository.uuid

DC.Identifier

repository.parent.uuid

DC.Relation

repository.root.uuid

DC.Relation

of the resource and will test them for accuracy. They will verify that all components are present and correct and ready for use. They will review the stated goals of the assignment and verify that those are addressed by the materials included in the solution. They will attempt to install and use any materials that are supposed to be available for the eventual user of the resource. After this review, the material is added to the collection. A user will have a reasonable amount of confidence that the submission is what is described and that it is suitable for the stated purpose. It is possible that the suitability of the assignment will change over time and the validity of the reviewers’ assessments will have less weight as time passes. Some repository organizations may choose to institute a repeated review of a resource after some period. This may be especially suitable if a resource has not had a lot of downloads or if the downloads have not been recent.

6.4 Content Quality: Moderator vs. User Ratings There are several models for addressing questions of content quality. These are decisions to be made by a repository organization; however, the consequences of those decisions will be significant to the submitter and to the user of a resource. The submitter will be conscious of choices that will affect the way his or her work is treated once it is released for public access. The user will be concerned about the confidence with which a posted resource can be used or adapted.

In the case of relatively open submission followed by user comments, the resource submitted becomes accessible to the public quickly, but there is little control over the quality of the submission. There would usually be at least a brief review to be sure that the submission is real and not spam, but it may otherwise be unchecked. After the submission becomes available, comments by users become the effective review and provide valuable information to other potential users. This is the familiar Amazon.com model of feedback on a resource. There is always the possibility of a glowing review written by the author of a resource or by a colleague. Some reviews may be falsely negative due to a misunderstanding of the instructions or an attempt to use the material in a way that was not intended. Some

There are two basic models of content quality control: Pre-entry review and post-entry comments. In the first case, an editorial review panel gives careful attention to a submitted resource and approves it for inclusion in the collection. The intention is that the reviewers will pay close attention to the stated characteristics


Dublin Core

- 177 -


at one time. There will be instances of revision by the author(s) of the assignment. Some revisions will be minor updates: just wording changes to make something more clear or corrections of typing errors. A repository will usually want to replace the original submission with the revised version. Most will not want to present two nearly identical versions. The assignment metadata format includes fields to identify the submission and to date the original entry and its modification. An interface designed to accept a user submission must provide access to an existing version of an assignment so that the owner can make revisions.

submitters may hesitate to subject their work to this type of feedback in case negative comments might cause them harm in their departments.

6.5 Incentives for Contributors Another issue of concern to repository managers is how to solicit quality submissions. This has been a problem for various assignment sharing efforts in the past, where the relatively high cost of preparing work for sharing and the relatively low benefit to the original author have together made it difficult to develop critical mass. While the assignment-sharing format itself does not address this concern, there are potential steps that a repository manager can take to help address this particular problem.

Another class of revision originates not with the owner of the submission, but with another person who has used the assignment or considered using it, and has determined that modification will make the assignment better, at least for his or her own students and circumstances. Such reuse should be subject to the permissions associated with the original submission and should acknowledge the original with a link in the repository.root.uuid field. In addition to acknowledging the source of the derived assignment, this allows the repository to generate an ancestry tree for a given assignment, showing the various descendents of a common root and also the the various ancestors of a given assignment.

First, repositories can track many useful statistics about the works of those who contribute, including both authors of assignments and writers of reviews. For example, it is possible to track the number of views, number of downloads, and average user rating for a contributed assignment. It is also possible to track an assignment’s rank in terms of ratings, or rank in terms of downloads. A repository would provide more value to contributors if it provided clean summary reports of an individual’s impact in the community, including summary statistics over all their contributions, highlights of their “top five” (or top ten) contributed assignments, information about the number of individuals (or number of different institutions) represented among their downloader’s, or information about the number of variant assignments contributed by other authors based on one’s original root assignments. The ACM Digital Library has taken steps in this direction already by publishing bibliometric data on articles in their collection. By providing useful tools that allow contributors to make a stronger case that their assignments are received positively by the community and are having a broader impact, a repository can give contributors help in promotion and tenure documentation.

The nature of the revisions will vary and some will be nearly new works with a given source providing only a general idea of the project. Others will be nearly the same assignment, but with some significant modification that casts the problem in a new domain or extends it to include new objectives. A repository that is able to identify related assignments will provide a valuable resource to users, allowing them to see the variations on a starting point and, perhaps, inspiring still more adaptations.

7. WHERE TO FROM HERE? While the result of this working group is a proposal for a interchange format for assignments, this step is the beginning of a process rather than the end. As part of this activity, new ideas about how to provide a successful, sustainable repository for sharing assignments have begun to develop. One issue is whether to build on existing repository efforts, or work on a nextgeneration implementation that is more grounded in social networking, reviews, feedback, adaptation, and connection with non-repository services like automated grading tools and plagiarism detection tools. Another issue is how best to tie any assignment sharing or cataloging activities in with efforts under the umbrella of the National Science Digital Library, such as CITIDEL [5].

At the same time, if user ratings are tracked over time, it may be possible for a repository to provide individualized reports on a contributor’s evolving track record. By presenting a summary of how one’s ratings have changed over time, the repository can provide a measure of value of the contributor’s work that would not be available otherwise. If repositories provide a way to collect information on student feedback rather than just instructor feedback, innovative assignments could be rewarded. In fact, repositories can look at existing auction and social networking systems for other ideas. One possibility that arises from this arena is the concept of “badging”: giving community participants special icons or graphical “trophies” based on their achievements. Contributors who submit more than ten (or 25, or …) assignments may earn entry into a bronze, silver, or gold circle, and then receive an appropriately colored symbol by their name throughout the repository. Alternatively, separate awards could be given based on the number of “5-star-rated” assignments one has developed, or based on the number of independent reviews one has written of assignments developed by others. Badging is helpful both in increasing community participation, and in providing some documentation of a member’s degree of contributions.

At the same time, questions arise of how students might use such a repository, rather than just instructors. Some students may wish to find assignments for practice, while others may be pursuing self-study rather than a structured academic course. Also, collecting information about student perceptions of programming assignments would provide a useful tool for instructors. Hansen and Eddy described manual efforts to collect similar information for their own assignments [11], but the inclusion in a repository of such information could significantly boost its value. Overall, there are many exciting directions to explore, and the working group is looking forward to seeing new efforts in assignment sharing infrastructure development in the future.

6.6 Versioning Several factors will influence the number of versions of a programming assignment that will be available from a repository


- 178 -


'07. ACM, New York, NY, 271-275. DOI= http://doi.acm. org/10.1145/1227310.1227407

8. ACKNOWLEDGEMENTS This work is supported in part by the National Science Foundation under grants DUE-0618663 and DUE-0633594. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. The authors would also like to acknowledge the contributions of our on-line working group participants who were not able to attend the ITiCSE conference: Manuel Perez, Michael Caspersen, Ed Gehringer, Cay Horstmann, Kenrick Mock, and Greg Wilson.

[12] hCard Microformat. org/wiki/hcard

Available from: http://microformats.

[13] Joyce, D., Knox, D., Gerhardt-Powals, J., Koffman, E., Kreuzer, W., Laxer, C., Loose, K., Sutinen, E., and Whitehurst, R. Developing laboratories for the SIGCSE computing laboratory repository: guidelines, recommendations, and sample labs. In the Supplemental Proceedings of the Conference on Integrating Technology into Computer Science Education: Working Group Reports and Supplemental Proceedings (Uppsala, Sweden, June 01 05, 1997). ITiCSE-WGR '97. ACM, New York, NY, 1-12. DOI= http://doi.acm.org/10.1145/266057.266058

9. REFERENCES [1] ACM. Computing Classification System. Available from: http://www.acm.org/class/1998. [2] Carbone, A., Hurst, J., Mitchell, I., and Gunstone, D. Principles for designing programming exercises to minimise poor learning behaviours in students. In Proceedings of the Australasian Conference on Computing Education (Melbourne, Australia). A. E. Ellis, Ed. ACSE '00, vol. 8. ACM, New York, NY, 26-33. DOI= http://doi.acm.org/ 10.1145/359369.359374

[14] Knox, D., Wolz, U., Joyce, D., Koffman, E., Krone, J., Laribi, A., Myers, J., Proulx, V., and Reek, K. Use of laboratories in computer science education: guidelines for good practice: report of the working group on computing laboratories. In Proceedings of the 1st Conference on Integrating Technology into Computer Science Education (Barcelona, Spain, June 02 - 06, 1996). ITiCSE '96. ACM, New York, NY, 167-181. DOI= http://doi.acm.org/10.1145/ 237466.237644

[3] Cassel, L. N., Davies, G., Fone, W., Hacquebard, A., Impagliazzo, J., LeBlanc, R., Little, J. C., McGettrick, A., and Pedrona, M. The computing ontology: Application in education. SIGCSE Bulletin 39, 4 (Dec. 2007), 171-183. DOI= http://doi.acm.org/10.1145/1345375.1345439.

[15] Leach, P., Mealling, M., and Salz, R. RFC4122: A Universally Unique IDentifier (UUID) URN Namespace. Available from: http://tools.ietf.org/html/rfc4122

[4] Citation Formats. Available from: http://microformats.org/ wiki/citation-formats [5] Computing and Information Technology Interactive Digital Educational Library Repository: CITIDEL Home. Available from: http://www.citidel.org

[16] Mitchell, S. M. and Lutters, W. G. Assessing the value of computer science course material repositories. In Proc. 19th Conf. Software Engineering Education and Training Workshops. IEEE Computer Society, Washington, DC, 2006, pp. 2-5.

[6] Dublin Core Metadata Element Set. http://dublincore.org/documents/dces/

Available from:

[17] Open Source Initiative. Open Source Licenses. Available from: http://www.opensource.org/licenses

[7] Edwards, S.H. Improving student performance by evaluating how well students test their own programs. Journal of Educational Resources in Computing,. 3, 3 (Sep. 2003), 1. DOI= http://doi.acm.org/10.1145/1029994.1029995

[18] Parlante, N. Nifty Assignments. Available from: http://nifty. stanford.edu

[8] Edwards, S.H. The Web-CAT Wiki. http://web-cat.org/.

[19] Parlante, N. JavaBat: Java Practice Problems. Available from: http://www.javabat.com

Available from:

[20] Phillips, A. and Davis, M., eds. RFC4646: Tags for Identifying Languages. Available from: http://tools.ietf.org/ html/rfc4646

[9] Gallagher, L. and Carnahan, L. A general purpose registry/repository information model. Information Technology Laboratory National Institute of Standards and Technology. 2nd Draft. Available from: ftp://xsun.sdct. itl.nist.gov/regrep/GPregrepModel.pdf

[21] Rel-License Microformat. Available microformats.org/wiki/rel-license [22] SIGCSE: Education Links. sigcse.org/topics

[10] Grissom, S., Knox, D., Copperman, E., Dann, W., Goldweber, M., Hartman, J., Kuittinen, M., Mutchler, D., and Parlante, N. Developing a digital library of computer science teaching resources. In Working Group Reports of the 3rd Annual SIGCSE/SIGCUE ITiCSE Conference on integrating Technology into Computer Science Education (Dublin, Ireland). D. Joyce, Ed. ITiCSE-WGR '98. ACM, New York, NY, 1-13. DOI= http://doi.acm.org/10.1145/ 316572.358289

http://

Available from: http://www.

[23] Sun Microsystems. Class java.util.Properties. Available from: http://java.sun.com/javase/6/docs/api/java/util/ Properties. html#load(java.io.Reader) [24] Tungare, M., Yu, X., Cameron, W., Teng, G., PérezQuiñones, M. A., Cassel, L., Fan, W., and Fox, E. A. Towards a syllabus repository for computer science courses. In Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education (Covington, Kentucky, USA, March 07 - 11, 2007). SIGCSE '07. ACM, New York, NY, 55-59. DOI= http://doi.acm.org/10.1145/1227310.1227331

[11] Hansen, S. and Eddy, E. Engagement and frustration in programming projects. In Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education (Covington, Kentucky, USA, March 07 - 11, 2007). SIGCSE


from:

- 179 -


/assignment.html /images/screenshot.png /images/interface.png /software-tests/public/simple-ex-in.txt /software-tests/public/simple-ex-out.txt /software-tests/public/big-ex-in.txt /software-tests/public/big-ex-out.txt /software-tests/reference/input1.txt /software-tests/reference/expected-out1.txt /software-tests/reference/input2.txt /software-tests/reference/expected-out2.txt /solution/1/DiceExperiment.java /solution/2/DiceExperiment.java

APPENDIX A: Example ZIP File Contents Packaged assignments can contain a variety of assignment information. They can also be developed and evolved incrementally. To illustrate the variety that is possible, this appendix presents an evolving set of versions of one example assignment, showing what the zip file organization would be for each step. The full metadata file corresponding to this example is shown in Appendices B and C, so only the zip file structure will be presented here. To begin, consider the simplest case: packaging just an assignment writeup. In this situation, the zip file would contain only two entries: /assignment.properties /assignment.html

Finally, the instructor might eventually create a basic Eclipse project (or a project setup for another IDE) that students can use as a starting point for the assignment. This starting project might include a stubbed-out shell for the main program, as well as various support files: /assignment.properties /assignment.html /images/screenshot.png /images/interface.png /software-tests/public/simple-ex-in.txt /software-tests/public/simple-ex-out.txt /software-tests/public/big-ex-in.txt /software-tests/public/big-ex-out.txt /software-tests/reference/input1.txt /software-tests/reference/expected-out1.txt /software-tests/reference/input2.txt /software-tests/reference/expected-out2.txt /solution/1/DiceExperiment.java /solution/2/DiceExperiment.java /start/.classpath /start/.project /start/build.xml /start/simple-ex-in.txt /start/DiceExperiment.java /solution/2/DiceExperiment.java

After revising the assignment writeup, suppose the author wishes to add a couple of images to the writeup: /assignment.properties /assignment.html /images/screenshot.png /images/interface.png The image files can be placed in an appropriate subdirectory, and then included in the writeup using relative URLs. After the assignment has been used in class the first time, the instructor may have developed a sample solution to include--in this case, the solution consists of just one program source file: /assignment.properties /assignment.html /images/screenshot.png /images/interface.png /solution/DiceExperiment.java Perhaps a sample solution volunteered by one of the better students in the course would be included as well: /assignment.properties /assignment.html /images/screenshot.png /images/interface.png /solution/1/DiceExperiment.java /solution/2/DiceExperiment.java

Although this example is contrived, it does illustrate the basic structure of the assignment packaging zip file. It also shows how new contributors might ease their way into more full-featured assignments by starting out slow, and then incrementally adding resources as time permits. The hypothetical metadata for this example appears in Appendix B.

After working with the assignment in a course, the instructor may then decide to experiment with automated grading support. For this, automated tests might be developed to exercise student solutions. In this case, suppose that the instructor starts out with simple text input files that are to be piped into each student solution: /assignment.properties /assignment.html /images/screenshot.png /images/interface.png /software-tests/input1.txt /software-tests/input2.txt /solution/1/DiceExperiment.java /solution/2/DiceExperiment.java

APPENDIX B: An Example Metadata File in Properties Format This appendix provides a complete metadata file for the example assignment from Appendix A, in Java properties format. # ============================================= # Assignment Metadata File # ============================================= # Required Properties # ------------------------------------------------------------------title = Dice Rolling Experiment

Later, to help students out, the instructor may begin providing sample input files and corresponding expected output files to students so they can check their own work against expectations. The tests intended for student use would be separate from the more carefully crafted tests used for assessment: /assignment.properties


abstract = Build a class that simulates rolling two pair of dice, \ one containing two 6-sided dice, and the other containing a \ two-sided die and a 10-sided die. The program will read in \ the number of rolls to make, roll each pair the requested \ number of times, and then present the average sum of each \

- 180 -


A306987F-E867-4C6E-A3CB-39C0FAB71973 repository.acl = SomeGroupName

roll for both pairs. This allows you to carry out a lab \ experiment to experimentally test whether one of these \ pairs of dice tends to give a larger combined value, on average.

# Custom Properties # ------------------------------------------------------------------custom.property = some value

programming.language = java author1.name.first = Jane author1.name.last = Hacker author1.email = [email protected] author1.url = http\://... author1.affiliation = Villanova University, \ Dept. of Computing Science author1.phone = +1 234 547-8901 date.created = 2008-12-31 14:05:27 UTC date.modified = 2008-12-31 14:05:27 UTC languages = en copyright = (c) 2008 Villanova University license = Creative Commons Attribution-Noncommercial \ 3.0 United States License license.url = http\://creativecommons.org/licenses/by-nc/3.0/us/

APPENDIX C: An Example Metadata File in XML This appendix provides the same metadata file from Appendix B, but in XML format. The XML DTD for the metadata schema is available on-line at the URL listed in the XML file. Dice Rolling Experiment

# Recommended Properties # ------------------------------------------------------------------type = program keywords.topics = loops, accumulation, average, mean, \ random numbers keywords.prerequisites = main programs, variables, \ assignment statements keywords.other = experiment, lab learning.objectives = This assignment is designed to \ allow students to practice writing basic looping structures ... prerequisite.assignments = \ 410053AF-41FC-425E-BBB4-DF83AA3348C7 command.build = javac *.java command.test = for file in in*.txt; \ do \ java DiceExperiment < $file > $file.out; \ done

Build a class that simulates rolling two pair of dice, one containing two 6-sided dice, and the other containing a two-sided die and a 10-sided die. The program will read in the number of rolls to make, roll each pair the requested number of times, and then present the average sum of each roll for both pairs. This allows you to carry out a lab experiment to experimentally test whether one of these pairs of dice tends to give a larger combined value, on average. Jane Hacker [email protected] http://... Villanova University, Dept. of Computing Science +1 234 547-8901 en (c) 2008 Villanova University Creative Commons Attribution-Noncommercial 3.0 United States License

# Optional Properties # ------------------------------------------------------------------programming.language.vendor = Sun programming.language.version = 1.4+ course = CS1, CSC1051: Algorithms & Data Structures I info.url1 = \ http\://csc.villanova.edu/~csc1051/assignments/dice.html info.cite1 = http://doi.acm.org/10.1145/1384271.1384303 estimated.difficulty = 3 estimated.time = 1 week estimated.size = 120 # Assigned Properties # ------------------------------------------------------------------repository.uuid = \ 611B46BB-0DA6-40DB-8988-932AEB6A79A4 repository.version = 1.2 repository.parent.uuid = \ 2932F022-53E2-4288-8CA6-871FB93033BE repository.root.uuid = \


loops, accumulation, average, mean, random numbers main programs, variables, assignment statements experiment, lab

- 181 -


http://csc.villanova.edu/~csc1051/assignments/dice.html http://doi.acm.org/10.1145/1384271.1384303

CS1, CSC1051: Algorithms & Data Structures I program javac *.java for file in in*.txt do java DiceExperiment < $file > $file.out done

SomeGroupName

This assignment is designed to allow students to practice writing basic looping structures ...

410053AF-41FC-425E-BBB4-DF83AA3348C7


- 182 -


Developing a Common Format for Sharing ... - Semantic Scholar

Developing a Common Format for Sharing ... - Semantic Scholar

Suggest Documents

A Format for Phylogenetic Placements - Semantic Scholar

A Rule Format for Associativity - Semantic Scholar

Format for Manuscripts - Semantic Scholar

Developing a Collaborative Agenda for ... - Semantic Scholar

A PROCEDURE FOR DEVELOPING INTUITIVE ... - Semantic Scholar

Developing a Documentation System for ... - Semantic Scholar

A Bootstrapping Approach for Developing ... - Semantic Scholar

A METHODOLOGY FOR DEVELOPING HIGH ... - Semantic Scholar

DEVELOPING A METHODOLOGY FOR DESIGN ... - Semantic Scholar

Developing a Manufacturing Science for ... - Semantic Scholar

Sharing good NEWS across the world: developing ... - Semantic Scholar

Format 1 - Semantic Scholar

ITT Format - Semantic Scholar

A Common Semantic Basis for BDI Languages - Semantic Scholar

Password Sharing: Implications for Security ... - Semantic Scholar

Resource Sharing Architecture For Cooperative ... - Semantic Scholar

Observable Sharing for Functional Circuit ... - Semantic Scholar

Common Criteria - Semantic Scholar

Common Subsequences - Semantic Scholar

Format Guide for ITA Authors - Semantic Scholar

Electronic format submission for AP2000 - Semantic Scholar

Electronic format submission for AP2000 - Semantic Scholar

Format for Springer Verlag books - Semantic Scholar

Electronic format submission for Ursi2004 - Semantic Scholar