submissions by standard HTML form elements. ... By using the Back button of the browser, the form can be modified and resubmitted. Students are able to repeat ...
Interdisciplinary Conference on Electrical, Electronics and Computer Engineering in the Third Millenium (ICEECEE) 2000
TOPKAPI: A Tool for Performing Knowledge Tests Over the WWW Guido Rößling and Bernd Freisleben Department of Electrical Engineering and Computer Science University of Siegen Hoelderlinstr. 3, D-57068 Siegen, Germany {roessling, freisleb}@informatik.uni-siegen.de In this paper, we present the TOPKAPI tool for generating, administering and evaluating flexible multiple-choice questions over the WWW that completely frees the teacher from the evaluation process. Each answer may be commented by the author to provide students with additional feedback. TOPKAPI also supports easy reuse of components.
1
Introduction
One of the issues teachers must address is the level of knowledge gained by the students. This is important both for pinpointing shortcomings in the course materials and general understanding problems common to a larger group of students. However, many students are reluctant to ask questions in large courses. Thus, assessing the level of understanding is very difficult. One popular way is using multiple-choice questions. The larger the course, the more benefits can be gained from using multiple-choice questions, as this is often the only way to force students to actively participate in the course. As administration of multiple-choice questions is time-consuming, automated correction is highly desirable. Special scanning software can be used to limit analysis to the answer portions. However, this approach still requires human supervision and offers no feedback to participants. The WWW, on the other hand, allows for easy automation of submissions by standard HTML form elements. Computer-based tests allow automating the submission process and can also support automatic evaluation. However, teachers still need access to special software for putting their test on the WWW. The effort needed to generate new multiple-choice tests should not be underestimated, and often includes repeating the same steps for each test. Thus, teachers would profit from having a tool that allows easy configuration and reuse of standard components. In contrast to regular exams performed over the WWW, our main focus is offering students a possibility for selfevaluation. The tests are expected to pinpoint individual deficiencies, increase student's understanding of the material, and enhance their motivation to repeat the material already covered. It should be stressed that incorrect answers can be instructive. In fact, if commented properly, they may even be of greater help in understanding than picking the right answer at once. However, if the feedback consists only of answer classifications, it offers no insights to the student, nor does it give hints what the correct solution might be. In this paper, we present a tool that supports the automatic generation of Web-based knowledge tests. It supports giving feedback detailing why an answer was incorrect and giving additional information. The tool can be used by teachers to design multiple-choice questions, administer them, and extract statistics about the students' performance. The tool uses component reuse and offers internationalization aspects. The paper is organized as follows. In section 2, we present some desirable features of such a tool from both the learner and the teacher perspective. Section 3 contains the underlying concept of knowledge tests supported by our tool. Section 4 gives an evaluation of our tool. Section 5 concludes the paper and outlines areas of future work.
2
Desirable Features of Multiple-Choice Tools
In the following, a given set of questions, answer choices, ratings and optional comments will be called a catalogue. Catalogues set in a specific context are called questionnaires. 2.1
Features for Students
We consider the following features necessary for students for pedagogical, motivational or privacy reasons. •
Provide motivating feedback by explaining why answers are wrong, give hints or listing related topics
•
Allow repeated submissions for learning from mistakes; for self-evaluation tests only
Interdisciplinary Conference on Electrical, Electronics and Computer Engineering in the Third Millenium (ICEECEE) 2000
•
Display previous results
•
Show results by points or percentage achieved
•
Support authentication
•
Place only modest soft- and hardware demands
•
Generate automatic index of questionnaires for easy access without memorizing URLs or adapting WWW pages
2.2
Features for Teachers
Coming up with questions and a set of plausible answers is rather time-consuming. While this process cannot be automated yet, the tool should try to free authors from as much administrative work as possible. Therefore, we hold the following features to be desirable for teachers. •
Easy generation of questionnaires
•
Limited need for administrative tasks
•
Limited redundancy, especially regarding author or course information and layout
•
Platform independence
•
Automatic answer logging for automating submission evaluations
•
Support for automatic evaluation for a single questionnaire or all questionnaires for a given student or all students
•
Automatic indexing of questionnaires, preventing authors from building it themselves in HTML
3
Features of TOPKAPI
TOPKAPI is an acronym for “Tools for organizing and performing knowledge assessments with possible interactivity”.
3.1
Features for Students
To reach the widest audience possible, we chose HTML forms for the input. Thus, the users need only have a connection to the WWW using any browser including text-based browsers like lynx. TOPKAPI provides a separate tool for indexing questionnaires according to their classification. After selecting a questionnaire, students are presented with a list of questions and possible answers, concluded with a pair of “submit” and “reset” buttons. Upon submission of the filled-in form, the student is presented with a precise evaluation containing the percentage of points reached and a commented listing of the chosen answers with ranking and any instructional comments the author has provided. By using the Back button of the browser, the form can be modified and resubmitted. Students are able to repeat this process until they are satisfied with their result. For grading purposes, the number of possible submissions can also be limited. Each questionnaire starts with a greeting message containing the best score achieved by the student. Students are also provided with a separate tool listing the scores reached for all tackled questionnaires. Students can administer their own passwords and change them online. To prevent password abuse, we use the UNIXbased crypt algorithm for password encryption. Note that this means that even the system administrator has no access to student passwords.
Interdisciplinary Conference on Electrical, Electronics and Computer Engineering in the Third Millenium (ICEECEE) 2000
3.2
Features for Teachers
The following additional features of are restricted to the author of a given questionnaire which is checked by login and password. All features can be used and administered directly over the World-Wide Web. TOPKAPI provides a built-in debug mode presenting all possible answers with their classification and reaction information. The author can test questionnaires before they are published. The tool also provides an evaluation mode for keeping track of how often each answer was chosen. Authors can thus determine problem areas in course materials. A separate tool generates a table of all registered students, listing the questionnaires taken and the best result achieved, as well as the average percentage reached. Finally, on registration of a given questionnaire in a context, index pages containing all questionnaires for this context are generated on demand. The tool responsible for indexing provides a simple entry page listing all categories, for example “algorithms; data structures” with the number of questionnaires listed. A page for each element of the classification hierarchy is provided, for example listing all sorting algorithms.
3.3
Catalogue Components and Settings
TOPKAPI uses a number of inputs for building a single form as shown in figure 1. Each questionnaire contains information about the author and the context. As these are likely to be reused, they are represented by objects.
Figure 1: Components and interaction of TOPKAPI Evaluator is a predefined object for evaluating the user answers that can be specialized for other rankings. Date and Title are simple texts used in the form header and footer. The Classifiers object contains the supported classification types. The default contains entries for Correct, Wrong and Partially Correct. Catalogue contains the question catalogue including all answers with classification and reactions. Reusing a given questionnaire in a different context only requires changing the context and/or author attribute, providing a new access key and adapting the log file name. All these attributes are modeled as objects, and can thus easily be exchanged. We also provide language support for English and German. Supporting other languages requires translating several short phrases in a text file without touching the source code. Questionnaire registration consists only of generating a simple description file and adding a single line to the registration file.
3.4
Some Implementation Details
Each questionnaire consists of a list of question entries. Each entry contains a question text, question type and a set of answers, combined with reactions and classifications. The entries are collected in a simple text file. After generating a new questionnaire, the author should enter it into the registry containing the keys shown in figure 1. A new access key and a target file for storing the questionnaire submission data must also be provided. Immediately afterwards, the new questionnaire is accessible and ready for testing. After testing, the questionnaire is made available to the students by entering its access key in the context-specific registration file also used for generating contextspecific index pages.
Interdisciplinary Conference on Electrical, Electronics and Computer Engineering in the Third Millenium (ICEECEE) 2000
On choosing a catalogue, the relevant information is extracted from the registration and a HTML form is generated, including the set of questions and their possible answers, as well as „submit/reset'' buttons. TOPKAPI supports Radio buttons, checkboxes and selections question types. Due to the difficulty of evaluating text input, this type is currently not supported. On submission of the form, the answers are evaluated and the result page is shown. All entries are logged in the general access log and a special log file for the current questionnaire.
4
Evaluation
TOPKAPI has been used over a year now for our introductory courses in computer science at the University of Siegen and the Berufsakademie Mannheim, Germany. Recently, TOPKAPI was also adopted at the Darmstadt University of Technology. Although using TOPKAPI was not compulsory, more than 7500 questionnaires were submitted. We have nearly achieved our goal of freeing the author from administration. Authors now need only enter the registration information for new questionnaire, which is typically achieved in a few seconds. Compared to other approaches [WebT:98, Peel:94, Wark:96], TOPKAPI provides greater flexibility for both teachers and students. This is most clearly visible by the flexible response teachers can embed, which is absent in most other programs. Furthermore, the dynamic generation of index pages has proved a great boon to the usability, as we currently provide 128 questionnaires used in six separate introductory courses to computer science. Access to TOPKAPI requires a login that can be generated online within seconds. To alleviate possible privacy concerns, we stress that the login can be chosen freely and that we do not log information like the Web server used by the student. Thus, we have no way of associating a given login with a person. Our students responded to this by stating that they had “no” (33%) or “little” (45%) private concerns. It is also possible to build profiles for evaluating learning strategies. As the questionnaires are mainly used for selfevaluation, we are currently not interested in actually grading individual students. For evaluating learning strategies, student IDs could be mapped to a sequence of numbers so that any student information the login might contain is lost before evaluation. We will try to analyze the students' strategies in ”solving'” the questionnaires from this evaluation. In our evaluation at the end of the term, 61% rated the offer of the online knowledge tests as (very) helpful for understanding the topics presented, with 71% asking for more (or far more) inclusion of questionnaires in the course. We are planning to provide a complete evaluation in the near future. Due to the limited space, we cannot provide an example of a TOPKAPI session. However, TOPKAPI is freely accessible on the World-Wide Web [Topk:99].
5
Conclusions
In this paper, we presented TOPKAPI, a tool for generating, administering and evaluating multiple-choice tests over the WWW. Currently, our system is in use at the University of Siegen, Berufsakademie Mannheim and the Darmstadt University of Technology, Germany. Most questionnaires are written completely in German. However, a translation of several questionnaires to English is in progress. Due to the easy integration of internationalization aspects, authors need only translate the catalogues themselves. Most components can be dynamically exchanged with little redundancy. TOPKAPI currently offers 128 questionnaires, though several of them actually only differ in the course-specific settings. We plan to extend our system for full support of HTML select elements for „fill in the spaces'' questions where text flows around the selection entries. This would make the tool highly usable for the types of tasks often set in language examinations like TOEFL (Test of English as a Foreign Language). We also strive to complete the LaTeX front-end. TOPKAPI can also be easily adapted to online exams or other usage areas like market surveying.
6
References
[WebT:98] Herr, G. WebTutor, 1998. Available at http://sponsor.iti.informatik.tu-darmstadt.de/Demo/WebTutor/. [Peel:94] Peel, A. Computer Aided Assessment through Hypermedia. Active Learning 1, 1 (Dec. 1994), 35--37. [Topk:99] Rößling, G. TOPKAPI Home Page, 1999. Available at http://www.informatik.uni-siegen.de/cgibin/roesslin/TIP.pm?context=bacs1 [Wark:96] Warkentyne, H. M.~K., Smith, I., and Forte, E. Implementation and Evaluation of a WWW Multiple Choice Question Server. Lecture Notes in Computer Science 1108 (1996), 228--243.
