Proc. 1st Int'l Workshop on Pen-based Learning Technology (PLT), pp.113-118. May 2007.
Paper Architecture and an Exam Scoring Application Masaki Nakagawa, Narcis Lozano and Hideto Oda Department of Computer and Information Sciences Tokyo University of Agriculture and Technology 2-24-16 Naka-cho, Koganei, Tokyo 184-8588, Japan
[email protected],
[email protected]
Abstract This paper presents paper architecture and an exam scoring application which runs on the architecture. The paper architecture is unified software architecture that allows the use of several types of electronic pen and paper devices without having to modify the main application. Electronic ink written by students in exams is captured, transformed into a standard format and then sent to the paper architecture server. There the ink is segmented and processed. We have taken a hybrid approach for the scoring application. Selection type questions are automatically recognized and scored. For free text questions we have developed an interface that allows teachers to mark them interactively. It provides question reorganization, empty questions marking, and other techniques to reduce the scoring time. A preliminary evaluation comparing the exams scoring by using the application and by hand showed considerable reduction of the scoring time.
1. Introduction Research in pen interfaces started from those for tablets, and then shifted to those for display integrated tablets like PDAs or tablet PCs. In recent years, however, electronic pen devices on real paper are gathering attention. They are much cheaper and have the merit of real paper. Electronic ink (a time sequence of pen-tip coordinates, digital ink or ink in short when clear) captured is stored in memory and processed when the pen device is connected to the computer. These devices usually consist of a pen and a positioning device that can track the pen-tip position and store the coordinates in memory [1, 2] or send them electronically. There are also other types of pen devices that calculate pen-tip position by using special
paper printed with dots [3]. When people write, real ink is left on paper and at the same time electronic ink is stored. This fact makes these devices especially suitable for some applications. On the other hand, their interactivity is lower than the display integrated tablets. Often the electronic ink is not processed until the device is connected to a PC except that it is sent to the PC on real time. Even if it is sent on the fly, the ink and any computation can not be fed back to the real paper unlike display integrated tablets. This is especially noticeable for the correction of recognition errors. With display integrated tablets, on the other hand, errors can be detected as the user writes and corrected interactively. Nevertheless, these devices (hereafter, we call them as pen and paper devices) succeed the merit of paper and add the merit of computing. There are several promising applications such as diagnosis taking by medical doctors, invoice preparation in front of customers by sales persons and so on. Paper examination seems one of suitable applications. Students can concentrate on questions and teachers can mark their answers as they have been doing on real paper exams with the computer support. We have prepared the method to employ various types of pen and paper devices and abstract them so that we can develop applications on the unified and abstracted “Paper Architecture” using the concept of lazy recognition [4]. Processing and recognition can be delayed after all handwritten data have been inputted. Then, we made an exam scoring application by which teachers mark exams using a PC with a mouse or a pen, which is more efficient than working on real exam papers. Test results are sent directly from the scoring application to the students by e-mail as well as paper exams are returned to the students. Although there are test scoring systems that use scanned images such as [5], these are often limited to selection type questions. Because of the inherent
difficulty of segmenting foreground and background in images, they usually require the answers to be fulfilled in a separated sheet of paper, which can be confusing to the students. On the other hand, pen devices can automatically segment the track of the pen-tip from the background, require far less storage capacity than images and can retrieve additional information such as writing speed, stroke order or pressure. The structure of this paper is as follows. Section 2 presents Paper Architecture, unified software architecture for pen and paper devices. Section 3 describes an editor to prepare a template which defines how to process electronic ink. Section 4 presents an exam scoring application developed using Paper Architecture. Section 5 shows the results of a preliminary evaluation experiment of the exam scoring application and Section 6 draws the conclusion.
2. Paper architecture We prepared Paper Architecture in order to run the same application with different pen and paper devices, without having to reprogram the main application. With this we aim to overcome the lack of a standard for pen and paper devices. This lack of a standard entailed that developers had to reprogram their applications for each kind of device they wanted to support. Paper Architecture abstracts the communication between a pen and paper device and a computer using the concept of lazy recognition [4].
2.1. Ink capture The process of inputting electronic ink into the system is shown in Figure 1. A client gets the ink from the device in the proprietary format. Then it segments the ink into pages, when this is applicable to the device. It then proceeds to map the pen-tip coordinates to a standard coordinate system. Finally it stores the ink in the InkML format [6]. A client must be created for each pen and paper device we want the system to support. Because the client requires only the most basic functions of the pen
and paper device, it can be easily developed. The file generated by the client is then sent to the server, through which it becomes the input to the main application.
2.2. Ink processing and output The processing of the electronic ink contained in the input files takes place in the server. The flow is described in Figure 2. Because the input files that the server receives contain only ink, streams of pen-tip coordinates, we need a definition of the structure of the document (position and type of the data, etc.) in order to process the ink. For this, we use template files. These files are stored in the server in order to improve the security and the maintenance of the applications. To decide which template must be used, a document identifier, an arbitrary length of digits, is used as the name for the template. The document identifier can be embedded or uniquely determined in the Anoto pen and paper [3]. For general, however, we reserve a special area in the document for the user to specify an identifier without recognition error, which is described later. When the ink arrives to the server this area is segmented, the content identified and the identifier used to retrieve the corresponding template. Templates are XML files containing a definition of the structure of the document and associated data. In order to improve reusability and simplify the design, the structure is defined using processing elements (PE in short) with associated functionality: questions, text input fields, drawing input fields, etc. The template is created interactively using a GUI tool to place necessary PEs. Using the template, the ink is first segmented, associated with the suitable PEs and then processed according to the parameters of the PEs. The results generated by the different PEs are then grouped and exported as an XML output file. This output file can be used to update a database in the server or sent to the user to be employed as an input to another application, such a visualization tool, a web browser, etc.
Figure 1. Data input flow. Figure 2. Ink processing flow.
2.3. Processing elements Figure 3 shows an example of documents. A document is divided into areas with certain parameters associated with each of them to define how the ink must be processed. We call these areas processing elements (PEs). A PE is implemented over an abstract class that receives ink and returns the result as an XML document. For this version of Paper Architecture we have developed a set of PEs that covers most operations that are needed for ink processing. If additional PEs are needed, they can be easily added. This kind of design is used for increasing the reusability of the component of the system. The current set of PEs is composed of: (1) Labeler: this is to store raw electronic ink, without performing any recognition process. This PE has only one parameter, an optional text label. The output of this PE in the result file is the segmented ink contained in the bounding box of the PE stored in InkML format and the text label. (2) Question labeler: it stores ink and an additional parameter for a text string of the expected correct answer. This PE is designed to be used as questions in exams. The answer can be compared automatically with the input using some recognition or matching technique or it can be shown in a marking interface so the teacher can score the question manually as it is done in the application in section 3. (3) Field: it stores ink to be processed by a recognition engine with several parameters: • Name: a text to identify the field in the output result file. • Rows: a number of rows that compose input of long text strings. • Cells: a number of cells that compose each row of the field. • Types: a type or types of the data we expect the user to input (digit, English, Japanese, etc.). This parameter determines which recognizer will be used by Paper Architecture to recognize ink. (4) Question field: it is a field and an additional parameter for a text string of the expected answer. It is processed the same as with the field, but when the result text string from the recognition engine arrives, it is compared with the correct answer. If the answer matches the user input, the question is marked as Correct in the result file; Incorrect otherwise. (5) Checkbox: this is to input a Boolean value. If there is ink within the boundary of the bounding box for this PE, the input is recognized as TRUE; otherwise it is recognized as FALSE. This is then
Virtual keyboard for Document ID
Virtual keyboard for User ID
Question field
Question checkbox
Labeler
Figure 3. An example of documents. written in the result file, along with an identifier for this PE. (6) Question checkbox: it is a checkbox with an additional parameter for the expected answer (TRUE or FALSE). If the answer matches the user input, the question is marked as Correct in the result file; Incorrect otherwise. (7) Virtual keyboard: this is used to input a sequence of characters or digits, without needing to use character recognition. Therefore, it is the most robust input method. A virtual keyboard is composed of a series of cells, arranged in one or more rows as shown in Figure 4. Each cell has a character (letter or digit) associated, that we call key. For input a value using a virtual keyboard, users write traces in the cells in order, much like they will do pressing keys in a keyboard. The virtual keyboard takes the advantage that pen 0
1
2
3
4
5
6
7
8
9
Result: 04724
Figure 4. A Virtual keyboard.
devices not only store the x and y coordinates of the pen-tip points, but also temporal data, or at least the order in which the points were written. Once the traces inside the cells for the PE are segmented, the traces are analyzed by order and the sequence of keys of the cells that have been ticked is written in the result output file, along with an identifier for the PE. Although robust to errors than other fields, this PE is not well suited for the input of a long sequence of characters, because the order of the traces can not be confirmed by the user just looking at the ink marks in the paper. We employ virtual keyboards for inputting document identifiers, student IDs, etc, because errors in the recognition of these identifiers would usually cause the whole processing of the document to fail, and therefore we needed the most reliable PE available.
3. Template preparation editor In the previous section we presented the processing elements (PEs) used in Paper Architecture. These PEs are contained in template files, XML files which are used for processing the ink. In this section we will describe the process for preparing a template file. Because template files are based on the XML, we can create them simply using a text editor. However, it is difficult to set the boundaries of the PEs only with a text editor. Therefore we developed an interactive template editor that uses the printed document as guide, and that allows PEs to be placed easily and precisely. In order to create a template file, the first step is to get a printed copy of the document we want to use. Then we proceed to use a pen and paper device to put marks on the edges of fields, check boxes, and other PEs of the document as shown in Figure 5. We can also fulfill the correct answers of the question PEs (question fields and question checkboxes). Then, we proceed to load the ink of the marks and correct answers into the template editor. The interface of the editor is shown in Figure 6. The controls on the lower part of the screen (1) are used to navigate between the different pages. In the upper right side of the screen (2), there are buttons for selecting which PE we want to place on the template. After pressing one of the buttons we proceed to draw a PE (3) by defining the limits of the PE, following the marks that we put by the pen and paper device in the previous step. In the case of question PEs (question fields and question checkboxes), the editor analyzes the contents in the bounding box and, if there is ink, it recognizes the answer automatically, using a built-in recognition engine. Thanks to this feature, the templates for exam applications, with the complete
Marking of the edges of the processing elements in the printed document.
Figure 5. Marking of the elements. information about the correct answers can be created in a few steps. Using the textboxes (4) on the right size of the screen we can modify the parameters that define the functionality of the PE. The number and titles of these textboxes change according to the type of the PE. PEs usually occur in groups. In order to take advantage of this fact, we added the option of adding several PEs in array in just one step. The user fills in the textboxes in the lower-right part of the screen (5) with the number of PEs per row and the number of rows, and then proceeds to set the limits of the area that will occupy all the PEs. The editor then divides this area and places the PEs automatically. After we have finished editing the template we save it in the template folder in the server.
(2)
(3)
(4)
(5)
(1)
Figure 6. Template editor interface.
4. Exam scoring application In order to exploit the merit of pen and paper devices and to test the capabilities of Paper Architecture, a human assisted exam scoring application was developed. This application is aimed to assist teachers to mark and score exams, and to analyze and store the student results. Most research in exam scoring interfaces has focused in fully automatic scoring. The user input is segmented, converted to text, and then compared to the answer to decide if it is right or not. However, this approach has several problems. Cursive handwritten character recognition is an error prone process, and errors in segmentation and recognition may lead to false evaluation of answers as incorrect. On the other hand, questions typically can have two or more than two possible correct answers. In automatic scoring systems, the teacher must think beforehand all possible answers, which is often impossible. Also, in the case of free text answers, the system should be able to fully understand it, in order to evaluate it. Despite there has been research in automatic evaluation of this type of questions [7, 8], these methods are really meant for evaluation result estimation, not for unassisted automatic marking. Because of these problems, most exam scoring systems deal only with selection type questions. However, with this kind of questions, exams become a selection process. Users only have to be able to decide if an answer is right or wrong, but may not learn the contents thoroughly. For example, in the case of language tests, the user may learn to discern the meaning of vocabulary, but not to spell it correctly. On the other hand, it becomes difficult to test the reasoning and expression abilities of the students, being this type of question appropriate mostly just for memorization based contents. In our scoring system we have taken a hybrid approach. Automatic scoring can be used for selection type questions and for questions that the teacher considers that all possible answers can be limited (dates, country capitals, for example). For questions that can have multiple answers, or that require to write free text answers, a human assisted scoring interface is used. Instead of recognizing ink, it is shown to the teacher so that the teacher can evaluate the answers, using the interface designed especially for this purpose. The process is as follows: (1) Exam preparation The teachers first prepare the exams using a text processor and print it. Using this printed document as a
guide, they create the template file adding question PEs with the GUI tool. For answers that are scored automatically the correct answer is assigned to the question as a text string. For questions to be evaluated using the human assisted scoring interface the correct answer may be given as text or as electronic ink. Finally, they save the template or templates in the server. The identifier is automatically assigned to the template. Then, they prepare copies of the exam paper, which are distributed to the students. (2) Answering by students The students answer the exam using the pen and paper device. They fill in the exam identifier given to them by the teacher and their own student identifiers. We employ virtual keyboards for them to input these identifiers. The students then answer the questions. When they finish, their answer files are stored into the server and the paper exams written by the students are also gathered.
Figure 7. Marking interface. (3) Processing by the server The server identifies the exam using the identifier and retrieves the corresponding template. Following the document structure of the template, the server segments ink of answers to each question. For automatic scoring questions, ink is recognized and compared with the correct answers and evaluated. The rest of the questions are kept in the InkML format. (4) Marking by teachers The teachers load the exam template file in the tool. They then load the answer files of the students processed by the server. For answers that have to be scored manually, they open the interface shown in Figure 7. On the left of the screen a tree view of the student exams allows the teachers to access
hierarchically the answers ordered by the question number or student ID. Once the question has been selected, the answers are shown in the screen. In the bottom part of the screen, the correct answer is shown either as a text string or electronic ink, depending of the way it was inputted in the template design step. On the right of each answer there is a small interface for evaluating the answer into several ranks. There is also a textbox in case the teachers want to give the answer a more precise scoring. The teachers can also write over the answer using the mouse or a pen device. The interface to mark the exams allows the teachers to reorganize the answers unbound from the paper. When scoring real paper exams, teachers must correct one after another. They have to look which question is, think the correct answer, look the student answer and evaluate it. Our tool allows not only this way of scoring, but also another. Instead of scoring all the questions for one student, it groups all the students’ answers by question. The teachers first mark all the students’ answers for question 1, then those for question 2, and so on. By organizing questions in this fashion, they do not have to be constantly bothered thinking which question they are scoring or what the correct answer is, and they can focus their attention to the student answers instead. The teachers can automatically evaluate the questions left blank by the students as incorrect. They can also hide the answers already scored so that they can focus their attention to unmarked questions. Once all the answers have been marked, the results can be exported. These are presented in a spreadsheet with overall statistics of the exam as well as of each student. Test results are sent directly from the tool to the students by e-mail as well as paper exams are returned to them. The data can be also incorporated into LMS which can provide much richer services.
Table 1. Experimental result. Participant
Manual marking time (min.)
Comp. assisted marking time (min.)
Time reduction ratio (%)
24 11 22 24 24 23 20
13 9 12 12 14 11 12
45.83 18.18 45.45 50.00 41.67 52.17 40.00
No. 1 No. 2 No. 3 No. 4 No. 5 No. 6 No. 7
6. Conclusions This paper presented Paper Architecture to abstract device dependency of pen and paper devices and an exam scoring application employing them. Electronic paper exams allow students to concentrate on questions and teachers to mark questions more effectively. The scoring application employs automatic scoring for selection type questions and human assisted scoring for free text questions. Question reorganization, one-click question evaluation, statistics export and automatic e-mailing of test results to the students were also implemented. There are several remaining issues. We have not exploited the dynamic information from the students’ answers though the speed or fluency of writing is available from electronic ink. It might signal something about the student confidence. Another issue is in the evaluation. We must evaluate marking error rates, mental burdens and so on.
7. Acknowledgement This work is supported by Grant-in-Aid for Scientific Research under the contract number (B)17300031 and the MEXT Research and Education Fund for Promoting Research on Symbiotic Information Technology.
5. Preliminary evaluation experiment
References
We prepared 15 exam answer sheets composed of free text inputs (with different distribution of correct, partially correct, incorrect and empty answers) using the pen and paper device [1]. We compared the time for 7 participants to score 15 exam sheets manually and that to score them using our tool. We briefly instructed the participants in the use of the tool. After this we employed the tool options to auto-score the empty questions and hide the already marked questions. The participants then scored the questions through the interface using the mouse. The results of the experiment are shown in Table 1. It shows large reduction of the marking time.
[1] [2] [3] [4] [5] [6] [7] [8]
Acecad at http://www.acecad.com.tw Pegasus Notetaker at http://www.pc-notetaker.com Anoto at http://www.anoto.com Nakagawa M. et al.: Lazy recognition as a principle of pen interfaces, INTERCHI’93 Adjunct Proc. 89-90. 1993. Dejan, G. et al.: A simple system for automatic exam scoring using optical mark reader, Proc. 2nd Balkan IFAC Int’l Symp., Ohrid, R. of Macedonia, 2000. InkML at http://www.w3.org/2002/mmi/ink Burstein, J. et al.: Automated scoring using a hybrid feature identification technique, Proc. 17th Int’l Conf. on Comp. Linguistics, 1, 206-210. Montreal, Canada, 1998. Wild, F. et al..: Parameters Driving Effectiveness of Automated Essay Scoring with LSA, Proc. 9th Int’l Computer Assisted Assessment Conf., 485-494, Loughborough, UK, 2005.
