Automatic Checklist Generation for the Assessment of ... - CiteSeerX

Automatic Checklist Generation for the Assessment of UML Models Tom Gelhausen, Mathias Landh¨außer, and Sven J. K¨orner Institute for Program Structures and Data Organization University of Karlsruhe 76131 Karlsruhe, Germany {gelhausen, lama, koerner}@ipd.uni-karlsruhe.de

Abstract. Assessing numerous models from students in written exams or homework is an exhausting task. We present an approach for a fair and transparent assessment of the completeness of models according to a natural language domain description. The assessment is based on checklists generated by the tool Sumoχ. Sumoχ directly works on an annotated version of the original exam text, so no ‘gold standard’ is needed.

1

Introduction

If we teach modeling we also need to grade models. But the task of modeling comprises a certain degree of freedom, and this renders exam questions on that task somewhat awkward: Assessments are frequently rejected by the examinees. This is either due to mistakes of the examiners or due to unreasonableness of the examinees. No doubt, the examiners make mistakes as the task of assessing numerous models is exhausting, requires fast perception, and great mental flexibility. But as frequently, students tend to challenge assessments on a pursuit for a better grade. Experience shows, that especially modeling tasks lend themselves to discussing every single detail of an assessment in terms of correctness and adequacy. Instant insight on the students’ side seems aspirational to solve both problems: a) discover true errors made by the examiners and b) avoid exhausting discussions during homework reviews and post-exam reviews. We identified the need for an evaluation procedure that meets the following requirements: The assessments should be systematic, fair, transparent, and cumulative in a sense that they consist of simple atomic decisions which themselves are (ideally) unquestionable. An assessment based on a sample solution is problematic, as one tends to use this sample as a yardstick for all of the students’ solutions. We discourage the use of such a ‘gold standard’, since it stultifies the examiner’s perceptivity. Our approach uses comprehensive and selfexplanatory checklists to assess UML models1 . Based on a technique which we initially designed to automatically generate domain models [1], we developed 1

Please note that the approach is not limited to class diagrams. We currently support class and sequence diagrams, state charts and OCL.

2

Sumoχ (SalE -based UML MOdel eXamination). It works on a semantic annotation of the original exam question’s text and is therefore able to generate considerably less biased checklists than one would create by hand. The Sumoχ-checklists are designed to measure the completeness of the content of a model according to a given domain description. Established metrics on models measure quality aspects such as similarity and class relationship complexity (cf. Section 5). In this respect, general checklists on model quality and the domain-dependent Sumoχ-checklists complement each other. We start with the explanation of the approach and how it can be used to generate checklists. We then evaluate the complete process using the checklists to assess students’ homework and exams. The lessons we learned during the evaluation are discussed afterwards. The following section focuses on the related work and also serves as foundation for our idea since it shows the necessity of such a method. The article closes with a conclusion.

2

The Approach

Assume you want to give your students a modeling task as it is depicted in Figure 1. You provide a text describing the application domain and you expect your students to model it concisely regarding this text.

Exercise 17: Model the following domain in UML. You may use OCL to specify constraints. The game of chess is played between two opponents. They alternately move their pieces on a square board called a chessboard. The player with the white pieces commences the game. A player has the move, after his opponent made his move…

Fig. 1. Example modeling task.

We propose to use a systematic checklist to grade the results from the students. Table 1 shows an excerpt of such a checklist for the text above. This checklist was generated by Sumoχ and has already been filled out by an imaginary examiner (in script style). It contains one section per sentence of the original domain description. For example, row 1 introduces the first section of this checklist. Each section lists all elements that could be modeled. For the first section, opponents and the Multiplicity 2 are exemplified (rows 2-6 and 7-10). The opponents in turn give an example for a linguistic structure, a constituent in this case, that can be modeled in various ways: They could be modeled as a class (row 2) or

3

as a role (-name) within a UML association (row 3). Sumoχ also proposes that opponents could be modeled as an instance of another class (row 4) – possibly meaningless in this case but obviously meaningful in the case of the element player below (row 30). However, our imaginary examiner did not find a class, a role, or an instance that covered the concept of opponents. Instead, he found a note from the student that the classes Player and Opponent were unified (row 6).

1 2 3 4 5 6 7 8 9 10 ... 27 28 29 30 31 32 33 34 35 36 37 38 ...

Element of Phrase

Modeled...

Phrase: The game of chess is played between two opponents. “opponents” as class as role as instance Differently (please specify): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Explicitly not modeled (please specify): Opponent- and Player-classes are equivalent . . . . Multiplicity “2” of “opponents” as multiplicity in an OCL constraint Differently (please specify): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Explicitly not modeled (please specify): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :::::::::::: etc. :::::::::::: Phrase: The player with the white pieces commences the game. “player” as class as role as instance Differently (please specify): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Explicitly not modeled (please specify): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Attribute “white” of “pieces” as Boolean function (with a parameter) as scalar function as state of “pieces” as attribute of “pieces” Differently (please specify): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Explicitly not modeled (please specify): . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . :::::::::::: etc. ::::::::::::

yes no wrong

Row

Table 1. Example of an Assessment Based on a Sumoχ-Questionnaire.

7 7 7 7 7 7

7 7 7

7 7 7 7

Obviously, our approach maps linguistic structures2 to model elements. This approach originated in Abbott’s 1983 article “Program Design by Informal English Descriptions” [2]. Our method incorporates various suggestions that have been published in the domain of (automatic) model extraction since then (see [3] for a survey of approaches). We collected all UML fragments we found that were generated from single (simple or complex) linguistic structures; Table 2 enumerates some of the resulting mappings. Up to now, we identified a total of 35 rules3 to turn linguistic structures into questions about model elements. Applying these 2

3

On a syntactic level such structures are expressed via noun phrases or adverbials, for instance. But actually, we do not map syntactic patterns on the text source, but rather semantic patterns on the internal discourse model (cf. Table 2). The exact number significantly depends on the intelligence of the rule processor. The number 35 would apply to a ‘human processor’. A computer program is less fault-tolerant according to the applicability of the single rules and thus requires (in our case) a total of 120 GrGen.NET-rules for all variations.

4 Table 2. Linguistic Structures matched to UML (excerpt). Linguistic Struc- Explanation ture object with the role An acting person or thing AG executing an action object with the role Person or thing affected by PAT an action or on which an action is being performed object with the role An action, executed by AG ACT (+AG +PAT) on PAT, or a relation between AG and PAT

ACT1

OMN PARS

PARS

ACT2

ACT3



Class, role, or instance Class, role, or instance

Method named ACT at the AG class with PAT param, association named ACT between AG and PAT classes, state or transition named ACT, etc. A complex action (OMN) Subcall of ACT2 and ACT3 from composed of multiple other ACT1 in sequence diagram or in actions (n×PARS). a comment on method ACT1

::::::::::::

1

UML model element

etc.

::::::::::::

Listing 1. SalE -Annotated Domain Description



[ #The game_of_chess|PAT #is played|ACT #between *two opponents|AG ].

2 3 4 5 6

[ They|AG move|ACT their|POSS pieces|{HAB,PAT} $alternately