Learner Modelling for Intelligent CALL - CiteSeerX

In Anthony Jameson, Cécile Paris, and Carlo Tasso (Eds.), User Modeling: Proceedings of the Sixth International Conference, UM97. Vienna, New York: Springer Wien New York. © CISM, 1997. Available on−line from http://um.org.

Learner Modelling for Intelligent CALL Maureen Murphy★ and Michael McTear School of Information and Software Engineering, University of Ulster, N. Ireland

Abstract. The demand for software for Computer-Assisted Language Learning (CALL) is increasing considerably. However a drawback of most, if not all, of the currently available CALL software is that it cannot provide very helpful feedback to the learner. The aim of this project was to work towards providing a more adequate and user-oriented interface for CALL. The project focused on the design of an application called CASTLE which takes into account the strengths, weaknesses, preferences and level of proficiency of each individual student when tutoring. This was accomplished by developing a module that provides detailed linguistic analysis of the learner’s response to the exercises of the program, a module that creates a dynamic model of the learner, and a module that controls the system’s reactions to the learner’s input and the structure of the materials offered to the learner.

1 Introduction Computer-Assisted Language Learning (CALL) systems are becoming increasingly popular within the educational, training and business worlds. They have many advantages, for example, giving learners more independence from classrooms and allowing them the option to work on their material at any time of the day. Once implemented, it can be expected that the cost for CALL systems is considerably lower than for classroom teaching, and when used in conjunction with traditional classroom study, students can study more independently, leaving the teacher more time to concentrate on aspects of language teaching unsuitable for the computer, such as pronunciation and work on spoken dialogue. There are many CALL packages commercially available with excellent GUIs, multimedia features and well designed tutorial sessions. However, invariably, these packages fall short when it comes to providing the learner with individualised teaching and flexible feedback, necessary features if true learning is to take place. As a result, CALL systems are often perceived by learners and teachers as dumb and inflexible, which is demotivating for the learner and restricts the independent use of CALL systems considerably. If CALL applications are to be used more widely, and with more confidence, they need to be able to provide this support for individual learners, with course materials as well as the system’s reactions to the learner’s input being adapted to the individual learner’s level of proficiency, preferred learning strategies, and other aspects such as their knowledge of other languages. Such adaptivity is made possible through the use of a Learner Model which stores characteristics of the learner relevant to the system’s tutoring strategies. —————————— ★

This work has been funded under the EU’s Telematics Applications of Common Interest—Language Engineering LE1-1615.

302

M. Murphy and M. McTear

This paper is concerned with the specification and design of a Learner Modelling component for a CALL system called CASTLE. CASTLE has been developed as part of the RECALL1 project, which has sought to investigate the current market and future direction of CALL software.

2 Requirements Specification In spite of the general observation that it would be advantageous if CALL systems were able to adapt to the needs and levels of proficiency of individual learners, it is difficult to obtain specific requirements relating to adaptivity and individualisation from users of CALL systems. This is due to the fact that such enhancements have not yet been provided in commercially available systems and, as a result, users have difficulty in visualising exactly what form such adaptivity and individualisation should take. The RECALL team adopted 3 approaches for gathering the requirements for the Learner Modelling component: 1. requirements elicited in previous market studies and reported in project deliverables; 2. requirements identified in a literature review; 3. requirements gathered as a result of a questionnaire distributed to language tutors. On the basis of these sources, a set of 25 requirements for CALL systems in general, and for adaptivity and individualisation specifically, were gathered. These included requirements for flexibility, individualised learning activity, practice in specific areas of a language where the system has identified that the learner is weak, and adaptivity to the individual learner.

3 Functional Architecture and Design The Learner Module is one of the components of a functional architecture which has been developed to provide the required CASTLE functionality. The system components and the functional relations between them are illustrated in Figure 1. 3.1 CASTLE Scenario Through discussions with language teachers it became apparent that an eclectic approach is often used in contemporary language teaching. This approach embraces the traditional grammatical approach in combination with the more functional communicative approach (Thume, 1992). Consequently, in order to model the adept language tutor as closely as possible, CASTLE provides a range of communicative role-play scenarios which are supported by a grammatically based remedial tutoring facility. Learners are able to type almost free input to questions posed by the system. Input is restricted according to a set of allowable words, but the learner may enter a range of sentence structures using a number of different interaction modes. Thus, CASTLE allows the user to practice building (syntactically) correct sentences while mastering typical conversational situations. When the learner encounters problems, CASTLE guides her through a set of grammaticallybased remedial exercises which underline the basic formal structures of the language.

—————————— 1

RECALL: Repairing Errors in Computer Aided Language Learning.

Learner Modelling for Intelligent CALL

303

Stereotype Library Acquisition Rules

Learner Module

Domain Model

GUI

Exercise Library

Test Library

Tutoring Module

Feedback Library

Learner Model

Error Classification

Context Model

Grammar Pages

Error Grammar

Diagnosis Module

Lexicon

Grammar

Figure 1. CASTLE architecture.

The Context Model is a central repository for storing all relevant data about the current state of the system, i.e., it holds all the information essential for the current session to operate. This includes the learner model information such the current learner’s preferred feedback media and the exercise and item definitions of the current exercise, and the cause of the learner’s error, as identified by the Diagnosis Module. The Diagnosis Module checks the learner input for correctness and provides a detailed linguistic error description in the case of erroneous input. The learner input is analysed by means of a chart parser operating on a grammar, an error grammar and a lexicon. In order to describe the error in some detail, the error description is based on a hierarchy of error classes. The two main functions of the Tutoring Module are to tailor the language exercises for each learner and to determine when to intercede in the lesson, using the tutoring approaches, strategies and techniques incorporated into the system. The Tutoring Module contains an explicit representation of the grammatical and linguistic concepts within the domain and how they are related, a full set of exercises, a feedback library which stores canned feedback text and a series of adaptive templates, and a full set of grammar pages which are available to the learner on demand.

304

M. Murphy and M. McTear

3.2 Components of the Learner Module The Learner Module has four main components: a Learner Model Library, a Stereotype Library, Implicit Acquisition Rules, and an Inference Engine. Contents of the Learner Model. The Learner Model contains the model of the individual learner which is to be regularly updated in the course of her interactions with the system. Some items of information carried within this model will, however, be stable or relatively stable and not subject to regular updating. Based on a functional architecture for user modelling systems proposed by Benyon and Murray (1993), we can identify three different groups of information contained in the Learner Model: 1. Profile Model: personal learner details; 2. Student Model: the system’s estimate of the learner’s (grammatical) proficiency in the language and proneness to commit errors; 3. Cognitive Model: relatively stable characteristics of the learner. Most of the items in the Profile Model as well as those found within the Cognitive Model are stable and unlikely to require updating, whereas items under Student Model are dynamic and will require updating as the learner interacts with the system. Table 1 outlines the characteristics which have been identified under each category and provides information on the possible value types of the attributes, how and when these values are acquired, and whether the values are static or dynamic. Updating the Learner Model. In order to update the Learner Model, Context Model information is passed and analysed in the Learner Module. This information includes the error classes, the potential causes of errors which have occurred for each item, the response strategies which were selected by the Tutoring Module and the level of help that was sought by the learner. The explicit Domain Model (handled by the Tutoring Module) is a major source of data for the update mechanisms found within the Acquisition Rules knowledge source. This model was based on three popular commercial language courses and refined by members of the RECALL pedagogic team. The curriculum upon which CASTLE was based is divided into a number of partitions: e.g. Present Simple; Past Simple; Present Continuous; Present Perfect. These partitions have an implicit ordering which indicates that Partition 1 is normally taught before Partition 2, etc. Within each partition there are a number of lower level sections, each of which is made up of grammatical and linguistic items. Within a partition the different sections may be taught in any order, or there may be an ordering defined by the tutor. It was also found that there was a precondition relationship present among many of the grammatical and linguistic items which indicated that mastery of one item was dependent on mastery of other items within the language. Thus, there are two primary relation types within the Domain Model: precondition and temporal (i.e. taught before).

Learner Modelling for Intelligent CALL

305

Table 1. Characteristics of the Learner Model.

Type

Item

Description

How acquired

Profile

Name Native language Level of qualifications

Free text Free text None, primary, secondary, undergraduate, postgraduate Free text and 1 (Poor), 2 (Fair), 3 (Good), 4 (Very Good), 5 (Fluent) Business/ study/ recreation 0 (None), 1 (Poor), 2 (Fair), 3 (Good), 4 (Very Good) Time Entry from domain model and proficiency score Entry from hierarchy of Error Classes and system estimate of proneness Estimate of likely causes of errors Sound, graphics, natural language, animation Gap-filling, MCQ, Image Descriptions, Comprehension Yes / no

[Background language and proficiency]2 Motivation

Student Model

Initial target language proficiency Time elapsed [Domain topic and proficiency] [Error and proneness to commit this error] [Likely cause]

Cognitive Model

Preferred feedback media Preferred exercise types Interest in Grammar Usage of polite Form

—————————— 2

Yes / no

User input User Input

When acquired First First

Modified by User User

User input

First

User

User input

First

User

User input

First

User

Calculated Stereotype/ inferred

Each session Continuous

System System

Inferred

Continuous

System

Diagnosis Module User input

Continuous

System

Continuous

User and system

User input

First

User and system

User input/ stereotype User input

First

User and system User

First

Items surrounded by [ ] indicate that there may be zero or multiple entries for these items in the resulting Learner Model. For instance, a Learner may know multiple background languages which have associated proficiency values.

306

M. Murphy and M. McTear

−

The precondition relation denotes the fact that in order to know a topic, the learner must know underlying, supporting topics. The temporal relation is looser in that it indicates that one topic should be taught before another. Thus the temporal relation is defined between the partitions while the precondition relation holds between the topics in each of these partitions. Figure 2 illustrates a portion of the Domain Model used within CASTLE. The precondition relationships are drawn between nodes whilst the temporal relationships defined by the partitions are colour coded. To illustrate, mastery of topic ‘do+does’ (2c: topic c within section 2) depends upon mastery of (simpler) topics ‘verbs - positive sentences’ (2b) and ‘to - do verbs’ (1a).

11a

6d

10a

5a

4a

1a 1a

7b

9b

9a

8a

2a

1b

1c

6c

10c

6a

10d

4b

3a

7a

2c

2b

2d

5d

3b

3c

6b

5c

4c

4d

8b

5b

8c

8d 10b

Adjectives Pronouns Cnouns Wh-Words Infinitives Vocabulary Partition 1

Partition 2

Partition 3

Figure 2. Grammar Domain Model.

Learner Modelling for Intelligent CALL

307

Assigning the Initial Proficiencies. The Domain Model is used by the Stereotype Library at the start of the learner’s first interaction with the system. Based on a simple test the learner is assigned to one of the four stereotype groups defined within CASTLE (novice, beginner, intermediate, advanced). The primary information held within the stereotypes is the list of grammar and linguistic items or topics which language tutors believe that members of this stereotype should be familiar with. To illustrate, the proficiencies of the novice language learner (who typically has a poor knowledge of the language) is defined as follows: e.g.,

stereotype_topics(Language, Stereotype, Proficiency,Topics). stereotype_topics(english, novice, 1,[2a, 2b, 3c, ...]). stereotype_topics(english, novice, 2,[1a, 1b, 1c,....]).,

where Proficiency is either 0,1,2,3,4 and Topics is a list of lists of topic numbers which have this initial Proficiency value. The Inference Engine asserts the relevant proficiency facts for maureen, a novice, into the database using the following predicate: e.g.,

learners_achievement(User, Language, Topic, Proficiency,Source). learners_achievement(maureen, english, 2b, 1, stereotype). learners_achievement(maureen, english, 1a, 1, stereotype).

with the Source argument always set to stereotype. Updating the Proficiencies. This default topic information is updated during each interaction as the learner answers the questions or items within the exercises. The learner is given a score based on how well she answers the questions, and also how many times she uses the built-in help facility provided by the Tutoring Module. There are three levels of (individualised) help available for each user input, ordered so that more and more help is given until Level 3, when the answer is offered. The best result, from the user’s point of view, is that she should provide a syntactically correct input, without any need to make use of the hint facility. If the learner is able to pose a syntactically correct answer on the first attempt, the proficiency values for the underlying grammatical constructs which were used are updated positively. If, however, she enters a syntactically incorrect sentence, the Diagnosis Module must analyse the input and return an error description. This error description contains the general cause of the error (e.g. over-generalisation or interference) and the category of the error. These categories map to topics in the Student Model (see Table 1). The Context Module records the number of attempts the learner has made and the number of levels of hint she has had to use in order to answer the question correctly. The Implicit Acquisition Rules source holds a series of general update rules which are used to update the proficiency values for the topics based on the level of help which is provided to the learner. For example, update_topic(Topic):learners_achievement(User, Language, Topic, 2, stereotype), level_help(User, Language, Topic, 3), retract(learners_achievement(User, Language, Topic, _,_)), assert(learners_achievement(User, Language, Topic, 1, inferred)).

308

M. Murphy and M. McTear

As the learner enters more input within the communicative scenario, the Student Model records the number of errors which has been made per category. When the learner makes three errors of the same category the Tutoring Module recommends a remedial exercise which tutors on the domain topic associated with the error category. The remedial exercises are much more directed than the conversational or communicative free input found within the role-playing scenario. In order to select suitable remedial exercises the Tutoring Module must identify which prerequisite topics may need to be introduced/ re-introduced in order that the learner can master the topic which is causing problems. The first step is to identify the prerequisite topics, and in turn the proficiencies associated with each of these pre-requisites in the Learner Model. The Student Module ranks the pre-requisites based on the proficiency values and the sources of this information. The Source argument provides an implicit belief in the accuracy of the value. Thus, the Tutoring Module would prefer a topic which has a tested Source argument over one which has an inferred value as CASTLE is more confident in the former value being true. Within the remedial exercise the Context Module again records the average level of help which is provided to the learner when answering the exercise questions. This value is used to update the proficiency value for the remedial topic(s). Domain topics do not need to be explicitly taught and tested in order to achieve a proficiency level. The Implicit Acquisition Rules holds a series of rules which are able to infer proficiency levels of topics which are related to topics which have been tested. This is achieved by using the precondition relation defined between topics. For instance, if a complex topic has been tested and is related to a number of prerequisite (and simpler) topics in the Domain Model, the system assumes that the learner probably knows these simple concepts, and updates the Proficiency argument and Source argument accordingly if the Source argument is set to stereotype. The Learner Module also maintains an account of the learner’s proneness to commit errors. A number of error types have been identified in work for the RECALL Diagnosis Module. The problem that arises for the Learner Module is that of estimating a user’s proneness to commit errors of each of these types. A reliable index of a user’s liability to commit an error of type E is given by comparing the number of times E has been committed in the course of the session with the total number of questions answered or sentences entered, if the exercises require composition of full sentences. Suppose, for instance, that a learner has to input answers to 10 individual questions. If it happens that she commits 15 errors of type E it is clear that her error-proneness with regard to E is very high; if she had committed only two, the proneness would have been low. Since a user’s error-proneness for a given error type E is measured by her committing that error a given number of times in a set of exercises, it is represented as a percentage figure, calculated as follows: Proneness = (No. of errors / No. of questions) * 100. The learner’s proneness to commit errors is expressed as a number of error_proneness facts. For example, error_proneness(maureen,english,word_order,30). The Learner Model also maintains a representation of the learner’s overall proficiency in the language. The overall-proficiency takes the form of a percentage measure, calculated from a cumulative total of the learner’s proficiency scores in all the topics that she has completed. This

Learner Modelling for Intelligent CALL

309

estimate is used primarily by the learner during runtime when she asks for an overview of the system’s estimate of her knowledge of the target language.

4 Learner Module Interactions From the Learner Module’s perspective there are four main processes in the operation of CASTLE: 1. 2. 3. 4.

selection of exercises; interpretation and diagnosis of the learner’s input; system’s response to the input; updating the Learner Model (as described previously).

The Learner Module is involved in all four processes, and continuously interacts with the central Context Model in order to accept, validate, assimilate and update the Learner Model information. 4.1 Selection of Topics There are two cases which need to be accommodated in the selection of exercises at the beginning of a session. The first case occurs when the learner indicates that she has not used CASTLE before and the second case corresponds to the learner’s use of the system at later sessions. In the latter case the learner enters a unique identifier which is passed to the Learner Module. The appropriate Learner Model is retrieved from the Learner Models knowledge source. The Learner Module provides the GUI with a summary of what topics have been covered to date in the curriculum and how well the learner is performing within the curriculum. The learner has either the option of selecting from all the available topics and scenarios within the curriculum or have CASTLE select appropriate topics based on the contents of the Learner Model. These topics are subsequently used to select a suitable communicative scenario. If the learner selects a topic, the Tutoring Module may advise that: 1. the choice is appropriate; 2. the choice is inappropriate because it is beyond the learner’s current proficiency; 3. the choice is inappropriate because it is below the learner’s current proficiency, i.e. too elementary given the standard already attained. The Tutoring Module determines which of these three cases applies by consulting the Learner Model information in the Context Model. If the choice is appropriate no further action is necessary. If it is inappropriate i.e. 2 or 3 above, a message will be displayed to the user, informing her of the inappropriateness of the choice and a suggestion of an alternative which is more in line with her current proficiency. 4.2 Interpretation and Diagnosis of the Learner’s Input When the learner types in an answer to a question (item) the input is passed to the Tutoring Module and subsequently to the Context Model. If there is a set of anticipated correct and incorrect answers tagged to a question, the learner’s answer is checked against these. If the input does not match with these answers it is passed through the spell checker. These three activities obviously do

310

M. Murphy and M. McTear

not involve any learner information. If the input is not recognised at this stage it is sent to the Diagnosis Module for a full linguistic analysis. Rather than exhaustively searching the Error Grammar knowledge base, the Diagnosis Module requests learner information from the Context Model in order to prune the search. This information includes the learner’s background knowledge (for the selection of possible interference errors); the proneness to commit certain errors and the causes of errors which have already been made. Having found a hypothesis, or set of hypotheses to explain the error, the Diagnosis Module passes back the hypothesis(es) in the form of an error classification and a cause to the Context Model. At the end of an exercise the Learner Module retrieves relevant information from the learner model area within the Context Model to update the long-term Learner Model using the Acquisition Rules knowledge source. 4.3 Response to the Learner’s Input If the learner’s answer to a question is correct, no Learner Model information is required as a simple affirmation of the correctness of the response is given to the learner. However the Context Model will note that the learner answered the question correctly. If, however, the learner’s input is incorrect the Tutoring Module needs to select an appropriate response from the Feedback Library. In the case where the Diagnosis Module has sent multiple hypotheses for the cause of the error, the Tutoring Module may also have to select the most likely cause/error before selecting a response. This may involve presenting a similar or simpler exercise to clarify where the learner may have gone wrong. For the selection of a response the Tutoring Module needs to ascertain the type of response the learner prefers, the response strategies which have already been adopted and the amount of help that has been sought on previous occasions.

5 Related Work It is clear that the primary goal of a learner modelling component is to develop a consistent and theoretically sound model which can be used to evaluate each learner’s language proficiency. McCoy et al. (1996) present an interesting architecture which involves the following Input/Feedback Cycle: 1. The user enters a portion of text. 2. The Error Identification component tags all errors (one sentence at a time). − A syntactic parse is carried out. If more than one parse is found then the ‘best’ is selected (through a scoring mechanism). − If a syntactic mal-rule is used in the parse then the sentence and the mal-rule annotation (indicating the type of error which has been made) is passed to the Response Generator. 3. The Response Generator processes this data with information from the Learner Model and the History Model to decide which errors to correct in detail and how they should be corrected. 4. The Response Generator selects an appropriate teaching strategy. 5. The Learner can enter corrections and have them rechecked when the Response Generator responds. This process depends on a categorisation of errors as mal-rules for the diagnosis of learner errors and on a model of language acquisition to support the Response Generator in the selection of an

Learner Modelling for Intelligent CALL

311

appropriate teaching strategy. McCoy et al. (1996) propose a framework called SLALOM (Steps of Language Acquisition in a Layered Organisational Model) to represent the acquisition sequence of target rules. SLALOM divides language into a set of feature hierarchies (such as morphology, types of noun phrases, types of relative clauses) according to their relationship and complexity. Features of similar complexity are grouped together into layers representing stereotypical levels of language ability. These layers allow the system to make inferences. A similar hierarchy has been used in the CASTLE system for the diagnosis and treatment of the learner’s errors. A simpler type of representation is to be found in the VP2 system, where the user’s knowledge of a language is represented simply as a grammar of that language (Schuster and Finin, 1986). The sole purpose of VP2 is to account for transfer of language structures from a speaker’s native language to the target language, so this is adequate: a more comprehensive CALL system requires much more. The use of overlays involves viewing the user’s current state of knowledge as a subset of that of the system (or domain model). This may be an important part of a complete student model but since it allows no representation of the student’s mistakes and misconceptions, its diagnostic value seems to be minimal. Value-attribute pairs have been widely used, e.g. in GRUNDY (Rich, 1983) and also in Mr. Collins (Bull et al., 1995), to indicate those characteristics (preferences in the one case, proficiency in the other) which the user is believed to have, the value assigned to them and the confidence the system has in the rating of values for the various attributes. Bull et al. (1995) recommend two further types of information which could be stored in the Learner Model in addition to mal-rules and the student’s knowledge of other languages. These are: 1. the learning strategies which are employed by the learner (studying prior to exercises, guessing, questioning etc.); 2. the learner’s confidence in her performance, which can be expressed through the assignment of belief measures. The discussion of the student model within this system is based on belief measures, where the learner assigns a (modifiable) belief measure based on their assessment of performance and the system assigns a belief measure based on the learner’s average result of the last five attempts to use a grammatical rule. Concerning the acquisition of information, Bull et al. (1995) see their model as being acquired explicitly through asking the learner to explain her choice of a (wrong) answer when the system itself is unable to choose between two or more alternative explanations— when, for example, it is unclear whether there was a confusion of two grammatical rules or a transfer of a rule from another language. Also included in this system is the idea that the learner model should be available for inspection by the learner, thereby eliminating grounds for worry about confidentiality and perhaps enabling the model itself to function as a learning resource. Whilst it was acknowledged that Bull’s approach was theoretically interesting, the RECALL team adopted a more pragmatic approach to modelling. Rather than having dual measures of the learner’s proficiency in the language, the system maintains an estimate of the learner’s proficiency which the learner may view, and indeed modify at any time. In addition, if there are any ambiguities regarding the source of the error, the Tutoring Module may decide to present the learner with a more restricted remedial exercise with a different interaction mode which should eliminate some of the potential causes—e.g. a multiple choice question rather than free text should eliminate attentional errors. The Tutoring Module will also on occasion admit to the learner that it is unable to identify

312

M. Murphy and M. McTear

the source of the error but is happy to carry on nonetheless, rather than initiating a complex subdialogue between system and learner to get to the source of the problem.

6 Conclusion CASTLE employs simple, yet effective learner modelling techniques to provide a level of adaptivity which has not been present in commercial CALL products to date. Initial reaction from the evaluation study has been positive although it is acknowledged that the facilities provided by the HTML-based GUI are rather limited. In addition, the interactions between the Prolog-based Learning and Tutoring Modules, the C-based Diagnosis Module and the HTML interface are not ideal. Future work will concentrate on liasing with language tutors to refine the learner modelling update mechanisms, incorporating more multimedia features and investigating how CASTLE can be delivered in a distributed client-server architecture.

References Benyon, D., and Murray, D. (1993). Applying user modeling to human-computer interaction design. Artificial Intelligence Review 6:43–69. Bull, S., Pain, H., and Brna, P. (1995). Mr. Collins: A collaboratively constructed, inspectable student model for intelligent computer assisted language learning. Instructional Science 23:65–87. Laurillard, D. (1992). Principles for computer-based software design for language learning. Computer Assisted Language Learning 4(3):141–152. Merrill, D. C., Reiser, B. J., Ranney, M., and Trafton, J. G. (1992). Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. The Journal of the Learning Sciences 2(3):277–305. McCoy, K., Pennington, C. A., and Suri, L. Z. (1996). English error correction: A syntactic user model based on principled “mal-rule” scoring. In Proceedings of the Fifth International Conference on User Modeling, 59–66. Rich, E. (1983). Users are individuals: Individualizing user models. International Journal of ManMachine Studies 18:199–214. Thume, K. H. (1992). Studien zur Entwicklung und Effektivität von computergestütztem Frendsprachenerwerb. Regensburg: Roderer. Schuster, E., and Finin, T. (1986). VP2: The role of user modelling in correcting errors in second language learning. In Cohn, A. G., and Thomas, J. R., eds., Artificial Intelligence and Its Applications. 197–209.