Student Modelling in an Intelligent Tutoring System for the ... - CiteSeerX

Download PDF
10 downloads 0 Views 699KB Size Report
Comment
attempts to diagnose the underlying misconception of the mistake. In order to provide individualised help, the system holds a profile for every student, the long ...
Educational Technology & Society 3(4) 2000 ISSN 1436-4522

Student Modelling in an Intelligent Tutoring System for the Passive Voice of English Language Maria Virvou Department of Informatics, University of Piraeus 80, Karaoli and Dimitriou St., Piraeus 185 34, Greece [email protected] Tel: +301 4142269 Fax: +301 4112463

Dimitris Maras Department of Computer Science, University of Athens 30, Panepistimiou St., Athens 106 79, Greece [email protected] Tel: +301 5778782

Victoria Tsiriga Department of Informatics, University of Piraeus 80, Karaoli and Dimitriou St., Piraeus 185 34, Greece [email protected] Tel: +301 4142131 Fax: +301 4112463 ABSTRACT This paper describes an intelligent multimedia tutoring system for the passive voice of the English grammar. The system may be used to present theoretical issues about the passive voice and to provide exercises that the student may solve. The main focus of the tutor is on the student's error diagnosis process, which is performed by the student modelling component. When the student types the solution to an exercise, the system examines the correctness of the answer. If the student's answer has been erroneous it attempts to diagnose the underlying misconception of the mistake. In order to provide individualised help, the system holds a profile for every student, the long term student model. The student’s progress and his/her usual mistakes are recorded to this long term student model. This kind of information is used for the individualised error diagnosis of the student in subsequent sessions. In addition, the information stored about the student can also be used for the resolution of an arising ambiguity, as to what the underlying cause of a student error has been.

Keywords Intelligent Tutoring System, Multimedia, Student Modelling, Error Diagnosis, Language Learning

1. Introduction This paper describes an intelligent tutoring system for the domain of the passive voice of the English grammar for Greek students. There is increasing interest in the use of computer-assisted language instruction, especially in cases when the language taught is not the mother tongue of the students (Bos and van de Plassche 1994; Heift 1998; Kunichike et al. 1998). The need for tutoring systems that may provide individualised support to errors is even greater when students are taught more than one foreign languages simultaneously, as is very often the case for Greek students. However, as Murphy and McTear (1997) point out, many packages of computer assisted language learning fall short when it comes to providing the learner with individualised teaching and flexible feedback. Therefore, the multimedia tutor that we have developed has been based on Intelligent Tutoring Systems (ITSs). In this way, the system combines an attractive multimedia interface and adaptivity to individual student needs. Indeed, many researchers (e.g. McGraw 1994) point out that the integration of AI technologies and hypermedia may offer better support to students than any of these alone. The Passive Voice Tutor complies with the architecture of most ITSs, which consists of the domain knowledge, the student modeller, the advice generator and the user interface (Hartley and Sleeman 1973; Burton and Brown 1976; Wenger 1987). The main focus of the system described is on the student modelling component and especially on the error diagnosis process. The facility of individualised error diagnosis is particularly important for students, who can benefit from advice tailored to their problems. As Nagata (1997) notes, computer-assisted language instruction, in conjunction with

139

contemporary natural language processing technology, holds out the promise of unlimited, immediate feedback pinpointed to the specific grammatical errors made by the student. However, the underlying cause of a mistake may be difficult for a tutoring system to spot. For example, Hollnagel (1991; 1993) makes an important distinction between the underlying cause or genotype of an error and the observable manifestation or phenotype of the error. In addition, Mitrovic et al. (1996) argue that ambiguity may be a problem, since there may be different explanations of observed incorrect user’s actions. In the case of the passive tutor, error diagnosis is performed by a short term and a long term student model (Rich 1979; 1983). The short term student model is responsible for analysing the student’s mistakes in the current session. On the other hand, the long term model of each student contains information about previous mistakes or correct knowledge of the student. The long term student model is used for ambiguity resolution of competing hypotheses about errors. For example, if a student has been recorded to have been prone to spelling mistakes, then the system may use this piece of information for the interpretation of a mistake by favouring the cause of wrong spelling rather than lack of grammar knowledge at some points of the interaction. In this way we achieve a bi-directional interaction between the long term and the short term student model for the benefit of more individualised error diagnosis and help to the student. The Passive Voice Tutor uses a categorisation of common student mistakes, so that it may identify the cause of a particular error. The categorisation has resulted from an empirical study that involved both students and teachers. In addition, this empirical study, provided information about the kind of exercises to be encoded into the system, as well as the underlying teaching strategy to be followed by the Passive Voice Tutor. The domain representation of the Passive Voice Tutor contains knowledge about how to convert a sentence from one voice to another and knowledge about prerequisite grammatical concepts, such as the conjugation of verbs, irregular verbs, etc. In addition, the system is capable of constructing dynamically new exercises for the student to solve. This means that it incorporates syntactic and semantic knowledge concerning the words that may constitute a meaningful sentence. Hence, the system is able to show a wide range of new examples to students and to perform error diagnosis in new exercises. The ultimate objective for educational software is that it should be educationally beneficial and it is exactly in such environments that it is important to track how usability contributes (or not) to educational goals (Jones et al. 1999). Therefore, in this research, we have evaluated the error diagnosis and advice generation components of the Passive Voice Tutor, so as to ensure that the system can be useful and effective in helping students learn.

2. Related Work The problem of teaching a language through a computer assisted approach is very significant and has attracted a wide interest from the community of both educators and computer scientists. With regard to second language learning ITSs concerned with formal aspects of natural language, Chanier et al. (1992) distinguish two types of systems: computational-errors and early deeper analysis systems. Computational-errors systems, on the one hand, have a large linguistic coverage, but depth limited error analysis procedures. On the other hand, early systems with deeper error analysis have a narrow linguistic coverage but a quite exhaustive description of the possible errors, which may have occurred in their domain. The Passive Voice Tutor is a system with deeper error analysis similar with other systems such as the English Tutor (Fum et al. 1992), the VP2 (Schuster 1986), and Spengels (Bos and van de Plassche 1994). In this section we highlight the difference of our approach, as compared to these three systems that are representative of systems with deeper error analysis. English Tutor is an intelligent system, which comprises a Verb Generation Expert. This expert is devoted to tense generation and is capable of solving fill-in exercises. Each exercise consists of one or more sentences in which some verbs have to be conjugated into the appropriate tenses. In the English Tutor, student modelling is based on a methodology called backward model tracing. This methodology is used to analyse the reasoning process of the student by reconstructing step by step and in reverse order the chain of reasoning that s/he has followed in giving his/her answer. In order to do this, both correct domain specific knowledge and a catalogue of stereotyped mal-rules is used. There is also a meta-bug library that describes generic ways of altering correct knowledge and is used in cases where the catalogue of mal-rules is insufficient for the explanation of the student’s behaviour. One advantage of the Passive Voice Tutor against the English Tutor is that it contains stereotyped mal-rules for more than one levels and types of knowledge. For example, it has mal-rules concerning

140

typos, spelling mistakes, verb transformation errors, etc. In both systems, multiple hypotheses may be generated if an error occurs. However, the hypotheses resolution in the case of the Passive Voice Tutor is performed using the long-term student model, a component which does not exist in the English Tutor. Furthermore, our system has also the ability to construct dynamically new exercises for the student to solve. One other system that focuses on error diagnosis is VP2. VP2 is an ITS that is aimed at teaching English to Spanish people. The specific domain of interest in VP2 is English verbal constructions formed from a verb plus particle or verb plus prepositional phrase. The system apart from modelling the grammatical concepts of the English language models the Spanish language as well. The two grammar models are used in the process of the error analysis. VP2 first uses the English grammar model in order to parse a student’s answer. In the case that there is an error that cannot be recognised using this model, the system tries to perform the parsing using the Spanish model of grammar. In this way, the system is able to recognise errors that are related to the mother tongue of students. The Passive Voice Tutor is similar to VP2 in that it contains explicit representations of frequent errors of Greek students, which are due to the influence of their mother tongue. In addition, there are explicit representations of errors that are due to the influence of other foreign languages (e.g. French) that a Greek student may learn at the same time as English. However, a limitation of VP2 is that its diagnostic value seems to be minimal, since it allows no representation of the students’ mistakes and misconceptions. In the case of the Passive Voice Tutor, on the other hand, a personal long term model is held for every particular student, allowing the system to provide more individualised tutoring and diagnosis. Finally, a system that bears a lot of similarities to the Passive Voice Tutor is Spengels. Spengels is a knowledgebased tutor for the conjugation and spelling of English verbs. The system is addressed to Dutch students who have an elementary spelling knowledge of English. The domain knowledge consists of morphological rules, such as: infinitive + s → third person, singular of present tense (e.g. talks) infinitive + ed → past tense (e.g. talked). It also contains spelling alternation rules that are used to alter the standard morphological rules (e.g. y→ ie; such as try → tried). Spengels contains eight bug classes: 1) syntactic bugs, 2) overgeneralisations of a formula application, 3) application of a morphological mal-rule, 4) application of a buggy spelling rule, 5) typos, 6) analogy bugs, 7) compound bugs, and 8) miscellaneous bugs. It uses a fuzzy set theory to model the degree to which a student masters a feature or rule and it presents a short overview of the student’s performance at the end of each session. The Passive Voice Tutor has a wider domain and buggy knowledge than Spengels. The domain of Spengels (verb transformation) is covered in its prerequisite grammatical concepts. The passive voice generates several hypotheses about possible student errors at many levels. Similarly to the Passive Voice Tutor, Spengels also forms a long term student model. However, in Spengels the long term student model is only used to show to the student a short overview of his/her performance. In contrast, the Passive Voice Tutor uses the long term student model to refine hypotheses about possible student errors, as well as to show a progress report to the student.

3. Architecture of the Passive Voice Tutor The architecture of the Passive Voice Tutor follows the main line of Intelligent Tutoring Systems (ITS) architectures. It is widely agreed that the major functional components of an ITS architecture are the domain knowledge, the student modeller, the advice generator and the user interface (Hartley and Sleeman 1973; Burton and Brown 1976; Wenger 1987). In this section we will briefly describe the domain knowledge, the advice generator and the user interface, while the student modelling is described in detail in Section 4. The domain knowledge of the Passive Voice Tutor consists of procedures concerning the conversion of a sentence from active into passive voice and vice versa, a vocabulary of English words used in the construction of exercises, the semantic relations of the words of the vocabulary and procedures and rules about prerequisite grammatical concepts. The domain knowledge of the Passive Voice Tutor is responsible for performing the following tasks: Ø Ø

Parsing the exercise sentence. Identifying the semantic relationships between the subject, the verb and the object and judging whether the exercise sentence makes sense.

141

Ø Ø Ø Ø

Transforming the object/subject of the active/passive sentence into the subject/object in the passive/active sentence, respectively. Identifying the tense of the verb used in the active/passive sentence. Conjugating the verb into the appropriate tense in the passive/active sentence. Transforming the subject/object of the active/passive sentence into the object/subject in the passive/active sentence, respectively.

In order to illustrate a part of the transformation from the active into the passive voice of an exercise sentence, we present some PROLOG predicates. article(X, subject_passive(Y)):article(X, object_active(Y)). add(verb(be, Tense),verb_passive(participle(A)):verb(A, Tense). add(by, object_passive(C)):subject_active(C), explicit(C). The tasks performed by the domain knowledge of the Passive Voice Tutor are supported by a knowledge base that represents the vocabulary used in the exercise sentences. For each word, a number of attributes are associated with it. For example, in the case of a noun one attribute represents whether it is uncountable or not, and other attributes concern the Greek translation of the noun and translations to other languages, such as French etc. In case of a verb, attributes represent whether the verb is irregular, along with the irregular form of the past and past participle of the verb, etc. In addition, the words included in the vocabulary are related through a semantic net, so that the system may be able to identify whether a sentence makes sense or not. The advice generator is responsible for acting when the user makes an error. In such case, it tries to respond in the most appropriate way by informing the user about what the cause of the error has been and by showing to him/her the relevant part of the theory. For example, if a student has made a mistake that is related to the use of verb tenses then the advice generator will show to the student the part of the theory that deals with this subject. Furthermore, the advice generator is the component that is responsible for constructing new exercises, as well as showing to the learner a set of grammar pages on demand. For example, in one mode of function of the Passive Voice Tutor, the student creates exercises and the system is trying to solve them. The student may create sentences in active voice by selecting subjects, verbs and objects from three lists respectively. When the student has chosen the basic elements of the sentence, s/he may enrich the active sentence using certain supplementing words. Such words include articles, pronouns, and the words “a lot of”, “a few”, “some”, “many”. In addition, the student may also choose singular or plural form for the subject and object as well as s/he can select a tense for the verb. While constructing an active sentence, the system has the ability to semantically check the created sentence, through the semantic net of the domain knowledge that represents the relations between words of the vocabulary. Irrespective of the semantic correctness of the sentence, the system transforms it into the passive voice whenever it is asked to. Finally, the user interface is quite important for this kind of application, because it can stimulate the student’s interest in learning. In addition, it conveys the functionality of a computer application to the user and translates the user’s input into a machine-specific format (Plass 1998). The user interface of the Passive Voice Tutor, is a multimedia user interface, which involves animations, sounds and a limited form of natural language so that it can attract the student’s interest in the subject.

4. Modelling the Student Knowledge The student modeller is responsible for preserving the system’s estimation of the learner’s proficiency in the domain as well as his/her proneness to commit errors. The emphasis on the student modelling component of the Passive Voice Tutor has been put on the bi-directional interaction of two sub-components: the long term and the short term student model (Rich 1979; 1983; Jones and Virvou 1991).

142

The Passive Voice Tutor constructs a student model, which serves as a source of information that can be used for the interpretation of the student’s actions and possible mistakes in solving exercises. The student modeller checks the student’s answer against the expert’s answer and in a case of an error, it performs error diagnosis. While performing error diagnosis, the student’s answer is checked against the set of the erroneous versions that the system is able to identify. One important source of errors (although not the only one) is considered to be the interference of the mother tongue and other foreign languages that the student is familiar with. Error diagnosis is performed by the short term student model. Another responsibility of the student modeller is to form the long term student model. The long term student model constitutes a history model of the student’s weaknesses and progress. The long term student model influences the process of error diagnosis. For example, if a student has been recorded to have frequently made typographic errors but no grammar errors at all, then in case of ambiguity the former cause is favoured. In addition, as Cook and Kay (1994) have shown, students benefit from viewing their own student models. Therefore, this kind of information is used not only for refining the error diagnosis process but also for presenting it to the user.

4.1. Error Diagnosis in Exercises An empirical study that we conducted among human teachers and their students showed that both teachers and students were very interested in knowing which categories of error individual students were prone to and which grammatical concepts they had mastered. This result was in accordance with results from other empirical studies as well (e.g. Bos and van de Plassche 1994). Human tutors also considered it important that the type of exercises should be similar to the type used in exam papers of state schools. Therefore, the student, while working with the Passive Voice Tutor, is given three types of exercise to select from (Virvou and Maras 1999a): 1. Multiple choice exercises: Multiple choice questions are the most widely used and highly regarded of the selection-type questions (Gronlund 1992). The information provided by the empirical study was used in order to develop a library that held the multiple choice questions and the erroneous answers that could be presented to the student. While selecting the multiple choice question to provide the student, the Passive Voice Tutor requests input from the student modelling component, as to what the main difficulties of the particular student are. In this way the selected question is relevant to those parts of the theory that the student has been recorded to have weaknesses in. Furthermore, the results of the empirical study have been used for the analysis of possible students’ mistakes in the design of the diagnostic component of the student modeller. Each erroneous student’s answer is checked against the set of the erroneous versions that the system stores in its bug library. 2. Exercises where the user is asked to rewrite a sentence using passive voice: In this case, the student is given a sentence in the active voice and s/he is asked to rewrite the sentence in the passive voice. At this kind of exercise, the diagnosis of the students’ mistakes is done in a more sophisticated way than in the case of multiple choice exercises. In order to reason about the correctness or the cause of an erroneous answer, the Passive Voice Tutor performs a restricted natural language processing while parsing the student’s input. This is due to the fact that the student in this kind of exercise has more freedom in the creation of his/her answer. 3. Exercises where the user is asked to rewrite a sentence using active voice: In those exercises the user is asked to rewrite a sentence, given the passive voice into the active voice. The system operation in a case like this is similar to the one described in the exercises where the user is asked to rewrite a sentence using passive voice. As Brown (1997) points out, given the advantages of individual, time -independent language testing, computerbased testing will no doubt prove to be a positive development in assessment practice. The Passive Voice Tutor contains knowledge about how to solve an exercise correctly and in several faulty ways. The student modelling component of the Passive Voice Tutor uses a combination of buggy and overlay techniques to error diagnosis (Wenger 1987). Buggy procedures are related to the conversion of an active into a passive sentence and the opposite, as well as to the prerequisite grammatical concepts. Each one of these procedures is associated with a certain category of error. For example, a common mistake that students seem to make is the one illustrated in Figure 1. In this case, the student has neglected the rule that says that the word ‘with’ is used instead of ‘by’ in the case where the object of the passive sentence is material and not an agent. Similarly, a buggy procedure associated with prerequisite grammatical concepts may be the use of the suffix ‘ed’ while forming the past participle of an irregular verb.

143

The system is also able to identify not known procedures of transformation. For example, a Greek student may have a difficulty in understanding the following conversion: Active Sentence: Passive Sentence:

People believe that she is clever. She is believed to be clever.

This is so because there is no such passive voice structure in the Greek language.

Figure 1. Erroneous student’s answer while converting a sentence from active into passive voice Error diagnosis is performed by the Passive Voice Tutor in the Solving Exercises Mode. In multiple choice exercises error diagnosis is simple. For every erroneous answer that the student may select, there is an associated misconception. Therefore, depending on the erroneous selection that the student has made, a corresponding error message is presented, explaining the cause of the mistake. In the case of exercises where the student is asked to rewrite a sentence, error diagnosis becomes more sophisticated since in this case the student is allowed to be more creative than in multiple choice exercises. In the “rewrite” type of exercise the student is given a sentence in one voice (active or passive) and is asked to rewrite the sentence using another voice. The system incorporates knowledge about how to convert a sentence from one voice to another correctly. However, if the student’s answer differs from the system’s expectation then the system performs error diagnosis. The cases where the system performs error diagnosis result from the following steps of a parsing algorithm: 1. The system counts the words that constitute the sentence that the student has given as an answer. This sentence could be either in the active or in the passive voice. If the number of the words in the student’s answer is different from the ones that the system expects, then error diagnosis is performed. 2. The system is parsing the sentence typed, so as to verify whether one of the essential words, “by” or “with”, is present. If neither of the two is included in the student’s answer, then error diagnosis is performed. 3. Next, the correctness of the form of the verb “be” in the student’s answer is checked. The system first constructs the correct form of the verb based on the active sentence. If the student’s answer is different from the one expected by the system, then error diagnosis is performed. 4. In this step, the system is looking for spelling mistakes in the student’s answer. If it finds a spelling mistake, then it performs error diagnosis. 5. Finally, the system performs a syntactic analysis so as to ensure that the components of the sentence are in the right order. If the words of the sentence are not in the right order, then error diagnosis is performed. Error diagnosis is performed by matching each identified error against the buggy knowledge and corresponding explanations of the system’s knowledge base. In case more than one categories of error and/or explanations

144

match the identified mistake, then ambiguity resolution is performed, using the priorities of the categories of errors and the long term student model. As Chapelle (1998) points out, when errors are recognised, the process of the learner’s self-correction is also believed to be beneficial, particularly because the items for which self-correction occurs may be those for which learners’ knowledge is fragile. Therefore, after identifying a student’s mistake and providing a corresponding error message, the Passive Voice Tutor expects the student’s correction, instead of providing the correct answer. The answer of an exercise may be presented only on the student’s demand.

4.2. Categories of Error and Explanations Common student mistakes have been classified into eight broad categories of error that may be recognised by the system (Virvou and Maras 1999b). These errors are associated with the conversion of passive into active voice and vice versa and also the prerequisite grammatical concepts, such as irregular verbs. The categories of error are: 1. Accidental slips Accidental slips are occasional actions, which are not systematic and which the learner himself/herself can correct. For example, the student may have deleted some words and may have forgotten to complete the sentence. Trivial typographic errors are not considered as accidental slips. They are completely ignored by the system. Therefore when they occur the system considers the sentence correct. For example, the absence of a fullstop at the end of the sentence, absence of some space between words, redundant commas etc. are ignored. 2. Spelling mistakes A spelling mistake occurs when the user has typed the expected word so that one letter is redundant or missing or two neighbouring letters have been interchanged. For example, the student has typed “mather” instead of “mother”. 3. Article mistakes For example, the user may have used “a” instead of “an”. 4. Conversion mistakes For example, the user may have not converted the active verb “write” to the past participle “written”. 5. Irregular verb mistakes For example, the user may have used the usual ending “-ed” to create the past participle of an irregular verb (e.g. s/he may have typed “buyed” instead of “bought”). 6. Noun mistakes For example, the user may have typed “book” instead of “books”. 7. Verb tense mistakes For example, the user may have typed “been” instead of “being”. 8. Singular/Plural mistakes For example, the user may have typed “flowers has been put in the vase” instead of “flowers have been put in the vase”. Each category of error (apart from accidental slips) may also be associated with a variety of explanations about the possible cause of the mistake. Explanations have been based on identified strategies that learners may use in order to simplify the task of learning a second language (Richards 1974): Ø Language Transfer of Interference: The learner uses his/her previous mother-tongue (or possibly some other foreign language distinct from the target language) experience as a means of organising the second language. In the Passive Voice Tutor, this explanation is associated with each category of error, depending on the particular error. There are buggy procedures associated with this particular cause of an error for almost every category of error. For example, the spelling mistake “analyseis” instead of “analyses” is often made by Greek students for the plural of the Greek word “analysis”, because in Greek the plural is “analyseis”. Similarly, an example of singular/plural mistake that may be attributed to language interference could be: “Greek people is believed to have been used to sunny weather.” instead of “Greek people are believed to have been used to sunny weather.” This is so, because in Greek the translation of “people” is always considered singular. Ø Overgeneralisation of the target language rules: The learner has acquired a particular piece of linguis tic knowledge and some strategies which s/he had found helpful in organising the facts about the target language, but applies them too widely. This

145

Ø

Ø

Ø

Ø

explanation is also associated with many of the categories of error, depending on the particular error. For example, a conversion mistake that is associated with this explanation is: “It is believed by people that he committed the crime.” instead of “It is believed that he committed the crime.” In this case, the agent should not have been mentioned. However, the student has used an overgeneralisation of the rule for the conversion that includes the agent. Ignorance of rule restrictions: For example, in conversion mistakes a student may type: “The room was filled by smoke.” instead of “The room was filled with smoke.” This error may be due to the fact that the student may not know the restriction that “with” is used instead of “by” when the object of the passive voice sentence stands for materials rather than agents. Incomplete application of rules: Incomplete application of rules involves a failure to learn more complex types of structure because the learner finds s/he can achieve effective communication by using relatively simple rules. For example, in conversion mistakes the student may have typed: “The book was written.” instead of “The book was written by John Smith.” False concepts hypothesised: They may derive from faulty comprehension of a distinction in the target language. For example, verb tense mistakes may be attributed to this cause if the student has not understood correctly how to use verb tenses. Transfer on training: Errors may also come from the influence of a teacher and/or textbook writer who inevitably emphasises some aspect of the target language and neglects others.

The long-term student model records how often a student has made a certain type of error and what underlying cause was considered relevant. In some cases a mistake of the user may be attributed to more than one categories of error and/or explanations. In such a case, an amb iguity arises and the system should consult the student’s long-term model in order to resolve it. For example, if a user is given the sentence: “Alice brought flowers” and is asked to convert it to passive voice, the correct answer would be “Flowers were brought by Alice”. However, if the student types the sentence “Flowers were brought Alice” where “by” is missing then this mistake may be attributed to two categories of error. It could either be an accidental slip or a conversion mistake. If the particular student has not previously made any conversion mistakes but has frequently made accidental slips then this would have been recorded in his/her long-term student model. In this case the system would favour the accidental slip as the most probable cause of the ambiguous mistake. Question number

1

2

3

student answers

System’s Suggestion

System: My father will buy a book. Student: My father will be bought by a book. System: My sister reads magazines. Student: Magazines are readed by my sister.

Sentence conversion error. Notice that in Passive Voice subject and object exchange places.

System: Jim drives expensive cars. Student: Expensive cars are drive by Jim.

Irregular verb error. Notice that the verb is irregular and does not take the suffix "-ed" Verb transformation error. Notice that in Passive Voice we use the past participle of the verb. In such a case an ambiguity arises.

146

Degree of Compatibility with Human Tutors 100%

100%

80%

System: Columbus discovered America. Student: A merica is discovered by Columbus. 4

5

6

7

8

9

10

System: John has painted my car. Student: My car has be painted by John. System: Catherine cuts a flower. Student: An flower is cut by Catherine. System: My son brings the letters. Student: The letter is brought by my son.

Verb tense mistake. Notice that in Simple Past we use the words "was" or "were". The verb is not in Simple Present. This mistake might be attributed either to carelessness, or it could be a serious error. Verb transformation error. Incorrect form of the verb “be”.

Incorrect article in subject. The words that begin with consonant take this article. Singular – Plural confusion in the subject of the passive sentence.

Object is missing. Such an error could System: The dog ate the be attributed either to carelessness, or it steak. might be a more serious mistake. Student: The steak was eaten. System: Peter wrote a poem. Spelling error. This mistake might Student: A poem was writen have been due to the student’s by Peter. carelessness. System: The hairdresser cut Verb transformation error. Notice that my hair. the verb is irregular and does not take Student: My hair was cutted the suffix "-ed" by the hairdresser. Table 1. Tutors’ error diagnosis evaluation results

95%

90%

100%

90%

90%

100%

100%

5. Evaluation of the Student Modelling Component The correctness of the processes of the error diagnosis and advice generation is considered very important for the overall performance of an intelligent tutoring system. As Dix et al. (1993) pointed out a design can be evaluated before any implementation work has started to minimise the cost of early design errors. They also say that most techniques for evaluation at this stage are analytic and involve using an expert to assess the design against cognitive and usability principles. The overall performance of the Passive Voice Tutor has been evaluated by human tutors and students (Virvou et al. 2000). However, here we focus on the presentation of the evaluation that was relevant to the student modelling component and its contribution to advice generation. The evaluation methods employed at this phase of the evaluation comprised qualitative techniques, such as questionnaires (Virvou & Du Boulay 1999; Virvou & Kabassi 2000; Virvou & Tsiriga 2000). We compare the Passive Voice Tutor’s error analysis and advice generation with that of 10 human tutors who were asked individually to give their comments on the sample. Tutors were given a set of erroneous answers to questions relating to the domain of the passive voice of the English language, which were generated using the system’s domain knowledge and bug library. Each one of the questions was accompanied with an erroneous answer. For each question, each tutor had to evaluate the suggestion of the system concerning the underlying misconception of the error. In addition, tutors were asked whether they thought that the mistake was made due to the student’s carelessness or if it was a serious error. Table 1 illustrates the analysis of a sample of the evaluation results. The first column provides the system’s question along with an erroneous answer of a student. In the second column we show how the Passive Voice Tutor explained the related misconception and in the third column, we give the degree of compatibility of the system’s suggestions to the human experts’ suggestions for each mistake.

147

The results of the evaluation showed that the Passive Voice Tutor was successful at achieving a high degree of compatibility with the human experts’ opinion. We also found out that there were limitations concerning the ambiguity resolution (for example in question 3). A problem with the evaluation was that the human tutors involved in it lacked experience with other educational software packages. In addition, due to the limited period of time in which the evaluation of the system was conducted, the system’s restrictions related to temporal aspects of the student modelling component could not be spotted by the human tutors. Temporal aspects may be related to the students’ progress in learning over time or forgetting, etc. Therefore, we plan to conduct a second phase of evaluation, which is going to last for a longer period and will focus on the temporal aspects of the long term student model.

6. Conclusions The Passive Voice Tutor is an educational program that combines the attractiveness and user- friendliness of a multimedia interface with individualised help that an ITS can provide. In particular, its student modelling component forms a short and a long term student model for each user and performs error diagnosis. The ambiguity resolution of competing hypotheses about students’ errors is based on the long term student model, which contains information about previous mistakes or correct knowledge of the student. In order to perform error diagnosis, the system bears a detailed categorisation of common students’ mistakes along with their explanations. The error diagnosis process of the Passive Voice Tutor is especially focused on errors due to interference of the mother tongue of Greek students as well as of other languages that they may learn at the same time (e.g. French). Given the results of the evaluation, of the student modelling component, it is shown that the explanations that the Passive Voice Tutor can give are considered satisfactory. Furthermore, our system has also the ability to construct dynamically new exercises for the student to solve. For each exercise, the system is performing semantic analysis, so as to ensure that the provided sentence makes logical sense. Within the future plans of this research, is the improvement of the student modelling component of the Passive Voice Tutor, so that it may cope with temporal aspects of the student learning process. These temporal extensions will include the modelling of the fact that the student is learning some new concepts while working with the Passive Voice Tutor, as well as that s/he may forget some already possessed concepts after some time has passed. In addition, the adaptation of the difficulty of the exercises provided to the student according to his/her proficiency level, will also be addressed in future releases of the Passive Voice Tutor.

Acknowledgements The authors wish to thank Katerina Kabassi for her helpful comments.

7. References Bos, E. & van de Plassche, J. (1994). A Knowledge-Based, English Verb-Form Tutor. Journal of Artificial Intelligence in Education, 5 (1), 107-129. Brown, J. D. (1997). Computers in language testing: Present research and some future directions. Language Learning & Technology, 1 (1), 44-59. Burton, R. R. & Brown, J. S. (1976). A tutoring and student modeling paradigm for gaming enviroments. Computer Science and Education, ACM SIGCSE Bulletin, 8 (1), 236-246. Chanier, T., Pengelly, M., Twidale, M. & Self, J. (1992). Conceptual Modelling in Error Analysis in ComputerAssisted Language Learning Systems. In M. L. Swartz & M. Yazdani (Eds.) Intelligent Tutoring Systems for Foreign Language Learning, Berlin: Springer-Verlag, 125-150. Chapelle, C. A. (1998). Multimedia CALL: Lessons to be Learned from Research on Instructed SLA. Language Learning & Technology, 2 (1), 22-34. Cook, R. & Kay, J. (1994). The Justified User Model: A Viewable, Explained User Model. In Goodman, B., Kobsa, A., and Litman, D. (Eds.) Proceedings of the Fourth International Conference on User Modeling, Hyannis, Mass.: The Mitre Corp, 145-150.

148

Dix, A., Finlay, J., Abowd, G. & Beale, R. (1993). Human-Computer Interaction, NY: Prentice-Hall. Fum, D., Giangrandi, O. & Tasso, C. (1992). The Use of Explanation-Based Learning for Modeling Student Behavior in Foreign Language Tutoring . In M. L. Swartz, M. Yazdani (Eds.) Intelligent Tutoring Systems for Foreign Language Learning, Berlin: Springer Verlag, 151-170. Gronlund, N. E. (1993). How to make achievement tests and assessments, Boston, MA: Allyn and Bacon. Hartley, J. R. and Sleeman, D. H. (1973). Towards intelligent teaching systems. International Journal of ManMachine Studies, 5, 215-236. Heift, T. (1998). An Interactive Intelligent Language Tutor Over the Internet. In Ottmann, T. & Tomek, I. (Eds.) Proceedings of ED-MEDIA, ED-TELECOM 98, World Conference on Education Multimedia and Educational Telecommunications, Vol. 2, Charlottesville, VA: AACE, 508-512. Hollnagel, E. (1991). The Phenotype of Erroneous Actions: Implications for HCI Design. In G. R. S. Weir and J.L. Alty (Eds.) Human-ComputerInteraction and Complex Systems, London: Academic Press Ltd, 73-112. Hollnagel, E. (1993). The Phenotype of Erroneous Actions. International Journal of Man-Machine Studies, 39, 1-32. Jones, A., Scanlon, E., Tosunoglu, C., Morris, E., Ross, S., Butcher, P. & Greenberg, J. (1999). Contexts for Evaluating Educational Software. Interacting with Computers, 11 (5), 499-516. Jones, J. & Virvou, M. (1991). User modelling and advice giving in Intelligent Help Systems for UNIX. Information and Software Technology, 33, 121-133. Kunichike, H., Takeuchi, A. & Otsuki, S. (1998). Automatic Generation of Questions for a Comprehension Test in English Learning. In Ottmann, T. & Tomek, I. (Eds.) Proceedings of ED-MEDIA, ED-TELECOM 98, World Conference on Education Multimedia and Educational Telecommunications, Vol. 1, Charlottesville, VA: AACE, 767-772. McGraw, K. L. (1994). Performance Support Systems: Integrating AI, Hypermedia and CBT to Enhance User Performance. Journal of Artificial Intelligence in Education, 5 (1), 3-26. Mitrovic, A., Djordjevic-Kajan, S. & Stoimenov, L. (1996). INSTRUCT: Modeling students by asking questions. User Modeling and User Adapted Interaction, 6 (4), 273-302. Murphy, M. & McTear, M. (1997). Learner Modelling for Intelligent CALL. In A. Jameson, C. Paris, and C. Tasso (Eds.) Proceedings of the Sixth International Conference on User Modeling, Vienna: Springer, 301-312. Nagata, N. (1997). An experimental comparison of deductive and inductive feedback generated by a simple parser. System, 25 (4), 515-534. Plass, J. L. (1998). Design and Evaluation of the User Interface of Foreign Language Multimedia Software: A Cognitive Approach. Language Learning & Technology, 2 (1), 35-45. Rich, E. (1979). User Modelling via Stereotypes. Cognitive Science, 3 (4), 329-354. Rich, E. (1983). Users are Individuals: Individualizing User Models. International Journal of Man-Machine Studies, 18, 199-214. Richards, J. C. (1974). Error Analysis: Perspectives on Second Language Acquisition, London: Longman (Reprinted in 1984). Schuster, E. (1986). The role of native grammars in correcting errors in second language learning. Computational Intelligence, 2, 93-98.

149

Virvou, M. & Du Boulay, B. (1999). Human Plausible Reasoning for Intelligent Help. User Modeling and UserAdapted Interaction, 9 (4), 321-375. Virvou, M. & Kabassi, K. (2000). An Empirical Study Concerning Graphical User Interfaces that Manipulate Files. In Bourdeau J. & Heller R. (Eds.) Proceedings of ED-MEDIA 2000, World Conference on Education Multimedia and Educational Telecommunications, Charlottesville, VA: AACE, 1117-1122. Virvou, M. & Maras, D. (1999a). An intelligent multimedia tutor for English as a second language. In Collis, B. & Oliver, R. (Eds.) Proceedings of ED-MEDIA 99, World Conference on Educational Multimedia, Hypermedia & Telecommunications, Vol. 2, Charlottesville, VA: AACE, 928-932. Virvou, M. & Maras, D. (1999b). Error Diagnosis in an English Tutor. In Bullinger, H.-J. & Ziegler, J. (Eds.) Human-Computer Interaction: Communication, Cooperation, and Application Design, Proceedings of the 8th International Conference on Human-Computer Interaction – HCI International '99, Vol. 2, Mahwah, NJ: Lawrence Erlbaum Associates, 657-661. Virvou, M. & Tsiriga, V. (2000). Involving effectively teachers and students in the life cycle of an intelligent tutoring system. Educational Technology & Society, 3 (3), 511- 521. Virvou, M., Tsiriga, V. & Maras, D. (2000). Evaluation of an ITS for the Passive Voice of the English Language Using the CIAO! Framework. In Bourdeau J. & Heller R. (Eds.) Proceedings of ED-MEDIA 2000, World Conference on Education Multimedia and Educational Telecommunications, Charlottesville, VA: AACE, 17221723. Wenger, E. (1987). Artificial Intelligence and Tutoring Systems, Los Altos, CA: Morgan Kaufman.

150