Learning Assistant — Personalizing Learning Paths in e ... - IEEE Xplore

7th Computer Information Systems and Industrial Management Applications

Learning Assistant – personalizing learning paths in e-learning environments Halina Kwasnicka, Dorota Szul, Urszula Markowska-Kaczmar, Pawel B. Myszkowski Wroclaw University of Technology [email protected]

Abstract

to Christians in Roman Empire ([17]). Radio can be considered as the next medium of distance learning. The School of Air ([8]) is known as a station of different learning courses. University of Iowa initiated Educational TV. Video courses are popular in some universities, e.g., University in Madison. Satellite TV provided a new impetus to distance learning. Computers bring additional attractiveness of distance learning: e-lessons and other educational materials could be used not only as distance learning but also in schools, during lessons with teachers. CD and DVD enable storing and transfer lectures in multimedia form. One can use them in school as well as in selfeducation [12]. But landmark in education was observed with developing of Internet. The global computer network joins people and resources around the terrestrial globe. The way of knowledge transferring, the place and time of learning evolve, with them – methods of learning/teaching also evolve. The modern form of distance learning is developed as e-learning. Inside e-learning we distinguish:

The paper presents an agent called Learning Assistant, which is responsible for defining individual learning paths for pupils in e-learning environment. The Assistant is able to infer using metadata described pupils and didactic materials; this inference is a basis for building the individual learning path for each pupil. To build a learning path for a new pupil the agent uses information collected during introductory tests. A SOM neural network is used for grouping similar pupils. WebTeacher is e-learning environment in which Learning Assistant works. This environment is shortly presented in the paper. Next, we present the idea of personalization – we consider the individual’s pupil characteristic and a group of similar pupils characteristic. Data structures described didactic materials and pupils are also shortly explained. The performed experiments allow formulate some conclusions, they are described very shortly. Summary ends the paper.

• CBT (Computer Based Training), as the medium of knowledge transferring are used CD, floppy disks, DVD etc.,

1. Introduction

• WBT (Web Based Training), it uses WWW as a medium for learning areas of knowledge.

Internet becomes accessible for a wide part of contemporary societies and its role continuously widens. One of the interesting usage of Internet is e-learning. In this paper we focus on possibilities of support the learning process through Internet. We have built a kind of intelligent agents, called Learning Assistant, using an architecture of programmable agent and a SOM neural network [1]. Our Learning Assistant’s task is defining a learning plan for a given pupil who has to be familiar with assumed knowledge. Education changes during centuries, as new technologies are developed. Initially, knowledge between generations was transferred orally. Images, Egyptian papyruses are the next phase of people education. Discovering the method of printing in the middle of fifteenth century changed the form of education – people could transmit their knowledge by means of written documents ([2]). One can say that printing starts distance learning, e.g., Saint Paul sent his letters

978-0-7695-3184-7/08 $25.00 © 2008 IEEE DOI 10.1109/CISIM.2008.51

Web Based Training works in three modes: synchronous, asynchronous and individual. Synchronous mode, called also real-time mode, enables interactions between teachers and pupils similar as during normal (stationary) lessons. Asynchronous mode does not allow for on-line interactions. Computer systems store didactic materials in databases and present users a content using HTML format. This mode does not force regular and in-time learning, users must take care about systematic training. The individual mode uses tools similar as the asynchronous one. A pupil decides about the tempo and subjects of learning. This approach is useful in corporations for improving education of their personnel. We can enumerate the main advantages of elearning as: • learning in any place, in any time, ”just in time” – it

308

means, one can start learning in any time and on the suitable (preferred) level, also a cost per pupil is lower,

concept effect relationship ([15]). In ITES these relationship are used to define the relationships between learning materials. Learning materials are visualized as a diagram consisted of chapters, sections, subsections, and key concepts that must be learn. Defined connections between concepts and queries verifying possessed knowledge are used in the further learning process. Mathematical formulas are proposed to point out a set of concepts that should be repeated in the learning process. The authors of the system claimed that experiments have shown that the system supports the traditional education.

• individualization of learning process – a pupil learns in preferred style and tempo, a number of pupils can be increased without lowering the quality of learning, • high attractiveness of learning – it allows to join wide knowledge and didactic skill of good specialists with new technology, • the didactic materials can be relatively easily updated comparing with books.

2.2

In this paper we propose an intelligent agent, Learning Assistant, it is a module included into WebTeacher system useful in e-learning process. WebTeacher and Learning Assistant were developed and tested as two master thesis at Wroclaw University of Technology [14, 18]. The proposed approach is based on the idea of software agent architecture and SOM neural network. The developed assistant uses metadata concerning pupils’ activities and learning materials. The agent builds the learning path for a given pupil using inference mechanism. The problem is with a new pupil, when he/she logs into the system the first time. In our approach the agent uses information collected during the tests which all students have to perform before start the learning process.A SOM neural network is used for grouping similar pupils. The paper is structured as follows. The next section introduces shortly e-learning systems, including WebTeacher. Section third presents the Learning Assistant. An experiment and its results are described in section fourth. The last, fifth section summarizes the paper.

ELENA uses artificial intelligence techniques to provide friendly educational environment with a number of different functionalities [4, 5, 6]. It supports e-learning in distributed learning environment using semantic net technology. The aim of the system is to join personalization with an open, dynamic network repository. Learning in the open environment requires very effective personalization method. The main questions are: • How we can support identification and profiling of pupils in distributed environment? • How to integrate ability of personalization with other essential functionalities supporting pupils? The system architecture allows for personalization on the basis of distributed services [5]. Knowledge about the pupils and resources is represented using semantic web technology [6]. The architecture of the system proposed by Dolog [5] is a basis for developing e-learning system working in distributed environment. Semantic nets can be used for metadata representation, it meets standards concerning normalization and structuralization, recommended by W3C and IEEE. Functionalities defined in terms of services are implemented as autonomic modules. The main disadvantage of the presented approach is that only prototype of the system was developed and presented in [5]. Therefore it is impossible to evaluate the system.

2 E-learning systems – a short overview Developed WebTeacher system uses some ideas taken from two projects: ITES and ELENA. They are presented below.

2.1

System ELENA

System ITES

ITES was proposed in 1997 as a computer system which supports learning problems concerned with elaboration of individual courses of learning [10, 11]. ITES is based on CORAL project (Cooperative Remotely Accessible Learning) [16]. The system consists of a diagnostic module and an expert system which generates a learning path for each pupil based on pupils answers and relations of test queries with learning subjects [11]. Students learn new concepts and new connections between known already concepts. This knowledge can be represented in the form of conceptual map [13]. Some relations (connections) influence the ability of new concepts learning, they are called

2.3

WebTeacher system

WebTeacher [14] consists of two main parts: a system that is an e-learning environment for teachers and pupils and a module that adjusts a learning process to the individual needs and possibilities of pupils. The first part of the system manages the didactic (learning) materials (contents), users’ accounts, and reports. Also it is responsible for didactic material presentation. In this paper we focus on the second module, namely, module of learning paths personalization [18].

309

WebTeacher is a general purpose system, its architecture disregards learning contents and possible pupils (i.e., grammar or high school, university, etc.). The important requirement is that it should be useful for different institutions, so its configuration should allows for easy adjustability the system to the particular needs of users. The system should improve efficiency and comfort of the learning process. Users (pupils) reach the system by Internet. WebTeacher distinguishes two actors: a teacher and a pupil. The teacher manages learning contents and services supporting learning process. He also supervises the learning progress of pupils. The pupil takes the didactic content provided by teachers, check the progress of his learning process using tests (or other examination forms) supplied by teachers, uses services supporting learning process, traces his course of learning and reports some problems that he noticed and other remarks. One instance of WebTeacher can contain didactic contents concerning only one discipline (domain). It consists of eight packages in which a particular group of functionalities is realized (see Fig. 1).

teractive animations, slides and active hyperlinks to other objects. A section is a logical element that groups lectures, each lecture is a logical group of issues, each issue is a presentation of an elementary portion of knowledge. Two kinds of issues are defined: basic issues (atomic elements) and examples. They are described using metadata which are used in defining a study program (a learning path) for each pupil and for a chosen by him section. Verifying objects (materials) consists of test queries, which are also described using metadata. Test queries are grouped into verifying tests. Both, issues and queries are stores in a form of XML files. Three levels of difficulties are defined for basic issues: • Easy (for beginners) – simple subjects are easily presented, with a number of examples, etc., • Medium (for intermediate pupils) – a given subject is presented in a compact way to avoid pupils become bored, • Hard (for advanced pupils) – simple parts are omitted. Following metadata are defined for basic issues: keywords, importance (five levels scale), option required (true, false). Three types of connections between issues are defined: precondition (one issue is necessary before another), structural connection (introduction, extension, summary), sequence connections (section number to which a given issue belongs, lecture number, and issue number). Each example is connected exactly with one basic issue, but a number of examples can refer to a given basic issue. Examples are included into a learning path only if the Learning Assistant decides that there is such a need. A test query is an atomic element of verifying objects (materials). Each query concerns exactly one basic issue. Queries are described by its difficulty level (five levels scale) and its type:

Reports Tests

Core

Information

Lectures

AI Assistant

SOM

Dictionary

• knowledge – enumerate..., when..., etc..., • understanding – e.g. a query concerns a concept in a new formulation,

Figure 1. Interconnections between packages in WebTeacher system

• application – calculate..., assign..., • problem solving – synthesis or evaluation, selection of solution from a set of solutions, etc.

The connections between didactic materials (objects) play an important role in the individual learning path generation. Therefore a number of metadata describing didactic materials and connections between them as well as metadata describing pupils are proposed in WebTeacher. WebTeacher defines two kinds of didactic materials – learning materials (learning objects) and verifying materials (verifying objects). The first group – the learning – has constant structure, the users see it as a e-Book in which the learning objects are structured similarly to written books. It means that e-Book consists of a number of sections. Each section consists of a number of lectures but each lecture consists of a number of issues. E-Books can contain in-

The level of knowledge (skills) of a pupil is described by connections: know, understand, is able to use. The level of a given pupil skill is evaluated by inference mechanism of Learning Assistant.

3 Learning Assistant – personalization in elearning systems Learning Assistant is the module responsible for the learning process personalizing. It helps the users to pass

310

over the learning contents from the selected area by creating an individual learning path for each pupil. The kind and scope of the learning content should depend on the pupil’s need, skill and his prior knowledge. The assistant should take into account also speed of learning and ability of creative using the possessed knowledge. The personalization module analyzes a behavior of groups of pupils, and looks for the regularities which can be useful for personalization of the other (new) pupils. We propose a solution based on an agent architecture and a SOM neural network.

3.1

that a human being has seven types of mutually independent intelligences, but one influences the others. They are: linguistic, logical-mathematical, musical, bodily-kinesthetic, spatial, interpersonal and intrapersonal. Scientists distinguish four types of style of thinking: concrete sequential, concrete nonlinear, abstract nonlinear, and abstract sequential. The above opinion given by didactics and psychologists suggest that pupils with similar levels of adequate types of intelligence and similar cognitive processes should reveal similarities in their learning processes, i.e., gaining new knowledge [7, 9]. Summing up, groups of pupils with similar psychophysical predispositions and similar cognitive processes should reveal similarities in the learning processes. It is a basis for the group personalization. The problem is how we can measure these specific predispositions of pupils. We have introduced an initial (entry) test for each new pupil. The test consists of two parts. The first part is a test proposed by [9] (http://www.bion.pl/testy/test-stylmyslenia.htm) – it describes the pupil thinking styles. The second part is a set of logical puzzles – they were developed on the basis of popular Intelligence tests. Answers for the questions contained in this test allow to evaluate the levels of five types of skill: logical thinking, spatial imagination, language skill, perceptiveness, and calculus skill. Analysis of fulfilled test allows the system to define the coefficients for styles of thinking and levels of skill. This nine-dimensional vector characterizes each pupil. Having the data characterizing pupils we have used a SOM neural network to define groups of similar pupils. Our system can use three neighborhood functions (Buble, Gausian, ’Mexican hat’) and three functions of the learning coefficient modification (linear, hyperbolic, exponential). Next phase of group personalization is searching the patterns in particular group users. Two elements are used for this task: difficult questions in the verifying tests i.e., questions with wrong pupils answers, and the frequent keywords, i.e., keywords that pupils used them during learning process. We have defined a number of coefficients:

Functionalities of Learning Assistant

Functionalities of Learning Assistant are contained in two packages: AI assistant is an implementation of assistant logic, and SOM is an external tool for training the neural network (see Fig. 1). A core of the AI assistant architecture is a programmable agent, which develops an individual plan for each pupil using an inference mechanism and metadata concerning learning contents and pupils. The initial test should define initial (a priori) knowledge of the candidate. During the learning process, the agent uses the information about previous courses and results of tests, therefore the agent has abilities of memorizing and knowledge actualization. The agent uses some characteristic of pupils gathered from registration forms (i.e., features that can influence the learning course) together with the previous results of pupils for grouping the similar pupils. The agent observes pupils during their activity and tries to discover some regularities for pupils classified to the same group. If such regularities are discovered, we can assume that the new pupil classified to the same group will behave in similar way. This information can be very useful for a learning path planning task. Searching the group of similar students and patterns of their behavior is not an easy task, therefore we have used a SOM neural networks that support inference mechanism of the agent by clustering pupils with similar characteristics.

3.2

Personalization

• Ωs – a frequency of keyword s, it is a ratio of a number of searching with using keyword s in group G to a number of all searching with using keywords in group G,

Learning Assistant uses two kinds of information when it generates a learning path for a given pupil: individual information about this pupil and regularities discovered in a group to which the pupil is classified. We distinguish individual personalization – it is built on the basis of individual data about the pupil, and group personalization – it uses patterns about the group (cluster) of pupils. According to psychology, predispositions and functionality of cognitive processes of pupils are the crucial factor in a process of knew knowledge gaining. Each pupil has a specific set of skills, specific way of thinking and perceptions [3, 7]. Howard Gardner in ”Multiple intelligences and education” (http://www.infed.org/thinkers/gardner.htm) writes

• Ψs – a popularity of keyword s, it is a ratio of a number of pupils in group G which used keyword s to a number of all pupils in group G, • Ωp – a frequency of wrong answers to question p, it is a ratio of a number of wrong answers to question p given by pupils of G to a number of all answers to question p by pupils of G,

311

• Ψp – a popularity of question p, it is a ratio of a number of pupils in group G who have answered to question p to a number of all pupils in group G.

3.3

k THEN calculate f requencyof searchingkeywordk and popularityof keywordk. The next two rules modify difficulties of verifying questions. 1. IF pupil U answered well to question p THEN diminish a difficulty of p . 2. IF pupil U answered wrongly to question p THEN enlarge a difficulty of p . The second group of rules (RS) consists of four types of rules. The first type encloses rules adding to the course issues connected with these questions, to that pupil answered wrongly. 1. IF pupil U answered well to question p connected with issue Z and p is knowledge type THEN add to the course of U issue Z on the adequate level for U . 2. IF pupil U answered wrongly to question p connected with issue Z and p is knowledge type and U does not know issue Z THEN add to the course of U issue Z on the adequate level for U and add to the course of U an example of Z. 3. IF pupil U answered wrongly to question p connected with issue Z and p is knowledge type and U knows issue Z THEN add to the course of U an example of issue Z. The second type of RS add to the course issues connected with these questions, to which a given pupil answered wrongly. They are similar to above. The third type of RS can add to the course issues in case when the pupil’s knowledge concerning these issues have not be verified yet. The fourth type of RS use a profile of the group to which the given pupil belongs. The rules from the third group (RM) actualize a plan of course, accordingly to the learning progress of the pupil. Modifiers control the progress using short tests. These rules cannot remove from the course any issue that has not been presented to the pupil yet. But they can add to the course issues from behind the section selected by the pupil.

Knowledge Base

Learning Assitent has one goal – to support users in their learning process by selection a scope and a type of didactic materials. Knowledge Base is a set of rules useful during plans generation and modification. The learning program generated for learning a given section we call simply course or course plan. The rules allow for actualization of some information concerning pupils and didactic materials. Each rule has a form: IF premise THEN conclusion or IF premise THEN action. All rules are divided into three groups, taking into account the goal of using them. The first group of rules are metadata modifiers (RMt) – they modify metadata of didactic materials, groups of pupils, and pupils. The second one are selectors (RS) – they are used for plans generation using metadata concerning pupils, didactic materials and connections between them. The third group of rules are modifiers (RM) – they use also results of verifying tests. The first group of rules (RMt) contains three types of rules. Six rules concern a level of knowledge: 1. IF pupil U answered to question p connected with issue Z and p is knowledge type question THEN enlarge a level of U knowing Z. 2. IF pupil U answered to question p connected with issue Z and p is understanding type THEN enlarge a level of U knowing Z and enlarge a level of U understand Z. 3. IF pupil U answered to question p connected with issue Z and p is application type THEN enlarge a level of U knowing Z and enlarge a level of U understand Z and enlarge a level of U can apply Z. The next three rules are similar, but they concern the wrong answers and diminishing of appropriate levels. Enlarge level is realized as

3.4

enlargelevel = level ∗ 0.64 + 0.4.

Performance of Learning Assistant

At the beginning, a pupil selects a section for learning (selection section). The system checks if the pupil has open course which is not finished yet (exists plan). If yes, the system starts to present next steps of the course (next step), see Fig. 2. If not, the pupil is asked to select a level of learning (select level) which he wants, and the process of course plan generation starts. A new plan of course is generated on the basis of the final test concerning the finished by the pupil courses from the same section (has history) or, for a new pupil who wants to start learning the current section, on the basis of initial (entry) test (generate preTest, execution preTest). This entry test (preTest) evaluates an initial knowledge of the pupil. The default period of considered

diminish level is realized as Diminishlevel = level ∗ 0.4. These rules cause that the answer to the last question plays a big role. The system puts the stress on the learning progress of pupils. The next two rules produce actions that calculate wrongly answers and searching materials with using keywords: 1. IF pupil U belonging to group G answered to question p THEN calculate f requencyof wronganswerstop and popularityof questionp. 2. IF pupil U belonging to group G has searched keyword

312

time (history) is three months, but a teacher can change this parameter. In course generation process (generate Course) the RS are used. After generation a plan of course, Learning Assistant presents the pupil issues from the sequence of issues (next Step). The course always contains a test after an issue. After each test the plan of course is modified. When the whole course is realized (empty Course), the final test is generated and presented to the pupil (generate endTest, present endTest). On the basis of results of this test, the knowledge of the agent is actualized (update KB) and the evaluation of the pupil knowledge is calculated (calculate note). At the end, the final note is presented together with congratulations of finishing the course (congratulations).

Computer Science at Wroclaw University of Technology. The students were encouraged to attend in this experiment. The system was filled with didactic materials concerning Theory of logical structures. The best 24 students were involved in didactic materials preparation. In the next phase, a group of students started learning process with using the developed system. The group was divided into two subgroups: standard subgroup consists with 36 students and the experimental subgroup with 28 students. Students from the standard group learned without Learning Assistant. It means that they decided what didactic material and in what sequence they learned. For the students from the experimental group, Learning Assistant was responsible about a learning path. Students from both subgroups have learned the same area of knowledge. The main aim of the experiment was to verify efficiency of developed Learning Assistant and its possibilities to support learning process. The results of this experiment can be a little deceitful because the students were stimulated by their lecturer to take a part in the experiment. The didactic materials were not very high quality. But we have observed that the average notes at the end of learning time were higher in the experimental subgroup – 50.8% and 58.3% respectively. A standard deviation in the standard subgroup was higher (23.3%) than in the experimental subgroup (15.2%). It suggests that Learning Assistant has brought good results especially for the lower level students. But outstanding students achieve better results when they work independently, searching interesting for them information. At the end of our experiment, students have been asked to fill anonymously a query-sheet. A number from 0 to 10 points could be the indicated as an answer for a part of questions that concerned the system functionalities, some other questions were binary valued (yes/no). Concerning Learning Assistant, the results are following: 85% of students evaluates positively the courses contained in the system; 80% of students declared willingness to use such a system for standalone learning, 66% of students evaluated the initial test verifying their basic, initial knowledge as a good solution, almost 90% of students think that the wider explanations of reasons of repetition/adding a given issue to the learning path would be helpful for them. Taking into account that the system is only a prototype, and that didactic material was prepared not by the very good lecturers but by the best students, the obtained results are quite satisfying. We can suppose that the system could receive much better evaluations after improving only didactic material.

Start Selection section

exists plan ? Yes next step

Yes update KB

No slelect level

after test ? has history ?

No

Yes

chose step: issue or test modification course

No

empty Course ? No

Yes

generate preTest

generate endTest execute preTest present endTest update KB update KB calculate note

generate Course

congratulations Stop

Figure 2. A schema of Learning Assistant

4 Testing the system

5 Summary

The proposed intelligent agent, called Learning Assistant was implemented as a part of WebTeacher system. An experiment was conducted with the second year students of

The paper presents a prototype e-learning system in which the intelligent Learning Assistant support the learning process. We have tried to solve the problem of assigning

313

the new pupil to the group of similar in the sense of learning skill pupils. We are convinced that personalization of learning courses (paths) is the most important feature of the e-learning processes. Therefore the individual personalization was important part of our work. The proposed rules can be tuned or changed as we gain experience. The experiment, despite of its weakness, reveals some advantages of our approach to personification the learning process automatically. The experiment, however imperfect, it indicates the directions of future work. The first is enrichment of the system in additional commentaries. The user should receive the explanation why some additional issues are added to his learning path. The learning progress should be better commented – the system should be more a virtual teacher. Similar conclusion can be drawn from other experiments with distance learning – Learning dyslexic children by enjoying oneself with computers [12]. Relatively high interaction with computer causes the learning less boring task. An agent (animated character) that is able to inform pupils about news/innovations in the system, motivate pupils, etc. should be desirable functionalities in the system, especially for younger pupils [12]. Using data mining methods for discovering new knowledge about the skills, preferences and quality of the learning process seems to be very interesting area of further study. Independence of personalization mechanism from the area and scope of didactic materials allow for using the system in different educational institutions as a tool supporting education process.

[9] K. Gozdek-Michaelis. Grow up your brilliant mind. In Polish), FILAR, Warszawa, 2005. [10] G.-J. Hwang. Development of an intelligent testing and diagnostic system on computer networks. In Proceedings of the National Science Council of ROC(D), pages 1–9, Taiwan, 1999. [11] G.-J. Hwang. A conceptual map model for developing intelligent tutoring systems. Comput. Educ., 40(3), 2003. [12] H. Kwasnicka. Learning dyslexic children by enjoying oneself with computers. In (In Polish), e-University – methods and tools, pages 45–64, Rzeszow, Poland, 2002. the University of Information Technology and Management in Rzeszow. [13] R. McAleese. The knowledge arena as an extension to the concept map: Reflection in action. Interactive Learning Environments, 6(3):251–272. [14] M. Podbielski. A computer network system assisted learning process. (In Polish). Master Thesis, Wroclaw University of Technology, Wroclaw, 2005. [15] D. Salisbury and P. F. Merrill. Research on drill and practice strategies. ournal of Computer-Based Instruction, 11(1):19– 21. [16] C.-T. Sun and C. Chou. Experiencing coral: design and implementation of distantcooperative learning. IEEE Transactions on Education, 39(3), 1996. [17] D. Swierk. E- learning - a great tool of modern education. (In Polish) 2003. [18] D. Szul. An agent assisted the learning process i e-learning system. (In Polish). Master Thesis, Wroclaw University of Technology, Wroclaw, 2005.

References [1] R. Agarwal, A. Deo, and S. Das. Intelligent agents in elearning. SIGSOFT Softw. Eng. Notes, 29(2):1–1, 2004. [2] T.-W. Chan, C.-W. Hue, C. Y. Chou, and O. J. L. Tzeng. Four spaces of network learning models. August 2000. [3] T. Dartnal. Artificial Intelligence and Creativity. Kluver Academic Publishers, The Netherlands, 1994. [4] P. Dolog, R. Gavriloaie, W. Nejdl, and J. Brase. Integrating adaptive hypermedia techniques and open rdf-based environments. In Proc. of the Twelfth International World Wide Web Conference, Hungary, 2003. [5] P. Dolog, N. Henze, W. Nejdl, and M. Sintek. Personalization in distributed e-learning environments. In WWW Alt. ’04: Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, pages 170– 179, New York, NY, USA, 2004. ACM. [6] P. Dolog and W. Nejdl. Challenges and benefits of the semantic web for user modelling. In Proc. of Workshop on Adaptive Hypermedia and Adaptive Web-Based Systems at WWW, Hungary, 2003. [7] M. Gerd. Psychology of Education. A practical textbook for teachers. In Polish), Psychological Editions, Gdansk, 2002. [8] J. Gooch. They blazed the trail for distance education. http://www.uwex.edu/disted/gooch.htm 1998.

314