E-Learning Systems with Artificial Intelligence in

E-Learning Systems with Artificial Intelligence in Engineering Wojciech Kacalak and Maciej Majewski Koszalin University of Technology, Faculty of Mechanical Engineering Raclawicka 15-17, 75-620 Koszalin, Poland [email protected], [email protected]

Abstract. This paper presents a new concept of intelligent e-learning systems with intelligent two-way speech communication between an elearning system and the user. The system uses intelligent methods for analysis, evaluation and assessment of user knowledge and skills as well as e-learning process control, supervision and optimization. Developed as a prototype for mobile technologies, the communication system by speech and a natural language between the intelligent e-learning system and external users consists of intelligent mechanisms for biometric user identification, speech recognition, word and sentence recognition, sentence meaning analysis, and user reaction assessment. Also discussed are selected problems of the new concept of intelligent e-learning systems using intelligent speech communication. The discussion focuses on recognition and evaluation of spoken natural language sentences with use of hybrid neural networks. Keywords: E-learning system; User-computer interaction; Speech interface; Artificial intelligence; Mobile technology; Cybernetics.

1

Introduction

For the purposes of this research, mobile technologies can be understood as a classification of all tasks in which receiving and passing of data, observation, supervision and control of processes, decision taking, passing of rights, access to means, initiation of undertakings and tasks, as well as prediction of results, can take place without limitations involving movement and location of tasks and computer devices. Mobile technologies aided by artificial intelligence methods produce higher quality, especially when it is necessary to use the expert’s assistance regarding distant effects and decision connections, heuristic circumstances of decisions, sudden changes of conditions, while the usage of this knowledge can take place via remote communication. Building and applying intelligent personal computer and remote mobile systems of realization, control, supervision and optimization of distance learning processes which communicate users with an e-learning system by speech and a natural language, is an important goal. These systems enable a remote supervision of the e-learning process quality using spoken commands in a natural D.-S. Huang et al. (Eds.): ICIC 2009, LNCS 5754, pp. 918–927, 2009. c Springer-Verlag Berlin Heidelberg 2009

E-Learning Systems with Artificial Intelligence in Engineering

919

language and allow to achieve a high quality e-learning process. It is very significant for development of new effective and flexible e-learning systems. It can also contribute for increase of efficiency and decrease of costs of e-learning processes. The system for remote realization, control, supervision and optimization is an innovative solution making it possible to exploit better the e-learning methods used nowadays. It also allows for more complete advantages of applied e-learning processes. The presented solution can be included to the attempts of creating the standard of intelligent personal computer and mobile systems for realization, control, supervision and optimization of e-learning processes using two-way communication by speech and a natural language between the e-learning system and the user for distance learning systems. This subject area will soon become one of the most important for the technological development of e-learning systems.

2

The State of the Art

Speech communication between the e-learning system and the user involves spoken language understanding which is composed of speech recognition and natural language understanding. Many sites in USA, Canada, Europe, China and Japan have been researching spoken language understanding systems. The recent advances in development of prototypes of such systems are described in articles in [1,5,7]. Vocabularies for these systems are usually about tens of thousands of words, and the speech and language are spontaneous. The speech recognition utterance error rates in the 2009 benchmarks was about 5% to 15%. The utterance understanding error rates range from 6% to 41%. These error rates may be compatible with many potential applications. The integration of speech recognition and natural language in applications is faced with many of the same challenges that each of the components face: accuracy, robustness, portability, speed, and size. The integration also gives rise to some new challenges which include: integration strategies, coordination of understanding components with system outputs, the effective use in natural language of a new source of information from speech recognition, and the handling of spontaneous speech effects. With few exceptions, current research in spoken language systems has focused on the understanding of spoken input. However, many if not most potential applications involve a collaboration between the human and the computer. In many cases, spoken language output is an appropriate means of communication that may or may not be taken advantage of, because of lack of coordination of understanding components with system outputs. This paper proposes an approach to deal with the above mentioned problem. As the Internet gains wide popularity around the world, e-learning is taken by the learners as an important study aid. E-learning systems offer new possibilities in learning, because a user can get immediate feedback on solutions to problems, learning paths can be individualized, etc. On-line learning is a growing business. The number of organizations working on online learning and the number of courses available on the Internet is growing rapidly. E-learning creates a tremendous opportunity for specific institutions and training organizations to

920

W. Kacalak and M. Majewski

provide on-demand education and training via virtual learning environments. At present, a lot of e-learning tools with varying functionality and purposes exist [2,3,4]. E-learning is an alternative concept to the traditional tutoring system. E-learning forms the revolutionary and new way to empower a workforce with the necessary skills and knowledge [6]. Towards this goal, various e-learning systems have been developed during the last years, however, most of them form old-fashioned applications, missing functionalities like sophisticated capabilities which could be delivered with use of artificial intelligence methods.

3

A New Concept of Intelligent E-Learning Systems

The new concept involves e-learning systems which use artificial intelligence methods and are equipped with intelligent two-way speech communication between the e-learning system and the user (fig. 1). According to the new concept, the e-learning system contains intelligent methods for analysis, evaluation and assessment of the user’s knowledge and skills as well as e-learning process control, supervision and optimization. The complete intelligent e-learning system through hybrid artificial neural networks is equipped with an intelligent twoway voice communication system which can optionally use mobile technology (fig. 2). The communication system by speech and a natural language between the intelligent e-learning system and its users consists of intelligent mechanisms of biometric user identification, speech recognition, word and sentence recognition, sentence meaning analysis, and user reaction assessment. The intelligent Any distance from e-learning system

Spoken sentences in natural language

INTELLIGENT TWO-WAY SPEECH COMMUNICATION LAYER THROUGH ARTIFICIAL NEURAL NETWORKS User identification

USER

Speech recognition Voice messages

User’s errors

Word recognition Sentence recognition Sentence meaning analysis Reaction assessment

User’s utterances or answers representing knowledge

INTELLIGENT E-LEARNING SYSTEM THROUGH ARTIFICIAL NEURAL NETWORKS

User’s command: e-learning process parameters, progress decision Chosen level of e-learning supervision automation

Analysis, evaluation and assessment of user knowledge&skills

Examination, course, results Process information

Fig. 1. Concept of an intelligent e-learning system

E-learning process control, supervision and optimization

E-Learning Systems with Artificial Intelligence in Engineering E-learning system communication with mobile technology

E-learning process control, supervision and optimization

Analysis, evaluation and assessment of user knowledge&skills

INTELLIGENT E-LEARNING SYSTEM THROUGH ARTIFICIAL NEURAL NETWORKS

User-system communication cycle

Processing text to voice messages

INTELLIGENT TWO-WAY SPEECH COMMUNICATION THROUGH ARTIFICIAL NEURAL NETWORKS User of e-learning system

User biometric identification and authentication

Mobile technology communication with Tablet PC

Processing spoken sentences to text

921

Mobile technology communication with e-learning system

User sentence meaning analysis Tablet PC communication with mobile technology

Fig. 2. E-learning system for mobile technology

e-learning system (fig. 3A) can be developed for personal computers as well as various mobile technology devices (fig. 3B). The advantages of intelligent e-learning systems using intelligent two-way communication by speech and a natural language between the e-learning system and the user include the following: – More robustness against user’s errors and more efficient realization, control, supervision and optimization of the e-learning process with the chosen level of supervision automation. – Improvement of the co-operation between a user and an e-learning system in respect to the richness of communication. – Achievement of a higher level of organization of a distance learning process that is equipped with an intelligent two-way speech communication system, which is relevant for its efficiency, flexibility, ergonomics and economy. – E-learning decision and optimization systems can be remote elements with regard to an e-learning system. The complete intelligent e-learning system is shown in abbreviated form on fig. 4. Upon biometric identification and authentication of the user by the mobile system, the automatic recognition process of the user’s utterance or answer in a natural language is performed. The sentences produced in continuous speech are processed to text and numeric values with the module for processing spoken sentences to text on a Tablet PC. The speech recognition engine is a continuous density mixture Gaussian Hidden Markov Model system. The speech signal is transmitted from the PDA to the Tablet PC and after a successful utterance recognition, a text sentence in a natural language is sent to the PDA. Individual words treated here as isolated components of the text are processed with

922


(A)

(B)

ANALYSIS, EVALUATION AND ASSESSMENT OF USER KNOWLEDGE & SKILLS Module of resources of knowledge acquisition content and knowledge testing content

Module of estimation of user’s utterance or answer using neural networks

Knowledge remembering level assessment module using hybrid neural networks

Knowledge remembering state evaluation module using hybrid neural networks

Module of diagnosing knowledge understanding state using probabilistic neural networks

Sentence recognition module using neural networks

MEANING ANALYSIS OF USER UTTERANCE OR ANSWER

Module for analyzing knowledge understanding level using probabilistic neural networks

Module of error indication in utterance or answer

Sentence analysis module Sentence syntax analysis module

Answer correctness assessment module

Word recognition module using neural networks

Module of grade assessment of affiliation of answers to correct answer category

Word analysis module

Answer correction module

Module for processing spoken sentences to text Spoken sentence archive module

User authentication module User identification module using neural networks

E-LEARNING SYSTEM COMMAND EXECUTION

Error control module

SPEECH COMMUNICATION

USER IDENTIFICATION

USER'S SPOKEN SENTENCES IN NATURAL LANGUAGE

Letter string analysis module

Module for signaling user commands accepted for execution User reaction assessment module Command execution module E-learning process control, supervision and optimization

VOICE MESSAGES FROM THE E-LEARNING SYSTEM TO THE USER Module for processing text to voice messages

Module of distribution of e-learning content


Fig. 3. (A) Complete intelligent e-learning system; (B) Prototype of the intelligent e-learning system for mobile technology: an example question and answer by speech and a natural language and the user knowledge evaluation and assessment

the letter string analysis module. The letters grouped in segments are then processed by the word analysis module. The analyzed word segments are inputs of the evolvable fuzzy neural network for recognizing words. The network uses a training file containing also words and is trained to recognize words as sentence components, with words represented by output neurons. In the next stage, the recognized words are transferred to the sentence syntax analysis module which uses sentence segment patterns. It analyses and divides sentences into segments with regards to meaning, and codes sentences as vectors. They are sent to the sentence segment analysis module using Hamming neural networks equipped with sentence segment patterns. The sentence become inputs of the sentence recognition module. The module uses a 3-layer evolvable fuzzy Hamming neural network either to recognize the sentence and find its meaning or else it fails to recognize it (fig. 5). The neural network of this module uses a training file containing patterns of possible meaningful sentences. The word and sentence recognition modules contain Hamming neural networks which feature evolvable architectures and learning (fig. 6).


923

Communication transmitter from PDA to E-learning system User’s recognized meaningful sentences

E-learning content information

SENTENCE SEGMENT ANALYSIS MODULE USING NEURAL NETWORKS Words in segments as sentence components SENTENCE SYNTAX ANALYSIS MODULE Recognized words WORD RECOGNITION MODULE USING EVOLVABLE FUZZY NEURAL NETWORKS Analyzed word segments

ANN learning file Words as sentence components

WORD ANALYSIS MODULE USING NEURAL NETWORKS Letters in segments as word components LETTER STRING ANALYSIS MODULE Text sentences in natural language Tablet PC transmitter to PDA

ERROR CONTROL MODULE

MODULE FOR PROCESSING SPOKEN SENTENCES TO TEXT Speech recognition engine on Tablet PC

Vocabulary

PDA transmitter to Tablet PC

MODULE OF ESTIMATION OF UTTERANCE OR ANSWER USING NEURAL NETWORKS

KNOWLEDGE REMEMBERING LEVEL ASSESSMENT MODULE USING HYBRID NEURAL NETWORKS

MODULE FOR ANALYZING KNOWLEDGE UNDERSTANDING LEVEL USING PROBABILISTIC NEURAL NETWORKS

LEARNING CONTEXT

KNOWLEDGE RANGE AND CRITERIA (STANDARDS)

KNOWLEDGE REMEMBERING STATE EVALUATION MODULE USING HYBRID NEURAL NETWORKS

MODULE FOR DIAGNOSING KNOWLEDGE UNDERSTANDING STATE USING PROBABILISTIC NEURAL NETWORKS

MODULE OF ERROR INDICATION IN UTTERANCE OR ANSWER

ANSWER CORRECTNESS ASSESSMENT MODULE

LEARNER CHARACTERISTICS MODULE OF ESTIMATION OF LEARNING EFFECTIVENESS AND TOPIC RELEVANCE

ANALYSIS, EVALUATION AND ASSESSMENT OF USER KNOWLEDGE & SKILLS

Meaningful sentences

Analyzed sentence segments

MODULE OF RESOURCES OF KNOWLEDGE ACQUISITION CONTENT AND KNOWLEDGE TESTING CONTENT

Evaluation and assessment of user knowledge & skills

ANN learning files

MEANING ANALYSIS OF USER UTTERANCE OR ANSWER

SENTENCE RECOGNITION MODULE USING EVOLVABLE FUZZY NEURAL NETWORKS

MODULE OF GRADE ASSESSMENT OF AFFILIATION OF ANSWERS TO CORRECT ANSWER CATEGORY

ANSWER CORRECTION MODULE

SPOKEN SENTENCE ARCHIVE MODULE Speech signal Communication Cycle between the User and the E-learning system

REACTION TIME ANALYSIS MODULE

USER REACTION ASSESSMENT MODULE USING FUZZY LOGIC

BIOMETRIC USER IDENTIFICATION MODULE USING NEURAL NETWORKS

COMMAND EXECUTION MODULE

Password input characteristics recognition Signature pattern recognition Knowledge pattern recognition Personality recognition

USER Voice message Speakers Speech signal MODULE FOR PROCESSING TEXT TO VOICE MESSAGES Speech synthesis engine


User’s knowledge

User’s password

E-learning progress decisions MODULE OF E-LEARNING PROCESS CONTROL AND SUPERVISION USING NEURAL NETWORKS E-learning process parameters

PROCESS STATE SIGNALING MODULE

E-LEARNING SYSTEM CONFIGURATION E-learning process details

MODULE FOR SIGNALING USER’S LEARNING RESULTS

E-LEARNING PROCESS OPTIMIZATION MODULE USING NEURAL NETWORKS Process optimization

ADDITIONAL INFORMATION MODULE

MODULE OF DISTRIBUTION OF E-LEARNING CONTENT

E-learning system communication transmitter to PDA

Fig. 4. Architecture of the complete intelligent e-learning system

E-LEARNING SYSTEM KERNEL

USER AUTHENTICATION MODULE

COMMAND EXECUTION

MODULE FOR SIGNALING USER COMMANDS ACCEPTED FOR EXECUTION

Microphone User’s spoken sentence

Sentence syntax analysis and coding

924


[...] ADAPTIVE ALGORITHM ASSOCIATIVE BIDIRECTIONAL CRITIC DISCRETE DRIVER

SENTENCE SYNTAX ANALYSIS MODULE z: seg.1, seg.2, seg.3, seg4.

SENTENCE SEGMENT ANALYSIS MODULE

[...]

[...] HEURISTIC IS [...] LEARNING MEMORY METHOD [...] REINFORCEMENT [...]

algorithm discrete bidirectional associative memory WORD RECOGNITION MODULE WORD ANALYSIS MODULE algori-thm discret-ely direction-al associat-ed memor-ies LETTER STRING ANALYSIS MODULE the algorithm seems correct referring to discretely two directional associated memories

sentence recognition cycle

z : algorithm discrete bidirectional wi : method is discrete bidirectional

associative memory associative memory

SENTENCE RECOGNITION MODULE

Hamming distance DH= 2 to the class representing sentence: ‘method is discrete bidirectional associative memory’

Hamming distance DH= 9 to the class representing sentence: ‘method is driverreinforcement learning’



[...]

[...]



Fig. 5. Illustration of a cycle of meaning analysis of the user’s utterance or answer

After the meaning analysis process of the user’s utterance or answer, their recognized meaningful sentences are processed for analysis, evaluation and assessment of the user’s knowledge and skills. The e-learning system estimates the user’s utterances or answers, assesses the knowledge remembering level, analysis the knowledge understanding level, evaluates the knowledge remembering state, and consecutively diagnoses the knowledge understanding state. The intelligent methods are applied for these tasks using hybrid and probabilistic neural networks. The system also indicates errors in the utterances or answers, assesses the answer correctness, estimates learning effectiveness and topic relevance, assigns the answers to the correct categories, and makes corrections. The command execution tasks include assessment of the reaction time and reactions of the user, and control of the e-learning process with regards to the definition of progress decisions as well as parameters for the e-learning process. The process supervision is performed using neural networks. The e-learning system kernel is composed of an e-learning system configuration module, modules for intelligent optimization of the e-learning process, as well as signaling the process state and user’s learning results. The speech communication to the user involves processing text to voice messages.

E-Learning Systems with Artificial Intelligence in Engineering Classification to sentence classes (Set of possible meaningful sentence patterns) C1

C2

N31

Output: binary image of the recognized sentence

C3

N32

N3p

[...]

Output layer

a1

a2

a3

N22

N21

Evolvable learning of ANN


Cp

N33

ap Recursive layer MAXNET


N2p

N23

Evolutionary optimization: z, W 1

1

b1

b3

N12

N11

Mpp

1

1

b2

bp

Chromosome W=weights

N1p

N13

Binary distance layer

N02

N03

N04

N05

x1

x2

x3

x4

x5

Fuzzy N06 connectionist structure

p=number of patterns

Minimum Hamming distance fitness function F(G)=DHmin(z,wi)

Wpn

N01

925

Genotype G ={Chz, ChW }


N0n

Chromosome z=input pattern

[...]

b

x6

xn

xi={-1,+1}

a

WORD b

WORD 1 WORD 2 WORD 3

Input of the net: n=a*b

a

WORD a

Sentence syntax analysis

4 3 2 1

WORD 4 WORD 3 WORD 2 WORD 1

Sentence segment analysis using neural networks

b


Evolvable architecture of ANN

Binary image of a sentence

a

Words into a sentence , 1

}

Output: images of the recognized words

WORD b

WORD 1 WORD 2 WORD 3

Bit = { 0 (for ANN: -1) Represented word class

ABCDEFGH I J KLMNOPQRSTUVWXYZ 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Classification to word classes (Set of word patterns as sentence components) C2

C1 N32

N31

a1

a2

N21

N11

ap

N2p

1

1

Chromosome Evolvable b2 W=weights architecture of ANN N12

bp N1p

Evolvable learning of ANN

x2

a

ABCDEFGH I J KLMNOPQRSTUVWXYZ

Image of recognized word ‘ASSOCIATIVE’

Mpp

p=number of patterns

Evolutionary optimization of parameters: z, W

Genotype G ={Chz, ChW } Input of the net: n=26*a

Fitness function F(G)=DHmin(z,wi)

b ABCDEFGH I J KLMNOPQRSTUVWXYZ

Wpn Chromosome z=input pattern

Fuzzy connectionist structure

xi={-1,+1} x1

N3p

N22

1

b1

Cp

A S S O C I A T I V E

xn

Binary images as isolated components of the text created by the word analysis module

a

1 2 3 4 5 6 7 8 9 10 11 12 13 14

A S S O C I A T

a

ABCDEFGH I J KLMNOPQRSTUVWXYZ

Binary image of letter string ‘ASSOCIAT’

Fig. 6. Evolvable fuzzy neural networks for word and sentence recognition

926


4

Experimental Results

The prototype of the intelligent communication system was developed for Windows Mobile for a PDA (Pocket PC) and a Tablet PC for Windows. As shown in Fig. 7A, the ability of the implemented evolvable fuzzy neural network to recognize a word depends on the number of letters of that word. For best performance, the neural network requires a minimum number of letters of each word being recognized as its input. The ability of the evolvable fuzzy neural network to recognize the sentence depends on the sentence length as shown in Fig. 7B. Similarly, for best sentence recognition, the neural network requires a certain minimum wordcount of the given sentence. (A)

(B)

Lmin = 0,86 * L 0,72

W min =1,13*W 0,61 6

8 17

7

10

11

5 8

4 5

3 2

2

3

6

9

12

15

16 Minimum number of words Wmin

Minimum number of letters Lmin

5 14

6 13

10

7

4

3

2

0

2

3

2

3

4

5

6

4

5

6

7

8

9

7

8

9

1

1

1

1

4

0 0

2

4

6

8

10

12

14

16

18

0

Number of letters of the word L

1

10

11

Number of words of the sentence W

Fig. 7. Experimental results: (A) Sensitivity of spoken word recognition: minimum number of letters of the word being recognized vs. number of word letters; (B) Sensitivity of spoken sentence meaning recognition: minimum number of words of the sentence being recognized vs. number of sentence component words

The experimental implementation of the intelligent e-learning system prototype allowed for an attempt of its evaluation. The testing has enabled to determine for the system the following attribute values [6]: – Suitability, expressing the degree of appropriateness of the system to the tasks which have to be accomplished. – Learnability, which conveys how easy it is for the user to learn the system and how rapidly the user can begin to work with it. – Error rate, which reflects the error ratio while working with the system.

5

Conclusions and Perspectives

A speech interface between users and e-learning systems using the natural language, is ideal because it is the most natural, flexible, efficient, and economical


927

form of human communication. Application of hybrid neural networks allows recognizing sentences of similar meanings but of different lexical and grammatical patterns, which will undoubtedly be the most important way of communication between humans and computer systems. This presented approach can be included to the methods of natural language processing. The condition of the effectiveness of the presented system is to equip it with mechanisms of meaning analysis of the user’s utterance or answer, as well as analysis, evaluation and assessment of the user’s knowledge and skills. In the e-learning systems, the condition of high quality communication between users and e-learning systems is the analysis of the e-learning software and process state before an utterance is performed by the user, and use of artificial intelligence for the analysis, evaluation and assessment of the answer or command. The developed flexible intelligent e-learning system can be extended for various applications. The experimental results of the proposed system show its promising performance for further development and experiments. The system described in this paper is a conceptually new approach to the problem of lack of coordination of understanding components with system outputs in a spoken language system, although it is an appropriate means of communication that may or may not be taken advantage of.

References 1. Ayres, T., Nolan, B.: Voice Activated Command and Control with Speech Recognition over WiFi. Science of Computer Programming 59(1-2), 109–126 (2006) 2. Govindasamy, T.: Successful Implementation of E-Learning. The Internet and Higher Education 4(3-4), 287–299 (2002) 3. Ismail, J.: The Design of an E-Learning System. The Internet and Higher Education 4(3-4), 329–336 (2002) 4. Larkin, M.: Handheld Use Increasing for E-Learning and Clinical Decision Making. The Lancet 361(9351), 93 (2003) 5. Paek, T., Chickering, D.M.: Improving Command and Control Speech Recognition on Mobile Devices: Using Predictive User Models for Language Modeling. User Modeling and User-Adapted Interaction 17(1-2), 93–117 (2007) 6. Rosenberg, M.J.: E-Learning: Strategies for Delivering Knowledge in the Digital Age. McGraw-Hill, New York (2001) 7. Wald, M., Bain, K.: Universal Access to Communication and Learning: the Role of Automatic Speech Recognition. Universal Access in the Information Society 6(4), 435–447 (2008)