E-Learning Systems with Artificial Intelligence in Engineering Wojciech Kacalak and Maciej Majewski Koszalin University of Technology, Faculty of Mechanical Engineering Raclawicka 15-17, 75-620 Koszalin, Poland
[email protected],
[email protected]
Abstract. This paper presents a new concept of intelligent e-learning systems with intelligent two-way speech communication between an elearning system and the user. The system uses intelligent methods for analysis, evaluation and assessment of user knowledge and skills as well as e-learning process control, supervision and optimization. Developed as a prototype for mobile technologies, the communication system by speech and a natural language between the intelligent e-learning system and external users consists of intelligent mechanisms for biometric user identification, speech recognition, word and sentence recognition, sentence meaning analysis, and user reaction assessment. Also discussed are selected problems of the new concept of intelligent e-learning systems using intelligent speech communication. The discussion focuses on recognition and evaluation of spoken natural language sentences with use of hybrid neural networks. Keywords: E-learning system; User-computer interaction; Speech interface; Artificial intelligence; Mobile technology; Cybernetics.
1
Introduction
For the purposes of this research, mobile technologies can be understood as a classification of all tasks in which receiving and passing of data, observation, supervision and control of processes, decision taking, passing of rights, access to means, initiation of undertakings and tasks, as well as prediction of results, can take place without limitations involving movement and location of tasks and computer devices. Mobile technologies aided by artificial intelligence methods produce higher quality, especially when it is necessary to use the expert’s assistance regarding distant effects and decision connections, heuristic circumstances of decisions, sudden changes of conditions, while the usage of this knowledge can take place via remote communication. Building and applying intelligent personal computer and remote mobile systems of realization, control, supervision and optimization of distance learning processes which communicate users with an e-learning system by speech and a natural language, is an important goal. These systems enable a remote supervision of the e-learning process quality using spoken commands in a natural D.-S. Huang et al. (Eds.): ICIC 2009, LNCS 5754, pp. 918–927, 2009. c Springer-Verlag Berlin Heidelberg 2009
E-Learning Systems with Artificial Intelligence in Engineering
919
language and allow to achieve a high quality e-learning process. It is very significant for development of new effective and flexible e-learning systems. It can also contribute for increase of efficiency and decrease of costs of e-learning processes. The system for remote realization, control, supervision and optimization is an innovative solution making it possible to exploit better the e-learning methods used nowadays. It also allows for more complete advantages of applied e-learning processes. The presented solution can be included to the attempts of creating the standard of intelligent personal computer and mobile systems for realization, control, supervision and optimization of e-learning processes using two-way communication by speech and a natural language between the e-learning system and the user for distance learning systems. This subject area will soon become one of the most important for the technological development of e-learning systems.
2
The State of the Art
Speech communication between the e-learning system and the user involves spoken language understanding which is composed of speech recognition and natural language understanding. Many sites in USA, Canada, Europe, China and Japan have been researching spoken language understanding systems. The recent advances in development of prototypes of such systems are described in articles in [1,5,7]. Vocabularies for these systems are usually about tens of thousands of words, and the speech and language are spontaneous. The speech recognition utterance error rates in the 2009 benchmarks was about 5% to 15%. The utterance understanding error rates range from 6% to 41%. These error rates may be compatible with many potential applications. The integration of speech recognition and natural language in applications is faced with many of the same challenges that each of the components face: accuracy, robustness, portability, speed, and size. The integration also gives rise to some new challenges which include: integration strategies, coordination of understanding components with system outputs, the effective use in natural language of a new source of information from speech recognition, and the handling of spontaneous speech effects. With few exceptions, current research in spoken language systems has focused on the understanding of spoken input. However, many if not most potential applications involve a collaboration between the human and the computer. In many cases, spoken language output is an appropriate means of communication that may or may not be taken advantage of, because of lack of coordination of understanding components with system outputs. This paper proposes an approach to deal with the above mentioned problem. As the Internet gains wide popularity around the world, e-learning is taken by the learners as an important study aid. E-learning systems offer new possibilities in learning, because a user can get immediate feedback on solutions to problems, learning paths can be individualized, etc. On-line learning is a growing business. The number of organizations working on online learning and the number of courses available on the Internet is growing rapidly. E-learning creates a tremendous opportunity for specific institutions and training organizations to
920
W. Kacalak and M. Majewski
provide on-demand education and training via virtual learning environments. At present, a lot of e-learning tools with varying functionality and purposes exist [2,3,4]. E-learning is an alternative concept to the traditional tutoring system. E-learning forms the revolutionary and new way to empower a workforce with the necessary skills and knowledge [6]. Towards this goal, various e-learning systems have been developed during the last years, however, most of them form old-fashioned applications, missing functionalities like sophisticated capabilities which could be delivered with use of artificial intelligence methods.
3
A New Concept of Intelligent E-Learning Systems
The new concept involves e-learning systems which use artificial intelligence methods and are equipped with intelligent two-way speech communication between the e-learning system and the user (fig. 1). According to the new concept, the e-learning system contains intelligent methods for analysis, evaluation and assessment of the user’s knowledge and skills as well as e-learning process control, supervision and optimization. The complete intelligent e-learning system through hybrid artificial neural networks is equipped with an intelligent twoway voice communication system which can optionally use mobile technology (fig. 2). The communication system by speech and a natural language between the intelligent e-learning system and its users consists of intelligent mechanisms of biometric user identification, speech recognition, word and sentence recognition, sentence meaning analysis, and user reaction assessment. The intelligent Any distance from e-learning system
Spoken sentences in natural language
INTELLIGENT TWO-WAY SPEECH COMMUNICATION LAYER THROUGH ARTIFICIAL NEURAL NETWORKS User identification
USER
Speech recognition Voice messages
User’s errors
Word recognition Sentence recognition Sentence meaning analysis Reaction assessment
User’s utterances or answers representing knowledge
INTELLIGENT E-LEARNING SYSTEM THROUGH ARTIFICIAL NEURAL NETWORKS
User’s command: e-learning process parameters, progress decision Chosen level of e-learning supervision automation
Analysis, evaluation and assessment of user knowledge&skills
Examination, course, results Process information
Fig. 1. Concept of an intelligent e-learning system
E-learning process control, supervision and optimization
E-Learning Systems with Artificial Intelligence in Engineering E-learning system communication with mobile technology
E-learning process control, supervision and optimization
Analysis, evaluation and assessment of user knowledge&skills
INTELLIGENT E-LEARNING SYSTEM THROUGH ARTIFICIAL NEURAL NETWORKS
User-system communication cycle
Processing text to voice messages
INTELLIGENT TWO-WAY SPEECH COMMUNICATION THROUGH ARTIFICIAL NEURAL NETWORKS User of e-learning system
User biometric identification and authentication
Mobile technology communication with Tablet PC
Processing spoken sentences to text
921
Mobile technology communication with e-learning system
User sentence meaning analysis Tablet PC communication with mobile technology
Fig. 2. E-learning system for mobile technology
e-learning system (fig. 3A) can be developed for personal computers as well as various mobile technology devices (fig. 3B). The advantages of intelligent e-learning systems using intelligent two-way communication by speech and a natural language between the e-learning system and the user include the following: – More robustness against user’s errors and more efficient realization, control, supervision and optimization of the e-learning process with the chosen level of supervision automation. – Improvement of the co-operation between a user and an e-learning system in respect to the richness of communication. – Achievement of a higher level of organization of a distance learning process that is equipped with an intelligent two-way speech communication system, which is relevant for its efficiency, flexibility, ergonomics and economy. – E-learning decision and optimization systems can be remote elements with regard to an e-learning system. The complete intelligent e-learning system is shown in abbreviated form on fig. 4. Upon biometric identification and authentication of the user by the mobile system, the automatic recognition process of the user’s utterance or answer in a natural language is performed. The sentences produced in continuous speech are processed to text and numeric values with the module for processing spoken sentences to text on a Tablet PC. The speech recognition engine is a continuous density mixture Gaussian Hidden Markov Model system. The speech signal is transmitted from the PDA to the Tablet PC and after a successful utterance recognition, a text sentence in a natural language is sent to the PDA. Individual words treated here as isolated components of the text are processed with
922
W. Kacalak and M. Majewski
(A)
(B)
ANALYSIS, EVALUATION AND ASSESSMENT OF USER KNOWLEDGE & SKILLS Module of resources of knowledge acquisition content and knowledge testing content
Module of estimation of user’s utterance or answer using neural networks
Knowledge remembering level assessment module using hybrid neural networks
Knowledge remembering state evaluation module using hybrid neural networks
Module of diagnosing knowledge understanding state using probabilistic neural networks
Sentence recognition module using neural networks
MEANING ANALYSIS OF USER UTTERANCE OR ANSWER
Module for analyzing knowledge understanding level using probabilistic neural networks
Module of error indication in utterance or answer
Sentence analysis module Sentence syntax analysis module
Answer correctness assessment module
Word recognition module using neural networks
Module of grade assessment of affiliation of answers to correct answer category
Word analysis module
Answer correction module
Module for processing spoken sentences to text Spoken sentence archive module
User authentication module User identification module using neural networks
E-LEARNING SYSTEM COMMAND EXECUTION
Error control module
SPEECH COMMUNICATION
USER IDENTIFICATION
USER'S SPOKEN SENTENCES IN NATURAL LANGUAGE
Letter string analysis module
Module for signaling user commands accepted for execution User reaction assessment module Command execution module E-learning process control, supervision and optimization
VOICE MESSAGES FROM THE E-LEARNING SYSTEM TO THE USER Module for processing text to voice messages
Module of distribution of e-learning content
SPEECH COMMUNICATION
Fig. 3. (A) Complete intelligent e-learning system; (B) Prototype of the intelligent e-learning system for mobile technology: an example question and answer by speech and a natural language and the user knowledge evaluation and assessment
the letter string analysis module. The letters grouped in segments are then processed by the word analysis module. The analyzed word segments are inputs of the evolvable fuzzy neural network for recognizing words. The network uses a training file containing also words and is trained to recognize words as sentence components, with words represented by output neurons. In the next stage, the recognized words are transferred to the sentence syntax analysis module which uses sentence segment patterns. It analyses and divides sentences into segments with regards to meaning, and codes sentences as vectors. They are sent to the sentence segment analysis module using Hamming neural networks equipped with sentence segment patterns. The sentence become inputs of the sentence recognition module. The module uses a 3-layer evolvable fuzzy Hamming neural network either to recognize the sentence and find its meaning or else it fails to recognize it (fig. 5). The neural network of this module uses a training file containing patterns of possible meaningful sentences. The word and sentence recognition modules contain Hamming neural networks which feature evolvable architectures and learning (fig. 6).
E-Learning Systems with Artificial Intelligence in Engineering
923
Communication transmitter from PDA to E-learning system User’s recognized meaningful sentences
E-learning content information
SENTENCE SEGMENT ANALYSIS MODULE USING NEURAL NETWORKS Words in segments as sentence components SENTENCE SYNTAX ANALYSIS MODULE Recognized words WORD RECOGNITION MODULE USING EVOLVABLE FUZZY NEURAL NETWORKS Analyzed word segments
ANN learning file Words as sentence components
WORD ANALYSIS MODULE USING NEURAL NETWORKS Letters in segments as word components LETTER STRING ANALYSIS MODULE Text sentences in natural language Tablet PC transmitter to PDA
ERROR CONTROL MODULE
MODULE FOR PROCESSING SPOKEN SENTENCES TO TEXT Speech recognition engine on Tablet PC
Vocabulary
PDA transmitter to Tablet PC
MODULE OF ESTIMATION OF UTTERANCE OR ANSWER USING NEURAL NETWORKS
KNOWLEDGE REMEMBERING LEVEL ASSESSMENT MODULE USING HYBRID NEURAL NETWORKS
MODULE FOR ANALYZING KNOWLEDGE UNDERSTANDING LEVEL USING PROBABILISTIC NEURAL NETWORKS
LEARNING CONTEXT
KNOWLEDGE RANGE AND CRITERIA (STANDARDS)
KNOWLEDGE REMEMBERING STATE EVALUATION MODULE USING HYBRID NEURAL NETWORKS
MODULE FOR DIAGNOSING KNOWLEDGE UNDERSTANDING STATE USING PROBABILISTIC NEURAL NETWORKS
MODULE OF ERROR INDICATION IN UTTERANCE OR ANSWER
ANSWER CORRECTNESS ASSESSMENT MODULE
LEARNER CHARACTERISTICS MODULE OF ESTIMATION OF LEARNING EFFECTIVENESS AND TOPIC RELEVANCE
ANALYSIS, EVALUATION AND ASSESSMENT OF USER KNOWLEDGE & SKILLS
Meaningful sentences
Analyzed sentence segments
MODULE OF RESOURCES OF KNOWLEDGE ACQUISITION CONTENT AND KNOWLEDGE TESTING CONTENT
Evaluation and assessment of user knowledge & skills
ANN learning files
MEANING ANALYSIS OF USER UTTERANCE OR ANSWER
SENTENCE RECOGNITION MODULE USING EVOLVABLE FUZZY NEURAL NETWORKS
MODULE OF GRADE ASSESSMENT OF AFFILIATION OF ANSWERS TO CORRECT ANSWER CATEGORY
ANSWER CORRECTION MODULE
SPOKEN SENTENCE ARCHIVE MODULE Speech signal Communication Cycle between the User and the E-learning system
REACTION TIME ANALYSIS MODULE
USER REACTION ASSESSMENT MODULE USING FUZZY LOGIC
BIOMETRIC USER IDENTIFICATION MODULE USING NEURAL NETWORKS
COMMAND EXECUTION MODULE
Password input characteristics recognition Signature pattern recognition Knowledge pattern recognition Personality recognition
USER Voice message Speakers Speech signal MODULE FOR PROCESSING TEXT TO VOICE MESSAGES Speech synthesis engine
SPEECH COMMUNICATION
User’s knowledge
User’s password
E-learning progress decisions MODULE OF E-LEARNING PROCESS CONTROL AND SUPERVISION USING NEURAL NETWORKS E-learning process parameters
PROCESS STATE SIGNALING MODULE
E-LEARNING SYSTEM CONFIGURATION E-learning process details
MODULE FOR SIGNALING USER’S LEARNING RESULTS
E-LEARNING PROCESS OPTIMIZATION MODULE USING NEURAL NETWORKS Process optimization
ADDITIONAL INFORMATION MODULE
MODULE OF DISTRIBUTION OF E-LEARNING CONTENT
E-learning system communication transmitter to PDA
Fig. 4. Architecture of the complete intelligent e-learning system
E-LEARNING SYSTEM KERNEL
USER AUTHENTICATION MODULE
COMMAND EXECUTION
MODULE FOR SIGNALING USER COMMANDS ACCEPTED FOR EXECUTION
Microphone User’s spoken sentence
Sentence syntax analysis and coding
924
W. Kacalak and M. Majewski
[...] ADAPTIVE ALGORITHM ASSOCIATIVE BIDIRECTIONAL CRITIC DISCRETE DRIVER
SENTENCE SYNTAX ANALYSIS MODULE z: seg.1, seg.2, seg.3, seg4.
SENTENCE SEGMENT ANALYSIS MODULE
[...]
[...] HEURISTIC IS [...] LEARNING MEMORY METHOD [...] REINFORCEMENT [...]
algorithm discrete bidirectional associative memory WORD RECOGNITION MODULE WORD ANALYSIS MODULE algori-thm discret-ely direction-al associat-ed memor-ies LETTER STRING ANALYSIS MODULE the algorithm seems correct referring to discretely two directional associated memories
sentence recognition cycle
z : algorithm discrete bidirectional wi : method is discrete bidirectional
associative memory associative memory
SENTENCE RECOGNITION MODULE
Hamming distance DH= 2 to the class representing sentence: ‘method is discrete bidirectional associative memory’
Hamming distance DH= 9 to the class representing sentence: ‘method is driverreinforcement learning’
[...] ADAPTIVE ALGORITHM ASSOCIATIVE BIDIRECTIONAL CRITIC DISCRETE DRIVER
[...] ADAPTIVE ALGORITHM ASSOCIATIVE BIDIRECTIONAL CRITIC DISCRETE DRIVER
[...]
[...]
[...] HEURISTIC IS [...] LEARNING MEMORY METHOD [...] REINFORCEMENT [...]
[...] HEURISTIC IS [...] LEARNING MEMORY METHOD [...] REINFORCEMENT [...]
Fig. 5. Illustration of a cycle of meaning analysis of the user’s utterance or answer
After the meaning analysis process of the user’s utterance or answer, their recognized meaningful sentences are processed for analysis, evaluation and assessment of the user’s knowledge and skills. The e-learning system estimates the user’s utterances or answers, assesses the knowledge remembering level, analysis the knowledge understanding level, evaluates the knowledge remembering state, and consecutively diagnoses the knowledge understanding state. The intelligent methods are applied for these tasks using hybrid and probabilistic neural networks. The system also indicates errors in the utterances or answers, assesses the answer correctness, estimates learning effectiveness and topic relevance, assigns the answers to the correct categories, and makes corrections. The command execution tasks include assessment of the reaction time and reactions of the user, and control of the e-learning process with regards to the definition of progress decisions as well as parameters for the e-learning process. The process supervision is performed using neural networks. The e-learning system kernel is composed of an e-learning system configuration module, modules for intelligent optimization of the e-learning process, as well as signaling the process state and user’s learning results. The speech communication to the user involves processing text to voice messages.
E-Learning Systems with Artificial Intelligence in Engineering Classification to sentence classes (Set of possible meaningful sentence patterns) C1
C2
N31
Output: binary image of the recognized sentence
C3
N32
N3p
[...]
Output layer
a1
a2
a3
N22
N21
Evolvable learning of ANN
[...] ADAPTIVE ALGORITHM ASSOCIATIVE BIDIRECTIONAL CRITIC DISCRETE DRIVER
Cp
N33
ap Recursive layer MAXNET
[...] HEURISTIC IS [...] LEARNING MEMORY METHOD [...] REINFORCEMENT [...]
N2p
N23
Evolutionary optimization: z, W 1
1
b1
b3
N12
N11
Mpp
1
1
b2
bp
Chromosome W=weights
N1p
N13
Binary distance layer
N02
N03
N04
N05
x1
x2
x3
x4
x5
Fuzzy N06 connectionist structure
p=number of patterns
Minimum Hamming distance fitness function F(G)=DHmin(z,wi)
Wpn
N01
925
Genotype G ={Chz, ChW }
[...] ADAPTIVE ALGORITHM ASSOCIATIVE BIDIRECTIONAL CRITIC DISCRETE DRIVER
N0n
Chromosome z=input pattern
[...]
b
x6
xn
xi={-1,+1}
a
WORD b
WORD 1 WORD 2 WORD 3
Input of the net: n=a*b
a
WORD a
Sentence syntax analysis
4 3 2 1
WORD 4 WORD 3 WORD 2 WORD 1
Sentence segment analysis using neural networks
b
[...] HEURISTIC IS [...] LEARNING MEMORY METHOD [...] REINFORCEMENT [...]
Evolvable architecture of ANN
Binary image of a sentence
a
Words into a sentence , 1
}
Output: images of the recognized words
WORD b
WORD 1 WORD 2 WORD 3
Bit = { 0 (for ANN: -1) Represented word class
ABCDEFGH I J KLMNOPQRSTUVWXYZ 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Classification to word classes (Set of word patterns as sentence components) C2
C1 N32
N31
a1
a2
N21
N11
ap
N2p
1
1
Chromosome Evolvable b2 W=weights architecture of ANN N12
bp N1p
Evolvable learning of ANN
x2
a
ABCDEFGH I J KLMNOPQRSTUVWXYZ
Image of recognized word ‘ASSOCIATIVE’
Mpp
p=number of patterns
Evolutionary optimization of parameters: z, W
Genotype G ={Chz, ChW } Input of the net: n=26*a
Fitness function F(G)=DHmin(z,wi)
b ABCDEFGH I J KLMNOPQRSTUVWXYZ
Wpn Chromosome z=input pattern
Fuzzy connectionist structure
xi={-1,+1} x1
N3p
N22
1
b1
Cp
A S S O C I A T I V E
xn
Binary images as isolated components of the text created by the word analysis module
a
1 2 3 4 5 6 7 8 9 10 11 12 13 14
A S S O C I A T
a
ABCDEFGH I J KLMNOPQRSTUVWXYZ
Binary image of letter string ‘ASSOCIAT’
Fig. 6. Evolvable fuzzy neural networks for word and sentence recognition
926
W. Kacalak and M. Majewski
4
Experimental Results
The prototype of the intelligent communication system was developed for Windows Mobile for a PDA (Pocket PC) and a Tablet PC for Windows. As shown in Fig. 7A, the ability of the implemented evolvable fuzzy neural network to recognize a word depends on the number of letters of that word. For best performance, the neural network requires a minimum number of letters of each word being recognized as its input. The ability of the evolvable fuzzy neural network to recognize the sentence depends on the sentence length as shown in Fig. 7B. Similarly, for best sentence recognition, the neural network requires a certain minimum wordcount of the given sentence. (A)
(B)
Lmin = 0,86 * L 0,72
W min =1,13*W 0,61 6
8 17
7
10
11
5 8
4 5
3 2
2
3
6
9
12
15
16 Minimum number of words Wmin
Minimum number of letters Lmin
5 14
6 13
10
7
4
3
2
0
2
3
2
3
4
5
6
4
5
6
7
8
9
7
8
9
1
1
1
1
4
0 0
2
4
6
8
10
12
14
16
18
0
Number of letters of the word L
1
10
11
Number of words of the sentence W
Fig. 7. Experimental results: (A) Sensitivity of spoken word recognition: minimum number of letters of the word being recognized vs. number of word letters; (B) Sensitivity of spoken sentence meaning recognition: minimum number of words of the sentence being recognized vs. number of sentence component words
The experimental implementation of the intelligent e-learning system prototype allowed for an attempt of its evaluation. The testing has enabled to determine for the system the following attribute values [6]: – Suitability, expressing the degree of appropriateness of the system to the tasks which have to be accomplished. – Learnability, which conveys how easy it is for the user to learn the system and how rapidly the user can begin to work with it. – Error rate, which reflects the error ratio while working with the system.
5
Conclusions and Perspectives
A speech interface between users and e-learning systems using the natural language, is ideal because it is the most natural, flexible, efficient, and economical
E-Learning Systems with Artificial Intelligence in Engineering
927
form of human communication. Application of hybrid neural networks allows recognizing sentences of similar meanings but of different lexical and grammatical patterns, which will undoubtedly be the most important way of communication between humans and computer systems. This presented approach can be included to the methods of natural language processing. The condition of the effectiveness of the presented system is to equip it with mechanisms of meaning analysis of the user’s utterance or answer, as well as analysis, evaluation and assessment of the user’s knowledge and skills. In the e-learning systems, the condition of high quality communication between users and e-learning systems is the analysis of the e-learning software and process state before an utterance is performed by the user, and use of artificial intelligence for the analysis, evaluation and assessment of the answer or command. The developed flexible intelligent e-learning system can be extended for various applications. The experimental results of the proposed system show its promising performance for further development and experiments. The system described in this paper is a conceptually new approach to the problem of lack of coordination of understanding components with system outputs in a spoken language system, although it is an appropriate means of communication that may or may not be taken advantage of.
References 1. Ayres, T., Nolan, B.: Voice Activated Command and Control with Speech Recognition over WiFi. Science of Computer Programming 59(1-2), 109–126 (2006) 2. Govindasamy, T.: Successful Implementation of E-Learning. The Internet and Higher Education 4(3-4), 287–299 (2002) 3. Ismail, J.: The Design of an E-Learning System. The Internet and Higher Education 4(3-4), 329–336 (2002) 4. Larkin, M.: Handheld Use Increasing for E-Learning and Clinical Decision Making. The Lancet 361(9351), 93 (2003) 5. Paek, T., Chickering, D.M.: Improving Command and Control Speech Recognition on Mobile Devices: Using Predictive User Models for Language Modeling. User Modeling and User-Adapted Interaction 17(1-2), 93–117 (2007) 6. Rosenberg, M.J.: E-Learning: Strategies for Delivering Knowledge in the Digital Age. McGraw-Hill, New York (2001) 7. Wald, M., Bain, K.: Universal Access to Communication and Learning: the Role of Automatic Speech Recognition. Universal Access in the Information Society 6(4), 435–447 (2008)