Oscar: An Intelligent Adaptive Conversational Agent Tutoring System
Annabel Latham (
[email protected]) Intelligent Systems Group School of Computing, Mathematics & Digital Technology Manchester Metropolitan University
Introduction Intelligent Tutoring Systems (ITS) The Index of Learning Styles (ILS) Conversational Agents Oscar Conversational ITS (CITS) Experimental Study Results Conclusions and Further Work
www.AnnabelLatham.com
[email protected]
Intelligent Tutoring Systems
3 main ITS approaches [1]: ◦ Curriculum sequencing ◦ Intelligent solution analysis ◦ Problem solving support
How do ITS adapt to the student?
www.AnnabelLatham.com
[email protected]
The Index of Learning Styles (ILS) PERCEPTION DIMENSION Preferred type of information SENSORY external
INPUT DIMENSION Preferred way to receive external information
INTUITIVE internal
VISUAL diagrams
VERBAL explanations
LEARNING STYLE
PROCESSING DIMENSION How information is converted into knowledge ACTIVE discussion
REFLECTIVE introspective consideration
UNDERSTANDING DIMENSION Progression towards understanding SEQUENTIAL continual steps
GLOBAL large jumps
‘Learning and Teaching Styles in Engineering Education’, Felder & Silverman 1988 Assessed by questionnaire
www.AnnabelLatham.com
[email protected]
Learning Styles and ITS Adapt content according to preference How to adapt to learning style (LS)?
◦ One dimension of learning style – but only reacting to one aspect of LS ◦ Several dimensions – but need many types of learning material Choose strongest preference – but still only one aspect of LS – it may change for different subjects
◦ Proposed system: select adaptation for each question depending on student LS strength and question adaptation strength www.AnnabelLatham.com
[email protected]
Conversational Agents Natural language communication with computers Intuitive to use Allow direct, non-linear access to information Web-based guidance, tutoring, customer support, user interfaces
www.AnnabelLatham.com
[email protected]
Oscar Conversational ITS Conversational ITS which dynamically predicts and adapts to each student’s learning style during tutoring Directs a tutoring conversation (‘mixed initiative’) Incorporates:
◦ Curriculum sequencing ◦ Intelligent solution analysis ◦ Problem solving support www.AnnabelLatham.com
[email protected]
www.AnnabelLatham.com
[email protected]
Oscar CITS Generic Architecture Learning Styles Adapter
Tutorial Knowledge Base
Scripts
Student Model
Controller
Graphical User Interface
Conversational Agent
www.AnnabelLatham.com
[email protected]
User
Methodology Overview
Learning Styles:
◦ Analysis of ILS model: Table of learner behaviour mapped to preferred teaching styles Domain-independent tutor material categories
◦ Mapping of ILS to tutoring conversation scenarios:
Different styles of question (question templates) Different styles of tutor material (e.g. examples, movies)
Domain-independent Adaptation Algorithm ◦ Consider neutral learners ◦ Combine strength of learner preference and adaptation strength
www.AnnabelLatham.com
[email protected]
Methodology (continued)
Conversational Tutorial: ◦ SQL tutoring conversation scenarios captured from human tutors ◦ Scripting CA (38 contexts, over 400 rules) ◦ Several versions of learning material ◦ Each question scored for adaptation to each learning style
Design of student model, controller, GUI
www.AnnabelLatham.com
[email protected]
Example CA FAQ Rule a:0.01 p:50 * *select* p:50 *select* * p:50 * *select* p:50 *select* * p:50 * *select* p:50 *select* * r: The SQL SELECT command is used to retrieve data from one or more database tables. * www.AnnabelLatham.com
[email protected]
Experimental Methodology
Hypothesis: students presented with material matched to learning styles perform better than those using unsuited material 70 undergraduate Computing participants 3 experimental groups: ◦ Neutral ◦ Adapt ◦ Mismatch
Performance measured by counting number of correct tutorial question answers
www.AnnabelLatham.com
[email protected]
Experimental User Interaction START
Anonymous Registration
Formal ILS Questionnaire
Pre-tutorial MCQ Test (‘pre-test’) Conversational Tutoring Session Post-tutorial MCQ Test (‘post-test’) Test Results Comparison and Oscar’s Feedback User Evaluation Questionnaire
www.AnnabelLatham.com
END
[email protected]
Results Experimental Group
No. Students
Avg. Correct Answers
Neutral
8
72%
Adapt
32
73%
Mismatch
14
61%
All Participants (Total)
54
70%
Mean test score improvement 21% 91% would use Oscar CITS:
◦ 77% instead of a book ◦ 50% instead of attending class
“it encouraged me to think rather than simply giving me the answer”
www.AnnabelLatham.com
[email protected]
Conclusions 12% better performance when adapting to learning styles compared to mismatched material Positive student learning experience because:
◦ Users value conversational tutoring ◦ Oscar CITS has improved tests scores by 21% on average
Domain-independent adaptation algorithm: ◦ Considers both learning style and strength of tutor material adaptation available ◦ Allows staged development of tutor material ◦ Adaptation changes to choose best fit for each question
www.AnnabelLatham.com
[email protected]
Current Progress & Further Work Current progress Automatic, dynamic prediction of all ILS dimensions Further work Authoring Toolkit Voice recognition
www.AnnabelLatham.com
[email protected]
