An Affective and Cognitive Tutoring System for Learning Programming María Lucía Barrón-Estrada1, Ramón Zatarain-Cabada1, Francisco González Hernández1, Raúl Oramas Bustillos2, Carlos A. Reyes-García3 1
Instituto Tecnológico de Culiacán, Juan de Dios Bátiz s/n, Col. Guadalupe, Culiacán Sinaloa, 80220, México 2 Universidad de Occidente, Carretera a Culiacancito, Km. 1.5, Culiacán, Sinaloa, 80054, México 3 Instituto Nacional de Astrofísica, Óptica y Electrónica (INAOE) Luis Enrique Erro No. 1, Sta. Ma. Tonanzintla, Puebla, 72840, México {rzatarain, lbarron}@itculiacan.edu.mx,
[email protected]
Abstract. In this paper, we present a multiplatform and Intelligent Tutoring System for learning Java (Java Sensei). The learning system combines state-of-the-art action selection, motivation through emotions, a modern recommendation mechanism, and multimodal instructional and selection learning. Java Sensei architecture works with a collection of modules and processes, each with its own effective representations and algorithms. The learning system was implemented under different learning methodologies like problem-solving for the pedagogical module, knowledge space for the expert module, and overlays for the student module. One of the main contributions of this work was the integration of cognitive and affective information in a behavioral graph which is used by a learning companion to show emotions and empathy to the student. Java Sensei was tested with different groups of university students with which we obtained positive results. In addition to providing a detailed description of the implementation and evaluation of Java Sensei, we also provide some proposals of future work in our intelligent tutoring systems. Keywords: Intelligent Tutoring Systems, Affective Computing, Artificial Neural Networks, Fuzzy Logic, Mobile learning, Knowledge Spaces.
1
Introduction
Learning a programming language is the first big challenge for a student of computer science or a related field [1]. Some of the issues that contribute to this obstacle are: the teaching methods employed by the instructor, the difficult nature of computer programming, the study methods, abilities and attitudes employed by the student, the nature of the art of programming, the lack of prior knowledge of novice students, and the psychological influence that the student suffers from society [2, 3, 4, 5]. Many researchers have proposed and developed methodologies and tools to help students learn programming. For example the ACM has SIGCSE (Special Interest Group on Computer Science Education) where issues related to the development, implementation and/or evaluation of programs and curricula are discussed [6]. Today, in the field of Artificial Intelligence, affect recognition is an emerging research area of significant theoretical and applied awareness to a number of academic
fields including machine learning, computational linguistics, computer vision and speech and language, neuroscience, educational psychology, and pedagogy [7]. The link between affect computing and student learning has been a topic of increasing attention in recent years. Due to the importance of this connection of emotions and learning, one of the current aims of many intelligent tutoring systems is to recognize and manage student emotional states by means of affective mediations. Intelligent Tutoring Systems (ITS) try to simulate a human tutor to provide personalized instruction taking into account not only cognitive aspects of students [8] but also affective elements [9]. This paper presents the implementation of an Intelligent Learning Environment (ILE) for Java programming. This system works in a flexible and interactive multiplatform environment that considers both cognitive and affective states of students. The paper is organized as follows: section 2 describes related work of the main topics. Section 3 shows and explains the architecture of the ILE Java Sensei. Affect recognition and feedback is illustrated in section 4. Section 5 presents the experiments and results produced with the system. Conclusions and future work are discussed in Section 6.
2
Related Work
Research on intelligent tutoring systems for programming language has focused on different programming languages like Java, C-C#, or PHP. Wiggins et al., [10] implemented The JavaTutor System, an affective tutoring system that uses natural language dialogues for human-to-human or human-to-computer dialogues using machine learning techniques. The system takes into account cognitive aspects, where it applies the ACT-R theory of cognition for knowledge representation. Affect focused on nonverbal behavior obtained through bodily expressions and detection is performed by different hardware tools. CSTutor is a tool [11] to help students learn programming in C#. The tool incorporates anchored instruction for the domain representation of the programming language and game theory, instead of affect recognition, for motivation improvement. The PHP ITS was developed to teach students to program in PHP scripting language [12]. The tutor makes an analysis written by the student and translates to the equivalent of an abstract syntax tree (AST) code. Then, the AST is used to create events or trigger actions. The facts that exist after all AST nodes are processed are called the final state. To check the correctness of a program, the tutor analyzes whether all predicates in the goal for a particular task are present in the final state. Through this method, the student writes the solution of the task defined in its own way, as long as the final state of his/her program meets the specified target. CTutor [13] is an intelligent learning environment (ILE) where students perform exercises programming in C. The idea of this system is to facilitate the effectiveness of learning when students navigate through complex problems and thereby provide an advanced education system to enhance the learning process. This ILE allows teachers to create new programming exercises and manage them from the system interface. The domain in the system is adjustable for each solution of each exercise. From the perspective of a student the CTutor interface is a tool with controls to represent the solution of the exercises. The tool uses a constructivist pedagogical orientation where
the authors give special weight to feedback as the main source of information; moreover, its domain is formed based on a constraint-based model. On the other hand, CTutor does not support affect recognition.
3
Java Sensei Architecture
In this section we present the main architecture of Java Sensei: the affective environment for learning Java programming. Figure 1 shows the different layers and components of our architecture. Sensei Java architecture is designed in seven layers with a client-server structure with a fat client. The aim of developing a thick client is to reduce the processing load on the server.
Fig. 1. Java Sensei Architecture
Next, we explain the function of each layer. Web Layer: The Web Layer is the layer of the client, and is made up of modules that need to run in the web browser when the user interacts with the system. This layer contains the following components: - Web Interface: This component represents the web user interface, so this component is responsible for interaction with the customer, the use of external libraries and centralizing interactions with other layers, mainly with the Service Layer. - Example-Tracing Engine: This element is the semantic and visual interpretation of all courses in the system. Within this component, the following tasks are performed: creation of tracing trees, extraction of tree information, and interaction with the Web Interface to build the GUI. In addition, the events generated by the example-tracing logic and notification of state changes are managed. - Pedagogical Agent: This component creates the visual representation of a learning companion and it is based on the pedagogical information obtained from the server layer. The actions performed are: extraction of pedagogical data for the visual representation of the agent and extraction of information of affective information from the student using the tree created by Engine Example-Tracing. - Camera in Background: This component manages the capture of the array of photographs in the background during exercise sessions in the system. It is responsible for starting and stopping the photography sessions, clear the memory of a photograph queue, and manage the time between each capture; also it is responsible for converting the array of photographs into an understandable format for the Server Layer. Service Layer: It is the layer where the components related to the handling of external requests to the server are located. In this layer the components must be accessible from the Web and use the HTTP communication protocol. This layer holds the next components: - Restful Services: This component is responsible for receiving incoming web requests from the Web Layer. Its role is to accept requests and ensure its integrity and completeness. It also gives them a format so they can be worked on by the components comprising the Server Layer. Server Layer: The Server Layer is the layer that contains the modules of the server, which is composed of modules that need a strong interaction with the Domain Layer and Student Layer; also it makes decisions using a high burden of processing and memory resources. Within this layer there are some key components for making instructional decisions. - Tutoring Module: It is responsible for taking appropriate cognitive and affective educational decisions for the student as: assessing the quality of student response, assessing the overall ability of the student and use the Affective Framework component for obtaining affective information. - Adaptation Module: It uses the perceptions and preferences of the student in relation to the exercises and help resources to build a recommendation system.
-
4
Affective Framework: This component represents the affective student assessment. It uses cognitive and affective information from the students provided by the Tutoring Model. It uses a back propagation neural network to recognize emotions through the face and uses a fuzzy logic system for affective pedagogical strategy. Domain Layer: This layer contains one component: The Content Manager, which is responsible for interacting with the Server Layer and Data Management Layer. The Content Manager represents the tutor domain or knowledge of the Java language. This knowledge is represented using the theory of knowledge spaces which is encoded using JSON files. Student Layer: It represents the student's knowledge compared to the domain model. This layer contains one component: The Student Profile Management. The tasks of the component are: calculate the overall ability of the student (the total number of correct exercises) and quality of the current response (the number of errors that the student makes in an exercise); verify changes in the domain model to upgrade the overlay student model; and store student preferences regarding exercises and resources. The component also interacts with the Data Management Layer to store/retrieve student data in JSON format. Data Management Layer: This layer contains two components: Entities and Plain Objects. The component Entities are data structures that encapsulate domain and student information that are used in the Data, Domain, and Student Layers respectively. The component Plain Objects contains information in plain text in JSON format. Data layer: This layer manages the data for the Domain and Student Modules represented in JSON and MongoDB formats.
Affective Recognition and feedback
Figure 2 shows the steps followed in the recognition and affective feedback from the system. In the first step the ILE Java Sensei receives the data and the student's collection of images from the browser which are later sent to the ILE. In the second step the ILE starts a cycle with several student faces being sent to the neural network, and decides, at the end of the cycle, the current emotion. This current emotion is obtained based on the most frequent student emotion. In the third step the ILE sends the updated student model to the fuzzy system which performs the fuzzy rules to output the tutor actions. In the last step, the tutor actions are sent to the browser to build the actions of the pedagogical agent. Pedagogical Agent: The pedagogical agent is responsible for transmitting messages that a human tutor would perform. For this, it has been provided with both facial expressions and dialogues that can communicate to the student.
Fig. 2. Affect Recognition and Response
These expressions are based on facial expressions investigated by Ekman [14]. Positions of the eyebrow, mouth, and eye opening are taken into account. These positions are entered as parameters in the FaceGen Modeller [15] software. In Figure 3 we observe the four facial expressions made by the pedagogical agent of Java Sensei. These emotions were based on the pedagogical agent working with AutoTutor [16].
Fig. 3. Pedagogical Agent Fuzzy System and Neural Netwkork: The fuzzy logic system seeks to represent the way a human tutor acts when he/she faces various scenarios related with cognitive and emotional situations of a student. These situations have to do with the predefined actions. The fuzzy system uses the emotional values obtained by the neural network
which recognizes the affective state of the student when he/she works with the ILE Java Sensei. The fuzzy logic system works with four input fuzzy variables and three output ones. A total of 144 fuzzy rules were constructed. To define these rules we use Fuzzy Control Language, a standardized language for creation of fuzzy rules. The implementation was done with jFuzzyLogic [17]. The emotion recognition system was built in three steps: the first one was an implementation to extract features from face images in a corpus used to train the neural network. The second step was the implementation of the neural network. We used the Java-based algorithms implemented in NeuroPH [18] to implement classification by using a neural network (feed-forward method). The third step integrated feature extraction and emotion recognition with the fuzzy system. For training and testing the neural network, we used the corpus RAFD (Radboud Faces Database), which is a database with 8040 different facial expressions that contain a set of 67 models including men and women. In the training process we obtained a successful rate of 86 % which we consider is good enough for our system. Once the emotional state is extracted from the student, the emotional state is sent to the fuzzy system. Adaptation Module: The ILE must be able to adapt to the student. The ILE collects the rating produced by the student concerning the exercises and resources that the student visits. The rating of one exercise is measured using Likert scale [19] where one indicates total disagreement with the exercise and five indicates total agreement. At the beginning all the exercises are rated with a value of two. We used a recommendation system created with Apache Mahout [20]. This tool helps to find the next problem the student must solve. The system implements the K-Nearest Neighbor (kNN) approach and the Pearson Correlation. As more users are included, the rating system generates the best recommendations and thereby it gets more adapted to the changes. Pedagogical Module: The system uses the pedagogical model known as "problem solving" [21] where the student learns as he solves problems with a certain structure. The system uses three types of strategies for solving problems. Type 1 is used to evaluate theoretical concepts with “true-false” exercises. Type 2 questions present a complete program or piece of code and ask the student to determine the output of the code. Type 3 is a combination of the two previous types and allows creating more complex exercises involving "n" number of steps to reach the solution. Expert Module: This model contains the expert knowledge that the student wants to learn and practice. The system represents the knowledge using JSON files (a variation of XML files). Knowledge representation uses the knowledge space theory [22]. The expert model represents six basic skills that students must master to be modeled by a graph representing the Knowledge Space. These skills are: Introduction to Java, Variables and calculations, selection, iteration, methods, and arrays. Example-Tracing Engine: An ILE combines hypermedia techniques with other methods used by Intelligent Tutoring Systems (ITS). We applied Example-Tracing [23, 24] in order to implement the ITS in Java Sensei. Example-Tracing ITS has the advantage of providing step-by-step guiding to the students and providing with multi-
ple strategies to the problem solution, including optimal or sub-optimal solution or miss-conception management. Example-Tracing exercises can be easily adapted with the pedagogical model implemented with “problem solving” and they are represented by a graph that is traversed by the student (figure 4). One of the main contributions of our work is the integration of cognitive and affective information in a behavior graph which is used by the pedagogical agent to show emotion and empathy to the student. Next, we describe the algorithm to implement the affective-example-tracing: a) b) c)
d)
The ILE loads the JSON file with the corresponding exercise. From the JSON file, a behavior graph is created and an initial step is executed. As students traverse the graph (by clicking with the mouse), different options can be performed and different paths are traversed. The options are: i. Initial step (IS) ii. Error step (ES) iii. Optimal step (OS) iv. Sub-optimal step (SOS) v. Final Optimal step (FOS) vi. Final Sub-optimal step (FSS) Every step triggers two events: i. Actual Emotion Evaluation. ii. Tutor Action Evaluation.
Every time a student chooses some of the options in the graph, the engine calls the fuzzy logic system to get the tutor action (feedback, intervention, etc.) and to get the emotion information so the pedagogical agent can build the graph. Figure 4 illustrates an example. At the left we can observe how the student solved the problem in an optimal way. At the right the student made a mistake.
Fig. 4. Affective Example-Tracing
5
Experiments and Results
In order to test the ILE, we made different experiments with different kind of students of the Instituto Tecnológico de Culiacán. The system was first tested with graduate students in our research lab. This test has the aim to test the functionality of the ILE. Next, we show (figure 5) a small session of a student browsing in the ILE. To access the ILE, the student must use a browser with a PC, laptop, or mobile device (e.g. a smartphone), and a Facebook account allowing the use of the Web cam. The left part of the example shows a problem with the student making a mistake, at the end of the code. In the right part of the problem the student finish successfully the exercise.
Fig. 5. Affective Example-Tracing The following evaluation was applied to Computer Systems students (Engineering) taking currently a Java course. In this case we consider two aspects: the software functionality and the software effectiveness. With respect to software functionality we compared Java Sensei against three similar ILEs: JavaTutor System [25], JITS [26] y jLatte [27]. We considered 9 features to evaluate [6] and Likert scale from 1 to 5, where 1 represents an incomplete feature and 5 represents a complete feature the ILE accomplish. Table 1 shows the results obtained from the software functionality evaluation. The results show that Java Sensei and JavaTutor System are the ILEs that get better results in the evaluation. The evaluated systems meet almost every feature of an ILE/ITS system except for three features (Mixed Initiative) and 6 (self-improvement). Feature 7 (Affect recognition and Handling) which is a recent area causing that few systems incorporate this feature. The same applies to features 8 and 9. It is expected that modern ITS migrate to the Web and Mobile platforms.
Table 1. Comparisons Results in Software Functionality.
-------------ILE----------Feature
JavaSensei
Generativity
3
Student and Expert Modeling Mixed Initiative Interactive Learning
JITS
jLatte
5
4
4
5
5
5
5
2
2
1
1
4
5
4
5
Instructional Modeling Self-improving
5
5
5
5
1
1
1
1
Affect Recognition and Handling Web Platform
4
5
1
1
5
5
1
1
Mobile Platform
5
1
1
1
TOTAL
34
34
23
24
JavaTutor System
For evaluation of the effectiveness of the system, we conducted a test with a group of 20 students in the subject Object Oriented Programming (OOP), looking to measure the learning outcome of the ILE JavaSensei on the topic of inheritance. The selection of students was not performed randomly, but by availability of participants. To access the ILE, students used a web browser with support for HTML5 and front camera in desktops, laptops, and smartphone, entering the system from their Facebook account (see left part of figure 6).
Fig. 6. Students using Java Sensei The evaluation design methodology consisted of measuring the knowledge of two groups of student: one experimental group who learns by using the ILE and one control group who learns by using traditional teaching. The time given to the pretest was 15 minutes (see right part of figure 6), the time for the ILE session was 70 minutes, and the time for the posttest was 20 minutes. Figure 7 shows the evaluation of both
tests. The results in the posttest indicate that the experimental group obtained better scores (in most cases) than the control group.
Fig. 7. Pretest and Posttest Results
6
Conclusions and Future Work
The ILE has been implemented with open source code in programming languages and software libraries like Java, Javascript, MongoDB, JSON, jFuzzyLogic, Apache Mahout, Weka, jQuery, jQueryMobile among others, and can be accessed from different platforms (operating systems, web browser, computers, etc.). Right now, a group of programming Java professors is applying more tests to the ILE with students from our university (Instituto Tecnológico de Culiacán). For future work, we want to include gamification techniques in order to work more the motivational part of the system. We also want to work with emotions that affect the process of learning (engagement, boring, frustration, etc.). Last we are developing a new version oriented to more collaboration work among students and authoring new exercises by instructors. Funding The work described in this paper is fully supported by a grant from the DGEST (Dirección General de Educación Superior Tecnológica) in Mexico under the program “Projects of Scientific Research and Technological Innovation”.
References 1. Rodrigo, M.M.T., Baker, R.S.J. d, Jadud, M.C., Amarra, A.C.M., Dy, T., Espejo-Lahoz, M.B.V., Lim, S.A.L., Pascua, S.A.M.S., Sugay, J.O., Tabanao, E.S.: Affective and behavioral predictors of novice programmer achievement. SIGCSE Bulletin. 41, 156–160 (2009) 2. Matthíasdóttir, Á. How to teach programming languages to novice students? Lecturing or not. in International Conference on Computer Systems and Technologies-CompSysTech ( 2006) 3. Jenkins, T. On the difficulty of learning to program. in Proceedings of the 3rd Annual Conference of the LTSN Centre for Information and Computer Sciences (2002)
4. Gomes, A. and A.J. Mendes. Learning to program-difficulties and solutions. in International Conference on Engineering Education–ICEE (2007) 5. Bernard, M. and E. Bachu, Enhancing the Metacognitive Skill of Novice Programmers Through Collaborative Learning, in Metacognition: Fundaments, Applications, and Trends, Springer. p. 277-298 (2015) 6. SIGCSE. 2015 [cited 2015]; Available from: http://www.sigcse.org/. 7. Picard, R. Affective Computing: From Laughter to IEEE. IEEE Transactions on Affective Computing 1, 11-17 (2010) 8. Woolf, B.P., Building Intelligent Interactive Tutors. Amherst, Massachusetts, Estados Unidos: Morgan Kaufmann Publishers (2009) 9. Harley, J. M., Lajoie, S. P., Frasson, C., & Hall, N. C. An Integrated Emotion-Aware Framework for Intelligent Tutoring Systems. InArtificial Intelligence in Education (pp. 616-619). Springer International Publishing (2015) 10. Wiggins, J. B., Boyer, K. E., Baikadi, A., Ezen-Can, A., Grafsgaard, J. F., Ha, E. Y., Wiebe, E. N. JavaTutor: An Intelligent Tutoring System that Adapts to Cognitive and Affective States during Computer Programming. Paper presented at the Proceedings of the 46th ACM Technical Symposium on Computer Science Education (2015) 11. Budi, H., & Jim, R. Incorporating Anchored Learning in a C# Intelligent Tutoring System. Paper presented at the Consortia Proceedings of the 21st International Conference on Computers in Education. Indonesia: Asia-Pacific Society for Computers in Education (2013) 12. Weragama, D., & Reye, J. (2012). Designing the knowledge base for a PHP tutor. Paper presented at the Intelligent Tutoring Systems (2012) 13. Kose, U., & Deperlioglu, O. Intelligent Learning Environments within Blended Learning For Ensuring Effective C Programming Course. International Journal of Artificial Intelligence in Applications, 3(1) (2012) 14. Ekman, P. Facial expressions. Handbook of cognition and emotion, 16, 301-320 (1999) 15. Todorov, Alexander, and Nikolaas N. Oosterhof. "Modeling Social Perception of Faces [Social Sciences]." Signal Processing Magazine, IEEE 28.2: 117-122 (2011) 16. D’Mello, S., et al. AutoTutor detects and responds to learners affective and cognitive states. in Workshop on Emotional and Cognitive Issues at the International Conference on Intelligent Tutoring Systems (2008) 17. Cingolani, P. and J. Alcalá-Fdez, jFuzzyLogic: a java library to design fuzzy logic controllers according to the standard for fuzzy control programming. International Journal of Computational Intelligence Systems, 6(sup1): p. 61-75 (2013) 18. Sevarac, Z. Neuroph-Java neural network framework. Retrieved in July, 2015 19. Likert, R., A method of constructing an attitude scale. Scaling: a sourcebook for behavioral scientists. Chicago: Aldine, 1974: p. 233-243 (2012) 20. Anil, R., Dunning, T., & Friedman, E. Mahout in action (pp. 145-183). Shelter Island: Manning (2011) 21. Kumar, R., et al. Comparison of Algorithms for Automatically Building Example-Tracing Tutor Models. in Educational Data Mining (2014) 22. Doignon, J. P., & Falmagne, J. C. Knowledge spaces. Springer (1999) 23. Aleven, V., et al., A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2): p. 105-154 (2009) 24. Aleven, V., et al. The cognitive tutor authoring tools (CTAT): preliminary evaluation of efficiency gains. in Intelligent Tutoring Systems. Springer (2006) 25. Wallis, M. JavaTutor-A Remotely Collaborative, Real-Time Distributed Intelligent Tutoring System for Introductory Java Computer Programming-A Qualitative Analysis (2011) 26. Sykes, E. R., & Franek, F. An Intelligent Tutoring System Prototype for Learning to Program Java TM. IEEE Xplore. Proceedings of the The 3rd IEEE International Conference on Advanced Learning Technologies (2003) 27. Holland, J., Mitrovic, A., & Martin, B. J-LATTE: a Constraint-based Tutor for Java (2009)
