Lecomps5: A Framework for the Automatic Building of Personalized Learning Sequences Carla Limongelli1, Filippo Sciarrone3, Marco Temperini2, and Giulia Vaste1 1
Roma Tre University, Dept. of Computer Science and Automation, Via della Vasca Navale 79 – 00146 Rome, Italy {limongel,vaste}@dia.uniroma3.it 2 Sapienza University, Dept. of Computer and Systems Science, Via Ariosto 25 – 00184 Rome, Italy
[email protected] 3 Open Informatica srl, E-learning Division, Via dei Castelli Romani 12/A – 00040 Pomezia, Italy
[email protected]
Abstract. In the context of distance learning, Adaptive Web-based Educational System focus on personalization and adaptation, that is on “learner’s satisfaction”. In this paper we address the other side of the coin, that is the "teacher’s satisfaction" problem, which is quite seldom taken into account in educational systems. We present a new version of the Lecomps5 Web-based Educational System, a system capable of providing personalization and adaptation on the basis of learner’s knowledge, learning styles and learning progresses. In this new version, a framework provides the teacher with an easy and flexible tool for managing learning material, expressing different didactic strategies and sequencing personalized courses by means of an embedded planner. Such functionalities are supported by the system basing on evaluations of learner’s knowledge, learning styles, and learning progresses. We report on a first controlled experiment, we made to evaluate the “teacher’s satisfaction”.
1 Introduction Modern research in distance learning focuses most attention on personalization and adaptation of courses to learner’s needs, as opposite to the traditional "one-size-fitsall" approach [2]. Lecomps5, presented in [9], is a Web-based Educational System, in which the learning experience can be personalized, making adaptive the learning content together with its delivery. The personalization of the learning experience is sought for several reasons: individualized content, built up through appropriately designed learning resources, can be expected to be better accepted by the learner, as (s)he might better understand it and consider it relevant to her apparent needs; this, in turn, may have good effects on learner’s motivation and collaboration, and then in overall satisfaction. Such effectiveness can also let gain in efficiency, as better motivated learners may be more likely to qualify, and in good time. All the above motivations might be considered as different facets of the term “learner’s satisfaction”. M.D. Lytras et al. (Eds.): WSKS 2008, LNAI 5288, pp. 296–303, 2008. © Springer-Verlag Berlin Heidelberg 2008
Lecomps5: A Framework for the Automatic Building
297
As a matter of facts, the aspect of "teacher’s satisfaction" is quite seldom taken into account, while discussing about adaptive Web-based Educational Systems. Producing learning material is a task that needs a considerable effort by the teacher, but, often, the teacher is also asked to define appropriate metadata or rules in order to sequence the material in a suitable way. The first task can not be performed in an automatic way, but the second one can be made easier by suitable tools. So, we are seeking for a light-as-possible approach to course personalization, trying to focus on the aspects we think might affect personal satisfaction of both learner and teacher, good quality of the learning content, efficiency of its use and reuse, and good support to initial and continuing adaptation of the course to the learner (from both the content and presentation viewpoints). In this work we address the “teacher’s satisfaction” problem. In particular, we focus on the sequencing problem: Lecomps5 is improved with the Pdk Planner [5], and now provides the teacher with an easy and flexible tool for course configuration. If we examine some available adaptive educational systems, we see that sequencing is generally performed following two main approaches: sequencing given step-by-step to the students, through techniques such as adaptive link annotation and direct guidance, and sequencing that plans the entire learning path at the beginning, then modifying it, when the study does not succeed as it should. The first approach is applied, for example, in the AHA! System [6] and in the ELM-ART system [11]. The second approach is used in the DCG system [3] and in the IWT system [4]. Frequently, the sequencing techniques, are rule-based, such as in AHA!, or graph-based, such as in IWT. These techniques, however, lack in attention to teacher’s needs, either requiring heavy teacher’s work, or being useless for expressing different pedagogical approach or didactic preferences that the teacher might desire. The Lecomps5 system provides the teacher with an easy tool for managing learning material, for expressing different didactic strategies, and for sequencing personalized courses, both on the basis of student’s knowledge, learning styles, learning progresses during the study, and on the basis of teacher’s didactic strategies. In the rest of the paper, Sec.2 illustrates the architecture of Lecomps5; Sec.3 details on the Pdk planner and shows advantages in its use for sequencing strategies from the teacher’s point of view; Sec.4 presents an experimental system evaluation in a real instructional environment. Finally, in Sec.5 conclusions and future works are drawn.
2 The Lecomps5 System We present a new version of the Lecomps5 system, presented in [9], where, with respect to the old version, it is possible to provide adaptation by means of a planner embedded in the system. This updated version is a web-based learning environment supporting teacher’s, learner’s and administrator’s functionalities, capable to generate personalized and adaptive courses on the basis of the student’s starting knowledge on the domain of interest, and on the basis of the student’s learning styles. A personalized course, related to a given subject matter, is characterized by the Target Knowledge (TK) and by the Starting Knowledge (SK). TK is the knowledge to be acquired by the student through the course. SK is the learner’s knowledge about the topic prior of the course. In the system, knowledge is represented by atomic elements, called
298
C. Limongelli et al.
Knowledge Items (KI). A course is composed by a set of Learning Components (LC), i.e., learning objects enriched with the specification of the Required Knowledge (RK, prerequisites) and the Acquired Knowledge (AK), related to the learner’s fruition of the component’s learning content (both expressed as sets of KI); a value for the effort needed on the component by the learner is also specified; moreover the component contains different presentations of the learning material, according to the dimensions of Felder and Silverman’s learning style theory [7]. The real acquisition of the AK of a given LC can be evaluated through post-tests, also included in the LC and related to the concepts managed there. All the LCs related to a given subject matter are collected together into a pool, that is a sort of knowledge database. The teacher defines prerequisite relationships among LCs. This task is made easy by the graphic visualization of such relationships, in a graph of LCs, as shown in Fig.1. Lecomps5 configures the personalized course for a given learner basing on her SK, measured by a pre-test, her TK, pre-stated by the teacher, and navigating the LCs, as arranged by the teacher in the graph. During the configuration process LCs are selected, such that their overall AK, together with the SK, covers the TK. The automated configuration of the course is achieved by the Pdk planner, as shown in next Section. LCs editing is performed through the FCK Web editor1, while the graphical visualization of LCs in a pool and in personalized courses is obtained by producing SVG interactive web pages through the Graphviz system [8].
Fig. 1. The graph represents prerequisite relationships among Learning Components of the pool
3 The Planner Here we show how automated planning can help the teacher to apply a didactic approach and how it can support sequencing and adaptation, handling with different learning styles as well. 1
Available at http://www.fckeditor.net/ web site.
Lecomps5: A Framework for the Automatic Building
299
Course configuration problems can easily be seen as planning problems, where the learner is the executing agent, the initial world state is the initial student model (including her initial cognitive state and learning styles), and actions in the plan correspond to learning components of the pool (see for example [10] for an introduction to automated planning problems). When a learner executes an action, (s)he is offered fruition of a learning material according to her learning style. To understand such a material, the learner may need other knowledge (the action preconditions); after studying that learning material, and after a possible test, (s)he may be assumed to possess additional knowledge (the action effects). The plan goal corresponds to the course TK and course configuration corresponds to synthesizing a sequence of actions leading to the goal. In Artificial Intelligence (AI) planning, planning languages are used to specify problems in a uniform and simplified way. Also for the case of course sequencing we want to use a tool that allows a plain specification of requirements for teachers and learners. Here we focus on logic-based planners, which can exploit some important functionalities such as: domain validation, redundant actions detection or control knowledge specification, helping the teacher during the learning components arrangement. In AI planning, the term control knowledge means the additional information that can enrich the planning domain (given as mere list of actions with their preconditions and effects) and guide the plan synthesis. For instance, once a pool is arranged, a teacher might want to specify that a given LC must be studied before another one also if there is not a prerequisite relationship between them, or that a sequential1 student wants to alternate explanations and examples, while a global2 student prefers to see first all the explanations and then examples. What is needed is a language that allows the teacher to specify such kind of “control knowledge''. The Pdk3 (Planning with Domain Knowledge) planner conforms to the “planning as satisfiability'' paradigm, and the logic used to encode planning problems is the propositional Linear Time Logic (LTL) [12]. The related planning language PDDL-K [5] guides the user into the specification of control knowledge. Pdk accepts PDDL-K as input language, translates the problem description into its LTL representation and reduces planning to model search. Moreover, in the didactic context the user is the teacher that can decide: 1. at which level some concepts must be given. Following Bloom’s Taxonomy [1] we can differentiate LCs, for example at three levels: Knowledge, Application and Evaluation, 2. to sequence the LCs differently for different students’ learning styles, e.g.: examples and explanations can be differently sequenced for global or sequential students, 3. that some contents are mandatory for all the students, even if they already know them, and wants to force the sequence to present these contents, 4. different didactic strategies: for example the teacher could decide to explain recursion either with the induction principle or activation records, 5. that knowledge can be “classified'' in different categories (theoretic, exercises, examples, etc.), each category enjoying common properties: the teacher can specify 2 3
According to Felder and Silverman learning styles theory. Available at http://pdk.dia.uniroma3.it/
300
C. Limongelli et al.
that an action of a given category (for instance, an example) must always follow a given kind of action (theoretical material). 6. to define alternative prerequisites for given LC (see dashed arrows in Fig.1). The use of control knowledge in planning domain description languages enriches the expressivity of relations among concepts and helps both in configuring optimized courses and managing the LC pool, as it will be shown in the next section. Moreover, Pdk provides tools to support the debugging phase, by exploiting the fact that the planning problem is entirely encoded as a logical theory. In course sequencing, these tools can help in detecting “conceptual holes” such as about: 7. action executability: an action is not executable when its preconditions can never be satisfied; this happens in presence of loops in the components graph. 8. control rules redundancy, and coherence with the PDDL-K pool representation. Some of the above cited features (1, 2, 4, 6) are already available in Lecomps5; the others can be easily implemented through suitable interfaces.
4 The Evaluation of the System We carried out a first evaluation of the Lecomps5 system with the aim of measuring the efficiency and effectiveness of the proposed framework from the teacher’s point of view. Efficiency stands for the costs-benefits ratio, i.e., what is the required overall workload, in terms of time spent to prepare the course, compared to the didactical benefits due to the system’s use. By effectiveness we mean how good is the course proposed by the system, compared to the one proposed by the teacher himself. We followed the classical schema of a controlled experiment, performed in an environment where the human plays a fundamental role. Our main research questions are: - RQ1: Does Lecomps5 help the teacher to prepare his didactic plans? - RQ2: Is Lecomps5 able to generate reasonable didactic plans? - RQ3: What is the degree of teacher satisfaction after using Lecomps5? We implemented an experimental plan, by firstly selecting a sample of twenty computer science teachers, and then applying the following steps: test administration, experimental data gathering and results evaluation. Test Administration The chosen learning domain was Recursion, supported by an experimental pool with twelve LCs. A sample LC was Ricorsione:introduzione (Recursion:introduction); another was Ricorsione:esercizi (Recursion:exercises). According to our research questions, we submitted our sample to two different tasks. In the first task, hand-written, teachers were required to write down two distinct learning sequences, to deal with two different learner’s profiles in the same Recursion learning pool. To this aim first we prepared a questionnaire, suited to measure the student’s SK on the domain; then we simulated two different sets of answers for two distinct student profiles, referenced in the following as Profile A and Profile B and, finally, we submitted these profiles to the teachers and had back the proposed learning sequences. Such sequences were to be built only by LCs from the experimental pool, made available to
Lecomps5: A Framework for the Automatic Building
301
the teacher. In the second task, teachers were required to actually use Lecomps5 and to build a learning sequence accomplishing the same learning goal as in the first task. Teachers were invited to firstly complete the specification of the learning components of the pool (submitted incomplete on purpose), by stating their RKs and AKs. Then the system was used to automatically configure two learning sequences, one for each of the above mentioned learner’s profiles. Data Gathering We gathered the following data for every teacher involved in the experimentation: two learning sequences, hand-written, one happy sheet on the satisfaction degree in the use of Lecomps5, two learning sequences produced by the system, and a questionnaire in which each teacher assessed the learning sequences produced by the system and compared them with those produced by himself. Statistical Analysis and Results In Fig.2 the experimental results, concerning the teacher assessments for both profiles A and B are shown. In the figure, the x-axis is the ordinal scale ranging from –10 to +10; the y-axis, for every value of the ordinal scale, reports the frequency of teachers choosing that value. For the profile A, i.e., a student with an empty starting Knowledge on the recursion domain, the teachers assessed a good similarity degree between their hand-written course, compared to the one produced by the system. In fact, the 40% of the sample, i.e., eight teachers, gave “6” as similarity degree, that is the two courses were similar enough. The remaining 60% of the sample, i.e., twelve teachers, assessed “8” as similarity degree, i.e., a very high value of similarity degree. In both cases, for this profile, the teachers judged as positive the courses produced by the system. For the profile B, i.e., a student with a poor starting knowledge, 20% of the teachers, i.e., 4 teachers, evaluated the work of the system very similar to their handwritten work, giving “8”. Four teachers, that is the 20%, gave “0”, assessing in this way a neutral position with respect to the course produced by the system. Finally, twelve teachers, i.e., 60%, gave “4”, assessing a sufficient degree of similarity. In conclusion, we can say that both our experimental frequency distributions are entirely contained in the right part of the “0” point and consequently we can deduce that for both profiles the system performed well, as assessed by our sample.
Fig. 2. Teacher assessment on similarity degree
302
C. Limongelli et al.
In Fig.3, we show the distribution frequency of the answers to question Q2, with the same ordinal scale of the previous case, for both A and B student profiles. By this question we want to measure the didactic quality of the course produced by the system, also in case it is different from the one produced by the teacher. Again, for the A profile, the frequency distribution is shifted right of the “0” point, indicating that courses produced by the system are indeed reasonable. The 33,3% of the teachers gave “8” and “10”, assessing in this way a very good capability of the system in producing didactically valid courses. Values of “2” and “6” were assessed by the 16,7% of the sample. For the B profile, the frequency distribution was less positive, but however, with the 84% of the values in the right part with respect to the “0” point.
Fig. 3. Teacher assessment on reasonable quality of the course produced by the system
We submitted an happy sheet to the teachers, asking for their degree of satisfaction in using the system, together with other questions concerning the time spent to complete the task, with the goal of gathering some indications about usability as well. For example to the question: How much do you consider the Lecomps5 system useful in preparing learning courses?, the 50% of the sample answered “useful”, 25% “very useful” and 25% “not so useful”, with respect to an ordinal scale (“useless”, “not so useful”, “indifferent”, “useful”, “very useful”). This first experimentation of the system gave positive indications about usefulness and didactic reliability of the system. Our sample’s assessment seems to say that automatically produced courses are reasonable enough, and similar enough to those produced by the teacher.
5 Conclusions and Future Work The issue of how to define, manage and deliver personalized didactic courses is of great relevance in the evolving knowledge society. Nowadays, the knowledge that one can use comes out to be an asset of economic value. So then, transmitting knowledge (in general) and training learners (in our particular case), by using the internet and in a most efficient way, is a facet of how technology can be used to produce wealth. In such formative actions, personalization is a key issue, allowing to gain in efficiency and effectiveness.
Lecomps5: A Framework for the Automatic Building
303
In this paper we presented an upgraded version of the Lecomps5 system, a Webbased Educational System, addressing the "teacher’s satisfaction" problem. The teacher, through the system, and through a graphical environment, is able to configure the learning components which the knowledge domain is based on. We performed a controlled evaluation of the system by means of statistical analysis with encouraging results supporting our approach. We plan, as next steps in our ongoing research, to extend the PDDL-K language with new syntactic elements that would help the teacher in arranging the pool of learning components and to improve the usability of the system in order to allow the teacher to communicate in a more effective way his didactical strategies to the system.
References 1. Bloom, B.S.: Taxonomy of Educational Objectives. David McKay Company Inc. (1964) 2. Brusilowsky, P.: Adaptive Hypermedia. Journal of User Modeling and User-Adapted Interaction 11, 87–110 (2001) 3. Brusilowsky, P., Vassileva, J.: Course sequencing techniques for large-scale webbased education. International Journal of Continuing Engineering Education and Life-long Learning 13, 75–94 (2003) 4. Capuano, N., Gaeta, M., Micarelli, A., Sangineto, E.: Automatic student personalization in preferred learning categories. In: 3rd International Conference on Universal Access in Human-Computer Interaction (2005) 5. Cialdea Mayer, M., Limongelli, C., Orlandini, A., Poggioni, V.: Linear temporal logic as an executable semantics for planning languages. J. of Logic, Lang. and Inf. 1(16), 63–89 (2007) 6. De Bra, P., Smits, D., Stash, N.: Creating and delivering adaptive courses with AHA! In: Nejdl, W., Tochtermann, K. (eds.) EC-TEL 2006. LNCS, vol. 4227, pp. 21–33. Springer, Heidelberg (2006) 7. Felder, R.M., Silverman, L.K.: Learning and teaching styles in engineering education. Engineering Education 78(7) (1988) 8. Gansner, E.R., North, S.C.: An open graph visualization system and its applications to software engineering. Software – Practice and Experience 30(11), 1203–1233 (2000) 9. Limongelli, C., Sciarrone, F., Temperini, M., Vaste, G.: LECOMPS5: A Web-Based Learning System for Course Personalization and Adaptation. In: IADIS International Conference e-Learning 2008, Amsterdam, The Netherlands, July 22 - 25 (accepted, 2008) 10. Russel, S., Norvig, P.: Artificial Intelligence: A Modern Approach. Prentice-Hall, Englewood Cliffs (2003) 11. Weber, G., Brusilovsky, P.: Elm-art: An adaptive versatile system for web-based instruction. International Journal of AI in Education 12(4), 351–384 (2001) 12. Wolper, P.: The tableau method for temporal logic: an overview. Journal of Logique et Analyse 28, 119 (1985)
