Using Knowledge Discovery Techniques to Support Tutoring in an Ill ...

Using Knowledge Discovery Techniques to Support Tutoring in an Ill-Defined Domain Roger Nkambou1, Engelbert Mephu Nguifo2, and Philippe Fournier-Viger1 1

Université du Québec à Montréal, Laboratoire GDAC, Montréal (QC), Canada Université Lille-Nord de France, Artois, F-62307 Lens, CRIL, F-62307 Lens CNRS UMR 8188, F-62307 Lens , France [email protected], [email protected], [email protected]

2

Abstract. Domain experts should provide relevant knowledge to a tutoring system so that it can guide a learner during problem-solving learning activities. However, for ill-defined domains this knowledge is hard to define explicitly. As an alternative, this paper presents a framework to learn relevant knowledge related to procedural tasks from users’ solutions in an ill-defined procedural domain. The proposed framework is based on a combination of sequential pattern mining and association rules discovery. The resulting knowledge base allows the tutoring system to guide learners in problem-solving situations. Preliminary experiments have been conducted in CanadarmTutor.

1 Introduction An ill-structured problem is defined by Simon [15] as one that is complex, with indefinite starting points, multiple and arguable solutions, or unclear strategies for finding solutions [17]. Domains that include such problems and in which, tutoring targets the development of problem-solving skills are said to be ill-defined (within the meaning of Ashley et al. [16]). According to Aleven, Ashley, Lynch and Pinkwart [1], ill-defined domains present a number of unique challenges for researchers in Intelligent Tutoring Systems and Computer Modeling, such as “(1) defining a viable computational model for aspects of underspecified or open-ended domains; (2) developing feasible strategies for search and inference in such domains; (3) providing feedback when the problem-solving model is not definitive; (4) structuring of learning experiences in the absence of a clear problem, strategy, and answer; (5) user models that accommodate the uncertainty of ill-defined domains; and 6) user interface design for ITSs in ill-defined domains where usually the learner needs to be creative in his actions, but the system still has to be able to analyze them”. The method of cognitive task analysis that aims at producing effective problem spaces or task models by observing expert and novice users is a good solution for capturing different ways of solving problems. This supports model and knowledge tracing, coaching, errors detection, and plan recognition. However, this process is very time-consuming [2] and cannot be applied easily for ill-defined domains. Constraint based modeling (CBM) was proposed as an alternative [3]. It consists of specifying sets of constraints on what is a correct behavior, instead of providing a B. Woolf et al. (Eds.): ITS 2008, LNCS 5091, pp. 395–405, 2008. © Springer-Verlag Berlin Heidelberg 2008

396

R. Nkambou, E.M. Nguifo, and P. Fournier-Viger

complete task description. Though this approach was shown to be effective for some ill-defined domain, a domain expert has to design and select the constraints carefully. Alternatively, our proposal aims at learning a knowledge base that can replace (or represent) the problem space, from users’ interactions. It combines two knowledge discovery techniques. Sequential patterns mining is used to discover frequent actions sequences among the recorded usage of expert, intermediate and novice users. Association rules discovery finds associations between these significant actions sequences, relating them together. The goal is to use solutions from users as a source of knowledge about the ill-defined problem. In this paper, we show how the proposed framework is used to discover new domain knowledge that the tutor uses to track learners’ actions and provides them with relevant hints. The framework is applied in the CanadarmTutor [4], a simulation-based tutoring system that we have developed to teach astronauts how to operate a robot manipulator deployed on the International Space Station (ISS). During the robot manipulation, operators do not have a direct view of the scene of operation on the ISS and must rely on cameras mounted on the manipulator and at strategic places in the environment where it operates. Furthermore, for a given robot manipulation problem, there are many possibilities for moving the robot to a goal position and thus, it is not possible to define a complete and explicit task model. In fact there is no simple ‘legal move generator’ for finding all the possibilities at each step. Hence, CanadarmTutor operates in an ill-defined-domain [15]. The paper is organized as follows. First, we present some related works and their limitations. Second, we describe the framework and some techniques that are used. We then present a few possible tutoring services based on the framework. Finally, we describe an experiment of using the framework in CanadarmTutor and illustrate how the results enable CanadarmTutor to provide more realistic tutoring services.

2 Related Works Creating cognitive tutors usually rests on the implicit assumption that one should predefine a task model describing correct and incorrect solution paths. CTAT [2] offers a set of tools that allows ITS designers to specify the behavioral graph (BG) of a task, presenting correct and buggy paths. BGs (sometimes transformed into production rules) are used to track student actions. The behavior recorder can automate the translation of user actions into a BG. This concept was improved by the BND (Bootstrapping Novice Data) approach [5]. BND records the actions of many students in a log file which is then used to create a common BG that can be improved by designers. However, the BND approach is devoid of learning, reducing the approach to a simple way of storing or integrating raw user solutions into a structure, as in [6] and [7]. This is very limiting because the system does not try to extract useful knowledge from those solutions, which could enrich the problem space. In CanadarmTutor, to automatically detect errors of a student learning to operate the manipulator and to produce illustrations of correct and incorrect motions in training and give feedback to the learner, our first solution was to integrate a special path-planner based on probabilistic roadmap approach into the system. The pathplanner we developed ([4]) acts as a domain expert and can calculate the arm's moves avoiding obstacles and consistent with the best available cameras views to achieve a

Using Knowledge Discovery Techniques to Support Tutoring in an Ill-Defined Domain

397

given goal. The path-planner enables CanadarmTutor to answer several learners’ questions such as: How to..., What if…, What next…, Why… and Why not. However, solution paths provided by the path-planner are sometimes too complex and difficult to follow by the users. To sustain more effective learning, an effective problem space that captures real users’ knowledge is needed. We decided to create a partial effective solution using the cognitive tutor approach [8]. We modeled the spatial knowledge for the Canadarm manipulation task as semantic knowledge. To achieve this, we discretized the 3D space into 3D sub spaces named elementary spaces (ES). Spatial knowledge is encoded as relationships such as (1) a camera can see an ES or an ISS module, (2) an ES comprise an ISS module, (3) an ES is next to another ES or (4) a camera is attached to an ISS module. The procedural knowledge of how to move the arm to a goal position is modeled as a loop where the learner must recall a set of cameras for viewing the ESs containing the arm, select the cameras, adjust their parameters, retrieves a sequence of ESs to go from the current ES to the goal, and then move to the next ES. CanadarmTutor detects all the atomic actions like camera changes and entering/leaving an ES. It was not possible to go into finer details like how to choose the joint(s) to move from an ES to another. As [9] stated, modeling complete possible solutions for a given goal is not realistic in ill-defined domain. That is exactly the case in CanadarmTutor. The CBM approach may represent a good alternative to the cognitive tutor approach [3, 9]. However, in the CanadarmTutor context, it would be a difficult work for domain experts to describe relevance and satisfaction conditions. In fact, there would be too many conditions and many possible ideal solutions for each problem; the domain is too much complex for this approach. Contrary to these approaches, we are proposing a solution to create a more general, flexible, albeit sometimes partial, BG-like structure by inferring association rules between actions or action sequences, providing meta-knowledge to the ITS. In fact, both novice and expert domain users can provide primitive action sequences required to achieve typical tasks in the application domain. These sequences, whether good or buggy, may then be used to teach procedural knowledge associated with the task, thereby continually enhancing the system's intelligence. They can be used for supporting valuable tutoring services without the need of a clear and complete representation of problem space. We believe that, this approach represents an effective and complementary solution for tutoring ill-defined domain.

3 The Framework The framework that we propose goes through four processes to learn rules: 1) Given log files containing users’ plans, the first step consists in generating frequent sequential patterns; 2) Those patterns are used for creating a meta-context where each plan is linked with the frequent patterns that appear in it; 3) Using the meta-context, a generic base of association rules is produced; it shows how frequent sequential patterns are related; 4) The generic base and the set of frequent patterns are transformed into a knowledge base that will be used by the tutoring system.

398


3.1 Finding Frequent Sequential Patterns Using PrefixSpan The problem of mining sequential patterns is stated as follows [10]. Let D be a transactional database containing a set of transactions (here also called plans) and a set of sequence of items (called actions in our context). An example of D is depicted in table 1. Let A = {a1, a2,…, an} be a set of actions. We call a subset X ⊆ A an actionset and |X|, its size. A sequence s = (s1, s2, … , sm) is an ordered list of actionsets, where si ⊆ A, i ∈ {1,…,m}, and where m is the the size of s (also noted |s|). A sequence sa = (a1, a2,…, an) is contained in another sequence sb = (b1, b2,…, bm) if there exists integers 1 ≤ i1 < i2 < … < in ≤ m such that a1 ⊆bi1 , a2 ⊆ bi2 , . . . , an ⊆bin. The relative support of a sequence sa is defined as the percentage of sequences s ∈ D that contains sa, and is denoted by supD(sa). The problem of mining sequential patterns is to find all the sequences sa such that supD(sa) ≥ minsup for a database D, given a support threshold minsup. Consider the dataset of table 1. The size of the plan P2 is 6. Suppose we want to find the support of the sequence sa = (1 {9 31}). From Table 1, we know that sa is contained in the sequences for plan 1 and plan 3 but is not in the sequence for plan 2. Hence, the support of sa is 2 (out of a possible 7), or 0.28. If the user-defined minimum support value is less than 0.28, then sa is deemed frequent. Table 1. A Data Set of 7 Successful plans

PlanID P1 P2 P3 P4 P5 P6 P7

Sequences of actions 1 2 25 46 48 {9 10 11 31} 1 25 46 54 79 {10 11 25 27} 1 2 3 {9 10 11 31} 48 2 3 25 46 11 {14 15 16 48} 74 2 25 46 47 48 49 {8 9 10} 1234567 25 26 27 28 30 {32 33 34 35 36}

A subsequence or pattern, P, is closed if there exists no superset of P with the same support in the database. A closed pattern induces an equivalence class of pattern sharing the same closure, i.e. all the patterns belonging to the equivalence class are verified by exactly the same set of plans. Those patterns are partially ordered, e.g. considering the inclusion relation. The smallest elements in the equivalence class are called minimal generators, and the unique maximal element is called the closed pattern. Many algorithms have been proposed to efficiently mine sequential patterns or other time-related data [10], [11], [12]. We chose PrefixSpan [12] as it is one of the most promising approach for mining large sequence databases having numerous patterns and/or long patterns, and also because it can be extended to mine sequential patterns with userspecified constraints. PrefixSpan is a projection-based, sequential pattern-growth approach that recursively projects a sequence database into a set of smaller projected databases. Sequential patterns are grown in each projected database by exploring only locally frequent fragments. Table 2 shows some sequential patterns extracted by PrefixSpan from the data in table 1 using a minimum support of 25%.


399

Table 2. Examples of sequential patterns extracted by PREFIXSPAN

Sequential patterns 1 25 46 48 1 25 46 {10 11} 1 {9 10 31} 1 {9 11 31} 1 {9 10 11 31} 1 46 {10 11}

Sequence patterns’ labels S1 S2 S6 S7 S8 S13

Although a sequential pattern may not allow reaching a tutor goal in a problem-solving situation, two or more sequential patterns linked together might do so. Thus the patterns found by Prefixspan will be linked by discovering associations between them. Association rules is a powerful technique originally applied for market basket analysis that mine associations between items from a list of transactions. Since the number of extracted rules can be very high and include many redundancies, minimal and non redundant representation of association rules called generic base have been proposed such. Among previous studies on mining of generic bases, we chose IGB [13] as it efficiently extracts more compact generic bases without information loss, i.e. it has a valid and complete axiomatic system allowing the derivation of all the association rules. 3.2 Extracting Generic Rules between Patterns Using IGB IGB [13] is a new informative generic base. Its rules are correlations between minimal premise and maximal conclusion (in term of items number). It was shown that this kind of rules is the most general (i.e., conveying the maximum of information). They are two types of generic rules: (1) factual rules having an empty premise; and (2) implicative rules having a non empty premise. IGB base is generated by a dedicated algorithm which takes as input the meta-context of initial plans, the minimum support minsup (as defined in PrefixSpan), and the minimum confidence, minconf. The meta-context of initial plans (see example in Table 3) is the set of plans rewritten with the frequent sequential patterns obtained with PrefixSpan. IGB algorithm checks for each non empty closed pattern P if its support is greater or equal to minconf. If it is the case, then the generic rule Ø ÆP is added to IGB base. Else, it iterates on all frequent closed actionsets P0 subsumed by P. For those having support at Table 3. Part of the crisp meta-context of frequent sequences built from dataset in table 1

PlanID P1 P2 P3 P4 P5 P6 P7

Frequent sequential patterns S1 S2 S4 S5 S6 S7 S8 S9 S10 S95 S97 S98 S113 S116 S118 S1 S5 S6 S7 S9 S98 S1 S2 S3 S4 S5 S6 S8 S10 S95 S97 S98 S113 S116 S118 S2 S3 S6 S7 S9 S10 S2 S4 S5 S7 S9 S10 S95 S1 S2 S3 S7

400


least equal to supD (P)/minconf, the algorithm iterates on the list of minimal generators associated to P0. During this iteration, we look for the smallest minimal generator gs, such that there does not exist a generator g0 subsumed by gs which is already inserted in the list L of smallest premises. Then, IGB algorithm iterates on all elements of the list L to generate rules of IGB which have the following form: gs Æ (P – gs). By dividing the sub-sequence occurrence by the plans’ occurrence, we obtain the relative support associated to the sub-sequence. Let us consider a minsup of 2 (25%), meaning that a valid sequence should occur in at least 2 input-plans, we can obtain the meta-context which part is shown in table 3. Each sub-sequence can appear in a plan with a certain confidence which is its relative support (in table 3, we consider a crisp context where dichotomic values (0 or 1) are assigned when a subsequence appears or not in a plan). Using this meta-context as input, IGB computes a set of generic meta-rules, part of which is shown in table 4. These meta-rules combined with frequent sequential patterns will constitute the knowledge that will be used to support tutoring services. Table 4. Examples of generic meta-rules extracted by IGB

Meta-rules S10 ===> S9 S9 ===> S7 S9 ===> S5 S5 ===> S10

Support 4 4 4 4

Confidence 0.8 0.8 0.8 0.8

Expanded meta-rules … 1 {10 31} ===> 1 {9 11 31} … …

4 Supporting Tutoring Services Using Learned Rules As mentioned before, tutoring systems should provide useful tutoring services to assist the learner, such as coaching, assisting, guiding, helping or tracking the students during problem-solving situations. To offer these services, a tutoring system needs some knowledge related to the context. The knowledge base namely procedural task knowledge (PTK) obtained from the framework serves to that end. The next paragraphs present some examples of services that can be supported. Assisting the User to Explore Possible Solutions. Domain expert users can explore, validate or annotate the PTK. The validation consists of removing all meta-rules with a low confidence, meaning that those rules can not significantly contribute to help the student. Annotation consists of connecting some useful information to meta-rules lattice depicting semantic steps of the problem as well as hints or skills associated to a given step. A meta-rule lattice annotated in this way is equivalent to [2]’s BN, except that BNs are built from scratch by domain experts. For student users, exploring PTK will help them learn about possible ways of solving problem. They can be assisted in this exploration using an interactive dialog with the system which can prompt them on their goals and helps them go through the rules to achieve these goals. This kind of service can be used when the tutoring system wants to prepare students before involving them in real problem-solving situation.


401

Tracking the Learner. Plan recognition is very important in tutoring systems. PTK is a great resource to this process. Each student’s action can be tracked by searching the space defined by meta-rules lattice so that we can see the path being followed by detecting patterns that the learner follows. Guiding the Learner. When solving a problem, a classic situation is when the student asks the tutor what to do next from the actual state. PTK allows the tutor to produce the next most probable actions that the student should execute and prompt him on that, taking into account uncertainty related to rules’ confidence and other possible patterns that match with the current state.

5 Experiments and Results We have set up two scenarios consisting each of moving the load to one of the two cubes (figure 1a). A total of 15 users (a mix of novices, intermediates and experts) have been invited to execute these scenarios using the CanadarmII robot simulator. A total of 155 primitive actions have been identified. Figure 1b shows part of an example log file from a user’s execution of the first scenario.

… EnterCorridor(C1) {SelectCamera(Monitor1,CP8), SelectCamera(Monitor2,CP10), SelectCamera(Monitor3,CP9)} SelectJoint(WE) bigMove(WE,decrease) LeaveCorridor(C1) smallMove(WE,decrease) SelectJoint(SP) bigMove(SP, decrease) SelectCamera(Monitor1,CP2) smallMove(SP,decrease) EnterCorridor(C2) mediumMove(SP,decrease)

Fig. 1. (a): Environment setup for the two plan’s database. (b): An entry of experimental scenarios.

We obtained a database with 45 entries each corresponding to a given usage of the system. A value indicating the failure or success of the plan has been manually added at the end of each entry. The framework presented in section 3 was applied. A unique number was assigned to each action. After coding each entry of the traces database using PrefixSpan data representation, we obtained a binary file containing plans’ data for the two scenarios. This file was sent as input to the rest of the process. After executing PrefixSpan, the first stage of the experiment consisted of finding sequential patterns from the input data, we obtained a total of 76 significant patterns (with a support greater than .5) for the first scenario, and 82 for the second scenario. At the second stage, we created a binary context where each row represents a plan

402


data and each column stands for a set of patterns. The goal at this stage was to mine association rules between sequential patterns. Because all the sequential patterns shared the same first few actions and because they are closed patterns, initially the IGB approach did not find any rules. To overcome this difficulty, we filtered the patterns to remove the first common actions to all plans. Then, we regenerated the sequential patterns using PrefixPan. Using the IGB approach with a minimum support of 0.2, we obtained a PTK of 37 meta-rules. One such meta-rule (rule #31) connects the two following patterns. The first pattern (pattern #25) is to select the CP6 camera, which gives a close view of the arm in its initial position, slightly decrease the yaw of the CP6 camera to have a better view, select the WE joint and decrease a little bit its rotation value. The second pattern (pattern #55) was to select the SP joint and decrease its rotation value. These two patterns constituted a possible safe and effective strategy to move the arm toward the goal. Moreover, the confidence level of the rule (0.8), its relative support (0.25), and its expertise level annotation (“intermediate”) indicated respectively that the second pattern is a common follow-up to the first pattern, that the rule is widespread among users, and that this rule is usually employed by intermediate users. These rules were then coded and integrated in a new version of CanadarmTutor that uses this knowledge base to support tutoring services. To recognize a learner’s plan, the system proceeds as follows. The first action of the learner is compared with the first action of each frequent pattern in the PTK. If the actions do not match for a pattern, the system discards the pattern. Each time the learner makes an action, the system repeats the same process. It compares the actions done so far by the learner with the remaining patterns. When a complete pattern has been identified, the software looks for association rules that link the completed pattern to other patterns that could follow. Because the generated PTK is partial, it is possible that at any given moment a user action does not match with any patterns. If this situation arises, the algorithm tries two possibilities, which are ignoring the last user action or ignoring the current action to match for each pattern. This heuristic rule makes the plan recognizing algorithm more flexible and has shown to improve its effectiveness. Furthermore, a marking scheme has been implemented to detect and avoid association rules cycles. One utility of the plan recognizing algorithm is to assess the expertise level of the learner (novice, intermediate or expert) by looking at the rules and patterns s/he applied. The plan recognizing algorithm also plays a major role in the CanadarmTutor tutoring service for guiding the learner. It allows determining the possible actions from the current state according to the PTK. This functionality is triggered when the student selects “What should I do next?” in the interface menu. The algorithm returns a set of possible actions with the associated pattern(s) or rule(s). The tutoring service then selects the action among this set that is associated with the rule or pattern that has the highest utility value, and that is the most appropriate for the estimated expertise level of the learner. Whereas the utility value of a pattern is the pattern’s relative support, the utility of a rule is calculated as the product of the relative support and the confidence of the rule. For example, the utility value of rule #31 is 0.2 (0.8 * 0.25). Utility values and expertise levels ensure that in every case the likely most useful action is proposed to the user. In the rare cases where no actions can be identified, the system asks the FADPRM path planner to generate a path to go from the current configuration to the goal.


403

Figure 2 illustrates a hint message given to a learner upon request during scenario 1. The guiding tutoring service recognized that the student carried out pattern #25, that rule #31 matched with the highest utility value, and that rule #31 correspond with the estimated expertise level of the learner. Therefore, the system suggested pattern #55, which is selecting the SP joint and decreasing its rotation value. By default, two steps are showed to the learners in the hint window depicted in figure 2. However, the learner can click on the “More” button (fig. 2) to ask for more steps or click on the “another possibility” button to ask for an alternative. The sentences in natural language depicted in figure 2 to describe the actions of pattern #55 are an example of tutoring resources that can be used to annotate the PTK

. Fig. 2. A hint generated by the guiding tutoring service

An empirical test with this version has been conducted with the same users of the system’s version relying on FADPRM. We found that the system behavior in terms of guiding the user during the two scenarios significantly improved compared to the behavior observed in the version relying solely on the path planner. The system can now recommend good and easy-to-follow actions sequences. The system can also recognize users’ plans and anticipate failures or successes, thus infer user profiles by detecting the path they follow. The PTK produced by our framework is sometimes too large and contains non useful rules (because of the amount of sequential patterns). However, this is not harmful for the tutor behavior but it may slow the performance as the system need to go through this huge knowledge base each time the user executes an action. We are now working to improve the quality of the PTK. We are also looking for a way of managing unsuccessful plans data. We believe that this could allow better learning guidance, as the tutor could easily identify sequence patterns that lead to failure or success.

6 Conclusion We proposed a knowledge discovery framework to learn procedural knowledge associated to a task. We showed how the framework contributes to enhance an intelligent tutoring system’s domain knowledge in an ill-defined procedural domain. The experiment showed that this enables CanadarmTutor to better help learners. Prefixspan and IGB are parameters of our framework proposal, and thus can be replaced by other convenient tools. Since these tools and the input and output of the

404


framework are domain independent, the framework can be potentially applied to any ill-defined procedural domains where the problem can be stated in the same way. For future works, it would be interesting to find some ways of filtering the resulting meta-rules and integrating unsuccessful paths. The work of Kum et al. [17] provides some suggestions on a model that may help to filter sequential patterns. Different proposals have also been made on qualitative measures of association rules. We will also carry out further tests to clearly measure the benefit of the approach in terms of tutoring assistance services. Another important aspect that we will focus on is binding users’ skills with discovery sequences. In fact, we found that, it will be interesting to compute a subset of skills that characterized a pattern by finding common skills demonstrated by users who used that pattern. This will allow a thorough cognitive diagnosis of missing and misunderstanding skill for the users who demonstrate part of that pattern and therefore, help them to acquire correct skill to be able to solve the goal. Acknowledgments. Our thanks go to the Canadian Space Agency, the NSERC (Natural Sciences and Engineering Research Council) for their logistic and financial support. The authors also thanks the current and past members of the GDAC and PLANIART research teams who have participated to the development of the CanadarmTutor. We also thank the authors of PrefixSpan and IGB for providing their programs.

References [1] Aleven, V., Ashley, K., Lynch, C., Pinkwart, N.: Proc. of the Intelligent Tutoring Systems for Ill-Defined Domains Workshop. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053. Springer, Heidelberg (2006) [2] Aleven, V., McLaren, B.M., Sewall, J., Koedinger, K.: The Cognitive Tutor Authoring Tools (CTAT): Preliminary evaluation of efficiency gains. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 61–70. Springer, Heidelberg (2006) [3] Mitrovic, A., Mayo, M., Suraweera, P., Martin, B.: Contraint-based tutors: a success story. In: Proc. of the Industrial & Engineering Application of Artificial Intelligence & Expert Systems, pp. 931–940 (2001) [4] Nkambou, R., Belghith, K., Kabanza, F.: An Approach to Intelligent Training on a Robotic Simulator Using an Innovative Path-Planner. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 645–654. Springer, Heidelberg (2006) [5] McLaren, B., Koedinger, K.R., Schneider, M., Harrer, A., Bollen, L.: Bootstrapping Novice Data: Semi-Automated Tutor Authoring Using Student Log Files. In: Proc. of the Workshop on Analyzing Student-Tutor Logs. ITS 2004 (2004) [6] Blessing, S.B.: A Programming by Demonstration Authoring Tool for Model-Tracing Tutors. In: Murray, T., Blessing, S., Ainsworth, S. (eds.) Authoring Tools for Advanced Technology Learning Environments: Toward Cost-Effective Adaptive, Interactive and Intelligent Educational Software, pp. 93–119. Kluwer Academic Publ. (2003) [7] Jarivs, M., Nuzzo-Jones, G., Heffernan, N.T.: Applying Machine Learning Techniques to Rule Generation in Intelligent Tutoring Systems. In: Proc. of ITS 2006, pp. 541–553 (2006)


405

[8] Fournier-Viger, P., Nkambou, R., Mayers, A., Dubois, D.: Automatic Evaluation of Spatial Representations for Complex Robotic Arms Manipulations. In: Proc. of ICALT 2007, pp. 279–281 (2007) [9] Mitrovic, A., Koedinger, K.R., Martin, B.: A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling. In: User Modeling 2003, pp. 313–322 (2003) [10] Agrawal, R., Srikant, R.: Mining Sequential Patterns. In: Proc. of the Int. Conf. on Data Engineering, pp. 3–14 (1995) [11] Zaki, M.J.: SPADE: An Efficient Algorithm for Mining Frequent Sequences. Machine Learning Journal 42(1-2), 31–60 (2001) [12] Pei, J., Han, J., et al.: Mining Sequential Patterns by Pattern-Growth: The PrefixSpan Approach. IEEE Trans. Knowledge and Data Engineering 16(10), 1–17 (2004) [13] Gasmi, G., Yahia, S.B., Mephu Nguifo, E., Slimani, Y.: A new informative generic base of association rules. In: Proc. Ninth Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 81–90 (2005) [14] Kum, H., Chang, J.H., Wang, W.: Benchmarking the effectiveness of sequential pattern mining methods. Data and Knowledge Engineering (DKE) 60(1), 30–50 (2007) [15] Simon, H.A.: Information-processing theory of human problem solving. In: Estes, W.K. (ed.) Handbook of learning and cognitive processes, vol. 5, Human information (1978) [16] Lynch, C., Ashley, K., Aleven, V., Pinkwart, N.: Defining Ill-Defined Domains; A literature survey. In: Proc. of the Intelligent Tutoring Systems for Ill-Defined Domains Workshop ITS 2006, pp. 1–10 (2006) [17] Fields, A.M.: Ill-structured problems and the reference consultation: The librarian’s role in developing student expertise. Reference services review, Emerald 34(3), 405–420 (2006)