Dynamically Adapting Training Systems Based on User Interactions∗ R. M. Weerakoon
P.Chundi
Computer Science Department University of Nebraska-Omaha
Computer Science Department University of Nebraska-Omaha
[email protected] [email protected] M.Subramaniam University of Nebraska-Omaha Computer Science Department
[email protected] ABSTRACT Game-based simulation systems are increasingly being used to train users in several applications across government, industry, and academia. Designing game-based training systems that can measurably improve learning while providing an engaging training experience is a challenging problem. In this paper, we describe a novel framework that tightly integrates game-based training systems with instructional components using data analysis to address this problem. Intelligent training systems based on this framework dynamically adapt both the training and the instructional components to measurably improve learning in play sessions. We propose a three phase approach to automatically identify points in a play session to predict high-value future scenarios, validate predictions, and prescribe actions. A case study using the KDD Cup 2010 educational data set is described illustrating the effectiveness of the proposed approach.
Keywords Simulation, Adaptation, Sequence mining
1.
INTRODUCTION
Modeling and simulation have a long history in providing users hands-on learning in realistic and risk-free environments. The recent advent of serious computer games has lead to a resurgence of game-based simulation systems to train users across several domains including emergency and first-responder services, military, health-care, corporate as well as school and college education and many others [1, 5, 13]. The ability of these immersive game-based training systems to engage and motivate users to master skills is ∗This work is partially supported by NSF grant DUE1044627.
becoming increasingly evident [2]. Designing game-based training systems that foster measurable learning without compromising the engaging qualities of these systems is a challenging activity. Most of the current game-based training systems [5, 1], do not contain an inbuilt instruction component and hence provide minimal support for measuring learning progress. Even systems where instructional support is explicitly present, the training and the instructional aspects are combined in an ad hoc manner and do not adequately exploit each others capabilities to provide training with measurable learning outcomes. In this paper, our overall objective is to aid the design of game-based training systems incorporating rich instructional components to provide training with measurable learning outcomes. Towards this goal, we propose a novel framework that combines game-based training systems with data analysis methods to customize training to measurably improve learning. The data analysis methods used are inspired by the impressive successes obtained in the field of educational data mining and intelligent tutoring systems [11]. The proposed framework integrates a game-based training system with a data mining system to develop what we call an intelligent training system (ITS). In ITS, the training scenarios as well as the data analysis functions are dynamically adapted based on the observed user behavior. A typical user play session on the ITS proceeds in three phases – prediction, validation, and prescription. In the prediction phase, training scenarios are generated based on the current play configuration, user responses are recorded and processed to move to the next play configuration. The recorded user responses are analyzed to determine if the responses have sufficient and meaningful information about a player learning behavior. Once this is ascertained, (multiple) hypotheses about future scenarios that are likely to improve future learning are formulated. In the validation phase, the training system is dynamically adapted to generate scenarios that can aid in validating the available hypotheses. The mining system is adapted to evaluate the user responses in the context of the available hypothesis to identify the valid hypotheses, if any, and rank them. The ITS enters the prescription phase once one or more of the hypothesis are val-
idated. The training system generates scenarios based on the validated top ranked hypotheses and the mining system measures improvement in learning and signals a transition of the ITS to the prediction phase once the improvement in learning stabilizes. The proposed framework achieves a close integration of the training and the mining systems. Not only that the mining system results are used to dynamically adapt the training content and the presentation of the scenarios but we also leverage the trainer to validate mining predictions on-thefly. Dynamically adapting the content and presentation in an ITS is a lot more challenging since this could require arbitrarily changing simulation time and state including rolling back to previous visual presentation with the appropriate context. In this sense, ITSs are significantly different from intelligent tutoring and other adaptive learning systems [12]. Further, in the latter systems, usually, the user behavior predicted by the mining systems are used to adapt the learning content and the effectiveness of the predictions is validated externally, possibly with manual help. The threephase based approach takes this one step further by dynamically changing mining based on the ITS phase. We illustrate the proposed ITS framework using the KDD Cup data set from 2010. The KDD Cup 2010 featured an educational data mining competition where the participants were asked to analyze students past behavior and predict their future performance on algebra problems. We model each problem solving activity by a student as a play session in the ITS. The individual steps of the problem correspond to individual training scenarios in the ITS. Our preliminary results on this case study are highly promising. They show that the proposed framework can potentially provide a targeted and measurable training experience in solving the problems in this data set. The rest of this paper is organized as follows. Section 1 describes the design model of ITS in terms of game-based training and data analysis. An overview of the ITS framework is presented. Section 2 gives the system architecture of the ITS. Section 3 illustrates the ITS framework on the KDD 2010 data set. Section 4 concludes the paper.
2.
DESIGN FRAMEWORK
In this section we describe the proposed design framework for ITS. After a brief overview of game-based training and mining systems, the framework underlying ITS is described.
2.1
Game-based Training
Game-based training systems have a long history in providing training for mission critical applications. Game-based training is becoming more widespread and is being used to train students in liberal arts, sciences, and technology. Large scale collaborative game-to-teach projects at the MIT comparative media labs and the Wisconsin Educational Technology have developed a number of games in the areas of history (Revolution), electricity and magnetism (Supercharged ), and others [5, 13]1 . Inspired by these works, the last author, working with construction engineering domain experts, has 1
More details are in http://www.educationarcade.org/
recently built a game based training system, VICE (Virtual Instruction in Construction Education) [3] that provides project-based education to construction engineering students. The VICE system is built using Adobe Flash CS3 professional version 9.0. with defined preference rules. Various multimedia resources including sound, images, animation, interactive flash, and other 3 dimensional construction objects created using VICO software’s constructorT M 2 are connected as interactive events in VICE. The VICE system has been well received by the construction engineering academia [4] and industry [9]. Usually, game-based training systems support a number of game stories with varying game objectives. For instance, one of the objectives in the Supercharge [5] game is to successfully navigate a charge through a field of positive and/or negative charges. In VICE, an objective in one of the game stories is to construct a house for a given set of requirements while making optimal use of available resources and time. Typically, players can customize the game objectives in these systems for a more targeted training experience. In each play session of a game-story, a player (or a team) chooses roles (e.g: student, teaching assistant, instructor, architect), selects degree of difficulty (e.g: novice, medium, expert), learning modes (e.g: supervised, unsupervised) to start playing. Each play session consists of players interacting with a sequence of visual media-rich scenarios to achieve the chosen objectives. Similar to the popular video games, there is no explicit adversary in these game stories. Player response to each scenario is scored based on its contribution towards achieving the objective at hand. Cumulative score obtained at the end of a play session rates a player’s performance in that session. Scores used in these games usually measure learning at a very high level and often provide limited insight about the strengths and weakness of a player. Game-based training systems typically codify domain knowledge in a knowledge base consisting of i) a set of knowledge components with enabling pre-conditions and effects specified by post-conditions, and ii) a knowledge map describing the interdependency between the knowledge components. Each game story is modeled as a problem solving activity. The feasible solutions to each problem are limited to those that can be built using the available knowledge components. These solutions are stored in the form of a template called a solution plan, a sequence of play situations. User responses are used to move from one play situation to the next. In each play session, based on parameters, a solution plan is first extracted from the knowledge base by identifying the initial and final situations for the problem at hand. Each extracted plan is customized based on values of the parameters such as the learning modes, player roles, to create what is called an actionable solution plan(s). An actionable solution plan annotates the solution plan with all possible responses that a player may provide to move from one situation to the next one in the plan. Each path in the actionable solution plan from the initial to the final situation denotes one feasible solution to the problem. All the possible responses connecting any two situations are ranked 2 VICO provides construction software and services to commercial industry.
in terms of their contribution to the game objectives. In order to play, the media-rich scenarios are generated for each play situation, the player responds to the scenario reaching the next play situation, obtains a score computed based on the rank assigned to that response in the actionable solution plan. The play session ends upon reaching the final situation with the final score. To obtain the highest score, a player (team) must choose the top ranked response in every situation starting from the initial to the final one in the actionable solution plan. Note that in training systems like VICE that model complex construction projects, the solution space too large to obtain the complete solution plan at any step. Usually, incremental planning is performed by extracting a partial plan up to certain depth and then extracting the next one based on the situation reached at the end of the first plan and so on. Replanning is performed whenever user actions deviate from the existing solution plans.
2.2
Mining Responses in Training Systems
As users train with systems such as VICE, the system typically records the scenarios that are presented to the users along with user responses. The recorded data can be analyzed using data mining techniques to measurably improve learning. Applying data mining to educational systems including intelligent tutoring and adaptive learning systems is an emerging research activity. Educational data mining methods have been proposed to analyze the data collected from educational systems (see [11] for an excellent survey.). According to [11], there are numerous applications of data mining to the educational data – providing feedback to the instructors, predicting student success, identifying student groups based on learning patterns, developing concept maps, to name a few. In particular, the student usage data has been analyzed in e-learning environments to validate and/or evaluate educational systems and to improve the quality of education received by students. A related goal of these approaches has been to learn the student model to predict their learning behavior. Many data mining methods, such as clustering, association rule mining, text mining, and sequence mining have been used for analyzing the data. Sequence mining techniques that can analyze sequence of user responses to scenarios can be immensely beneficial in developing training systems with measurable learning. In this context, Kock and Paramythis in [8] describe an approach based on automatic clustering to analyze sequence of user responses to identify learner profiles, problem solving styles and concrete problem solving patterns. In [8], the authors identify the problem of integrating these analysis techniques in the adaptation cycle of an educational system and mention several potential directions for achieving such an integration. In this paper, we have presented a simple alternative approach for tightly integrating sequence data analysis techniques into the adaptation cycle. Further, the data analysis techniques in this paper are based on notions such as stability and entropy measures of sequences.
2.3
Intelligent Training System (ITS)
The learning aspects of game-based training systems can be vastly improved by using a mining system in the loop so that user responses can be analyzed to measurably study player learning. Below, we first introduce some simple measures used for response data analysis. Then, we describe our framework that combines training and mining systems by dynamically adapting both during a play session. Each recorded user response includes information about the knowledge component(s) of the scenario, the overall result of the response ( correct/wrong, optimality score), and the mode of interaction (including elapsed time, hints taken) and so on. We define a benefit function with these recorded fields as the inputs. The function outputs a numeric value in the range (−1, 1) that measures the contribution of this scenario to user’s learning based on the observed response. A benefit value close to 1 indicates that the user is likely to be proficient in handling this scenario whereas that close to -1 indicates scope for further improvement in handling the concerned scenario. Given a sequence of recorded user responses having the parameter values described above, we define a cumulative benefit function with the sequence of tuples of values as inputs. The function outputs a numeric value in the range (-1, 1), which has the same meaning as the benefit function. The cumulative benefit function for a sequence is simply the average of the benefit function values of the individual elements of the sequence. More detailed information regarding the benefit associated with a sequence of user responses is represented using a benefit vector, a list of triples where the first element of each triple specifies the knowledge component, the second element specifies the number of occurrences of (a scenario based on) this component, and the last element is the average contribution of these occurrences to the cumulative benefit of the entire sequence. We say a sequence of user responses is discernable if there exist elements in the sequence whose benefit value dominates that of the remaining elements. A sequence where all elements occur uniformly or vary minimally are not discernable sequences. Usually, sequences where most frequently occurring elements make a large positive contribution to the cumulative benefit or a large negative contribution are meaningful sequences for predicting user behavior. Well known information theoretic measures such as entropy can be used to identify discernable sequences. If a sequence of responses has a sufficient low entropy value then the associated benefit vector is used to analyze the contributions of the elements to the cumulative benefit. In order to meaningfully integrate data analysis with training, first, we need to ensure that the usage of the training system has reached a stage where analysis of user responses can produce meaningful results. For instance, the initial stages of a play session may be spent by users in getting familiar with the system and the concepts. Data collected during such a stage may be noisy to produce meaningful results. Usually, this means that the system usage must reach some sort of steady-state for analysis. In our integrated framework, we leverage data analysis to continually moni-
tor the cumulative benefit to determine whether or not the training system has reached a steady-state. Data analysis is also used to check if the associated sequence is a discernable sequence. Once a discernable sequence is obtained then the benefit vector is used to identify knowledge components that significantly contribute (either positively or negatively) to the current state of learning. Based on the occurrences of the scenarios based on these components, hypotheses (usually several) about future scenarios based on these components that are most likely to lead to improvement in learning are formulated. These hypotheses are used as additional constraints by the training system to generate relevant scenarios whose user responses can aid in validating the hypotheses. The responses to these generated scenarios are recorded and analyzed to identify valid hypotheses, which are ranked to determine high-value predictions. Based on these highvalued predictions, prescriptions for generating high-valued scenarios to improve future learning are generated and used to conduct training until learning improvements continue. Once the learning reaches a steady-state, the process repeats. Note that as long as the training system does not reach a steady-state of learning or does not produce a discernable sequence of responses, the training and the associated analysis are continued as usual. It is also possible that none of the hypotheses that are formulated are validated based on the scenarios generated. This is determined based on the formulated hypothesis. In such cases new hypothesis are formulated based on the observed responses if possible or the entire process is re-started. In rare situations, where a single hypothesis is formulated, we can skip validation, and move directly to prescription. By exploiting the synergy between simulation and data analysis, ITS can automatically fine tune the interaction parameters unlike many of the current tutoring systems. Further, unlike ITS, tutoring systems do not have adequate sequence mining capabilities and hence cannot identify a steady-state of user responses. Finally, most of these systems do not perform on-the-fly hypothesis validation and require human intervention for the same.
3.
ITS – SYSTEM ARCHITECTURE
The ITS system architecture (Figure 3) has a modular structure with four basic components namely: (a) game-based trainer, (b) sequence miner, (c) integration layer, and (d) knowledge bases. This is a generic architecture and can be used to create different ITSs across different domains by integrating the corresponding game-based trainers interacting with the codified domain knowledge. The sequence miner component can also be customized with appropriate benefit, cumulative benefit and other measure functions to perform domain specific user response. The UCM (User Interface/Control object /Model layers) is a scalable architecture [6, 7] based on the well-known MVC (model-view-controller) architecture [10] that has been commonly used to design game-based trainers and simulators. The UCM architecture separates the modules in a trainer into three layers, namely i) interface layer that controls in-
terface elements such as buttons, dials, gauges, equipment, movie clips, and so on, ii) Control layer that contains mechanisms for coordinating the interface elements and mediates the user interface with the functionality of the simulation model, and iii) model layer that includes computations and processes required to realize the functionality in the simulation model. The interface layer communicates with the control layer which communicates with the model layer. The communication is achieved by exchange of messages. Typically, during a play session, a user response to a scenario is captured by the interface layer in terms of the values of the different interface elements (and their states) and communicated to the control layer. The control layer interprets the responses based on the state of the relevant interface elements and generates the appropriate actions to be performed on the simulation model. The model layer incorporates these actions into the current model configuration in the available solution plan and moves the model to the next configuration. The actions are used by the scorer module to generate a score for the user response. The score and scenario corresponding to the new model configuration is sent to the control layer, which based on the state of the interface elements generates updates to display the next scenario to the user to repeat the process. At the start of the play session, the initial parameter values of the session set by the user at the interface sent to the control layer are communicated to the model layer to create the initial model configuration, which is then used to extract the appropriate solution plan for the play session. The sequence data miner component comprises of a collection of processes and computations used to perform data analysis of user responses and correlate them to the knowledge components. It comprises of two modules, namely, i) data, noise, and stability analyzer and ii) a hypotheses generator. The first module comprises of a library of customizable routines to measure benefit of individual response, cumulative benefit, and discernibility of a sequence of responses, as well as routines to compute benefit vector, and others. The hypothesis generator module contains routines that can formulate a set of hypotheses based on a sequence of user responses. It also has routines that can validate a set of hypotheses against a sequence of user responses and check the conformance of user responses against a set of validated hypotheses. The specific routines used in a particular instance of an ITS depend on the application domain. The integration layer, seamlessly combines the game-based trainer and the sequence data miner to create an intelligent training framework where the trainer is controlled by the learning outcomes determined the miner and the miner actions are controlled in turn by the user responses to the simulation scenarios. The integration layer comprises of two main modules, namely, the user response data base and the phase controller. The user response data base records user responses to each scenario along with the associated knowledge components used to generate the scenario, and the benefit and other related function values computed by the sequence miner. Accesses to the user response data base are performed by the sequence miner and are managed by the integration layer.
Intelligent Training System Game-based Trainer
Model Updates
Solution Plan
Model Layer
Interface values
Control Layer
Response
Situation & Ranked Responses
Interface Updates
Interface Layer
Game Scenario
KB Correlated Response Logs
Configuration
Scorer
Integration Layer
Phase Controller Predict
Hypothesis Generator
Sequential Data Miner Stability and Noise Detector
Stable?/ Reset
Stable?/ Hypos
Validate
Prescribe Resolution ?/ Prediction
Figure 1: Intelligent Training System Architecture The control layer communicates the relevant aspects of a user response and the score computed to the integration layer which relays them to the sequence miner, which analyzes this data with other historical data, computes the required benefit and other related values and returns them to the integration layer to update the user response data base.
The behavior of the sequence miner in the validation phase is different from that in the prediction phase. Each user response to a scenario with the concerned hypothesis is analyzed to determine the trend, if any, in the cumulative benefit due to this scenario. Hypotheses showing significant promise for improving learning objectives are considered validated and are ranked.
The phase controller module co-ordinates the operation of the entire ITS. It divides the operation of the ITS into three phases, namely, prediction, validation, and prescription phases. The phase controller starts an ITS in the prediction phase. In this phase, scenarios are generated by the trainer based on the problem chosen for the play session and scored without any interaction with the miner. The miner in this phase analyzes each user response until it detects a stable and discernable sequence of responses. Upon this, the miner formulates hypotheses predicting future scenarios that are most likely to improve learning.
Upon, determining a validated set of hypotheses, the phase controller transitions the ITS to the prescription phase. In the prescription phase, the trainer is adapted to generate scenarios based on the top ranked hypotheses. The sequence miner has the same functionality as done in the prediction phase and causes the ITS to transition to the prediction phase upon reaching a stable sequence of user responses. A stable sequence of user responses in the prescription indicates that the prescription has produced the desired effect or that it has worn off. In either case, ITS enters the prediction phase to search for new prescriptions.
After communicating the hypotheses to the control layer in the trainer, the phase controller transitions the ITS to the validation phase. In the validation phase, the hypotheses affect changes to both the model and the interface layer. First, in the model layer, the knowledge components referred to in the hypotheses are identified. User responses in the solution plans that are not connected with these knowledge components are pruned so that scenarios based on these knowledge components are only generated. The pruned actions are communicated to the control layer which then suitably adapts the interface elements. This may involve adding/removing certain simulation elements.
4.
CASE STUDY
The KDD Cup 2010 featured an educational data mining competition where the participants were asked to analyze students past behavior and predict their future performance on algebra problems. The data set contained summaries of the logs of student interaction with an intelligent tutoring system. Of the two data sets provided we use the Algebra 2008-2009 data set in this paper. There are around 9 millon student interaction steps in the ITS data set. Each user interaction is recorded using the fields – student ID, problem hierarchy consisting of step name, problem name, unit name, and section name, along with the knowledge components
used in the problem, and the number of times a problem has been viewed, whether the student was correct on the first attempt for this step (value 0 stands for incorrect and value 1 stands for being correct), the number of hints requested, and the duration taken for the step. To use the KDD data set for the ITS framework, each problem in the data set was mapped to a play story of the ITS. Each play session of the ITS corresponds to a student’s attempt to solve the given problem. This requires the student to solve the sequence of steps belonging to the problem. Each step of the problem corresponds to one visual scenario of the trainer. To start a play session based on the chosen play story, the trainer retrieves a solution plan from the knowledge base containing problems and their solutions. As described earlier the solution plan is a sequence of situations where each situation corresponds to a step of the problem of the KDD data set. The actionable solution plan where all the possible user actions to move from one situation to the other are added to the solution plan is relatively simpler for the KDD data set. In this case the actionable solution plan is generated from the solution plan by adding just two actions between any two successive situations – an action for correctly solving the situation and an action for not solving it correctly. The primary learning objective in the KDD data set is to solve each problem correctly. Consequently, the user actions corresponding to correct solutions are ranked higher than the incorrect ones. A user response to this scenario obtains a score N = 1 if the response is a correct solution to the step and obtains a score N = 0 otherwise. Corresponding to each situation in the actionable solution plan the trainer generates the corresponding visual scenario by suitably updating the interface elements such as the timer, the hint and answer buttons, and other relevant equipment such as calculator and protractor). User responses including the chosen answer, the number of hints taken and duration spent in the scenario are extracted, logged in the user response data base, and used to perform analysis by the sequence miner. The ITS starts with the prediction phase and transitions into the other two phases as described earlier based on the analysis results. Consider a scenario Si involving the set of knowledge components Ki whose user response taking Di duration (positive number), and Hi hints (non-negative number) has received a score Ni . The benefit function for the scenario and the response is ½ 1/(Diα + Hiα ) if Ni = 1 Bi (Si , Ki , Di , Hi ) = −1 + 1/(Diα + Hiα ) if Ni = 0 where α is a positive number, a user specified threshold denoting the impact of duration and hints for solving a scenario. Given a sequence of scenarios S = [S1 , · · · , Sn ] with benefit values [B1 , · · · , Bn ] of scenarios in S, the cumulative benefit of S is the average of the benefit values B1 , . . ., Bn . We use CBi,j to denote the cumulative benefit of the subsequence of S containing scenarios Si through Sj . We
use |S| to denote the number of scenarios in S. Let {K1 , . . . , Kp } be the set of knowledge components of the scenarios in S. In this case, p ≤ n. That is, each scenario belongs to a different knowledge component in which case p = n. In case when some knowledge components repeat, then p < n. Let m be a positive number less than n and β, 0 < β < 1 be a user specified threshold. Sequence S is m-stable if there exists u (1 ≤ u ≤ n - m) such that |CBu,j - CBu,j+1 | ≤ β, for all j = u, · · · u + m. To find discernable sequence, we use the notion of entropy. The entropy of a sequence S is computed using the probabilities of knowledge components appearing in S (|S| = n). Suppose each knowledge component Ki appears with P a frequency of f ki in S so that pi=1 f ki = n. Then, the entropy of S is simply −(f ki /n) × log(f ki /n). Sequence S is a discernable sequence if the entropy of S is below is a user specified threshold. In general, entropy of a sequence where each scenario belongs to a distinct knowledge component is higher than those where scenarios belong to a few knowledge components. Let S be a m-stable and discernable sequence of scenarios. For each knowledge component Ki in S, let f ki be its occurrence frequency and CBi be its cumulative benefit. Based on these two values, we can determine the contribution of each knowledge component to a student’s learning. A knowledge component with a high occurrence or f ki value, and/or high benefit CBi value in S can be considered a valuable scenario for a student. Let Ki be a knowledge component. If Ki has a low occurrence in S then the ITS ignores Ki due to lack of actionable data. On the other hand, if Ki has a high occurrence, then the ITS hypothesizes a plan for increasing the future benefit of these knowledge components by considering Ki cumulative benefit value. If Ki has a low cumulative benefit, then the ITS takes the action to improve the familiarity with this scenario by increasing the frequency of this scenario, i.e; generate more play sessions where this scenario is included. If Ki has a high cumulative benefit, then the ITS poses an increased challenge to the student by requiring the student to perform this scenario in smaller duration and with fewer hints. Let KH denote all the knowledge components in S with a high occurrence value. Now, the ITS is in validation phase monitoring the scenarios once again. In this phase, for each knowledge component in KH , benefit is calculated by using a higher value of α if the knowledge component had a high cumulative benefit in the prediction phase (increased challenge by penalizing hints and longer duration). A smaller value of α is used if the knowledge component had a low cumulative benefit value. The ITS identifies another m-stable and discernable sequence S 0 . The knowledge components in KH are examined for their cumulative benefit. Let Ki ∈ KH . If the cumulative benefit value of Ki in S 0 is higher than that of S, then the hypothesized plan is successful for Ki . If the hypothesized plan does not increase the cumulative benefit of any of the knowledge components in KH , the ITS will
Correct?
Duratio n
Hints
Know.Comp
Bene fit
1
S1
1
1
0
Read Numerals
1
2
S2
1
1
0
Identify Units
1
3
S3
0
2
2
Define variables
-0.75
4
S4
1
3
2
Write expression
0.2
5
S5
1
2
1
Negative slope
0.33
6
S4
1
3
3
Write expression
0.17
7
S5
1
1
1
Negtive slope
0.5
8
S5
1
1
1
Negative slope
0.5
Figure 2: Stable, Discernable Sequence of Scenarios.
return to prediction phase. Otherwise, it will enter the prescription phase and continues to use the hypothesized plan until cumulative benefit values keep improving. We now give a simple example illustrating the actions of an ITS. In this example, we use the following settings: α = 1 for all steps, β = 0.05, m = 5. The sequence of user scenarios are displayed in Figure 2. For this example, we assume that the entropy of the sequence must be less than 0.69 (this value is the entropy of a sequence containing the same number of distinct steps as the example sequence and each step appears exactly once). For these values, the above sequence is m-stable and discernable at step 8. Then, the cumulative benefit of each of the knowledge components is analyzed. The occurrence frequency and cumulative benefit of each distinct knowledge component is as follows – h Read Numerals, 1/8, 1 i, h Identify Units, 1/8, 1 i, h Define Variable, 1/8, -0.75 i, h Write Expression, 2/8, 0.185 i, h Negative Slope, 3/8, 0.4 i. Based on these values, we determine that write expression, negative slope are the two knowledge components with a high occurrence frequency and the rest of have a low occurrence frequency. Among these two components, write expression has a low benefit. Then the system plans to include more play sessions that include write expression and sets its alpha value to 0.5. (Small value of α means that a user can take longer time and more hints to solve a scenario without much penalty.) For negative slope knowledge component, the plan is to increase the play sessions displayed to the user with this knowledge component and set α to 1.5 (This action will reduce the benefit of a scenario with a longer duration and/or high number of hints. Figure 3 displays two possible scenarios for the continuation of the play session. Table in Figure 3(A) shows a sequence of scenarios for which the cumulative benefit of the above knowledge components increase. Table in Figure 3(B) shows an alternative for which the cumulative benefit of write expression knowledge component reduces. We now describe an experiment where we analyzed a small subset of the KDD data set. We chose data of 20 stu-
No .
Scen ario
Correct?
Duratio n
Hints
Know.Comp
Bene fit
9.
S1
1
1
0
Read Numerals
1
10.
S4
1
2
1
Write expression
0.41
11.
S5
1
2
2
Negative slope
0.17
12.
S5
1
1
0
Negative slope
1
13.
S4
1
2
2
Write expression
0.35
Scen ario
No .
No .
Scen ario
Correct?
Duratio n
Hints
Know.Comp
Bene fit
9.
S1
1
1
0
Read Numerals
1
10.
S4
0
2
1
Write expression
-0.59
11.
S5
1
2
2
Negative slope
0.17
12.
S5
1
1
0
Negative slope
1
13.
S4
1
2
2
Write expression
0.35
Figure 3: Sequence of Scenarios in Validation Phase.
dents based on low entropy values. The data set included the scenarios of the following 20 students – stu ec5da6f77d, stu 7b6d0a18f6, stu b4581374fd, stu de6ab8afa0, stu 90242420ae, stu 4e15f8dd70, stu 4e15f8dd70, stu e3a8cf507f, stu 7c66794422, stu 10127b1302, stu 317ab04ae8, stu d4de23f6b5, stu c1d27b0e16, stu 7677b38bd3, stu 538557b88c, stu f30efa2d77, stu a63b30c044, stu 2e902bca6a, stu 7c1f202e4f, stu 3b2176de2f, stu 3b2176de2f, stu 924a363092. We set α = 1, m = 10, and β = 0.15. We constructed an mstable and discernable sequence S of scenarios for the above students. Based on S, we analyzed the benefit vector containing the frequency and cumulative benefits of knowledge components to generate hypotheses for each student. The number of hypotheses generated is based on the number of knowledge components with a high occurrence frequency in S. We then computed another m-stable and discernable sequence S 0 for each student. In Figure 4(A), we display a chart that shows, for each student analyzed, the lengths of S and S 0 . As can be seen from the chart, |S| is about 14 for most students whereas |S 0 | is around 15. These data show that the m-stable, discernable sequences can be extracted from data. It can be easily seen that the length of S depends on m as well as β. Sequence S is longer for small β values. Using S 0 , we counted the number of students for which at least one hypothesis was validated. For each student we counted the number of knowledge components that had high occurrence frequency and high cumulative benefit in both S and S 0 or those that had high occurrence frequency and low cumulative benefit in S and changed to high occurrence frequency and high cumulative benefit in S 0 . Figure 4(B) displays the number of hypotheses generated
-
'
& !
!
!
%
$
# "
(
(
)
*
(
)
(
,
+
ing can be predicted, the predictions become validated so that prescriptions to measurably improve learning can be generated, and finally points where the prescriptions have achieved their effect (or worn off). The effectiveness of intelligent training systems is illustrated using a case study from the KDD Cup 2010 education data set. We plan to conduct further experiments using this data set as well as our construction engineering system VICE. The current approach focusses on measurably improving learning for a single user. We also plan to extend the proposed approach to handle multiple users.
-
B
J
E
D
;
E
E
I
>
H G
F
E
DE
=
C
B
=
;
@A
;
=
T
V
VU
U
L
QR U
YZ
XU
T
V
VU
U
L
QR
L
[
[
U
Z
/
Z
S
\
^
]
0
M
L
1
S
P
K
O
2
3
WU
S
[
4
5
M
L
S
P
N
K
O
6
7
8
4
9
9
.
+
6.
Figure 4: (A)Length of S and S 0 for 20 students. (B)Number of hypotheses generated and the number of hypotheses validated for each student. for each student and the number of hypotheses that were validated. As can be seen from the figure, for most students, 7 hypotheses were generated. There are 11 students where at least one hypothesis is validated in S 0 . For the other 9 students, none of the hypotheses generated were validated. Although this is a very small experiment, the results are encouraging. The experimental results show that the proposed approach of constructing stable and discernable sequences to predict user learning and to validate hypotheses generated from data mining using these stable sequences can be successful in practice. We plan conduct experiments at a much larger scale to validate these preliminary findings in near future.
5.
CONCLUSIONS
We have proposed a novel, simple framework for combining game-based training systems with data analysis methods to develop intelligent training systems. These systems enable users to ”learn-by-doing” in realistic, risk-free environments with measurable improvements to learning. The proposed modular system architecture underlying intelligent training systems allows plug-and-play with different gamebased trainers and data miners. The training and the mining systems are tightly integrated and dynamically adapted based on the phases of the intelligent training system. The proposed approach employs data analysis to automatically identify points in a play session where the user responses have meaningful and stable information about the their learning behavior so that future scenarios likely to improve learn-
REFERENCES
[1] A. Derryberry. Serious games: online games for learning. Adobe Whitepaper, November 2007. [2] M. Dondlinger. Educational video game design: A review of the literature. Journal of Applied Educational Technology, 4(1):21–31, 2007. [3] J. Goedert, Y. Cho, M. Subramaniam, H. Guo, and L. Xiao. A framework for virtual interactive construction education (vice). Intl. Journal of Automation in Construction(Elsevier), 20(1):76–87, 2011. [4] J. Goedert, Y. Cho, M. Subramaniam, and L. Xiao. Virtual interactive construction education (vice) using bim tools (best paper). 3rd Intl. Conference on Construction Engineering and Project Management, 2009. [5] H.Jenkins, E.Klopfer, K.Squire, and P.Tan. Entering the educational arcade. ACM Journal of Computers in Entertainment, 4(1):21–31, 2003. [6] I. Horrocks. Constructing User Interface with Statecharts. Addison-Wesley, 1999. [7] J. Kaye and D. Castillo. Flash MX for Interactive Simulation. Onward Press, 2002. [8] M. Kock and A. Paramythis. Activity sequence modelling and dynamic clustering for personalized e-learning. Journal of User Modeling and User-Adapted Interaction, 21:51–97, 2011. [9] M.Subramaniam, J. Goedert, D. Khazanchi, and J. Shockley. Attracting the next generation: A multi-dimensional framework for construction education. FIATECH, 2009. [10] T. Reenskaug. The model-view-controller:its past and present. JavaZone, 2003. [11] C. Romero and S. Ventura. Educational data mining: A review of the state of the art. IEEE Trans. on Systems, Man and Cybernetics, 40(6):601–619, 2010. [12] M. Singley, P. Fairweather, T. Wolf, and D. Lam. Pastel: Pattern-driven adaptive simulations. INTERACT, LNCS 4663, page 3, 2007. [13] C. Steinkuehler. Massively multiplayer online games as an educational technology: An outline for research. Journal of Educational Technology, 48(1):10–21, 2008.
