Proceedings of the 18th Conference on Behavior Representation in Modeling and Simulation, Sundance, UT, 31 March - 2 April 2009 [paper: 09-BRIMS-017, pp. 141-142]
The Predictive Performance Optimizer: Mathematical Modeling for Performance Prediction Tiffany S. Jastrzembski Kevin A. Gluck Air Force Research Laboratory 6030 S. Kent St Mesa, AZ 85212 480-988-6561
[email protected],
[email protected] Stuart Rodgers AGS TechNet 10887 Miriam Lane Dayton, OH 45458 937-903-0558
[email protected] Michael Krusmark L3 Communications at Air Force Research Laboratory 6030 S. Kent St Mesa, AZ 85212 480-988-6561
[email protected] Keywords: mathematical model, performance prediction, prescription, optimization
1. Research Goal
1.1 The Predictive Performance Optimizer
The overarching goal of our research is to translate basic cognitive science into an applied, state-of-the-art “cognitive tool” (Lajoie & Derry, 1993) - which we have termed the Predictive Performance Optimizer (PPO) – to provide teachers, trainers, and learners of all types with a new generation of adaptive training assistance that dynamically assesses performance effectiveness, prescribes training regimens, and accurately predicts future performance.
The Predictive Performance Optimizer performs three distinct functions. Firstly, the tool tracks the performance of a learner using historical performance data (see Figure 1.1), akin to the student modeling approach taken by intelligent tutors. However, unlike intelligent tutoring systems, PPO includes a mechanism to represent memory decay as a function of time passed to training time amassed. This decay mechanism inherently provides PPO with a second functionality the ability to make predictions of future performance following periods of non-use (see Figure 1.1).
This project has leveraged more than a century of research in the domain of human memory to develop a cognitive mathematical model that informs the implementation of cognitive mechanisms and processes responsible for human performance (Jastrzembski, Gluck, & Gunzelmann, 2006). Our model builds upon and extends past work (e.g., Pavlik & Anderson, 2005) by providing new capabilities in the realms of performance prediction and training regimen prescription. As such, the model functions by capitalizing on learning signatures and mathematical regularities in the human memory system to best schedule the timing and frequency of future training events that will maximize performance around specified training goals, constrained as necessary by the logistical practicalities of the real world.
Figure 1.1. PPO tracks the performance of the learner and extrapolates mathematical regularities from training history to make accurate predictions of performance.
In Figure 1.1 above, PPO tracks the performance of a team of F-16 fighter pilots flying simulators at the Air Force Research Laboratory over the course of a week. They return three months later to complete three additional training events. PPO accurately predicted future performance of these pilots very well, achieving a correlation of 0.98 to human data (RMSD = 0.009). Of critical importance, PPO’s underlying mathematical representation of human memory captures the spacing effect. This phenomenon reveals that practice occurring more slowly over time becomes more durable, and therefore more stable – thus, thoughtful consideration of initial spacing of training produces less decay at future times (see Figure 1.2). Panel A: Well-spaced, distributed training history
Figure 1.3. PPO allows users to compare performance implications across various training regimens.
2. Summary In conclusion, PPO is a new technology option based on state-of-the-art applied cognitive science developed to aid instructors and trainees with performance predictions and optimization of future training prescriptions around specified goals.
3. References Panel B: Poorly-spaced, massed training history
Figure 1.2. Retention implications for massed and distributed training at 1, 2, 3, and 4 month lags. The accurate ability to capture the spacing effect leads to PPO’s third major functionality – the capability to prescribe training regimens. PPO extrapolates from the learning signatures using known historical data to schedule the timing and frequency of future training events on the basis of maximizing performance around user-specified training goals. PPO then automatically generates graphical depictions of future performance to allow the user to assess and compare performance implications of training plans, and provides the user with optimal training prescriptions associated with training goals (see Figure 1.3). We assert that PPO may also be used to help instructors and trainers answer relevant questions regarding trainee readiness. For example, in the case of a warfighter being deployed in one month, this tool allows a training manager to assess how much training that individual must receive to perform at a specified level of effectiveness, and additionally helps determine whether or not deployment timetables are feasible.
Jastrzembski, T.S., Gluck, K.A., & Gunzelmann, G. (2006). Knowledge tracing and prediction of future trainee performance. I/ITSEC Annual Meetings, Orlando, December 4-7. Lajoie, S.P, & Derry, S.J. (Eds.) (1993). Computers as Cognitive Tools. Hillsdale, NJ: Erlbaum. Pavlik, P. & Anderson, J.A. (2005). Practice and forgetting effects on vocabulary memory: An activation-based model of the spacing effect. Cognitive Science, 29, 559-586.
Author Biographies Tiffany S. Jastrzembski is a Research Psychologist at the Air Force Research Laboratory focused on developing the underlying mathematical model in PPO to capture the dynamics of human memory. Stuart Rodgers is a computer scientist focused on implementing cognitive models of human performance and other adaptive, reactive, and autonomous software systems. He is Director at AGS TechNet, Dayton, OH. Kevin A. Gluck is a Senior Research Psychologist at the Air Force Research Laboratory and enthusiastic collaborator on the Predictive Performance Optimizer project. Michael Krusmark is a Research Scientist with L-3 Communications at the Air Force Research Laboratory and collaborator on the PPO project.
