effectiveness of simulation results for decision support. Keywords-simulation, system dynamics, systems engineering, experience accelerator, acquisition ...
Simulation-Based Decision Support for Systems Engineering Experience Acceleration Douglas A. Bodner
Jon P. Wade, Alice F. Squires, Richard R. Reilly, Peter G. Dominick, George Kamberov
Tennenbaum Institute Georgia Institute of Technology Atlanta, GA 30332 USA
Stevens Institute of Technology Castle Point on Hudson Hoboken, NJ 07030 USA
William R. Watson Department of Curriculum and Instruction Purdue University West Lafayette, IN 47907 USA Abstract—With much of the current systems engineering workforce reaching retirement age, there is a critical need to develop work experience among new systems engineers. Increasingly, educators and employers are turning to educational technology-based approaches to aid in this process. One such effort is the Experience Accelerator, an on-going research project aimed at maturing program-level systems engineers in Department of Defense acquisition programs. In this experiential learning approach, learners assume the role of a lead program systems engineer for development of a new unmanned aerial system (UAS), and they must make appropriate decisions and trade-offs at key points during the development lifecycle to keep the program on track and recover from problems that occur. Problems can occur in such areas as requirements, schedule, quality, cost, and customer expectations. The acquisition program is represented with a set of system dynamics simulation models that include such activities as sub-system development, system integration and system test. This paper focuses on the simulation-based approach used to provide internal decision support for learners as they engage in a learning experience within the Experience Accelerator. We present findings to date involving simulation modeling of acquisition programs and effectiveness of simulation results for decision support. Keywords-simulation, system dynamics, systems engineering, experience accelerator, acquisition program
I.
INTRODUCTION
A. Motivation At a time when systems are becoming more complex, there is a shortage of available systems engineering expertise, particularly within U.S. Department of Defense (DoD) programs [1]. Much of the systems engineering workforce is This material is based upon work supported, in whole or in part, by the U.S. Department of Defense through the Systems Engineering Research Center (SERC) under Contract H98230-08-D-0171. SERC is a federally funded University Affiliated Research Center managed by Stevens Institute of Technology.
978-1-4673-0750-5/12/$31.00 ©2012 IEEE
nearing retirement age. Thus, developing expertise and skills among the next generation of systems engineers is a major concern. This challenge is compounded, though, by fiscal constraints that are expected to result in fewer DoD programs to provide on-the-job training. Remaining programs will likely be more critical, thus providing less room to allow learning-bymaking-mistakes for inexperienced systems engineers. Therefore, alternative skill development methods besides onthe-job training are needed. One way to address this problem is the use of educational technology. Educational technology includes such approaches as distance learning and interactive learning environments, and has been used in a variety of domains. B. Experience Accelerator Background The Systems Engineering Experience Accelerator is one example of educational technology used to develop skills for systems engineers [2]. The Experience Accelerator provides an interactive learning environment targeted to support lead program systems engineers in DoD acquisition, and encompasses a competency model [3] for skills that a systems engineer at that level should possess. The model is organized into six broad categories: systems and critical thinking, systems engineering technical leadership, systems engineering implementation, technical management, program management, and other broad-based professional competencies. In this interactive learning experience, the learner assumes the role of a lead program systems engineer for a major DoD acquisition program addressing development of a new unmanned aerial system (UAS). The learner interacts with a simulated program environment with the goal of keeping the technical aspects of the program on track and identifying and recovering from problems. Note that the term “learner” is used in this paper to refer to the user of the Experience Accelerator. The learner starts the experience replacing a former lead program systems engineer on a program that just completed the
preliminary design review. The learner guides the program’s technical aspects through successive reviews until it achieves initial production and deployment. At several points between each review, the learner is presented with the current program status and given the opportunity to make decisions and recommendations for program adjustments. Decisions may, for instance, be recommendations to reallocate or add resources (labor, equipment or facilities), increase the frequency of design reviews, extend the schedule, or reduce capabilities. The Experience Accelerator uses a system dynamics-based simulator engine to represent the progress of the program, and its outputs are provided as data and charts for the learner to use as decision support in identifying and addressing problems. Note that the terms simulation models and simulator are used to refer to the system dynamics simulation models, which are part of the overall simulated experience presented to the learner. The goal is to create an experience that teaches the learner key systems engineering lessons and trade-offs across the development lifecycle in an accelerated time-frame, rather than simply relying upon on-the-job training over the course of a multi-year program. We present results to date involving simulation modeling of acquisition programs and effectiveness of simulator results for decision support. II.
EXPERIENTIAL LEARNING AND SIMULATED WORLDS
A. Experiential Interactive Learning Environments Experiential interactive learning environments are a type of educational technology used increasingly for training and skill development in a variety of fields. In such environments, some of which may be likened to a digital game, a learner interacts with a simulated world, representing a domain of interest for learning. Key lessons are presented about the domain and its constituent elements and about effective decision-making within the domain in terms of strategy, tactics and operations. The intent is that these lessons become generalized and transfer from the simulated world to actual settings in the domain of interest. Such environments build on the concept of experiential learning designed to accelerate learning beyond the rate of traditional means [4]. In recent years, research has demonstrated that such environments are well-suited to education in today’s knowledge economy [5]. In particular, these environments encourage a constructivist approach to learning, in which the learner solves problems, reasons, thinks critically, and uses knowledge both actively and reflectively [6]. In a role-based experience, the learner becomes familiar with actions, vocabulary and ways of thinking associated with a domain of interest [7]. Hence, the challenge is to design the experience and simulated world in a manner that supports effective and generalized/transferrable learning. B. Simulated Worlds Using System Dynamics A simulated acquisition program is an important element of our particular learning experience. One simulation methodology for such worlds is system dynamics, well-known for simulating complex systems that have non-linear behavior and non-intuitive interactions between system elements [8].
System dynamics has been used to model and analyze a number of engineering project management applications, especially software development [9, 10, 11]. System dynamics employs a representation based on accumulation points (stocks), flows between stocks, feedback sub-systems and various equations that govern relationships between system elements. The basic formalism is a continuous flow system based on differential equations. However, discrete effects (e.g., step functions, pulse functions, etc.) have been added to the underlying continuous representation. System dynamics has been implemented in a number of commercially available and open source software packages. In a defense acquisition program context, system dynamics can be used to model such program elements as progress on tasks, rework, skill levels of labor, training of personnel, communication overhead, requirements creep, defect generation, defect identification and correction, equipment and facility resources, cost accruals, and earned value management (EVM). EVM is used by Department of Defense and many industries to judge a program’s cost and schedule performance relative to a baseline [12]. III.
LEARNING EXPERIENCE
Since the learner in our experience assumes the role of lead program systems engineer just after the preliminary design review, replacing the previous lead program systems engineer, the learner is tasked immediately with identifying any existing problems with the program. The learner then progresses through a set of phases that end with important program milestones, as shown in Table 1. It should be noted that Phase 1 provides introductory material to the learner about the experience, and there is a reflection phase following Phase 5. TABLE I. Phase
PHASES FOR EXPERIENCE Phase Description
Phase activity focus
Ending milestone
2
System pre-integration
Critical design review
3
System integration
Flight readiness review
4
Flight test
5
Limited production
Production review Integrated review
readiness system
The learner monitors the program via metrics categorized into (i) key performance parameters (KPPs) and technical performance measures (TPMs), (ii) quality measures, (iii) progress on meeting entrance criteria for reviews, and (iv) schedule and cost (via earned value management). Problems can occur in any of these areas. The Experience Accelerator provides two means by which the learner can identify problems – program status charts and interaction with non-player characters (NPCs). The learner engages with NPCs representing various government and contractor roles (e.g., prime contractor lead systems engineer) to determine root causes for the problems indicated by metrics. The NPC interaction is designed and implemented as a set of hub-andspoke dialog sessions whereby the learner selects questions to
ask an NPC and is led to new possible questions based on the answer. The emphasis is on asking the right sequence of questions to obtain the correct root cause information. The learner interacts with the simulated world only at certain key points. That is, every few months of simulated time, the learner is given a chance to review the program status and then make a set of decisions to adjust program performance based on feedback received from charts or NPCs. Assuming that adjustments are made, the simulator executes a system dynamics simulation model of the program for the next 3-6 months (depending on the particular phase), advancing its state with the decisions made by the learner. This new program state is used to generate charts corresponding to the status of various metrics, and it is used to inform the state-dependent dialog for the NPCs, as shown in Fig. 1. In the software architecture for the Experience Accelerator, the simulated UAS program is separate from the simulated NPC [13]. Status charts
Simulated program
Program state
NPCs
Figure 1. Learner interaction with status information
The system dynamics methodology allows modeling of such phenomena as lags in system behavior. That is, similar to real program behavior, actions do not immediately produce desired results. Hiring, for instance, to increase staff and address a scheduling problem takes time and must be done early enough in the program phase to overcome the issue. A similar phenomenon is Brooks’ Law, a generalized principle in software development that adding too many people to a project behind schedule worsens the schedule problem at least temporarily [14]. This is modeled by training requirements that divert skilled people, as well as communication overhead. Thus, the role of the simulator is to demonstrate to the learner subtle effects in program behavior based on the recommendations, as well as how early problem identification and recovery is critical. Since the lead program systems engineer is not in charge of the program, the learner’s decisions are cast as recommendations to the acquisition program manager NPC, who in turn would make final decisions. In this version of the Experience Accelerator, the assumption is that the
recommendations are accepted and implemented. Example recommendations are to reallocate or add resources (labor, equipment or facilities), increase the frequency of design reviews, extend the schedule, or reduce capabilities. In future versions, the learner may be expected to convince the program manager or other stakeholders to accept a recommendation. In the future, a set of problems, or challenges, given to a learner will be tuned to the learner’s skill level as measured by assessment instruments (questionnaires) that the learner fills out prior to the learning experience. IV.
SIMULATION AND DECISION SUPPORT
A. System Dynamics Models This section describes system dynamics models for the preintegration, system integration, flight testing and limited production phases. There are separate models for each phase, although a number of variables span multiple phases. The state in between phases is kept via use of text files. The main focus here is on the pre-integration phase (Phase 2). These models are implemented using extensions to an existing open-source Java™ codebase (SystemDynamics [15]). One of the key principles of the Experience Accelerator is the use of open-source code to facilitate use of the technology by that element of the systems engineering community interested in skill development. The models themselves are XML files, while they are executed by the simulator code. The pre-integration phase models the design and development of sub-systems for the UAS and the planning for the overall system critical design review. Currently, the airframe/propulsion, the command-and-control system, and the ground station are modeled as sub-systems. A different contractor is responsible for each sub-system, with the lead contractor being responsible for system integration. Other subsystems (e.g., self-defense system) may be added in the future. The engineering workforce for each contractor has different specialties such as design and quality, as well as different skill levels (currently modeled as experienced vs. inexperienced). Design engineers obviously address technical tasks associated with system readiness, while quality assurance personnel identify and remediate defects. Training effects, such as the time needed for experienced staff to mentor junior staff, and communication overhead are modeled. The contractors can hire either experienced or inexperienced engineers. Inexperienced engineers gain experience at a particular rate, and of course experienced engineers have a higher productivity than inexperienced ones. Productivity rate impacts progress on various design and development tasks associated with each sub-system. There is an attrition rate, in addition, as personnel may choose to leave the program. These modeling constructs are similar to those described in [9, 11]. The key performance parameter modeled for the UAS is range R. In pre-integration, the range KPP is an estimate based on several technical performance measures, including weight (ratio of initial fueled weight Wi to final weight Wf), aerodynamic efficiency (ratio of lift L to drag D), and propulsion efficiency (ratio of constant speed V to specific fuel
consumption c). This relationship is implemented using the Breguet range equation, a standard approximation equation relating these factors [16], as shown in (1).
criteria for production readiness review: airworthiness certification milestones, affordability demonstrated, digital production files, production facilities, production line demonstrated, and production personnel trained.
R = (V/c) (L/D) ln (Wi/Wf)
Finally, the limited production phase (Phase 5) demonstrates the results of the acquisition program in terms of meeting customer expectations. The range KPP is measured as average fleet range. Cost per unit of air vehicles and ground stations are also tracked, as well as the number of vehicles and ground stations produced over time. This phase is posed as a way for the learner to see results from the previous phases, and thus entrance criteria for the next review are not modeled.
(1)
Quality is measured by the number of software design defects currently outstanding in the command-and-control system, a software-intensive system. Defects are categorized as either critical or general. Such defects are introduced to the command-and-control sub-system as undiscovered defects that originate with a specific rate. The defects are then discovered at a rate dependent on design review frequency and review effectiveness. They are addressed at a rate dependent on the productivity rate and number of quality assurance staff (accounting for productivity differences between experienced vs. inexperienced engineers). One of the main goals of the pre-integration phase is to ensure readiness for the critical design review. Readiness is measured by progress on entrance criteria for the review. If a program is not “ready” for the critical design review (or any review) by the scheduled time for the review, then typically the review is delayed. Obviously, this results in schedule and cost problems. Defense acquisition programs use many such criteria, some of which are standard across programs, and some of which depend on the type of system being developed (e.g., aircraft vs. missile system). These are measured here as a percentage toward completion. To simplify the set of criteria used to a manageable number, we use the following: •
Detailed test requirements determined,
•
Digital design data files released,
•
Software design description finished,
•
Structural loads released,
•
Sub-system design and verification, and
•
Verification and validation of system integration lab performed.
The integration phase (Phase 3) models the progress on integrating the sub-systems in preparation for the flight readiness review (FRR). Similar to the previous phase, the range KPP is tracked via estimates based on the constituent TPMs. The software quality metrics move from design defects to implementation defects, as code is implemented and tested. A new set of entrance criteria is introduced and includes such criteria as airframe/engine certification for flight, airworthiness certification milestones, flight critical software demonstrated and verified, ground vibration testing, safety of flight certified, software safety of flight testing, structural static testing, and test cards completed. The flight testing phase (Phase 4) models progress on flight tests and preparation for the production readiness review. Here, there are two important elements of the program being modeled. First is the progress of the UAS via flight testing and demonstration. Second is progress on production facilities and capabilities. These two elements are reflected in the entrance
While costs are tracked in Phase 5, these are not currently implemented as part of a comprehensive cost modeling across all phases. Rather, such a comprehensive model of schedule and cost is under development via earned value management. EVM uses several key concepts to inform program leaders of the status of schedule and cost [17]. First is budgeted cost of work scheduled (BCWS), which provides a baseline for program task execution. Budgeted cost of work performed (BCWP) compares actual task execution with the schedule using the budgeted cost figures. To compare actual vs. budgeted cost, the actual cost of work performed (ACWP) is tracked. All three figures are tracked over time; the costs are mapped out as accruals of total cost budgeted or incurred over a time horizon. For purposes of the simulated world, BCWS is an input, since it is determined prior to program initiation. BCWP and ACWP are tracked as the program simulation progresses and costs are incurred. Cost categories include labor, facilities and test equipment. Cost efficiency is computed at any time t as the ratio of BCWP to ACWP where those terms are measured as a function of t. Similarly, cost efficiency at any time t is computed as the ratio BCWP to BCWS measured at t. It is desirable that both efficiency terms be greater than or equal to 1.0. In any simulation, one critical issue is verification and validation of the system dynamics models. Verification usually refers to internal consistency (does the implemented model accurately portray the modeling intent), and validation typically refers to external consistency (does the implemented model reflect the real system modeled). A UAS was selected for the experience due to the increased importance of such system and subsequent increased acquisition needs in this area. In terms of verification and validation, we did not use an existing program against which to perform validation. Rather, our approach is to develop a modeling intent (i.e., experience design) and validate the design using a panel of subject matter experts (SMEs) with extensive experience in aeronautical system acquisition. The SMEs are then charged with verifying that the implemented models are in line with the design intent. Thus, our approach is to convey an authentic learning experience based on generic lessons to be learned versus one based on an actual program(s). B. Output Artifacts During each cycle, the simulator generates a set of charts based on system dynamics model results, and these are provided to the learner as a set of artifacts. These charts reflect the status of various metrics described previously. The charts
typically show the status compared to a target and potentially to a plan for the status across a time horizon. For example, consider quality, one measure of which is the number of software defects currently outstanding in the command-and-control system versus a targeted maximum allowable number of defects. Fig. 2 shows a chart that measures general (i.e., non-critical) software errors in the command-and-control system across a span of six months. While a large number of errors have been resolved, the number outstanding has exceeded the target for maximum errors. Obviously, this should cause concern for the learner.
points in the program. It starts at 4% above the requirement, and then the expectation is that weight or some other factor will cause a drop in the estimate. There is good news at the outset of the program, as the estimate is above both the plan and the requirement. However, estimate decreases somewhat rapidly as compared to the plan. This should be a cause of concern for the learner. Note that the range estimate is reported in discrete units (i.e., small changes due to changes in the underlying TPMs are generally not reported). Finally, there are charts for entrance criteria. Fig. 4 shows progress on the release of digital design files as an entrance criterion for the critical design review. The plan is expressed as a step function with expected progress by six month increments. Progress essentially is a continuous function due to the number of digital files. For this particular program instance, the schedule is met for the first year-and-a-half, while the expected progress in the last six months did not reach 100%. The learner has to use judgment as to whether it is advisable to continue with the planned CDR without 100% of the design files. Note that this chart is for primary components, and there is a similar chart for secondary components. Also, this chart covers the Phase 2 period prior to the critical design review.
Figure 2. Software quality chart
The range KPP is also tracked via a status chart. The range requirement is 5,000 miles. Fig. 3 depicts a range chart with a planned range shown, along with the requirement and the actual range estimate as measured over a time of two years. This time corresponds to Phase 2 (the two years prior to scheduled critical design review). Note that the y-axis is expressed as a percentage differential of the requirement. Thus, the range estimate starts at 105% of the requirement, and the range plan starts at 104%.
Figure 4. Progress on digital design files for CDR
C. Challenges and Learner Responses As implied from the charts presented before, a set of challenges is built into the system dynamics models for the learner to identify and attempt recovery. These charts serve as decision support for the user to identify the underlying challenge, make deeper investigation of root causes, and then make the appropriate recommendation. Consider the quality chart presented in Fig. 2. The error rate may be higher than expected, or the error remediation team may not be as productive as needed. The learner has several choices, three of which might be: Figure 3. Range chart
The planned range is the expectation of senior program personnel as to what the range estimate should be at various
•
Add resources by hiring junior quality staff for the command-and-control sub-system,
•
Add resources by hiring senior quality staff, or
Extend the schedule to allow time for the remediation team to catch up to the error backlog.
systems engineering students that measure their perceptions and level of competence pre and post experience.
In the simulation, hiring junior staff requires training time for them to become fully productive (which requires senior staff time, reducing overall productivity in the short term). Extending the schedule, obviously, would increase cost, and would delay the integration even if the other sub-systems are on schedule. The best recommendation is likely to hire senior staff who can improve productivity sooner (although there still is a Brooks’ Law effect due to increased communication overhead). Thus, the learner should use the chart to recognize that the number of errors is on a trajectory to exceed the target before it does so.
An area of future research would be to determine changes to the artifact charts that would improve their effectiveness as decision support instruments. For instance, there are rules of thumb that experienced systems engineers recognize as danger signals (e.g., once a program is significantly off-track cost-wise in the first part of allocated program costs, it is difficult-toimpossible to recover). If these rules of thumb are identified and included, do they improve decision-making? Do they improve learning, or is failure a better teacher?
Consider the range chart in Fig. 3. The estimated range is below the planned value, and this may indicate issues with drag or weight that need to be corrected before the situation worsens. Thus, the learner should consult a weight or drag chart, or engage with one of the NPCs such as the lead systems engineer for the prime contractor to get answers to questions. Depending on the underlying problem, the learner may need to recommend a focused effort on weight or drag reduction or a reallocation of weight from one sub-system to another.
The authors would like to acknowledge all members of the Experience Accelerator team and the subject matter experts that have aided with authenticity of the simulation results and overall learning experience.
•
Finally, consider the entrance criterion chart in Fig. 4. There did not seem to be cause for concern for the first 18 months. There is not much time left to remediate the shortfall in the last few months before the CDR. The learner must decide, therefore, if it is acceptable to proceed with CDR or to recommend a schedule extension. In many instances, the digital design files are in continued need of revisions and additions, especially early in the program lifecycle, and thus the former option is likely more appropriate. Note that there are thresholds defined whereby the program goes into various levels of breach. This could be due to being significantly over budget, behind schedule, or short of requirements. If the breach is significant enough, the program is cancelled. The goal for the learner is address situations such as identified here early and not let multiple problems accumulate and cause a breach. V.
CONCLUSIONS AND FUTURE WORK
This paper presents a simulation-based method for providing internal decision support to learners in an experiential interactive learning environment designed to promote skill development for systems engineers. The underlying system dynamics models are described, as well as the output artifacts used by a learner to identify and remediate problems. Several output artifacts and associated problems, among the many modeled in the program lifecycle, are discussed in detail. We continue to validate these models and artifacts using subject matter experts. Further evaluation of the simulation-based decision support for learning effectiveness and evaluation of the overall learning experience are on-going. This will be done using formative surveys of systems engineering instructors and evaluations with
ACKNOWLEDGMENTS
REFERENCES [1] [2]
[3]
[4] [5]
[6] [7] [8] [9] [10]
[11] [12] [13] [14] [15] [16] [17]
NDIA, Top Systems Engineering Issues in US Defense Industry, Systems Engineering Task Group Report, Final-v11-9/21/2010, 2010. A. F. Squires, et al., “Investigating an innovative approach for developing systems engineering curriculum: The Systems Engineering Experience Accelerator,” in Proc. 2011 ASEE Annual Conf., 2011. A. F. Squires, J. P. Wade, P. G. Dominick, and D. Gelosh, “Building a Competency Taxonomy to Guide Experience Acceleration of Lead Program Systems Engineers, in Proc. Conf. Systems Engineering, 2011. D. Kolb, Experiential Learning. Englewood Cliffs, NJ: Prentice Hall, Inc., 1984. Federation of American Scientists (2006). Harnessing the power of video games for learning. [Online]. Available: http://www.fas.org/gamesummit/Resources/Summit%20on%20Educatio nal%20Games.pdf. M. P. Driscoll, Psychology of Learning for Instruction, 3rd ed.. Boston: Pearson Education, Inc., 2005. D. W. Shaffer, How Computer Games Help Children Learn. New York: Palgrave Macmillan, 2006. J. D. Sterman, Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston: McGraw-Hill, 2000. T. Abdel-Hamid and S. Madnick, Software Project Dynamics: An Integrated Approach. Englewood Cliffs, NJ: Prentice Hall, 1991. S. Ferreira, J. Collofello, D. Shunk, and G. Mackulak, “Understanding the effects of requirements volatility in software engineering by using analytical modeling and software process simulation,” Journal of Software and Systems, pp. 1568-1577, October 2009. R. J. Madachy, Software Process Dynamics. Hoboken, NJ: WileyInterscience, 2008. Q. Fleming and J. Koppelman, Earned Value Project Management, 3rd ed., Project Management Institute, 2005. J. Wade, G. Kamberov, D. Bodner, and A. Squires, “The Architecture of the Systems Engineering Experience Accelerator,” unpublished. F. P. Brooks, Jr,. The Mythical Man-Month. Reading, MA: AddisonWesley, 1975. J. Melcher (2009). SystemDynamics. [Online]. Available: http://sourceforge.net/projects/system-dynamics. G. J. J. Ruijgrok, Elements of Airplane Performance. VSSD, 2009. Defense Acquisition University, Defense Acquisition Guidebook, 2011.
