A Methodology for the Probabilistic Assessment of System ... - CiteSeerX

5 downloads 34290 Views 4MB Size Report
Nov 1, 2001 - the best of companions. ... taught by example what it means to be a teacher, and I can only hope to follow ... and that a warm hug, even by email, is one of the most wonderful things there is. ..... CAMPAIGN CODE SELECTION .
A Methodology for the Probabilistic Assessment of System Effectiveness as Applied to Aircraft Survivability and Susceptibility

A Thesis Dissertation

Submitted to the Academic Faculty In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in Aerospace Engineering

By

Danielle Suzanne Soban

Georgia Institute of Technology Atlanta, GA November 1, 2001 Copyright  2001 by Danielle S. Soban

ACKNOWLEDGEMENTS It is said that a journey of a thousand miles begins with a single step. That single step took place many years ago, and I have been so fortunate to have had along the way the best of companions. The people who have walked beside me on this journey have provided an unending supply of love, support, professional and technical experience, and unfailing encouragement. This work is theirs as much as it is mine. To my advisor, Dr. Dimitri Mavris, thank you for believing in me and not taking no for an answer. You have opened a door on a world of professional experiences and opportunities, and I am grateful for your guidance. To Dr. Schrage and the rest of my thesis committee, Mr. Gene Fleeman, Dr. James Craig, and Dr. Bill Bell, thank you for sharing with me your wisdom, your experience, and your time. To the ASDL family, Michelle, Elena, Andy, and all of the rest: we’ve laughed together and we’ve cried together, we’ve moved mountains and we’ve played in the sand. Thanks for making this crazy place so interesting to come to every morning. And to Dr. Dan Biezad: you have taught by example what it means to be a teacher, and I can only hope to follow in your substantial footsteps. Todd, my love, my partner, you have filled the journey with joy and adventure. You pushed when I needed pushing, you lent your shoulder when I needed a place to lay my head. I wake each day amazed that you walked into my life and then decided to stay. This journey may be over, but I look forward to sharing with you all of the new adventures that await us.

iii

To my huge and crazy family, thanks for showing me it’s okay to be a little nuts, and that a warm hug, even by email, is one of the most wonderful things there is. Your love has filled my sails and your support has kept me afloat. And finally, to my mother, Suzanne. You have always held the beacon that illuminates my way, and I have never been lost. You have been my inspiration, my staunchest supporter, and my dearest friend. I will never stop learning from your wisdom and your courage. Thanks, Mom.

Dani

November, 2001 Atlanta, GA

iv

TABLE OF CONTENTS ACKNOWLEDGEMENTS ......................................................................................................... iii

TABLE OF CONTENTS ............................................................................................................... v

LIST OF FIGURES....................................................................................................................... xi

LIST OF TABLES...................................................................................................................... xvi

NOMENCLATURE .................................................................................................................. xvii

SUMMARY................................................................................................................................ xxii

INTRODUCTION .......................................................................................................................... 1 MOTIVATION ................................................................................................................................ 1 NEED FOR A MILITARY SYSTEM EFFECTIVENESS FRAMEWORK.................................................... 3 The Changing Economy and its Effect on Decision Making................................................... 3 The Link to System Effectiveness ............................................................................................ 5 Measures of Effectiveness .................................................................................................................6 Use of System Effectiveness Metrics.................................................................................................6 Resource Allocation .....................................................................................................................7 Requirements Definition...............................................................................................................7 Trade Studies Between System Components................................................................................8

Lack of Overall System Effectiveness Methodology................................................................ 9 Semantics and a Surplus of Synonyms ..............................................................................................9 Difficulty Accessing Government and Classified Material..............................................................14

SYSTEM OF SYSTEMS APPROACH ............................................................................................... 14 A Shifting Paradigm.............................................................................................................. 15

v

The Theater as the System..................................................................................................... 17 Mathematical Modeling ........................................................................................................ 21 Use of Probability Theory................................................................................................................22

RESEARCH QUESTIONS ............................................................................................................... 24 INCORPORATING UNCERTAINTY: THE RESPONSE SURFACE METHOD AND MONTE CARLO ANALYSIS................................................................................................................... 26 SIZING AND SYNTHESIS .............................................................................................................. 26 METAMODELS ............................................................................................................................ 27 RESPONSE SURFACE METHODOLOGY ......................................................................................... 29 Response Surface Equations ................................................................................................. 30 Design of Experiments .......................................................................................................... 31 USING RESPONSE SURFACE METHODOLOGY................................................................................ 33 Setting up the Problem.......................................................................................................... 33 Screening and Pareto Plots .................................................................................................. 34 Prediction Profiles ................................................................................................................ 36 Technology Impact Forecasting ........................................................................................... 38 Overall TIF Environment.................................................................................................................39 Technology Mappings and k-factors................................................................................................40 Monte Carlo Simulation...................................................................................................................43

Analysis: the Probability Graphs.......................................................................................... 44 SUMMARY OF INCORPORATING UNCERTAINTY AND RESPONSE SURFACE METHODOLOGY ........... 46

THE NATURE OF MILITARY MODELING .......................................................................... 49 MODELS AND MODELING ........................................................................................................... 49 Types of Models .................................................................................................................... 50 Conceptual Models vs. Computer Models ............................................................................ 53 How Models are Used........................................................................................................... 55

vi

Transparency ........................................................................................................................ 57 CLASSIFICATION OF MILITARY MODELS .................................................................................... 58 SIMTAX................................................................................................................................. 59 Other Taxonomies................................................................................................................. 61 Hierarchical Modeling.......................................................................................................... 62 Decomposition Levels .....................................................................................................................65 Engineering Models....................................................................................................................65 Mission Models ..........................................................................................................................67 Campaign Models.......................................................................................................................68

The Military Code Continuum .............................................................................................. 71 SUMMARY OF MILITARY MODELING ............................................................................................ 74

PRELIMINARY INVESTIGATION: APPLYING CURRENT METHODS TO CAMPAIGN LEVEL ................................................................................................................................. 76 CAMPAIGN CODE SELECTION ..................................................................................................... 76 ITEM ..................................................................................................................................... 77 CASE I: THEATER LEVEL CASE .................................................................................................. 78 Scenario for Case I ............................................................................................................... 79 Inputs and Outputs................................................................................................................ 81 Results from Case I ............................................................................................................... 82 CASE II: SURVIVABILITY TEST CASE.......................................................................................... 85 Scenario for Case II .............................................................................................................. 85 Inputs and Outputs................................................................................................................ 88 Results from Case II.............................................................................................................. 90 SUMMARY OF PRELIMINARY INVESTIGATION ............................................................................. 92 RESULTING ISSUES AND SOLUTIONS TO PRELIMINARY INVESTIGATION .......... 93 IDENTIFICATION OF THREE PRIMARY ISSUES ............................................................................... 93

vii

Level of Detail....................................................................................................................... 93 Model Integration ............................................................................................................................94 Zooming .....................................................................................................................................95 Model Abstraction ...........................................................................................................................96

Human in the Loop Dilemma ................................................................................................ 97 Scenario Significance.......................................................................................................... 100 PROPOSED SOLUTIONS ............................................................................................................. 100 “Abstragration”: A Linked Analysis Environment ............................................................. 101 Creation of the Conceptual Analysis Model ..................................................................................102 Applying “Abstragration”..............................................................................................................103

Full Probabilistic Environment .......................................................................................... 104 Tree Diagrams/Decision Trees.......................................................................................................105 Impact Dials...................................................................................................................................108

Summary of Solutions ......................................................................................................... 108 PROPOSAL OF NEW METHOD-POSSEM .......................................................................... 110 SUMMARY OF RESEARCH ......................................................................................................... 110 THE POSSEM FLOWCHART ..................................................................................................... 112 Difference between Analyst and Analysis Tool ................................................................... 112 CREATE THE CONCEPTUAL MODEL .......................................................................................... 114 IDENTIFY KEY DECISION NODES .............................................................................................. 116 CREATE LINKED ANALYSIS ENVIRONMENT ............................................................................. 117 CREATE FULL PROBABILISTIC ENVIRONMENT.......................................................................... 118 ANALYSIS ................................................................................................................................. 119 POSSEM PROOF OF CONCEPT............................................................................................ 122 SURVIVABILITY CONCEPTS....................................................................................................... 122 The Need to Bring Survivability into Preliminary Design Process..................................... 124

viii

The Paradigm Shift and Survivability............................................................................................127 The Link to System Effectiveness..................................................................................................130

EXAMPLE: CREATE THE CONCEPTUAL MODEL......................................................................... 130 Answers to Key Questions................................................................................................... 131 What problem are we trying to solve? ...........................................................................................131 What level of detail do we need? ...................................................................................................132 What tools are needed and available? ............................................................................................134 FLOPS ......................................................................................................................................134 ITEM ........................................................................................................................................136 Mission Level Code..................................................................................................................147

Baselines ............................................................................................................................. 148 Aircraft Baseline: F/A-18C............................................................................................................149 Design Mission.........................................................................................................................151 Validation .................................................................................................................................152

System Inputs and Outputs.................................................................................................. 155 Inputs: Engineering Level..............................................................................................................157 Outputs: Campaign Level ..............................................................................................................159

Scenario .............................................................................................................................. 160 Summary of Conceptual Model........................................................................................... 164 EXAMPLE: IDENTIFY KEY DECISION NODES............................................................................. 165 EXAMPLE: CREATE LINKED ANALYSIS ENVIRONMENT ............................................................ 167 Engineering Level: FLOPS................................................................................................. 169 Radar Cross Section Mapping........................................................................................................170

Campaign Level: ITEM....................................................................................................... 172 Mission Level Mapping....................................................................................................... 175 Detectability...................................................................................................................................176 Maneuverability .............................................................................................................................183

Linking the Codes Together ................................................................................................ 186

ix

EXAMPLE: CREATE FULL PROBABILISTIC ENVIRONMENT ........................................................ 188 Creation of the Metamodels................................................................................................ 189 The Complete Probabilistic Analysis Environment ............................................................ 192 Adding the Probabilistics...............................................................................................................193

EXAMPLE: ANALYSIS ............................................................................................................... 195 Engineering Level Results................................................................................................... 196 Screening Test ...............................................................................................................................197 Prediction Profile ...........................................................................................................................200

Mission Level Results.......................................................................................................... 206 Prediction Profile ...........................................................................................................................206

Campaign Level Results...................................................................................................... 207 Prediction Profiles .........................................................................................................................208

Complete Environment Results ........................................................................................... 214 Effect of Varying Design Variables...............................................................................................215 Effect of Varying Threat Variables................................................................................................219 Effect of Fully Probabilistic Environment .....................................................................................224

CONCLUDING REMARKS ..................................................................................................... 232 RESEARCH QUESTIONS AND ANSWERS ..................................................................................... 238 RECOMMENDATIONS ................................................................................................................ 240 REFERENCES ........................................................................................................................... 245 VITA............................................................................................................................................ 253

x

LIST OF FIGURES FIGURE 1 – WEAPONS SYSTEM EFFECTIVENESS EXAMPLE............................................................................ 12 FIGURE 2 – PARADIGM SHIFT: BRINGING KNOWLEDGE FORWARD IN DESIGN PROCESS ............................... 15 FIGURE 3 – PARADIGM SHIFT: FROM PERFORMANCE BASED QUALITY TO EFFECTIVENESS BASED QUALITY ............................................................................................................................................................. 17 FIGURE 4 – THE AIRCRAFT AS THE SYSTEM .................................................................................................. 17 FIGURE 5 – SYSTEM OF SYSTEMS FORMULATION .......................................................................................... 20 FIGURE 6 – EXAMPLE OF A PARETO PLOT ..................................................................................................... 36 FIGURE 7 – EXAMPLE OF A PREDICTION PROFILE .......................................................................................... 37 FIGURE 8 – PROCESS TO CREATE TIF ENVIRONMENT AND ASSESS TECHNOLOGY SCENARIOS ..................... 39 FIGURE 9 – EXAMPLE K-FACTOR NOTIONAL SHAPE FUNCTION FOR A WEIGHT REDUCTION TECHNOLOGY DIAL .................................................................................................................................................... 41 FIGURE 10 – EXAMPLES OF A PROBABILITY DENSITY FUNCTION AND A CUMULATIVE PROBABILITY FUNCTION ............................................................................................................................................ 45 FIGURE 11 – MODEL CATEGORIZATION ........................................................................................................ 52 FIGURE 12 – CHARACTERISTICS OF A CONCEPTUAL MODEL ......................................................................... 55 FIGURE 13 – SIMTAX CATEGORIZATION OF WARFARE SIMULATION .......................................................... 60 FIGURE 14 – TRADITIONAL PYRAMID OF MILITARY MODELS ....................................................................... 64 FIGURE 15 – ALTERNATE HIERARCHICAL STRUCTURE OF MILITARY MODELS I........................................... 64 FIGURE 16 – ALTERNATE HIERARCHICAL STRUCTURE OF MILITARY MODELS II.......................................... 65 FIGURE 17 – ENGINEERING LEVEL MODEL FEATURES .................................................................................. 67 FIGURE 18 – MISSION LEVEL MODEL FEATURES .......................................................................................... 68 FIGURE 19 – COMMON CAMPAIGN CODES IN USE TODAY ............................................................................ 69 FIGURE 20 – CAMPAIGN LEVEL MODEL FEATURES....................................................................................... 70 FIGURE 21 – THE MILITARY CODE CONTINUUM ........................................................................................... 74 FIGURE 22 – THE ITEM GRAPHICAL INTERFACE AND ENVIRONMENT .......................................................... 77

xi

FIGURE 23 – APPLYING CURRENT METHODOLOGY USING ITEM ................................................................. 78 FIGURE 24 – FLORIDA SCENARIO WITH AIR SUPERIORITY OPERATIONS SITUATION ..................................... 79 FIGURE 25 – BLUE AIRCRAFT MODELED FOR SCENARIO .............................................................................. 80 FIGURE 26 – RED SAM SITE DETAIL ............................................................................................................ 80 FIGURE 27 – PARETO PLOTS FOR THEATER SCENARIO SCREENING TEST ...................................................... 84 FIGURE 28 – PREDICTION PROFILE FOR THEATER SCENARIO SCREENING TEST ............................................ 85 FIGURE 29 – FLORIDA SCENARIO USED IN THEATER LEVEL SURVIVABILITY STUDY .................................... 87 FIGURE 30 – SAM SITE WEAPON COMPARISON FOR THEATER LEVEL SURVIVABILITY STUDY .................... 87 FIGURE 31 – PREDICTION PROFILE FOR THEATER LEVEL SURVIVABILITY STUDY ........................................ 92 FIGURE 32 – A ZOOMING APPROACH: BREAKDOWN OF RDTE..................................................................... 96 FIGURE 33 – FLOWCHART FOR DECISION-MAKING FOR ITEM...................................................................... 98 FIGURE 34 – NOTIONAL EXAMPLE OF DECISION TREE DIAGRAM ............................................................... 106 FIGURE 35 – PROPOSED FULL PROBABILISTIC ENVIRONMENT .................................................................... 107 FIGURE 36 – SUMMARY OF ISSUES AND PROPOSED SOLUTIONS .................................................................. 109 FIGURE 37 – THE POSSEM FLOWCHART ................................................................................................... 113 FIGURE 38 – CREATE CONCEPTUAL MODEL STEP OF POSSEM.................................................................. 116 FIGURE 39 – IDENTIFY KEY DECISION NODES STEP OF POSSEM............................................................... 117 FIGURE 40 – CREATE LINKED ANALYSIS ENVIRONMENT STEP OF POSSEM .............................................. 118 FIGURE 41 – CREATE FULL PROBABILISTIC ENVIRONMENT STEP OF POSSEM .......................................... 119 FIGURE 42 – ANALYSIS AND FINAL STEP OF POSSEM ............................................................................... 120 FIGURE 43 – VOLPE’S SURVIVABILITY 3-D APPROACH .............................................................................. 123 FIGURE 44 – RELATIONSHIP BETWEEN SURVIVABILITY AND LIFE CYCLE COST .......................................... 125 FIGURE 45 – EFFECT OF SURVIVABILITY ON FORCE SIZE ............................................................................ 126 FIGURE 46 – EFFECT OF SURVIVABILITY ON FORCE EFFECTIVENESS .......................................................... 127 FIGURE 47 – CREATE CONCEPTUAL MODEL STEP OF POSSEM.................................................................. 131 FIGURE 48 – FLOPS ANALYSIS FLOWCHART ............................................................................................. 136 FIGURE 49 – HIERARCHICAL STRUCTURE OF ITEM DATABASE .................................................................. 139

xii

FIGURE 50 – SAM ENGAGEMENT FLOW CHART ........................................................................................ 142 FIGURE 51 – SURFACE TO AIR MISSILE ENGAGEMENT PROCESS ................................................................. 145 FIGURE 52 – SAM INTERCEPT OPPORTUNITIES .......................................................................................... 146 FIGURE 53 – THREE VIEW OF BOEING F/A-18C .......................................................................................... 150 FIGURE 54 – DESIGN MISSION USED FOR SIZING AND MATCHING THE F/A-18C ....................................... 151 FIGURE 55 – F404-GE-402 ENGINE USED ON THE F/A-18C MODEL .......................................................... 153 FIGURE 56 – MATCHED DRAG POLARS FOR F/A-18C ................................................................................. 155 FIGURE 57 – DETERMINING SYSTEM LEVEL INPUTS AND OUTPUTS ............................................................ 156 FIGURE 58 – SCENARIO GEOGRAPHY FOR POSSEM EXAMPLE................................................................... 161 FIGURE 59 – SCENARIO WEAPONS .............................................................................................................. 162 FIGURE 60 – SCENARIO SAM SITE.............................................................................................................. 162 FIGURE 61 – SCENARIO VEHICLES .............................................................................................................. 163 FIGURE 62 – SCENARIO AIRBASES .............................................................................................................. 163 FIGURE 63 – IDENTIFY KEY DECISION NODES STEP IN POSSEM ............................................................... 165 FIGURE 64 – DECISION NODES FOR POSSEM SURVIVABILITY EXAMPLE ................................................... 167 FIGURE 65 – CREATE LINKED ANALYSIS ENVIRONMENT STEP IN POSSEM............................................... 168 FIGURE 66 – USE OF PRE-EXISTING RESPONSE SURFACE EQUATION FOR RADAR CROSS SECTION ............. 171 FIGURE 67 – ENGINEERING LEVEL MAPPING .............................................................................................. 171 FIGURE 68 – AIRCRAFT DATA WINDOW IN ITEM ...................................................................................... 173 FIGURE 69 – ENGINEERING LEVEL MAPPINGS AND CAMPAIGN LEVEL MAPPINGS ...................................... 175 FIGURE 70 – SECONDARY MISSION USING VARIABLE ALTITUDE................................................................ 183 FIGURE 71 – COMPLETE LINKED ANALYSIS ENVIRONMENT CREATED FOR SURVIVABILITY EXAMPLE....... 188 FIGURE 72 – CREATE FULL PROBABILISTIC ENVIRONMENT STEP IN POSSEM........................................... 189 FIGURE 73 – COMPLETE PROBABILISTIC ANALYSIS ENVIRONMENT ............................................................ 193 FIGURE 74 – ANALYSIS STEP IN POSSEM .................................................................................................. 196 FIGURE 75 – ANALYSIS AND RESULTS OVERVIEW ...................................................................................... 196 FIGURE 76 – PARETO PLOTS FOR ENGINEERING LEVEL SCREENING TEST 1 ................................................ 199

xiii

FIGURE 77 – PARETO PLOTS FOR ENGINEERING LEVEL SCREENING TEST 2 ................................................ 199 FIGURE 78 – PREDICTION PROFILE FOR ENGINEERING LEVEL ..................................................................... 205 FIGURE 79 – PREDICTION PROFILE FOR MISSION LEVEL ............................................................................. 207 FIGURE 80 – PREDICTION PROFILE FOR CAMPAIGN LEVEL, CASE 1 ............................................................ 212 FIGURE 81 – PREDICTION PROFILE FOR CAMPAIGN LEVEL, CASE 4 ............................................................ 214 FIGURE 82 – PROBABILITY DENSITY FUNCTIONS FOR SURVIVABILITY OF BLUE-1 AIRCRAFT WITH PROBABILISTIC DESIGN ENVIRONMENT ............................................................................................. 217 FIGURE 83 – PROBABILITY DENSITY FUNCTIONS FOR SURVIVABILITY OF BLUE-2 AIRCRAFT WITH PROBABILISTIC DESIGN ENVIRONMENT ............................................................................................. 218 FIGURE 84 – OVERLAY CHARTS FOR AIRCRAFT SURVIVABILITY WITH PROBABILISTIC DESIGN ENVIRONMENT ................................................................................................................................... 219 FIGURE 85 – PROBABILITY DENSITY FUNCTIONS FOR SURVIVABILITY OF BLUE-1 AIRCRAFT WITH PROBABILISTIC THREAT ENVIRONMENT ............................................................................................ 221 FIGURE 86 – PROBABILITY DENSITY FUNCTIONS FOR SURVIVABILITY OF BLUE-2 AIRCRAFT WITH PROBABILISTIC THREAT ENVIRONMENT ............................................................................................ 222 FIGURE 87 – COMPARISON OF PDF AND CDF FOR AIRCRAFT SURVIVABILITY, COMPOSITE CAMPAIGN .... 223 FIGURE 88 – OVERLAY CHARTS FOR AIRCRAFT SURVIVABILITY WITH PROBABILISTIC THREAT ENVIRONMENT ................................................................................................................................... 224 FIGURE 89 – PROBABILITY DENSITY FUNCTIONS FOR SURVIVABILITY OF BLUE-1 AIRCRAFT WITH FULLY PROBABILISTIC ENVIRONMENT .......................................................................................................... 226 FIGURE 90 – PROBABILITY DENSITY FUNCTIONS FOR SURVIVABILITY OF BLUE-2 AIRCRAFT WITH FULLY PROBABILISTIC ENVIRONMENT .......................................................................................................... 227 FIGURE 91 – OVERLAY CHARTS FOR AIRCRAFT SURVIVABILITY WITH FULLY PROBABILISTIC ENVIRONMENT ........................................................................................................................................................... 228 FIGURE 92 – COMPARISON OF PDFS AND CDFS FOR BLUE-1 SURVIVABILITY, COMPOSITE CAMPAIGN ..... 229 FIGURE 93 – OVERLAY CHART COMPARING CONTRIBUTIONS OF DESIGN AND THREAT ENVIRONMENTS TO FULLY PROBABILISTIC ENVIRONMENT FOR BLUE-1 SURVIVABILITY ................................................. 229

xiv

FIGURE 94 – CUMULATIVE EFFECTS OF VARYING SETS OF INPUT DATA FOR SURVIVABILITY RATE OF BLUE-1 .............................................................................................................................................. 231

xv

LIST OF TABLES TABLE 1– EXAMPLE DESIGN OF EXPERIMENTS TABLE.................................................................................. 32 TABLE 2– SEVERAL DOES AND REQUIRED EXPERIMENTAL CASES ............................................................... 33 TABLE 3 – INPUTS FOR THEATER LEVEL SCENARIO ...................................................................................... 81 TABLE 4 – OUTPUTS FROM THEATER SCENARIO ........................................................................................... 82 TABLE 5 – INPUT VARIABLES FOR THEATER LEVEL SURVIVABILITY STUDY ................................................ 89 TABLE 6 – OUTPUT VARIABLES FOR THEATER LEVEL SURVIVABILITY STUDY ............................................. 90 TABLE 7 – SURVIVABILITY ENHANCEMENT CONCEPTS ............................................................................... 124 TABLE 8 – DESIRED CAPABILITIES AND RESULTING MODEL FEATURES IN ITEM....................................... 138 TABLE 9 – SELECTED CHARACTERISTIC DATA OF BOEING F/A-18C........................................................... 150 TABLE 10 – GENERAL ENGINE SPECIFICATIONS FOR F404-GE-402............................................................ 152 TABLE 11 – WEIGHTS MATCHING FOR THE F/A-18C IN FLOPS ................................................................. 154 TABLE 12 – SYSTEM LEVEL INPUTS ............................................................................................................ 158 TABLE 13 – SYSTEM LEVEL OUTPUTS ......................................................................................................... 159 TABLE 14 – OUTPUT PARAMETERS TRACKED FROM FLOPS....................................................................... 169

xvi

NOMENCLATURE ε

Error term in a response surface equation

λ

Wavelength of the radar

σ

Radar cross section

ANOVA

Analysis of Variance

b0

Intercept term in a response surface equation

bi

Regression coefficients for the first order terms in an RSE

bii

Coefficients for the pure quadratic term in an RSE

bij

Coefficients for the cross-product terms in an RSE

CCD

Central Composite Design

CDF

Cumulative Distribution Function

CINCPAC

Commander in Chief Pacific

COFA

Concept Feasibility Assessment

dB

Decibels

DoD

Department of Defense

DoE

Design of Experiments

Ek

Estimated Kill

FedEx

Federal Express

FLOPS

Flight Optimization System (computer code)

Fmaneuver_elem

Maneuverability factor for aircraft in the raid element

Freldet,r,e

Relative detectability of raid element

G

Gain or electronic amplification of the radar xvii

GUI

Graphical User Interface

Hs

Height of the radar for the SAM site

Hr

Altitude of the raid

IPPD

Integrated Product and Process Design

ITEM

Integrated Theater Engagement Model (computer code)

JHAPL

Johns Hopkins Applied Physics Laboratory

JTCG/AS

Joint Technical Coordinating Group on Aircraft Survivability

k

Total number of variables considered in an RSE

k

Boltzmann’s constant (1.38 X 10-23 Watt-second/°K)

k1

Weighting factor for excess energy

k2

Weighting factor for turning rate

k3

Weighting factor for turning radius

k-factor

Multiplicative factor to a variable

krcs/pw

Multiplicative factor on RCS for parasitic weight

L

Loss factor accounting for atmospheric attenuation of radar energy

L/D

Lift to Drag ratio

MoE

Measures of Effectiveness

MORS

Military Operations Research Society

Nac_damage, e Number of aircraft damaged Nac_elem

Number of aircraft or weapons in the raid element

NASA

National Aeronautics and Space Administration

MoP

Measures of Performance xviii

Ndetected

Total expected targets

Ndet

Number of raid elements detectable at minimum distance

Nsalvos_elem Number of salvos launched at each raid element Nsim_engage Number of simultaneous engagements for the site Nweps_avail Number of weapons available Nr,e

Number of original raid elements

P

Value to be converted into decibels

P(dB)

Value in decibels

PDF

Probability Density Function

Ph

Probability of Hit

Pk

Probability of Kill

Pk/h

Probability of Kill given Hit

Pksalvo_elem Probability of kill of the salvo against the raid element Pksam, elem Probability of kill of the SAM against the raid element Pmin

Minimum receiver signal threshold

P0

Reference value when converting to decibels

Pop_fc, s

Probability the site's fire control is operational

Pop_radar,s

Probability that SAM radar is operational

POSSEM

PrObabilistic System of Systems Effectiveness Methodology

Ps

Probability of Survival

Ps,sizing

Excess energy from aircraft sizing

Ps,sizing

Excess energy from the alternate mission xix

Pt

Radar transmitter power

R

Desired response term in a response surface equation

Rcpa

Closest point of approach of raid element to SAM site

RCS

Radar Cross Section

RCSold

RCS value from existing RSE

RCSnew

RCS value after multiplying by parasitic weight factor, krcs/pw

Rdet_unlim,e

Non-horizon-limited radar range for SAM site

Rdetect, s

Site's maximum detection and tracking range against the raid element with the largest relative detectability

RDTE

Research, Development, Test, Evaluation

Rh

Horizon-limited radar range for SAM site

RSE

Response Surface Equation

Rsam

SAM range

RSM

Response Surface Methodology

Rtrk,s,

Tracking range of the SAM site

Ssam

SAM speed

Sraid

Raid speed

Sratio

Speed ratio = Sraid / Ssam

Ssalvo

SAM salvo size

SAM

Surface to Air Missile

TIES

Technology Identification/Evaluation/Selection (methodology)

TIF

Technology Impact Forecasting (methodology) xx

TNR

Threshold to Noise Ratio

Trad,s

Turning radius from the aircraft sizing

Trad,alternate

Turning radius from the alternate mission

Trate,s

Turning rate from the aircraft sizing

Trate,alternate

Turning rate from the alternate mission

Treact

Reaction time of SAM fire

Ts

System noise temperature and includes internal radar and electronics and external random noise effects

UPS

United Parcel Service

WWII

World War II

xi ,xj

Independent variables in a response surface equation

xxi

SUMMARY Significant advances have been made recently in applying probabilistic methods to aerospace vehicle concepts. Given the explosive changes in today’s political, social, and technological climate, it makes practical sense to try and extrapolate these methods to the campaign analysis level. This would allow the assessment of rapidly changing threat environments as well as technological advancements, aiding today’s decision makers. These decision makers use this information in three primary ways: resource allocation, requirements definition, and trade studies between system components. In effect, these decision makers are looking for a way to quantify system effectiveness. Using traditional definitions, one can categorize an aerospace concept, such as an aircraft, as the system. Design and analysis conducted on the aircraft will result in system level Measures of Effectiveness.

System effectiveness, therefore,

becomes a function of only that aircraft’s design variables and parameters. While this method of analysis can result in the design of a vehicle that is optimized to its own mission and performance requirements, the vehicle remains independent of its role for which it was created: the warfighting environment. It is therefore proposed that the system be redefined as the warfighting environment (campaign analysis) and the problem be considered to have a system of systems formulation. A methodology for the assessment of military system effectiveness is proposed. Called POSSEM (PrObabilisitic System of System Effectiveness Methodology), the methodology describes the creation of an analysis pathway that links engineering level changes to campaign level measures of effectiveness. xxii

The methodology includes

probabilistic analysis techniques in order to manage the inherent uncertainties in the problem, which are functions of human decision making, rapidly changing threats, and the incorporation of new technologies. An example problem is presented, in which aircraft survivability enhancements are added to a baseline aircraft, and the effects of these additions are propagated to the campaign analysis level.

xxiii

1

Danielle Soban

CHAPTER I

INTRODUCTION

Motivation Assessing the success and effectiveness of today’s complex systems becomes an increasingly challenging problem. Demands for increased performance, lower system life cycle costs, longer operating capacities and improved productivity and efficiency must be balanced against limited resources, scant and sometimes unknown data, the identification and resolution of conflicts and problems, and resource allocation [1]. Consideration of these tradeoffs dictates the need for an integrated and systematic methodology that can identify potential problem areas and assess system effectiveness during all phases of the system’s life cycle. This analytical framework must also support decision-making between alternatives and options while assessing the consequences of such decisions. In the current world military environment, system effectiveness takes on a new meaning. In the past, military aircraft design has been characterized by an emphasis to design for optimum performance. Aircraft success was defined in terms of the aircraft’s ability to perform at least as well as the requirements to which it was designed, effectively ignoring adaptability to rapidly changing threat environments. Performance was characterized by such attributes as speed, payload capacity, etc. Recent imperatives,

Danielle Soban

2

however, have shifted the emphasis from performance to overall system effectiveness as a key measure of merit for the aircraft. Today, system effectiveness must not focus only on the aircraft’s performance, but instead on its ability to satisfactorily complete its mission, against a wide variety of threats and situations, at an affordable life cycle cost. Aircraft survivability is a key metric that contributes to the overall system effectiveness of military aircraft as well as to a lower life cycle cost. As shown by Volpe [2], linear changes in survivability produce exponential changes in force effectiveness. In addition, a more survivable aircraft is both safer and more reliable (in peacetime operations) and thus reduces life cycle cost. Survivability, defined as the capability of an aircraft to avoid and/or withstand a man-made hostile environment, is a function of both susceptibility (denying the target and degrading the threat) and vulnerability (the inability of the aircraft to withstand the damage caused by the hostile environment) [3]. Susceptibility and vulnerability, in turn, are functions of characteristics such as shape, performance, agility, stealth, mission planning, mission requirements, and threat environment. Because these characteristics are themselves functions of basic design parameters and requirements, as are the other more traditional design disciplines, it becomes both necessary and cost-effective to consider survivability as a design discipline in its own right, allowing tradeoffs to occur at the preliminary design stage, thus optimizing the aircraft to an overall system effectiveness metric with a resulting reduction in design cycle time. The aircraft designer, therefore, must have a complete and thorough understanding of the interrelationships between the components of survivability and the

3

Danielle Soban

other traditional disciplines as well as how they affect the overall life cycle cost of the aircraft. If this understanding occurs, the designer can then evaluate which components and technologies will create the most robust aircraft system with the best system effectiveness at the lowest cost. Thus, there exists a need for an integrated and efficient framework that can rapidly assess system effectiveness and technology tradeoffs for today’s complex systems.

Need for a Military System Effectiveness Framework

The Changing Economy and its Effect on Decision Making In recent years, the world has been changing at a remarkable pace.

A

revolutionary new economy has risen. This economy is based on knowledge rather than conventional raw materials and physical labor [4]. With this new economy comes new emphasis on technology and its impact, especially in the warfighting environment. Almost all of the world’s countries spend a significant amount of their budget on the research, development and procurement of increasingly sophisticated weapons and warfare technologies [5].

This is necessary because countries need to maintain or

enhance their military capabilities in order to maintain their supremacy over their adversaries. In addition, strong and capable military capabilities serve as a deterrent to other countries that might otherwise turn aggressive.

However, the high cost of

maintaining these capabilities must be balanced against limited resources. Former U.S. Secretary of Defense Dick Cheney is credited with the statement “budget drives strategy,

Danielle Soban

4

strategy doesn’t drive budget” [4]. Military decision makers need to understand and assess the benefits and consequences of their decisions in order to make cost efficient, timely, and successful choices. Along with changes in the world’s economy come changes in the way war is fought.

Substantial progress has been made in both weapon lethality and military

technology. In addition, the battlefield of today has become increasingly complex, with interactions and their consequences becoming more and more difficult to isolate and understand. Because of the rapid advance of these developments, the decision makers are often left with ambiguous information and relatively short time spans to conduct analysis. Often, these changes occur so rapidly that previous analysis is rendered obsolete. For example, an aircraft that is designed to incorporate a certain avionics suite will often find that those avionics are obsolete by the time the aircraft comes into production. The inherent uncertainty in this information makes definitive analysis difficult and implies that the use of scenario-based probabilistic methods to understand and interpret this information is most appropriate. Overall, military decision makers need to be able to rapidly and efficiently answer questions such as those raised by Jaiswal [ 5]: What is the effectiveness of a weapon system or tactical plan in a plausible combat scenario? If the various factors influencing the performance of a system can be expressed qualitatively, can the performance be quantified? What force mix should be deployed for a specified mission?

Danielle Soban

5

How many types of weapons should be deployed on various sites to provide cost-effective defense? How should weapons be assigned to targets to achieve a specified objective? Who is likely to win? These questions all point to the need for a military analysis capability that takes places at the theater level. Decision makers must be able to take rapidly changing information and technologies and combine them with projected situations in order to make decisions and understand their consequences.

The Link to System Effectiveness What these decision makers are looking for is a quantification of system effectiveness. In this case, the system of interest is the warfighting theater or campaign. The primary tool of today’s military decision makers is the campaign analysis environment. These environments are modeling tools in the form of computer codes that model force-on-force engagements. They are often quite complex and vary in their abilities to capture different aspects of the warfighting environment. It is common for campaign analysis tools to have a detailed primary force models with only rudimentary modeling of secondary forces. For example, an Army code may have complex and sophisticated models for ground troops movement and support vehicle logistics, but have a relatively simplistic air campaign model, or even no air campaign model at all. True joint force models (models that capture all aspects of the warfighting environment with an equal level of analysis) are relatively few.

Danielle Soban

6

Measures of Effectiveness The output of these tools are system effectiveness quantifiers, or Measures of Effectiveness (MoEs). An MoE is a metric used to indicate the quality of a system [1]. It may be a measurable quantity or calculated from other output parameters, or it can also take the form of a weighted combination of several other metrics. These metrics often consist of final calculations of damage done or resources used. The following are some typical examples of campaign level MoEs: Number of Red aircraft shot down by Blue aircraft Number of damaged runways Distance in kilometers to halt Red advance Number of returning aircraft from a specific mission Each theater or campaign tool provides either its own set of hardwired MoEs or enough output data for the user to create his own system effectiveness metrics (or both). It is through the shrewd choice of these metrics that the decision maker links the MoEs to the answers to questions such as those posed above. Use of System Effectiveness Metrics There are three primary ways that decision makers utilize system effectiveness information: resource allocation, requirements definition, and system component trade studies.

7

Danielle Soban

Resource Allocation Most countries, when considering their military wants and needs, must deal with limited and often strict budgets. Different government agencies, often with competing agendas, must all vie for a finite set of resources. In addition, these agencies will often make decisions in isolation, negating the chance for potentially mutually beneficial, and cost effective, decisions.

Deciding how to allocate precious funds and resources,

therefore, becomes a key issue. System effectiveness concepts, when applied to the theater level, give the decision makers a way to link dollars to campaign level metrics. Comparisons may be made between dissimilar components of the system. For example, there may be a need to assess whether additional resources should be supplied to a missile program or an aircraft program. Straight one-on-one comparison of these two types of vehicles may be difficult because of their inherently different capabilities and performance. But when placed in the context of the overall system (the warfighting environment), their individual (or even combined!) effect on the overall system effectiveness metrics can be assessed and appropriate decisions made.

Requirements Definition Another way system effectiveness metrics aid the decision maker is in the development of requirements for system components.

Given the performance and

capabilities of a system component, a campaign analysis tool can use that information to assess the effect of that component. But this assessment capability may be turned around. By varying the capabilities and performance characteristics of a notional system

Danielle Soban

8

component, the optimal settings of these characteristics can be obtained that maximize system effectiveness. An aircraft may be used as an example. The question to be considered may be: what is the optimal aircraft strike speed needed to obtain a specific campaign objective? A notional aircraft is modeled and the strike speed allowed to vary in the campaign analysis until the selected MoEs reach their optimal value(s). Now the ideal strike speed is known for that class of vehicle. This information can be used to define a design goal for future aircraft, or it may be used to assess the potential of modifying existing aircraft to achieve the new strike speed. In this way, complete requirements for new system components and new technologies may be developed. Finally, sensitivities of the values of specific requirements may be assessed. This can be tremendously useful information: can a difficult requirement be relaxed, allowing cost savings or trade-offs between other characteristics, at an insignificant or acceptable reduction of overall system effectiveness?

Trade Studies Between System Components Finally, system effectiveness metrics can be used to assess the differing values and effects of system and sub-system components. As mentioned earlier, it is often difficult to compare and contrast dissimilar sub-systems. By placing those sub-systems in a larger framework (or system), the changes they affect in the top-level metrics may be observed and quantified. For example, say it was of interest to consider which of two avionics packages would be better to use on an existing aircraft. Analyzing changes in individual aircraft performance with each of the avionics packages could be difficult or

9

Danielle Soban

indistinguishable. But if the aircraft, with the avionics packages, were placed as system components in the theater, the effect of the avionics packages could be assessed. In this case the avionics packages were allowed to fulfill their intended function within the larger system, and thus their effects more easily quantified.

Lack of Overall System Effectiveness Methodology Given the power of a system effectiveness consideration of the modern warfighting environment, coupled with its usefulness in decision making, it is surprising to find a lack of cohesive and accepted methodologies used to address campaign level system effectiveness in the open literature. To be true, there are a multitude of campaign level modeling tools, and the creation, use, and improvement of these tools is a flourishing endeavor [6]. In addition, many decision makers and analysts use these tools in their own individual way.

But finding information specifically detailing overall

methodologies is difficult. There are several possible reasons for this lack of obvious resources. These reasons are detailed below. Semantics and a Surplus of Synonyms In order to formulate a systems effectiveness framework, it is important to understand and clearly define the concepts of both “system” and “system effectiveness”. There is general agreement across fields and disciplines as to what constitutes a system. The following definition is representative of this agreement, and is an acceptable definition for the developing framework:

Danielle Soban

10

A system may be considered as constituting a nucleus of elements combined in such a manner as to accomplish a function in response to an identified need…A system must have a functional purpose, may include a mix of products and processes, and may be contained within some form of hierarchy…[7] However, the definitions of system effectiveness vary widely and are often application dependent. Some examples that illustrate the diversity of these definitions include: “The overall capability of a system to accomplish its intended mission” [11] “The probability that the system can successfully meet an operational demand within a given time when operated under specified conditions” [8] “A measure of the degree to which an item can be expected to achieve a set of specific mission requirements, and which may be expressed as a function of availability, dependability and capability” [9] The authors of an annotated bibliography on system effectiveness models in 1980 concluded “A wide range of definitions, and measures of system effectiveness are used without strong guiding logic” [11]. The words “system effectiveness” and the concept they represent first reared its head in the 1950s and 1960s [10,11]. However, these early formulations of system effectiveness were defined primarily as functions of the “-ilities”: reliability, availability, repairability, and maintainability. As such, the system effectiveness concept was applied to a single component or tool that itself was defined as the system. For example, a missile would be defined as the system, and its system effectiveness assessed based on its availability, reliability, etc. While this was a revolutionary concept at the time, these

Danielle Soban

11

definitions are not as useful if the theater itself is considered the system. Each component of the system may be assessed by its “-ilities” but these “-ilities” are inadequate to serve solely as the theater level Measures of Effectiveness. These pioneering definitions are somewhat still in use today [10,1], making research specifically on campaign analysis system effectiveness difficult to isolate. Figure 1 shows one way in which the “-ilities” are incorporated into the effectiveness of a weapons system, in this case an aircraft [12]. Each of the effectiveness components, survivability, readiness, capability, and dependability, are further subdivided into those weapons features that determine that particular characteristic. These features are all distinctions of that particular weapons system. These features are then balanced against the various costs associated with that weapons system, and a balance, or tradeoff, is conducted, optimizing the features with respect to cost. In this way an overall operational effectiveness is computed as a weighted sum of each of the individual features. Finally, “system effectiveness” holds different meanings for different communities and applications. Some organizations tailor their definitions and methods to apply to very specific problems [11]. A representative of the Potomac Institute for Policy Studies offers that the difficulty in finding information on system effectiveness lies in the broad connotations of the term: “System Effectiveness has many different "branches", primarily based upon the application area, e.g., military system effectiveness, policy analysis, information system effectiveness, reliability analysis, etc. Within each application area there are multiple areas for consideration, e.g., in policy analysis there is the study of health care

12

Danielle Soban

reform and its affect (sic) on society; the effect of transportation policies on a metropolitan area, etc.” [13]

Cost

Operational Effectiveness

• Acquisition cost • Acquisition cost

Survivability Survivability

Readiness Readiness

Capability Capability

Dependability Dependability

• Operation cost • Operation cost • Maintenance cost • Maintenance cost

• Susceptibility • Susceptibility • Vulnerability • Vulnerability

• Maintainability • Maintainability • Inherent availability • Inherent availability

• Performance • Performance • Maneuverability • Maneuverability

• Safety • Safety • Reliability • Reliability

• Reliability • Reliability • Logistic support • Logistic support

• Lethality • Lethality

• Maintenance defects • Maintenance defects • Design defects • Design defects • Operations • Operations

• Aircraft replacement • Aircraft replacement • Crew replacement • Crew replacement training training • RDT&E cost • RDT&E cost

Affordability Measurement Effectiveness = k 1(Capability)+ k 2(Survivability)+ k 3(Readiness)+ k 4(Dependability) + k 5(Life Cycle Cost)

Survivability Maneuverability Availability Safety Capability

RDT&E Cost O&S Cost Acquisition Cost

Figure 1 – Weapons System Effectiveness Example

In addition, “system effectiveness” is often synonymous with other concepts, such as “operations research” and “systems analysis”. However, even these other concepts umbrella a huge array of specific analysis approaches and definitions, and locating the unique niche of military system effectiveness is difficult. For example, Reference 5 is a very recent (1997) state-of-the-art book on Military Operations Research. This book uses the words “system effectiveness” only once in a brief, passing note.

Similarly,

Kececioglu [10] in his 1995 book devotes only one small section to system effectiveness

Danielle Soban

13

and defines it again in terms of mission reliability, operational readiness, and design adequacy, which is again difficult to apply to the theater. A new, consistent definition for system effectiveness, therefore, is necessary and must be justified by identifying key elements crucial to a useful and informative definition. First, the term “effectiveness” implies that some sort of quantification needs to occur. This quantification must necessarily be the result of some sort of systematic analysis of variables and metrics that represent the system performing its function. In addition, in order to perform the quantification, an intended or expected effect needs to be identified in order to properly model the results of the system performance. Combined, these concepts result in the following definition put forth by the author for use in formulating the framework for the probabilistic assessment of system effectiveness: System effectiveness is a quantification, represented by system level metrics, of the intended or expected effect of a system achieved through functional analysis. This definition will be used for the remainder of this dissertation and subsequent research. Another confusion arises when there is a lack of distinction between the modeling tools and the methodologies that use the tools. Research that asks the question “What is the current state of the art in system effectiveness methodologies?” often turn up only the codes that can be used in such methods. A true methodology should be a freestanding framework that is relatively independent of the tools it utilizes. As the tools improve in fidelity, they should be able to be substituted into the methodology with little or no

14

Danielle Soban

interruption. Because the answer to this question usually results in a listing of modeling codes rather than methods or frameworks, an inherent lack of such methodologies is indicated. Difficulty Assessing Government and Classified Material Originally, system effectiveness studies were confined to military and space systems. Agencies of the US Government, such as the Department of Defense and the National Aeronautics and Space Administration (NASA), were the ultimate customers. Because of this, the available literature on system effectiveness and the accompanying models were published primarily as technical reports, but rarely appear in widely published journals [11].

Today’s analysts appear to have new interest in system

effectiveness studies using campaign modeling, especially in the area of technology infusions. However, much of this work is classified or proprietary, limiting accessible publications and information. Finally, those non-government agencies that do make advances in theater modeling and system effectiveness may find it necessary to keep their in-house methods proprietary in order to retain their competitive edge. Because of these restrictions, some fundamental contributions to this field do not appear in this body of research.

System of Systems Approach In order to successfully formulate a system effectiveness methodology, it is imperative to clearly define the system and its components. The preceding sections

15

Danielle Soban

discussed the benefits to the decision maker of considering system effectiveness at the theater or campaign level. This endpoint represents an expanding progression of what is considered the system. The resulting “system of systems” formulation is a key concept in the development of the proposed methodology.

A Paradigm Shift In traditional design, most design decisions are made relatively early in the process, when the designer (or design team) has the least available knowledge about the proposed new system. Design decisions lock in financial commitments, so the bulk of the cost is committed early in the design process. As these decisions are made, design freedom falls off rapidly (Figure 2).

A paradigm shift, founded on the notion of

Integrated Product and Process Design (IPPD), is now widely accepted. IPPD seeks to bring more knowledge about the system life cycle to an earlier stage of the design process, in an attempt to delay cost commitments and also keep design freedom open [14]. In other words, the designer needs to understand and quantify the implications of her/his decisions earlier in the design process in order to effectively reduce cost.

Figure 2- Paradigm Shift: Bringing Knowledge Forward in Design Process

Danielle Soban

16

In addition, there is a parallel paradigm shift that considers what the measure of “goodness” is for a system. Traditionally, differing designs would be compared based on their performance. For example, the questions that would mark the “goodness” of an aircraft would be of the sort:

How fast does it fly? How far can it fly? How much payload can it support? All comparisons between competing designs would be based on performance. The new paradigm shifts this emphasis not to individual system performance but to system effectiveness (Figure 3). For an aircraft, this effectiveness would be illustrated by the answers to such questions as:

What is the exchange ratio? What is the damage per sortie? What is the maintenance hours per flight hours cost? Together, these two paradigm shifts represent a broadening view of the design process, expanding the ideas and concepts from detailed particulars to a “big picture” representation. This momentum will be carried forward, further expanding these basic concepts, to result in a system of systems depiction.

17

Danielle Soban

• Exchange ratio? • Damage per sortie?

Altitude

• How high? • How far? • How fast?

paradigm shift

Mach

Quality based on Performance

Quality based on Effectiveness

Figure 3- Paradigm Shift: From Performance Based Quality to Effectiveness Based Quality

The Theater as the System Using the traditional definitions, one can categorize an aerospace concept, such as an aircraft, as the system. Design and analysis conducted on the aircraft will result in system level Measures of Effectiveness. System effectiveness, therefore, becomes a function only of that aircraft’s design variables and parameters. The relationship between the aircraft’s input design parameters and its outputs (called responses, or MoEs) is

Vehicle Level

Metrics/Objectives

Responses

Constraints

MoEs

illustrated in Figure 4.

relationships between inputs and outputs Top Level Requirements

Vehicle/Design Econ. Vars.

Technology K-factors

Figure 4- The Aircraft as the System

18

Danielle Soban

While this method of analysis can result in the design of a vehicle that is optimized to its own mission and performance requirements, the vehicle remains independent of its role for which it was created. In other words, the aircraft is never placed in its correct context and evaluated as a system fulfilling its intended function. In order to place the aircraft in its correct context, the system must be expanded and redefined. No longer is the aircraft the sole system; rather let the aircraft’s intended environment become the system. For a military aircraft, this new, larger system is the warfighting environment: the theater. Thus, the theater (system) becomes a function of its components (systems in their own right, yet sub-systems here) and the overall formulation becomes a “system of systems”. There is, however, a missing level in this formulation. The outputs of the vehicle level (performance parameters) do not usually map directly as inputs to theater level modeling codes. Rather, the inputs at the theater level usually consist of probability of kill values, or effectiveness values that are the result of component vs. component encounters. There must be an intermediary mapping that takes the output of the vehicle level as its inputs, and in turn generates outputs that serve as inputs to the theater level. This concept is illustrated in Figure 5.

With this formulation comes a necessary

redefinition of output parameters, solely for clarity. The output responses of all sublevel analysis will be called Measures of Performance (MoPs) and the output of the top level system (in this case, the theater) will be called Measures of Effectiveness.

Thus,

referring to Figure 5, theater level MoEs are functions of vectors of subsystem MoPs (at

Danielle Soban

19

the engagement level) which are in turn functions of the requirements, design and economic variables, and technology factors associated with the vehicle level inputs. When the methodology is complete, there will exist a continuous mapping between vehicle level design parameters and theater level Measures of Effectiveness. Changes at the vehicle level can thus be propagated all the way to the theater level. Instead of optimizing an aircraft, for example, to its own pre-defined performance and mission constraints, the aircraft can now be optimized to fulfill theater level goals and objectives. In addition, as more system level components are treated as input variables, tradeoffs can be established not only at the individual component level, but across the components. In other words, the methodology will allow tradeoffs between, say, the effectiveness of a surface-launched cruise missile compared to an aircraft carrying a specified weapons load. Tradeoffs could also be made between the number of system components needed: two of aircraft “A” could produce the same effectiveness of five of aircraft “B”, but at less cost. Thus, the methodology becomes a key device for design decisions as well as resource allocation. Finally, the completed methodology can be used to actually determine the mission and design requirements for the vehicles themselves that comprise the system. By using the Measures of Effectiveness at the theater level as a measure of goodness, tradeoffs can be made between vehicle design and mission requirements. These requirements, when optimized to maximize the overall effectiveness of the system, become the requirements to which the vehicles are then designed.

20

MoEs

Danielle Soban

Theater Level System MoE = fn( MoP1, MoP2, MoP3, etc)

MoPs

MoEs become MoPs

Mission Level

Metrics/Objectives

Responses

MoP = fn( Xreq, Xdesign/econ, Xtech factors)

Vehicle Level

Constraints

MoPs

MoPs of vehicle become variables for next level

Top Level Requirements

Vehicle Design/ Econ . Vars

Technology k-factors

Figure 5- System of Systems Formulation

A note must be made at this point concerning system decomposition. The first system to be put forth was the aircraft itself. It becomes obvious, in the light of the previous discussion, that the aircraft itself is a system of systems. The aircraft is made up of a variety of major components, which can be seen as sub-systems. In turn, each of these subsystems can be seen to be functions of their components, so they, too, are each a system of systems. Going in the other direction, the engagement level can be seen as a function of not just one aircraft vs. aircraft scenario, but must be comprised of many differing engagements in order to generate a complete set of information for the next system in the hierarchy: the theater. System decomposition, therefore, can be understood to have a pyramid shape, with each level in the decomposition being subdivided into its components, which are in turn subdivided. The question becomes, then, where does one stop? How much subdividing is necessary, and how many levels are needed? The

Danielle Soban

21

answer to this depends on the definition of the problem that is being studied and the tools that are available. This leads to the idea of the “conceptual model”, which is discussed more thoroughly in Chapter III. It is up to the skill and experience of the designer or decision maker to accurately and adequately bound the problem and define the system and its components effectively.

Mathematical Modeling Once the system and its components have been clearly identified, an analysis environment must be created. The key word in the definition of system effectiveness is “quantification”. In order for the decision maker or designer to analyze the system effectively, the results of the analysis must be presented as quantifiable metrics. This involves restating a research goal or design decision into a question that can be answered quantitatively. Dixon [15] states this explicitly: “An engineering analyst must begin by defining quantitatively answerable questions”.

Mathematical methods, thus, become

primary tools in system analysis because of their ability to rapidly provide these calculable (quantifiable) metrics. In addition, mathematical modeling allows the user to understand and make informed decisions at various levels within the system hierarchy. With the “system of systems” concept comes an appreciation of the potential complexities and interactions involved.

Mathematical modeling offers significant benefits: “There are many

interrelated elements that must be integrated as a system and not treated on an individual basis. The mathematical model makes it possible to deal with the problem as an entity

Danielle Soban

22

and allows consideration of all major variables of the problems on a simultaneous basis [14].” Use of Probability Theory The paradigm shift of Figure 2 makes the argument that bringing knowledge forward in time results in better decision making. However, it must be recognized that this knowledge has an associated uncertainty with it. This lack of certain knowledge could be based on missing, unavailable, or incomplete information, the incorporation of a new technology as a function of its readiness level, or even an uncertainty in the modeling tools used in the analysis. The question becomes how to accommodate this uncertainty into the mathematical modeling and subsequent analysis. The answer to this is to incorporate basic probabilistic elements into both the modeling and the analysis, and, by extrapolation, the overall system effectiveness methodology. Understanding the sources of the uncertainty helps determine why a probabilistic approach is useful. Referring back to the “system of systems” hierarchy, it is clear that each subsystem level will have its own inputs. Perfect knowledge about these inputs is rare, and it is often that the designer or decision maker must make assumptions based on available data and personal experience. Using probabilistic inputs would allow the user to account for variation in his assumptions. Analysis based on these probabilistic inputs could provide useful information about the sensitivities of the inputs, which in turn could be translated into requirements definitions. By allowing the inputs to vary, the designer or decision maker could play “what if” games, using the models as a computationally and economically inexpensive way to explore the boundaries of the problem. And finally,

Danielle Soban

23

variable inputs would allow an investigation of the robustness of a solution (i.e. that solution whose performance parameters are invariant or relatively invariant to changes in its environment). Another major source of uncertainty can be found when considering the incorporation of a new technology. Modeling current technologies is straightforward, with the performance parameters of that technology generally known. However, current technologies may not be capable of meeting customer needs or design goals. In addition, current technology may be obsolete by the time the system is implemented.

This

necessitates a prediction capability concerning the impact of new technologies. Performance of a new technology is a function of its readiness level, but that function may or may not be completely defined. By modeling a new technology in a probabilistic fashion, one can explore various assumptions pertaining to the performance and the corresponding effects of that technology. In addition, knowledge gained through the probabilistic analysis of a new technology could be used to decrease the technology’s development time. Overall, the presence of uncertainty in most complex systems points to the use of probabilistic elements. Coupled with a mathematical modeling capability, an analysis environment can be created for incorporation into a system of systems effectiveness methodology.

24

Danielle Soban

Research Questions In summary, there exists a need for an integrated and systematic methodology that can assess military system effectiveness. This methodology will be used to aid the decision maker in resource allocation, trade studies between system components, and requirements definition.

The inherent presence of uncertainty in such a process

necessitates the use of a probabilistic analysis environment, facilitated through the use of mathematical modeling. These needs give rise to specific research questions: 1) What are the needed elements of a systems effectiveness methodology?

2) How does one define the system and its components?

3) How does one link design level metrics to system level Measures of Effectiveness?

4) Can a methodology be formulated and used to analyze the effect of incorporating survivability concepts at the vehicle level, yet assess them at the theater level?

5) How can uncertainties, such as unknown or incomplete data, rapidly changing technologies, and uncertainties in the modeling

25

Danielle Soban

environment be taken into account in a system effectiveness methodology?

The remainder of this thesis outlines the tools and approaches needed to answer these questions.

Once the methodology has been formulated, a proof of concept

implementation is presented to clarify the various steps and concepts of the proposed method. In addition, the proof of concept is used to identify any issues that were overlooked in the formulation process.

26

Danielle Soban

CHAPTER II

INCORPORATING UNCERTAINTY: THE RESPONSE SURFACE METHOD AND MONTE CARLO ANALYSIS As discussed in the previous chapter, the modeling and inputs of the complete system will necessarily contain uncertainties. These uncertainties can be functions of data availability, the incorporation of new technologies, modeling deficiencies, and the modeling inputs themselves; an uncertain threat environment or the exploration of alternate scenarios.

In order to incorporate these uncertainties into the system

effectiveness methodology, an extrapolation of the basic response surface methodology coupled with a Monte Carlo analysis technique will be used. The following sections summarize the basic concepts of the response surface method, and illustrate how this collection of tools and techniques may be used to form a foundation for the incorporation of uncertainty into the system of systems effectiveness methodology.

Sizing and Synthesis At the vehicle level, system design is facilitated through the use of multidisciplinary sizing and synthesis techniques. Synthesis is defined as “the combining of separate elements or substances to form a coherent whole” [16]. In vehicle design, a system can be decomposed, often by discipline, in order for individual analysis to be performed. The recomposition of these elements into an integrated whole is known as

27

Danielle Soban

synthesis. Although synthesis can be achieved through a number of differing algorithms, for aerospace vehicles synthesis is usually accomplished in conjunction with a sizing routine. Sizing is often a physics-based process in which the geometric configuration of a vehicle is “flown” through a specified design mission. At each point in the mission, fuel burn is calculated and is ultimately matched to the fuel available to the configuration. This fuel balance is conducted simultaneously with an engine scaling, which matches drag produced against thrust available (thrust balance). This iterative process results primarily in a vehicle gross weight, wing area, and corresponding sized (scaled) thrust, but results may also include component weights, performance data, disciplinary outputs, and economics. Synthesis and sizing is most often accomplished through the use of computer codes. These codes use algorithms that can be based on the physics of the problem, historical databases and extrapolations, or through mathematical representations of empirical relationships. It is clear to see the inherent sources of uncertainty in such models. In addition, uncertainty can be introduced into the sizing and synthesis process through the incorporation of new technologies, which may or may not be represented appropriately in an analytic algorithm. Finally, changes in the inputs may introduce uncertainty with the goal of determining the design sensitivities to these changing inputs.

Metamodels Synthesis and sizing tools are used to analyze and simulate an aerospace vehicle concept. Through the use of these tools, design knowledge is brought into the process

28

Danielle Soban

sooner.

This is the goal of the paradigm shift discussed earlier.

In addition, the

multidisciplinary nature of these tools allows the designer to conduct an overall system analysis of the concept.

The drawback of these tools, however, stems from their

complexity. Each of the decomposed elements of the analysis environment contains its own set of algorithms, databases, and relationships. These individual elements are then combined in the synthesis process, which usually contains an overall algorithm that ties the pieces together and calls the separate analyses as they are needed. The result is often a tool that is useful in its capabilities but is expensive in terms of computational efficiency and run times. As the fidelity of the synthesis components increase, the expense increases likewise. One option is to use lower fidelity models that run more quickly, and sacrifice the accuracy. However, an alternative approach is to represent the higher fidelity analyses using approximation techniques. These representations are called metamodels. A metamodel is, in essence, a model of a model. It is formulated so that it captures most, if not all, significant effects of the more complex model it is emulating. There are several methods used to create metamodels, such as neural networks, statistical techniques, universal kriging, interpolative procedures, and Gaussian processes. While each of these methods has its own advantages and drawbacks, the method that will be used as a foundation for the system of systems effectiveness methodology will be response surface methodology, which is a subset of the statistical techniques.

29

Danielle Soban

Response Surface Methodology The response surface methodology (RSM) is an efficient, multivariate approach to modeling that defines clear cause-and-effect relationships between design variables and system responses. The technique has been used extensively since the 1950s for a variety of applications in areas such as engineering, agriculture, and chemistry [17,18,19]. Originally, RSM was used in situations where the only way to establish a relationship between inputs and responses was through the use of costly and complex experiments. This technique allows the maximum amount of information to be gained from the fewest number of experiment executions, and thus provides trade study results in a more costeffective manner. RSM is based on a statistical approach to building and rapidly assessing empirical models [19,20].

In general, in order to thoroughly establish a cause-and-effect

relationship between given system variables and system responses, there must exist a complete set of knowledge about the system.

Because this complete knowledge is

difficult (and often impossible) to obtain and identify, this knowledge can be approximated with an empirically-generated deterministic relationship.

The RSM

methodology, employing a Design of Experiments (DoE) strategy, aids in this by selecting a subset of combinations of variables to run experimentally which will guarantee orthogonality (i.e. the independence of the various design variables) and will allow for the creation of a statistically representative model.

30

Danielle Soban

Response Surface Equations The primary representation of the metamodel generated using RSM is a polynomial formulation known as a response surface equation (RSE). Although a variety of forms of the equation may be used, the most common is a quadratic polynomial that may be viewed as a truncated Taylor series approximation. The form of this type of RSE k

k

i =1

i =1

k −1

R = bo + ∑ bi xi + ∑ bii xi2 + ∑

k

∑b

i =1 j = i +1

ij

xi x j + ε

is:

R

is the desired response term

b0

is the intercept term

bi

are regression coefficients for the first order terms

bii

are coefficients for the pure quadratic term

bij

are the coefficients for the cross-product terms

xi ,xj

are the independent variables

k

is the total number of variables considered

ε

is the error term

When using RSM to create a metamodel of a computer code, the error ε can be assumed to be zero due to the repeatability of the experiment. This also changes, slightly, the form of the Design of Experiments discussed below. If the non-linearities of the problem are not sufficiently captured using this form of the equation, then transformations of the independent and dependent variables can be found and alternate forms of the equation, such as logarithmic or exponential, may be used.

Danielle Soban

31

The simplicity of the resulting RSE makes its attraction clear. The equation may be easily manipulated for optimization purposes. For probabilistic assessments, the RSE can be used in conjunction with a Monte Carlo analysis, providing significantly reduced run times over conducting a Monte Carlo analysis around the original code. Overall, the polynomial form of the RSE makes it a good choice for a metamodel.

Design of Experiments In order to create the response surface equations used in response surface methodology, the Design of Experiments technique is used. The DoE is expressed as a table or matrix, specifying which values to use for each experimental run.

When

modeling a computer code, each case represents one execution of the code, with input variables (the factors) set to the values specified by the DoE table. The values are usually normalized to a high, low, or midpoint value of the variable to aid in the statistical analysis, and to avoid inappropriate weighting of the responses. An example DoE table is shown in Table 1. After each experiment or code execution takes place, the responses (or outputs) of interest are parsed and placed in the response column of the table. A least squares regression analysis is used to determine the coefficients of the response surface equation. An Analysis of Variance (ANOVA) is then conducted to determine the relative importance and significance of each of the variables. There will be one RSE for each response, and that response will be a function of all of the experimental factors.

32

Danielle Soban

Table 1- Example Design of Experiments Table

Experimental Case

Factor 1

Factor 2

Factor 3

…Factor n

Response 1 (R1)

…Response n (Rn)

1

-

-

0

-

R1-1

Rn-1

2

-

+

0

+

R1-2

Rn-2

3

+

-

0

0

R1-3

Rn-3

4

+

+

0

+

R1-4

Rn-4

5

0

-

-

0

R1-5

Rn-5

6

0

-

+

0

R1-6

Rn-6

7

0

+

-

-

R1-7

Rn-7

8

0

+

+

-

R1-8

Rn-8

9

-

0

-

+

R1-9

Rn-9

10

+

0

-

-

R1-10

Rn-10

11

-

0

+

0

R1-11

Rn-11 Rn-12 …

12

+

0

+

-

R1-12













0 +

Minimum Value Nominal (midpoint) Value Maximum Value

Values are non-dimensionalized

There are many different types of DoEs. To gain the most complete information, a full factorial set of experimental cases needs to be conducted. This arrangement tests all possible combinations of variables. Because the total number of combinations in a full factorial is exponential, as the number of variables increase, so do the number of cases that need to be run. For example, a twelve variable experiment would require 312 or 531,444 experimental runs. This quickly becomes impractical for larger numbers of variables. To reduce the number of runs required to create the metamodel, yet retain as much useful information as possible, other DoE designs are considered. These designs are called fractional factorials and are created to take into account the portion of a full factorial that are needed to represent certain effects of interest. Table 2 (recreated from

33

Danielle Soban

[19]) shows several common designs, along with the total number of experimental runs that need to be conducted.

These designs are discussed in detail in References

[19,20,21]. For most purposes, the face-centered central composite design (CCD) is considered to be an efficient trade between accuracy and computational cost [22].

Table 2- Several DoEs and Required Experimental Cases Experimental Cases Needed Design of Experiments

Equation 7 Variables

12 Variables

Full Factorial

2,187

531,441

3

Central Composite Design

143

4,121

2 + 2n + 2

Box-Behnken

62

2,187

n/a

D-Optimal

36

91

(n+1)(n+2)/2

n

n

Using Response Surface Methodology Response Surface Methodology may be employed in a systematic way to generate metamodels as well as other useful information gained in the Analysis of Variance. The steps and tools commonly used in RSM are described below.

Setting Up the Problem The first step in the RSM process is identical to the first step in all analysis methodologies: the design and set up of the analysis. This involves a clear problem

Danielle Soban

34

definition or analysis goal. The inputs of interest must then be clearly identified and appropriate responses determined. These inputs and responses are first a function of the problem/goal, but the experimental environment must also be taken into account. Variables and metrics that represent the effects of interest must exist in the modeling environment. Once the input variables have been determined, variable ranges must be set to limit the scope of the problem. A high and low value of each variable will define the design space to be explored. Care must be taken that the ranges are large enough to capture the effects of interest, yet are confined enough to clearly isolate results.

Screening and Pareto Plots One of the limitations to the RSM method is the number of variables for which DoEs exist. For CCD and Box-Behnken designs, the number of input variables can be software limited. Custom designs for more variables are available [23], but the creation of statistically sound models is not trivial. In addition, an increase in the number of variables brings with it the “curse of dimensionality”; an increase in the number of experimental cases needs to be conducted in order to create a viable metamodel. It is beneficial, therefore, to restrict the number of variables to as few as possible. One way of doing this is to conduct a screening test. A screening test utilizes a two level fractional factorial DoE, which tests the fit of a linear model. This fit only accounts for main variable effects, and no interactions are calculated. Once the screening DoE is complete, an Analysis of Variance is conducted, yielding a Pareto plot that clearly identifies the most statistically significant variables to

Danielle Soban

35

the response selected [19,20,21]. A Pareto plot is a bar chart that displays the frequency, relative frequency, and cumulative frequency of a set of variables to a specific response. For example, the Pareto plot shown in Figure 6 indicates that the X6 variable contributes most significantly to the variability of the response, while all variables below X3 are negligible. By using the Pareto plot, some variables may be eliminated from the higher order metamodel by noting the ones with little or no effect, and setting those variables to their nominal value. To choose the variables to eliminate, the Pareto Principle may be utilized.

The Pareto Principle, named after Vilfredo Pareto, a nineteenth century

economist and sociologist, was coined by Dr. Joseph Juran during his work on Total Quality Management. The principle states that a small number of causes is responsible for a large percentage of the effect- usually a 20% to 80% ratio [24]. Applied to RSM, this means that all of the variables that cumulatively affect 80% of the response should be kept and the rest of the variables discarded (set to their nominal values and no longer considered as variables). A new, higher order DoE using these variables should be conducted. In the example in Figure 6, this means that only three variables should be kept for higher order analysis: X6, X4, and X2. Choosing the variables that contribute the most to the variability of the response, using the Pareto Principle, completes the screening test of the overall RSM process. A new DoE is selected based on the remaining variables, and the experiments or computer runs are conducted accordingly. It should be noted here that a variety of statistical analysis packages, such as JMP [23], Minitab [25], and iSIGHTTM [26], are available to aid in the construction of the DoE, the ANOVA, and the subsequent analysis. Based on

36

Danielle Soban

ease of use, capabilities, and availability, the statistical package JMP was selected for use in this research.

Pareto Plot of Estimates Term

80% of Response

Orthog Estimate

X6

1738.9192

X4

1146.0329

X2

-1004.6644

X5

983.8613

X1

255.9926

X3

86.5018

X10

-1.1168

X9

0.3635

X7

0.0000

X8

0.0000

X11

0.0000

X12

0.0000

X13

0.0000

X14

0.0000

X15

0.0000

X16

0.0000

X17

0.0000

.2

.4

.6

.8

Individual Effect Cumulative Effect

Relative magnitude and direction of each effect

Figure 6- Example of a Pareto Plot

Prediction Profiles A key feature of the JMP software is its Prediction Profiler. This innovative tool allows the user to see the side-by-side effects of all variables on all responses. In addition, the user may change the values of the input variables, within the range that is valid for the metamodel, and the responses update instantaneously. The tool may also be used for optimization. An example Prediction Profile is shown in Figure 7. The prediction curve shown in Figure 7 is a graphical representation of the response in which the variable of interest is changing while all other variables are held constant.

In this way, the prediction profile can show the sensitivity of the response to

37

Danielle Soban

that input variable. As the slope increases, so does the influence of that variable, and the direction of the slope indicates a positive or negative relationship. As previously noted, the DoE formulation normalizes the inputs so that their low, midpoint, and high values are shown as –1, 0, and 1 respectively. By moving the hairline, the variable value can be changed, and the corresponding value of the response is instantaneously calculated. Altogether, the prediction profile is a useful real-time tool that can be used by the designer to gain insight into the problem and also to seek optimal configurations.

change value of variable by dragging this line along curve 7.4043

prediction curve and error bars

7.042221 5.543

current value of the response current value of the variable

0

0

0

X4

X5

X6

minimum/maximum values of the variable

Figure 7- Example of a Prediction Profile

At this point in the process, the response surface equations are available as metamodels. JMP provides the coefficients of the resulting regression analysis, and these numbers can be exported easily and converted to a form useful to the designer. The next section details a specific methodology that uses RSM at its foundation.

38

Danielle Soban

Technology Impact Forecasting The tools and methods described in the previous sections have been combined into several distinct analysis pathways. One of these pathways is called the Technology Impact Forecasting (TIF) methodology. This particular methodology is discussed here because its basic structure and features were used as an outline for the formulation of the proposed methodology. Although the methodology was originally designed specifically for the incorporation of new technologies onto an aerospace vehicle concept, the basic methodology can be expanded to include concepts that are not specifically technological in nature. TIF is a technique that generates an environment that allows the quantitative exploration of new technology concepts as applied to aerospace systems [27, 28]. Originally a stand-alone methodology, TIF was an evolutionary step in the creation of more

sophisticated

methodologies,

first

culminating

in

the

Technology

Identification/Evaluation/Selection (TIES) method [29, 30, 31], and then both being absorbed in the Concept Feasibility Assessment (COFA) framework [32].

These

subsequent methodologies, however, go into great detail that specifically address all aspects of technology inclusion for a concept, and ensuing concept selection based primarily on that technology. The TIF methodology, on the other hand, concerns itself with the creation of a top level environment that can quickly describe the impact of certain technologies, based on k-factor concepts, Monte Carlo analysis, and technology scenarios. These are the specific features that will be incorporated into the proposed methodology. For the system of systems formulation envisioned, the detailed analysis of

39

Danielle Soban

TIES and COFA are not necessary and the basic tools and concepts of TIF suffice. These tools and concepts are described in the following sections. Overall TIF Environment The TIF methodology is presented in Figure 8. In order to start the TIF process, it is assumed that a baseline aerospace concept is given, and that this concept is realistic but not necessarily feasible (i.e. satisfying all design constraints) or economically viable (i.e. satisfying all economic targets). This baseline should be a representative configuration usually before any candidate advanced technologies are added. It must have the ability to be easily modified, and care should be taken that the concept can still be sized with reasonable accuracy at the extremes of the variable ranges.

Variables and Responses

Technology Scenarios

TIF Environment

Synthesis Environment

Response Surface Equations (RSEs)

R = f(k1, k2, …)

max min max

Monte Carlo

Analysis

min max min -1 1-1 1-1 1-1 1 R1 R2 R3 R4

Baseline Definition

Figure 8- Process to Create TIF Environment and Assess Technology Scenarios

The next step is the same as in basic RSM methodology: define the problem, including appropriate inputs and responses. Then, an appropriate synthesis and sizing

Danielle Soban

40

environment needs to be determined that adequately models and sizes the concept under consideration. Typically this environment will be a computer code that will have as inputs the geometric definition of the concept, aerodynamic, propulsion, and structural (weights) data, and a mission profile. The code must also be capable of resizing the concept subject to a list of performance related constraints (i.e. sizing points). An economics package must be included or linked to provide the necessary economic analysis. Ideally the code will be user-friendly and lend itself to quickly changing inputs and multiple runs. Normally a shell script is created to facilitate the multiple runs and changing variables. From these initial steps, the RSM methodology is applied as described earlier. Before continuing, however, the concept of k-factors must be introduced, as well as their effect on determining the inputs of the problem. Technology Mapping and k-Factors A new technology concept is characterized by ambiguity and uncertainty with regards to its performance, cost, etc. This uncertainty is directly proportional to its development status and the uncertainty is at its greatest in the early phases of the technology’s development. In order to introduce these uncertainties into the model, variability must be added to each input variable. When applied to new technologies, the variability is introduced through the use of technology factors (in other words, a disciplinarian metric multiplier) referred to here as k-factors. A technology is mapped against a vector of k-factors representing primary and secondary effects. For example, Figure 9 shows an example shape distribution for the k-factor associated with wing

41

Danielle Soban

weight. This particular shape distribution would be appropriate for a technology that is expected to give a 7.5% decrease in wing weight, yet recognizes, through the use of a skewed distribution, that there is some chance of achieving either a greater or lesser change in wing weight. Other distribution shapes that may be used include a uniform distribution, used for when each value is as likely as another value, or a normal distribution, which is used when there is an equal uncertainty around a particular value.

k-Wing Weight

-9.00%

-8.25%

-7.50% -6.75%

% Reduction in Wing Weight Figure 9- Example k-factor Notional Shape Function for a Weight Reduction Technology Dial

Once defined, the k-factors become the variables used in the DoE table and subsequent RSE generation.

The responses, in the form of the response surface

equations, are functions of the individual k-factors. Each technology concept, then, becomes a vector of these variables. In essence, a k-factor becomes a “technology dial”. The methodology establishes a direct relationship between the technology dials and the responses of interest. By examining the prediction profile, created by varying the kfactors in the DoE instead of design variables, the designer can identify their sensitivities to the responses. Remembering that the k-factors directly represent parts of technology

42

Danielle Soban

concepts, the designer can clearly identify those factors which have a significant impact on the responses. Later, these sectional technology concepts, represented by their kfactors, can be grouped together (to create a vector) to form “technology scenarios”, with each scenario representing a complete technology concept. In this way, the designer can analyze both the benefits and the risks associated with a technology concept. A final advantage to use the k-factors is that they represent a smooth relationship between the k-factors and the system responses, based on the shape distributions given. This allows the designer to select not only the endpoints of the technology (in other words, having the technology “on” or “off”) but also lets him/her select an intermediate value of a technology improvement and assess its impact on the design. For example, if a k-factor represents a technology that impacts aircraft L/D ratio, the designer could “dial” in a maximum value of, say, 15% improvement and quantify this impact. However, the designer could also explore a 10% improvement, a 5% improvement, or any other value that is contained in the range of the k-factor that was used to produce its RSE. A technology scenario is created as follows. For each candidate technology, identify the key disciplinary metrics (represented by k-factors) that will be affected by that technology and decide by what amounts they will be affected. For example, an advanced aerodynamics scenario might affect, either positively or negatively (representing both benefits and drawbacks), the following variables: overall aircraft drag (an improvement), engine specific fuel consumption (a degradation due to possible increased engine bleed to power, say, a blown wing), and the systems learning curve (increased due to increased system complexity).

Together, this group of variables

Danielle Soban

43

represents one technology scenario. (Realize that each of the variables selected must have been used as a variable in the creation of the RSEs.) This step will be based on data provided by the discipline experts, empirical or historical databases, and the configuration designer’s own knowledge and intuition. Once a technology scenario is created, a Monte Carlo analysis is conducted to assess the impact of that technology scenario. Monte Carlo Simulation After using k-factor inputs and applying the RSM to create the RSEs, the next step is to import the RSEs into a Monte Carlo analysis package in order to conduct the analysis. Crystal Ball [33] is one such statistical package that works in conjunction with Microsoft Excel. Excel spreadsheet templates were created to allow the user to easily import the RSEs in the format they are provided by JMP. A new input file is created for each technology scenario to be explored. A shape function must be assigned to each variable affected by the scenario. These shape functions will determine the probability of achieving certain values of variables. Because the actual shape functions are subjectively selected and can heavily influence the results, it is up to the designer to use their database of knowledge and expertise to ensure the shape distributions are appropriate and reasonable. Variables that are not affected by the technology scenario are set at their most probable, or baseline, values. A Monte Carlo simulation, based on each scenario, is then conducted. The program achieves this by randomly choosing variable values based on the shape distributions given. The responses are then calculated through the use of the

Danielle Soban

44

RSEs. The results are probability distributions that indicate the likeliness of achieving a certain result. The precision of a Monte Carlo analysis is directly proportional to one over the square root of the number of simulations that are run [34]. In order to get a good statistical analysis, it is not unfeasible that the number of runs be on the order of 10,000 cases.

Because the RSEs are fairly straightforward equations, they require little

computational power, and running thousands of cases is a timely and feasible task. (Compare this to running the same number of cases through the synthesis code and the computational beauty of the RSEs becomes apparent.)

Analysis: the Probability Graphs Figure 10 shows examples of the two ways that the probabilistic results from the Monte Carlo simulation can be presented. The first is the probability density function (PDF), which depicts the frequency that a certain value is observed in the simulation. The second is the integral of the PDF, called the cumulative distribution function (CDF), which shows the probability or confidence of achieving a certain value. By examining the CDF in Figure 10, the designer can see that there is about a 10% chance of achieving a takeoff gross weight of 33,475 pounds or less, but a 100% chance of achieving a takeoff gross weight of less than 33,850 pounds (find 33,475 on the horizontal axis, follow it up to where it hits the curve, and read the corresponding probability from the vertical axis). The designer can interpret information from the probability distributions in a number of ways. If the distribution has quite a bit of variability, but some or most of it fulfills the requirement being examined, this would suggest the benefit of investing more

45

Danielle Soban

resources into that technology concept. This addition of resources could have the effect of narrowing the uncertainty associated with the technology. On the other hand, if the distribution indicates that the probability of meeting the requirement is low, then it might be more provident to examine other technology options before investing money into a technology that might not be sufficient to solve the problem. This kind of system-level investigation can also show how much the detrimental effects of the technology are penalizing the system. This information, shared with the disciplinary experts that engage in the development of the technologies, could be investigated to see how resources need to be allocated towards reducing the penalties, as opposed to improving benefits.

Forecast: TOGW 5,000 Trials

Frequency Chart 123 92.2 61.5 30.7 0

33,350.00

33,475.00

33,600.00

33,725.00

33,850.00

33,725.00

33,850.00

Forecast: TOGW 5,000 Trials

Cumulative Chart

1.000 .750 .500 .250 .000 33,350.00

33,475.00

33,600.00

Figure 10- Examples of a Probability Density Function and a Cumulative Probability Function

46

Danielle Soban

At this point the TIF environment is complete and the tool is ready for further analysis and use. If a response represents a design constraint and the CDFs show a low probability of achieving that response, the designer has a few options.

One is to

manipulate the shape functions to give a better probability of success. Realize, however, that this represents a potentially higher level of technology that must be achieved. For example, if an advanced aerodynamics scenario is created and the designer, based on information from his/her discipline experts, can expect a 10% improvement in L/D, but the CDF shows a low probability of success, the designer can rerun the simulation and see how much a 15% improvement will do. If it is determined that there is a high probability of success with the 15% improvement, the designer will need to go back to his/her discipline experts and have them determine whether such levels are realizable or even possible. Other options include redefining or reducing the constraint, or continuing to look at alternative technologies.

Either way, the designer has gained valuable

knowledge concerning the effect of integrating a technology on a vehicle system. The environment that has been created represents a specific class of aircraft with a specific range of variables. The tool, therefore, may be used to conduct studies on similar aircraft without having to recreate a new environment.

Summary of Incorporating Uncertainty and Response Surface Methodology Response Surface Methodology and Technology Impact Forecasting have been applied extensively to aerospace concepts [35]. These tools provide the designer with knowledge crucial to their design decisions, yet much earlier in the design process, where

47

Danielle Soban

these decision have the most economic effect. The statistical and probabilistic nature of these techniques allow for the consideration of uncertainty and risk when considering new technologies or assumptions made on insufficient or unavailable data. The goal of the current research is to use these basic tools and methods to aid in the formulation of an overall methodology at the system of systems level. For example, rather than having a sizing and synthesis environment for a single aerospace concept, the environment becomes the whole theater level simulation code.

Metamodels of the relationships

between system components, such as aerospace vehicles but also diverse components like ground vehicles or communication systems, may be developed using RSM. Screening for inputs of high effect will help scope the problem to a manageable level. And TIF concepts, including k-factors and technology scenarios, can be expanded to account for various uncertainties inherent in the problem (not just limited to the infusion of technologies). There have been endeavors to apply basic RSM at the theater level. Grier et al [36,37] uses factor analysis in an attempt to simplify response surface modeling of the campaign objectives for the campaign code THUNDER. Soban and Mavris [38] applied RSM to the campaign code ITEM to check for method feasibility as well as identify any shortcomings inherent in the application of the methods at the theater level. Results of this study lead to issues and solutions discussed in Chapter IV: Probabilistic Concepts. Finally, it should be realized that to truly understand the complex interactions and model the complete effects of system components, metamodel and probabilistic techniques need to be applied not only at the top level of the system of system

Danielle Soban

48

environment (the theatre level) but analysis needs to be conducted at the sublevels as well. This concept is discussed in greater detail in subsequent sections.

49

Danielle Soban

CHAPTER III

THE NATURE OF MILITARY MODELING

Models and Modeling Chapter I summarized the motivation for the development of a system of systems effectiveness methodology.

The uncertainty inherent in such a method will be

approached by utilizing the probability methods described in Chapter II. These methods, as well as the overall proposed methodology, depend, at their heart, on appropriate and adequate modeling tools. This chapter discusses in detail the nature and characteristics of military modeling.

An understanding of this nature is paramount to the successful

development of a system of systems effectiveness methodology. The world we live in is comprised of many complex systems. The naturally occurring complexities of biological systems, the intricacies of planetary and stellar motion, or the thermodynamic processes of weather are augmented with the results of our technological leaps and bounds of the last century: airplanes and automobiles, intricate economic infrastructures, even the harnessing and transportation of energy that is the backbone of our common household utilities. In our never-ending fascination with the world, we seek to study and analyze these complex systems. Commonly this analysis is conducted through modeling and simulation.

Danielle Soban

50

Modern military strategy is an ideal example of a system that is constantly increasing in complexity. The modern warfighting environment is comprised of vastly disparate sub-systems that must all combine together effectively to achieve a desired objective.

In addition, the rapid development of new technologies adds potential

capabilities that must be explored. Because of this, the use of models in modern military defense strategy continues to play an ever increasing role of importance [39].

Types of Models A model can be defined as a purposeful abstraction of a more complex reality [6, 40].

This reality could take the form of a real or imagined system, an idea, or a

phenomenon or activity. The model tries to capture the essence of this reality in a simplified representation. A military model is specifically defined as “a representation of a military operation and is an abstraction of reality to assist in making defense-related decisions” [6]. Models can be divided into two categories [6, 40, 41, 42]. The first is a physical, or iconic, model. A physical model is usually just that: a physical representation of the object it is representing. For example, a scale model of an aircraft that is used for wind tunnel testing is physical (iconic) model. Likewise, a map, a model car, a blueprint or scale model of a building all represent iconic models. In contrast to a physical model is an abstract, or mathematical model. This kind of model uses symbols and logic to create an abstraction of reality. Mathematical relationships are often utilized to represent the dynamic properties of the object. Examples of an abstract model include an aircraft

Danielle Soban

51

sizing and synthesis code, a biologic model that mimics population growth of bacteria, or an economic model of the stock market. Most military models in use today are abstract. The remainder of this research confines itself to the use of abstract models. Abstract models are further divided into descriptive and prescriptive [6, 41]. A descriptive model limits itself to replicating the behavior of what it is representing. As such, there is no value judgement placed on the behavior; no “goodness” or “badness” is represented. An example of a descriptive model is a combat simulation. Decisions based on information from descriptive models are made by inference, as there is no integral optimization structure inherent in the model. For example, a sizing and synthesis code, which is another example of a descriptive code, may indicate that an aircraft’s gross weight is 35,000 pounds, but the model itself does not specify whether this is an acceptable value or not. In contrast, a prescriptive model specifies a course of action, with an attached value judgement. A prescriptive model (sometimes called normative, which implies a representation of human behavior) may label an output as adequate, inadequate, or optimal. Linear programming, dynamic programming, game theory, and decision theory are all methodologies that indicate to their user an acceptable course of action. It is often difficult to separate a descriptive model from a prescriptive one. Descriptive models are often used prescriptively, to explore options and solutions by trial and error. Prescriptive models are often used for insight only, as some loss of fidelity is usually traded off in order to incorporate optimization schemes. The user of such models needs to understand

52

Danielle Soban

the abilities and limitations of each model in order to utilize them accurately and effectively. This overall categorization of models is shown in Figure 11.

Model Iconic (physical)

Abstract (mathematical)

Prescriptive (normative) Attached Value Judgement linear programming dynamic programming game theory decision theory

Descriptive No Value Judgement replicates behavior no integral optimization

Figure 11 – Model Categorization

Finally, a word needs to be said on simulation. There are those who do not distinguish between simulation and modeling. Caughlin [43] states that a model could be “…a mathematical relationship or a method (algorithm) of expressing that relationship- a simulation. Therefore, we consider a simulation to be a particular representation of a model and will not distinguish between them.”

This parity is echoed by the following

definition “1. A model is a representation of a system 2. A simulation is: a model the exercise of a model a Monte Carlo model …In this paper, model and simulation are used interchangeably” [44]

53

Danielle Soban

Other authors, however, claim a subtle yet distinct difference between modeling and simulation. Bennet [40] defines simulation as “a technique or set of techniques whereby the development of models helps one to understand the behavior of a system, real or hypothetical.”

This relationship is similarly defined by “Simulator: any

compilation system (such as a single processor, a processor network, or more abstractly, an algorithm) capable of executing a model to generate its behavior” [45]. Perhaps the clearest definition of the difference between the two is given by “Simulation is the process of building a model of a system and experimenting with or exercising the model on a computer for a specific purpose such as evaluation, comparison, prediction, sensitivity analysis, optimization, what-if analysis, or training. Simulation cannot be performed without a model.

Every simulation must use a model” [41].

For the

remainder of this study, the above definition and distinction between modeling and simulation will hold: the research will contain models and modeling of the system, and simulation will be conducted using those models in order to assist in the analysis.

Conceptual Models vs. Computer Models An important concept in the modeling and analysis of complex systems is the distinction between a conceptual model and a computer model. As referred to in the first chapter, there is often a lack in differentiation between analysis methodologies and the tools (models) used in those methodologies. Hillestad et al [39] states “Using a campaign model for analysis requires maintaining the proper relationship between the model and the analysis. What seems to have been lost in many of the modeling developments is that

Danielle Soban

54

analysis is a “process” and that the model is but one tool in that process.” Consequently, the first step in the analysis of a complex system should be the formulation of a plan of attack. This plan should clearly identify the goals of the analysis as well as the inputs and outputs necessary for a successful completion of the analysis. This plan, or algorithm, or methodology, becomes the conceptual model of the problem. Zaerpoor and Weber [46] state “it is of utmost importance to have a documented, coherent and justified conceptual model of the experiment before one reaches into the bag of validated computer models and blessed scenarios.” Further, “In making the conceptual model, one starts with the questions of interest and conceives of the leanest physical experiment that would result in satisfactory answers for the questions. The conceptual model is then an abstraction of this experiment.” Computer models, then, become the tools used to achieve the analysis that is prescribed by the conceptual model.

If the conceptual model is in the form of a

methodology, then careful investigation of the problem will identify the appropriate models to utilize within the methodology. It should also be noted that a methodology can be a stand-alone algorithm that represents a certain class of problems. The specific problem, however, determines which models to use. In this way, the methodology is not a specific combination of modeling tools, but rather a framework that allows an interchange of modeling tools as appropriate. This concept is illustrated in Figure 12. The methodology proposed in this research, POSSEM, is designed to be a conceptual model for the analysis of the effectiveness of military system of systems. Although

55

Danielle Soban

specific computer models were used in its development, the methodology is intended to be a framework of analysis, independent of specific computer models.

Conceptual Model...

First step in analysis problem Defines analysis pathway Clearly identifies and states the problem Identifies analysis goals Aids in determining which tools to use

Figure 12- Characteristics of a Conceptual Model

How Models are Used An endless variety of problems that need to be studied necessitate a corresponding endless variety of esoteric models. Thus, it can be said that there are an infinite number of uses for models. However, all of these models do have a unified purpose: they aid in analysis. The purpose of analysis, in turn, is to gain insight into a particular problem. Thus, the ultimate purpose of any model is to aid in gaining insight into the particular problem it represents. Often this insight is translated into decision-making capabilities. As discussed in the first chapter, this research is intended to result in the formulation of a

Danielle Soban

56

framework for the probabilistic assessment of system of system effectiveness in order to facilitate decision-making. With this in mind, the uses of models can be classified into three primary categories. Assessment- called “point estimation” by Zaerpoor et al [46], assessment is using a model to replace a laboratory experiment. This is done when a laboratory experiment is not necessarily economical or feasible. An example of this is an aircraft synthesis code, a less expensive and quicker way to model design changes than actually building and testing an aircraft prototype. Assessment usually uses models that are high fidelity, physics-based computer codes. Most engineering codes are used for assessment. Comparison- this use of models allows for a comparison of alternatives to be explored.

Comparisons usually use models of lower fidelity than those used for

assessment, and concentrate on identifying trends and ratios rather than explicit numbers. Hillestad et al [39] explains, “the model can be used to help identify and analyze the relative importance of various systems, operational concepts, and force structures.” Exploration- finally, exploration examines the relationship between input variables. As stated by Zaerpoor et al [46], exploration “is the method of choice when the number, range, and effect of unknown or unknowable variables dominate the range of outcome.”

This use of a model allows a broader insight into possible future scenarios as

well as the impact of new concepts, such as technologies. Before utilizing a model, it is necessary to understand what type of model it is and how it is primarily used for analysis. Too often, codes are used incorrectly to achieve specific analysis goals. “Is it widely assumed that a blessed scenario plugged into a

Danielle Soban

57

validated combat model will automatically result, not only in credible, but also in useful results” [46]. The distinction between conceptual models and computer models drives home this point: a conceptual model of a problem to be analyzed must be what determines the appropriate models to be used. This research strives to develop one such conceptual model: a methodology.

Transparency Sometimes called “visibility”, transparency in a model is the client’s and analyst’s ability to understand the model’s cause and effect relationships [6]. This understanding is a key factor to a successful analysis. It is the responsibility of the analyst to not only present results but also to explain how the outputs result from the inputs, and how the data and assumptions affect the resulting conclusions [39]. Further, the conclusions should be model-independent; they should follow logically from the inputs and assumptions. It is a circular relationship: “the ability to explain the results depends on transparency.

This in turn depends on both the analytic process and the analyst’s

understanding of the relationships in the campaign model-explanation of the results is only partially a function of the model” [39]. Transparency, thus, is very user specific. The analyst who is very familiar with mathematical relationships and equations that comprise the model may think the model very transparent, while the new user may have difficulty performing an accurate analysis based solely on the inputs and the outputs. The opposite of transparency is opacity, and is a common feature of complicated, complex simulations and computer models. Because the same model can be labeled as

58

Danielle Soban

transparent or opaque, depending on the user, a heavy emphasis on the model’s interface can be used to improve transparency. But when all is said and done, the primary factor in improving model transparency is a clear, precise analysis path. As stated by Hillestad et al [39], “In the end, trust in the model, analyst, and data will provide some comfort that mistakes have been avoided, but it will not substitute for the explanatory transparency developed by the analytic process.”

Classification of Military Models The vast number of existing military models [47, 48] presumes the need to organize them into some sort of classification system. Such a system would aid in the development, acquisition, and use of models, as well as provide guidelines for the choice of models to be used in a particular analysis. The challenge becomes to choose a classification system that is intuitive and useful. To start, the difference between a taxonomy and a catalogue needs to be clarified. A taxonomy is a classification system that divides things into ordered groups or categories. A catalogue, on the other hand, is a listing or itemized display that usually contains descriptive information.

A taxonomy, thus, can form an indispensable

foundation for a catalogue [49], yet the taxonomy itself may not provide essential information such as that found in the catalogue. For example, the taxonomy used to classify a rosebush contains no information as to whether that rose is pleasing to look at or has a nice fragrance. A useful classification system must thus be more than just a taxonomy. The goal of this section is to discuss some current classification systems in

59

Danielle Soban

use today. This will be done in order to understand how models relate to each other, and thus aid in formulating the foundation of the overall conceptual model being developed.

SIMTAX Responding to the Department of Defense (DoD) need for a wargaming/warfare simulation descriptive framework, the Military Operations Research Society (MORS) hosted a series of workshops that resulted in the development of SIMTAX-a taxonomy for warfare simulation [44].

This taxonomy was designed to aid the DoD in the

development, acquisition, and use of warfare models. Specific features of the taxonomy were designed to accomplish the following goals: classify warfare simulations, construct frameworks for comparing conflict models, and provide the foundation for a comprehensive wargames catalogue. The SIMTAX workshop concluded that warfare simulations needed to be classified according to three characteristics of equal importance and occupying the same relational level (in other words, they are not hierarchically related).

These three

characteristics are the purpose, the qualities, and the construction of the model. Figure 13 shows the traits and sub-categories associated with each characteristic. The purpose of the model describes why the model was built or to what purpose the model could be applied. Although it was conceded that a particular model may be used for a variety of purposes, it was agreed that a singular purpose could be determined that described the basic function of the model. The qualities of the model are those aspects that describe the model itself, such as its level of detail, its scope or its domain. Finally, the construction

60

Danielle Soban

of the model describes the design of the model, such as whether or not human participation is needed or how randomness is treated.

Warfare Simulation

Purpose

Qualities

Analysis

Domain Span Environment Force Composition Scope of Conflict Mission Area Level of Detail Processes Entities

Research & Evaluation Operation Support (decision aids) Training and Education Skills Development Exercise Driver

Construction Human Participation Required Not Required Time Processing Static Dynamic Treatment of Randomness Stochastic Deterministic Sidedness One-sided Two-sided Three or more-sided

Figure 13- SIMTAX Categorization of Warfare Simulation

SIMTAX is a useful way to classify existing models. It allows models to be grouped by common characteristics, thus aiding side-by-side comparisons of model capabilities.

It should be emphasized that SIMTAX was developed for warfare

simulations, and thus is not necessarily appropriate for the classification of all types of military models, for example, engineering models.

61

Danielle Soban

Other Taxonomies While SIMTAX provides an exhaustive framework for the classification of warfare simulations, Hughes [6] provides a sampling of other characteristics under which military models in general may be classified. Models by Application or Analytical Purpose- similar to the SIMTAX characteristic, Hughes’ monograph lists this as the primary categorization of military models. Models are divided into seven major categories of application: battle planning, wartime operations, weapon procurement, force sizing, human resources planning, logistic planning, and national policy analysis. In addition, two other categories may be extracted from those above and separated into distinct entities: command, control, communications, and intelligence, and cost models. Models by Technique or Level of Abstraction- military models can be categorized into one of the following four areas: analytical representations, computer simulations, war games, and field experiments. Models to Describe, Prescribe, and Predict- this category describes the descriptive, predictive, and normative traits as discussed earlier in this chapter. Ad Hoc and Standing Models- a standing model is similar to a legacy code. It implies a model that is continuously supported, updated, and improved. In contrast, ad hoc models are built with a specific decision in mind. Scientific and Sensible Models-

characteristics that classify something as

“scientific” include openness, explicitness, objectivity, the use of empirical data, quantification, and a self-correcting character [50]. A scientific model, therefore, should

62

Danielle Soban

have these inherent traits.

Military modeling, however, can often be described as

“sensible” rather than “scientific”. A sensible model is one in which the parameters and variables are set down, the equations (processes) are open to inspection, and the same inputs can be shown to produce the same or similar responses [6]. However, there are inherent assumptions and value judgements that contributed to the logic of the model. Analysis involving sensible models focuses more on the inherent logic of the model and the choice of inputs. The results can not be validated in the scientific sense, but if the model is accepted and its limitations recognized, the results that follow can be useful in aiding the decision-making process. Models by Scope and Scale- this final category separates models into three or four levels of aggregation. Rather than discuss only characteristics of each model, this sort of classification starts to take into account how the models relate to each other. The next few sections will discuss this idea of hierarchical modeling in more depth. It should be noted that Hughes’ categories cannot be used concurrently as a single taxonomy. The categories as stated are not mutually exclusive. However, each on their own provides a useful key to understanding the capabilities and nuances of individual models.

Hierarchical Modeling A well known way of relating military models to each other is to classify them as to their position in a defined hierarchy. This hierarchy is often portrayed as having a pyramidal shape (Figure 14) and is described by Hughes [6]: “Whatever the number of

63

Danielle Soban

echelons that is included in it, the bottom will contain mainly phenomenological models and the top a single macro model. The pyramid may represent the nesting of models in one interacting, organic whole; show how results are integrated from echelon to echelon; or merely be an abstract representation of a structure that relates independent model operations.” Typically, a hierarchical structure will be divided up into three or four aggregate levels. Ziegler [51] first introduced the idea of decomposing models in a hierarchical manner that corresponded to levels of detail. Later, he described this decomposition as having as its first level the most abstract behavioral description of the system. This is followed by sub-systems levels of increasing detail, until a limit is reached and further decomposition is not justified [52].

This type of decomposition is echoed in the

hierarchical structures of military models described today: the first level usually contains an encompassing theater or campaign level model, followed by engagement or mission models, with the lower levels being reserved for engineering sub-system models. Figure 15 shows a traditional pyramidal structure containing three levels. Figure 15 (created from data in Reference [56]) and Figure 16 (Reference [53]) portray two alternative decompositions. Each of these categorical levels will be discussed in more detail.

64

Danielle Soban

Theater or Campaign Model Mission Model Single or Multiple Engagement Model

Engineering Model

Figure 14- Traditional Pyramid of Military Models

Level I: System/Engineering Analysis deals with individual systems or components

Level II: Platform Effects Analysis component is associated with a platform

Level III: Mission Effectiveness Analysis

assesses contribution of platform to a combat mission environment, including Command and Control, time-sensitive maneuvers, and a defined enemy posture

Level IV: Force Effectiveness Analysis all activity associated with operations of joint campaigns against an enemy combined arms force

Figure 15- Alternate Hierarchical Structure of Military Models I

65

Danielle Soban

Decision-making Focus

Applicable Levels of M&S

Strategic Issues

Strategic

Operational Issues

Campaign

Tactical Issues System Design Issues

Mission Engagement Engineering

Figure 16- Alternate Hierarchical Structure of Military Models II

Decomposition Levels The examples of hierarchical levels given in the previous section illustrate an interesting point: although each example chooses its own delineation for its levels, and names them correspondingly, each individual hierarchy does encompass the entire spectrum of relational military models and analysis. In order to discuss and define these types of models, they will be divided into roughly three categories: engineering models, mission models, and campaign models. These model categories will now be discussed in more detail, with the understanding that they will overlap in definition as they pertain to individual hierarchies.

Engineering Models The level that comprises the bottom of all hierarchies is that of engineering models. These are usually detailed mathematical representations of individual systems or components that may or may not be associated with a platform. For example, a hierarchy may consider a jammer or a sensor to be an engineering model, yet another hierarchy

Danielle Soban

66

may lump them together onto a platform and have one model of that integrated system. Engineering models are usually physics-based, contain a high level of detail, and are of relatively short simulation timeframe. Inputs are usually design variables, sizing criteria, and new technologies. Basic mission requirements and constraints may be introduced at this level to aid in sizing. For example, an aircraft sizing and synthesis code typically needs a rudimentary mission profile to be input for sizing purposes. (An interesting investigation of the influence of mission requirements on the vehicle being sized can be found in Reference [54].) Outputs of engineering models often consist of geometric dimensions and performance data. Engineering codes are generally used to conduct tradeoff studies of design variables and technologies, and to calculate performance characteristics. These types of codes can usually be validated and they lend themselves well to real-world testing. However, analysis conducted using this type of model is limited to the scope of the model. As discussed in the first chapter, an airplane sizing and synthesis model can provide data as to the performance of a particular aircraft, yet does not provide information as to how well that aircraft will aid in reaching system (theater or campaign) level goals. Figure 17, from Reference [53], shows the engineering level features.

67

Danielle Soban

Engineering/Engagement View of the World Characteristics: -Short simulation timeframe -Narrow scope -High detail Detailed systems and subsystems Some physics-based interactions Strengths: -Amenable to real-world testing -Generally able to be validated -Data collection centralized (1 SPO) Limitations: -Limited scope / context Only answers question in a small arena

Figure 17- Engineering Level Model Features [53]

Mission Models The middle level of the traditional pyramid is occupied by mission models. These models, also referred to as engagement models, encapsulate one vs. one or many vs. many encounters of specific sub-system components. For example, a mission level code could be used to simulate air-to-air combat between multiple flights of aircraft. The focus on this level concerns the timing and details of a single mission. Scenario, strategy, support, and overall force capabilities are usually not considered [55]. The timeframe of mission models is usually at the hours or days level, and they involve a medium level of detail. The level of interaction is increased, and the model will usually contain aggregate systems and subsystems.

Unlike the engineering models, mission models are more

68

Danielle Soban

troublesome to validate, and data collection for input is often difficult. Figure 18 shows a representation of the mission level from Reference [53].

Mission-Level View of the World Characteristics: -Medium simulation timeframe (Hrs - days) -Up to theater in scope -Medium detail Entity-Based: Simulation entities map to real world entities Many interactions effects-based Systems & Subsystems aggregated Strengths: Scope: Variety of interactions Theater sized scenarios Limitations: -Not comparable to real world testing standards -Not easily validated -Data collection dispersed

Figure 18- Mission Level Model Features [53]

Campaign Models Campaign models, also called force models or theater models, are usually large single codes that encompass the effects of the total forces involved, including air, ground, and naval, as well as coalition forces [39, 56]. They are primarily used to answer questions and make decisions at the larger system level. For example, campaign analysis, aided by campaign models, is used to study the interactions of strategy, force allocation, and system capabilities.

Other features that dominate analysis are the effects of

69

Danielle Soban

command and control decisions, deployment and sustainment (logistics) issues, and the cumulative effects of decisions as considered in a time-spanning environment. There are many campaign (or theater) level models in use today. It is common that certain organizations favor specific codes for their analysis needs, and these codes often emphasize, in terms of modeling detail and capability, a particular force. Some of the more well-used codes and the organizations that are their primary users are listed in Figure 19.

USN GCAM ITEM

USA TAC WAR VIC

USMC COMBAT IV

USAF THUNDER

OSD TAC WAR ITEM

Figure 19- Common Campaign Codes in Use Today

There are inherent limitations to campaign models. The first is that an inordinate amount of experience and information is needed by the user to use them effectively. This echoes a key point made earlier in this chapter: transparency of the model is user dependent, and the quality of the analysis is often directly related to the experience of the analyst. In addition, considerable detail is needed in both the scenario definitions and in the component descriptions in order to have a complete analysis. Yet at the same time, the complexity and run time of the code necessitates that detail be kept at a minimum. One issue in the use of campaign codes that must be noted is the ability of the user to influence the outcome.

Campaign codes are often based on empirical

relationships and experience mapped into coded equations, as opposed to direct physics-

70

Danielle Soban

based mathematics. It is difficult to model such complexities as human judgement and error. As noted previously, the campaign level codes need to be used by an analyst with a great deal of insight, skill, and experience with the problem being modeled. The input assumptions crucially drive the results. As such, the analyst has the capability, either wittingly or unwittingly, of unduly influencing the results towards a particular outcome. Another problem is the length of the campaign to be modeled. As the campaign time increases, difficulty in retaining fidelity of the model also increases. A key feature of campaign time is that as the campaign progresses, tactics and decisions evolve based on experiences and results so far [6]. This “human in the loop” problem is considered in more detail in subsequent sections. The campaign level is depicted by Reference [53] and is shown in Figure 20.

Campaign View of the World Characteristics: -Long simulation timeframe (months)

-Up to global in scope -Low detail Interactions effects-based Units & entities aggregated Strengths: -Scope: Wide variety of interactions over time -Full campaign Limitations: -Far from real world testing standards -Difficult to validate -Data collection widely dispersed (multi-service) -Reliance on abstraction

Figure 20- Campaign Level Model Features [53]

Danielle Soban

71

The Military Code Continuum Given the features of each type of military model, the traditional pyramid formulation can be replaced and enhanced by a new concept, similar to that in Reference [53], of a military code continuum. Figure 21 shows this continuum, and illustrates the primary two analysis tradeoffs of the continuum: detail versus complexity. As analysis moves from the engineering end of the spectrum to the campaign analysis end, the modeling codes increase dramatically in complexity yet lose an enormous amount of detail. The sheer number of entities that need to be modeled at the campaign level, coupled with an increasing number of decisions and interactions, soon lead to a modeling problem that is so complex that it becomes impractical to model the inputs with any level of detail. However, this necessary sacrifice of detail does come with a price. At the engineering level, where significant details is captured, the resulting metrics are very specific. Questions can be answered precisely. As the analysis moves towards the campaign level, the metrics become increasingly amorphous, with results that are more subjective and provide insight rather than explicit answers. Before proceeding, it is important to understand the differences between detail and complexity. Level of detail, or resolution, relates to how well and to what depths the object under consideration is being modeled [57]. It is a measure of the degree to which the model has captured the essence of the object. Complexity, on the other hand, is a computational feature of the model itself, and is a measure of the intricacy of the model.

Danielle Soban

72

How these two concepts relate to model accuracy is likewise important. The following intuitive assertions are repeated from Reference [57]: - Increased level of detail does not necessarily imply increased accuracy - Increased complexity does not necessarily imply increased accuracy - Increased level of detail usually does imply increased complexity - Complexity is directly related to computer runtime. As shown in Figure 21, complexity is represented by several modeling features. First is validation. When considering a highly detailed, physics-based model, validation becomes rather straightforward. Results can be compared and correlated to existing systems, and the equations verified. As the model moves up the continuum, however, the modeling becomes less physics based and more effects based. The results of campaign analysis provide more insight than deterministic answers, making validation very difficult. Indeed, these subjective answers and insight are the next area of increasing complexity. Metrics based on the answers resulting from the engineering end of the continuum are specific and easy to understand and use. At the campaign end of the spectrum, however, the metrics can become vague, leading to insight but rarely concrete answers. In addition, these metrics are usually the result of a multitude of effects, making isolation of primary effects difficult. Next is the skill of the user/analyst. While it is good practice to ensure that the user of any type of modeling code has a good idea of what and how is being modeled, it can still be said that an engineering code is more straightforward and implies “less” skill to use. If the user/analyst of an engineering code trusts the basic physics of the code,

73

Danielle Soban

then the use and analysis of that code is less complicated. On the other hand, there is a high level of experience needed to use and analyze a campaign code successfully. The lack of deterministic relationships in the model, coupled with vast number of entities and concepts modeled, necessitates a user that is thoroughly familiar with campaign concepts and can make appropriate assumptions. The next three features can be discussed together.

Moving up through the

continuum from the engineering level to the campaign level, the number of entities that need to be modeled increases, as well as the number of interactions and the numbers of decisions that need to be made. This in itself tremendously increases the complexity of the campaign end of the spectrum. Finally, we discuss timeframe. An engineering code is usually considered to be independent of a timeframe. Moving through the continuum, entities are grouped together and pitted against each other. Once a scenario becomes involved, time becomes a factor. Mission level codes may have a timeframe of minutes to hours. A typical campaign can be modeled at hours, days, or even weeks. The addition of a time element increases the complexity of the model. The level of detail at the engineering end of the continuum is very high. This is because only a single entity is being modeled, and so must be modeled in sufficient detail. As the continuum moves to the right, however, the sheer number of entities that must be modeled, as well as their interactions, necessitates that each of these entities cannot be modeled in the same level of engineering detail. This would cause huge costs both in terms of the setup and analysis of the code, as well as the computational runtime.

74

Danielle Soban

The capture of the physical process can be considered high at the engineering end, although this may or may not indicate a greater model accuracy.

COMPLEXITY Validation Metrics Necessary Skill of Analyst Number of Entities

Easy

Difficult

Specific/Answers

Subjective/Insight

Low

High

Few

Many

Number of Decisions

None

Number of Interactions

Few

Timeframe

Many (unmanageable)

Few

Many Hours

None

Weeks

Days

DETAIL Capture of Physical Process Level of Detail

High

Low

High

Low/None Squadron

Receiver

Radar

Aircraft

Flight

Group

Wing

Air Force

Command

RCVR

Component

Sub-system

Engineering

Platform

Unit

Mission

Force

Campaign

Figure 21 – The Military Code Continuum

Summary of Military Modeling The purpose of this chapter was to provide an overview of military models, what they are, and how they are related to each other. This understanding is necessary given the goal of this research: to develop a methodology to assess system of systems effectiveness within the military modeling environment.

Danielle Soban

75

The traditional pyramid of models was discussed, as well as its replacement with the military code continuum. It is this continuum and its features that have the biggest effect on the development of the methodology.

The next chapter discusses the

application of existing probabilistic methods directly to a campaign level code. In doing so, several key issues are identified, and the solutions to these issues make up the backbone of the proposed methodology. This backbone relies heavily on the concept of the military code continuum, and the relationships and characteristics of the types of models that make up the continuum.

76

Danielle Soban

CHAPTER IV

PRELIMINARY INVESTIGATION: APPLYING CURRENT METHODS TO CAMPAIGN LEVEL The first three chapters provided the background for the ensuing research. The next step was to begin the formulation of the methodology. The first investigative question that needed an answer was: can the existing statistical methods that were developed for the engineering level be applied directly the campaign level?

A

preliminary investigation of this question was conducted in order to identify any specific issues of concern that might arise from the direct extrapolation of the methods from the engineering level to the campaign level. Once these issues were identified, solutions to these issues could be proposed. Based on the incorporation of these solutions, an overall new methodology would be developed. The results of this investigation did indeed identify three major issues of concern that had a direct impact on the formulation of the system of systems effectiveness methodology: model abstraction vs. integration, the human in the loop dilemma, and scenario significance. The initial investigation will be discussed in this chapter, and the resulting issues and their proposed solution will be contained in the next chapter.

Campaign Code Selection In order to conduct the preliminary investigation, a suitable campaign level code was needed. The selected code would need to have input and output variables that were

77

Danielle Soban

relative, provided insight, and were easily manipulated. In addition, in order to be compatible with the existing methodology, the code needed to have the ability to run multiple cases quickly and efficiently.

ITEM The code selected for the study was ITEM (Integrated Theater Engagement Model) developed by SAIC, Inc [58] and is described more fully in Chapter VII. ITEM is an interactive, animated computer simulation of military operations in theater-level campaigns. It has fully integrated air, land, and naval (including amphibious) warfare modules and contains a strong emphasis on visualization. The inputs and output are GUI-driven (Graphic User Interface): an example of this interface is shown in Figure 22. ITEM is fully object-oriented in design and execution and contains a hierarchical structure of its database. Master Inter face Process Graphical User Interface

WARVIEW Graphical User Interface

Figure 22- The ITEM Graphical Interface and Environment

78

Danielle Soban

ITEM was inserted as the modeling code into the current probabilistic methods. A representation of this method is shown in Figure 23.

Integrated Theater Item Model (ITEM) Input Parameters: Effectiveness: Blue Weapons Salvo Size: SAM Sites PH of Red Weapons PK Red Weapon 5 vs. Blue Weapon 4 etc

Response Surface Methodology

\ \

\ \

\

\

\

\

\

\ C2 \ C3

\

\

Prediction Profile

\

\

Pareto Chart

+ Monte Carlo =

X1 C4

Dynamic Contours

Probability

\

\

\

\

C1

\

\

X2

\

\

\

X1 X2 X3 X4 ...

\

\

\

X1

Measures of Effectiveness: Total Damage Red SAM Sites Total Damage Red Airbases Total No. Blue Aircraft Destroyed etc

Metric

Probability Distribution

Figure 23 – Applying Current Methodology Using ITEM

Case I: Theater Level Case The first case to be investigated was a general theater level case described in the scenario below. This was the first attempt to directly apply the current probabilistic methods to a campaign level code. Inputs and outputs are described below, as well as initial results.

79

Danielle Soban

Scenario for Case I The scenario, operational situation, and inputs and outputs were provided by Johns Hopkins Applied Physics Laboratory (JHAPL). The test scenario chosen was a fictional conflict in which South Florida attacks North Florida with missiles and aircraft, and the scenario and variable ranges were selected by JHAPL in order to facilitate an unclassified study. An Air Superiority Operational Situation was constructed, and is summarized in Figure 24. Two Blue aircraft were modeled, and are called Strike 1 and Strike 2. The variables that are available to model aircraft are shown in Figure 25 and the difference between the two aircraft specified.

It is important to note how few

variables in ITEM are used to represent an aircraft, and the disconnect between these and traditional design variables, such as those used in an aircraft synthesis code, becomes obvious. The Red Surface to Air Missile (SAM) sites are shown in Figure 26, with their main goal to protect the Red airbase.

Day 1 Ship-launched cruise missile strikes on four SAM sites

Red Airbase protected by SAM Sites

Objective: Increase survivability of two aircraft strikes on Day 2

Day 2 Two aircraft strikes from Blue airbase on red airbase protected by the four SAM sites.

Ships/ Task Groups

Blue Airbases

Objective: Render airbase inoperational

Figure 24 - Florida Scenario with Air Superiority Operations Situation

80

Danielle Soban

Strike 2

Strike 1 Mission Duration

2.5 hrs

Mission Duration

Turnaround Time

5 hrs

Turnaround Time

Max. Speed

1600 kts

Max. Range Relative Detectability

4000 nm

Maneuverability Degradation

Variable

0.4

Shelterable

3 hrs 6 hrs 1500 kts

Max. Speed Max. Range

4500 nm

Relative Detectability

0.25

Maneuverability Degradation

Variable

Shelterable

Maximum Altitude

25000 ft

Standard Conventional Load

L1, L2

Maximum Altitude

25000 ft

Standard Conventional Load

L1, L2

Figure 25 – Blue Aircraft Modeled for Scenario

Radius of SAM Site SAM 3 SAM 1

Airbase

Communications Range of SAM Site

SAM 2

Figure 26 – Red SAM Site Detail

SAM 4

81

Danielle Soban

Inputs and Outputs The inputs to the scenario as provided by JHAPL are shown in Table 3. “PK” is the incremental probability of kill and “EK” represents the expected number of targets destroyed per hit.

The minimum and maximum values bound the ranges for the

variables, with the average of the two values being used as the midpoint, or baseline, value. The output Measures of Effectiveness are given in Table 4. Note that the number (percentage) of aircraft destroyed is the same as 1- (number of aircraft survived).

Table 3 - Inputs for Theater Level Scenario Minimum Maximum PK of Cruise Missile versus SAM Fire Control System

0.65

0.95

Maneuverability Degradation of Strike 1 Aircraft

20%

60%

Maneuverability Degradation of Strike 2 Aircraft

40%

80%

EK of Strike 1 AC / SCL 1 Loadout versus Airbase Runway

1

2

EK of Strike 1 AC / SCL 2 Loadout versus Aircraft in Open

2

4

EK of Strike 2 AC / SCL 2 Loadout versus Aircraft in Open

1.5

2.5

PK of SAM-1 Missile versus Strike 1 Aircraft

0.1

0.3

PK of SAM-1 Missile versus Strike 2 Aircraft

0.1

0.4

PK of SAM-2 Missile versus Strike 1 Aircraft

0.1

0.2

PK of SAM-2 Missile versus Strike 2 Aircraft

0.1

0.3

Reliability (i.e. P(hit)) of SAM-1 Missile

0.55

0.85

Reliability (i.e. P(hit)) of SAM-2 Missile

0.55

0.85

82

Danielle Soban

Table 4 - Outputs from Theater Scenario

Measures of Effectiveness Expected Number of Strike 1 Aircraft that are Destroyed Expected Number of Strike 1 Aircraft that Survive Expected Number of Strike 2 Aircraft that are Destroyed Expected Number of Strike 2 Aircraft that Survive Expected Number of SAM-2 Missiles Launched Expected Number of Runways Destroyed by Aircraft Strikes Expected Number of Aircraft in Open Destroyed by Aircraft Strikes

Results from Case I Preliminary results are shown in the form of a 2-level DoE screening test. This is a first order fit between the input variables and the output variables, and a total of 129 cases were run. Pareto plots showing the magnitude of the contributions of the different input variables to the variability of the response is shown in Figure 27. For the expected number of aircraft destroyed for both Strike 1 and 2, it can be seen that the Pk of the cruise missile against the SAM fire control system is the highest variable contributor. This confirms the Day 1 objective: increasing the Pk value of the cruise missiles does indeed increase the survivability of the strike aircraft. (It is important to remember that the Pareto plot shows the greatest contributors to the variability of the response. In Figure 27, the Pk of the cruise missile is shown to contribute the highest amount to the expected number of aircraft destroyed. This is not to say that an increase in the Pk value increases the number of aircraft destroyed. In fact, the opposite is true, as shown in

Danielle Soban

83

Figure 28.) Likewise, the Pk of the cruise missile has an overwhelming effect on the number of SAM2’s launched against the strike aircraft. SAM1 was represented in the model, but because of the geometry involved, was in effect a non-player. The variable was kept in as a sort of sanity check; if SAM1 was to show an effect, the model would have been unrealistic, identifying an error somewhere in the model. Similarly, some of the results are obvious: the Pk of Strike 2 aircraft against aircraft in the open does indeed have the highest influence on the expected number of aircraft in the open destroyed. Again, this shows the model and system are behaving as expected. Figure 28 is the prediction profile for the screening test. While the Pareto plot identifies the chief contributors to the response, the prediction profile identifies magnitude and direction of the impact of the input variables, as well as shows the simultaneous impact of the other variables. As noted in the Pareto plots, the prediction profile shows that increasing the Pk of the cruise missile does indeed increase the survivability of both strike aircraft. In addition, the number of SAM2’s launched is decreased. There is little change in the number of airbase runways destroyed or the number of aircraft in the open destroyed, which makes intuitive sense. A correlation is also seen between the degradation of the maneuverability of the strike aircraft and the number of these aircraft destroyed. Other results are checked for intuitive correctness.

84

Danielle Soban

.2

Term

.4

.6

.8

.2

Term

Pk of Cruise Missile vs. SAM Fire Con.

Pk of Cruise Missile vs. SAM Fire Con.

Maneuv. Degradation of Strike 1 A/C

Pk of SAM2 vs Strike 2 A/C

Pk of SAM2 vs. Strike 1 A/C

Maneuv. Degradation of Strike 2 A/C

Ph of SAM2

Ph of SAM2

Pk of SAM1 vs. Strike 2 A/C

Pk of SAM1 vs. Strike 1 A/C

Pk of Strike 1 A/C vs. Aircraft in Open

Pk of SAM2 vs. Strike 1 A/C

Pk of Strike 2 A/C vs. Aircraft in Open

Maneuv. Degradation of Strike 1 A/C

Strike 1 A/C vs. Runways

Ph of SAM1

Pk of SAM1 vs. Strike 1 A/C

Strike 1 A/C vs. Runways

Pk of SAM2 vs Strike 2 A/C

Pk of Strike 2 A/C vs. Aircraft in Open

Ph of SAM1

Pk of SAM1 vs. Strike 2 A/C

Maneuv. Degradation of Strike 2 A/C

Pk of Strike 1 A/C vs. Aircraft in Open

Expected Number of Strike 1 A/C Destroyed

.2

Term

.4

.6

.6

.8

Expected Number of Strike 2 A/C Destroyed .2

Term

.8

Pk of Cruise Missile vs. SAM Fire Con.

Pk of Strike 2 A/C vs. Aircraft in Open

Maneuv. Degradation of Strike 2 A/C

Pk of Strike 1 A/C vs. Aircraft in Open

Ph of SAM1

Ph of SAM2

Strike 1 A/C vs. Runways

Ph of SAM1

Pk of SAM2 vs. Strike 1 A/C

Pk of SAM1 vs. Strike 1 A/C

Pk of SAM1 vs. Strike 1 A/C

Pk of SAM2 vs Strike 2 A/C

Pk of SAM2 vs Strike 2 A/C

Maneuv. Degradation of Strike 2 A/C

Maneuv. Degradation of Strike 1 A/C

Strike 1 A/C vs. Runways

Pk of Strike 2 A/C vs. Aircraft in Open

Pk of SAM1 vs. Strike 2 A/C

Pk of SAM1 vs. Strike 2 A/C

Pk of SAM2 vs. Strike 1 A/C

Pk of Strike 1 A/C vs. Aircraft in Open

Maneuv. Degradation of Strike 1 A/C

Ph of SAM2

Pk of Cruise Missile vs. SAM Fire Con.

Expected Number of SAM2 Launched

.4

.6

.8

Expected Number of Aircraft in the Open Destroyed .2

Term

.4

.4

.6

.8

Strike 1 A/C vs. Runways Pk of Cruise Missile vs. SAM Fire Con. Maneuv. Degradation of Strike 1 A/C Pk of SAM2 vs. Strike 1 A/C Pk of SAM1 vs. Strike 2 A/C Pk of Strike 2 A/C vs. Aircraft in Open Pk of SAM1 vs. Strike 1 A/C Pk of Strike 1 A/C vs. Aircraft in Open Maneuv. Degradation of Strike 2 A/C Pk of SAM2 vs Strike 2 A/C Ph of SAM1 Ph of SAM2

Expected Number of Runways Destroyed

Figure 27 - Pareto Plots for Theater Scenario Screening Test

85

0.218 0.05

3.996

0.234

Strike 2 Survived

Strike 1 Survived

0.004

Strike 2 Destroyed

Strike 1 Destroyed

Danielle Soban

3.94 3.782

0.05 0.004 1.996 1.94 1.766

A/C Runways SAM2 in Open Destroyed Missiles Destroyed by Launched by A/C Strikes A/C Strikes

3.281 1.92

0.575 4 2.98 1.96 8 6.75 4.96 .65

.95 20%

60% 40%

Pk of Maneuv. Cruise Missile Degradation vs. of SAM Strike 1 Fire Control System

80% 1.0

Maneuv. Degradation of Strike 2

2.0 2.0

Ek of Strike 1 A/C w/ SCL-1 vs. Runway

4.0 1.5

Ek of Strike 1 A/C w/ SCL-2 vs. A/C in Open

2.5 0.1

Ek of Strike 2 A/C w/ SCL-2 vs. A/C in Open

0.3

Pk of SAM1 Missile vs. Strike 1

0.1

0.4

Pk of SAM1 Missile vs. Strike 2

0.1

0.2 0.1

Pk of SAM2 Missile vs. Strike 1

0.3 0.55

Pk of SAM2 Missile vs. Strike 2

0.85 0.55

Phit of SAM1 Missile

0.85

Phit of SAM2 Missile

Figure 28 - Prediction Profile for Theater Scenario Screening Test

Case II: Survivability Test Case For the second test case, a survivability assessment was chosen. This was in anticipation of the final application selected to demonstrate the proposed methodology: apply survivability enhancement technologies at the engineering (vehicle) level, yet assess their effectiveness at the campaign level.

Scenario for Case II Because aircraft were selected as the baselines, an unclassified notional air superiority scenario was chosen. The scenario needed to be simple enough to be able to clearly identify results of tradeoffs (especially since this first case is a proof of concept)

86

Danielle Soban

yet complex enough to capture important interactions between variables.

An

extrapolation of the basic scenario used in the above section was chosen. This scenario models a situation in which South Florida attacks North Florida and is shown in Figure 29. Two airbases are established in South Florida and the notional aircraft stationed there. The target is an airbase in North Florida that is protected by four SAM sites. The SAM sites are distributed around the airbase.

Each sites has identical

features, including the same number of launch rails, reload time, antenna height, and engagement radius. However, there are two different kinds of weapons defined that differ in terms of range, speed, and altitude (Figure 30). It must be noted that some of the values, such as SAM speed, can be considered unrealistic. These numbers were provided as proof of concept numbers in order to maintain the unclassified nature of the study. The results, therefore, should be considered quantifications of trends rather than absolute figures. Two of the airbases have one type of weapon and the other two have the second type.

87

Danielle Soban

Ships Blue Airbases

Figure 29 - Florida Scenario used in Theater Level Survivability Study

no factor no factor

Weapon Min Range (nm) Max Range (nm) Speed (kts) Min Alt (ft) Max Alt (ft) Rel. Detectability Ph Engagement Time (min)

RSAM-1 2 20 600 2000 18000 0.5 0.9 1.5

RSAM-2 1 15 800 1000 14000 0.5 0.9 0.5

Figure 30 - SAM Site Weapon Comparison for Theater Level Survivability Study

88

Danielle Soban

Features of the South Florida (Blue) airbases were not crucial to the study. The two notional aircraft were assigned to each Blue airbase and a surplus of weapons was available. The North Florida (Red) airbase featured runways, shelters, revetments, and aircraft in the open. These aircraft did not mobilize to defend the airbase, but were rather targets. The only defense of the Red airbase came from the surrounding SAM sites. The air campaign consisted of hourly attacks on the Red airbase for a total of 8 hours. Blue airbases alternated attacks, so each airbase launched a total of four strikes, consisting of five aircraft per strike mission. Each aircraft was assigned one of five targets: a SAM fire control, a SAM radar, a runway, a shelter, or aircraft in the open. Flight paths were straight line from airbase to airbase. SAM sites and Red airbase features were allowed a variable repair rate. There was no repair rate for Blue aircraft. SAM sites defended automatically when detecting incoming aircraft.

Inputs and Outputs Input choices were limited to those available in the synthesis code.

Those

variables most closely modeling susceptibility/survivability concepts were chosen. For aircraft, the variables selected were detectability, maneuverability, and turnaround time. The detectability variable models the relative detectability of the aircraft, and is a scaling factor used to reduce the probability of detection of opposing forces against the aircraft. It is directly analogous to radar cross section (RCS), and will be discussed in more detail later in this chapter. Likewise, the maneuverability variable measures the aircraft’s

89

Danielle Soban

ability to evade defensive systems and is a scaling factor used to reduce the probability of hit of engaging weapons against the aircraft [58]. Several threat variables were chosen. The engagement range of SAM sites was allowed to vary, but all four sites varied at the same time. Additionally, repair rates for the SAM fire control and radar systems, as well as airbase runways, shelters and aircraft in the open were selected as variables of interest. Table 5 summarizes the variables and their ranges.

Table 5 - Input Variables for Theater Level Survivability Study

Variables Detectability Strike 1 Detectability Strike 2 Maneuverability Strike 1 Maneuverability Strike 2 Turnaround Time Strike 1 Turnaround Time Strike 2 Track Range SAM Site Repair Rate SAM Fire Control Repair Rate SAM Radar Repair Rate Runways Repair Rate Shelters Repair Rate Aircraft in the Open

Low

Baseline

High

0.7

0.5

0.3

1

0.8

0.6

0.6 0.9

0.4 0.7

0.2 0.5

0.25

0.75

0.75 10 0.1

0.5 1 20 0.2

0.1

0.2

0.3

0.1 0.1

0.2 0.2

0.3 0.3

0.1

0.2

0.3

1.25 30 0.3

Appropriate output Measures of Effectiveness were chosen to illustrate the effect of the changing variables. The surviving percentage of each aircraft was tracked, as well

90

Danielle Soban

as the number of each type of SAM weapon fired. Finally, the number of runways and aircraft in the open destroyed was tabulated. These outputs are shown in Table 6.

Table 6 - Output Variables for Theater Level Survivability Study

Results from Case II Figure 31 shows the prediction profile for the survivability test case. The first thing to notice is that none of the repair rates for the airbase components have a significant effect on the responses. This is shown by a relatively flat line. There is a slight slope on the repair rate for runways when compared to the number of runways destroyed that does show a small effect of increasing runway repair rate and decreasing number of runways destroyed.

If the variable ranges on the repair rates had been

increased, or the air campaign plan decreased, the repair rate could have had more impact. The same result is seen with turnaround time for the two aircraft. The range around turnaround time was just too small to have an impact, given the air campaign plan. Track range for the SAM sites shows considerable effect on the responses. As the track range increases, more weapons are fired, and more aircraft are killed. This is

Danielle Soban

91

logical and intuitive. In addition, the destruction of the airbase components decreases. Note the deflection of the runway response to track range at its aft end. This indicates that the runway destruction begins to increase in rate towards the end of the air campaign. This could indicate that SAM site damage has an effect on its ability to protect the airbase components.

Interestingly, more SAM-2 weapons are fired at the aft end of the

campaign, and the number of SAM-1 weapons appears to decrease slightly. Maneuverability can be seen to have a slight effect on the number of airbase components destroyed, with the Strike 2’s maneuverability playing a slightly more significant role. Because maneuverability primarily affects the interaction between aircraft and SAM site, this result is somewhat intuitive. Once the aircraft has penetrated the defenses successfully, the maneuverability ceases to affect the actual kill. But it is interesting to note that increasing the maneuverability of Strike 2 (the slower, higher load-carrying aircraft) has a more significant impact than increasing that of Strike 1. Increasing the maneuverability of both aircraft does have a direct impact on the survivability of that aircraft. Detectability shows less promise. For Strike 2, increasing the detectability has negligible effect on the airbase components destroyed, as well as no discernable effect on increasing its survivability. For Strike 1, increasing its detectability does have a small impact on the number of airbase components destroyed and a small, interesting impact on its own survivability. Overall, it was discovered that the air campaign plan had a significant effect on the quality of the results. Force mix was tried as a variable and was found to

92

Danielle Soban

depend too heavily on the specific order of air strikes.

This points to significant

interactions in the code that need to be explored more fully. For a very simple air campaign, it was shown that increasing the percentage of Strike 1 aircraft in the overall number of aircraft did increase that aircraft’s survivability.

However, this increase

decreased the survivability of the Strike 2 aircraft, and at a more significant rate.

% Strike 1 Survived % Strike 2 Survived

SAM-1 Weapons Fired SAM-2 Weapons Fired Runways Destroyed

Aircraft in Open Destroyed

Detect. Strike 1

Detect. Strike 2

Maneuv. Strike 1

Maneuv. Strike 2

Turnaround Time Strike 1 (hrs)

Turnaround Track Range Repair Rate Repair Rate Repair Rate Repair Rate Repair Rate SAM SAM Radar Runways Shelters A/C in Open Time SAM Sites Fire Control (hr) (hr) (hr) (hr) Strike 2 (nm) (hr) (hrs)

Figure 31 - Prediction Profile for Theater Level Survivability Study

Summary of Preliminary Investigation Overall, it was shown that in general, statistical methods could be applied singularly to a campaign level code.

Most inputs to the code could be varied

appropriately, and metamodels could be formed relating the Measures of Effectiveness to those inputs. There were several issues, however, that were identified that could limit the usefulness of simply applying the current methods to the campaign level code. These issues, and their proposed solutions, are discussed more completely in the next chapter.

93

Danielle Soban

CHAPTER V

RESULTING ISSUES AND SOLUTIONS TO PRELIMINARY INVESTIGATION The previous chapter discussed the application of existing statistical methods directly and only to a campaign level code.

Although some specific results were

discussed in the previous chapter, the current chapter will focus on issues that were identified that could possibly limit the usefulness of a direct application of these probabilistic methods.

Solutions to these issues will be proposed, and a new

methodology will be formulated based upon these solutions.

Identification of Three Primary Issues

Level of Detail The first key observation that was made applying the current probabilistic methods to ITEM was that there were insufficient variables at the campaign level in place to model the aircraft in the detail that the analysis warranted. Kenneth Musselman, in a panel discussion at the 1983 Winter Simulation Conference, noted, “Aggregated measures are used to draw conclusions about system performance, while the detailed dynamics of the system go virtually unnoticed…A decision based solely on summary performance could lead to unacceptable results…It makes practical sense for us to learn how to properly amplify these details and to incorporate them into the evaluation process.” This is an illustration of a larger issue first discussed when describing the

Danielle Soban

94

military code continuum (Figure 21). Because of the increased complexity necessary to a campaign level code, a sacrifice is made in code detail. It would be impractical in terms of code development and run time to model each and every entity and interaction in the campaign code to an engineering level. Yet, how can one accurately assess the impact of, say, a technology applied at the engineering level, if the technology is not modeled in sufficient detail to have an impact? This is a currently a prominent issue in military campaign analysis [46,56,57,59,60]. Model Integration With the military code continuum fresh in mind, the most obvious solution might be to simply link detailed codes together. Recent efforts have been concentrated in this area [46,57,59], called “model integration”. This concept involves replacing a coarse model of an entity or technology with a more detailed model. In other words, it is moving a model or a part of a model “up” the continuum, from the engineering level to the campaign level. The result is a mixed fidelity simulation, and care must be taken that all of the links and hooks between the codes are complete and compatible. The drawbacks to model integration are similar to those associated with increasing the detail in a campaign analysis itself. If too many entities are modeled in too much detail, run time suffers dramatically. Also, creating the links between the models can be costly in terms of time, and validation must occur to ensure that the detailed model is indeed modeling the same thing as the coarse model.

Danielle Soban

95

Zooming Model integration is in reality nothing more than the time-honored method called zooming. Zooming can be defined to be “the act of expanding the view of a specific area of interest [56].” A software zoom, therefore, can be described as varying the fidelity of a model, or parts of it, in order to gain a more focused examination. An example of zooming as applied to the current probabilistic methods discussed in Chapter II is shown in Figure 32 [61]. In this example, an economic modeling code was used, and shows the direct operating costs and dollars per revenue passenger mile as a function of RDTE (research, development, test, evaluation) cost, production cost, and operating and support costs. For analysis purposes, it was desired to have a software zoom of the RDTE cost. This is shown in the bottom of the figure, where RDTE becomes a function of nine other economic variables. The original metrics, thus, are no longer just functions of the initial three economic variables, but also of the zoomed RDTE variable, with its nine components. The conceptual model, as the first step in the analysis procedure, is used to identify which entities in the analysis are viable candidates for model integration and zooming. Overall, these techniques are used to amplify the effects of entities of interest and to provide insight into their behavior.

96

Danielle Soban

1.3313

DOC 0.7028 1.3426

DOC+I 0.6959 1.3523

$ / RPM 0.7228 -.2

.2 k_RDTE Cost

-.2

.2 k_Production Cost

-.2

.2 k_O&S Cost

0.0367

k_RDTE

. 0189 -0.0092 0.33

0.02

2nd Tier Electrical Group

Raw Mat Test Rotor

0.24

0.49

Initial Tooling Hardware Design Rotor Rotor

0.33

0.17

Manufacturing Software Design Rotor Flight Mngt & Controls

0.21 Iron Bird

0.21 Hardware/ Software Integration

0.21 Wind Tunnel Hrs

Figure 32 – A Zooming Approach: Breakdown of RDTE Model Abstraction A new approach to solve this level of detail problem is called “model abstraction”. This approach does not involve the linking of detailed codes together with the aggregate model, but instead capturing the essence of the codes and linking that essence to the aggregate model [57]. This kind of formulation could improve model accuracy at a reduced execution time. Indeed, as stated in Reference 57, “the goal of model abstraction is to sufficiently reduce the complexity of a model, without suffering (too great) a loss in accuracy.” Although a comprehensive taxonomy of model abstraction techniques can be found in References 62 and 63, in general these techniques can be classified into three broad categories: model boundary modification, model behavior modification, and model

97

Danielle Soban

form modification.

The first, model boundary modification, involves relaxing the

boundaries placed on input variables, sacrificing some accuracy and detail for decreased complexity and run time. Given an aircraft aerodynamics analysis code, an example of this would be reducing the number of analysis points along the surface of the wing. The second model abstraction technique is model behavior modification. This technique would involve fixing input variables that have lesser impact on the final analysis metrics to a median or baseline value. By fixing these variables, complexity is again reduced. Finally, model form modification involves a full replacement of the model with some sort of surrogate. This could be in the form of a table lookup or a metamodel. The probabilistic methods described in Chapter II themselves involve, at their heart, model abstraction techniques. Model behavior modification is employed through the use of screening tests, which identify which input variables contribute the least to the variability of the response. Metamodels are used in the true model abstraction ideal: they are used to reduce the run time and complexity of the model in order to facilitate the incorporation of the probabilistic techniques.

Human in the Loop Dilemma The second issue identified in the preliminary investigation is called the “Human in the Loop Dilemma”. A major difference between engineering level and campaign level analysis codes is how the user interacts with the code. In a traditional vehicle sizing code, for example, the user will supply a set of inputs and the code will iterate on a sizing scheme to converge the vehicle according to the laws of physics and empirical

98

Danielle Soban

relationships. In ITEM and other similar theater codes, however, the user becomes an integral part of the analysis process. This means that the user periodically evaluates the effect of her/his decisions and can then change the parameters (either from that point or change initial input parameters and rerun the simulation) to provide improved results. ITEM was specifically designed to incorporate the use of human judgement to make strategic decisions based on the state of the forces at any given time. Figure 33 shows a typical analysis scheme for using a theater level code that includes the human interaction.

Define Campaigns

Set Up Forces for Single Campaign

Schedule Events for Time Step

Run Events for Time Step

Satisfactory Results?

Y Campaign

Y

Done?

N

N Modify Events for Time Step

ITEM Analyze Data from Campaigns

Figure 33 - Flowchart for Decision-Making for ITEM [58]

The major advantage of a human in the loop model is that it most closely mimics reality. Actual wargamers are employed, and the decisions that they make tend to be realistic and the resulting analysis is useful. However, the logistics of running such a model can be daunting. It often takes weeks to prepare to run a single simulation. The model databases must be properly loaded, the gamers thoroughly briefed, and a proper simulation facility, with all of the appropriate software and hardware, must exist and be operable. If there are any flaws in any of these “systems”, the gaming is delayed. In addition, the model itself is often run in real time, and the analysis must wait until the

Danielle Soban

99

results have been compiled, downloaded, and presented in a format compatible for analysis. The analysis of a single scenario, thus, could take days, weeks, or even months to complete, start to finish. A full human in the loop model is the epitome of complexity. The alternative to having the human in the loop is to use some sort of embedded rules (expert systems) to make decisions. There are some theater level codes that do this. The key drawback to this is that the rules have an inherent lack of flexibility to simulate real operational plans. In addition, these rules lack transparency in assessing cause and effect relationships. An example of this drawback in illustrated in the following example. Say that an embedded rule system is used to model the decisions made in a particular scenario. The results are summarized as follows: “The analysis shows that there is an 85% probability that this scenario (with its inputs) results in the loss of two aircraft carriers in the first four days of the event.” What is wrong with this statement? In the real world, losing two aircraft carriers is so completely unacceptable that, after the loss of the first carrier, the decisions (inputs) would be changed in order to ensure that a second carrier would not be lost. With embedded rules, unrealistic results such as these could be modeled and decisions based upon these results. The ideal model situation, therefore, is one in which the flexibility and capability of the human gamer/analyst is kept intact, yet the model itself has a minimum of complexity and realistic runtimes.

100

Danielle Soban

Scenario Significance The final issue identified by the preliminary investigation was the impact of the threat environment itself. It was found that the campaign scenario had a significant effect on the outputs. Usually when conducting a campaign analysis, a particular scenario is specified and used for the remainder of the analysis. Given a particular scenario, it can be fairly straightforward to optimize a technological and tactical solution that is applies only to that scenario. However, as discussed in the introduction, today’s world is full of rapidly changing threats and technologies. An optimized solution for one particular scenario may differ dramatically from the solution to a subtly different scenario. This lack of robustness can have significant implications to today’s decision makers, who need to consider a wide variety of ever changing threats and technological advances. Thus, the proposed methodology must be able to take into account a wide variety of changing threat environments and conduct analysis accordingly.

Proposed Solutions The previous section identified three primary issues resulting from the preliminary investigation of applying the existing probabilistic methodology directly to a campaign level code. These issues pose the challenge, and their solutions will become the backbone of the formulated system of systems effectiveness methodology. Two key framework concepts are proposed that result in an analysis environment that is both fully linked (in a combination of model abstraction and integration) and is fully probabilistic in nature.

Danielle Soban

101

“Abstragration”: A Linked Analysis Environment Although the word is rather awkward, “abstragration” does imply the literal combining of both the model abstraction and the model integration philosophies. It is proposed that just such a melding must occur, in the form of a linked analysis environment, in order to address some of the issues raised in this and previous chapters. Returning to the first chapter, it must be remembered that the primary goal of this research was to develop a methodology that could rapidly assess system effectiveness and technology tradeoffs for today’s complex systems, in order to aid decision makers. It was further postulated that this methodology must rely on a “system of systems” formulation, which would include some way of mapping vehicle level design variables and technologies to campaign (system) level Metrics of Effectiveness. From this, it is clear than a modeling/analysis environment must be created that links together models from the engineering level, through the mission level, to the campaign level. Chapter III outlined the basic hierarchical structure of the traditional pyramid of military models and discussed why a military code continuum was a better formulation of the hierarchy. Attention then turned to the problem of how to link these models together in the creation of just such an environment. Confirmed in the preliminary investigation of Chapter IV, it was found that the varying levels of detail and complexity from one end of the continuum to the other was a significant issue. The concepts of model integration (varying fidelity) and model abstraction (replacing codes with their “essence” to reduce complexity) must both be applied in order to create an appropriate environment. The

Danielle Soban

102

linked analysis “abstragration” environment will be created using the following tools and concepts. Creation of the Conceptual Analysis Model While this step will be included as a first step in the overall methodology, its completion is crucial before attempting to create the linked analysis environment. A conceptual model was defined in Chapter III to be an idea, algorithm, or “game plan” for the analysis. Thus, the first thing an analyst or decision maker needs to do is to clearly define the problem under investigation, understand and note which variables and technologies will be studied, and which metrics of effectiveness will provide the most appropriate information. This process will identify which models will be needed, where they are in the military code continuum, and to what detail each entity needs to be modeled. An example is now provided of a “first cut” at a conceptual model. Suppose the problem under consideration is to gain insight into the effect of a new aircraft radar. Let’s start at one end of the military code continuum. An engineering code that models the physics of the radar would seem like a good and necessary tool to use. However, just by itself, this tool could only give performance data of the radar. But in order to really assess the effect of this new radar system, it needs to be placed in its correct context: the radar needs to assigned to a platform, and that platform needs to be assessed as a component in the larger system, the warfighting environment. So by itself, this code does not provide the necessary information, even though the level of detail is superb. Moving to the other end of the continuum we may be tempted to start with a full blown campaign

Danielle Soban

103

analysis code. This would give us information (metrics) at the needed system level. However, such a code will be so “top level” that any inputs for aircraft radar would be limited to one or two variables at most, if any inputs exist at all. What is needed, therefore, is a link between the two extremes. The detail of the engineering code is needed yet the data needs to be assessed at the system (campaign) level. It should be noted that a direct link between the two extremes is not practical, and indeed violates the military code continuum. There needs to be an intermediate code at the mission level. The radar needs to be placed on an aircraft and that aircraft needs to be placed into a one vs. one or few vs. few situation in order to assess this new system’s performance. This data is then passed on to the campaign code. Because there is a clear analysis path from the campaign code all the way back to the radar code, transparency is enhanced and a proper assessment may be conducted. This example of a conceptual model does not include the definitions of the specific inputs and outputs, but the concept is clear. Applying “Abstragration” In some cases it may be that an appropriate analysis environment may be created by simply identifying and linking codes within the military code continuum. This would be an example of pure model integration. More often, however, this could be impractical for a variety of reasons, such as increased run time. In that case, the concept of model abstraction is brought into play and applied to the conceptual model. The screening test method of Chapter II could be employed to identify key variables, and parts or all of various models could be replaced with the “essence” of those entities. Because the new methodology is an extrapolation of those methods discussed in Chapter II, these

104

Danielle Soban

abstractions would be in the form of metamodels. Zooming technique are applied to those areas identified in the conceptual model as needing a more detailed analysis, or to enhance transparency in cause-and-effect relationships.

Full Probabilistic Environment The linked analysis environment created as a result of the conceptual model defines the codes and models to be used, as well as how they are to be linked together and which of the models may be “replaced” by metamodels. This completes the first framework concept used to address the issues identified in the preliminary investigation. The second framework concept is the inclusion of a full probabilistic environment. The original probabilistic methods described in Chapter II were developed to be applied to engineering level codes, specifically, aerospace vehicles. This was done in order to understand the cause and effect relationships between design variables and system responses.

The methods were then enhanced to allow the incorporation of

uncertainty and risk into design decisions as well as assess the impact of new technologies. The first part of the full probabilistic analysis environment will retain these capabilities at the engineering level.

Distributions will be assigned to key design

variables and the resulting performance metrics will be in the form of cumulative probability distributions. This first element of the full probabilistic environment will retain all of the key characteristics described in Chapter II and will be implemented in a similar fashion.

Danielle Soban

105

The second element of the full probabilistic environment is developed in response to the Human in the Loop problem. The primary characteristic to having a human as an integral part of the modeling and analysis process is the uncertainty that the human brings. Decisions and assumptions are made based on human experience, knowledge, and judgement. Yet even though these things can be mimicked by a rule-based process, the human being still retains the flexibility and creativity to throw in an anticipated decision. The ideal modeling environment will seek a compromise: retain the experience, knowledge, and judgement of the human while also allowing for flexibility and uncertainty that will not limit the solution. This will be accomplished in the proposed methodology by implementing a two step process: identify the key decision nodes in the analysis, and place probability distributions around the considered paths. Tree Diagrams/Decision Trees The tool that will be used to identify the key decision points in the analysis will the tree diagram/decision tree concept. A tree diagram is an organized list that shows all possible outcomes of an event. Reference 64 defines it as a “tool that systematically maps out in increasing detail the full range of paths and tasks that need to be accomplished in order to achieve a primary goal and every related subgoal”. A typical graphical representation of a tree diagram is shown in Figure 34. A decision tree is an extrapolation of the basic tree concept, and represents a decision process [65]. The nodes represent points in time where a decision can be made, and the branches emanating from the node represent the specific decisions. Often the probability of an event occurring can be associated (written beneath) the corresponding

106

Danielle Soban

branch. The most common way that decision trees are used is to determine an optimal decision for a complicated process. This is achieved by moving sequentially backwards from the terminal nodes in the network, and calculating the expected gains at the intermediate nodes. The “best” decision is the one with the maximum expected gain. In addition, decision trees are commonly used to determine the rules for rule-based algorithms.

In this sense, they are almost to be considered deterministic, and the

resulting rules are most definitely deterministic in nature.

Terminal Node

Decision Node

Probability Assigned (.5) to Node

Decision Paths

Terminal Nodes

Figure 34- Notional Example of Decision Tree Diagram

The proposed methodology will use the tree diagram/decision tree process to chart and identify key decision nodes in the analysis process. Typically, these would be those decision points in the scenario that have the most impact on the Measures of Effectiveness. Preliminary analysis could take advantage of screening test processes to refine the nodes. Once identified, analyst experience and knowledge is used to assign

107

Danielle Soban

probability distributions to each decision branch. This differs from traditional decision tree analysis in that a distribution will be assigned to the branch, not a single probability. These distributions will then be carried through the subsequent analysis. The final element of the full probabilistic environment addresses the issue of the changing threat environment.

As stated earlier, it can be fairly straightforward to

optimize a solution given a particular specified threat. Yet the ideal solution will always be the most robust solution. In order to obtain the most robust solution, a probabilistic threat environment will be used, in which threat parameters will be allowed to vary probabilistically.

This will aid in assessing the sensitivity of the Measures of

Effectiveness to those threat parameters. The three basic elements involved in the probabilistic threat environment are shown in Figure 35.

geometry, technology, requirements maps vehicle measures of performance to effectiveness values

Design Variables Mission Level Scenario

Threats Decision Nodes

System Measures of Effectiveness

Figure 35 – Proposed Full Probabilistic Environment

Danielle Soban

108

Impact Dials The use of the full probabilistic environment allows the creation of an analysis environment that is analogous to the technology scenarios discussed as part of the Technology Impact Forecasting techniques of Chapter II. The previous section discussed the concept of having probabilistic inputs as various locations in the analysis continuum (the engineering level, the decision nodes, and the threat inputs). To capture the effects of these probabilistic inputs, metamodels that relate the outputs at different system levels to these varying inputs will be created.

These metamodels can then be combined,

independent of the modeling environment in which they were created, into a new analysis environment. Because the metamodels are portable and have very quick execution times, they can be ported into, say, a spreadsheet format. The analyst will then have at their disposal a tool that can be manipulated rapidly, accurately, and efficiently. Because the metamodels were created using variable inputs, the spreadsheet analysis environment would allow the analyst the ability to vary all of the inputs (within the ranges to which the metamodels are valid) and assess their impact on the system outputs. In effect, the environment will allow the analyst to manipulate “impact dials”, similar to the k-factors discussed earlier.

Summary of Solutions The key issues were identified during the preliminary investigation discussed in Chapter IV. These were the level of detail problem, the human in the loop, and the significance of the scenario. Solutions to each of these issues were proposed and were

109

Danielle Soban

often based on extrapolations of existing probabilistic techniques. Figure 36 summarizes the solutions and the techniques that will be applied to each issue. The next chapter discusses the formal methodology for bringing all of these concepts together into a cohesive framework.

Solution Issue

Linked Analysis Environment

Full Probabilistic Environment

“Abstragration”

Level of Detail

Human in the Loop

Scenario Significance

Metamodels Zooming Tree Diagrams/Decision Trees Distributions around key decision points Impact Dials Probabilistic Threat Environment Impact Dials

Figure 36 – Summary of Issues and Proposed Solutions

110

Danielle Soban

CHAPTER VI

PROPOSAL OF NEW METHOD- POSSEM

Summary of Research Before presenting the new methodology, it will be useful to stop and summarize the research up to this point. Chapter I provided the motivation for the research. It discussed how the changing world climate affects today’s military decision makers. These decision makers need a way to quantify system effectiveness for three primary purposes: resource allocation, requirements definitions, and trade studies between system components. This can be accomplished by utilizing the system of systems concept. Rather than designing/optimizing an aerospace vehicle to its own performance, the vehicle instead needs to be placed in its correct context- the warfighting environmentand optimized to the new system (theater) level Measures of Effectiveness. In order to do this, there must exist a continuous mapping from the vehicle level to the theater level. The creation of this mapping is a key component of the research. The second chapter reviewed current probabilistic methods that have been applied to the aerospace vehicle level. The impetus for the development of these methods was to incorporate uncertainty and risk into the preliminary design phase of the vehicles. The methods were further expanded to allow the incorporation and effect of new technologies to be analyzed, resulting in the Technology Impact Forecasting environment and the

111

Danielle Soban

concept of k-factors. Extrapolations of these methods will be used in the formulation of the new research methodology. Chapter III discussed the nature of military modeling. It defined different types of models and discussed the difference between a conceptual model and a computer model. Classification of military models was outlined, as well as some taxonomies in use today. Hierarchical modeling concepts led to the idea of military code continuum, which classifies models into three levels: engineering, mission, and campaign.

It is this

continuum which became the foundation of the analysis environment used to formulate the system of systems effectiveness methodology. In Chapter IV, the current probabilistic methods of Chapter II were applied directly to a campaign level code. This was done in order to validate the appropriateness of applying probabilistic methods to a campaign level code, and to identify any issues which might arise. Three issues of concern were identified: the level of detail problem, the human in the loop issue, and the significance of the scenario (ever-changing threat environment and robust design). Chapter V offers solutions to the issues identified in Chapter IV. The concepts of model abstraction and model integration were discussed and combined into the awkwardsounding “abstragration”. The concept of zooming was explained, and together a linked analysis environment was proposed.

Further, a fully probabilistic environment was

envisioned, with probabilistic design variables, a probabilistic threat environment, and the identification, through the use of tree diagrams, of key decision nodes in the scenario, to which probability distributions would be applied.

112

Danielle Soban

The POSSEM Flowchart All of the preceding concepts and ideas are now combined into one cohesive methodology. This chapter will discuss, in a general and intuitive fashion, the proposed methodology.

The actual details of the implementation will be more thoroughly

explained through the use of an example, which is the subject of Chapter VII. Called the PrObabilistic System of Systems Effectiveness Methodology, or POSSEM, the framework outlines a step by step process to assess the effectiveness of a complex military system. The entire framework is shown in Figure 37, and each component of the process will be discussed in detail.

Difference Between Analyst and Analysis Tool At this point an important distinction needs to be made between the role of the analyst and the role of the analysis tool. The POSSEM framework is an analysis tool. It does not conduct analysis. Rather, it provides a clear, concise path to follow to aid the analyst in their assessments. As discussed in Chapter III, too often today modeling tools are confused with the actual analysis process. Just because a tool has been created and validated does not mean that anyone who can operate the tool is automatically going to generate useful, correct, and pertinent analysis.

The tool can only be successfully

operated by someone who is thoroughly familiar with the problem and has some understanding of both the inputs and the outputs used by the tool. The important, implicit assumption in the POSSEM framework is that it is to be used by an appropriate analyst.

113

Danielle Soban

POSSEM 2.0

Probabilistic System of Systems Effectiveness Methodology

Create Conceptual Model Inputs/Outputs

Establish Baseline(s)

Key Questions:

Design/Control Variables Impact Factors Measures of Effectiveness Mission Requirements Tactics

Vehicles Technologies

What problem are we trying to solve?

Define Scenario

Sub-systems

What level of detail do we need? What tools are needed and available?

Operational Situations Threats (probabilistic)

Identify Key Decision Nodes Decisions Assign Probability Distributions

Decision Trees

Create Linked Analysis Environment RCVR

Receiver

Software Zoom

Radar Code

a ftw So

Choose codes based on Conceptual Model

re nk Li

Weapon Code

Force on Force

Many vs. Many

Aircraft Synthesis

Missile Synthesis

Software Link One vs. One

“Abstragration” Zooming

Theater

Weapon Code

Aircraft Synthesis

Engineering Codes

Mission Codes

Campaign Codes

Create Full Probabilistic Environment Using Linked Analysis Environment Design of Experiments CASE 1 2 3 4 5 6 7 8

SREF -1 -1 -1 -1 1 1 1 1

SWEEP -1 -1 1 1 -1 -1 1 1

TURBO -1 1 -1 1 -1 1 -1 1

R1 y1 y2 y3 y4 y5 y6 y7 y8

Assign Distributions Design Variables Technologies Threats

Prediction Profile

R2 y11 y12 y13 y14 y15 y16 y17 y18

Monte Carlo

Probability Distributions

R1

R2

R3

Analysis Campaign Level

Responses = fn(Theater Level MoEs) System MoE = fn(MoP1, MoP2, etc)

Mission Level

MoP = fn(Xreq, Xdes, Xtech)

Engineering Level

X = fn(time-dependant variables)

Impact Dials

RESOURCE ALLOCATION REQUIREMENTS DEFINITION TRADE-OFF STUDIES BETWEEN SUB-SYSTEM COMPONENTS

Figure 37 – The POSSEM Flowchart

114

Danielle Soban

Create the Conceptual Model The first step in the POSSEM process is the most crucial: the creation of the conceptual model. As defined in Chapter III, the conceptual model is the “plan of attack” used towards solving a particular problem. It is the necessary up front work that the analyst needs to do before even considering running a single computer code.

The

conceptual model is a careful consideration of the problem at hand, and results in the identification of key elements that are subsequently used in POSSEM.

As part of

POSSEM, the conceptual model is created by answering three key questions: What problem are we trying to solve? What level of detail do we need? What tools are needed and available? The first question, what problem are we trying to solve?, serves to aid the analyst in identifying the basic goals of the analysis. Answers to this question provide information that aids in identifying what Measures of Effectiveness are needed, what input variables are appropriate, and, to some extent, what modeling tools may be necessary. A clear understanding of what the analysis goals are is crucial to a successful analysis. The next question, what level of detail is needed?, is an often overlooked element. Too many times the analyst will let the capability of the tools drive the analysis, rather than the other way around. The analyst needs to decide, before conducting any code executions, how good is good enough. What level of fidelity on the answer is needed? What basic assumptions can be made that simplify the problem without placing the analysis at risk? Which components need to be modeled in great detail and which can be

Danielle Soban

115

modeled more coarsely? The answers to this question will determine which types of codes and at what level in the continuum are needed. The final question, what tools are needed and available?, serves to recognize that as much as we would like to stay philosophically pure, analysts do sometimes have limitations on their available resources. A survey of appropriate modeling tools needs to be conducted, and the appropriate tools, at the appropriate level of detail, need to be selected. If an appropriate tool does not exist that meets the pure requirements of the analyst, a less suitable tool may be substituted. But this pre-analysis will allow the analyst to understand the limitations of their tool, and adjust their analysis accordingly. Once these three questions have been answered, the analyst will then have the resources and information to conduct the initial problem setup.

This involves

establishing the baseline vehicles and technologies, determining the specific inputs and outputs of the problem, and defining the scenario most suitable for the investigation of the problem.

But, as shown in Figure 38, the answers to the questions and the

establishment of the problem setup is an iterative process. Tradeoffs must be conducted between the three questions and the resulting three areas of setup. For example, knowing what problem is trying to be solved keys directly into what level of detail is needed to solve that problem. The level of detail needed may or may not be driven by what tools are available. The scenario that is defined must include in its inputs and outputs those entities that are to be studied.

116

Danielle Soban

A solid conceptual model creates a solid foundation for subsequent analysis. It allows the analyst to more thoroughly understand the problem at hand, and provides crucial insight and information useful to the remainder of the analysis. Create Conceptual Model Inputs/Outputs

Establish Baseline(s)

Key Questions:

Vehicles

What problem are we trying to solve?

Technologies Sub-systems

Define Scenario

Design/Control Variables Impact Factors Measures of Effectiveness Mission Requirements Tactics

What level of detail do we need? What tools are needed and available?

Operational Situations Threats (probabilistic)

Figure 38 – Create Conceptual Model Step of POSSEM

Identify Key Decision Nodes The next step to POSSEM is shown in Figure 39. This step works with the scenario defined during the creation of the conceptual model, and is used to help combat the human in the loop problem. The goal is to retain the flexibility and uncertainty of having a human involved in the decision and assumption making process, yet create an environment in which the computer codes may be run quickly and efficiently. To do this, the analyst conducts a pre-processing of the scenario/campaign.

Tree diagrams are

constructed and used to identify the key decision nodes. For a very complex scenario, the screening techniques of Chapter II may be employed to help identify which of the decision nodes contribute most to the variability of the response, and which can be set to their most likely value.

117

Danielle Soban

Once the decision nodes have been identified, the analyst uses their skill and experience to assign probabilities to each path. This completed environment will then be used as part of the full probabilistic environment.

Identify Key Decision Nodes Decisions Assign Probability Distributions

Decision Trees

Figure 39 – Identify Key Decision Nodes Step of POSSEM

Create Linked Analysis Environment The creation of the modeling environment in which to conduct the analysis is the next step (Figure 40). Using information generated in the conceptual model, modeling codes are selected that, together, will create an environment to which the answers to the problems posed in the conceptual model may be answered. During this step, the concepts of both model abstraction and model integration must be applied. Starting first with model integration, models are selected that form a continuous modeling path through the continuum, from the engineering level to the campaign level. Care must be taken to select the appropriate codes at the appropriate level of detail. Software zooming may be necessary to isolate and highlight a particular effect. Once the codes have been selected, the concept of model abstraction is applied. Those codes and areas that may be replaced

118

Danielle Soban

by metamodels will be chosen, in order to increase efficiency and runtime, with an acceptable loss of fidelity. The final step in the creation of the linked analysis environment is to link the various codes together in a computing environment. This could take the form of scripts that take the outputs of one code and feed it into the other, or the creation of a graphical interface or shell that conducts all the necessary data transfer. This step is not to be considered trivial by any means, and the successful creation of a linked analysis environment is a major achievement in the process.

Create Linked Analysis Environment RCVR

Receiver

Software Zoom

Radar Code

e ar ftw So

Missile Synthesis

Choose codes based on Conceptual Model

nk Li

Weapon Code

Force on Force

Many vs. Many

Aircraft Synthesis

Software Link One vs. One

“Abstragration” Zooming

Aircraft Synthesis

Theater

Weapon Code

Engineering Codes

Mission Codes

Campaign Codes

Figure 40 – Create Linked Analysis Environment Step of POSSEM

Create Full Probabilistic Environment Once the linked analysis environment has been created, it can be used to implement the full probabilistic environment of Figure 41. This involves applying the probabilistic methods described in Chapter II to the linked analysis environment. To this end, ranges are placed on the selected input variables, and a Design of Experiments is

119

Danielle Soban

conducted. Metamodels are created for those parts of the linked analysis environment identified in previous steps. Intermediate prediction profiles may be created at each juncture point for analysis purposes. Distributions are also placed around key threat variables, to model a changing threat environment.

These distributions are carried

throughout the entire analysis. Finally, code runs are conducted around the decision points in the scenario. The results of the code runs conducted in this step will be a series of linked metamodels. These metamodels are then imported into a spreadsheet environment for final analysis.

Create Full Probabilistic Environment Using Linked Analysis Environment Design of Experiments CASE 1 2 3 4 5 6 7 8

SREF -1 -1 -1 -1 1 1 1 1

SWEEP -1 -1 1 1 -1 -1 1 1

TURBO -1 1 -1 1 -1 1 -1 1

R1 y1 y2 y3 y4 y5 y6 y7 y8

R2 y11 y12 y13 y14 y15 y16 y17 y18

Prediction Profile

Assign Distributions Design Variables Technologies Threats

Monte Carlo

Probability Distributions

R1

R2

R3

Figure 41 – Create Full Probabilistic Environment Step of POSSEM

Analysis The final step of POSSEM is to use the generated metamodels and data to conduct the analysis (Figure 42). This is done by creating a spreadsheet environment that uses the metamodels to create analysis paths that link the outputs of one level of the continuum to the inputs of the next level. In this way, there is a traceable computational path that links the final Measures of Effectiveness down through the engineering level inputs. At each point along the analysis path, wherever there were probabilistic inputs, the spreadsheet

120

Danielle Soban

will allow those inputs to be changed (within their ranges of applicability) and the results updated in real time through the use of the metamodels. This is the “Impact Dial” environment, and is a valuable tool for the analyst. With this tool the analyst can explore the impacts of various assumptions rapidly and efficiently. The final goal of the method is for the analyst to use this information to answer the questions posed in the conceptual model, aiding in resource allocation, trade studies between system components, and requirements definitions.

Analysis Campaign Level

Responses = fn(Theater Level MoEs) System MoE = fn(MoP1, MoP2, etc)

Mission Level

MoP = fn(Xreq, Xdes, Xtech)

Engineering Level

X = fn(time-dependant variables)

Impact Dials

RESOURCE ALLOCATION REQUIREMENTS DEFINITION TRADE-OFF STUDIES BETWEEN SUB-SYSTEM COMPONENTS

Figure 42 – Analysis and Final Step of POSSEM

This chapter discussed and outlined, in an intuitive manner, the steps of the proposed Probabilistic System of Systems Effectiveness Methodology, or POSSEM. The

Danielle Soban

121

actual details of the implementation of POSSEM are best explained through the use of an example, which is the subject of the subsequent chapter.

122

Danielle Soban

CHAPTER VII

POSSEM PROOF OF CONCEPT In order to illustrate the details involved in the implementation of POSSEM, a representative problem was chosen and the methodology applied. The chosen example involved assessing the effects of survivability enhancements on a baseline aircraft. This particular problem was chosen because survivability enhancements are most easily and beneficially applied to the design stage of an aircraft, yet their impact is most properly assessed at the campaign level.

This allows an analysis across the full military

continuum, making it an ideal example. The chapter begins with a discussion of aircraft survivability and provides a detailed motivation for the selection of this problem. The chapter then continues with the full implementation of POSSEM as applied to aircraft survivability.

Survivability Concepts Aircraft combat survivability is defined to be “the capability of an aircraft to avoid and/or withstand a man-made hostile environment” [3]. Survivability is made up of two key components, susceptibility and vulnerability. Susceptibility is the inability of the aircraft to avoid threat elements that make up the hostile environment. It is measured by the quantity PH, the probability that the aircraft can be detected and hit. Vulnerability, the inability of the aircraft to withstand damage caused by a hostile environment, is measured by PK/H, representing the probability that the aircraft is killed if hit. The overall

123

Danielle Soban

probability of kill for the aircraft, PK, is related to susceptibility and vulnerability by the following equation: PK = PH PK/H The overall metric representing survivability is the probability of survival PS. It is related to the probability of kill by: PS = 1-PK Volpe [2] combines the concepts of susceptibility and vulnerability into three key tasks that must be accomplished in order to increase survivability: deny the targeting, degrade the threat, defeat if hit. The various elements of susceptibility and vulnerability are grouped appropriately under these headings (Figure 43).

Deny Targeting

Degrade the Threat

Defeat, if Hit

•Signatures

(if targeted) •Jamming

•Hardening

•Long range sensors

•CM/ECM

•Shielding

•Stand-off weapons

•Maneuverability/Agility

•Separation

•Passive attack

•Tactics

•Redundancy

•Mission Planning

•Sealing

•Altitude

•Filtration

•Speed

SUSCEPTIBILITY

VULNERABILITY

Figure 43 - Volpe’s Survivability 3-D Approach [66]

124

Danielle Soban

Ball [3] translates Volpe’s tasks into specific survivability enhancement concepts (Table 7). If care is taken during the preliminary design process, survivability features such as these can be added to the aircraft at little or no penalty in terms of cost, weight, or performance.

Table 7 - Survivability Enhancement Concepts [3] Susceptibility Reduction Threat Warning Noise Jammers and Deceivers Signature Reduction Expendables Threat Suppression Tactics Maneuverability Enhancements High Altitude Flight High Speed

Vulnerability Reduction Component Redundancy (with seperation) Component Location Passive Damage Supression Active Damage Supression Component Shielding Component Elimination

The Need to Bring Survivability into Preliminary Design Process In today’s environment of military conflicts, political instabilities, economic uncertainties, and limited resources, we are being asked to do more with less [2]. Shrinking budgets limit the opportunities available for development and procurement of new warfighting tools. Concepts that do successfully compete for monetary support must guarantee a high probability of success as well as a substantial useful life span. In addition, our forces are being asked to generate more sorties for more diverse missions with fewer assets [2]. Each aircraft is experiencing increased utilization. In this high demand environment, aircraft survivability is key.

125

Danielle Soban

During peacetime operations, a more survivable aircraft is safer and more reliable. This translates into lower maintenance costs and increased availability. This in turn makes the aircraft very affordable over its life cycle (Figure 44). The same argument can be made during wartime. A more survivable aircraft will need less maintenance. More of the aircraft will return from a given sortie, and these aircraft will have less damage than their less survivable counterparts. More unharmed aircraft means a larger reserve of aircraft, and availability increases. All of these contribute to the increase of aircraft life cycle, increasing its affordability. This increase in aircraft life cycle aids in offsetting the research and development cost of the more survivable aircraft.

Survivable

Safer More Reliable

Lower LCC

(during peacetime operations)

affordable

Figure 44 - Relationship between Survivability and Life Cycle Cost

Although survivability is a relatively new field, there have been studies that show that small changes in survivability can reap huge benefits. Data from [67] shows that during Desert Storm, the allies flew an average of 51 missions per aircraft (103,200 sorties with 2021 aircraft). Figure 45 shows the affect of survivability on force size as a function of missions flown [2]. For these 51 sorties, changing the survivability from 98% to 99% changes the surviving force from 36% to 60%. Figure 46 shows that an initial force of 100 with a 98% probability of survival attacks 3151 targets.

When the

126

Danielle Soban

probability of survival is raised to 99%, the number of targets attacked jumps to 3970, indicating that a 1% change in survivability produces a 26% change in force effectiveness. Such high potential benefits clearly illustrate the need to increase aircraft survivability. To summarize, increased aircraft survivability translates into an aircraft that lasts longer in combat and has a higher probability of returning unscathed or with less damage. The more aircraft that return, the larger the size of the attack force at any given time, and the larger the attack force, the more weapons are fired and the more targets are attacked. Finally, and perhaps most importantly, the greater the probability of survival for the

SURVIVING FRACTION OF A/C

aircraft, the greater the probability that the pilot will return safely. 1

PROBABILITY OF SURVIVAL

0.9

0.999

0.8 0.7 0.6

0.995

0.5 0.4

0.99

0.3 0.2 0.98 0.95 0.9

0.1 0 0

10

20

30

40

50

60

70

80

90

100

NUMBER OF SORTIES MR93-1076-003

Figure 45 - Effect of Survivability on Force Size [2]

127

Danielle Soban

PROBABILITY OF SURVIVAL

10,000

- STARTING FORCE 100 A/C

9000

0.999

- 1 TARGET PER A/C 0.995

TARGETS ATTACKED

8000 7000

0.99 6000 5000 0.98

4000 3000 2000

0.95

1000

0.9

0 0

10

20

30

40

50

60

70

80

90

100

NUMBER OF SORTIES MR93-1076-004

Figure 46 - Effect of Survivability on Force Effectiveness [2]

The Paradigm Shift and Survivability The paradigm shift of Chapter I is now invoked. The goal of the paradigm shift is to bring more and better information to light earlier in the design process. This concept applies to the design of an aircraft incorporating survivability. Until recently, preliminary aircraft design focused on tradeoffs between the traditional disciplines: aerodynamics, propulsion, structures, etc. However, issues such as technological advances and life cycle considerations have induced a shift of emphasis in preliminary design to include nonconventional disciplines. These disciplines, often called the “-ilities”, include disciplines such as maintainability, reliability, and safety, as well as crossover disciplines such as economics and stability and control. One of these disciplines is that of survivability.

Danielle Soban

128

It is crucial that survivability be considered during the preliminary design process. As stated by Sapp [68]: “But, perhaps the most important thing to realize at this point is that it (survivability) must be addressed from the first day the need for a new aircraft is envisioned. That need must be tied to a clear-cut mission requirement and must, at the same time, be matched with the threat it can expect to face…Once a design is frozen, once the lines are drawn in ink, the chance to optimize is lost.” This need is recognized by such entities as the Joint Technical Coordinating Group on Aircraft Survivability (JTCG/AS), whose primary goal is to advance and establish the design discipline of survivability [69]. Survivability is a function of both susceptibility (denying the target and degrading the threat) and vulnerability (the inability of the aircraft to withstand the damage caused by the hostile environment) [3]. Susceptibility and vulnerability, in turn, are functions of characteristics such as shape, performance, agility, stealth, mission requirements, and threat environment. Because these characteristics are themselves functions of basic design parameters and requirements, as are the other more traditional design disciplines, it becomes both necessary and cost-effective to consider survivability as a design discipline in its own right. The advantages to considering survivability during the preliminary design process are clear. The first of these is reduction in design cycle time. While it was common up until WWII to have very short time spans between conception and production, today’s complex systems have gestation periods measured in years [68]. The shorter realization times were a necessary function of the wartime era that fostered them, but didn’t allow

129

Danielle Soban

adequate research and development time to optimize the configuration.

Today’s

realization times are so long that often technology advancements surpass the production timelines of the aircraft, and the aircraft is produced with inferior technology simply because the conception phase occurred such a long time ago.

By considering

survivability earlier in the design process, the effects of survivability concepts can be predicted earlier and trade-offs conducted much sooner in the process. Like the paradigm shift illustrates, more information earlier in the design process reduces design cycle time. Additionally, adding survivability features to aircraft after they are in production is costly and inefficient. When it was realized during the conflicts in Southeast Asia that US aircraft were being shot down in large numbers, there began a frantic rush to develop and add survivability concepts to existing airframes [3,68]. The resulting additions were usually in the form of “humps, bumps, and bulges” [68] that added weight and drag even as they decreased performance and payload. Often the trade-off between decreased performance and increased protection eliminated potential concepts that could have easily been integrated into the original design.

Finally, if the survivability concepts are

considered in the preliminary design process, optimization of the overall configuration can be made. For example, an optimal stealthy configuration for an aircraft is not necessarily the most aerodynamic one. If the trade between stealth and aerodynamics was conducted at the preliminary design phase, an optimal setting of both could be implemented.

130

Danielle Soban

The Link to System Effectiveness Although the concepts and decisions that affect survivability are functions of preliminary design variables, the effect of these concepts are realized most not at the aircraft performance level but at the theater level. The primary metric for survivability is the probability of survivability (Ps), a metric that needs to be evaluated by putting the aircraft in an appropriate scenario and then analyzing its performance. Measures of this performance can be represented by system level Measures of Effectiveness: number of returning aircraft, number of targets attacked, etc. In this way, there is a clear connect between survivability concepts and system effectiveness metrics.

To consider one

without the other is to miss a significant piece of the overall puzzle. As stated in [3]: “The survival of a military aircraft operating in a hostile environment depends upon many diverse factors, such as the design of the aircraft, the skill and experience of the crew, the armament carried, the onboard countermeasures equipment, the offboard supporting systems, and the tactics employed. The cost of modern aircraft weapon systems, coupled with the requirement that the system be effective, makes imperatives the consideration of the aircraft’s survivability throughout the life cycle of the system.”

Example: Create the Conceptual Model An example of the proposed methodology, POSSEM, will now presented. The following sections will show a step by step implementation of the method. Each section will be comprised of one complete step in POSSEM, followed by the results and analysis of the problem.

131

Danielle Soban

The first step of POSSEM is to create the conceptual model. This is the most crucial step of the process, and is indeed, the most crucial step in any analysis problem. The conceptual model clarifies to the analyst what the key components of the problem are, and helps define a clear analysis pathway for its solution. In addition, it aids in identifying the tools and resources needed to solve the problem. The process of creating a conceptual model is iterative, with preliminary tradeoffs conducted that balance inputs and outputs, scenarios, and baseline characteristics. Figure 47 shows the steps in the creation of the conceptual model, repeated here for easy reference. Create Conceptual Model

Key Questions: What problem are we trying to solve?

Inputs/Outputs

Establish Baseline(s) Vehicles Technologies Sub-systems

Define Scenario

Design/Control Variables Impact Factors Measures of Effectiveness Mission Requirements Tactics

What level of detail do we need? What tools are needed and available?

Operational Situations Threats (probabilistic)

Figure 47 – Create Conceptual Model Step of POSSEM

Answers to Key Questions The first step in the creation of the conceptual model has the analyst considering three key questions. Iterations on the answers to these questions provide the foundation for the model. What problem are we trying to solve? Given the discussion in the first part of this chapter, the following conclusions can be drawn. First, survivability is an important capability for a military aircraft to possess.

Danielle Soban

132

Secondly, survivability can be divided into susceptibility and vulnerability. Susceptibility is function of aircraft geometry, aircraft performance and tactics, and stand-off weapons and sensors that the aircraft carries. Vulnerability is a function of the detailed design of the aircraft, and is affected by structural design, system redundancies, and materials selection. Finally, it is difficult or impossible to determine the survivability of an aircraft without placing it in a combat situation and assessing it there. From these conclusions we pose the problem: What is the effect of adding survivability concepts to an aircraft? Note that not all of the information used in the first paragraph was necessary to formulate the question, yet this information will be used to refine the conceptual model as part of the iterative process. What level of detail do we need? In order to assess the effects of survivability, it is obvious that some sort of modeling capability is needed that can model survivability concepts. If these concepts are to be placed on an aircraft, a modeling tool that provides a detailed model of the aircraft and can distinguish between various survivability concepts added to that aircraft is necessary. Assuming that the survivability concepts will affect the shape of the aircraft, as well as its size and its performance, the tool must be able to account for such effects. All of these ideas are pointing in the direction of an aircraft sizing and synthesis code. This code must be able to resize the aircraft given some geometric and payload changes, and must output performance characteristics as well as sizing results.

Danielle Soban

133

Moving to the other end of the analysis spectrum, these changes to the aircraft must be assessed at the appropriate level. It is postulated that this is the campaign level. Remembering the fluidity of the military code continuum, it may be that the effectiveness of the aircraft with the survivability concepts may be properly assessed at some sort of mission level, or a level in between. This information will be used in conjunction with the tools and resources that are available to iterate on a solution. In any case, the question of level of detail that is needed can be answered. A modeling tool capable of having as its outputs metrics that reflect a change in aircraft survivability must be used. Thus, the tool must model, in a coarse degree, some basic features of an aircraft carrying out its mission. Details of the aircraft are not as important to have modeled as the capabilities of that aircraft. Also, the code must take as its input some sort of scenario or combat situation. Finally, there must be a cohesive analysis path between the inputs and outputs of the problem. If a sizing and synthesis code is used for the inputs, and a campaign level code used for the outputs, then some sort of middle level linking might become necessary. This will be determined once the codes have been chosen and the specific inputs and outputs identified. The answer to the second key question then becomes: At the aircraft level, a significant level of detail is needed to properly model the effects of survivability concepts added to that aircraft. This implies a standard sizing and synthesis code. At the campaign level, the modeling code must be able to take aircraft capabilities as inputs, as well as provide a somewhat detailed scenario. Outputs must include metrics that reflect the effects of changes in aircraft

134

Danielle Soban

survivability. Middle level links between these two extremes may become necessary. What tools are needed and available? Finally, the available tools and resources must be taken into account. In a perfect world, availability would not drive analysis requirements. However, it must be realized that sometimes analysis must be compromised for sake of expediency and, indeed, the capability of performing any sort of analysis at all.

The beauty of making these

considerations part of the conceptual modeling process is that these potential negative effects can quite possibly be mitigated. By acknowledging modeling limitations, the analysis pathway can be designed to minimize these drawbacks, while still providing quality analysis. In addition, knowing which tools are available and being familiar with their capabilities, a more robust and complete analysis could result. The capabilities of the tools could point to analysis pathways not previously considered. This said, a return to the question is made in light of the specific example at hand. Several tools that were appropriate to the study were available and familiar to the researcher. These tools and their capabilities are described below. FLOPS So far in the development of the conceptual model it was determined that an aircraft sizing and synthesis code was needed. The code available and chosen by the researcher was FLOPS (Flight OPtimization System). FLOPS is a preliminary design and analysis program developed and supported by NASA Langley Research Center [70]. It is capable of modeling most subsonic and supersonic fixed-wing aircraft, including notional designs.

FLOPS is comprised of several interconnected analysis modules,

135

Danielle Soban

including weight and balance, mission trajectory, mission performance, aerodynamics, engine cycle analysis, takeoff and landing, community and airport noise, and economics. The code is capable of sizing an aircraft by performing a fuel balance given a specific geometry, scalable engine data (or using the cycle analysis), and mission. Once the aircraft is sized, FLOPS can perform standard mission analysis tasks, such as integral and point performance (range, endurance, fuel burn, excess power, etc). Figure 48 shows the analysis flow for FLOPS. Information that defines basic geometry, aerodynamics, propulsion, and the sizing mission are input. Drag polars are then calculated using input data, with internal algorithms being used to account for any missing or undefined elements or data in the regime of interest. An engine deck is then either read as input or the internal cycle analysis package is used to create propulsion data and to calculate tables for optimal maneuver schedules. The mission module uses this information to “fly” the aircraft through the mission, conducting a fuel balance until the aircraft is converged and thus sized. FLOPS may also be used to fly a previously sized aircraft through a mission to determine performance characteristics.

At this point,

additional analysis modules may be called to calculate such things as takeoff and landing performance, noise analysis, and economics. FLOPS was written as a single, fully portable program in Fortran. The source code is easily manipulated to add additional analysis capabilities. The input and output formats are intuitive and helpful, with a variety of debugging flags available.

136

Danielle Soban

FLOPS Input File

FLOPS

Input Block 1: Control Information Analysis Type = . . . Analysis Options = . . . ... Input Block 2: Weights & Geometry Wing Area = . . . Wing Weight = . . . ... Input Block 3: Aerodynamic Data CL = f(alt,Mach) CD = f(alt,Mach) ... Input Block 4: Propulsion System Data Thrust = f(alt,Mach,PC) Fuel Flow = f(alt,Mach,PC) ...

Aero. Analysis Routines

Propulsion Analysis Routines

Optimal Maneuver Schedules

Piecewise Integration Through Time: ∆t

∆(Distance)=(Speed)∆t

Distance=Σι ∆(Distancei)

∆(Weight)=(dFuel Burn/dt)∆t

Weight=Σι ∆(Weighti)

∆(Altitude)=(dAltitude/dt)∆t

Altitude=Σι ∆(Altitudei)

∆(Speed)=(Acceleration)∆t No

Speed=Σι ∆(Speedi)

Vehicle Sizing Loop: Revise Weight Estimates

Analyze Next Mission

Input Block 7: Performance Points SEP @ . . . Max Mach @ . . . ...

New Guess on Fuel Weight

No

Output Block 1: Input Echo Analysis Type = . . . Analysis Options = . . . ... Output Block 2: Aerodynamic Analysis Analysis Results = . . . Drag Polars = . . . ... Output Block 3: Propulsion Analysis Analysis Results = . . . Engine Deck = . . . ... Output Block 4: Mission Analysis Data Optimal Maneuver Schedules = . . . Mission Time History = . . . ...

Etc.

Input Block 5: Mission Description Cruise Distance = . . . Loiter Time = . . . ... Input Block 6: Performance Points SEP @ . . . Max Mach @ . . . ...

FLOPS Output File

Last Mission? Yes

No

Total Fuel Used = Available Fuel? Yes

Call Other Analysis Modules, Write Case Output to File

Output Block 5: Performance & Weights Point Performance = . . . Vehicle Weight Breakdown = . . . ... Output Block 6: Other Analyses TO/Land Time Histories = . . . Acoustic Noise Analysis = . . . Vehicle LCC Analysis = . . . Output Block 7: Additional Missions Mission Time History = . . . Point Performance = . . . ...

Figure 48 – FLOPS Analysis Flowchart [71] ITEM The development of the conceptual model points to the need for a campaign level code capable of modeling, to a somewhat coarse degree, aircraft and their capabilities and performance. The code additionally must be capable of flying these aircraft in some sort of a scenario, and output indicative effectiveness parameters. ITEM was chosen and was briefly described in Chapter IV. ITEM was originally developed in response to a request from the staff of the Commander in Chief Pacific (CINCPAC) to the Defense Nuclear Agency. The request was for a capability that would allow them to perform theater level analysis of joint force

137

Danielle Soban

expeditions taking place in the Pacific Theater [58]. Initial studies of this request led to the conclusion that the model must have the following capabilities: Simulate campaigns that reflect any Red or Blue campaign strategy or combat doctrine Integrate air, land and naval operations Simulate alternative campaigns quickly to explore many force mixes or courses of action Provide maximum transparency decisions and campaign outcomes

between

strategic

These desired capabilities led to modeling decisions in the development of ITEM, which are summarized in Table 8.

The first capability led to the decision to use human

judgement in the making of strategic decisions. Embedded rule systems were excluded because of their lack of flexibility and transparency (leading to the problems discussed in Chapter V). As a result, ITEM is an interactive model that may be stopped at any time in the analysis to allow the user to control her/his decisions (refer back to Figure 33), and full output of the model is displayed at these stopping points. A full Monte Carlo approach was ruled out. The second capability required that the simulation events cover a wide range of warfare areas, with these events interlaced in time to simulate realistic sequencing of events. This led to a relatively short event period of time (on the order of an hour). The third capability led to the goal of including no unnecessary level of detail, minimizing setup time and long run times. This is a common goal and feature of theater level codes, as discussed in Chapter III and shown in the military code continuum.

138

Danielle Soban

Table 8 – Desired Capabilities and Resulting Model Features in ITEM

Desired Capability

Model Feature Use of human judgement

Simulate campaigns that reflect any red or blue campaign strategy or combat doctrine

Embedded rules excluded Interactive model with mid-campaign outputs Full Monte Carlo analysis excluded

Large warfare areas for events Integrate air, land and naval operations

Events interlaced in time Short event period (one hour)

Simulate alternative campaigns quickly to explore many force mixes or courses of action

Provide maximum transparency between strategic decisions and campaign outcomes

No unnecessary detail to minimize setup and run times Event scheduling as straightforward/easy as possible

Achieved through incorporation of all of above

ITEM was written in C++ within a UNIX environment. It thus uses an objectoriented approach and is highly graphics intensive. ITEM uses a hierarchical structure of its database, allowing for the rapid creation of objects. This structure is shown in Figure 49. To create an object, all other objects that depend on that one must clearly be defined. For example, in order to create an aircraft, first the supplies, weapons, standard conventional loads, and emitters for that aircraft must be defined. Once all objects have been defined, scenarios are developed that include the geographic locations of the

139

Danielle Soban

campaign, as well as maneuver schemes and plans (for example, an air campaign plan or an amphibious assault plan). The war game is then executed by specifying the number of time steps to progress. Output files from the execution are available for post-processing analysis.

WEAPON

SUPPLY TYPE

TABLE CLASS OBJECTS SENSOR C O R E

SCL

AIRCRAFT

GROUND COMPONENT

LANDING CRAFT

SHIP CLASS

C AIRBASE O M MPA M CAP A STATION N D ACI / STATION C A MCM M AREA P A I AREA OF G INTEREST N

GROUND UNIT TYPE

SHIP SURFACE SHIP

GROUND UNIT

GENERIC COMPONENT

MOBILE SSM SITE

USER DEFINED INSTALLATION

SUPPLY DEPOT SEAPORT

TASK GROUP TG OPAREA

FIXED SSM SITE

GROUND FORCE

SUBMARINE

GROUND CORPS

AMPHIBIOUS ASSAULT AREA

SAM SITE

MOBILE SSM FIELD

GROUND ARMY

SPA

SEA AREA

EW/GCI SITE SENSOR COVERAGE AREA

LAND AREA

WEATHER AREA

SIDE INDEPENDENT AREAS

MINEFIELD

Figure 49 – Hierarchical Structure of ITEM Database

At this point it would be helpful to convey to the reader the basic computational structure of ITEM. ITEM uses the concept of fractional aircraft and weapons in its calculations. Damage assessments, strike capabilities, and other elements are calculated using multiplicative probabilities. This results in fractional elements being sent to attack, fractional weapons being launched, and fractional damage being done. While at first this may not seem intuitive, these numbers can be made analogous to a percentage. For

Danielle Soban

140

example, if the damage report comes back for an aircraft and is reported as 0.3, that can be interpreted as 30% of an imaginary fleet of 100 aircraft are damaged. This way of looking at things allows the analyst to conduct smaller, scaled down versions of actual combat scenarios. A summary of the logical structure involved in an air strike and defense will now be presented as an example of ITEM’s computational structure. The information given below has been summarized from Reference 58, and the interested reader is referred there for a more detailed discussion. First, air launch events are examined. Air launches are scheduled in the air campaign section of ITEM, and consist of selecting a time for the launch to occur, the aircraft and weapons package to be used, and the target(s) and their priority to be attacked. The raid elements are processed sequentially and the first step is to obtain the number of aircraft and weapons at the source. If the raid element cannot be supported by the source, then the raid is cancelled. (For example, a user might have specified launching an aircraft from a SAM site). The following is an example of the logic structure used to determine the raid element availability:

If, attacker is an aircraft, and if, source is an air base aircraft_avail = ac_readyairbase, aircraft and for every weapon, wep_i, in scl_load, scl_wep_i_avail = num_wepsairbase, wep_i Else, if source is a ship aircraft_avail = ac_readyship, aircraft and for every weapon, wep_i, in scl_load, scl_wep_i_avail = num_wepsairbase, wep_i Else attacker is a missile num_missile_avail = num_wepssm site, missile or num_missile_avail = num_wepship, missile

Danielle Soban

141

Similar logic structures are used to test for inclement weather. If the raid is launched, raid elements and weapons are subtracted from the inventory. The interactions between the raid elements and the defense assets are now modeled. A simple attack by aircraft against a defending SAM site protecting an airbase will now be described. The process is summarized in Reference 58: The module first determines whether the raid is detectable and engageable by checking to see if the closest point of approach (or minimum distance to the route segment end points) is inside the envelope of the SAM system (i.e., less than maximum range, less than maximum altitude, and greater than minimum altitude). If it is not, the SAM system versus raid event terminates. Next, it computes the location of the first intercept point against the most detectable raid element (i.e., the raid element with the longest detection range). If no intercept is possible for this raid element, then no intercepts are possible against any of the less detectable raid elements. If the intercept is beyond SAM range, firing is delayed until an intercept is possible. If the intercept is inside the envelope, the outcomes of the intercept (i.e., expected raid element kills and SAM expenditures are computed). Next, other raid elements are selected sequentially and the outcomes are computed. After all of the raid elements have been processed for the current intercept point, the module computes the next intercept point and the process continues until: (a) the site is out of ammunition, (b) all of the raid elements are destroyed, or (c) the next intercept point falls outside the SAM envelope. If the intercept falls inside the minimum SAM range, the intercept is delayed until the raid reaches minimum range on the outbound leg, unless the SAM is on a ship, in which case no more intercepts are attempted. The above process is shown in Figure 50.

142

Danielle Soban

Start

Raid elements detected ? yes

no

Compute the next intercept opportunity

Opportunity inside SAM envelope ? yes

no

Get the next raid element

Compute the intercept outcomes

All raid elements yes done ?

no

yes

Figure 50 – SAM Engagement Flow Chart [58]

The first step is to determine the number of raid elements detected. The horizonlimited radar range for the SAM site is computed as follows: Rh = 1. 23 ( Hs + Hr ) where, Hs = the height of the radar for the SAM site Hr = the altitude of the raid For each raid element, the aircraft’s relative detectability is used to adjust the nonhorizon-limited radar range for the SAM site. This is done by adjusting the tracking range, Rtrk,s, of the SAM site:

143

Danielle Soban

Rdet_unlim,e

=

[

Rtrk,s Frel det,r,e

]

1/ 4

with Rdet_unlim,e = non-horizon-limited radar range for SAM site = relative detectability of raid element Freldet,r,e The adjusted detection range for the SAM site is then the lower of the two radar ranges, either the horizon-limited radar range or the non-horizon-limited radar range. The closest point of approach of the raid element to the SAM site, Rcpa, is then calculated.

The number of aircraft in the raid element detectable at the minimum

distance is defined as Ndet and is computed as follows: For each raid element e, If, Rcpa ≤ Rmin_det,e then, Ndet,e, = Nr,e with Nr,e being the number of original raid elements The individual raid elements that are detectable are summed to generate the total expected number of targets detected for the entire raid, Ndetected, Ndetected

=

P op_ radar,s ∑ Ndet,e ∀e

with Pop_radar,s = probability that SAM radar is operational No event occurs unless at least part of a raid is detected at the minimum distance, that is: Ndetected

>

0.0

The geometry of the engagement process is shown in Figure 51. The trajectory occurs in an X-Y plane which crosses the X-axis at the closest point of approach (Xoffset

144

Danielle Soban

in the figure).

The trajectory is therefore parallel to the Y-axis.

The following

definitions are used in the calculations. Rsam = sam range = Rmax,ws Ssam = sam speed = Sws Sraid = raid speed = Sr Sratio = speed ratio = Sraid / Ssam Ssalvo = sam salvo size = Nsalvo,s Treact = reaction time = Teng,ws Nweps_avail number of weapons available = Nrail,s = Nsim_engage number of simultaneous engagements for = the site = Nmax_eng.s Pop_radar, s = Pop_fc, s = Nac_elem = = Nr,e

probability the site's radar is operational probability the site's fire control is operational number of aircraft or weapons in the raid element

Fmaneuver_elem = raid element = Fman,ar

maneuverability factor for aircraft in the

Rdetect, s = site's maximum detection and tracking range against the raid element with the largest relative detectability = max [Freldet,e] Rtrk,s

145

Danielle Soban

Rdetect

Sraid

Y

Ydetect

Rsam

Yfire

Yintercept Yfire

Treact

Treact

Ssam Yintercept

Ssam

X

Xoffset

raid

Figure 51 – Surface to Air Missile Engagement Process [58]

This geometry is used to calculate the first point of detection, Ydetect. The speed of raid multiplied by the reaction time of the SAM site is used to calculate the position of Yfire, which is the first opportunity of the SAM site to fire a weapon. Geometry is again used in conjunction with the speed ratio to calculate the point of intercept, Yintercept. Because ITEM allows up to a specified number of engagements at one time, more than one intercept and firing opportunities may be calculated. These are functions of the number of targets successfully detected and tracked, and the number of weapons available. This is shown in Figure 52. The number of salvos fired is then calculated. Nsalvos_fired = min(Nsalvos_avail, Ndetected, Nsim_engage) Pop_fc,s Ptrk,s

146

Danielle Soban

where, Nsalvos_avail Ndetected

= =

Nweps_avail / Ssalvo P op_ radar,s ∑ Ndet,e ∀e

Y

Sraid longest range intercept

Nmax_eng intercepts

Fire Point 1 Intercept Point 1

Fire Point 2

Ssam

Intercept Point 2

Ssam

Fire Point 3 Intercept Point 3

Ssam

X

Xoffset

Ssam Fire Point 4 Intercept Point 4

raid

Figure 52 – SAM Intercept Opportunities [58]

Finally, the outcome of the intercepts is calculated.

Examination of these

equations makes clear the fractional characteristics of the assessments. The number of salvos launched at each raid element is made proportional to the number of aircraft detected in the raid element divided by the total number of aircraft detectable. Nsalvos_elem = Nsalvos_fired (Nac_elem Pop_radar,s / Ndetectable) only for elements which are detectable.

147

Danielle Soban

The number of aircraft damaged in the element is the product of the number of salvos, the salvo Pk, and a maneuver factor for the raid element. The salvo probability of kill is, Pksalvo_elem

=

1 - (1 - Pksam, elem)salvo

= =

PKsam,ws,re Ssalvo

with Pksam, elem salvo

and, the number of aircraft damaged is Nac_damage, e

= Nsalvos_elem Pksalvo_elem Fmaneuver_elem

At this point the number of aircraft remaining in the raid element is decremented and the weapon inventory of the SAM ship or site is decremented. Mission Level Code The astute reader would now expect some mention of a mission level analysis code, in accordance with the military code continuum. Indeed, the outputs from the engineering level code (FLOPS) do not map directly into the inputs of the campaign level code (ITEM) and there does exist a need for a mapping between these two. But the reader is reminded that the development of the conceptual model is an iterative technique. Because the researcher is presenting an example of POSSEM, it made more sense to address the mission level mapping where it actually came into play: during the creation of the linked environment. The deficit in the analysis pathway is discovered at that point,

Danielle Soban

148

and the reader is urged to continue to that section, after the baseline section, for a more thorough discussion of the middle level mapping techniques used in this example.

Baselines Armed with the answers to the three key questions, the development of the conceptual model is continued. The next three steps: establishing the baselines, defining the inputs and outputs, and developing the scenario, are iterative in nature. Considerations of one influence the others, and decisions for these steps must be arrived at concurrently. However, in order to effectively record the process, the baseline step will be undertaken first. The goal identified of the analysis was defined to be the examination of the effect of adding survivability traits to an aircraft. This implies most obviously a baseline aircraft to which to apply the survivability traits. Several issues were considered during the selection of the baseline. First was the history of survivability itself. Earlier research, documented at the beginning of this chapter, indicated that survivability considerations did not become a primary concern until the conflicts in Southeast Asia beginning in the 1950’s. Because aircraft of that time were not designed to incorporate susceptibility into their features, using a baseline aircraft from this time could be misleading. In other words, the technological jump of redesigning a Korean or Vietnam war era aircraft to have major survivability features might be too distinct. Any change along those lines would be too significant. It would make more sense to choose a more modern aircraft that could incorporate increased survivability as one of its several, integrated features.

Danielle Soban

149

The survivability features would not dominate the aircraft, and a more realistic assessment of those features could ensue. Another consideration of the baseline is the availability of information in order to create and validate a model. Because a detailed sizing and synthesis code will be used, enough information would need to be available to “match” and validate the model. Drag polars, engine data, detailed sizing mission profiles, and geometric aspects would be needed. With these considerations in mind, the baseline aircraft chosen for the example was the Boeing F/A-18C. Aircraft Baseline - F/A-18C The Boeing F/A-18C Hornet was chosen as a representative modern fighter aircraft for this study. In addition to the F/A-18C having the performance characteristics of a modern fighter, geometric and performance data was readily available, allowing the creation of an accurate model. The F/A-18C is a twin engine, mid-wing, multi-mission tactical aircraft [72]. It was first flown in 1978, with operational deployment of the A/B models in 1983 and the C/D models in 1987. The Hornet has a ceiling of 50,000 ft, a maximum range (with external tanks) of 1,379 miles, and a maximum speed of over Mach 1.8. A three-view of the aircraft is shown in Figure 53 and characteristic data is shown in Table 9.

150

Danielle Soban

Figure 53 – Three View of Boeing F/A-18C [72]

Table 9 – Selected Characteristic Data of Boeing F/A-18C [72] Wing Span

37.5 ft

Aircraft Length

56 ft

Aircraft Height

15.29 ft

Wing Area

400 ft2

Max Gross T/O Weight

36,710 lbs

Internal Fuel Capacity

10,860 lbs

Engines Maximum Speed

Two General Electric F404-GE-400 Turbofans Mach 1.8+

151

Danielle Soban

Design Mission The design mission used to size the baseline aircraft was a fighter escort mission taken from the Standard Aircraft Characteristic [73] sheets, and is shown in Figure 54. The mission consists of takeoff and climb segments, a cruise of 311 nautical miles at best Mach number and altitude followed by a one minute combat segment.

The return

segment, also at best Mach number and altitude, includes a 20 minute sea level loiter. The weapons loading for the mission was two AIM-9 missiles on the tips of the wings, and a gun with ammunition.

Intermediate Thrust Climb 42,550 ft

Actual Modeled ( ) Cruise at Optimum Mach and Altitude (39,910 ft)

(40,000 ft)

41,300 ft (37,796 ft)

39,300 ft (38,904 ft)

38,100 ft

Reserves: 20 minutes Loiter at S.L. plus 5% of T/O Fuel Start & Taxi, Accelerate to Climb Speed 4.6 minutes at Intermediate Thrust, SLS

Combat at 10,000 ft 1 minute at Maximum Thrust Mach 1.0 (missiles retained)

Combat Radius =311 nmi

Figure 54 – Design Mission Used for Sizing and Matching the F/A-18C [75]

152

Danielle Soban

Validation The F/A-18C was modeled in the FLOPS synthesis code described earlier. The validation of the modeled aircraft was conducted by matching the output from the sizing code to the data in References 72 and 73. To do this, the geometry of the aircraft was added to the input file, as well as aerodynamic data read from figures in the references above. An external matched engine deck was also linked to the code [74] and its general characteristics are shown in Table 10. The engine is an F404-GE-402, which is an increased performance derivative of the F404. It features a dual-spool mixed flow turbofan architecture, 3X7X1X1 turbomachinery configuration. The engine is shown in Figure 55.

Table 10 – General Engine Specifications for F404-GE-402 Category Thrust

Specification 17,700 lb

SFC (max A/B)

1.74 lbm/lbf-hr

SFC (IRP)

0.81 lbm/lbf-hr

Airflow (SLS)

146 pps

Weight

2,282 lb

Length

159 in

Diameter

35 in

Danielle Soban

153

Figure 55 –F404-GE-402 Engine Used on the F/A-18C Model

The code was then executed and the outputs matched to the data. If there was a discrepancy, internal scaling factors in FLOPS were used to match the data [75]. Table 11 [76] compares the output weights from FLOPS with the actual weights. It should be noted that the weights were matched using scaling factors on the internal weight calculation equations, and were not input directly into FLOPS.

154

Danielle Soban

Table 11 – Weights Matching for the F/A-18C in FLOPS F/A18C Weight Breakdown Comparison Group

F/A18C

Baseline Model

Wing

3,919

3,918

Tail Group

1,005

1,006

Body

5,009

5,009

Alighting Gear

2,229

2,228

4,420

4,417

921

922

1,078 1,061 206 84 351 592 1,864 948 631 641 180 207 114 252 58 25,770

1,078 1,062 206 84 352 592 1,864 948 631 642 180 207 115 252 58 25,771 1,410

Propulsion Group Engines Engine Section Gear Box Controls Starting System Fuel System Flight Controls Auxiliary Power Plant Instruments Hydraulics Electrical Avionics Armament, Gun, Launchers, Ejectors Furnishings, Load/Handling, Contingency Air Conditioning Crew Unusable Fuel Engine Fluids Chaff, Ammunition Miscellaneous Operating Weight Empty Missiles (2) AIM-7F (2) AIM-9L Mission Fuel Takeoff Gross Weight

1,020 390 10,860 38,040

10,857 38,038

The matched drag polars for the F/A-18C are shown in Figure 56 [75].

155

Danielle Soban

Altitude = 36,089 ft 1.6

1.4

1.2

Mach

1

CL

1.8 1.5 1.2

0.8

0.9 0.6 0.2

0.6

0.4

0.2

0 0.00000

0.10000

0.20000

0.30000

0.40000

0.50000

0.60000

CD

Figure 56 – Matched Drag Polars for F/A-18C

System Inputs and Outputs The next step in the iterative process of the conceptual model is system level inputs and outputs. By “system level” it is meant the general inputs and outputs needed to provide the answers to the problem being analyzed. This does not mean that at this point in the process every input and output to every code being utilized needs to be defined.

Those specific inputs will be determined during the mapping and modeling

portion of POSSEM, taking place in the Create Linked Analysis Environment step. The concept of system level inputs and outputs is illustrated graphically in Figure 57.

156

Danielle Soban

Linked Modeling Environment Contains intermediate mappings

Code 1

Code 2

Code 3

Figure 57 – Determining System Level Inputs and Outputs

The reader is again reminded that the conceptual modeling process is iterative. At this point in the process, the analyst would need to have at least a basic idea of the scenario in order to formulate specific outputs. At the same time, having an idea of which outputs would be needed in order to analyze the given problem would help drive the scenario selection. The general problem goal of assessing survivability concepts must now be bracketed into a realistic testing circumstance. While the ideal survivability test case would involve trading not only between individual survivability concepts but between susceptibility and vulnerability concepts as well, it was thought that this would be too ambitious for an initial test case. Vulnerability analysis involves detail design studies, involving tools which are not currently available for use in methodology, but which may be added later. In addition, much work has already been done in the area of vulnerability analysis, with studies on susceptibility trades being more rare. It was thus decided that the initial test case of the systems effectiveness methodology would incorporate survivability trades by considering susceptibility concepts only.

The three key

susceptibility concepts that will be traded and analyzed during the study will be stealth,

157

Danielle Soban

tactics, and maneuverability.

These three concepts were chosen because of their

diversity, as well as their inherent capability to be modeled using existing tools. Inputs: Engineering Level Inputs were sought in FLOPS that would mimic the effects of adding survivability concepts to the baseline aircraft, based on the three areas chosen above. The first consideration was stealth, and the primary indication of stealth is radar cross section. Therefore, inputs were sought that would reflect a change in the baseline radar cross section. Reference 77 contains a study of the effect of changing geometric variables on radar cross section, and also uses the same baseline aircraft and modeling code. In concurrence with Reference 77, the following geometric variables were chosen: leading edge wing sweep, wing area, aspect ratio, taper ratio, and wing thickness to chord ratio. These variables were chosen in response to what was considered the primary radar area of interest: +/- 30 degrees in azimuth and elevation from a nose-on direction [77]. In addition, a varying parasitic weight factor was added as a variable to account for stealth. This technique was used in a previous study of the effects of stealth on notional aircraft, and is meant to model stealth enhancements, such as radar-absorbing paint, added to the aircraft in the form of a weight penalty [78]. Finally, a secondary mission was input for the sized aircraft. The mission altitude of this secondary mission was allowed to vary, mimicking a change in tactics: a low-low mission. While this tactic would decrease an aircraft’s susceptibility to defensive radar, a performance penalty is paid. Maneuverability in FLOPS is not an input, but rather a performance output of the code. As such, it comes into play not as an engineering code

158

Danielle Soban

input, but as an output, and will be allowed to vary at a different point in the analysis, as a function of its lower level system dependencies. The complete set of inputs to the engineering level code FLOPS is shown in Table 12. Ranges for the variables are indicated. For the geometric variables, the same ranges were used as in the study contained in Reference 77. Using the same variable ranges enhances conformity of data and also allows use of the existing resulting response surface equations for radar cross section, discussed in subsequent sections. The parasitic weight variable was allowed to range from 0 lbs to 1000 lbs, a range consistent with the study in Reference 78. Finally, secondary mission altitude was allowed to range from 0 to 10,000 feet, the altitude of the combat section.

Table 12 – System Level Inputs Variable

Low Value

High Value

15 degrees

30 degrees

400 sq ft

500 sq ft

Aspect Ratio

3

4

Taper Ratio

0.3

0.4

Thickness/Chord Ratio

0.04

0.06

Parasitic Stealth Weight

0 lbs

1000 lbs

0 ft

10,000 ft

Wing Leading Edge Sweep Wing Area

Secondary Mission Altitude

159

Danielle Soban

Outputs: Campaign Level The outputs at the system level will be those metrics obtained from the campaign level code that give insight into the questions posed in the first part of the creation of the conceptual model. These Measures of Effectiveness are shown in Table 13.

Table 13 – System Level Outputs % Blue Aircraft 1 survived % Blue Aircraft 2 survived No. of Weapons fired by RSAM1 No. of Weapons fired by RSAM2 Damage to RSAM1 Damage to RSAM2 Damage to RSAM3 Damage to RSAM4 No. of Runways Destroyed No. of Shelters Destroyed No. of Revetments Destroyed No. of Aircraft in the Open Destroyed No. of Cruise Missiles fired

The first two MoE’s are the survivability percentages for two Blue aircraft. The aircraft are identical, yet each carries a different weapons load. This was done to achieve transparency with respect to the weapon: it allows the analyst to isolate the effect of two different weapons. The next metric counts the number of cruise missiles launched, and can be used to determine the cost versus benefit ratio of the missile launch on Day 1 (the scenario will be discussed in more detail in the next section). The remaining metrics are measures of damage done to the Red side. The number of surface to air missiles that are used by each of the two engaged SAM sites is recorded. Damage at the airbase is tallied

Danielle Soban

160

by damage to the runways, the shelters, revetments, and number of aircraft in the open destroyed. Finally, the damage count to each of the four SAM sites is totaled.

Scenario The next step in the conceptual model was to determine the scenario to be used. Recall that this step is done iteratively with the selection of the inputs and outputs, and the determination of the baselines. To that end, realize that the outputs discussed in the previous section had to be determined from a basic outline of the scenario to be used. This just reinforces the iterative nature of the technique. The developed scenario had to fulfill some basic premises. First and foremost, it had to provide the environment in which the basic conceptual question(s) could be answered.

In addition, it needed to be detailed enough to showcase significant

interactions, yet simple enough to aid in transparency, and also to keep computational time (expense) to a minimum. In light of recent world events, and in order to focus attention on the method itself (avoiding preconceptions associated with certain regions of the world and military engagements), a neutral scenario location was selected. Just as Florida was used in the preliminary investigation, the test scenario will occur in the state of California, signifying a fictional confrontation between North and South California. Southern California was selected as the Blue side, leaving Northern California to represent the Red side. To build on previous knowledge, a somewhat similar geographical setup was developed.

However, the new scenario is considered more robust, and significant

161

Danielle Soban

analysis went into the selection of its details and geography. This geography is shown in Figure 58.

RSAM-4

RSAM-3 RSAM-1 RSAM-2

Southeast Airbase

Surface Ships South Airbase

Figure 58 – Scenario Geography for POSSEM Example

A Red airbase is located in the vicinity of Sacramento, and is protected on roughly cardinal points by four surface to air missile sites, named RSAM1, RSAM2, RSAM3, and RSAM4. The Blue side has a Task Force and surface ship stationed just off the coast of Monterey Bay. The ship is armed with cruise missiles. In addition, Blue has

162

Danielle Soban

two airbases located south and southeast of the Red airbase. There are two types of Blue aircraft stationed at each airbase.

Both are based on the notional baseline aircraft

described earlier, and differ only by their weaponry, which are in the form of air to ground bombs. Scenario element details are shown in Figure 59, Figure 60, Figure 61, and Figure 62. It is recognized here that some of the variable values are considered unrealistic.

These values were based on the numbers used in the preliminary

investigation, which were selected by JHAPL in order to keep the scenario unclassified. Because of these numbers, the reader is cautioned to view the results as trends and as a proof of concept of the method rather than as absolute numbers.

Type Min. Range Max. Range Speed Probability of Hit Launched Anti-

SLCM-1 ATG-1 ATG-2 SAM-1 SAM-2 cruise missile air to ground bomb air to ground bomb surface to air missile surface to air missile 0 0 0 0 0 450 10 10 30 25 * * * 500 500 0.9 0.9 0.7 0.6 0.8 ship air air land land land land land air air * not a factor

Figure 59 – Scenario Weapons Track Range (nm) Track Azimuth Limit (deg) Prob. of Track Max Engagement Salvo Size Number of Rails Stockpile Time to Reload (min)

RSAM-1 60 360 0.9 2 1 2 100 10

RSAM-2 50 360 0.8 4 1 4 100 15

RSAM-3 60 360 0.9 2 1 2 100 10

Figure 60 – Scenario SAM Site

RSAM-4 50 360 0.8 4 1 4 100 15

163

Danielle Soban

Mission Duration Speed Range Relative Detectability Maneuverability Max. Altitude (ft) Standard Conventional Load

Blue 1 Aircraft 2.5 750 750 variable variable 30,000 4 ATG-1

Blue 2 Aircraft 2.5 750 750 variable variable 30,000 4 ATG-2

Red Aircraft * 2.5 550 750 0.75 0.05 25,000 0

*in this scenario, Red Aircraft is target only. Stationed at Red airbase

Figure 61 – Scenario Vehicles

Runways Shelters Revetments Aircraft in Open No. of Blue 1 Aircraft No. of Blue 2 Aircraft No. of Red Aircraft Weapons (ATG-1,2)

Blue Airbase 1 4 20 10 70 50 50 0 400

Blue Airbase 2 4 20 10 70 50 50 0 400

Red Airbase 3 4 20 10 70 0 0 100 0

Figure 62 – Scenario Airbases

On Day 1, cruise missiles are sent from the surface ship to attack the four SAM sites. One missile is sent to target each site sequentially, and they are sent an hour apart. A twelve hour span occurs with no attacks, during which time the SAM sites attempt repairs. A second volley, identical to the first, begins again. The first day attacks are meant to damage the SAM sites, increasing the success of the Day 2 mission. On Day 2, the Blue airbases use the notional aircraft to attack both the SAM sites and the protected airbase. Four strike packages are created, with six aircraft in each. The first two aircraft, of the same type, are sent to target the first SAM site, and are tasked

Danielle Soban

164

with attacking the radar and the fire control systems associated with that SAM site. The next four aircraft of the strike package are identical, but of the other type than that attacking the SAM site. These four aircraft target airbase facilities; runways, shelters, revetments, and aircraft in the open. For the next strike package, the next SAM site is targeted, and the type of aircraft sent to the targets alternates. Only RSAM1 and RSAM2 are legitimate targets, as the geometry shows. The strike packages occur four hours apart, and both airbase and SAM sites are allowed to attempt repair during the no-strike hours. The attacks from the second airbase are identical to the first.

Summary of Conceptual Model The conceptual model is now completed for the example survivability case. It is the most important step in the process, and one that entails quite a bit of preparation and pre-analysis, as the length of this section attests. To summarize, the conceptual model is an iterative process that defines the entire analysis pathway for the problem. First, the problem must be well defined. This is accomplished by answering three key questions. In so doing, the scope of the problem is clearly outlined, and the necessary tools are identified. Next, baselines are established that will be used in the investigation. System level inputs and outputs are defined which are applicable to an appropriate scenario. Once the conceptual model is completed, the analyst may return to the rest of the POSSEM process. The example survivability problem is continued with the next step.

165

Danielle Soban

Example: Identify Key Decision Nodes The next step in POSSEM is to use the scenario developed in the conceptual model and identify the key decision nodes that occur. The motivation for this is to attempt to rectify the Human in the Loop problem discussed in Chapter V. These decision nodes will represent the places in the code that a human operator would normally stop and evaluate their inputs and decisions up to that point in the model run. Decision trees are used to aid in the mapping of the decisions. Figure 63 is a repeated figure of the decision node step in POSSEM.

Identify Key Decision Nodes Decisions Assign Probability Distributions

Decision Trees

Figure 63 – Identify Key Decision Nodes Step in POSSEM

Using the scenario discussed earlier, the first decision point occurs at the beginning of Day 1. The decision is whether or not to use cruise missiles fired from the surface ship to attack the Red airbase. The motivation for this attack is to damage the defensive SAM sites, increasing the effectiveness of the attacking aircraft on the successive day. The drawback of this decision is the potential cost associated with using the cruise missiles, and the potential depletion of that weapon source.

166

Danielle Soban

The second decision concerns which Blue airbase will be used to attack the Red airbase. Examination of the geometry in Figure 58 shows that an attack from the south would invade the defenses of RSAM-2. However, the attack route would take the aircraft directly over the SAM site, maximizing exposure time to defensive weapons. An attack from the southeast airbase would provide a route between RSAM-1 and RSAM-2, potentially exposing the aircraft to both SAM sites, yet minimizing exposure to both. Therefore, the second decision point is whether to attack from the south or the southeast. Because the example presented here is a proof of concept, it was decided that two decision points would be enough to demonstrate the efficacy of the method.

The

resulting decision tree is shown in Figure 64. The next step was to provide probabilities to each of the decision nodes. Because the use of cruise missiles do not put human life at risk, and comparing the relative costs of a cruise missile and a military strike aircraft, it was thought the most likely decision would be to use the cruise missiles on Day 1. Thus, a probability of 0.7 was assigned to the use of cruise missiles. Concerning the selection of the attacking airbase, it was thought that the southeast airbase might be less risky for the aircraft than the south airbase.

However, it is assumed there was insufficient

intelligence about the defensive capabilities (or perhaps even the exact locations) of the SAM sites, so the southeast attack was only considered slightly more favorable than an attack from the south. A 0.4 probability was assigned to an attack from the south, and a 0.6 probability assigned to a southeast attack. The resulting decisions will be modeled using the linked analysis environment developed in the next step.

167

Danielle Soban

Beginning of Campaign

Use Cruise Missiles

0.7

Day 1

0.4

0.6

Attack from South

Attack from Southeast

0.3

0.4 Attack from South

Do Not Use Cruise Missiles

0.6 Attack from Southeast

Day 2 Figure 64 – Decision Nodes for POSSEM Survivability Example

Example: Create Linked Analysis Environment Now that the conceptual model has been created and the scenario and its decision nodes have been identified, it is time in the process to create the linked analysis environment.

The conceptual model was classified the most important step in the

process. The creation of the analysis environment is now labeled the most difficult and time consuming step in the process. Even after the codes have been selected, linking them together in a cohesive, seamless way is by no means trivial, and setbacks at this step could endanger the entire process. The outline of the step is shown repeated in Figure 65. The goal of this step is to take the analysis codes decided upon in the conceptual model

168

Danielle Soban

and create software links between them, in order to use the finished product as an analysis tool.

Create Linked Analysis Environment RCVR

Receiver

Software Zoom

Radar Code

Missile Synthesis

k in eL ar ftw So

Weapon Code

Force on Force

Many vs. Many

Aircraft Synthesis

Choose codes based on Conceptual Model Software Link One vs. One

“Abstragration” Zooming

Aircraft Synthesis

Theater

Weapon Code

Engineering Codes

Mission Codes

Campaign Codes

Figure 65 – Create Linked Analysis Environment Step in POSSEM

The first step in this process is to examine the system level inputs and outputs determined by the conceptual model. The goal is to have a smooth analysis pathway that links the system level inputs to the system level outputs through the military code continuum. Assuming that middle level mappings are necessary, the outputs of one level must map into the inputs of the next. Part of this step, therefore, is to determine both these middle level mappings and the appropriate intermediate inputs and outputs. Each level in the analysis pathway will now be discussed, including inputs and outputs, and the links between them.

169

Danielle Soban

Engineering Level: FLOPS The inputs to the engineering level modeling tool FLOPS were presented and discussed previously, and are shown in Table 12. The selected output parameters for FLOPS are shown in Table 14. These consisted mostly of basic geometric sizing results (various weights) and performance parameters. The performance parameters will be used to map into maneuverability metrics at the middle level. The alternate range output is a result of the secondary mission. It is in effect the combat radius capable of the sized aircraft when performing the new secondary mission. Realize this number changes as the altitude of the secondary mission changes. It is a measure of the capability of the aircraft. While some of the output data will not directly be mapped as inputs into the next level, they were retained for insight and discussed in the analysis section. (Note that parsing outputs parameters from the existing FLOPS output files is a computationally inexpensive task.)

Table 14 – Output Parameters Tracked from FLOPS Tracked Outputs Takeoff Gross Weight (lbs) Operating Empty Weight (lbs) Takeoff Field Length (ft) Landing Field Length (ft) Alternate Range (nmi) Excess Energy Sizing (fps) Excess Energy Alternate (fps) Turn Rate Sizing (deg/s) Turn Rate Alternate (deg/s) Turn Radius Sizing (ft) Turn Radius Alternate (ft)

Danielle Soban

170

Radar Cross Section Mapping As mentioned earlier, Reference 77 contained a study that, in part, mapped aircraft geometric variables to radar cross section. This was accomplished by coupling geometric modeling tools together with a radar cross section prediction code and applying the methods of Chapter II. Because the baseline aircraft of that study was the F/A-18C and the variables and ranges used were identically chosen for use in this study, the resulting response surface equation was considered to be valid and useful in the present study. Figure 66 illustrates how the response surface equation was used. The equation of Reference 77 was a function of not only the wing geometric features but also the horizontal and vertical tail features.

Yet the results of that study, shown in the

accompanying prediction profile, showed little effect from the empennage variables on the radar cross section. Those variables were thus eliminated from the current study. In the equation, the values corresponding to the empennage variables were set to their nominal (midpoint) values and added to the intercept term. The reduced equation was then imported and combined with the other FLOPS output. Results are discussed in the analysis section. This completes the engineering level mapping. Figure 67 shows the engineering level mapping, including the inputs and outputs. The grey areas represent the mappings that still need to take place to achieve the complete cohesive analysis pathway. To try and identify what needs to be in the grey box, the outputs at the campaign level are examined and, working backwards, the inputs to that level are determined. In an ideal

171

Danielle Soban

situation, the outputs from the engineering level would map directly as the inputs to the campaign level. Realistically, however, this does not happen and a middle level mapping becomes necessary. The campaign level is now examined

intercept term (area, aspect ratio, sweep, taper ratio, t/c) RCS = b0 + f( horizontal tail geometry, vertical tail geometry, wing geometry)

retained variables for current study

set to their nominal values and added to intercept

Figure 66 – Use of Pre-existing Response Surface Equation for Radar Cross Section

Engineering

Campaign

Mission (middle)

System Level Outputs

System Level Inputs

% Survived Aircraft B-1

Parasitic Weight Detectability

Platform related

Secondary Mission Altitude Leading Edge Sweep Wing Area Aspect Ratio Taper Ratio Thickness/Chord

Radar Cross Section

Krcs

Excess Energy Sizing Excess Energy Alternate

Maneuverability

Turning Rate Sizing

Detectability

% Survived Aircraft B-2

Maneuverability

Number of SAM-1 Fired

Track Range RSAM1

?

Track Range RSAM-2

% Runways Destroyed % Shelters Destroyed

Repair Rate RSAM-1

% Revetments Destroyed

Repair Rate RSAM-2

% Aircraft in Open Destroyed % Damage RSAM-1

Turning Rate Alternate

Kexcess energy

Turning Radius Sizing

Kturning rate

Repair Rate Runway

Turning Radius Alternate

Kturning radius

Probability of Hit SAM-1

Alternate Range

Number of SAM-2 Fired

Probability of Hit SAM-2

Takeoff Gross Weight Operating Empty Weight Takeoff Field Length Landing Field Length

Figure 67 – Engineering Level Mapping

% Damage RSAM-2 % Damage RSAM-3 % Damage RSAM-4 Number of Cruise Missiles Fired

Danielle Soban

172

Campaign Level: ITEM At the campaign level, the outputs have already been identified. These are the outputs that will help the analyst gain insight into the problem being explored. Having previously identified ITEM as the campaign modeling tool of choice, the task is now to examine the inputs to ITEM and to select those most appropriate. Remembering the elements of the scenario to be used in the analysis, it is clear that the input variables associated with these elements must be examined. The scenario contains two Blue aircraft. These aircraft are the notional aircraft carried forth from the engineering level, and are modeled after the F-18C baseline. The aircraft differ by the standard conventional load each carries. The variables available in ITEM for an aircraft element are shown in Figure 68 [79]. The primary variables of interest in this study are the relative detectability and the maneuverability factors. Relative detectability, as discussed in more detail in the next section, is directly analogous to radar cross section. Both factors vary from 0 to 1. For detectability, a value of 1 means there is no reduction in aircraft detectability. A value of 0 means the aircraft is never detectable and thus never vulnerable to defensive systems.

The aircraft

maneuverability factor is a scaling factor on the probability of hit of defensive weapons. A value of 1 means there is no reduction in probability of hit. A value of 0 means the aircraft will never be hit. (The mathematical calculations used in ITEM involving these factors will be discussed in more detail in the analysis section). These two factors were selected as appropriate aircraft variables in the study. Other aircraft-specific inputs were set to values appropriate for the scenario.

173

Danielle Soban

Figure 68 - Aircraft Data Window in ITEM [79]

It should be noted at this point that the variables used to represent an aircraft in ITEM are quite few. This reinforces the argument put forth in this research that it is indeed necessary to link to a higher fidelity aircraft modeling tool in order to propagate changes made at the engineering level of an aircraft through to their effect at the campaign level. ITEM, and many similar campaign codes, simply do not contain the detail necessary to conduct such an analysis. Of course, as stated in Chapter III, this is a

174

Danielle Soban

normal characteristic of campaign level codes: a tradeoff is made between analysis detail and analysis breadth. The next set of variables was chosen to address one of the three primary issues identified in Chapter V.

It was postulated that a varying threat environment was

necessary in order to obtain the most robust analysis. Inputs were thus selected that would allow variability in the threat environment of the given scenario. These inputs included the tracking range of the RSAM-1 and RSAM-2 sites, and well as their repair capabilities. The last two variables are the probabilities of hit for the SAM-1 and SAM-2 weapons. (Realize that RSAM-3 and RSAM-4 are in essence non-players for the air attack events, due to the geometry of the problem. They were retained to mimic a cohesive defense environment around the airbase, and also come in to play during the Day 1 cruise missile attacks). In summary, the varying threat environment includes tracking ranges that shrink and grow, varying weapons capabilities, and changing repair rates at the defensive facilities. The complete set of ITEM inputs are shown in Figure 69, which also shows the mapping between variables at this point.

175

Danielle Soban

Engineering

Mission (middle)

Campaign System Level Outputs

System Level Inputs

% Survived Aircraft B-1

Parasitic Weight Detectability

Platform related

Secondary Mission Altitude Leading Edge Sweep

Radar Cross Section

Wing Area

Excess Energy Sizing

Aspect Ratio Taper Ratio Thickness/Chord

Excess Energy Alternate Turning Rate Sizing

Krcs

?

Maneuverability

Number of SAM-1 Fired

Track Range RSAM1 Track Range RSAM-2 Repair Rate RSAM-1

Number of SAM-2 Fired % Runways Destroyed % Shelters Destroyed % Revetments Destroyed % Aircraft in Open Destroyed

Kturning rate

Repair Rate Runway

% Damage RSAM-1

Kturning radius

Probability of Hit SAM-1

Kexcess energy

Turning Radius Sizing Turning Radius Alternate

Takeoff Gross Weight

% Survived Aircraft B-2

Maneuverability

Repair Rate RSAM-2

Turning Rate Alternate

Alternate Range

Detectability

Probability of Hit SAM-2

% Damage RSAM-2 % Damage RSAM-3 % Damage RSAM-4 Number of Cruise Missiles Fired

Operating Empty Weight Takeoff Field Length Landing Field Length

Figure 69 – Engineering Level Mappings and Campaign Level Mappings

Mission Level Mapping At this point the current analysis pathway was examined and a missing link discovered: some outputs of FLOPS did not map directly into inputs of ITEM (as is shown graphically in Figure 69). The two links that needed to be established were the mapping of radar cross section into ITEM, and the mapping of performance values, as a measure of maneuverability, into ITEM. These mappings correspond to the mission level in the military code continuum. This does not necessarily imply that a mission level code must be used. In a process similar to that of the conceptual model, candidate modeling codes and mapping capabilities are examined and the necessary level of detail is determined. For this particular problem, it was determined that the primary goal was to

176

Danielle Soban

establish an appropriate linked relationship between engineering level inputs and campaign level outputs. No direct analysis was needed at the mission level. Thus, a mission level code was not required to conduct the necessary mappings specific to the example being presented, but mission level relationships were needed. These mappings are discussed in the following sections. Detectability The first mapping needed was between the output of radar cross section and a similar input variable in ITEM.

The variable of “detectability” was examined.

According to Reference 79, the detectability variable was described as “The relative detectability of the aircraft. This is a scaling factor used to reduce the probability of detection of opposing forces against the aircraft in the engagement modules in ITEM. If the Relative Detectability is set to 1.0 there is no reduction in detectability. If it is set to 0.0, the aircraft will not be detected and therefore will not be vulnerable to defensive systems. Values between 0.0 and 1.0 reduce the probability of detection and, hence, the probabilities of kill for defensive systems proportionately.” This definition implies some sort of proportional relationship between detection range of defensive weapons and the detectability factor.

To truly understand what that

relationship is, and therefore use the variable correctly, Reference 58 was consulted. This reference describes how the non-horizon-limited radar range for a tracking SAM site against a raid element (aircraft) is computed.

The relative detectability of the raid

element is labeled Freldet,r.e and the SAM site’s tracking range is

177

Danielle Soban

Rtrk,s The non-horizon-limited radar range is then calculated as Rdet_unlim,e = Rtrk,s [Freldet,r,e]1/4 To understand the significance of this equation, the radar range equation is invoked [80]. 1

Rmax

 Pt G 2 λ2σL  4 =  3  (4π ) Pmin 

where Pt is the radar transmitter power G is the gain or electronic amplification of the radar λ is the wavelength of the radar σ is the radar cross section L is a loss factor accounting for atmospheric attenuation of radar energy Pmin is the minimum receiver signal threshold, determined by the background noise of the environment and the number of false alarms allowed and where

Pmin = TNR * kTs TNR is the Threshold to Noise Ratio k is Boltzmann’s constant (1.38 X 10-23 Watt-second/°K) Ts is the system noise temperature and includes internal radar and electronics and external random noise effects

For a given radar system, the equation reduces to

178

Danielle Soban

R = c4 σ Note the similarities with the non-horizon-limited radar range equation from ITEM! From this, it is clear that the detectability variable in ITEM is directly analogous to radar cross section. Thus, the radar cross section values output as a result of the mapping done at the engineering level can be directly mapped to the detectability variable in ITEM. The parasitic weight term added as a variable must also be mapped to radar cross section. In the referenced study in which parasitic weight was added to a notional vehicle, only the detriments of such an effect were catalogued; primarily the changes in takeoff gross weight and performance penalties [78]. It was concluded that to model the positive effects of adding stealth in the form of parasitic weight to the aircraft, a multiplicative scaling factor on the actual RCS should be used. The next issue was what range should be used for this scaling factor. Research in the open literature was not particularly fruitful: assigning values of stealth reduction due to stealth reduction techniques is highly classified. Because the parasitic weight was supposed to model an improvement analogous to radar-absorbing paint, research was conducted in that area. Words such as “significant” and “dramatic” were used to describe the reduction in RCS, but again, no numbers were ever found. Once source, however, did make the claim that “the USAF uses a type of radar-absorbing paint known as ‘Iron Ball’ which reduces the RCS of F-15 and F-16 fighters by 70% to 80%” [81]. Assuming that the F/A-18C is roughly analogous to these aircraft, it was assumed that such an improvement could also be achieved by this aircraft. Therefore, it was thought appropriate to examine an 80%

179

Danielle Soban

improvement in radar cross section due to radar-absorbing paint, modeled as parasitic weight. Realize, however, that this is merely a proof of concept study, and that the number is in fact a variable in the study. If better numbers are found for this range, due to new information or expert opinion, the methodology is such that the environment will already exist and the range may be changed without re-running cases. This is one primary advantage of the methods discusses in Chapter II and used herein. One final note is made on the multiplicative factor used on the RCS equation before discussing its implementation. The radar cross section is reported using the decibel scale. This is a logarithmic unit defined by: P(dB) = 10 log10(P/P0) Usually square meters are used for the reference P0, making the resultant unit dBm2. In order to use a multiplicative factor directly on the RCS equation, the RCS values were converted back into m2, and then the multiplicative factor was applied. The mechanics behind the RCS mapping are now discussed. Recall that the geometric variables from the engineering level were mapped to a radar cross section value using a response surface equation from Reference 77. This value will be referred to as RCSold: RCSold = fn(aspect ratio, wing area, wing sweep, thickness/chord) A multiplicative factor is used on RCSold to account for the change in radar cross section due to the addition of the parasitic weight:

180

Danielle Soban

RCSnew = krcs/pw RCSold

As stated earlier, the range on the krcs/pw factor will range from 0% improvement in radar cross section to 80% improvement in radar cross section. Recalling that the radar cross section will be mapped into the detectability variable in ITEM, and further recalling that by definition the variable must vary from 0 to 1, with 1 being no reduction in detectability and 0 being never detected, the krcs/pw factor must be applied appropriately. Thus, the krcs/pw factor ranges from a value of 1 (no reduction of RCS) to 0.2 (80% reduction in RCS). To calculate the resulting detectability factor, a reference value of RCS must be chosen. The maximum value of RCS possible, given the ranges of the variables, was shown to be 0.0541 m2. All other possible values represent a decrease in RCS, so this value is chosen as the reference value. The detectability value then becomes:

Dectectability =

Detectability =

RCS new 0.0541

k rcs/pw RCS old 0.0541

A comment is made here regarding the value of the RCS used to nondimensionalize the detectability. It is thought that the value here, representing the highest value of RCS given the variable ranges, is off by at least two orders of magnitude. The model fits of this number are discussed later in the analysis section, and possible suggestions for the low value of this number are given. One possibility is that the

181

Danielle Soban

modeling code used to generate this value did not incorporate the effect of the engines and inlets on the RCS. The RCS and corresponding response surface equation were found to be acceptable as incorporated into the research of Reference [77], and was thus considered adequate to use in the proof of concept of POSSEM. When calculating the maximum and minimum values possible for detectability, it is found that the detectability ranges from a value of 1 to a value of 0.157. While this may seem like quite a range, it must be remembered that the detectability factor is directly analogous to radar cross section, and, as such, the track range of defensive systems will be decreased by the fourth root of detectability value. Thus, significant changes in detectability are required to make significant changes in track range. Returning to the original parasitic weight range, the relationship between parasitic weight and the k factor can be described as:

∆ parasitic weight =

(1 - k

rcs/pw

0.8

)

∗ 1000

Thus, it can be seen that the relationship between changes in parasitic weight and radar cross section was assumed to be linear. Finally, one more mapping through radar cross section to detectability is performed. At the engineering level, the effect of tactics was modeled by changing the altitude of the secondary mission. This mission was completed using the sized aircraft from earlier in the process. The rerun, or alternate, mission is shown in Figure 70. The purpose of this mission was to vary the altitude of the cruise section from sea level to 10,000 feet in order to model an aircraft flying close to the terrain to avoid enemy radar.

Danielle Soban

182

Performance metrics were tracked, as well as alternate range, in order to quantify the performance and range penalty the aircraft would pay for this kind of tactic. The altitude of the flight should have a direct, positive effect on aircraft detectability: the lower the aircraft flies, the lower its detectability. However, when an aircraft engages in terrain following, the goal is to try and fly completely under the defensive radar, and thus be completely undetectable. This is a discrete event, and one that is difficult to model using the techniques employed in this research. To combat this, it was decided that an exponential scaling factor on detectability was most appropriate. At altitudes closer to sea level, the detectability is very low. As the aircraft increases its altitude, however, its detectability will rise at an exponential rate. (Realize, however, that when the altitude gets sufficiently high, the detectability again decreases. High altitude reduction in detectability was not modeled). This form of modeling has the advantage of a fairly sharp increase in detectability, attempting to model a discrete change in a continuous way. This formulation has another advantage: because the exact altitude of the defensive radar is unknown, an exponential curve helps to model the uncertainty in that exact altitude when the aircraft goes from being undetectable to detectable.

183

Danielle Soban

Variable Altitude (Sea Level to 10,000 ft)

Combat at altitude 2 minutes at Maximum Thrust Mach 1.0 (missiles retained)

Range calculated using available fuel

Reserves: 20 minutes Loiter at S.L. plus 5% of T/O Fuel Start & Taxi, Accelerate to Climb Speed 4.6 minutes at Intermediate Thrust, SLS

Figure 70 – Secondary Mission Using Variable Altitude

The final formulation for the detectability factor is  k rcs/pw RCS old  Detectability = (0.1) e (0.00023026 )(altitude )   (for altitude < 10,000ft)  0.0541 

Maneuverability The next mission level mapping needing to take place was the mapping of performance variables into the maneuverability variable available in ITEM. According to References 79 and 58, the aircraft maneuverability variable is a direct multiplier of the probability of hit of engaging weapons: “The measure of an aircraft’s ability to evade defensive systems. This is a scaling factor used to reduce the probability of hit of engaging weapons against the aircraft in the engagement modules in ITEM. If the Maneuverability Factor is set to 1.0 there is no reduction in

184

Danielle Soban

probability of hit. If it is set to 0.0, the aircraft will not be hit and therefore will not be vulnerable to defensive systems. Values between 0.0 and 1.0 reduce the probability of hit and, hence, the probabilities of kill for defensive systems proportionately.” In order to conduct an appropriate mapping, it is important to understand what aircraft maneuverability is. There is not, however, a universally agreed upon metric to measure maneuverability.

As stated in Reference 82, “Recent studies have made

considerable progress toward the understanding of aircraft maneuverability, agility, and related control power requirements. Researchers still fail to agree on basic ideas with regard to definitions of these qualities and, consequently, how to measure them.” Stinton [83] uses the following definition: “manoeuverability is the ability to change direction: the greater the change in a given time, the more manoeuverable is the aircraft.” That maneuverability is some function of the ability of the aircraft to change direction is echoed in Raymer [84], who gives a partial list of measures of merit for fighter aircraft: turn rate, corner speed, load factor, and excess power.

A definition for vertical

maneuverability is given by von Mises [85] as the inverse of the minimum radius of curvature for an aircraft performing a vertical loop. Finally, Shaw takes a stand and defines maneuverability in the following way: “Turn performance is the ability of an aircraft to change the directions of its motion in flight. This direction is defined by the velocity vector, which may be visualized as an arrow pointing in the direction of aircraft motion and having a length proportional to the speed of that motion. Maneuverability is defined in this text as the ability of a fighter to change the direction of its velocity vector. The terms maneuverability and turn performance, therefore, may be considered synonymous.”

185

Danielle Soban

Although these definitions differ in specifics, they share a common, intuitive feeling as to the relationship between performance metrics and maneuverability. An appropriate mapping, thus, of performance metrics to a maneuverability factor is proposed as a weighted, non-dimensionalized equation that is a function of excess power, turning rate, and turning radius. Remember that these performance metrics were tracked at two places for each design; once in the sizing mission, and once in the alternate mission. It was felt that analysis of both of these trackings would add insight to the analysis. The resulting formulation of the maneuverability mapping is therefore:

Maneuverab ility =

min Ps , alternate  k 2  min Trate sizing Trad alternate  min Trate alternate  k 3  Trad sizing k 1  min Ps , sizing + + +  +  +   2  Ps , sizing Ps , alternate  Trate alternate  2  Trate sizing 2  max Trad sizing max Trad alternate 

with k1 + k 2 + k 3 = 1 where min Ps,sizing is the minimum value of excess energy from the sizing mission Ps, sizing is the current value of excess energy from the sizing mission min Ps,alternate is the minimum value of excess energy from the sizing mission Ps,alternate is the current value of excess energy from the sizing mission min Tratesizing is the minimum value of turning radius from the sizing mission Tratesizing is the current value of turning radius from the sizing mission min Tratealternate is the minimum value of turning radius from the alternate mission

Danielle Soban

186

Tratealternate is the current value of turning radius from the alternate mission Tradsizing is the current value of the turning radius from the sizing mission max Tradsizing is the maximum value of the turning radius from the sizing mission Tradalternate is the current value of the turning radius from the alternate mission max Tradalternate is the maximum value of the turning radius from the alternate mission The three k factors are the weighting factors for each of the performance metrics. This allows the analyst to explore the various effects of each of the performance metrics, and also allows the analyst to use her/his own experience and data to provide appropriate weighting factors.

Linking the Codes Together At this point, there is a clear and uninterrupted analysis pathway established between the engineering level inputs and the campaign level outputs. All necessary intermediate inputs and outputs have been identified, and all essential mappings have been created. Returning to the issues identified in Chapter V, this step addresses the level of detail issue. So far, through the selection of the codes and the links created between them, the analyst has performed model integration. Remember, however, that it has been postulated that the optimum analysis capability comes from a blending of model

Danielle Soban

187

integration and model abstraction techniques (“abstragration”). The analysis pathway is thus examined to see if model abstraction techniques would be beneficial. The goal of the analysis in this example problem is to add survivability enhancements to a baseline aircraft and propagate the effects of these enhancements to the campaign level. The analysis pathway consists of two fairly complex modeling codes and a set of intermediate mapping relationships. A variety of inputs and outputs are necessary at each level. In addition, a probabilistic threat environment is desired, and all resulting pathways from the defined decision nodes must be analyzed. In order to complete the analysis in a computationally efficient way, it was decided that the modeling codes themselves should be subjected to model abstraction techniques. The response surface metamodeling techniques discussed in Chapter II and used for the preliminary results were therefore applied. The complete analysis pathway of the linked analysis environment, employing both model integration and model abstraction techniques, is now shown in Figure 71.

188

Danielle Soban

Engineering

Mission (middle)

Campaign System Level Outputs

System Level Inputs

% Survived Aircraft B-1

Parasitic Weight Detectability

Leading Edge Sweep

Radar Cross Section

Wing Area

Excess Energy Sizing

Aspect Ratio Taper Ratio

Excess Energy Alternate

Krcs

Maneuverability

% Survived Aircraft B-2

Maneuverability

Number of SAM-1 Fired

Track Range RSAM-2 Repair Rate RSAM-1

Turning Rate Sizing

Thickness/Chord

Number of SAM-2 Fired

Track Range RSAM1

Repair Rate RSAM-2

Turning Rate Alternate

Kexcess energy

Turning Radius Sizing

Kturning rate

Repair Rate Runway

Turning Radius Alternate

Kturning radius

Probability of Hit SAM-1

Probabilistic Threat Variables

Platform related

Secondary Mission Altitude

Detectability

Probability of Hit SAM-2

Alternate Range

% Runways Destroyed % Shelters Destroyed % Revetments Destroyed % Aircraft in Open Destroyed % Damage RSAM-1 % Damage RSAM-2 % Damage RSAM-3 % Damage RSAM-4 Number of Cruise Missiles Fired

Takeoff Gross Weight Operating Empty Weight Takeoff Field Length Landing Field Length

Response Surface Equations FLOPS

Middle Level Mappings

Response Surface Equations ITEM

Figure 71 – Complete Linked Analysis Environment Created for Survivability Example

Example: Create Full Probabilistic Environment At this point in the process it is time to start doing some computational work. The conceptual plan has been defined and the analysis environment created. These are the tools to be used to create the data for the analysis. The full probabilistic environment is the way in which this data is created. Figure 72 is repeated here to show the elements of the probabilistic environment. These are, in effect, elements that are described in Chapter II.

These elements will be combined with and applied to the Linked Analysis

Environment.

189

Danielle Soban

Create Full Probabilistic Environment Using Linked Analysis Environment Design of Experiments CASE 1 2 3 4 5 6 7 8

SREF -1 -1 -1 -1 1 1 1 1

SWEEP -1 -1 1 1 -1 -1 1 1

TURBO -1 1 -1 1 -1 1 -1 1

R1 y1 y2 y3 y4 y5 y6 y7 y8

Prediction Profile

R2 y11 y12 y13 y14 y15 y16 y17 y18

Assign Distributions Design Variables Technologies Threats

Monte Carlo

Probability Distributions

R1

R2

R3

Figure 72 – Create Full Probabilistic Environment Step in POSSEM

Creation of the Metamodels First, the metamodels, in the form of response surface equations, must be created for both the engineering level and the campaign level (as identified in the Create Linked Analysis Environment step). The process for doing this is identical to that discussed in Chapter II.

First, a Design of Experiments is created using the statistical analysis

package JMP, for the number of input variables. This file was then imported onto the Unix environment in which FLOPS is run. A Tk/tcl script was then written that used the DoE table and a baseline FLOPS input file to replace the baseline input variables with the values specified in the DoE table. A script was then used to run each of the FLOPS cases and parse the responses from the resulting output files. This data was then imported back into the JMP environment for analysis and the creation of the response surface equations. Creating the response surface equations for ITEM was a much more difficult process. ITEM is normally run using a graphical interface on a workstation. While a batch mode of ITEM did exist and was made available, the batch mode worked only with a binary data file. Thus, the method employed above, using a script to replace the input values, would not work. Yet running each of the cases through the GUI would be rather

Danielle Soban

190

impractical. A process that maximized the efficiency of running the cases was finally developed, utilizing both scripting techniques and graphical interfaces. First, a binary data file was created using the ITEM graphical interface. This data file was saved. At the same time, the graphical interface was used to save an ASCII file. This ASCII file, however, could only contain data pertaining to the elements of the campaign. The campaign itself (the timelines, strike packages, etc) could only be saved in the binary format. Also, the saved ASCII file had to have the same name as the binary file, or the ASCII file would not load back into the graphical interface. The ASCII file was then used as a baseline file and subjected to a script, similar to the FLOPS script, that would use the DoE table to find and replace the variables of interest. Realize that this process limits the choices of variables for use: only element characteristics could be chosen. The results of the script were numbered ASCII files, one for each DoE case. The ITEM program was then started in the graphical interface mode. The original binary data file was loaded. Each ASCII file was then loaded through the graphical interface and saved as a binary. Realize, however, that because the file names had to match, each numbered ASCII case had to be renamed right before it was loaded, then loaded, then saved again in binary format as its numbered name. This changed the name of the original binary file, so that file needed to be reloaded before the next ASCII file was loaded. This was the only way to combine the existing campaign plan with the changed element data into a single binary data file. A script was then used to run the binary files in batch mode, and to parse the resulting data.

191

Danielle Soban

The process described above is discussed here to illustrate a key point in using this metamodeling technique and methodology at the campaign level. Many modeling codes use graphical interfaces for data input and output. The metamodel method relies for its efficiency on the ability to rapidly run hundreds of cases and create a metamodel from the results.

When faced with manually running hundreds of cases through a

graphical interface, careful trade studies must be conducted to ensure that the method does indeed save execution time in the long run. For this example, there were four decision paths that needed to be explored. Each path required 129 code runs. Each set of 129 runs took one hour of run time to create the ASCII files, and the better part of a day for someone to manually create the binary files for the batch mode. Depending on the decision path and the number of inputs and outputs desired, the batch mode took between four and eight hours to run on a workstation for each set of 129 cases, and another two hours to parse the data. Factoring in the “doh!” effect (stupid errors that are a natural result from one person performing a tedious, repetitive task for several hours), plus adding the setup time for the original creation of each scenario and campaign, the cases here took a total of one week to create and run. This does not factor in the learning curve for the use of the code, or the development time for figuring out the how to make the metamodel process work. The lesson here reinforces the importance of the conceptual model. It is imperative to be aware of the benefits and limitations that come with each code selected, and to use it properly and efficiently. In this particular case, the benefits of the analysis capabilities of the code, coupled with the ability to run the code in a semiautomatic way, was clearly worth the somewhat difficult setup process.

Danielle Soban

192

The Complete Probabilistic Analysis Environment The resulting output data from both FLOPS and ITEM were imported back into JMP and the resulting response surface equations were created. An Excel spreadsheet environment was then created utilizing the response surface equations and the middle level mapping relationships. The input/output screen of this spreadsheet is shown in Figure 73. Values for the variables at each level (engineering, mission, and campaign) can be input directly into this screen. The equations are then invoked and the responses are automatically updated. With this tool, the analyst can put in whatever combination of variables (within the ranges for which the equations are valid) they wish to explore, and instantaneously see the effects. No more cases need to be run: an entire analysis space has been captured that links engineering variables all the way through to campaign level metrics of effectiveness. A primary goal of POSSEM is to provide the analyst the method of creating this analysis tool.

Danielle Soban

193

Figure 73 – Complete Probabilistic Analysis Environment

Adding the Probabilistics The real power of the analysis environment of Figure 73 is not the ability to explore single value effects, but rather to use the environment to explore the analysis space in a fully probabilistic manner. To do this, the probabilistic analysis package Crystal Ball [33] is used within the Excel environment. Probability distributions can be placed around any input value, and a Monte Carlo analysis conducted. This capability fulfills the promise of Figure 35: a fully probabilistic environment results. The design variables are allowed to vary probabilistically. At the mission level, k-factors are added

194

Danielle Soban

to allow the user to vary the effect of parasitic weight on radar cross section, and to allow the user to define which performance metrics, and by how much, affect the aircraft’s total maneuverability.

These factors can be set to a specific value, or allowed to vary

probabilistically. And at the campaign level, a probabilistic threat environment is created by assigning distributions to the threat variables. The final infusion of probabilistic variables comes from the decision points. Remember that there were two decision points, with two options each, for a total of four possible scenarios. Response surface equations were created for each of these scenarios and are shown in Figure 73 as Camp1 through Camp4. The next column is called “Combined Campaign”. This value results from applying a binomial distribution around each of the decision points, with the values of their likeliness corresponding to the probability of achieving that decision. When the Monte Carlo is run, results that are tallied come from whichever scenario corresponds to that “chosen” by the binomial distribution values. This results in a cumulative combined campaign output. To summarize, there are four areas in which probabilistic inputs can be applied. Distributions may be placed around the design variables, the mission level k-factors, and the campaign level threat variables. A binomial distribution is placed around the defined values for the decision points, and these, too, may be changed by the analyst to explore the effect of changing the decision likelihood. The resulting analysis environment is powerful, fast, and allows the analyst to play a myriad of exploratory “games”, providing information in a timely, intuitive manner.

195

Danielle Soban

Example: Analysis The final step of the POSSEM process is to perform the analysis (as shown in Figure 74). This step uses the data and analysis environment created in the previous steps to provide insight and quantifications to the analyst. To help the reader follow the progression of results, a roadmap of sorts is provided in Figure 75. First, the response surface equations for the engineering level, the mission level, and the campaign level will be examined and discussed. Even before they are incorporated into the probabilistic analysis environment, they provide a wealth of information and insight. Then the analysis will turn to the full probabilistic analysis environment itself, and some example analysis “games” will be played to illustrate the usefulness of the environment. First, design variables will be varied probabilistically while the threats are held constant. Next, the design variables will be held constant and the effects of a probabilistic threat environment are shown. Finally, both the design variables and the threat variables are varied simultaneously in a probabilistic manner, and the results presented. The reader is reminded at this point that some of the variable ranges used in the scenario were extrapolated from ranges used in the preliminary investigation. To keep with the unclassified nature of this project, some of these variable ranges are considered unrealistic.

The results, therefore, should not be considered as absolute, but rather

indicative of trends and relationships between the variables. This is appropriate as the goal of the implementation example of POSSEM is as a proof of concept, demonstrating its use and potential.

196

Danielle Soban

Analysis Campaign Level

Responses = fn(Theater Level MoEs) System MoE = fn(MoP1, MoP2, etc)

Mission Level

MoP = fn(Xreq, Xdes, Xtech)

Engineering Level

X = fn(time-dependant variables)

Impact Dials Figure 74 – Analysis Step in POSSEM

Engineering Level Results FLOPS

Response Surface Equations

Pareto Plots

Prediction Profiles

Mission Level Results Mapping Relationships

Response Surface Equations

Prediction Profiles

Campaign Level Results ITEM

Response Surface Equations

Prediction Profiles

Full Probabilistic Results Probabilistic Design Variables (Threats Constant) (Design Variables Constant) Probabilistic Threat Environment Probabilistic Design Variables

Probabilistic Threat Environment

Figure 75 – Analysis and Results Overview

Engineering Level Results The first set of results presented will be at the engineering level. The inputs described earlier in this chapter were selected to model survivability enhancements as

Danielle Soban

197

applied to the baseline F/A-18C. The probabilistic methods of Chapter II were applied to this baseline and utilized the sizing and synthesis code FLOPS. Effects of these variable inputs on the vehicle itself may be examined as part of the overall results. Screening Test A screening test was conducted using the input survivability variables and the outputs of choice. The resulting metamodels were first-order (linear) fits and are shown in the form of Pareto plots (Figure 76 and Figure 77). By examining these Pareto plots, the primary contributors to the variability of that response can be identified. For takeoff gross weight and operating empty weight, the contributors are the same. Parasitic weight and wing area both place a significant, almost equal, role followed by aspect ratio. Parasitic weight is expected to be a major contributor to the stealth of the aircraft, and thus the survivability of that aircraft. Note, however, that a key variable in reducing the frontal area of the aircraft, wing sweep, plays a small role in the weights of the aircraft. Now, the weights of the aircraft will not be directly mapped to higher levels, so at first glance keeping weights as an output might not make sense. But aircraft weight ties directly into the cost of the aircraft. The analyst, therefore, can assess the cost penalty for achieving a more survivable aircraft. Mission altitude is seen as a non-player. This is a good sanity check, as the mission altitude variable does not come into play until after the vehicle has been sized, so it should have absolutely no effect on the aircraft weights. The next Pareto plot is that of alternate range. This response is a measure of the capability of the aircraft. It indicates how far the sized aircraft can fly with its current fuel load and with a combat session. It is analogous to the combat radius of the aircraft.

198

Danielle Soban

Mission altitude, the altitude at which the secondary mission is performed, is the primary driver, as it should be. The secondary driver is aircraft wing area, a measure of the aerodynamic capability of the aircraft, and the parasitic weight, which is a measure of the weight of the aircraft.

All of these make intuitive sense.

Mission altitude of the

secondary mission was chosen to model the effects of tactics in survivability. Aircraft will often try to make a low-level mission run in order to evade defensive radar. The output variable alternate range is an indication of the performance penalty the aircraft pays for this tactic. Takeoff field length and landing field length were tracked as a way to assess the impact on performance capabilities of the aircraft. These output metrics will not be mapped to higher levels. The primary drivers are wing area, parasitic stealth, and aspect ratio. These indicate performance tradeoffs for increasing aircraft stealth. The next six Pareto plots (Figure 77) are all performance metrics. The primary metrics were parsed from the sizing mission of the aircraft, during the combat section of the mission (it is necessary in FLOPS to have a combat section from which to pull these performance parameters). The secondary metrics were all parsed from the secondary mission, in which mission altitude was allowed to vary. Not surprisingly, the primary drivers for all six metrics are identical. Wing area, of course, is a measure of the aerodynamic capability of the aircraft, as is aspect ratio. Parasitic weight affects the aircraft weight. performance.

Aerodynamics and weight are always chief contributors to aircraft These performance parameters all play a part in the aircraft’s

maneuverability, and as such will be mapped up to the campaign level.

199

Danielle Soban

Term

Orthog Estimate

Term

Orthog Estimate

Parasitic Stealth

787.7507

Parasitic Stealth

660.0424

Wing Area Aspect Ratio Thickness/Chord Taper Ratio Wing Sweep

738.3215 203.2109 -114.0118 54.5903 41.0836

Wing Area Aspect Ratio Thickness/Chord Taper Ratio Wing Sweep

613.3712 170.1585 -95.4529 45.7062 34.3940

-0.0000

Mission Altitude

-0.0000

Mission Altitude

Operating Empty Weight

Takeoff Gross Weight

Term

Orthog Estimate

Mission Altitude

50.96381

Wing Area Parasitic Stealth Aspect Ratio Thickness/Chord Taper Ratio

-12.88259 3.95956 1.02102 -0.56810 0.27704

Wing Sweep

0.21012

Alternate Range Term

Orthog Estimate

Wing Area Parasitic Stealth Aspect Ratio

-160.5304 72.9344 -27.1130

Thickness/Chord Taper Ratio Wing Sweep Mission Altitude

-10.3036 4.9806 3.7666 0.0000

Term

Orthog Estimate

Wing Area Parasitic Stealth

-233.4648 52.8876

Aspect Ratio Thickness/Chord Taper Ratio Wing Sweep Mission Altitude

13.4164 -7.4086 3.4242 2.5214 0.0000

Landing Field Length

Takeoff Field Length

Figure 76 – Pareto Plots for Engineering Level Screening Test 1

Term

Orthog Estimate

Wing Area Parasitic Stealth Aspect Ratio

-39.63143 -13.83201 -3.52532

Thickness/Chord Taper Ratio Wing Sweep Mission Altitude

1.97200 -0.95098 -0.71129 0.00000

Term Wing Area Parasitic Stealth Aspect Ratio Thickness/Chord Taper Ratio Mission Altitude Wing Sweep

Excess Energy Primary Term

Orthog Estimate

Orthog Estimate -37.87422 -13.44135 -3.43349 1.91908 -0.91985 -0.83736 -0.69261

Excess Energy Secondary

Term

Orthog Estimate

Wing Area Parasitic Stealth Aspect Ratio Thickness/Chord

0.6051875 -0.3085626 -0.0795181 0.0444984

Wing Area Parasitic Stealth Aspect Ratio Thickness/Chord

0.5712028 -0.2998855 -0.0773469 0.0433077

Taper Ratio Wing Sweep Mission Altitude

-0.0213387 -0.0159846 -0.0000000

Taper Ratio Mission Altitude Wing Sweep

-0.0207395 -0.0188406 -0.0155721

Turn Rate Primary

Term

Orthog Estimate

Wing Area Parasitic Stealth Aspect Ratio Thickness/Chord Taper Ratio Wing Sweep Mission Altitude

-143.9388 73.4636 18.8640 -10.4904 5.0428 3.7977 0.0000

Turn Radius Primary

Turn Rate Secondary

Term Wing Area Parasitic Stealth Aspect Ratio Thickness/Chord Taper Ratio Mission Altitude Wing Sweep

Orthog Estimate -150.7171 79.2613 20.3970 -11.4164 5.4086 4.9572 4.1323

Turn Radius Secondary

Figure 77 – Pareto Plots for Engineering Level Screening Test 2

Danielle Soban

200

Prediction Profile Once the screening test was completed, giving insight to the analyst as to the primary drivers of the responses, the model runs were repeated with a central composite design consisting of 143 model runs. Response surface equations were created mapping the responses to the input variables. No variables were eliminated during the screening test: the variables of interest were few enough in number that their retention did not significantly affect the computation run time. The resulting metamodels are presented in the form of a prediction profile (Figure 78). As previously described, the response surface equation from Reference 77 was imported and added to the data, in order that all of the results be seen on one cohesive prediction profile. The prediction profile echoes the relationships seen in the screening test Pareto charts. Its primary advantage is to allow the analyst to see all of these results side by side. The prediction profile tool may be used interactively by the analyst on a computer. The hairlines may be clicked and dragged, changing the values of the variables in real time and the screen will update the values of the response metrics and show the shapes of those relationships in real time. Figure 78 is a snapshot of that interactive screen, with all variables set to their nominal (midpoint) values. The radar cross section response is the first response of interest. Although the data in Reference 77 was presented in decibels, for this study the data was also converted into square meters to provide a more intuitive reference. As shown in Figure 78, the model fits were not affected by the units of the data. The first thing to notice is the

Danielle Soban

201

unusual model fit shapes for radar cross section with respect to aspect ratio, taper ratio, and (to a lesser extent) wing area. At first glance these trends do not make intuitive sense, so the models were examined to see if any reasons for these trends could be identified. First, the correlation of the estimates was examined. The correlation is a function of the geometric design of the experiment, and may introduce error into the design regardless of the accuracy of the data (or how it represents “real life”). Examination of the data did reveal some factors that indicated some correlation. These were most probably functions of the 10 cases excluded from the 287 model runs by Hines [77]. However, they were not of such significance that correlation could be pointed to as the sole cause of the unusual model fits. The next area of investigation was the interactions of the variables. Unlike the correlation, the interactions are a function of the data. Examination of the data showed some undesirable interactions, especially with the horizontal tail geometry variables. Because in the current research the horizontal and vertical tail geometry was not varied (the variables were set to their nominal value and added to the intercept term in the response surface equation), the errors associated with these terms may have had a sort of trickling effect into the error of the resulting response surface equation. It should also be noted that any variable interaction with wing aspect ratio had a highly observable quadratic effect. However, the magnitude of these errors again precludes any substantial effect on the fit of the radar cross section response.

202

Danielle Soban

Finally, the data was examined to see if it was being influenced by higher order effects. An examination of the scatter plot for radar cross section showed a fairly nice scatter, indicating a lack of error from higher order effects. A final source of error could be the modeling itself. The true effects of aspect ratio and taper ratio could have been inadvertently disguised through mathematical relationships used in the set up and running of the models. The source of error could also be in the actual code or codes used to generate the data, although Hines does provide substantiation for the validation of the codes used.

Without knowing the precise

formulation of the model runs, no exact conclusions can be formulated about the odd shape of the model fits for radar cross section with respect to aspect ratio and taper ratio. Examination of the data from a statistical viewpoint indicates that, although there are some minor sources of concern that could contribute to modeling error, overall the model fit is satisfactory. One cause could be errors in the actual modeling itself. Another possibility is that the trends are, in fact, correct yet just not intuitive. It has been surmised that the model did not take into account the effect of the engine inlets. If so, the RCS values gained from the research should be viewed as a delta increment on RCS due to geometric variation, and independent of engine inlets. The next two variables contributing to radar cross section are wing sweep and thickness to chord ratio. Wing sweep can be seen to have a significant effect on radar cross section, and the trend is as one would expect. As wing sweep increases, radar cross section decreases. This is because the frontal area of the aircraft decreases as the wing is swept back, and radar cross section is a direct function of frontal area. Thickness to

Danielle Soban

203

chord has the same effect: increasing the ratio increases the frontal area of the aircraft, and thus increases radar cross section. The next two variables, parasitic stealth and mission altitude, have no effect simply because they were not included as part of the original response surface equation. This is why they are labeled “not modeled”. These two variables, however, should have an effect on the radar cross section, and their mapping was accounted for during the middle level mapping. When discussing the screening test results, each response was discussed with respect to its primary drivers. The prediction profile of Figure 78 shows the same information from a different perspective. It allows the analyst to see the simultaneous effect of each variable on all of the responses. Starting with aspect ratio, one can see that it has very little effect on any of the response metrics, except for the radar cross section (which is itself suspect, as discussed earlier). If the trend for radar cross section with respect to aspect ratio is to be believed, then the analyst can conclude that radar cross section is highly sensitive to aspect ratio, and the optimum value of aspect ratio must be very carefully defined. At the same time, changing aspect ratio has little effect on the other responses, indicating there are few penalties to be paid for the correct value of aspect ratio. If the trend for radar cross section with respect to aspect ratio is to be ignored, then aspect ratio in its entirety can be said to have little effect on the overall design. The same arguments may be made for the taper ratio variable. Wing area, according to Figure 78, can be seen to be an important variable with a myriad of effects. Increasing wing area increases the weight of the aircraft, and thus the cost. Yet at the same time, takeoff and landing field lengths are decreased. Excess

Danielle Soban

204

energy is decreased, so a performance penalty is taken, but turn rate is increased and turn radius is decreased so a performance benefit is seen. Radar cross section is affected minimally, so the analyst can conclude that, if wing area is going to be considered as a survivability consideration, its role is significant in how it affects the aircraft’s performance, i.e. maneuverability. Wing sweep shows very little effect on all responses except for radar cross section. This would indicate that optimizing the aircraft for radar cross section with respect to wing sweep may be accomplished with little effect on the rest of the design, as indicated by those responses. Parasitic stealth is another interesting variable. Increasing parasitic stealth gives penalties across the spectrum of design responses. The benefits of parasitic stealth are not shown until the middle (mission) level of the analysis is reached. This supports the idea that the military continuum must be used in order to properly assess the effects of survivability concepts. At the engineering level, only penalties can be seen, leading the analyst to make erroneous conclusions about the effect of parasitic stealth. Finally, the mission altitude variable shows an effect only on the altitude range for the secondary mission. This is intuitive: the secondary mission is conducted after the aircraft is sized, and thus should have no impact on those responses that are a result of the sizing. There is a potential for some effect to be seen in the secondary performance variables, but these effects are seen to be negligible. This makes sense as altitude in general should not significantly affect the performance metrics tracked.

205

Danielle Soban

not modeled until middle level RCS in dB

-13.7 0.054 RCS in m^2

Stealth

-12.7 -13.452

0.045173

Takeoff Gross Weight Operating Empty Weight

26276.39

Takeoff Field Length

38590.06

Alternate Range

Responses

Landing Field Length

Sizing Results

0.0426 40779

36883 28092

24839 3198 2874.53 2636.5 5140 4781.355 4512 335.3 226.4788

Excess Energy Sizing

652.1287

Excess Energy Alternate

619.5841

Turn Rate Sizing

14.55787

Turn Rate Alternate

13.82599

Turn Radius Sizing

3392.271

Turn Radius Alternate

3571.89

712

590.27 677.8

559.08 15.5

13.338 14.74

12.637 3701

3187.9 3907

Taper Ratio

5,000 Mission Altitude (ft)

Variables

Figure 78 – Prediction Profile for Engineering Level

10,000

0

500 Parasitic Stealth (lbs)

1000

0

0.05

Thickness/ Chord

0.06

30

22.5

Wing Sweep (dg)

0.04

15

0.35

0.4

0.3

450 Wing Area (sq ft)

500

4

3.5

Aspect Ratio

400

3352

3

Combat Performance

183.45

206

Danielle Soban

Mission Level Results The next set of results presented will be the mission, or middle, level results. The mapping relationships for detectability and maneuverability, discussed earlier, were used. A Design of Experiments was wrapped around the relationships, and the resulting prediction profile is shown in Figure 79. The Design of Experiments and response surface methodology was not used to create an RSE for the purpose of reducing computational time. Rather, it was used to convert the results into a similar format, that of the prediction profile, of the engineering and campaign level results. In addition, the prediction profile format allows the analyst insight into the nature of the relationships. Prediction Profile The first response, detectability, can be seen to be a function of radar cross section, altitude, and parasitic weight. The performance characteristics are seen to be non-players, as they were not included in the original mapping. Altitude is shown to have a quadratic effect, due to the exponential mapping relationship that was used. Parasitic weight and altitude both show more influence than radar cross section. The reader is reminded that the radar cross section is only a function of the geometric variables used as inputs to the engineering level, and do not reflect a total “true” radar cross section as the original RSE used did not take into account the effect of the aircraft inlets. Maneuverability is shown to be a function of the three performance attributes used in the mapping relationships.

Each of the sizing and alternate performance

attributes were combined. For example, the excess energy term is the combined excess

207

Danielle Soban

energy for both the sizing and the alternate missions, non-dimensionalized, and are both multiplied by the same k-factor, as discussed earlier. The trends show the same level of influence, as would be expected since the k-factors were all set to the same value. The

Maneuverability Detectability

trends are intuitive and correct. 0.8 0.14131 -0.034 1 0.920279 0.8481

590.27 712 13.34 15.50 3187.9 3701 .0426 .0541 651.14 14.42 3444.45 .04835 Excess Energy (ft/s)

Turning Rate (deg/s)

Turning Radius (ft)

RCS (m2)

0

5,000

10,000 0

Altitude (ft)

500

1000

Parasitic Weight (lbs)

Figure 79 – Prediction Profile for Mission Level

Campaign Level Results The results of creating the response surface equations at the campaign level are now examined. There were four sets of cases run for the scenario, each corresponding to one decision node pathway. Each set consisted of 129 model runs in ITEM, with the variables and responses being identical in each set. Because the trends were identical in all four cases, only two will be presented here, representing a case in which the cruise missiles are fired on Day 1, and one in which they were not.

Danielle Soban

208

Prediction Profiles The first set of results is presented in the form of a prediction profile. This set corresponded to the decision node pathway in which cruise missiles were fired on Day 1 and an attack from the south airbase was conducted on Day 2. The prediction profile is shown in Figure 80. The first response of interest is the percentage of Blue-1 aircraft that survived. There is a clear trend that as the aircraft’s detectability decreases, so does its survivability. This is a measure of the effect of decreasing the radar cross section of the aircraft. The same trend is seen for the Blue-2 aircraft, but notice that the effect is not quite as dramatic. Also remember that the difference between the two aircraft is the weapons load each carries. Blue-2 carries a less effective weapon, and, interestingly, the detectability has less of an effect. The same effect is seen for the maneuverability factors, but at a much lesser degree. The ranges for the maneuverability, however, were much tighter than those for the detectability. This was a natural fallout of the mappings: the changes implemented at the engineering level had a more profound effect on detectability than on maneuverability. The next variables to have an effect are the track range variables for SAM sites 1 and 2. The trends are slight, but note how the as the track range for RSAM-2 increases, the survivability of the two aircraft decreases. The slightly opposite trend is shown for the tracking range for RSAM-2. While at first glance this doesn’t seem to make a lot of sense, the response is caused by the interaction between the two variables. Notice in the geometry of the scenario that RSAM-2 is the dominant SAM site of interest when an attack occurs from the south. At some point the

209

Danielle Soban

tracking range for RSAM-1 become large enough for the SAM site to track an incoming raid, and so fleetingly becomes a player. The repair rates show some slight trends, with the only one of significance again that of RSAM-2. As the repair rate increases, the survivability of both aircraft decreases. Finally, the reliability, or probability of hit, of the SAM-1 weapon shows a logical trend: as the reliability increases, the survivability of the aircraft decrease. The total number of SAM weapons fired from RSAM-1 and –2 are next. As the detectability of the aircraft decrease, the aircraft become more susceptible to the defensive systems, and more weapons are fired. however, the trends are relatively flat.

Notice that for maneuverability,

This is because of the modeling of the

maneuverability in ITEM. The maneuverability of the aircraft has no effect on the whether or not the aircraft are detected and fired upon, but is rather a multiplier on the probability that the aircraft is hit after being detected. The repair rates of the two SAM sites have a direct effect on how many weapons are fired: an increase in repair rate allows more weapons to be fired. The damage trends at the airbase are all the same. As the aircraft become more detectable, the defensive systems allow fewer aircraft through, and the damage to the airbase decreases. Maneuverability has a slight effect on damage, again because an increase in aircraft maneuverability allows more aircraft to penetrate and attack the airbase. Increased track range of the dominant SAM site in the scenario, RSAM-2, has the effect of diminishing damage at the airbase, but interestingly, the opposite effect is

Danielle Soban

210

seen for the track range of RSAM-1. Increasing repair rate and probability of hit also have the intended effect of decreasing damage at the airbase. Damage to the SAM sites themselves was tracked and two of those SAM sites presented here. The effect of aircraft detectability and maneuverability is practically nil on the damage to the SAM sites. This is a combined effect of repair rate and aircraft damage from the SAM defensive systems. In other words, when an aircraft is sent directly to attack the SAM site, the variable ranges (such as attack speed) are such that a substantial amount of damage is done to the attacking aircraft. These aircraft then do relatively little damage to the SAM sites, and that damage is repaired fairly quickly. Note the damage to RSAM-4 as a function of the tracking range of RSAM-2. This is a function of geometry. Note in the scenario how the tracking range of RSAM-4 will intercept attacks to the airbase. Increasing the tracking range of RSAM-2 helps “protect” the fourth SAM site. This same effect is mimicked in the reliability of the SAM-2 weapons used at the RSAM-2 site. Finally, there is a direct correlation between repair rate and damage done to the same SAM site, an obvious relationship but one that is a good sanity check. Finally, the prediction profile for the number of cruise missiles fired was not presented. Because this was a Day 1 event and no offensive measures were triggered, the number of cruise missiles fired was completely independent of any of the variables, which all applied to Day 2 events. There were 20 cruise missiles fired during Day 1. While the trends shown in the prediction profile were for the most part intuitive and correct, the error involved in the model fit could have been smaller. Previous studies

Danielle Soban

211

with ITEM have shown that model fits using the response surface methodology are highly dependent on the scenario. The primary reason for this is the nature of discrete events that occur in ITEM. For example, if the tracking range is allowed to become too small, there will be a point in which the raiding aircraft will be outside the sphere of influence of the SAM site, and thus will never become detected. This is a discrete cliff that occurs in a relationship that is trying to be modeled in a continuous way. Although care was taken in the development of the scenario to limit the number of discrete events, some were allowed to remain, chiefly to illustrate a potential drawback of using this sort of metamodeling technique at the campaign level. R2 values for the responses varied from values of 0.984 to 0.999.

212

% Blue AC-1 Survived % Blue AC-2 Survived

90.526 95.65 92.29875 89.882 4.592 2.415308 1.41

Total SAM2 Fired

19.27647

% Runways Remaining

23.91

60.90837

% Shelters Remaining

Total SAM1 Fired

94.7 92.65937

35.26398

9.104 106.5

29.45 59.34

19.02

% Revetments Remaining

42.2 25.02866

% Aircraft in Open Remaining

Damage to Airbase

Responses

SAM’s Fired

Survivability

Danielle Soban

13.69057

13.41 22.84

% Damage to RSAM4 Fire Control % Damage to RSAM4 Radar

23.76074

% Damage to RSAM2 Fire Control

13.5 11.88037

13.15564

% Damage to RSAM2 Radar

Damage to SAM sites

7.485

26.31128

10.875 27

21.75 23.18

4.8988 46.35

9.7976

.016

.507

.998 .016

Detectability Blue A/C 1

.507

.998 .848

Detectability Blue A/C 2

.924

1.0 .848

Maneuverability Blue A/C 1

.924

1.0

Maneuverability Blue A/C 2

50

60

70

Track Range RSAM1 Site (nm)

40

50

60

Track Range RSAM2 Site (nm)

.05

.25

.45

Repair Rate RSAM1 Site

.05

.25

.45 .05

Repair Rate RSAM2 Site

0.25

.45 .45

Repair Rate Red Runway

0.65 Reliability SAM-1

.85 .65

0.8

.95

Reliability SAM-2

Variables Figure 80 – Prediction Profile for Campaign Level, Case 1

The next response surface equation to be examined corresponds to the fourth decision node. This is the set where there are no cruise missiles fired on Day 1, and the second day attack comes from the southeast. This makes both RSAM-1 and RSAM-2

213

Danielle Soban

significant, unlike the dominance of RSAM-2 in the previous case. The prediction profile for this case is shown in Figure 81. In general, the trends follow the same path, with the difference being in the magnitude of the effect. The most notable trend difference is in the tracking ranges of the RSAM-1 and –2 SAM sites.

This scenario switches the emphasis of the primary

defensive site from SAM-2 to a combination of both. Note that at the lower range of the tracking range for RSAM-1 there is actually a slight increase in aircraft survivability. While this is not intuitive, it is a function of the interacting track ranges and reliabilities of the combined defenses of the SAM sites. Unfortunately, in this case the geometry is such that the prediction profile by itself is not enough to promote transparency of the cause and effect relationships being modeled. If the SAM sites were separated so that the attack from the south only encountered one site and the attack from the southeast encountered another distinct site, transparency of those effects would be enhanced. However, this would model a rather unrealistic real-life scenario. In most cases, a combined defensive geometry would be encountered. The other primary difference in trends is the effect of the repair rates. In Case 4, the repair rates have a far less significant effect than in Case 1. This is a function of the damage achieved by the cruise missiles in Day 1 of the campaign. The prediction profiles, in general, provide the analyst with the ability to look at all of the effects of all of the variables on all of the responses at once. This helps provide insight into the interrelationships between the variables. As shown here, however, there are times when the interrelationships lack transparency when only considering the

214

Danielle Soban

prediction profiles. The probabilistic analysis tool presented in the next section adds to

% Blue AC-1 Survived % Blue AC-2 Survived

97.36849

Total SAM1 Fired

SAM’s Fired

100.7 97.25001

12.35029

Total SAM2 Fired

Survivability

the analyst’s insight.

91.762 100.5

91.811 15.91

3.326 15.62 0.471068

% Runways Remaining % Shelters Remaining

124.7 98.46109 36.375 70.57 56.38518

% Revetments Remaining

21.585

% Aircraft in Open Remaining

Damage to Airbase

Responses

-3.305

50.54 40.03353 15.085 27.43 21.65593

% Damage to RSAM1 Fire Control % Damage to RSAM1 Radar

Damage SAM sites

8.305 15.58 11.73991 1.725 31.15 23.47982 3.45

.016

.507 Detectability Blue A/C 1

.998

.016

.507

Detectability Blue A/C 2

.998

.848

.924

Maneuverability Blue A/C 1

1.0

.848

.924

Maneuverability Blue A/C 2

1.0

50

60 Track Range RSAM1 Site (nm)

70

40

50

60

Track Range RSAM2 Site (nm)

.05

.25

Repair Rate RSAM1 Site

.45

.05

.25

Repair Rate RSAM2 Site

.45

.05

0.25 Repair Rate Red Runway

.45

.45

0.65 Reliability SAM-1

.85 .65

0.8

.95

Reliability SAM-2

Variables

Figure 81 – Prediction Profile for Campaign Level, Case 4

Complete Environment Results The response surface equations discussed in the previous section were incorporated into the probabilistic analysis environment created in the previous POSSEM step. This environment can now be used by the analyst in a myriad of ways to explore the complete analysis space. Several examples of using this tool are presented here in order to demonstrate the power of the POSSEM process and its resulting analysis environment.

Danielle Soban

215

Recall that the project goal, established in the development of the conceptual model, was to answer the following question: What is the effect of adding survivability concepts to an aircraft? The probabilistic analysis environment will be used to explore this question. Although data exists for all of the metrics listed in Table 13, for clarity the response investigated will be the survivability rates of both aircraft. The individual effects of varying input parameters at different levels will be conducted. First, the design variables will be allowed to vary probabilistically and the results discussed. Then, the campaign level threat variables will be varied. Finally, all parameters will be varied simultaneously and the results discussed. Effect of Varying Design Variables First, the environment will be used to explore the effects of probabilistically varying the design variables on the campaign level responses. Uniform distributions are placed around each of the design variables. Realize that the distributions could be of any form, allowing the analyst the freedom to weight variables towards specific values based on professional experience, technological data, or simply to explore a “what if” situation. The other variables, mission level k-factors, decision nodes probabilities, and campaign variables, are set at their expected or nominal values. Figure 82 shows the probability density functions for each of the four decision nodes, plus the combined campaign, for the survivability of the Blue-1 aircraft. These graphs show how often a certain value of survivability is achieved given the input

Danielle Soban

216

distributions and calculating the responses through the metamodels. For Cases 1 and 3, the curves are fairly “backloaded”, showing a rise and peak of the survivability around the 93% area. Cases 2 and 4 show a double peak at the beginning and end of the distributions, with a valley in the middle. The important graph, however, is the combined campaign graph. This shows the overall results for Blue-1 survivability when the design variables are probabilistic within a constant threat environment. There are two clear areas within which the survivability of the aircraft falls. The first peak shows a range of survivability from about 91.63% to about 93.73%. This curve shows a very clear peak close to the 93.73% point, with a fairly small amount of variability. The second curve occurs from about 97.33% to about 100%. This graph has two small survivability peaks and a broader variability. Note that there are regions between the two peaks where there are no results. This indicates that, given the scenario and the input assumptions, the aircraft cannot attain those survivability values. Similar trends can be shown in Figure 83 for the Blue-2 aircraft’s survivability. Recalling that the Blue-2 aircraft carries a less effective weapon, the results show that the values of survivability achievable by this aircraft are also less. The analyst can conclude from these results that setting different values of the design variables has an effect in two survivability rate ranges. By only manipulating design variables, the analyst or designer cannot achieve values of survivability that fall within these two groupings. There is less variability in the lower of the two groups, yet high values of survivability are desirable, and there is more variability seen in this grouping.

217

Danielle Soban

Forecast: % Survivability Blue-1 Campaign 1 10,000 Trials

Forecast: % Survivability Blue-1 Campaign 2

Frequency Chart

10,000 Trials

Frequency Chart

.024

244

.020

204

.018

183

.015

153

.012

122

.010

102

.006

61

.005

51

0

.000

.000 92.68

92.93

93.18

93.43

93.69

Forecast: % Survivability Blue-1 Campaign 3 10,000 Trials

0 97.85

98.38

98.91

99.44

99.98

Forecast: % Survivability Blue-1 Campaign 4

Frequency Chart

10,000 Trials

Frequency Chart

26 Outliers

.021

210

.020

198

.016

157.5

.015

148.5

.011

105

.010

99

.005

52.5

.005

49.5

0

.000

.000 91.58

92.04

92.50

92.95

93.41

0 97.33

97.97

98.60

99.24

99.88

Forecast: % Survivability Blue-1 Composite Camp. 10,000 Trials

Frequency Chart

.048

482

.036

361.5

.024

241

.012

120.5

.000

0 91.63

93.73

95.82

97.91

100.01

Figure 82 - Probability Density Functions for Survivability of Blue-1 Aircraft with Probabilistic Design Environment

218

Danielle Soban

Forecast: % Survivability Blue-2 Campaign 1 10,000 Trials

Forecast: % Survivability Blue-2 Campaign 2 10,000 Trials Frequency Chart

Frequency Chart

.021

208

.021

208

.016

156

.016

156

.010

104

.010

104

.005

52

.005

52

0

.000

.000 92.49

92.79

93.09

93.39

93.69

Forecast: % Survivability Blue-2 Campaign 3 10,000 Trials

0 97.42

97.83

98.25

98.66

99.07

Forecast: % Survivability Blue-2 Campaign 4

Frequency Chart

10,000 Trials

Frequency Chart

.020

204

.022

217

.015

153

.016

162.7

.010

102

.011

108.5

.005

51

.005

54.25

0

.000

.000 90.59

91.14

91.69

92.25

92.80

0 96.75

97.27

97.79

98.31

98.83

Forecast: % Survivability Blue-2 Composite Camp. 10,000 Trials

Frequency Chart

.043

434

.033

325.5

.022

217

.011

108.5 0

.000 90.89

92.93

94.97

97.01

99.05

Figure 83 – Probability Density Functions for Survivability of Blue-2 Aircraft with Probabilistic Design Environment

To understand the contributions of the individual decision nodes to the combined campaign result, overlay graphs are used.

These are shown in Figure 84 for the

survivability rates of the two aircraft. For the higher values of survivability, it can be seen that only Cases 2 and 4 will give the desired rate of survivability or higher. These cases correspond to attacks from the southeast airbase, and the use of cruise missiles on Day 1 is irrelevant. This is an interesting result given that the objective of using the cruise missiles was to increase the survivability of the aircraft on the second day strikes. This objective was not realized, and it can be seen that the more important of the

219

Danielle Soban

decisions is clearly from which airbase the attacks should be originated. This illustrates the usefulness of a method such as POSSEM.

Frequency Comparison .147

% Survivability Blue-1 Composite Campaign .110

% Survivability Blue-1 Campaign 1

.073

% Survivability Blue-1 Campaign 2

.037

% Survivability Blue-1 Campaign 3 % Survivability Blue-1 Campaign 4

.000 91.00

93.50

96.00

98.50

101.00

Frequency Comparison .108

% Survivability Blue-2 Composite Campaign

.081

% Survivability Blue-2 Campaign 1

.054

% Survivability Blue-2 Campaign 2 % Survivability Blue-2 Campaign 3

.027

% Survivability Blue-2 Campaign 4 .000 90.00

92.50

95.00

97.50

100.00

Figure 84 - Overlay Charts for Aircraft Survivability with Probabilistic Design Environment

Effect of Varying Threat Variables In order to achieve a robust solution, it was postulated that the threat environment must be allowed to vary probabilistically. To achieve the sole effects of this, the design variables were held constant at their nominal values, and uniform distributions were placed around the threat variables. A Monte Carlo simulation was then performed.

Danielle Soban

220

The probability density functions for the two aircraft survivability rates are shown in Figure 85 and Figure 86. The curves are similar in shape to those generated by the probabilistic design variables, yet the trends are smoother and less abrupt. The minimum survivability rate for Blue-1 aircraft is around 91.20%, compared to 89.74% for Blue-2 aircraft. Again, this is because the second aircraft carries a less effective weapon. The band of unachievable values of survivability is very small in these results, compared to the band generated by holding the threats constant and varying the design parameters. Figure 87 is included to illustrate the difference between the probability density function graph and the cumulative distribution function graph. Both represent the data, and it is more usual to see a CDF in presenting results of this type of probabilistic method.

Closer inspection of the graphs, however, reveals why the PDF is more

appropriate. As noted earlier, the data tends to fall into two survivability groupings. When displayed on a CDF graph, the area between the two groupings is represented by a flat line. One could interpret this flat line on the CDF to mean that those values of survivability are invariant to any changes in the threat environment, yet there is a successful chance of achieving these values. This is an untrue interpretation of the results. As the PDF clearly shows, no combination of variable will allow the analyst to achieve those values. The usefulness, however, of a CDF presentation lies in the ease of interpreting confidence values. For example, looking at the CDF for Blue-1, it can easily be determined that for a 75% confidence level, the analyst can expect a survivability value of at least 93%. This is “bottom line” information that is very useful to the analyst. The lesson is that the presentation of the results is very important, as well as the correct

221

Danielle Soban

interpretation of those results. POSSEM aids analysis by formulating a clear analysis pathway. The correct use of the information generated by POSSEM, including the correct presentation of the results, is the responsibility of the analyst.

Forecast: % Survivability Blue-1 Campaign 2

Forecast: % Survivability Blue-1 Campaign 1 10,000 Trials

10,000 Trials

Frequency Chart

Frequency Chart

.021

213

.020

204

.016

159.7

.015

153

.011

106.5

.010

102

.005

53.25

.005

51

0

.000

.000 92.02

92.64

93.25

93.87

96.79

97.50

98.21

98.93

Forecast: % Survivability Blue-1 Campaign 4

Forecast: % Survivability Blue-1 Campaign 3 10,000 Trials

0 96.08

94.49

10,000 Trials

Frequency Chart

Frequency Chart

219

.022

221

.016

164.2

.017

165.7

.011

109.5

.011

110.5

.005

54.75

.006

55.25

0

.000

.022

.000 91.10

91.60

92.10

92.60

0 94.82

93.11

95.81

96.80

97.79

98.77

Forecast: % Survivability Blue-1 Composite Camp. 10,000 Trials

Frequency Chart

.028

275

.021

206.2

.014

137.5

.007

68.75 0

.000 91.20

93.11

95.01

96.92

98.82

Figure 85 – Probability Density Functions for Survivability of Blue-1 Aircraft with Probabilistic Threat Environment

222

Danielle Soban

Forecast: % Survivability Blue-2 Campaign 1 10,000 Trials

Forecast: % Survivability Blue-2 Campaign 2

Frequency Chart

10,000 Trials

Frequency Chart

.020

195

.023

231

.015

146.2

.017

173.2

.010

97.5

.012

115.5

.005

48.75

.006

57.75

0

.000

.000 91.61

92.52

93.43

94.34

95.26

Forecast: % Survivability Blue-2 Campaign 3 10,000 Trials

0 96.11

96.82

97.54

98.25

98.97

Forecast: % Survivability Blue-2 Campaign 4 10,000 Trials Frequency Chart

Frequency Chart

.020

197

.026

256

.015

147.7

.019

192

.010

98.5

.013

128

.005

49.25

.006

64

0

.000

.000 89.77

90.75

91.73

92.71

93.69

0 95.53

96.28

97.03

97.77

98.52

Forecast: % Survivability Blue-2 Composite Camp. 10,000 Trials

Frequency Chart

.040

399

.030

299.2

.020

199.5

.010

99.75 0

.000 89.74

92.18

94.61

97.05

99.49

Figure 86 – Probability Density Functions for Survivability of Blue-2 Aircraft with Probabilistic Threat Environment

223

Danielle Soban

Probability Density Functions

Cumulative Distribution Functions

Forecast: % Survivability Blue-1 Composite Camp. 10,000 Trials

Forecast: % Survivability Blue-1 Composite Camp.

Frequency Chart

10,000 Trials

.028

275

.021

206.2

.750

.014

137.5

.500

.007

68.75

.250

0

.000

.000 91.20

93.11

95.01

96.92

98.82

.030

299.2

.750

.020

199.5

.500

.010

99.75

.250

0

.000

92.18

94.61

97.05

99.49

93.11

95.01

96.92

98.82

Forecast: % Survivability Blue-2 Composite Camp.

399

89.74

0

10,000 Trials

.040

.000

10000

91.20

Forecast: % Survivability Blue-2 Composite Camp. 10,000 Trials Frequency Chart

Reverse Cumulative

1.000

Reverse Cumulative

1.000

10000

0 89.74

92.18

94.61

97.05

99.49

Figure 87 – Comparison of PDF and CDF for Aircraft Survivability, Composite Campaign

While the shapes of the curves are different, the overlay charts presented in Figure 88 show the same conclusion as that with the probabilistic design variables. Only Cases 2 and 4 allow success for the higher survivability values. Again this points to the relative ineffectiveness of the cruise missiles on Day 1 with respect to aircraft survivability, and highlights the importance of the selection of the attacking airbase. Note, however, that for the lower survivability value grouping, Case 1 has more of an effect on the higher values of survivability than Case 3. This means that, given the selection of the south airbase for the attack, it is more beneficial to use the cruise missiles on Day 1 than not.

224

Danielle Soban

Frequency Comparison .060

% Survivability Blue-1 Composite Campaign .045

% Survivability Blue-1 Campaign 1 .030

% Survivability Blue-1 Campaign 2 % Survivability Blue-1 Campaign 3

.015

% Survivability Blue-1 Campaign 4

.000 91.00

93.25

95.50

97.75

100.00

Frequency Comparison .060

% Survivability Blue-2 Composite Campaign .045

% Survivability Blue-2 Campaign 1 .030

% Survivability Blue-2 Campaign 2 % Survivability Blue-2 Campaign 3

.015

% Survivability Blue-2 Campaign 4

.000 89.00

91.75

94.50

97.25

100.00

Figure 88 – Overlay Charts for Aircraft Survivability with Probabilistic Threat Environment

Effect of Fully Probabilistic Environment

While looking at the effects of holding the design variables and the threat variables constant in turn is interesting and does provide some insight, the true power of POSSEM comes from allowing the analyst to view the data in a fully probabilistic environment. The Monte Carlo simulation is performed once again on the environment. The design variables are allowed to vary with their uniform distributions, as are the threat elements. The probability density distributions are shown in Figure 89 and Figure 90 for the aircraft survivability rates.

225

Danielle Soban

The first thing to notice is that the curves tend to smooth out as the number of probabilistic variables increases. The characteristic gap in the middle of the groupings found in the first two sets of charts is smaller here and not as pronounced. This gap is significantly smaller than either gap produced by looking at the design variable set and the threat variable set independently. It also illustrates the importance of looking at the data in a summative way.

If only the effect of the varying design variables was

examined, the analyst would come to the false idea that a wider spread of survivability rates was relatively immune to the effect of design variable changes.

Adding the

probabilistic threat environment shows that to achieve a robust solution, a smaller range of survivability rates remain relatively insensitive to design variable changes. Figure 91 shows the overlay charts for aircraft survivability within the fully probabilistic environment. Although the trends are the same, and Cases 2 and 4 dominate the higher survivability values, notice the decreasing variability of the responses. Given an attack from the southeast airbase and the desire for the higher survivability values, there is not a clear preference for the use of cruise missiles on Day 1. If this decision were to be considered, the cost of the cruise missiles must be weighed against the slight increase in the aircraft survivability. Figure 92 compares both the CDFs and PDFs for Blue-1. It is clear from these figures how the groupings move from two distinct entities towards each other, filling in the survivability value gap. variability.

In addition, the curves smooth out and decrease in

The characteristic flat spot shown in the CDF of the probabilistic design

variable graph becomes less prominent in the probabilistic threat environment, and is

226

Danielle Soban

practically non-existent in the fully probabilistic environment. Figure 93 is another way of presenting the same data, in the form of an overlay chart. When the curves are superimposed on each other, the relative contributions of the design environment and the threat environment are readily apparent.

Forecast: % Survivability Blue-1 Campaign 2

Forecast: % Survivability Blue-1 Campaign 1 10,000 Trials Frequency Chart

10,000 Trials

Frequency Chart

.022

217

.019

193

.016

162.7

.014

144.7

.011

108.5

.010

96.5

.005

54.25

.005

48.25

0

.000

.000 92.62

93.19

93.76

94.33

Forecast: % Survivability Blue-1 Campaign 3 10,000 Trials

0 96.77

94.91

97.69

98.60

99.52

100.43

Forecast: % Survivability Blue-1 Campaign 4

Frequency Chart

10,000 Trials

Frequency Chart

.021

214

.020

203

.016

160.5

.015

152.2

.011

107

.010

101.5

.005

53.5

.005

50.75

0

.000

.000 91.64

92.33

93.02

93.70

94.39

0 95.78

96.97

98.15

99.34

100.53

Forecast: % Survivability Blue-1 Composite Camp. 10,000 Trials

Frequency Chart

.027

271

.020

203.2

.014

135.5

.007

67.75 0

.000 91.46

93.72

95.98

98.24

100.51

Figure 89 – Probability Density Functions for Survivability of Blue-1 Aircraft with Fully Probabilistic Environment

227

Danielle Soban

Forecast: % Survivability Blue-2 Campaign 1 10,000 Trials

Forecast: % Survivability Blue-2 Campaign 2

Frequency Chart

10,000 Trials

Frequency Chart

.021

208

.021

210

.016

156

.016

157.5

.010

104

.011

105

.005

52

.005

52.5

0

.000

.000 92.22

93.16

94.10

95.04

95.98

97.62

98.49

99.37

100.24

Forecast: % Survivability Blue-2 Campaign 4

Forecast: % Survivability Blue-2 Campaign 3 10,000 Trials

0 96.74

10,000 Trials

Frequency Chart

Frequency Chart

.021

210

.020

196

.016

157.5

.015

147

.011

105

.010

98

.005

52.5

.005

49

0

.000

.000 90.12

91.46

92.80

94.14

0 95.95

95.47

97.00

98.04

99.09

100.13

Forecast: % Survivability Blue-2 Composite Camp. 10,000 Trials

Frequency Chart

.031

311

.023

233.2

.016

155.5

.008

77.75

.000

0 90.23

92.72

95.21

97.70

100.19

Figure 90 – Probability Density Functions for Survivability of Blue-2 Aircraft with Fully Probabilistic Environment

228

Danielle Soban

Frequency Comparison .057

% Survivability Blue-1 Composite Campaign .043

% Survivability Blue-1 Campaign 1

.028

% Survivability Blue-1 Campaign 2

.014

% Survivability Blue-1 Campaign 3 % Survivability Blue-1 Campaign 4

.000 91.00

93.50

96.00

98.50

101.00

Frequency Comparison .039

% Survivability Blue-2 Composite Campaign

.029

% Survivability Blue-2 Campaign 1

.020

% Survivability Blue-2 Campaign 2

.010

% Survivability Blue-2 Campaign 3 % Survivability Blue-2 Campaign 4

.000 90.00

92.75

95.50

98.25

101.00

Figure 91 – Overlay Charts for Aircraft Survivability with Fully Probabilistic Environment

229

Danielle Soban

Forecast: % Survivability Blue-1 Composite Camp.

10,000 Trials

Frequency Chart

.048

Forecast: % Survivability Blue-1 Composite Camp. 10,000 Trials

482

.036

361.5

.750

.024

241

.500

.012

120.5

.250

.000 91.63

93.73

95.82

97.91

0 100.01

Forecast: % Survivability Blue-1 Composite Camp. 10,000 Trials

Frequency Chart

206.2

.750

.014

137.5

.500

.007

68.75

.250

.000

0

.000

96.92

203.2

.750

.014

135.5

.500

.007

67.75

.250

.000

0

.000

98.24

95.01

96.92

Probabilistic Threats (Design Variables Held Constant)

98.82

Reverse Cumulative 10000

Fully Probabilistic Environment

0 91.46

100.51

93.11

1.000

.020

95.98

Reverse Cumulative

Forecast: % Survivability Blue-1 Composite Camp.

271

93.72

Constant)

100.01

0

10,000 Trials

.027

91.46

97.91

10000

91.20

Frequency Chart

95.82

1.000

98.82

Forecast: % Survivability Blue-1 Composite Camp. 10,000 Trials

93.73

Probabilistic Design Variables (Threats Held

Forecast: % Survivability Blue-1 Composite Camp. 10,000 Trials

.021

95.01

0 91.63

275

93.11

10000

.000

.028

91.20

Reverse Cumulative

1.000

93.72

95.98

98.24

100.51

Figure 92 – Comparison of PDFs and CDFs for Blue-1 Survivability, Composite Campaign

Frequency Comparison .042

Probabilistic Design Variables Only

.032

Probabilistic Threat Variables Only .021

All Probabilistic Variables .011

.000 91.00

93.50

96.00

98.50

101.00

Figure 93 – Overlay Chart Comparing Contributions of Design and Threat Environments to Fully Probabilistic Environment for Blue-1 Survivability

Danielle Soban

230

Figure 94 shows the cumulative charts plotted on the same axis. From this it is clear to see how varying different sets of input parameters effects the shape of the overall curve. The lower half of the design curve is shifted to the left of the overall curve. At higher survivability values, however, the curve is shifted to the right of the overall curve. The threat curve is offset at both low and high values to the left of the overall curve. What this means is that at the higher confidence levels (where the analyst prefers to be), looking at the effect of the design variables in isolation results in an overprediction of the survivability rate. For a 70% confidence, this overprediction is on the order of 0.44% for the Blue-1 aircraft. While this may not seem like a lot, returning to the conclusions of Reference 67, a small change in survivability can have huge impacts in overall force effectiveness. The graph also shows that the combined survivability is significantly greater than that achieved simply by considering the probabilistic threat environment. (Remember, however, that an underprediction is preferred in this case to an overprediction). In light of this example, it is clear that the analyst must consider the combined effects of design variable changes as well as a probabilistic threat environment in order to accurately assess aircraft survivability rates.

231

Danielle Soban

100% 90% 80% Confidence

70% 60%

Design Threat All

50% 40% 30% 20% 10% 0% 90.00

92.00

94.00

96.00

98.00

100.00

102.00

Survivability Rate Blue-1 Aircraft (%)

Figure 94 – Cumulative Effects of Varying Sets of Input Data for Survivability Rate of Blue-1

232

Danielle Soban

CHAPTER IX

CONCLUDING REMARKS

This research and consequent dissertation resulted from a simple observation. As aircraft designers and researchers, a lot of attention is paid to the geometric and performance outputs that are the result of a design. Yet how can the results really be a measure of the “goodness” of the aircraft when the aircraft is never placed in the context for which it was designed?

For military aircraft, this context is the warfighting

environment. Therefore, to really understand the effectiveness of an aircraft design, that aircraft must be placed in its intended environment and assessed there. In addition, the links between design changes at the engineering level and the resulting effectiveness parameters at the warfighting level must be made clear to both the analyst and the designer early enough in the design process for those effects to influence the design. A second observation resulted from the first. Much work has been done using probabilistic methods applied to aircraft design. This allows the designer to quantify the effects of the uncertainty that is inherent in the design process, as well as assess the effects of new technologies. Why not extrapolate these methods to the campaign level? This would help the analyst account for the uncertainties inherent in the warfighting environment, and also allow the consideration of a probabilistic threat environment. Research was conducted to examine whether the above observations were justified and if there existed a need to address these issues. The results of this research clearly indicated that there was a need for a military system effectiveness framework.

Danielle Soban

233

This was based on a rapidly changing world economy in which a significant amount of resources were being placed into the research and procurement of warfighting tools. Decision makers needed information that would allow them to allocate resources, make trade studies between system components, and develop appropriate requirements for future system designs. While the need for a system effectiveness framework was clear, it was also established that there was a lack of just such an analysis environment. Part of this lack was a function of differing definitions and connotations of key concepts and words, such as "system effectiveness” and “operations research”, hindering communication.

In addition, those agencies, both government and private, that did

develop some sort of analysis framework often did not publish results and were not open to the idea of sharing methods and results (often due to national security reasons and proprietary interests). This led to a lack of discussion in the open literature about such frameworks and methodologies.

Finally, there is a common misconception that an

analysis tool and a methodology are the same thing. Research showed that modeling tools abound, but there are few, if any, methodologies. A significant result of this research is the clear definition of the term “system effectiveness”, coupled with a clear, cohesive analysis methodology. Concluding that a methodology for the assessment of military system effectiveness would be a worthwhile contribution to the general body of knowledge, it was determined that a system of systems approach would be appropriate. By redefining the system from the aircraft to the overall campaign environment, useful system level

234

Danielle Soban

metrics could be examined, and the observation made in the first paragraph could be addressed. Current probabilistic methods were examined to ascertain their appropriateness for inclusion into the proposed methodology.

Design of experiments methods and

response surface equations methods were studied. The Technology Impact Forecasting method, with its use of k-factors for modeling uncertainties and technological additions, was considered a base concept upon which the methodology would be modeled. Attention then turned to researching military models and their characteristics. The probabilistic methods that were considered compelled the use of modeling tools to generate the appropriate data. The concept of a military modeling continuum was then developed to help justify the necessity of a smooth and uninterrupted analysis path between different levels of the continuum (and thus between the military models). A preliminary investigation was conducted to ensure that it was possible and appropriate to apply the current statistical methods to the theater level. Three primary issues were identified that would need to be addressed by the proposed methodology. The first is the level of detail problem. As shown in the military code continuum, the level of modeling detail in a code progresses from a high level at the engineering level to a low level at the campaign end. The use of model integration (linking together codes) and model abstraction (metamodeling) was proposed to address the level of detail problem. While often discussed in the literature as separate approaches, the research concluded that a combination of the techniques was the appropriate solution (“abstragration”). The second issue was called the human-in-the-loop problem, and

235

Danielle Soban

addressed the common practice of most campaign codes to have an analyst contribute in real time to the analysis of the code while it is running. While this can be a fairly straightforward practice when examining a single scenario, it does not allow the exploration of an entire analysis space in a timely manner. A solution, replacing the human with an embedded rule set, was rejected because of the possibility of generating unrealistic and fallacious results.

A compromise was sought to model the human

interaction, and the research presents the concept of key decision nodes around scenario pathways that are themselves modeled probabilistically. The final issue was that of scenario significance. Simply put, it is fairly straightforward to optimize a design if the threat environment is known. However, if a robust design is desired, the threats must be allowed to vary and the design is selected based in its insensitivity to the threat environment. All of the above information was combined in the proposal of a methodology to assess military system effectiveness. Called POSSEM (PrObabilistic System of Systems Effectiveness Methodology), the framework is a step by step methodology for conducting a system of systems analysis. The first step is the most important: create the conceptual model.

This is the upfront work that helps define the scope of the problem and

determines the tools to be used. The baseline concepts are selected and the scenario is established.

The next step starts to address the human-in-the-loop problem.

It

determines the key decision nodes in the scenario. POSSEM then continues by creating the linked analysis environment using the elements determined by the conceptual model. This is the most difficult step and often the most time-consuming.

Computational

236

Danielle Soban

pathways must be established between all modeling tools, and the inputs and outputs for the individual tools defined. The concepts of model integration and model abstraction are applied where appropriate. Once the analysis pathway has been clearly established, the environment is used in the application of the probabilistic techniques. All of the previous POSSEM steps led up to the development of the fully probabilistic environment.

Metamodels of the modeling codes are created where

appropriate, as determined by the conceptual model. The metamodels and any other mapping relationships are combined into a single computational entity, such as a spreadsheet. Probability distributions are then assigned to any or all of the variables and a Monte Carlo simulation is performed. The real power of POSSEM comes from the use of this environment for analysis. The individual or combined effect of varying input parameters, at any level, is an easy, inexpensive computational task. The analyst can play “what if” games rapidly and gain insight by exploring the entire analysis space without having to return to the original modeling codes. An example of the entire POSSEM process was performed using a survivability test case.

This example was ideal because survivability enhancements are best

incorporated at the preliminary design level (engineering) yet their effect is primarily seen at the campaign level. This allowed the creation of an analysis pathway that linked the engineering level all the way through the military code continuum to the campaign level. Design variables were allowed to vary probabilistically, and a probabilistic threat environment was incorporated. Results in the form of probability distributions and overlay charts were presented.

Danielle Soban

237

In the example, three components of susceptibility were modeled and used; stealth (in the form of radar cross section), maneuverability, and tactics (in the form of a low level attack approach). It should be realized, however, that there are other survivability parameters that were not addressed that would be important to overall aircraft survivability. The insights gained from the example are all dependent on only those variables used and addressed in the modeling process. Thus, it is reinforced to the reader that the example served as a proof of concept of POSSEM, and not as a definitive study of survivability and susceptibility. POSSEM is the response to the need for a military system effectiveness framework. It provides the analyst with a step by step methodology to create and use a fully probabilistic analysis environment. As a decision maker, the analyst can use the resulting data and insight for resource allocation, trade studies between system components, and the development of requirements for future systems. It is observed at this point that although the methodology was formulated to aid in the analysis of military system effectiveness, it can be applied to system effectiveness studies in general. POSSEM is a methodology, and, as such, it provides an analysis framework. The modeling tools used in POSSEM are independent of the process. Given a generic complex system of systems, POSSEM can still be applied. For example, the system of systems could be a cargo transportation system, similar to the United Parcel Service (UPS) or Federal Express (FedEx). Changes made at the engineering level, such as to the delivery trucks, the aircraft used, or even the procedures of the employees, could be propagated up to the “campaign” level, or the total system level. The effectiveness of

238

Danielle Soban

the entire network could be quantified using Measures of Effectiveness such as delivery time, direct operating costs, and return on investment.

The concepts inherent to the

POSSEM framework may be easily applied to this and other complex systems.

Research Questions and Answers

With the development and presentation of POSSEM now complete, a return to the research questions is appropriate. 1) What are the needed elements of a systems effectiveness methodology? The individual steps of POSSEM are the basic elements needed for a system effectiveness methodology.

It was determined that an initial element consisting of

upfront analysis was crucial to the success of the analysis. A linked analysis environment must be created, and used in a fully probabilistic manner. 2) How does one define the system and its components? Traditionally, the system was defined as the aerospace vehicle. In this research, the system is redefined to become the entire warfighting environment, and the aerospace vehicle becomes a component of the new system. This leads to a “system of systems” formulation, which is then incorporated into the POSSEM framework. 3)

How does one link design level metrics to system level Measures of Effectiveness?

239

Danielle Soban

The creation of the linked analysis environment in POSSEM is used to link design level metrics to system level Measures of Effectiveness.

The techniques of model

integration and model abstraction are used in combination, and their use is defined by the conceptual model. 4)

Can the methodology be used to analyze the effect of incorporating survivability concepts at the vehicle level, yet assess them at the theater level? Through the use of the linked analysis environment, survivability concepts can be

added to an aerospace vehicle and their effect propagated through the environment up to the theater level. 5)

How can uncertainties, such as unknown or incomplete data, rapidly changing technologies, and uncertainties in the modeling environment be taken into account in a system effectiveness methodology? Probabilistic techniques are applied to the linked analysis environment. This

introduces uncertainties into the modeling equations, allowing the effects of unknown data, new technologies, and varying threats to be quantified.

240

Danielle Soban

Recommendations

The lessons learned from pursuing a project of this magnitude are substantial. Some of these lessons will be passed on here in the form of recommendations and future work. Perhaps the most difficult part of beginning a research project such of this is the proper identification of the problem that needs solving. Research into this particular issue was clouded in part due to the differing definitions and connotations of key words, such as “system effectiveness”. One lesson learned from this part of the research is that researchers and analysts must take great care that the people they are communicating with share the same meanings of these key words. When possible, operating definitions of these words need to be introduced at the beginning of conversations and all parties agree with and understand the verbiage. This research proposed a definition of system effectiveness, and it is recommended that this definition be used consistently by the researcher and her professional organization(s) in all future work. It is further hoped that this definition find broad acceptance and use in the community at large. The resulting analysis framework POSSEM was shown in the research to be a very powerful tool for tradeoff studies and “what if” analysis. It gives the analyst a clear methodology to follow that results in the creation of a comprehensive and useful analysis tool that allows the exploration of an entire analysis space in a computationally inexpensive way. While just a few examples of the utility of such a tool were presented here, many more areas of use should be explored. For example, the scenario presented herein was a fairly straightforward proof of concept study. The method, however, allows

241

Danielle Soban

analysis of a much more complex scenario. The probabilistic threat environment could be expanded tremendously. One very interesting study would be to allow the threats, such as the SAM sites, to move geographically. This would mimic imperfect knowledge on the Blue side, a major source of uncertainty in the real world. In addition, more complex flight paths for the attackers could be explored, including attacks from behind the installation and around key threats. Attack paths that take the raids through a series of integrated defenses could be explored, and those defenses varied probabilistically. Finally, more advanced studies could be conducted that combine dissimilar attack components, such as tanks and aircraft working in concert. More research could be conducted into the behavior of the probabilistic methods involved in the methodology. For example, it was clear from the test case that increasing the number of probabilistic variables had the effect of “smoothing” the resulting distributions.

It is postulated that when the number of parameters being varied

probabilistically become too high, the resulting curves could approach normal distributions, and the transparency of cause and effect relationships diminished. Research into an optimal maximum number of parameters to be varied could be conducted, and a resulting screening process, similar in concept to that presented in Chapter II, could be presented and added as a part of the POSSEM framework. Several recommendations are made concerning the actual implementation of POSSEM. One of the original goals of the research was to extrapolate existing statistical and probabilistic methods up to the campaign level.

It was found that using the

traditional response surface methodology worked quite well at the lower end of the

Danielle Soban

242

military code continuum. As the continuum points out, codes at the lower end are usually physics-based codes incorporating rather precise mathematical relationships.

Thus,

replacing these mathematical relationships with metamodels often results in quite good modeling fits and a significant reduction in execution time without sacrificing a lot of accuracy. As one progresses through the military code continuum, however, the behavior of the codes starts to become more difficult to model using typical regression techniques. The research consistently showed that model fits at the upper end of the code continuum had more error than those at the lower ends. One major source of this error is the introduction of discrete events that are difficult to model with a continuous curve. For example, one of the probabilistic threat variables was the radar tracking range of the SAM site. Given the range of the variables, it is conceivable that there will occur a radar tracking range that is so small that the attacking aircraft is simply not detected. This is a discrete event: as the tracking range decreases, the detectability of the aircraft decreases, but at some point, it simply goes to zero. Trying to model this event with a continuous curve is difficult and sometimes not even achievable with acceptable error levels. On the other hand, it was interesting to note that the damage to the SAM sites curves had very little error, yet an examination of the data showed definite “clumps” of damage. This was due to the repair rate of the SAM sites combined with the distributed hourly nature of the attacks. In other words, the aircraft would do damage, then a few hours were allowed when the sites would attempt repairs, and then a new round of damage would occur. This led to a cyclical pattern of damage, repair, damage, repair that showed up in the data as damage clumps. A very good curve fit, however, could be made through the center of the

Danielle Soban

243

clumps. The point is one that is made elsewhere in this document: the method and its results cannot stand in isolation as a black box. The analyst needs to understand the behavior of the models and needs to be able to interpret this data correctly. To address these poor model fits at the campaign level, it is recommended that other model fit techniques be explored. The choice of modeling codes needs to be considered very carefully. Although an important step mentioned in the creation of the conceptual model, it was discovered through the research that the code selection impacts the implementation of the process to a much greater degree than originally postulated. Learning curves associated with the use of a previously unfamiliar code can be substantial. In addition, it is not uncommon for the actual capabilities of the code to not meet expectations set forth in the advertisement of that code.

The process through which the user interacts with the code is very

important in the implementation of the method:

if multiple code runs cannot be

performed easily and fairly rapidly, the usefulness of the method diminishes considerably. In the course of this research, it was attempted to bring several different codes into the process, all without success. This resulted in serious research delays. If this research had been a time critical analysis task, the task itself might have been jeopardized. The lesson learned is that the analyst should have considerable knowledge about the capabilities and features of the analysis codes that are considered for use within the POSSEM framework. Finally, it is recommended that more research be conducted into the refinement of the decision node technique used to address the human in the loop problem. While the

Danielle Soban

244

method was used extremely effectively in the example case presented, it must be pointed out that there were only two decision nodes in this scenario, resulting in four analysis pathways to be explored. Response surface equations were created for each of those pathways, which was computationally expensive, but still within the scope of the problem. However, if a more substantial amount of decision nodes were added to the process, the number of potential analysis pathways would increase exponentially. Depending on the modeling tool, creating a metamodel for each pathway may be computationally unrealistic. One way of addressing this issue could be to develop a sort of screening process for the decision nodes, enabling the selection of key nodes only. Another area of possible research could be to use design of experiment techniques to determine a statistically significant subset of analysis pathway combinations to model, rather than creating a metamodel of each and every decision. Finally, the technique of genetic algorithms could be explored for applicability to this particular area.

245

Danielle Soban

REFERENCES

1

Habayeb, A.R., Systems Effectiveness, Pergamon Press, Headington Hill Hall, Oxford, England. 1987.

2

Volpe V. and Schiavone, J.M., “Balancing Design for Survivability”, Grumman Aerospace and Electronics Group AIAA 93-0982, AIAA/AHS/ASEE Aerospace Design Conference, Irvine, CA 16-19 February, 1993.

3

Ball, Robert E., The Fundamentals of Aircraft Combat Analysis and Design, American Institute of Aeronautics and Astronautics, Inc., 1985.

4

Toffler, A. and Toffler, H., War and Anti-war: Survival at the Dawn of the Twenty-First Century, Little, Brown and Company, 1993.

5

Jaiswal, N.K., Military Operations Research Quantative Decision Making, Kluwer Academic Publishers, Boston, 1997.

6

Hughes, W.P. et al, Military Modeling for Decision Making, 3rd Edition, Military Operations Research Society, Inc.,1997.

7

Blanchard, Benjamin S., Logisitics Engineering and Management, 5th Edition, Prentice Hall Inc., 1998.

8

ARINC Res. Corp., Reliability Engineering. Englewood Cliffs, NJ. Prentice Hall 1964.

9

“Definitions of Effectiveness Terms for Reliability, Maintainability, Human Factors, and Safety”, MIL-STD-721B, US Department of Defense. August 1966.

10

Kececioglu, D., Maintainability, Availability, & Operational Readiness Engineering Handbook, Volume 1, Prentice Hall PTR, Upper Saddle River, NJ, 1995.

11

Tillman, F.A., Hwang, C.L., and Kuo, W., “System Effectiveness Models: An Annotated Bibliography”, IEEE Transactions of Reliability, Vol. R-29, No. 4, October 1980.

12

Aerospace Systems Design Laboratory, in-house document, 1999.

Danielle Soban

246

13

Seykowski, R., Research Associate, Potomac Institute for Policy Studies, electronic mail conversation, June 2000.

14

Fabrycky, W.J., Blanchard, B.S., Life-Cycle Cost and Economic Analysis, Prentice-Hall Inc., Englewood Cliffs, New Jersey, 1991.

15

Dixon, J.R., Design Engineering: Inventiveness, Analysis, and Decision Making, McGraw-Hill Book Company, New York, 1966.

16

The American Heritage Dictionary of the English Language, Third Edition, Houghton Mifflin Company, 1996.

17

Myers, R., Khuri, A., and Carter Jr., W., “Response Surface Methodology: 1966-1988”, Technometrics, Vol. 31, No. 2, May 1989.

18

Welch, W., Buck, R., Sacks, J., Wynn, H., “Screening, Predicting, and Computer Experiments”, Technometrics, Vol. 34, No. 1, February 1992.

19

Box, G.E.P., Draper, N.R., Empirical Model Building and Response Surfaces, John Wiley & Sons, New York, 1987.

20

Montegomery, D.C., Design and Analysis of Experiments, 3rd Ed., John Wiley & Sons, New York, 1991.

21

Box, G.E.P., Hunger, W.G., Hunter J.S., Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building, Wiley & Sons, New York, 1978.

22

Bandte, O., “A Probabilistic Multi-Criteria Decisions Making Technique or Conceptual and Preliminary Aerospace Systems Design”, Ph.D. Thesis, Georgia Institute of Technology, October 2000.

23

Bailey, M., Rothenberg, L., “JMP Software: New Features and Enhancements in Version 4 Course Notes”, SAS Institute, Inc., 2000.

24

U.S. Department of the Interior, Bureau of Reclamation Website: Decision Process Guidebook. http://www.usbr.gov/Decision-Process/toolbox/paretopr.htm, 2001.

25

Minitab Website: http://www.minitab.com/, 2001.

26

Engineous Sotfware, “iSIGHT: World Leading Desing Exploration

Danielle Soban

247

Technology”, http://www.engineous.com/isightv5.html, 2001.

27

Mavris, D.N., Soban, D.S., Largent, M.C., “An Application of a Technology Forecasting (TIF) Method to an Uninhabited Combat Aerial Vehicle”, AIAA 1999-01-5633, 1999 World Aviation Conference, 19-21 October, San Francisco, CA.

28

Mavris, D.N., Baker, A.P., Schrage, D.P., “Implementation of a Technology Impact Forecast Technique on a Civil Tiltrotor”, Proceedings of the 55th National Forum of the American Helicopter Society, Montreal, Quebec, Canada, May 25-27, 1999.

29

Kirby, M.R., Mavris, D.N., “Forecasting the Impact of Technology Infusion on Subsonic Transport Affordability”, World Aviation Congress and Exposition, Anaheim, CA, September 28-30, 1998. SAE-985576.

30

Mavris, D.N., Kirby, M.R., “Forecasting Technology Uncertainty in Preliminary Aircraft Design”, World Aviation Congress and Exposition Congress, San Francisco, CA, October 19-21, 1999. SAE Paper 1999-015631.

31

Mavris, D.N., Kirby, M.R., “Technology Identification, Evaluation, and Selection for Commercial Transport Aircraft”. 58th Annual Conference of Society of Allied Weight Engineers, San Jose, CA, 24-26 May, 1999.

32

Mavris, D.N. and DeLaurentis, D.A., “A Probabilistic Approach for Examining Aircraft Concept Feasibility and Viability”, Aircraft Design, Vol. 3, pg 79-101, Pergamon, 2000.

33

Decisioneering, Inc., “Crystal Ball, Computer Program and Users Guide”, Denver, CO, 1993.

34

Contingency Analysis Website: http://www.contingencyanalysis.com/glossarymontecarlosimulation.htm

35

Aerospace Systems Design Laboratory Publications Website: http://www.asdl.gatech.edu/publications/index.html

36

Grier, J.B., Bailey, T.G., Jackson, J.A., “Response Surface Modeling of Campaign Objectives Using Factor Analysis”, Military Operations Research, V4 N2 1999.

Danielle Soban

248

37

Grier, J.B., Bailey, T.G., Jackson, J.A., “Using Response Surface Methodology to Link Force Structure Budgets to Campaign Objectives”, Proceedings of the 1997 Winter Simulation Conference.

38

Soban, D.S., Mavris, D.N., "Formulation of a Methodology for the Probabilistic Assessment of System Effectiveness”, Presented at the AIAA 2000 Missile Sciences Conference, Monterey, CA, November 7-9, 2000.

39

Hillestad, R.J., Bennett, B., Moore, L, “Modeling for Campaign Analysis, Lessons for the Next Generation of Models, Executive Summary”, prepared for the United States air Force, Rand, 1996.

40

Bennett, B.S., Simulation Fundamentals, Prentice Hall International, 1995.

41

VSE User’s Guide, webpage: http://www.cslab.vt.edu/VSE/

42

Ackoff, R.L., Gupta, S.K., Minas, J.S., Scientific Method- Optimizing Applied Research Decisions, John Wiley & Sons, Inc., New York and London, 1962.

43

Caughlin, D., “Verification, Validation, and Accreditation (VV&A) of Models and Simulations Through Reduced Order Metamodels”, Proceedings of the 1995 Winter Simulation Conference.

44

Anderson, L.B., Cushman, J.H., Gropman, A.L., Roske, V.P., “SIMTAX: A Taxonomy for Warfare Simulation”, Military Operations Research Society, Workshop Report, October, 1989.

45

Ziegler, B.P., Praehofer, H., Kim, T.G., Theory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems, 2nd Edition, Academic Press, 2000.

46

Zaerpoor, F., Weber, R.H., “Issues in the Structure and Flow in the Pyramid of Combat Models”, 68th MORSS, US Air Force Academy, 2022 June 2000.

47

“Catalog of Wargaming and Military Simulation Models”, (UNCLAS) (Computer Diskette), Joint Staff, Washington, DC, Force Structure Resource and Assessment Directorate (J-8), 12th edition 7 Feb 92. ADM000108/NAA

Danielle Soban

249

48

SURVIAC, Survivability/Vulnerability Information Analysis Center, http://iac.dtic.mil/surviac/prod_serv/prod_serv.html

49

“SIMTAX, A Taxonomy for Warfare Simulation”, Military Operations Research Society, Workshop Report, 27 October 1989.

50

Enthoven, A.C., Smith, K.W., How Much is Enough? Shaping the Defense Program, 1961-1969, Harper and Row, New York, NY, 1971.

51

B. P. Ziegler, Theory of Modeling and Simulation, John Wiley, New York, 1976.

52

Rozenblit, J.W., Sevinc, S., Zeigler, B.P., “Knowledge-Based Design of LANs Using System Entity Structure Concepts”, Proceedings, Winter Computer Simulation Conference, 1987.

53

Air Force Studies & Analyses Agency's Verification, Validation and Accreditation page for the THUNDER analytical campaign simulation, http://www.s3i.com/AFSAAV&V/campaign.htm, June 2001.

54

Baker, A. P., Mavris, D. N., “Assessing the Simultaneous Impact of Requirements, Vehicle Characteristics and Technologies During Aircraft Design”, 39th Aerospace Sciences Meeting and Exhibit, Reno, NV, January 8-11, 2001, AIAA-01-0533.

55

Hillestad, R.J., Moore, L., “The Theater-Level Campaign Model: A Research Prototype for a New Generation of Combat Analysis Model”, MR-388-AF/A, Rand Corporation, 1996.

56

Sisti, A.F., “Large Scale Battlefield Simulation Using a Multi-Level Model Integration Methodology”, AFRL/IF Webpage: http://www.if.afrl.af.mil/tech/papers/ModSim/LgScale.htm, January, 2001.

57

Sisti, A.F., Farr, S.D., “Model Abstraction Techniques: An Intuitive Overview”, AFRL/IF, Rome, New York. Webpage: http://www.if.afrl.af.mil/tech/papers/ModSim/LgScale.htm, January, 2001.

58

“Integrated Theater Engagement Model (ITEM) Technical Manual”, Version 8.3, Science Applications International Corporation, 10260 Campus Point Drive, San Diego, California 92121 November 5, 1999.

59

Sisti, A.F., Farr, S.D., “Modeling and Simulation Enabling Technologies

Danielle Soban

250

for Military Applications”, AFRL/IF, Rome, New York. Webpage: http://www.if.afrl.af.mil/tech/papers/ModSim/LgScale.htm, January, 2001.

60

“What is Campaign Analysis? An Accreditation Perspective”, webpage: http://www.s31.com/AFSAAV&V/campaign.html, June, 2001.

61

Baker, A., Mavris, D., Schrage, D., Craig, J., “Annual Review of NRTC Task 9.2.1”, in-house document, Aerospace Systems Design Laboratory, Georgia Institute of Technology, November, 1998.

62

Frantz, F.K., “A Taxonomy of Model Abstraction Techniques”, Proceedings of the 1995 Winter Simulation Conference, Washington D.C., 3-6 December, 1995.

63

Caughlin, D., Sisti, A.F., “A Summary of Model Abstraction Techniques”, Proceedings of Enabling Technology for Simulation Science I Conference, Orlando, Florida, 22-24 April, 1997.

64

Brassard, M., The Memory Jogger Plus+ Featuring the Seven Management and Planning Tools, Revised Edition, GOAL/QPC, Methuen, MA 1996.

65

Bronson, R., Naadimuthu, G., Operations Research, 2nd Edition, Schaum’s Outline Series, McGraw Hill, New York, 1982.

66

Adapted from Reference 2.

67

Operation Desert Storm Combat Incident Database Version I, Survivability/Vulnerability Information Analysis Center, March 1991.

68

Sapp, C.N., “Survivability-A Science Whose Time Has Come”, U.S. Naval Institute Proceedings, December 1978.

69

Website of the Joint Technical Coordinating Group on Aircraft Survivability: http://www.aiaa.org/calendar/index.html

70

McCullers, L.A., “Flight Optimization System Release 5 User’s Guide”, NASA Langley Research Center, January, 1988.

71

Roth, B. R., “A Theoretical Treatment of Risk in Modern Propulsion System Design”, Ph.D. Thesis, Georgia Institute of Technology, 2000.

Danielle Soban

251

72

Jane’s Information Group, Jane’s All the World’s Aircraft 1994-1995, Sentinel House, Surrey, United Kingdom, 1995.

73

Standard Aircraft Characteristics F/A-18A Hornet, NAVAIR 00110AF18-1, McDonnell Douglas, October 1984.

74

Kachurak, P., “F/A-18C Substantiating Performance Data with F404-GE402 Engines” Report MDC91B0290.

75

Soban, D.S., Largent, M.C. , “Modeling of F/A-18C Presentation”, inhouse document, Aerospace Systems Design Laboratory, Georgia Institute of Technology. Used in support of Office of Naval Research Contract N00014-00-10132, 1999.

76

Soban, D.S., Largent, M.C. , “Weight Breakdown Presentation”, in-house document, Aerospace Systems Design Laboratory, Georgia Institute of Technology. Used in support of Office of Naval Research Contract N00014-00-10132, 1999.

77

Hines, N.R., “A Probabilistic Methodology for Radar Cross Section Prediction in Conceptual Aircraft Design”, Ph.D. Thesis, Georgia Institute of Technology, June 2001.

78

Mavris, D.N., Soban, D.S., Largent, M.C., “Determination of Notional Aircraft Performance Relationships, Final Report”, submitted to Institute of Defense Analysis (IDA), Contract Reference Number DB/02.500.360.98.48245, May, 1998.

79

“Integrated Theater Engagement Model (ITEM) User’s Manual”, Version 8.3, Science Applications International Corporation, 10260 Campus Point Drive, San Diego, California 92121 November 5, 1999.

80

Knott, E., Shaeffer, J., Tuley, M., Radar Cross Section, Artech House, Norwood, MA 1993.

81

“F-22 Raptor Information Site”, website: http://f22rap.virtualave.net/stealth.html.

82

“Virginia Tech Dynamics and Control Research”, website: http://www.aoe.vt.edu/ACRS.html.

83

Stinton, D., Flying Qualities and Flight Testing of the Airplane, AIAA

Danielle Soban

252

Education Series, American Institute of Aeronautics and Astronautics, Inc., Washington, D.C., 1996. 84

Raymer, D.P., Aircraft Design: A Conceptual Approach 3rd Edition, AIAA Education Series, American Institute of Aeronautics and Astronautics, Inc., Reston, VA, 1999.

85

von Mises, R., Theory of Flight, Dover Publications, New York, 1959.

Danielle Soban

253

VITA Danielle Soban was born in Simi Valley, California in the Spring of 1966. She was raised there and attended Simi Valley High School, where she graduated in 1984. She then attended California Polytechnic State University and graduated with her Bachelor of Science in 1991. She remained at Cal Poly, SLO to pursue her Master of Science in Aeronautical Engineering, which she obtained in 1996. Her Master’s Thesis was entitled “PREDAVOR: Development and Implementation of Software for Rapidly Estimating Stability Derivatives and Handling Qualities”. While pursuing her degree, Ms. Soban interned at NASA Ames Systems Analysis Branch and also completed a research study entitled “Special Uses of the C17- An Investigation of Three Market Niches Applicable to a Commercial Version of the McDonnell Douglas C17 Transport Aircraft” while under contract to McDonnell Douglas. Ms. Soban moved to Atlanta, GA in 1996 to pursue her doctorate in Aeronautical Engineering from the Georgia Institute of Technology. She was employed as a Graduate Research Assistant at the Aerospace Systems Design Laboratory at Georgia Tech, where she worked on a variety of projects, including applying probabilistic methods to an Uninhabited Combat Aerial Vehicle (UCAV), exploring the effects of adding parasitic stealth to a notional aircraft fighter, and the design of a High Speed Civil Transport (HSCT). She currently resides in Mableton, GA with her husband Todd and their cat Maggie.