Right-of-way rules. Flight crew. Airline/aircraft efficiency considerations. (i.e. Fuel, wind aloft, etc). Desired resolution action. FMS. Broadcast via. ADS-B out.
1
Safety Analysis Tool for Automated Airspace Concepts (SafeATAC) 31st Digital Avionics Systems Conference Williamsburg, VA October 2012
NASA Ames Tech Monitors: David Thipphavong Dr. Heinz Erzberger
Metron Aviation, Inc: Arash Yousefi, Ph.D., Richard Xie, Ph.D., Shubh Krishna, GMU: John Shortle, Ph.D., Yimin Zhang
Motivation Various NextGen automation concepts are proposed to provide increased levels of airspace capacity by • • • •
reducing human workload increasing airspace capacity enhancing safety in higher levels of traffic density Different concepts propose – varying levels of automation – different system architectures (i.e. centralized vs. distributed) – different roles/responsibilities for human operators (ATC vs. pilot automation)
Two types of SA system architecture: • Centralized: Automated Airspace Concept •
i.e. Advanced Airspace Concept (AAC)
• Distributed: Automated Flight Rules •
i.e. Automated Flight Rules (AFR)
System level safety-capacity analysis is needed to guide the decision makers in selecting capacity enhancing concepts that maintain target level of safety 2
Objective For selected NextGen concepts: • Establish safety-capacity trade off relations • Perform phase transition analysis to understand the conditions under which the system should transition from maintaining high capacity to maintaining safety • Sensitivity analysis to identify critical points of failure and required redundancies
Safety-driven methods for concept design & refinement • Required reliability measures of system components? (e.g. required Mean Time to Failure for onboard automated separation software) • Optimal system architecture? (e.g. required redundancies and safety nets)
3
Basic Definitions in our Safety Modeling System = Automated Separation Assurance (SA) System System mission= Provide Separation Services System failure = System fails to prevent a conflict Initiating event = There exist at least two aircraft with conflicting 4DTs Conflict = Los of Separation (LOS), Near Mid-Air Collisions (NMAC), or Mid-Air Collisions (MAC)
Describe System
Identify Hazards
Analyze Risk
Assess Risk
Treat Risk
Advanced Airspace Concept (AAC) • • •
Transmit aircraft state (position, velocity, flight plan) to central (ground ) host Detect conflicts and generate resolutions at the central host Transmit conflict resolutions from the central host to the pilot
Reference: Erzberger, H., 2009, “Separation Assurance in the Future Air Traffic System,” 2009 ENRI International Workshop on ATM/CNS, Mar. 5–6.
Automated Flight Rules (AFR) ConOps • • •
Transmit aircraft state vector (position, velocity, intent) and flight plan via ADS-B Detect conflicts and generate resolution options onboard for AFR flights The AFR pilot relies on onboard automation for CD&R RTA at TRACON boundary if any issued by the ATS via CPDLC All aircraft (IFR, AFR) state vectors (position, speed) via ADS-B in All aircraft planned trajectories via ADS-B in
Airline/aircraft efficiency considerations (i.e. Fuel, wind aloft, etc)
CR: horizontal or Onboard conflict vertical maneuvers or resolver speed adjustment.
Ownship Right-of-way aircraft state rules vector Reference: Wing, D., and M. Ballin, 2004, “Pilot in command: A feasibility assessment of autonomous flight management operations,” 24th International Congress of the Aeronautical Sciences.
Flight crew Desired resolution action
FMS
Broadcast via ADS-B out
Calculation of Conflict Probability Pr{Collision} = Pr{Aircraft on course for conflict}
Pr{Collision | Aircraft on course for conflict}
x Part I P • Assumes no conflict resolution • Establishes safety-capacity relationship Simulation runs (e.g. FACET)
Part II
• Analyze effect of intervention by AAC and AFR
SafeATAC
Simulation runs to estimate conflict rate Conflict types considered here: Conflict Type Loss of separation (LOS) Critical loss of separation (CLOS) Near mid-air collision (NMAC) Mid-air collision (MAC)
• •
Lateral Vertica l 5NM 1000 ft 1.1NM
100ft
500ft
100ft
100ft
30ft
Conflict rates provided through NAS simulators (e.g. FACET) Flight trajectories simulated using FACET with 1.5 x traffic schedule • Direct routes flown, no conflict resolution • Two cases: Great circle routes, airway routes • 50 simulation replications, varying departure times • Conflict detection for sectors in Chicago center.
Reference:
8
Belle, A., J. Shortle, A. Yousefi, and R. Xie, 2012, “Estimation of Potential Conflict Rates as a function of Sector Loading,” 5th International Conference on Research in Air Transportation (ICRAT 2012), Berkeley, CA, May.
Part II - SafeATAC
Model system-level events that lead to CD&R failure or success
Model the functional events that lead to events in DET
Model how components work together to perform a function
Model how component performance changes over the time
Reference: Yousefi, A. and R. Xie, 2011, “Safety-Capacity Trade-off and Phase Transition Analysis of Automated Separation Assurance Concepts,” 30th Digital Avionics Systems Conference (DASC), Seattle, WA, October 16-20. Xie, R. and A. Yousefi, 2012, “Safety Analysis of Primary and Secondary Conflicts for Automated Airspace Concepts,” 12th AIAA Aviation Technology, Integration, and Operations (ATIO) Conference, Indianapolis, IN, September.
Dynamic Event Tree (DET) for AFR Strategic Timeframe If failed Tactical Timeframe If failed TCAS Timeframe If failed
Pilot See-and-Avoid Timeframe If failed Conflict realized
Fault Tree Layer Dynamic Event Tree
Fault Trees (an example)
Reliability Block Diagram
Airborne ASAS to detect a potential conflict
Source of conflicting flight’s positions
Source of ownship’s positions
Component Modeling
functional
functional
functional
Pww
Pww
Pwn
Pwn
Pnw
Pnw
Pnn
Pnn
Non-functional
functional
Non-functional
Non-functional
Non-functional
User’s input Hardware: Pwn is a constant
Pilot: Pwn or Pnw is a function of time
Pnw
Software: Pwn or Pnw is a function of time
Time
Safety Analysis Tool for Automated Airspace Concepts (SafeATAC) Workflow Event Tree Reference:
Fault Tree
Reliability Block Diagram
Safety-Capacity Analysis
Yousefi, A., R. Xie, and S. Krishna, 2012, Safety Analysis Tool for Automated Airspace Concepts (SafeATAC) User Manual, Metron Aviation Technical Report to NASA Ames Research Center, Rep # NNH09ZEA001N, July.
SafeATAC Full Factorial Analysis (Sensitivity of Overall System Safety to Individual Component Reliability Metrics)
Component Reliability Data
Conflict Rate
1. Revise System Design 2. Revise Component Reliability
SafeATAC – Specifying Input Files
DET Structure CD&R Stages
Fault Tree
Reliability Block Diagram
Component Reliability Data
Conflict Rate
SafeATAC Interactive Visualization
Click Event 1.2.2
Dynamic Event Tree plot ot
Fault Tree diagram
Event Description Even
Click to see fault tree
Click to reveal Reliability Block Diagram
16
Reliability Block Diagram
Automatic Report Generation Click on the menu item to start analysis and generate report
17 17
Automatic Report Generation HTML Report Sample View 1-
18
Automatic Report Generation Report Sample View 2-
19
Experimental Results
Experimental results for alternative system architectures of AAC and AFR 1. Safety-Capacity analysis and Phase Transition for different ConOps 2. Safety net analysis 3. Impact of strategic and tactical CD&R on overall system safety within AAC and AFR 4. Full Factorial Analysis to Identify safety critical components 5. Transitional phases in implementation of AAC 6. Secondary conflict within distributed separation assurance
20
S-C Tradeoff for the Baseline 1.40E-09
Prob(NMAC / Flight)
1.20E-09
1.00E-09
8.00E-10
6.00E-10
4.00E-10
2.00E-10
0.00E+00 0
10
20
30
40
50
60
Flight Count per 15 minutes per sector
70
80
Safety Performance of AFR Baseline Case • Pr {Conflict Not Resolved | Trajectory Conflict} = 3.77 x 10-8 • Decomposed into each timeframe
CD&R Timeframe
CD&R failure probability
Strategic ASAS
3.24㽢10-5
Tactical ASAS
5.48㽢10-2
TCAS
2.26㽢10-2
Pilot See-andAvoid
0.938
99.9% of conflict will be solved by Strategic ASAS!
Pilot See-and-Avoid is the least effective means for CD&R
System Characteristics Phase Transition Performance Change in Transponder 2.00E-09
Baseline TRN Failure = 1e-3
1.80E-09
TRN Failure = 1e-5
Prob(NMAC / Flight)
1.60E-09 1.40E-09 1.20E-09 1.00E-09 8.00E-10 6.00E-10 4.00E-10 2.00E-10 0.00E+00 0
10
20
30
40
50
Flight Count per 15 minutes per sector
60
70
80
Safety Sensitivity Curve to Effectiveness of ASAS Strategic CD Software 1.00E-04
Prob(NMAC Not Resolved)
1.00E-06 1.00E-08 1.00E-10 1.00E-12 1.00E-14 1.00E-16 1.00E-18 1.00E-20 1.00E-22 0.00
0.10
0.20
0.30
0.40
0.50
0.60
ASAS Strategic Detection Software Transition Probability
0.70
Impact of TSAFE or AR on overall System Safety within AAC • Baseline safety level is 1.98 x 10-8, given a trajectory conflict • Experiment 1: • TSAFE is removed from the baseline AAC model, •
by setting the initial failure probability as 1, and probability of transitioning from nonworking to working as 0
• Safety level becomes 3.6 x 10-8 • Risk almost doubled (same order of magnitude)
• Experiment 2: • AR is removed from the baseline AAC model • Safety level becomes 2.22 x 10-7 • Risk is 12 times worse than the baseline
• Other factors that can be changed : • Length of strategic and tactical time frames • Time step size of each time frame • Performance of the components in each CD&R system 25
Impact of ASAS on overall System Safety within AFR • Baseline safety level is 1.46 x 10-7, given a trajectory conflict • Experiment 1: • ASAS Tactical Resolution is removed from the baseline AAC model, •
by setting the initial failure probability as 1, and probability of transitioning from nonworking to working as 0
• Safety level becomes 3.94 x 10-7 • Risk almost tripled
• Experiment 2: • ASAS Strategic Resolution is removed from the baseline AAC model • Safety level becomes 1.32 x 10-6 • Risk is 9 times worse than the baseline
• Factors in play include: • Length of strategic and tactical time frames • Time step size of each time frame • Performance of the components in each CD&R system
26
FFA to Identify Safety-Critical Components +
• Sparsity-of-effects principle: a system is usually dominated by single-factor effects or two-factor interactions • Based on that principle, FFA is conducted using a subset of all factors to analyze system behavior
Vary one parameter at a time
SSR
• Fractional Factorial Analysis (FFA)
+ – –
• In one FFA example, following components are chosen
Full Factorial x6
+
x8
x7
SSR
27
+
Transponder
x5
• ASAS Detection Software, • ASAS Strategic Resolution Software, • ASAS Intent-based Tactical Resolution Software, • ASAS State-based Tactical Resolution Software, and • Mode A/C/S Transponder
–
Baseline
x4
x2 –
x1 –
Transponder
x3 +
–
+
FFA Results
Increasing reliability of Strategic Resolution Software by 10x increases system safety by about 8x
Increasing reliability of other components by 10x (and associated component pairings) have little effect on overall safety
28
Safety Analysis of Transitional Phases in Implementation of AAC • A transitional phase bridges the current operation and a future ConOps • Current operations are • Controller in charge of CD&R in strategic timeframe • In tactical timeframe: URET detects, controller resolves • Safety level: 1.45 x 10-7, given a trajectory conflict
• An example of transitional ConOps is • Controller still in charge in strategic CD&R • In tactical timeframe: TSAFE detects and resolves • Safety level: 1.49 x 10-7, given a trajectory conflict
• AAC Baseline safety level: 1.98 x 10-8 • AR improves safety by more than 8 folds
29
Secondary Conflict on Distributed Separation Assurance • • •
Possible failure of intent sharing causes sidewalk conflicts in AFR (secondary conflict) A hypothetical AFR model contains perfect intent sharing between flights Perfect intent sharing will eliminate the sidewalk conflict, and increases the probability of resolving the primary conflict
Probabilities
Baseline (with secondary conflict)
AFR w/o secondary conflict
CD&R Success without secondary conflict Secondary Conflict
0.9999997514
0.99999985385
1.024e-7
0
30
Summary • Developed tool in Matlab to facilitate safety analyses (SafeATAC) • SafeATAC capabilities: – User-friendly interactive model implementation (i.e. DET, fault trees, reliability block diagrams & component reliability modeling) – Safety-capacity trade off relations – Phase transition analysis when transitioning from maintaining high capacity to maintaining safety – Sensitivity analysis to identify critical points of failure and required redundancies
• Sample Experiments •
Impact of strategic and tactical CD&R on overall system safety for both AAC and AFR –
• •
Full Factorial Analysis to Identify safety critical components Transitional phases in implementation of AAC –
•
Removing strategic CD&R has much more effect on decreasing safety levels
Adding TSAFE improves safety by more than 8-fold
Secondary conflict within distributed separation assurance –
Risk was small 31
Future Work • Modeling component reliability as a function of traffic load and complexity (i.e. ADS-B jamming as a function of traffic load) • Modeling risk factors that distinguish centralized and distributed systems (e.g. roll-call interrogation capability for centralized) • Verification and Validation (V&V) of SafeATAC Models • Safety analysis of UAS integration in the NAS
32
Questions? 33
