Jun 2, 2011 - 2011 General Electric Company - All Rights Reserved. Challenges In Uncertainty, Calibration,. Validation and Predictability of Engineering ...
Challenges In Uncertainty, Calibration, Validation and Predictability of Engineering Analysis Models
Dr. Liping Wang GE Global Research Manager, Probabilistics Lab Niskayuna, NY 2011 UQ Workshop University of Minnesota, MN June 02, 2011
Team Members GE Global Research: Arun Subramaniyan, Nataraj Chennimalai, Xingjie Fang, Giridhar Jothiprasad, Martha Gardner, Amit Kale GE Aviation: Don Beeson, Gene Wiggs, and John Nelson © 2011 General Electric Company - All Rights Reserved
1 GE – Aviation & Global Research
Outline • Motivation • History of Development – How far are we along the path? • GE capabilities • Technical Challenges & Solutions • Future Direction • Summary © 2011 General Electric Company - All Rights Reserved
2 GE – Aviation & Global Research
Motivation • Why Model Calibration, Validation, Prediction & Uncertainty Quantification?
Uncertainty Quantification (UQ) Deterministic Simulation
Input Factors (Xs)
Output Factors (Ys) .
X1
• What has been accomplished? Literature Review GE Experience Input Factors (Xs)
• Possible technical solutions and future direction
X 1 X2 Etc.
Y1 .
X2
Y2
Etc.
.
Etc.
Model parameters Model discrepancy
θ Etc. δ Output Factors (Ys)
Uncertainty: • Aleatory – Random and usually modeled by probability distributions. Methods include probability theory and classical statistics • Epistemic – Lack of knowledge. Methods include fuzzy logic or evidence/possibility theory
© 2011 General Electric Company - All Rights Reserved
3 GE – Aviation & Global Research
History of Development – How far are we along the path? •One task at a time (calibration, validation, prediction & uncertainty quantification) - since the 1980s Calibration - data matching, or inverse problems, or parameter estimation - applied to heat transfer, fluid mechanics, solid mechanics, etc. Verification & Validation (V&V) - introduced by DoD, AIAA, ASME, National Labs … Prediction - well-established physics models, calibrated empirical models, and meta-models (Response Surface, Kriging, Gaussian Process, Radial Basis Function, etc)
Uncertainty quantification - Monte Carlo, First Order Second Moment, moments based, polynomial chaos, etc
•All tasks simultaneously - first introduced by Kennedy and O’Hagan in 2001 (Bayesian framework) © 2011 General Electric Company - All Rights Reserved
4 GE – Aviation & Global Research
Kennedy & O’Hagan (2001) •What is Bayesian Statistics?
f (θ )
θ
pdf Prior
f (θ | y )
f ( y | θ ) f (θ )
f (θ | y ) =
∫ f ( y | θ ) f (θ )
L (θ ) =
f ( y |θ )
pdf θ
Given data
Likelihood function
θ
Posterior
•Kennedy & O’Hagan Hybrid Model Formulation: y(xi)= η(xi, θ) + δ(xi) + ε(xi), i=1,2,…,n Observations from the physical system
Output of a simulator, with design inputs (x (x) and calibration parameters (θ)
Discrepancy between the simulator and the physical system
Observation (measurement system) error
• Build & calibrate Gaussian Process (GP) models for both η and δ... Specify beliefs about θ , δ through prior probability distributions Use Markov Chain Monte Carlo (MCMC) to obtain parameter estimates • Similar approaches by Higdon et al. and Liu et al.
Most implementations are for single output © 2011 General Electric Company - All Rights Reserved
5 GE – Aviation & Global Research
Multiple Outputs •Implementation by Los Alamos National Lab (LANL) - Higdon, William et al. Principal Components Analysis (PCA) for dimension reduction & efficiency improvements Correlated outputs η ( x,θ ) = k1w1 ( x,θ ) + ... + k pη wpη ( x,θ ) δ ( x) = d1v1 ( x) + ... + d pδ v pδ ( x ) k1 , k 2 ,..., k pη and d1 , d 2 ,..., d pδ are the principal components
w & v are the GP models for simulator and model correction
More Applicable to Real Problems with LANL Implementation © 2011 General Electric Company - All Rights Reserved
6 GE – Aviation & Global Research
Maximum Likelihood Estimation (MLE) • Alternative approach to Bayesian (Xiong, Chen, Tsui and Apley) • Investigated three possible formulations
y ( x , Θ ) = η ( x ,θ ) + ε y ( x, Θ) = η ( x, θ ) + δ ( x ) + ε
best
y ( x, Θ) = η ( x,θ ( x)) + δ ( x) + ε • Implementation only for single output • Sensitivity analysis prior to MLE optimization to avoid numerical instability
© 2011 General Electric Company - All Rights Reserved
7 GE – Aviation & Global Research
Model Inadequacy Correction & Prediction (No Calibration) • Capture model inadequacy with no model calibration (Wang et al. and Chen et al.)
y ( x ) = η ( x ) + δ ( x) + ε • • • •
Closed form Bayesian posterior Solve GP hyper-parameters using MLE Improved efficiency for high dimensional design space Useful for well established physical models where calibration is not necessary or performed previously
© 2011 General Electric Company - All Rights Reserved
8 GE – Aviation & Global Research
V&V and Model Validation Metrics • Current and desired state of validation metrics
Most Common
(Oberkampf et al. 2004)
• Quantitative Metrics using classical hypothesis testing, Bayes factor, frequentist’s metrics, and area metrics • Quantitative Metrics using Kennedy & O’Hagan Bayesian
Desired State
Framework (Chen et al.) • Preliminary elements of model validation (Paez, Swiler, Mayes, Miller, et al. – 2009 International Modal Analysis Conference, Orlando, FL)
Customers Stakeholders
Analysis Modelers
• Epistemic uncertainty (Paez & Swiler, Paez) © 2011 General Electric Company - All Rights Reserved
Experimentalists
Validation Analysts 9 GE – Aviation & Global Research
GE Capabilities • Deterministic Inverse modeling since 2003 Methods development and implementation Efficient transient data matching using PCA based hybrid metamodels & zoning techniques Partial probabilistic data matching to update standard deviations
• Widely used across GE businesses Transient Analysis and Performance
Heat Transfer and Fluid Systems
Materials Design Acceleration – Material Modeling
23 Xs, 35 TCs simultaneously
GE90 – 78Xs
Others: Transient cycle models 3D transient clearances Undercowl heat transfer Empirical model tuning
© 2011 General Electric Company - All Rights Reserved
GP7000 – 100Xs GEnx – 29 Xs
Analysis time savings >50% Data mismatch reduced by half
10 GE – Aviation & Global Research
GE Capabilities Probabilistic Inverse modeling (Bayesian Hybrid Modeling) since 2006 Built on Kennedy & O’Hagan Bayesian Method and LANL Implementation Efficiency improvement (~2X), flexibility, robustness … Investigated possible formulations y(x) = η (x) y(x) = η (x) + δ (x) y(x) = η (x) + δ (x) + ε (x) y ( x ) = η ( x ,θ ) + ε
y ( x ) = η ( x ,θ ) + δ ( x ) + ε ( x ) y ( x ) = η ( x , θ ( x )) + δ + ε
Kennedy & O’Hagan
y ( x ) = η ( x , θ ( x )) + δ ( x ) + ε y ( x ) = η ( x , θ , δ ( x )) + ε
Key drivers of model inadequacy & insight to possible model improvements Validated with multiple benchmark problems
© 2011 General Electric Company - All Rights Reserved
11 GE – Aviation & Global Research
GE Capabilities Demonstration with challenging engineering problems 1.6
1.7
1.8
1.9
Performance Maps at Speed = 105% Model Discrepancy δ(x) & Updating
1.74
Hybrid Modeling
1.72
105% speed-line
1.7 1.68
PT_ratio
1.5
P T _ra tio
Test data : y(x) Test data uncertainty: ε(x) BRM model: η(x,θ) Design parameters: x Model parameters: θ
1.4
1.66 1.64
1 .3
Mean=0.1797 Std=0.39367
1 .2
1.62
1 .1
Missing vertical parts of high speed lines
1
-1
-0.5
0
0.5
Confidence Bounds
1
1.56
θ (calibrated) 50
55
60
65
70
75
80
85
90
95
Test Hybrid Modeling Calibrated Simulator only
1.58
0 .9 45
1.54 104 104.5
100 105 110 115 120
105
105.5
Corrected flow
106
106.5
107 107.5
108 108.5
109
Corrected flow
2 Speed Lines
3 Speed Lines 105%
1.75
1.8
105% 1. 75 1.7
δ(x)
1.6
Test HM Mean 90% CI
100%
450 Mean=0.036642 Std=0.1968
Test 1.7 1.65
1. 65
1.6
1.6
1.55
1. 55
1.5
1.5
1.45
400
Calibrated Simulator only Hybrid Modeli ng
350
100%
300 250 200
1. 45
Te st Calibrated Simulator only
150 100
95%
50
1.4
Hybrid Modeling 1.4 98
100
102
104
106
108
110
1.35 95
100
105
110
0 -0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Matches Test Data Well for Single & Multiple Speed Lines © 2011 General Electric Company - All Rights Reserved
12 GE – Aviation & Global Research
GE Capabilities Transient AIR Model (Benchmark problem) • • • •
6 calibration parameters 9 Outputs (Ys) 52 time points in each transient (> 3000 DoE points) Only 52 DoE (simulation) points used for Hybrid modeling
Calibrated Model
Calibrated & Discrepancy Adjusted Model y(t)= η(t, θ) + δ(t)
δ(t)
Discrepancy
η(t, θ)
BHM calibrates transient model accurately with very little data © 2011 General Electric Company - All Rights Reserved
13 GE – Aviation & Global Research 13
GE Capabilities Demonstration with challenging engineering problems TOW Uncertainty
IPE Status Match
• Characterizing model discrepancy and uncertainty in severity models • Main effects able to point high thrust severity for improvement of current models
• Improved calibration results by capturing model discrepancy • More confidence in solution with probabilistic estimation
Cycle Deck Calibrated Simulator Discrepancy-Adjusted Discrepancy calibrated simulator discrepancy-adjusted discrepancy 800
10
Posterior Distribution of HPT Efficiency
8
Exhaust Gas Temperature
750 6
θ 350
4 300
2
650 0
600
-2
200
150
100
-4 50
550 -6 500
250 Histogram Counts
ZT49
700
0
0.2 0.4 0.6 0.8
1 0
0.2 0.4 0.6 0.8
1 0
0.2 0.4 0.6 0.8
0
-0.04
-0.03
-0.02
-0.01
0 0.01 DE42DD
0.02
0.03
0.04
-8 1
ZPCN12
Fan Speed
Demonstration with IPE Status Match, TOW Uncertainty Severity Modeling, Cycle Deck Performance … © 2011 General Electric Company - All Rights Reserved
14 GE – Aviation & Global Research
GE Capabilities Extension to Model Validation Combustion Dynamics Acoustic fluctuations (p’) GE90 Fan Blade Row Model
Xs δ(x) θs ε(x)
Flame heat release fluctuations (q’)
δ(x) at untested points
δ(x) at tested points Calibrated Simulator
25
Computation v. Experiment
Freq1
θ (calibrated)
Model Discrepancy δ(x)
30
20
15
0.5
10
5
0 0 -40
-30
-20
-10
0
10
20
30
40
50
60
-0.5
δ(x)
25
-1 20
-1.5
15
10
-2 5
-2.5 0 -50
-40
-30
-20
-10
0
10
20
30
40
50
Enabler for Probabilistic Validation Metrics at Tested & Untested Points © 2011 General Electric Company - All Rights Reserved
15 GE – Aviation & Global Research
GE Capabilities Going-forward Continue improving MCMC speed issues for high dimensional (>100Xs) and solving challenging engineering data matching applications Model Validation for All Engineering Models (3-year program) (Flexible to all models based on data availability, affordable & accurate, account for all types of uncertainty, probabilistic quantitative metrics) 2011 Hot Gas Path Heat Transfer
2012 Combustion Dynamics
2013 All Engineering Models
Acoustic fluctuations (p’) co m b ustor e xit TT PT Vr Vth K ω
Xs
δ(x)
θs
g eom etr y tip cleara n ce, c ore s hift, film h ole d rillin g
ε (x) θ sεε(x)
Xs
Aero pu rg e flo w
Mechanical Flame heat relea se fluctua tions (q’)
δ(x) = y(x) – (η(x) + ε(x))
δ(x) = y(x) – (η(x, θ) + ε(x))
δ(x) = y(x) – (η(x, θ) + ε(x))
Complicated Physics, Unknown Uncertainty, High Dimension … Challenges Remain! © 2011 General Electric Company - All Rights Reserved
16 GE – Aviation & Global Research
Technical Challenges and Solutions • Curse of dimensionality Large number of calibration parameters. MCMC speed issues. Transient data matching PCA/Sensitivity, sparse matrix inversion, adaptive convergence criteria for MCMC, sequential MC or other optimization techniques …
• Source of Uncertainty Epistemic & Aleatory uncertainty. Probability, statistics, fuzzy logic or evidence/possibility theory
• Model inadequacy, uncertainty and characterization Identifiability of parameter calibration and model inadequacy Use as much knowledge as you can on the prior. Carefully choose the range. Uncertainty quantification of experiments. Better understand model inadequacy and key drivers Post-processing to model discrepancy terms. Bayesian model comparison for possible suggestion to model improvement
© 2011 General Electric Company - All Rights Reserved
17 GE – Aviation & Global Research
Technical Challenges and Solutions • Lack of data and extrapolation Limited test & simulation data No overlap between simulation & test data (extrapolation) Scientific knowledge, designer’s belief (prior)
• Confounding Effects High-dimensionality coupled with lack of data Challenging mathematical issues Scientific knowledge, designer’s belief (prior)
• Model validation and quantitative metrics Account for all source of uncertainty (epistemic & aleatory) Flexible for all analysis models - empirical, physics (no calibration) & metamodels … based on data availability (complete, partial and extrapolation) Affordable & accurate Confidence and probabilistic metrics
© 2011 General Electric Company - All Rights Reserved
18 GE – Aviation & Global Research
Technical Challenges and Solutions • Multiple sets of experimental data Mixed datasets – multiple speed lines, multiple engine data … Multiple modes of posterior distributions Numerical and speed issues
• Measurement error and uncertainty Uncertainty quantification for both epistemic and aleatory uncertainty Statistical analysis Outlier detection (sensor failure)
• Multi-physics & multi-fidelity models Direct simulations are prohibitively expensive Vast scale difference among the lowest atomic to the highest macroscopic scale Difficult to establish criteria & strategies when switching design space from one scale to another
© 2011 General Electric Company - All Rights Reserved
19 GE – Aviation & Global Research
Summary • Advancements in model calibration, validation, prediction and uncertainty quantification in the past three decades • Much research and many publications from industry, government, academia • GE has been very active • Technical challenges remain and are being worked
© 2011 General Electric Company - All Rights Reserved
20 GE – Aviation & Global Research
