Testing Expected Shortfall C. Acerbi and B. Szekely MSCI Inc. Workshop on systemic risk and regulatory market risk measures Pullach, Germany, June 2014
Testing Expected Shortfall C. Acerbi and B. Szekely MSCI Inc.
Workshop on systemic risk and regulatory market risk measures
Pullach, Germany, June 2014
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
1 / 59
Outline
1
Motivation and goals
2
Testing setting Basel VaR backtest Three tests for ES. Plus one
3
Results
4
Conclusions Post Scriptum
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
2 / 59
1
Motivation and goals
2
Testing setting Basel VaR backtest Three tests for ES. Plus one
3
Results
4
Conclusions Post Scriptum
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
3 / 59
Motivation
in the VaR/ES debate, backtesting has always been the main problem with ES. See for instance Yamai and Yoshiba (01) last obstacle for the adoption of ES in Basel N, finally occurred in 2013 but model testing still based on VaR
rich literature on VaR backtesting: Basel I (96), Kupiec (95), Christoffersen (98), Berkowitz (00), Engle and Manganelli (04), among others few works on ES backtesting: noticeably Kerkhof and Melenberg (04) Angelidis and Degiannakis (06)
Why is it difficult to test ES? Fundamental reasons? Practical aspects? Power of the test? Model risk?
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
4 / 59
Motivation
in the VaR/ES debate, backtesting has always been the main problem with ES. See for instance Yamai and Yoshiba (01) last obstacle for the adoption of ES in Basel N, finally occurred in 2013 but model testing still based on VaR
rich literature on VaR backtesting: Basel I (96), Kupiec (95), Christoffersen (98), Berkowitz (00), Engle and Manganelli (04), among others few works on ES backtesting: noticeably Kerkhof and Melenberg (04) Angelidis and Degiannakis (06)
Why is it difficult to test ES? Fundamental reasons? Practical aspects? Power of the test? Model risk?
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
4 / 59
Confusion
The nice thing about VaR is it’s more or less transparently back-testable. You know what you’re getting. With ES it’s all clouded up with assumptions about distribution and arbitrary choices. When have you breached it? What exactly are you testing? When you go into the tail you are never quite sure... RISK Magazine, last week
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
5 / 59
The drama of non–elicitability of ES Gneiting (11): VaR is elicitable, ES is not This negative result may challenge the use of the ES functional as a predictive measure of risk, and may provide a partial explanation for the lack of literature on the evaluation of ES forecasts, as opposed to quantile or VaR forecasts
elicitability is a subtle concept:
xˆ = arg minx E[S(x, Y )]
What most people understood ES is not backtestable, at all a magnum champagne bottle gift for the VaR nostalgic panic followed ES cannot be back-tested because it fails to satisfy elicitability ... If you held a gun to my head and said: ‘We have to decide by the end of the day if Basel 3.5 should move to ES, or do we stick with VaR’, I would say: ‘Stick with VaR’ Paul Embrechts, Imperial College, 2013 Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
6 / 59
The drama of non–elicitability of ES Gneiting (11): VaR is elicitable, ES is not This negative result may challenge the use of the ES functional as a predictive measure of risk, and may provide a partial explanation for the lack of literature on the evaluation of ES forecasts, as opposed to quantile or VaR forecasts
elicitability is a subtle concept:
xˆ = arg minx E[S(x, Y )]
What most people understood ES is not backtestable, at all a magnum champagne bottle gift for the VaR nostalgic panic followed ES cannot be back-tested because it fails to satisfy elicitability ... If you held a gun to my head and said: ‘We have to decide by the end of the day if Basel 3.5 should move to ES, or do we stick with VaR’, I would say: ‘Stick with VaR’ Paul Embrechts, Imperial College, 2013 Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
6 / 59
The drama of non–elicitability of ES Gneiting (11): VaR is elicitable, ES is not This negative result may challenge the use of the ES functional as a predictive measure of risk, and may provide a partial explanation for the lack of literature on the evaluation of ES forecasts, as opposed to quantile or VaR forecasts
elicitability is a subtle concept:
xˆ = arg minx E[S(x, Y )]
What most people understood ES is not backtestable, at all a magnum champagne bottle gift for the VaR nostalgic panic followed ES cannot be back-tested because it fails to satisfy elicitability ... If you held a gun to my head and said: ‘We have to decide by the end of the day if Basel 3.5 should move to ES, or do we stick with VaR’, I would say: ‘Stick with VaR’ Paul Embrechts, Imperial College, 2013 Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
6 / 59
The drama of non–elicitability of ES Gneiting (11): VaR is elicitable, ES is not This negative result may challenge the use of the ES functional as a predictive measure of risk, and may provide a partial explanation for the lack of literature on the evaluation of ES forecasts, as opposed to quantile or VaR forecasts
elicitability is a subtle concept:
xˆ = arg minx E[S(x, Y )]
What most people understood ES is not backtestable, at all a magnum champagne bottle gift for the VaR nostalgic panic followed ES cannot be back-tested because it fails to satisfy elicitability ... If you held a gun to my head and said: ‘We have to decide by the end of the day if Basel 3.5 should move to ES, or do we stick with VaR’, I would say: ‘Stick with VaR’ Paul Embrechts, Imperial College, 2013 Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
6 / 59
The drama of non–elicitability of ES Gneiting (11): VaR is elicitable, ES is not This negative result may challenge the use of the ES functional as a predictive measure of risk, and may provide a partial explanation for the lack of literature on the evaluation of ES forecasts, as opposed to quantile or VaR forecasts
elicitability is a subtle concept:
xˆ = arg minx E[S(x, Y )]
What most people understood ES is not backtestable, at all a magnum champagne bottle gift for the VaR nostalgic panic followed ES cannot be back-tested because it fails to satisfy elicitability ... If you held a gun to my head and said: ‘We have to decide by the end of the day if Basel 3.5 should move to ES, or do we stick with VaR’, I would say: ‘Stick with VaR’ Paul Embrechts, Imperial College, 2013 Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
6 / 59
The drama of non–elicitability of ES Gneiting (11): VaR is elicitable, ES is not This negative result may challenge the use of the ES functional as a predictive measure of risk, and may provide a partial explanation for the lack of literature on the evaluation of ES forecasts, as opposed to quantile or VaR forecasts
elicitability is a subtle concept:
xˆ = arg minx E[S(x, Y )]
What most people understood ES is not backtestable, at all a magnum champagne bottle gift for the VaR nostalgic panic followed ES cannot be back-tested because it fails to satisfy elicitability ... If you held a gun to my head and said: ‘We have to decide by the end of the day if Basel 3.5 should move to ES, or do we stick with VaR’, I would say: ‘Stick with VaR’ certainly not a VaR fanatic! → Paul Embrechts, Imperial College, 2013 Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
6 / 59
Examples of elicitable statistics the mean is elicitable xˆ = arg min EX [S(m, X )] m
S(m, x) = (X − m)2
a α–quantile is elicitable qα = arg min EX [S(q, X )] q
S(q, x) = (x − q)(α − (x − q < 0))
when α = 1/2 we retrieve the median S(µ, x) = |x − µ|
M = arg min EX [S(µ, X )] µ
there is no scoring function S that elicits ES ES = arg min EX [S(c, X )] c
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
S(c, x) does not exist
June 2014
7 / 59
Something is not quite right
if elicitable means backtestable isn’t it a bit strange that banks have always backtested VaR but never by exploiting its elicitability? even standard deviation is not elicitable? Kerkhof and Melenberg, back in (04), had found that ...contrary to common belief, ES is not harder to backtest than VaR if we adjust the level of ES. Furthermore, the power of the test for ES is considerably higher than that of VaR.
as a matter of fact, others reacted quite differently ES is not elicitable. So, what?
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
Dirk Tasche
June 2014
8 / 59
Something is not quite right
if elicitable means backtestable isn’t it a bit strange that banks have always backtested VaR but never by exploiting its elicitability? even standard deviation is not elicitable? Kerkhof and Melenberg, back in (04), had found that ...contrary to common belief, ES is not harder to backtest than VaR if we adjust the level of ES. Furthermore, the power of the test for ES is considerably higher than that of VaR.
as a matter of fact, others reacted quite differently ES is not elicitable. So, what?
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
Dirk Tasche
June 2014
8 / 59
Something is not quite right
if elicitable means backtestable isn’t it a bit strange that banks have always backtested VaR but never by exploiting its elicitability? even standard deviation is not elicitable? Kerkhof and Melenberg, back in (04), had found that ...contrary to common belief, ES is not harder to backtest than VaR if we adjust the level of ES. Furthermore, the power of the test for ES is considerably higher than that of VaR.
as a matter of fact, others reacted quite differently ES is not elicitable. So, what?
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
Dirk Tasche
June 2014
8 / 59
Something is not quite right
if elicitable means backtestable isn’t it a bit strange that banks have always backtested VaR but never by exploiting its elicitability? even standard deviation is not elicitable? Kerkhof and Melenberg, back in (04), had found that ...contrary to common belief, ES is not harder to backtest than VaR if we adjust the level of ES. Furthermore, the power of the test for ES is considerably higher than that of VaR.
as a matter of fact, others reacted quite differently ES is not elicitable. So, what?
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
Dirk Tasche
June 2014
8 / 59
Something is not quite right
if elicitable means backtestable isn’t it a bit strange that banks have always backtested VaR but never by exploiting its elicitability? even standard deviation is not elicitable? Kerkhof and Melenberg, back in (04), had found that ...contrary to common belief, ES is not harder to backtest than VaR if we adjust the level of ES. Furthermore, the power of the test for ES is considerably higher than that of VaR.
as a matter of fact, others reacted quite differently ES is not elicitable. So, what?
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
Dirk Tasche
June 2014
8 / 59
Something is not quite right
if elicitable means backtestable isn’t it a bit strange that banks have always backtested VaR but never by exploiting its elicitability? even standard deviation is not elicitable? Kerkhof and Melenberg, back in (04), had found that ...contrary to common belief, ES is not harder to backtest than VaR if we adjust the level of ES. Furthermore, the power of the test for ES is considerably higher than that of VaR.
as a matter of fact, others reacted quite differently ES is not elicitable. So, what?
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
Dirk Tasche
June 2014
8 / 59
1
Motivation and goals
2
Testing setting Basel VaR backtest Three tests for ES. Plus one
3
Results
4
Conclusions Post Scriptum
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
9 / 59
Setting we look at ES backtesting from a regulatory point of view profit–loss: independent (but not i.i.d.) Xt ∼ Ft , the real distributions, t = 1, . . . , T (= 250) Pt predicted (model) distributions VaR and ES (with Basel confidence levels) VaRβ = −P −1 (β) ESα =
−1 α
α = 1%
α
Z
P −1 (q) dq
α = 2.5%
0
we assume Pt continuous and strictly monotonic (just for simplicity, inessential here). Then ESα = −E[X |X + VaRα < 0] the assumption can be easily removed at the cost of heavier notation Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
10 / 59
ES estimators standard estimator of ESα for N i.i.d. draws Xi ∼ P [Nα] X α,N 1 c Xi:N + (Nα − [Nα]) X[Nα+1:N] ES (X ) = − Nα i
coherent ∀N, α, consistent, asymptotically normal, known variance generalizes the idea of ‘average of the Nα worst cases’ to Nα ∈ /N but biased. It always underestimates risk for finite N. No unbiased estimator known for unknown P conditional estimator; assuming VaRα is known exactly f ES
α,N
PN
i=1 (X ) = − P N
Xi 1Xi +VaRα ES, which is always true in non–crazy cases this means that you can set up a contest among models that forecast jointly VaR and ES we could call it joint elicitability of VaR and ES Lambert, Pennock, Shoham (08) call this property 2–elicitability and prove it for variance and mean
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
56 / 59
By the way, ES is elicitable well, not exactly but consider the scoring function S(v , e, x) = αe2 /2−ev (α−(x +v < 0))+(ex −2(v 2 −x 2 ))(x +v < 0)+2αv 2 then you have {VaR, ES} = arg min EF [S(v , e, Y )] v ,e
the only condition is that 4VaR > ES, which is always true in non–crazy cases this means that you can set up a contest among models that forecast jointly VaR and ES we could call it joint elicitability of VaR and ES Lambert, Pennock, Shoham (08) call this property 2–elicitability and prove it for variance and mean
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
56 / 59
By the way, ES is elicitable well, not exactly but consider the scoring function S(v , e, x) = αe2 /2−ev (α−(x +v < 0))+(ex −2(v 2 −x 2 ))(x +v < 0)+2αv 2 then you have {VaR, ES} = arg min EF [S(v , e, Y )] v ,e
the only condition is that 4VaR > ES, which is always true in non–crazy cases this means that you can set up a contest among models that forecast jointly VaR and ES we could call it joint elicitability of VaR and ES Lambert, Pennock, Shoham (08) call this property 2–elicitability and prove it for variance and mean
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
56 / 59
By the way, ES is elicitable well, not exactly but consider the scoring function S(v , e, x) = αe2 /2−ev (α−(x +v < 0))+(ex −2(v 2 −x 2 ))(x +v < 0)+2αv 2 then you have {VaR, ES} = arg min EF [S(v , e, Y )] v ,e
the only condition is that 4VaR > ES, which is always true in non–crazy cases this means that you can set up a contest among models that forecast jointly VaR and ES we could call it joint elicitability of VaR and ES Lambert, Pennock, Shoham (08) call this property 2–elicitability and prove it for variance and mean
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
56 / 59
By the way, ES is elicitable well, not exactly but consider the scoring function S(v , e, x) = αe2 /2−ev (α−(x +v < 0))+(ex −2(v 2 −x 2 ))(x +v < 0)+2αv 2 then you have {VaR, ES} = arg min EF [S(v , e, Y )] v ,e
the only condition is that 4VaR > ES, which is always true in non–crazy cases this means that you can set up a contest among models that forecast jointly VaR and ES we could call it joint elicitability of VaR and ES Lambert, Pennock, Shoham (08) call this property 2–elicitability and prove it for variance and mean
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
56 / 59
General score function
most general scoring function, for all W S W (v , e, x) = αe2 /2 + W αv 2 /2 − αev + e(v + x) + W (x 2 − v 2 )/2 (x + v < 0) with ES < W VaR
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
57 / 59
A scoring function of VaR and ES
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
58 / 59
Thanks!
Carlo Acerbi and Balazs Szekely
Testing Expected Shortfall
June 2014
59 / 59
