Assume that Ohm's law is not known, i.e. we know only that the current depends on voltage and resistance, current = f( voltage, resistance) = f( E, R) where f is ...
Statistical modeling, Simulation and Time series Leif Nilsson, Oleg Seleznjev Department of Mathematics and Mathematical Statistics • Statistical models • Simulation (Bootstrap) • Time series (Correlated data)
1
Statistical modeling In many cases it is very hard to mathematically model a situation without observations.
• How do humans react on a specific drug. • How do the sizes of piston rings behave in a production process. • How do processing conditions, e.g. reaction time, reaction temperature, and type of catalyst, affect the yield of a chemical reaction.
• When the government plans for the future they need good prediction models for e.g. the Gross National Product (GNP). By making observations and using statistical methods we can handle these kind of situations. 2
Statistical modeling Sometimes we know the model apart from one or more parameters. These parameter can then be estimated from data. Natural questions are
• what accuracy and precision do the estimates have? • how accurate will a prediction be? These questions can also be handled with statistical methods.
3
Statistical modeling When we have a mathematical model for a situation it can be of importance to verify how well this model agree with the reality. This can be done by comparing observations with the model (or simulated observations from the model). Here can statistical methods be useful. Sometimes it is more or less impossible to make observations, e.g. the amounts of large insurance losses, extreme floods, … The extreme value theory can be used to model such situations.
4
Statistical modeling Consider the flow of current in a thin copper wire. A mechanistic model is Ohm’s law: current = voltage/resistance (I=E/R) If we measure this more than once, perhaps at different days, the observed current could differ slightly because of small changes or variations in factors that are not completely controlled (temperature, measuring technique, …). A more realistic (statistical) model for the observed current might be I = E/R + ε = μ + ε, where the term ε includes the effect of all of unmodeled sources of variability that effects this system. 5
Statistical modeling Assume that Ohm’s law is not known, i.e. we know only that the current depends on voltage and resistance, current = f( voltage, resistance) = f( E, R) where f is an unknown function. A workable model could be developed from a first-order Taylor series expansion: current = β0 + β1E + β2R. If we take into account the unmodeled sources of variability that effect this system, we get an empirical model current = β0 + β1E + β2R + ε.
6
Statistical modeling What is a reasonable model for ε. Often we believe that there exists a large number of small effects acting additively and independently on the measurements, i.e. ε is a sum of these small effects. The Central Limit Theorem (CLT) tells us that the distribution of a sum of a large number of independent variables is approximately normal. In that case, ε is assumed to be normally distributed with expected value 0 and standard deviation σ, ε ∈N(0,σ 2 ) . This means that the measurements are also normally distributed with standard deviation σ.
7
Statistical modeling Sometimes the small effects are acting multiplicatively and independently on the measurements, i.e. ε is a product of these small effects. In that case, it is reasonable to believe that the measurements (and ε) are log-normally distributed, that is log(measurement) and (log(ε)) is normally distributed. Depending on the situation, one has to investigate what distribution the observations follow. There exists a large amount of distributions: binomial, Poisson, uniform, beta, Cauchy, gamma, lognormal, normal, Weibull,….
8
Statistical modeling Let Y represent the resistance of a resistor marked 3.9 kiloohms with a tolerance of 5%. Assume that we measure 40 similar resistors. Statistical model: Yj = μ + εj,
j=1,2,…,40.
where μ is the theoretical resistance of these resistors (not y necessarily 3.9). Before the measures are taken Yi is said to be a random variable. μ is estimated by the sample mean⎯y = 3.8801 What is εj representing?
9
Statistical modeling The measurements are varying. A theoretical standard deviation σ exists and is estimated with the sample standard deviation: s = 0.0030 n
s = ∑ ( yi − y ) 2 / (n − 1) 2
i =1
If data is normally distributed, the 95% of the measurements should roughly lie within the interval ⎯y ± 1.96*s = 3.8801 ± 0.0059 = ( 3.8742, 3.8860).
10
Statistical modeling We can test the hypothesis H0: μ =3.9 (alternative H1: μ ≠ 3.9) by comparing⎯y with μ = 3.9. The estimate of the standard deviation of ⎯y is s/√40 = 0.00047 Since the data consist of 40 measurements, the sample mean is (approximately) normally distributed with mean μ and standard deviation σ/√40 (CLT). A 95% confidence interval is calculated as ⎯y ± 1.96*s /√40 = 3.8801 ± 0.0009359 = (3.8792, 3.8810). We can reject H0: μ =3.9. How do we interpret the interval?
11
Statistical modeling We can also test the hypothesis H0: μ =3.9 (H1: μ ≠ 3.9) by computing p-values. If the calculated p-value is smaller than the pre-chosen level of significance α (usually α=0.05, correspond to 95% confidence interval), The null-hypothesis H0 is rejected. Let us calculate the p-value and confidence interval with Matlab (with Statistics toolbox) Data (resistance): 3.883 3.883 3.880 3.875 3.881 3.877 3.880 3.877 3.877 3.881 3.876 3.881 3.881 3.875 3.875 3.880 3.878 3.875 3.884 3.880 3.877 3.883 3.882 3.886 3.877 3.880 3.883 3.884 3.880 3.879 3.880 3.881 3.882 3.879 3.885 3.886 3.879 3.881 3.880 3.881 12
Statistical modeling [h,p,ci] = ztest(resistance,3.9,std(resistance),0.05) h=1 hist(resistance) 8 p=0 ci = 3.8792 3.8810 7 6 5 4 3 2 1 0 3.874
3.876
3.878
3.88
3.882
3.884
3.886
3.888
13
Bootstrap Simulation can be used to investigate the distribution of an estimator. In order to do this we need random sources. In Matlab we can use the command rand(n) to generate n uniformly distributed random numbers at the interval (0,1). For other distributions Matlab has functions, e.g. normrnd(mu,sigma) for normal distribution binornd(N,P) for binomial distribution
14
Bootstrap The term “Bootstrap” (Efron, 1979 ) is an allusion to the expression "pulling oneself up by one's bootstraps” (doing the impossible). Traditional statistical methods are either
based upon the large sample theory or
prior beliefs or
inferences based upon small samples.
In most situations, bootstrap usually produces more accurate and reliable results than the traditional methods. Bootstrap is especially useful when complicated estimators are to be analyze. 15
Bootstrap Non-parametric bootstrap: Assume that we are interested in an estimate of the observed sample (y1, y2, …, yn): e.g ⎯y, s, median, s /⎯y, …. Construct artificial samples (y1* , y*2 , ..., y*n ) by resampling with replacement from the observed sample. These artificial samples are regarded as new observed samples. Calculate the estimate with the resampled data: e.g ⎯y*, s*, … By doing this many times (say B times) we receive B estimates
( y1* , y*2 , ..., y*B ) that will behave approximately as the true estimator will behave. Let us illustrate this in Matlab. 16
Bootstrap Data (resistance): 3.883 3.883 3.880 3.875 3.881 3.877 3.880 3.877 3.877 3.881 3.876 3.881 3.881 3.875 3.875 3.880 3.878 3.875 3.884 3.880 3.877 3.883 3.882 3.886 3.877 3.880 3.883 3.884 3.880 3.879 3.880 3.881 3.882 3.879 3.885 3.886 3.879 3.881 3.880 3.881
CLT tells us that the sample mean will be approximately normally distributed with mean μ and standard deviation σ/√40. B = 10000; for i = 1:B sample=randsample(resistance,40,true); boot.copy(i)=mean(sample); end With replacement
or shorter boot.copy = bootstrp(B, 'mean', resistance);
17
Bootstrap mean(boot.copy) ans = 3.8801
hist(boot.copy,40) 900 800
std(boot.copy) ans = 0.00046793
700 600 500 400 300 200 100 0 3.878
3.8785
3.879
3.8795
3.88
3.8805
3.881
3.8815
⎯y = 3.8801, s/√40 = 0.00047, conf.int: ( 3.8791, 3.8811).
3.882
3.8825
18
Bootstrap Confidence intervals can be estimated by the 2.5th and 97.5th percentiles. quantile(boot.copy,[0.025 0.975]) ans = 3.8792 3.8810 We can do all this with one command: bootci(B,{@mean, resistance}) ans = 3.8791 3.8810 Since the sample should reflect the true distribution of the estimate we can see that 3.9 is not a reasonable value, hence we can reject H0: μ =3.9. ⎯y = 3.8801, s/√40 = 0.00047, conf.int: ( 3.8792, 3.8810).
19
Bootstrap Parametric bootstrap: * * * (y , y , ..., y Construct artificial samples 1 2 n ) by simulating samples
of size n from an estimated distribution, e.g. Normal distribution with mean 3.8801 and standard deviation 0.0030. Calculate the estimate with the simulated data many times. We receive B estimates ( y1* , y*2 , ..., y*B ) that behave approximately as the true estimator. Warning! For large enough n and B.
20
Bootstrap B=10000 mu=mean(resistance); s=std(resistance); for i = 1:B sample= normrnd(mu,s,[1,40]); boot.copy(i)=mean(sample); end
700
600
500
400
300
mean(boot.copy) ans = 3.8801 std(boot.copy) ans = 0.00047556 quantile(boot.copy,[0.025 0.975]) ans = 3.8792 3.8811
200
100
0 3.878
3.8785
3.879
3.8795
3.88
3.8805
3.881
3.8815
3.882
21
3.8825
Bootstraping the median There exist no explicit formulas for the distribution of the median of a sample (the sampling distribution of the median). For large sample and normally distributed observations: σmedian = 1.253 σ/√n (approx.) We can use simulation to look at the sample distribution of the median. mu=0; stdev=1; B=10000; n=20; for i = 1:B sample= normrnd(mu,stdev,[1,n]); sim.median(i)=median(sample); end 22
Bootstraping the median 4
2
median based on 100000 replicates
x 10
1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 -1.5
-1
-0.5
0
0.5
1
1.5
[mean(sim.median), std(sim.median)] ans = -0.0022 0.2709 1.253*1/√20 = 0.2802
23
Bootstraping the median Let us try bootstrap. First, simulate a sample of size n=20. original.data = normrnd(mu,stdev,[1,n]); median(original.data) ans = 0.2578 boot.copy = bootstrp(B, 'median', original.data); [mean(boot.copy), std(boot.copy)] ans = 0.2854 0.2556 4
2
median based on 100000 bootstrap replicates
x 10
1.8
We can estimate the variation but not the true median (0).
1.6 1.4 1.2 1
sum(boot.copy R)/B pval = 0.0270 Notice that this is a p-value for a one sided test
34
Correlated data hist(perm,20) 1200
1000
800
600
400
200
0 -1
-0.8
-0.6
-0.4
-0.2
0
R = 0.6145
0.2
0.4
0.6
0.8
1
35
References
A. M. Zoubir, D. R. Iskander, Bootstrap techniques for signal processing. Cambridge Un Press, 2004 A. M. Zoubir, D. R. Iskander, Bootstrap Matlab Toolbox http://www.csp.curtin.edu.au/downloads/bootstrap_toolbox.html
Time series Definition: A time series is a sequence of random variables (r.v.) ordered in time {Yt , t ∈ T}. A realization of a time series occurs when measurements are taken at successive times (often uniformly spaced) ,t = k Δ t , k = 0, ± 1, ... . Examples: • The annual population of lynx. • Daily stock prices • Brain activity measured with electroencephalography (EEG) • The annual crop yield of sugar beets • Daily maximum and minimum temperatures Note that the r.v:s are dependent and the order is of highest importance. 1
Time series Time Series Plot of Lynx 80 Time Series/Lynx
70
60
50
40
30
20
10
0
1850
1860
1870
1880
1890
1900
1910
1920
1930
Time (seconds)
Tstool in Matlab
2
Objectives of Time series analysis Modeling paradigm 1. Set up a family of probability models to represent data 2. Estimate parameters of a model 3. Check a model for goodness-of-fit Applications: 1. Description 2. Relations between two or more time series 3. Prediction of future values 4. Hypothesis testing 5. Process monitoring 3
Time series A time series {Yt, t=1,2,…} can contain • a trend Tt (slowly varying) • a long time cyclic influence Zt (e.g. business cycle) • a short time cyclic influence St (e.g. seasonal effect) • an error term (random) εt. Yt = Tt + Zt + St + εt, t = 1,2,…
In the Lynx data, it seems to be a cyclic behavior and no trend?!
4
Stationary models We will start to consider time series without any trend or cyclic behavior. Let {Yt, t=1,2,…} be a time series with E(Yt 2) < ∞. The mean function of {Yt, t=1,2,…} is defined as μY(t) = E(Yt). and the covariance function of {Yt, t=1,2,…} is defined as γY(r, s) = Cov(Yr , Ys) = E[(Yr - μY(r)) (Ys - μY(s))] for all integers r and s.
Stationary models {Yt, t=1,2,…} is (weakly) stationary if (i) μY(t) is independent of t. (ii) γY(t, t + h) is independent of t for each h (γY(h)=γY(t, t + h)). Let {Yt, t=1,2,…} be a stationary time series. The autocovariance function (ACVF) of {Yt} is γY(h) = Cov(Yt+h , Yt). The autocorrelation function (ACF) of {Yt} is ρY(h) = γY(h) / γY(0) = Cor(Yt+h , Yt). γY(0) = Cov(Yt , Yt) = V(Yt). The correlation/covariance between Yt and Ys depends only on the distance h=t-s, which is called the lag.
Stationary models Estimation of the ACF from an observed data {y1, y2, …, yn}. We assume that data comes from a stationary time series. The sample mean: μˆ Y (t ) = y =
1 n y ∑ i =1 i n
The sample autocovariance function:
γˆY (h) =
1 n −|h| ( yt +|h| − y )( yt − y ) ∑ t =1 n
The sample autocorrelation function:
ρˆY (h) =
γˆY (h) γˆY (0)
−n< h< n
−n< h< n
Examples of models “IID noise”: {Yt, t=1,2,…} is a sequence of independent r.v’s that are identically distributed with mean E(Yt) = 0 and variance V(Yt) = σ2 < ∞. “White noise”: {Yt, t=1,2,…} is a sequence of uncorrelated r.v’s with mean E(Yt) = 0 and variance V(Yt) = σ2 < ∞. IID noise ⇒ White noise IID noise ⇔ White noise + normal distribution μY(t) = E(Yt) = 0 γY(0) = Cov(Yt , Yt) = σ2 . γY(h) = Cov(Yt+h , Yt) = 0 , h > 0. Hence, stationary!
Examples of models “Random walk”: Yt = X1 + X2 + … + Xt = = Yt-1 + Xt, where {Xt, t=1,2,…} is IID noise. μY(t) = E(Yt) = 0 V(Yt) = Cov(Yt, Yt) = t σ2. Cov(Yt+h , Yt) = t σ2. Corr(Yt+h , Yt) = √t /√(t+h). Depend on t, not stationary!
t=1, 2, …,
Examples of models Multiple Time Series 100 ...random walk 1 50
0
-50
-100 20 ...random walk 2 0 -20 -40 -60 -80
0
500
1000
1500
2000
2500
3000
3500
4000
4500
Time (seconds)
Two realizations of random walk where Xt is N(0,1)-distributed
Examples of models MA(1) (first-order moving average) process: Yt = Xt + θ Xt-1, t = 0, ±1, ±2, …, where {Xt, t=1,2,…} is white noise and θ is a real-valued constant. μY(t) = E(Yt) = 0 γY(0) =
Cov(Yt, Yt) = (1 + θ2)σ2.
If h = ± 1 then γY(h) = Cov(θ Xt-1, Xt) = θσ2
(otherwise 0),
Hence, stationary! ρY(h) = γY(h) / γY(0) = θ/(1 + θ2) (otherwise 0).
Examples of models Multiple Time Series 10 Time Series/MA 1 5
0
-5
-10 10 Time Series/MA 2 5
0
-5
-10
0
50
100
150
200
250
300
350
400
450
Time (seconds)
Two MA(1), with θ = 2 and θ = -2, respectively.
Examples of models ACF of Ma(1) with parameter 2
ACF of Ma(1) with parameter -2
1.2
1
1
Conf.int. for ρY(h)
0.8
0.5 0.6
0.4
0 0.2
0
-0.2
0
1
2
3
4
5
6
7
8
9
10
-0.5
0
1
2
3
4
5
6
7
8
9
10
To calculate and plot the ACF ρY(h) we used the Matlab-script acf.m that can be found at the homepage (KTH): http://www.math.kth.se/matstat/gru/5b1545/homeworks_2004.html The script acvf.m calculate the autocovariansfunction. Notice the confidence limits in the plot. ACF for MA(1), with θ = 2 and θ = -2, respectively.
Examples of models Regression Ma 1: Yn = -0.042 + 0.397*Yn-1 6
4
2
0
-2
-4
-6
-8 -8
-6
-4
-2
0
2
4
Regression: yn = -0.042 + 0.397*yn-1
6
Examples of models Regression Ma 1: Yn = -0.0817 - 0.0993*Yn-2 6
4
2
Yn
0
-2
-4
-6
-8 -8
-6
-4
-2
0
2
4
Yn-1
Regression: yn = -0.0817 - 0.0993*yn-2
6
Examples of models Estimation of MA(1): Maximum likelihood estimation is a common technique to estimate parameters, here we make it intuitively. By using that ρY(1) = θ/(1 + θ2) and that ρˆ Y (1) = - 0.4246 ρˆ Y (1) = 0.3954 we can find estimates of θ for the two series. We find the estimates 2.04 and -1.8 respectively, which is quite close to the true values 2 and -2. Furthermore, γY(0) = (1 + θ2)σ2 and γˆ Y (0) = 5.7470 γˆ Y (0) = 4.9015 give estimates of the variance, 1.11 and 1.15. Close to the true variance 1.
Examples of models AR(1) (first-order auto-regressive) process: Yt = φ Yt-1 + Xt, t = 0, ±1, ±2, …, where {Xt, t=1,2,…} is white noise and -1 < φ < 1.
Yt =
j φ ∑ j= 0 X t − j ∞
μY(t) = 0 γY(0) = σ2 /(1 - φ2) γY(h) = σ2φ|h| /(1 - φ2) Hence, stationary! ρY(h) = γY(h) / γY(0) = φ|h|.
Examples of models Multiple Time Series 10 Time Series/AR 1 5
0
-5
-10 10 Time Series/AR 2 5
0
-5
-10
0
50
100
150
200
250
300
350
400
450
Time (seconds)
Two AR(1), with φ = 0.9 and φ = -0.9, respectively.
Examples of models ACF of AR(1) with parameter -0.9
ACF of AR(1) with parameter 0.9
1
1.2
0.8 1
0.6 0.8
0.4
0.2 0.6
0 0.4
-0.2
-0.4
0.2
-0.6 0
-0.8
-0.2
0
5
10
15
20
25
30
35
40
45
50
-1
0
5
10
15
20
25
30
35
40
Result from the Matlab-script acf: 1.0000 0.9125 0.8311 0.7561 0.6884 0.6312 0.5716 1.0000 -0.8986 0.8175 -0.7437 0.6805 -0.6163 0.5444 ACF for AR(1), with φ = 0.9 and φ = -0.9, respectively.
45
50
Examples of models Estimation of AR(1): Here we can use that ρY(h) = φ|h| for different h. If we choose h=1 ρˆ Y (1) = 0.9125 ρˆ Y (1) = - 0.8986 we have two estimates of φ (one for each time series).
The true ones: 0.9 and -0.9. Furthermore, γY(h) = σ2φ|h| /(1 - φ2) γˆ Y (0) = 5.5540 γˆ Y (0) = 5.4844 give estimates of the variance: 0.93 and 1.06. Close to the true variance 1.
Examples of models ARMA(1,1): A combination af a AR(1) and a MA(1). Yt = φ Yt-1 + Xt + θXt-1, t = 0, ±1, ±2, …, where {Xt, t=1,2,…} is white noise φ + θ ≠0. These examples of models can be extended to “higher dimensions”: AR(p), MA(q) and ARMA(p,q).
Models with trend Assume that we also have a trend in the time series: Xt = Tt + Yt, t = 1,2,…,n where {Yt, t=1,2,…,n} is a stationary time series. We need to estimate and eliminate the trend. 1.Finite moving average filer:
Tˆt =
q 1 X t+ j , ∑ j= -q 2q + 1
q < t < n−q
2.Polynomial fitting (Linear regression):
Tˆt = βˆ 0 + βˆ 1 t + " + βˆ k t k
Models with trend Linear trend: 1+0.1*t, stationary error: AR(1) with φ = 0.9. n=100; mu=0; stdev=1; for i = 1:2*n; Y(i+1)= 0.9*Y(i) + normrnd(mu,stdev,1); end; for i = 1:n; X(i)= 1+0.1*i+Y(n+i); end; Time series with trend, Solid line = True trend 14
12
10
X
8
6
4
2
0
-2
0
10
20
30
40
50 t
60
70
80
90
100
Models with trend
Smoothma from homepage (KTH): http://www.math.kth.se/matstat/gr u/5b1545/homeworks_2004.html
Moving average, q=2, 5, and 10: q=5; Xsm=smoothma(X,ones(q,1)/q)';
Time series with trend, Moving Average with q=2, 10, 20 14
12
10
X
8
6
4
2
0
-2
0
10
20
30
40
50 t
60
70
80
90
100
Solid (red): True trend, Dashed: q=2, Dotted: q=10, Dash-dot: q=20.
Models with trend Moving average, q=2, 5, and 10: Trend elimination: Ysm =X - Xsm; AC=acf(Ysm); AC([1:5]) ACF from the ‘trend eliminated’ series: ans = 1.0000 -0.5513 0.0866 -0.0643 -0.0253 (q=2) ans = 1.0000 0.5563 0.2690 0.0272 -0.0883 (q=10) ans = 1.0000 0.7292 0.5481 0.3847 0.2838 (q=20) ACF from the ‘true’ AR(1)-series: ans = 1.0000 0.8177 0.6600 0.4950 0.3776 True = 1.0000
0.9
0.81
0.729
0.6561
Models with trend Linear regression, k=1, 5 and 10: for i=0:k, A(:,i+1)=((1:n).^i)'; end beta=regress(X',A) Xreg=beta'*A'; Time series with trend, Regression with k=1, 5, 10 14
12
10
X
8
6
4
2
0
-2
0
10
20
30
40
50 t
60
70
80
90
100
Solid (red) = True trend, Dashed: k=1, Dotted: k=3, Dash-dot: k=10
Models with trend Linear regression, k=1, 5 and 10 : Trend elimination: Yreg =X - Xreg; AC=acf(Yreg); AC([1:5]) ACF from the ‘trend eliminated’ series: ans = 1.0000 0.7686 0.6097 0.4679 0.3820 (k=1) ans = 1.0000 0.6623 0.4377 0.2616 0.1837 (k=5) ans = 1.0000 0.8226 0.7068 0.6016 0.5348 (k=10) ACF from the ‘true’ AR(1)-series: ans = 1.0000 0.8931 0.7761 0.6959 0.6264 True = 1.0000
0.9
0.81
0.729
0.6561
Models with both trend and seasonality Assume that we also have a trend in the time series: Xt = Tt + St + Yt, t = 1,2,…,n where {Yt, t=1,2,…,n} is a stationary, St+d = St and
∑
d j=1
St+ j = 0 .
The period d is assumed to be known. w 1.Eliminate the trend with a moving average filter: If d is even use mt = (0.5Xt-q + Xt-q+1 + … + Xt+q-1 + 0.5Xt+q)/d, q < t ≤ n-q else use ordinary moving average filter. k
2.Estimate the seasonal component: For each k=1,2,.. d, calculate averages w k of d X k + jd – m k + jd , q < k + jd ≤ n − q ; sˆk = wk − d −1 ∑ wi , k = 1,..., d
{
}
1
Models with both trend and seasonality Let us add seasonality to our previous time series. S=repmat([-5,5,2,0,-2],1,20); Model for Season 6
4
S
2
0
-2
-4
-6
0
10
Here is d=5.
20
30
40
50 t
60
70
80
90
100
Models with both trend and seasonality The resulting time series. Time series with trend and season 20
15
X
10
5
0
-5
-10
0
10
20
30
40
Hard to see the seasonality!
50 t
60
70
80
90
100
Models with both trend and seasonality The resulting time series. Time series with trend and season 20
15
X
10
5
0
-5
-10
0
10
20
30
40
50 t
60
Now it is easier to see the seasonality!
70
80
90
100
Models with both trend and seasonality One can also regard the ACF in order to identify the length of the seasonality period. ACF of the the time series 1
0.8
0.6
0.4
0.2
0
-0.2
-0.4
0
10
20
30
40
50
60
70
80
90
100
Strong dependency between observations on distances 5,10,15,….!
Models with both trend and seasonality 1. The estimation of the trend with moving average (q=5). m=smoothma(X,ones(q)/q)'; The m-estimate of the Trend 12 10
8
m
6
4 2
0 -2
-4
0
10
20
30
40
50 t
60
70
80
90
100
Models with both trend and seasonality 2. The time series when the trend is eliminated. W=X-m; The trend eliminated time series, (X-m) 10 8 6 4
X-m
2 0 -2 -4 -6 -8 -10
0
10
20
30
40
50 t
60
70
We can more or less identify the true seasonality.
80
90
100
Models with both trend and seasonality 3. For each seasonal time point k, take the averages over the periods, k = 1,2,…, d. We get an estimate of the seasonality. for i=1:4; Shat(i)=mean(W(mod((1:100),5)==i)); end; Shat(q) = mean(W(mod((1:100),5)==0)); Shat = -4.7014 5.4425 2.3385 0.4023 -1.7619 True =
-5
5
2
0
-2
Models with both trend and seasonality 4. Withdraw the seasonality from the original time series X (with trend). The season eliminated time series 14 12 10
X-S
8 6 4 2 0 -2 -4
0
10
20
30
40
50 t
60
70
80
90
100
Models with both trend and seasonality 5. It remain to eliminate the trend. This could be done e.g. with moving average or regression. If we use moving average with q= 5 Season eliminated Time series, Moving Average with q=5 14 12 10 8
XT
6 4 2 0 -2 -4
0
10
20
30
40
50 t
60
70
80
90
100
Models with both trend and seasonality The trend and season eliminated time series: Season and trend eliminated time series 4 3
2
YTsm
1
0 -1
-2 -3
-4
0
10
20
30
40
50 t
60
70
80
90
100
ACF from the ‘trend and season eliminated’ series: ans = 1.0000 0.8429 0.6713 0.5376 0.4453 ACF from the ‘true’ AR(1)-series: ans = 1.0000 0.8931 0.7761 0.6959 0.6264 True =
1.0000
0.9
0.81
0.729
0.6561
Bootstrap for time series
Introduction The observations are time dependent: we cannot resample from the empirical distribution function Fn (or resampling with replacement) which does not reflect the time dependency. Two main approaches: -Using a parametric model for resampling -Using blocks of the data to simulate the time series.
– Typeset by FoilTEX –
1
Model based resampling
We use standard time series models(ARMA)for defining the data generating process. - Let {Xt} be a second order stationary time series, with zero mean and autocovariance function γk : for all t, k we have E(Xt) = 0; Cov(Xt, Xt+k ) = γk . - The autocorrelation function is ρk = γk /γ0 for all k = 0, 1, 2, .... - Some basic models
– Typeset by FoilTEX –
2
Moving Average, MA(1)
• The model Xt = Zt + θZt−1, t = . . . , −1, 0, 1, . . . • here {Zt} is a white noise process of innovations: Zt ∼ W N (0, σ 2), uncorrelated • the autocorrelation function is ρ1 = θ/(1 + θ2) and ρk = 0 for |k| > 1, the variance γ0 = (1 + θ2)σ 2 • Matlab Simulation of MA(1) series: Zt=randn(1,n); X=filter([1 b],1,Zt); % b = θ
– Typeset by FoilTEX –
3
AutoRegressive, AR(1)
• The model Xt = φXt−1 + Zt, t = . . . , −1, 0, 1, . . . 2 • here |φ| < 1, and {Z } is a white noise W N (0, σ ), Xt = t P∞ k 0 φ Zt−k
• the variance, Cov(Xt−k , Zt) = 0, k ≥ 1, γ0 = V ar(Xt) = V ar(φXt−1 + Zt) = φ2γ0 + σ 2; and γ0 = σ 2/(1 − φ2). • the autocorrelation function is ρk = φ|k| and for k = 1, φ = ρ1 = γ1/γ0 • Matlab Simulation AR(1) series: Zt=randn(1,n); X=filter(1,[1 a],Zt); % a = −φ – Typeset by FoilTEX –
4
AutoRegressive Moving Average, ARMA(p,q)
• The model Xt =
p X
φk Xt−k + Zt +
k=1
q X
θk Zt−k ,
k=1
{Zt} is a white noise W N (0, σ 2). • Conditions on the coefficients to obtain a stationary process. • Matlab Simulation of ARMA(1,1) series: Zt=randn(1,n); X=filter([1 b],[1,a],Zt); % b θ, a = −φ
– Typeset by FoilTEX –
=
5
The bootstrap
• Fit the model to the data • Constructed the residuals from the fitted model • Re-center the residuals(mean zero as Zt) • Generate new series by incorporating random samples from the fitted residuals in the fitted model
– Typeset by FoilTEX –
6
Example: AR(1) model
• We have a sample x1, . . . , xn, compute φˆ = ρˆ1: ˆ
