Using Autoregressions Multiple Spectral Estimation - Department of ...

TECHNOMETRICS?, VOL. 24, NO. 2, MAY 1982

Using Spectral

Periodic

for Autoregressions Multiple

Estimation H. Joseph Newton Instituteof Statistics TexasA&MUniversity College Station,TX77843 A newmethodof estimatingthespectraldensityof a multipletimeseriesbasedon theconceptof periodicallystationaryautoregressive processesis describedandillustrated.It is shownthat the methodcan oftenovercomesomedifficultiesinherentin the traditionalsmoothedperiodogram and autoregressive methodsand that additionalinsightsinto the structure spectral-estimation of a multipletimeseriescanbe obtainedby usingperiodicautoregressions. KEY WORDS:Multiple time series; Spectral-densityfunction; Autoregressiveprocesses; Periodicautoregressive processes.

1. INTRODUCTION Let {Y(t), t e Z}, Z the set of integers, be a ddimensional time series, that is, for each integer t, Y(t) consists of observations Y1(t),Y2(t),..., Yd(t)of the d univariate time series Y1,..., Ydat time t. For example, the time series {Y(t), t e Z} could consist of EEG (electroencephalogram) records observed at equally spaced time points at d points on a subject's head. Other examples could be monthly total ozone levels at d recording stations or daily water-use levels and daily maximum temperatures at a certain geographical location. Given data Y(1), ..., Y(T) on such a time series, Y, one seeks to determine dynamic relationships among its channels

gets large. Then f(w) is a (d x d) matrix whose (j, k)th element is given by fjk(w)) =

2v

f(cO)=

coe [-r,

E 27rV=-X R(v)e-VW,

j k = 1, ... d,

1. Regressing one time series on another (see Brillinger 1975, Ch. 8). 2. From a d-dimensional time series, constructing a time series of lower dimension (which is much easier to analyze) that contains a large amount of the information in the original series (see Parzen and Newton 1980). 3. For a bivariate series, determining if the individual channels contain common cyclical trends (see Jenkins and Watts 1968, p. 340).

basic tools; the autocovariance function R and the spectral-density function f of Y: v

Rjk(v)e-i l=-00

andfjj( ) is called the power spectral density or auto spectral density of Yj, whilefjk( ?) is the cross spectral density of channel Yj with channel Yk.We note that fjj is real valued, fjk is complex, and fk((o) =fkj(wo), that is, the matrix f(w) is Hermitian. The spectral-density function f of the time series Y has proven very useful in a wide variety of statistical problems, including

Y1, Y2, ..., Yd. This is done using two

R(v) = cov(Y(t), Y(t + v)),

-1

Z zr] (1.1)

where i = //1. We note that R(v) is a (d x d) matrix whose (j, k)th element is given by Rjk(v) = cov(Yj(t), Yk(t+ v)), j, k = 1, ..., d; that {Rjj(v), v e Z} is the

The purpose of this article is to review existing methods of estimating the spectral-density function f and to propose a new method, based on the concept of periodically stationary autoregressive time series models (Pagano 1978, Gladyshev 1961, Jones and Brelsford 1967), which the author found to be very useful in overcoming some of the difficulties in the existing methods. Most of the technical details and computational formulas are given in Section 5.

autocovariance function of the time series Yj; and that {Rjk(v), v e Z} is the cross-covariance function between the time series Yj and Yk. By saying that cov(Y(t), Y(t + v))is the same for all t, we are assuming that Y is covariance stationary. Further, for the summation in (1.1) to exist (and thus for f to exist), we need to assume that the covariance of Yj(t) with Yk(t+ v) becomes in some sense close to zero for all j and k as v 109

110

H. JOSEPH NEWTON

2. TRADITIONAL ESTIMATION METHODS We assume that E(Y(t))is zero or that any nonzero mean has been successfully removed from Y. Thus we write R(v) = cov(Y(t), Y(t + v)) = E(Y(t)YT(t + v)), where the notation AT denotes the transpose of the

matrix A. 2.1 Periodogram Given a sample realization Y(1), ..., Y(T) from the time series Y, a natural estimator of R(v) = cov(Y(t), Y(t + v)) = E{Y(t)YT(t+ v)} is the average of the (T - v) available cross products of lag v; Y(1)yT(1 + v), ..., Y(T

-

v)YT(T). However, dividing by T in-

stead of by T - v leads to an estimator of R that is positive definite, a property that is important in guaranteeing that the spectral-density estimators to be given are also positive definite. Thus we define RT(V)=

-

1 T-Ivl

t (=1

Y(t)YT(t+ IVI),

Ivl < T- 1,

to be the sample autocovariance of lag v. Then the periodogram (or sample spectral-density function) is given by truncating (1.1) at v = + (T - 1) and estimating R(v) by Rr(v): fT(wo) =

T-1

1 -

Z

RT(v)e-

v= -(T-1)

Then it is well known that fT(c) is approximately an unbiased estimator of f(o) for large T, that fT(col) and fT(wO2) are uncorrelated,

but that the variances and covariances of elements of fT(w) do not approach zero

as T approaches infinity. Thus the periodogram is a highly oscillatory function of w) containing many possibly misleading peaks (see Hannan 1970, p. 249). 2.2 Smoothed Periodogram Method A number of methods for smoothing the periodogram to eliminate spurious peaks have been proposed. They are usually of the form fT,

=M(Co)

2.3 Autoregressive Method Another method of smoothing the periodogram, which has often been successful in objectively choosing an amount of smoothing and leading to more well-defined peaks, is the autoregressive method (see Akaike 1974, Parzen 1977, Newton and Pagano 1981). One assumes that Y(t) can be adequately expressed as Y(t) + A(1)Y(t-1)

+...

+ A(p)Y(t- p)=

teZ,

(t),

(2.1)

for some (d x d) matrices A(1), ..., A(p), and a ddimensional white noise time series {e(t), t e Z}, that is,

E(?(t))=

0,

cov(?(t), ?(s)) = 0

for

t + s,

and

var(?(t))= l, a (d x d) positive definite matrix. The integer p is called the order of the autoregressive process Y. Thus for example if d = 2 and p = 1, we have Y,(t) 2(t)_

All(1) _A2(1)

A12(1) A22(1)_ Y,l(t- 1) _ Y2(t -)_

e,(t) _ 2(t)_

and in general the model (2.1) states that each Yj(t) is expressible as a linear combination of Yl(t- k), ... Yd(t-k)fork = 1, ..., p. The autoregressive method consists of the following steps (see Parzen 1977 for details). 1. Determine an order p (if it is not known a priori) that is optimal for data Y(1), ..., Y(T). 2. Estimate A(1), ..., A(p) and l. 3. Form an estimator fp of f using the estimated parameters in the expression for the spectral density of an autoregressive process of order p. The autoregressive spectral estimator fp^(o)of f(co)is given by fp(w)= G l(ei)^pG *(ei), where

E

kM

R()e-

where M is an integer called a truncation point and k( ) is a weighting function called a covariance kernel. If k( . ) is a positive function, then fT M is positive definite. By properly choosing M and k( ) for a particular time series, one can obtain an estimator having very good statistical properties (see Parzen 1957). However, these choices are extremely difficult to make in practice and most analysts suggest using more than one amount of smoothing (usually by using the same k( - ) and varying M) in an attempt to determine the overall nature of the true spectral density.

TECHNOMETRICS ?, VOL. 24, NO. 2, MAY 1982

Gp(z)=

(j)z, j=0

j=l

Ap(j)RT(j- v)=

O, 0 .....,p

5~ = 1 if v = 0 and 0 otherwise, p is chosen as the value of m minimizing CAT(m) = tr

(T

-

T- md T

1

A* (A-*) denotes the complex conjugate transpose (inverse complex conjugate transpose) of a complex matrix A, and tr(A)denotes the trace of the matrix A.

111

USING PERIODICAUTOREGRESSIONS The integer p plays the role of a smoothing parameter for the autoregressive method analogously to M's role in the smoothed periodogram method. Thus the larger p is, the less smooth fp is. The autoregressive method has the advantage that there exists rationale for objectively choosing an order p having certain optimality properties. Of course the analyst should be aware that often there is more than one order that should be considered as leading to possible spectral estimators. 3. PERIODIC AUTOREGRESSIVE METHOD The autoregressive method has a drawback in that it often requires estimation of a large number of parameters. For example, if d = 4 and p is chosen to be 6 (not an unusual case in many scientific areas), then there are 106 parameters to be estimated, that is, the 96 = 16 x 6 elements of A(1), ..., A(6) and the 10 = 4 x 5/2 distinct elements of the symmetric matrix ;. Often, many of these parameters are negligible in size and it would be very useful if one could find a method for easily determining which can be considered to be zero. The idea of periodic autoregression affords such a method. Essentially, the ddimensional autoregression can be decomposed into d one-dimensional processes that can be analyzed separately. Then a spectral estimator for Y can be constructed from these d-individual analyses. This decomposition also allows one to obtain further insights into the joint behavior of the individual channels in the multiple series. To illustrate this decomposition of a d-dimensional process into d one-dimensional processes, consider 1. Ifwe let d=2,p= A(1)=

1l1 __21

L="1 _0a2/0 l

522.

D '

ll

=

(l2

12

__012

1

[oD=

=

022-

10

0

L

0'22

-

a2

--

1 101a

then we have that $ = LDLT is the modified Cholesky decomposition (Wilkinson 1967) of l, and we can write L- Y(t) + L- A(1)Y(t - 1) = L-1(t) = q(t), where var(qt(t))= D, that is, we can write Y1(t) + al(1)Y2(t - 1) + x1(2)Y1(t--

1) =1

(t),

Y2(t) + aC2(1)Y1(t)+ Ca2(2)Y2(t- 1) + a2(3)Y(t-

1) = /2(t),

where r1I and r12 are mutually uncorrelated white noise time series with variances al and 0220'2/011, respectively, while La(l) = a12, aj(2)= all,

a2(1) = -(12/011,

a2(2) = a22 -(012/0

a2(3) = a21 - (12/oll)

11) O12,

and

al1.

Thus if we define the one-dimensional time series X by X(0) = Y1(0),X(1) = Y2(0),X(2) = Y1(1),and so on, we see that our single two-dimensional autoregressive model has been written as two one-dimensional models, one expressing elements of the first channel of Y in terms of the previous two elements of X and the other expressing the second channel as a function of the previous three elements. Such a time series X is called a periodically stationary autoregressive process (or simply a periodic autoregression) of orders two and three. In Section 5.1 we show how a general d-dimensional autoregressive process of order p can be written as a periodic autoregression of orders pd, pd + 1, ..., pd + (d - 1). Such a process (with the restriction that the orders be pd, ..., pd + (d - 1) removed) is of interest in its own right. For example, if X is a time series of monthly average temperatures at a certain location (i.e., d = 12), then it is reasonable to assume that the observation for a certain month is linearly related to observations for previous months, but a) the order of dependence, the coefficients of the linear combination, and the variance of the error will vary from month to month, and b) these quantities will be constant for the same month from year to year. We note further that from any periodic autoregression of period d and orders pi, ..., Pd, we can construct a d-dimensional autoregression. The periodic autoregressive method for estimating the spectral density of Y consists then of (see Sec. 5.1 for details) 1. Determining the orders Pil, ..., Pd of the periodic autoregressive time series X with the order restrictions removed. 2. Finding estimators ck(j) and dk, k = 1, ..., d, j = 1, ..., Pk of the coefficients and error variances for the d-individual models. 3. From these estimators reconstruct estimators p, A(1), ..., A(p), ..., and ; of the corresponding ddimensional autoregressive process. Corresponding to this model we obtain the spectral-density estimator The major advantage of the periodic autoregressive method then is that whereas the autoregressive method forces P1 = dp + 1, ..., d = d + (d - 1), the periodic method leaves these integers free to vary. This may greatly reduce the number of parameters needed to adequately model Y. What often happens in practice is that one of the p!J'sis large relative to the others and the ordinary autoregressive method is forced to choose a large value of p to capture this phenomenon (thus forcing the total number of

TECHNOMETRICS?, VOL. 24, NO. 2, MAY 1982

H. JOSEPH NEWTON

112

parameters to be pd2 + d(d + 1)/2). The periodic autoregressive method, however, leaves the other pj's unrestricted, giving a total of Pi + ** + Pd + d parameters. Also the values of 1, ..., P^ provide insight into the amount of dependence of the various channels on Y on past values of this and other channels. Finally, Pagano (1978) has shown that the coefficient estimators for the various channels are asymptotically normal and independent of each other and thus their estimation and testing can be done separately. 4. WORKED EXAMPLE To illustrate the methodology of the periodic autoregressive method, consider the two time series in Figure 1; the series being levels in a cow of the hormones LH and Prolactin measured at the same 10minute intervals over a 24-hour period, thus giving T = 144 pairs of observations. The estimated autospectra (all autospectra in this example are plotted on a logarithmic scale because of the highly oscillatory nature of the estimators and are normalized to be on the same scale so that they can be compared) obtained from the periodogram method are given in Figure 2, illustrating how highly variable this estimator is. Figure 3 gives the results of using the Parzen kernel

+61X13

1 --6x2

,

Prolactin

2.00i 000?

V|M|\VV t

l-

1 -4.00

0.08

0.00

0.i6

0.24

0.32

0.40

0.48

0.32

0.40

0.48

LH 4.00' 200 '

t

0.00

"1 10

xi < ?l22

< Ixl < 1

k(x)= {2(1 -Ixl)3,?

xl >

0,

with truncation points M1 = T/6 = 24 and M2 = T/3 = 48 for smoothing the periodogram. These plots (M2 has the higher peaks) show the importance of the amount of smoothing done on the resulting estimators.

-6.00 0.00

0.08

0.16

0.24

FREQ (cycles/10

minutes)

Figure 2. Log Periodograms of Hormone Series

Prolactin Series (7 = 683.01) 1600

In Table 1 we give the parameterestimates from the

|

AJ

autoregressive and periodic autoregressive analyses of the series. Note that 31 parameters are used in the 800autoregressive model while 21 are required in the A A % J\ IA| U 8 periodic autoregressive model. Further, we see that to 1 \AAAV t ^1h ^\ \AA V11 Y 400f4 model LH adequately at time t, the periodic autore> VV . ? 400 gressive model uses the values of LH and Prolactin at each of the previous eight times while Prolactin at time t only requires LH at time t and both LH and LHSeries ( = 17.16) Prolactin at time t - 1. This results in a smoother for Prolactin than for LH as shown by spectrum 31. Figure 4 (autoregressive autospectra) and Figure 5 (periodic autoregressive spectra). k 23-1 A\ I 1\ The cyclic nature of the LH series is indicated in A \ AIA./r 151 V \ of its estimated autospectra while the long-term v each V ^vU \ ^V ' ' ' ' of Prolactin is indicated by its estimates. smoothness 9 25 41 57 1 89 105 121 137 153 e note that the autoregressive estimator exhibits the Nr WObservation same general features as the smoothed periodogram estimator and that the periodic autoregressive estiFigure 1. Two- Hormone Time Series

1200-

X

A

.

V V\\\ kv[

t\A +A g

TECHNOMETRICS?, VOL. 24, NO. 2, MAY 1982

113

USING PERIODICAUTOREGRESSIONS

4.00-

Prolactin

4.00-

2.00-

2.00-

0.00

0.00-

-2.00-

-2.00

-4.00

-4.00-

-6.00 0.00

. 0.08

.

0.16

0.40

0.32

0.24

0.48

Prolactin

-6.00 0.C)0

.00-

I

I

I~~~~~

0.08

0.16

0.24

0.32

0.40

0.48

0.08

0.16

0.24

0.32

0.40

0.48

LH

2.00-

0.00-2.00 -

-4.00-

-6.00 0.08

0.00

0.16

0.32

0.24

FREQ (cycles/10

0.40

0.48

0.00

FREQ (cycles/10

minutes)

minutes)

Figure 3. Log - Smoothed Periodograms of Hormone Series ( Parzen kernel, truncation points 24 and 48)

Figure 4. Log Autoregressive Spectra of Hormone Series

Table 1. Fitted Models for Hormone Series

mator is essentially identical to the autoregressive estimator although it used 10 fewer parameters. In Figure 6 we give the squared coherency between LH and Prolactin obtained from the three spectral estimators (M = 24 for the Parzen estimator). These plots show little indication of any association at any frequency between the two hormones.

Autoregressive Model

, =r.44 .051 L.O .43J .13 .161 .04] [.01 A7 (5)= r-.03 .091 L.03 -.17]

A7(1)= f-.30-.11] .09 -.74

A7(2)=

A, (4) = f.15 -.02] L-[09 .06 A, (7)=

-.52

A, (3)

[.02 -.091 L-.004

A7(6)=

.06 l.06

.051 .02j

051

-0 8 .06J

Periodic Autoregressive Model A, (1)= [-.21 [.13

A (4)= [.12

I .01

A, (7) = -.46 L-.03

t8

P,2=16 p2= 3

-01

A. (2) =

-.74

12 [01

=

.141 .01]

.00] .oo00

AB(5)= [-.03 .081 L-.01 -.01]

.081 .OlJ

A,(8)=

-.02 |-.1 L 01 -.01]J

43 .03 [ .03 .45 A (3)= [.02 -.071 .00 -.01

A, (6) = [ .04

[ .00

.04 .00

5. TECHNICAL AND COMPUTATIONAL DETAILS In this section we provide some technical details and computational formulas for the methods just considered. 5.1 The Periodic Autoregressive Method 1. If Y is a d-dimensional autoregressive process of order p with parameters A(1), ..., A(p) and $, then the TECHNOMETRICS?, VOL. 24, NO. 2, MAY 1982

114

H. JOSEPH NEWTON

4.00.

Prolactin

1.00-

Periodic Autoregressive Estimate

0.80-

2.00

0.60-

0.00

0.40-

-2.00

0.20-

-4.00

-6.00

A fn U.UU

.

0.00

4.00 -

0.08

0.16

0.24

0.40

0.32

0.00

0.48

,

.-

0.08

0.16

0

.

0.24

0 .

0.32

0.40

0.448

0.32

0.40

0.48

LH 1.00-

Autoregressive Estimate

0.80-

0.60

0.40'

0.20 -6.00 0.00

0.08

0.24

0.16

0.40

0.32

0.48

0.00

O.(D0

I

0.08

0.16 . 0.16

0.24

FREQ (cycles/10 minutes) Figure 5. Log- Periodic Autoregressive Spectra of Hormone Series

1.00'

Parzen Smoothing Estimate (truncation point 24)

0.80 time series X = {..., YT(-1),

YT(O Yr(1) ), ...} is a

periodically stationary autoregressive process of period d, orders pi = pd, P2 = pd + 1, ..., Pd= pd + (d - 1),and parameters ak(j),

i,

I