a markovian local resampling scheme for nonparametric estimators in ...

Econometric Theory, 17, 2001, 540–566+ Printed in the United States of America+

A MARKOVIAN LOCAL RESAMPLING SCHEME FOR NONPARAMETRIC ESTIMATORS IN TIME SERIES ANALYSIS EF S T A T H I O S PA P A R O D I T I S University of Cyprus

DI M I T R I S N . PO L I T I S University of California, San Diego

In this paper we study the properties of a pth-order Markovian local resampling procedure in approximating the distribution of nonparametric ~kernel! estimators of the conditional expectation m~x;f! 5 E~f~X t11 !6Yt, p 5 x! where $X t , t $ 1% is a strictly stationary process, Yt, p 5 ~X t , X t21 , + + + , X t2p11 ! Á , and f~{! is a measurable real-valued function+ Under certain regularity conditions, asymptotic validity of the proposed resampling scheme is established for a class of stochastic processes that is broader than the class of stationary Markov processes+ Some simulations illustrate the finite sample performance of the proposed resampling procedure+

1. INTRODUCTION Let $X t , t 5 1,2, + + + % be a ~strictly! stationary real-valued stochastic process and denote by FYt, p ~{! the marginal distribution of Yt, p 5 ~X t , X t21 , + + + , X t2p11 ! Á and by FXt11 6Yt, p ~{6{! the conditional distribution of X t11 given Yt, p + In this paper we are interested in estimating the sampling behavior of a nonparametric ~kernel! estimator of the conditional expectation m~x; f! 5 E~f~X t11 !6Yt, p 5 x! where f~{! is a real-valued, measurable function+ Several conditional functionals of the process can be obtained by appropriately specifying the function f+ For instance, for f1~z! 5 z and f 2 ~z! 5 z 2 we have that m~x; f1 ! is the conditional mean, whereas m~x; f 2 ! 2 m 2 ~x; f1 ! is the conditional variance of s 6Yt, p 5 x! X t11 given that Yt, p 5 x+ Apart from the conditional moments E~X t11 the function m~x; f! can also be used to describe some other conditional functionals of the process that may be of interest+ As an example, for f~z! 5 1~2`, y# ~z! and y a real number we have that m~x; f! 5 FXt11 6Yt, p ~ y6x!, that is, the conditional distribution function mentioned previously+ We are grateful to the co-editor Joel Horowitz and to both referees for a number of useful comments+ This research has been supported in part by National Science Foundation Grant DMS-97-03964 and by a University of Cyprus Research Grant+ Address correspondence to: Efstathios Paparoditis, Department of Mathematics and Statistics, University of Cyprus, P+O+ Box 20537, CY-1678 Nicosia, Cyprus; e-mail: stathisp@ucy+ac+cy+

540

© 2001 Cambridge University Press

0266-4666001 $9+50

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

541

Given observations X 1 , X 2 , + + + , X T the nonparametric estimator of m~x; f!, which we consider in this paper, is given by T21

( f~Xt11 !Kh ~x 2 Yt, p !

m[ h ~x; f! 5

t5p

T21

+

(1.1)

( Kh ~x 2 Yt, p !

t5p

In this notation Kh ~{! 5 h 2p K~{0h! where K : R p r R is a nonnegative kernel satisfying *K~u!du 5 1, *uK~u! du 5 0 and h 5 h~T ! . 0 is the bandwidth used to smooth the observations+ Note that ~1+1! is the common Nadaraya– Watson estimator of m~x; f!+ Nonparametric estimators such as the one considered in this paper are frequently used as an end product in time series analysis for modeling or predictive purposes or, perhaps more important, as a guide in identifying a parametric model to be used in a subsequent stage+ For instance, a general univariate model that contains many of the linear and nonlinear classes discussed in the literature, including the popular nonlinear conditional heteroskedastic ~ARCH! model, is given by X t11 5 g~X t , X t21 , + + + , X t2p11 , «t ! where g : R p11 r R is a measurable function+ Here $«t % is an independent and identically distributed ~i+i+d+! sequence of random variables with mean zero and finite variance and «t is assumed to be independent of X s for s # t ~cf+ Tong, 1990, Ch+ 3!+ One way to gain insight into the nonlinearity features of such models given a set of observations is to estimate nonparametrically functionals such as the conditional mean or the conditional variance and to compare the estimates obtained with those expected under a particular hypothesis about the underlying model+ Such an approach of detecting and modeling nonlinearity in time series analysis has been proposed by several authors; see among others Tong ~1990!, Auestad and Tjøstheim ~1990!, and Tjøstheim and Auestad ~1994!+ Under certain regularity conditions, strong consistency and asymptotic normality of the estimator m[ h ~x; f! have been established by several authors; we mention here among others the papers by Robinson ~1983!, Roussas ~1990!, and Masry and Tjøstheim ~1995!+ The asymptotic normality of the estimator ~1+1! is, for instance, useful in constructing pointwise confidence intervals for m~x; f!+ However, there is an inherent difficulty in using the limiting Gaussian distribution for such purposes that is due to the fact that the mean and the variance of this distribution depend on unknown ~and difficult to estimate! characteristics of the process+ The aim of this paper is to show that a Markovian local bootstrap procedure proposed by Paparoditis and Politis ~1997! and which generates bootstrap replicates X 1* , X 2* , + + + , X T* that “reproduce” correctly the conditional distribution of X t11 given the “past” Yt, p2 leads to an asymptotically valid approximation of the distribution of L T ~x! 5 !Th p ~ m[ h ~x; f! 2 m~x; f!!+

(1.2)

542

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

The proposed procedure provides an alternative ~as compared to the normal! approximation of the distribution of L T ~x! and at the same time leads to consistent estimation of quantities such as the mean and the variance of the distribution of interest+ Although the local bootstrap procedure generates a pseudoseries X 1* , X 2* , + + + , X T* with a Markov dependence structure, we show in the present paper that the validity of this resampling scheme for the class of nonparametric estimators considered is not limited to the Markov case only+ To understand this fact intuitively note that the information regarding estimation of FXt11 6Yt, p ~{6{! and its moments typically lies in the “scatterplot” of X t11 vs+ Yt, p for t 5 1,2, + + + , T 2 p; the crucial feature of this scatterplot is the joint distribution of X t11 and Yt, p that is determined by the conditional distribution FXt11 6Yt, p ~{6{! and the marginal distribution FYt, p ~{!+ Therefore, capturing FXt11 6Yt, p ~{6{! and FYt, p ~{! by a bootstrap process may give valid distributional conclusions for general ~not necessarily Markov! stationary processes+ Bootstrapping nonparametric estimators in time series analysis has received considerable interest in recent years+ Franke, Kreiss, and Mammen ~1996! propose some alternative approaches for bootstrapping the ~pointwise! distribution of nonparametric estimators under the assumption that the process considered is generated by the nonlinear autoregressive process X t11 5 f ~X t ! 1 s~X t !«t11 with $«t % an i+i+d+ sequence+ These approaches include a wild bootstrap and a residual-based bootstrap+ Under the assumption that $X t % is a Markov process, Neumann and Kreiss ~1998! show validity of the wild bootstrap for local polynomial estimators of the conditional mean E~X t11 6 X t 5 x! and for supremum type statistics+ Compared to these approaches, our Markovian local resampling scheme avoids any kind of preliminary nonparametric estimation of unknown functions to generate the bootstrap pseudoseries+ Furthermore, its realm of validity includes but is not restricted to the Markov class+ The paper is organized as follows+ In Section 2 the Markovian local resampling scheme is briefly described and the bootstrap approximation to the statistic L T ~x! is given+ Section 3 is devoted to the ability of the Markovian local resampling procedure to approximate the distribution of L T ~x!+ After stating the set of technical assumptions imposed on the class of stochastic process considered, the main result is given+ The finite sample performance of the method proposed is illustrated by some simulated examples in Section 4, and all proofs are collected in Section 5+

2. THE MARKOVIAN LOCAL RESAMPLING SCHEME The Markovian local resampling scheme introduced by Paparoditis and Politis ~1997! generates bootstrap replicates by reshuffling the original data points according to a particular probability mechanism+ This local resampling scheme can be described by the following three steps+

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

543

1+ Select a resampling width b 5 b~T ! . 0 and a set of p starting values Yp,* p 5 * ~X p* , X p21 , + + + , X 1* !+ In practice, a simple choice is Yp,* p 5 Yp, p + Select further a resampling kernel W that is a probability density on R p satisfying W . 0 and *uW~u!du 5 0+ 2+ For any time point t 1 1 [ $ p 1 1, p 1 2, + + + , T % suppose that Yt,* p has been already generated+ Let J be a discrete random variable taking its values in the set N p, T21 5 $ p, p 1 1, + + + , T 2 1% with probability mass function given by

Y

P~J 5 s! 5 Wb ~Yt,* p 2 Ys, p !

(

Wb ~Yt,* p 2 Ym, p !

(2.3)

m[N p, T21

* * for s [ N p, T21 + The bootstrap replicate X t11 is then defined by X t11 5 X J11 + 3+ If t 1 1 , T then go to step 2+

At completion of the algorithm, a new bootstrap pseudoseries X 1* , X 2* , + + + , X T* is created with distributional properties that “mimic” those of FXt11 6Yt, p of the original series+ Note that the Markovian local resampling scheme described earlier “works” by assigning at each step of the resampling process the resampling probabilities ~2+3! to each of the observed original values X s11 [ $X p11 , X p12 , + + + , X T %+ By this algorithm the probability that the * bootstrap random variable X t11 takes the value X s11 depends on how close Ys, p is to the already generated segment of bootstrap values Yt,* p+ The closer Ys, p is to Yt,* p the larger the resampling probability ~2+3! will be provided the mass of the resampling kernel W is concentrated around its mean value * 5 zero+ It is easily seen that by the positivity of the resampling kernel P~X t11 T21 * * * X s11 6Yt, p ! . 0+ Furthermore, ( s5p P~X t11 5 X s11 6Yt, p ! 5 1+ Our procedure provides a ‘natural’ extension of Efron’s ~1979! classical i+i+d+ bootstrap to the Markovian case+ In effect, Efron’s ~1979! i+i+d+ bootstrap can be considered a special case of our local bootstrap if we allow the Markovian order p to be zero, in which case all observations have the same resampling probability, that is, P~J 5 s! 5 10~T 2 p!+ Given a bootstrap series X 1* , X 2* , + + + , X T* generated by the Markovian local bootstrap, we propose to approximate the distribution of the statistic L T ~x! by the distribution of the bootstrap statistic L*T ~x! 5 !Th p ~ m[ h* ~x; f! 2 m b* ~x; f!!,

(2.4)

where T21

m[ h* ~x; f! 5

* f~X j11 !Kh ~x 2 Yj,* p ! ( j5p T

( Kh ~x 2 Yj,* p !

j5p

(2.5)

544

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

and T21

( f~Xt11 !Wb ~x 2 Yt, p !

m b* ~x; f! 5

t5p

(2.6)

+

T

( Wb ~x 2 Yt, p !

t5p

Note that if x [ X p , where X p is the p-times Cartesian product of the set of * observed values X 5 $X 1 , X 2 , + + + , X T %, then m b* ~x; f! 5 E * ~f~X t11 !6Yt,* p 5 x!, * that is, the conditional expectation of the bootstrap processes $X t , t $ 1% is a kernel estimator of m~x; f! with a “smoothing kernel” W and a “smoothing bandwidth” b+ In the following discussion we assume throughout that the set of points x at which the unknown function m~x; f! is estimated is a subset of X p + Note further that in obtaining m[ h* ~x; f! the same formula, that is, the same kernel and the same bandwidth as in ~1+1!, is used with the only difference that the observations X t , t 5 1,2, + + + , T are replaced by the bootstrap replicates X t* , t 5 1,2, + + + , T+ 3. VALIDITY OF THE MARKOVIAN LOCAL BOOTSTRAP FOR KERNEL ESTIMATORS To state the main result of this paper the following set of technical assumptions is needed+ ~A1!+ ~i! FYt, p ~{! and FXt11 6Yt, p ~{6x! are absolutely continuous with respect to Lebesgue measure on R p and R and have bounded densities fYt, p ~{! and fXt11 6Yt, p ~{6x!, respectively+ ~ii! For all x1 , x2 [ R p , and y [ R ø $`%,

*E

y

2`

E

fXt11 , Yt, p ~z, x1 !dz 2

y

2`

*

fXt11 , Yt, p ~z, x2 ! dz # L~ y!7x1 2 x2 7,

where fXt11 Yt, p ~z, x! 5 fXt11 6Yt, p ~z6x! fYt, p ~x!, infy L~ y! . 0 and supy L~ y! 5 L , `+ ~iii! For all y1 , y2 [ R, 6 fXt11 6Yt, p ~ y1 6x! 2 fXt11 6Yt, p ~ y2 6x!6 # C~x!6 y1 2 y2 6 where C~x! . 0 and supx[R p C~x! 5 C , `+ ~iv! A compact subset S of R exists such that X t [ S a+s+ Furthermore, fXt11 6Yt, p ~{6x! . 0 for every x [ S, where S denotes the p-fold Cartesian product of S+

Assumptions ~A1!~ii! and ~iii! are smoothness assumptions on the joint and conditional densities and are common in nonparametric estimation problems such as those discussed here ~cf+ Robinson, 1983; Masry and Tjøstheim, 1995!+ The assumption of compactness of the support of fXt ~{! in ~A1!~iv! is of a rather technical nature+ It is imposed to simplify the technical arguments and

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

545

can be weakened by simultaneously strengthening other requirements ~for details, cf+ Paparoditis and Politis, 1997!+ ~A2!+ ~i! f is a real-valued, measurable function such that E6 f~ X t !6 m , ` for some m . 4+ ~ii! For all z 1 5 ~ y1 , x1 !, z 2 5 ~ y2 , x2 ! [ R p11 , 6f~ y1 ! fXt11 Yt, p ~ y1 , x1 ! 2 f~ y2 ! fXt11 Yt, p ~ y2 , x2 !6 # C7z 1 2 z 2 7, where 0 , C , `+

The next assumption deals with the dependence properties of $X t %+ Recall the definition of the strong mixing ~a-mixing! coefficient, that is, a~n! 5

sup j

`

6P~A ù B! 2 P~A!P~B!6,

n [ N,

A[B1 , B[Bj1n ` denote the s-algebras generated by the sets of random variwhere B1 and Bj1n ables $X 1 , X 2 , + + + , X j % and $X j1n , X j1n11 , + + + %, respectively+ Based on this definition, $X t , t $ 1% is a-mixing if a~n! r 0 as n r `+ In the subsequent discussion we make the following assumptions+ j

~A3!+ $X t , t $ 1% is a-mixing, and the mixing coefficient satisfies a~n! # Cr n for some positive constants C and r [ ~0,1!+ Remark 3+1+ Note that ~A3! is fulfilled if the process $X t , t $ 1% is a pthorder strictly stationary and geometrically ergodic Markov process and that this Markov class of stochastic processes contains several of the commonly used parametric and nonparametric models in time series analysis ~for several examples, see Tong, 1990, Ch+ 3!+ It is known that if $X t , t $ 1% belongs to this Markov class then it is geometrically absolutely regular ~i+e+, b-mixing!, which implies geometrically a-mixing ~see Doukhan, 1994!+ However, the class of stochastic processes satisfying ~A3! is broader than the Markov class described previously+ The following conditions are imposed on the smoothing and resampling parameters+ ~A4!+ ~i! K~{! is a product kernel; that is, for every x 5 ~ x 1 , x 2 , + + + , x p ! [ R p we have p K~x! 5 ) i51 k~ x i ! where k~{! is a bounded, Lipschitz continuous, and symmetric probability density the support of which is the interval @21,1# + ~ii! The smoothing bandwidth h satisfies h r 0 and Th p r ` as T r `+

546

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

~A5!+ ~i! The resampling kernel W is a bounded, Lipschitz continuous, and symmetric probability density on R p satisfying W . 0, *uW~u!du 5 0 and *7u7W~u!du , `+ ~ii! The resampling bandwidth b satisfies b 5 O ~T 2d ! for some 0 , d , 10~2~ p 1 2!!+

The next assumption is needed to get ~asymptotic! expressions for the bias term in estimating m~x; f! using the kernel estimator given in ~1+1!+ ~A6!+ ~i! W is two times continuously differentiable+ Furthermore, for i, j [ $1,2, + + + , p% we have *u is W~u!du 5 0 for s odd and *u i u j W~u!du 5 di, j W2 where di, j is Kronecker’s d and 0 , W2 , ` for all i [ $1,2, + + + , p%+ ~ii! fXt11 Yt, p ~ y, x! is two times continuously differentiable with respect to x, and the ~x ! ~x , x ! functions *c~ y! fXt11l Yt, p ~ y, x!dy and *c~ y! fXt11l1 Yt,l2p ~ y, x!dy are Lipschitz contin~x ! uous for c 5 1 and c 5 f and every l 1 , l 2 [ $1,2, + + + , p%+ Here, fXt11l Yt, p ~ y, x! and ~ x l1 , x l2 ! fXt11 Yt, p ~ y, x! denote first and second order partial derivatives of fXt11 Yt, p ~ y,{! evaluated at x+

Given the preceding set of assumptions, large sample validity of the proposed Markovian local resampling scheme in approximating the distribution of L T ~x! can be established+ By “large sample validity” we mean that the law of L*T ~x! ~conditionally on the data! is close to the law of L T ~x! with high probability, if the sample size T is large enough+ Our main theorem makes the preceding statement precise; here Kolmogorov’s distance d0 ~P, Q! 5 supx[R 6P ~ X # x! 2 Q~ X # x!6 between probability measures P and Q is employed+ THEOREM 3+1+ Suppose (A1)–(A6) hold and let T 102 h ~ p14!02 r Ch $ 0. We then have that, as T r `, d0 $L~L*T ~x!6 X 1 , X 2 , + + + , X T !, L~L T ~x!!% r 0

in probability.

Note that if T 102 h ~ p14!02 r 0 then the bias term !Th p ~Em[ h ~x; f! 2 m~x; f!! is asymptotically negligible, whereas for T 102 h ~ p14!02 r Ch . 0 this term converges weakly to B~x; f! 5 Ch

E

p

H

u 2 k~u!du ( m ~ x l ! ~x; f! fYt, pl ~x! 1 l51

~x !

J

1 ~x x ! m l l ~x; f! fYt, p ~x! , 2 (3.7)

~x ! fYt, pl ~x!

and m ~ x l x l ! ~x; f! denote first and second order parwhere m ~ x l ! ~x; f!, tial derivatives of m~x; f! and fYt, p ~x! with respect to x l ~cf+ Auestad and Tjøstheim, 1990!+ The case T 102 h ~ p14!02 r Ch actually corresponds to optimal smoothing ~minimizing the mean square error of m[ h ~x, f!!, where the case

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

547

T 102 h ~ p14!02 r 0 corresponds to undersmoothing that is suboptimal+ In most cases where resampling is used to estimate the distribution of a kernel smoothed estimator, suboptimal smoothing is used so that the bias term becomes negligible+ Quite remarkably, the Markovian local bootstrap procedure also approximates correctly the asymptotic mean ~bias term! of the statistic L T ~x!+ 4. NUMERICAL EXAMPLES Two simple numerical examples are presented in this section to demonstrate the finite sample performance of the bootstrap method proposed+ We consider realizations of length T 5 500 from the random coefficient autoregressive model ~cf+ Tong, 1990, p+ 111! X t 5 @0+1 1 bt #X t21 1 «t

(4.8)

and from the moving average model X t 5 «t 1 0+7«t21 ,

(4.9)

where $«t % and $ft % are i+i+d+ sequences independent from each other and satisfying «t ; N~0,1! and ft ; N~0,0+81!+ For model ~4+8! we are interested in estimating the distribution of the kernel estimator of the conditional variance v~ x! 5 Var~X t11 6 X t 5 x! 5 1 1 0+81x 2 + Using a smoothing bandwidth h 5 0+7 and Epanechnikov’s kernel, such an estimator of v~ x! is given by v[ h ~ x! 5 m[ h ~ x; f 2 ! 2 m[ h2 ~ x; f1 !+ For model ~4+9! we are interested in estimating the distribution of the kernel estimator m[ h ~ x! of the conditional mean m~ x! 5 E~X t11 6 X t 5 x!+ Note that under the Gaussianity assumption the conditional mean of X t 5 «t 1 u«t21 is linear and given by m~ x! 5 $u0~1 1 u 2 !%x+ The 5%, 50%, and 95% percentage points of the exact distributions of the statistics considered have been estimated using the corresponding percentage points of the bootstrap statistics !Th~ v[ h* ~ x! 2 vb* ~ x!! and !Th~ m[ h* ~ x! 2 m b* ~ x!! for several values of x in the interval ~23,3!+ Note that v[ h* ~ x! 5 m[ h* ~ x, f 2 ! 2 @ m[ h* ~ x, f1 !# 2 , vb* ~ x! 5 m b* ~ x; f 2 ! 2 @m b* ~ x, f1 !# 2 , m[ h* ~ x! 5 m[ h* ~ x, f1 !, and m b* ~ x! 5 m b* ~ x, f1 !+ The results of this simulation example are presented in Figure 1 for model ~4+8! and in Figure 2 for model ~4+9!+ The bootstrap estimates, shown in these figures by crosses, are mean estimates over 100 independent repetitions of the corresponding models+ For each repetition the distribution of the bootstrap statistic has been evaluated using 1,000 bootstrap replications, the Gaussian resampling kernel, and resampling width b 5 1+0+ The percentage points of the exact distribution, shown in these figures by circles, have been estimated using 10,000 replications of each of the models considered+ As these figures show, the bootstrap procedure proposed gives a very satisfactory estimation of the exact percentage points of the distribution of interest over the whole range of x-values considered+ We mention here that the accu-

548

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

Figure 1. Estimated exact and bootstrap estimates of the 5%, 50%, and 95% percentage points of the distribution of !Th~ v[ h ~ x! 2 v~ x!! for model ~4+8!+

racy of the bootstrap estimator decreases as the value of 6 x6 increases+ This can be seen in Table 1, where the estimated standard deviation of the bootstrap estimators is reported for both models considered and for some different values of x+ Table 2 gives ~for both models considered! the empirical coverage probabilities of 90% bootstrap confidence intervals and the corresponding mean lengths

Table 1. Standard errors of the bootstrap estimates of the 5%, 50%, and 95% percentage points of the distributions considered in Figures 1 and 2 Model ~4+8! X 22+5 21+5 0+0 1+5 2+5

Model ~4+9!

5%

50%

95%

X

5%

50%

95%

13+171 3+050 0+545 3+202 12+737

5+245 1+294 0+408 1+372 5+344

15+026 3+411 0+720 3+186 11+028

22+4 21+4 0+0 1+4 2+4

1+048 0+473 0+315 0+406 1+306

0+722 0+296 0+216 0+315 0+801

1+110 0+400 0+287 0+493 1+123

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

549

Figure 2. Estimated exact and bootstrap estimates of the 5%, 50%, and 95% percentage points of the distribution of !Th~ m[ h ~ x! 2 m~ x!! for model ~4+9!+

of these intervals based on 100 independent repetitions+ For model ~4+8! we calculate the bootstrap confidence intervals for the conditional variance v~ x! and for model ~4+9! the bootstrap confidence intervals for the conditional mean m~ x! using the values x 5 21+5, x 5 0, and x 5 1+5+ Also included in this table is a comparison of the Markovian local bootstrap with the block bootstrap method ~cf+ Künsch, 1989; Liu and Singh, 1992!+ Note that a ~1 2 a!100% bootstrap confidence interval for the conditional variance v ~ x! is defined here by @ v[ h ~ x! 2 q[ 12a02 , v[ h ~ x! 1 q[ a02 # where q[ p denotes the pth quantile of the bootstrap estimate of the distribution of v[ h ~ x! 2 v~ x!+ For the Markovian local resampling scheme this distribution is estimated using the distribution of v[ h* ~ x! 2 vb* ~ x! and for the block bootstrap method using the distribution of v[ h1 ~ x! 2 v[ h ~ x!+ Here v[ h1 ~ x! is the kernel estimator of v~ x! based on the replicate X 11 , X 21 , + + + , X T1 obtained using the block bootstrap method+ The bootstrap confidence intervals for the conditional mean m~ x! have been obtained in an analogous way, that is, by replacing in the preceding formulas v~ x!, v[ h ~ x!, v[ h* ~ x!, vb* ~ x!, and v[ h1 ~ x! by m~ x!, m[ h ~ x!, m[ h* ~ x!, m b* ~ x!, and m[ 1 h ~ x!, respectively, where m[ 1 h ~ x! is the kernel estimate of the conditional mean based on the block bootstrap series+ Finally, and to see the effects of varying the resampling parameters, both methods have been applied using several values of the resampling width b and of the block size B+

550

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

Table 2. Empirical coverage probabilities ~C+P+! and mean lengths of 90% bootstrap confidence intervals for the conditional variance of model ~4+8! and for the conditional mean of model ~4+9! x 5 21+5 C+P+

Length

x 5 0+0

x 5 1+5

C+P+

Length

C+P+

Length

Model (4.8) Block bootstrap B 5 10 B 5 15 B 5 20 B 5 25 Markov local bootstrap b 5 0+7 b 5 1+0 b 5 1+2 b 5 1+5

0+86 0+85 0+79 0+79

2+131 2+028 1+888 1+567

0+83 0+96 0+88 0+93

1+475 1+296 1+309 0+771

0+83 0+81 0+79 0+79

2+115 2+049 2+013 1+530

0+82 0+86 0+84 0+87

1+435 1+424 1+419 1+457

0+80 0+92 0+95 0+95

0+408 0+447 0+480 0+518

0+87 0+84 0+84 0+85

1+412 1+429 1+404 1+444

Model (4.9) Block bootstrap B 5 10 B 5 15 B 5 20 B 5 25 Markov local bootstrap b 5 0+7 b 5 1+0 b 5 1+2 b 5 1+5

0+76 0+82 0+85 0+93

0+352 0+349 0+349 0+340

0+85 0+89 0+91 0+89

0+244 0+248 0+240 0+239

0+86 0+89 0+91 0+85

0+345 0+352 0+340 0+345

0+88 0+87 0+93 0+93

0+387 0+386 0+393 0+395

0+90 0+92 0+93 0+93

0+271 0+272 0+271 0+274

0+88 0+92 0+94 0+94

0+384 0+389 0+388 0+388

As this table shows the local bootstrap procedure compares favorably with the block bootstrap procedure regarding the coverage probabilities of the confidence intervals for both classes of models considered+ In fact, both methods seem to behave very similarly taking into account the standard error !p~1 [ 2 p!0100 [ of the estimated coverage probability p+[ Apart from this overall behavior, it seems that in the case of the moving average model ~4+9! the block bootstrap method is slightly more efficient than the local resampling method in terms of the average length of the bootstrap confidence intervals obtained+ On the other hand, and in terms of the same quantity, the local bootstrap method is quite a bit more efficient than the block bootstrap method in the case of the random coefficient autoregressive process ~4+8!; this is not surprising because model ~4+8! is a true first order Markov process+

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

551

5. DERIVATIONS To prove Theorem 3+1 we make use of some lemmas that are established in what follows+ We begin with two lemmas concerning some asymptotic properties of the bootstrap series $X t* % and that are reproduced from and proved in Paparoditis and Politis ~1997!+ To state these lemmas we fix some additional notation+ * # x2 , + + + , For x 5 ~ x 1 , x 2 , + + + , x p ! [ R p let FT* ~x! 5 P~X t* # x 1 , X t21 * X t2p11 # x p ! be the stationary distribution function of the bootstrap Markov chain and F~x! 5 P~X t # x 1 , X t21 # x 2 , + + + , X t2p11 # x p ! the stationary distribution function of the original process $X t , t $ 1%; that is, we set F~{! 5 FYt, p ~{!+ LEMMA 5+1+ Suppose that Assumptions (A1)–(A3) and (A5) hold. Then conditionally on X 1 , X 2 , + + + , X T we have that, as T r `, sup 6FT* ~x! 2 F~x!6 r 0

a+s+

x[S

LEMMA 5+2+ Under the same assumptions as in Lemma 5.1 and conditionally on X 1 , X 2 , + + + , X T , T0 [ N exists such that for all T $ T0 the bootstrap series $X t* , t $ 1% is strongly mixing where the strong mixing coefficient a * satisfies a * ~n! # C® n for some constants C . 0 and ® [ ~0,1! that are independent of T. LEMMA 5+3+ Suppose that Assumptions (A1)–(A5) hold and let f be a realvalued, continuous function, y a fixed real number, and s a positive integer. We then have that, as T r `,

sup x[S

*

T21

h s21

( f~Xl11 !khs ~ y 2 Xl11 !Wb ~x 2 Yl, p !

l5p

T21

( Wb ~x 2 Yn, p !

n5p

2 f~ y! fXt11 6Yt, p ~ y6x!

E

*

k s ~u!du r 0

in probability. Proof+ Let N~x; f! 5 f~ y! fXt11 Yt, p ~ y, x! *k s ~u!du, NT ~x; f! 5

h s21 T

T21

f~X l11 !khs ~ y 2 X l11 !Wb ~x 2 Yl, p !, ( l5p

552

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

fbZ ~x! 5 T 21 ( T21 n5p Wb ~x 2 Yn, p !, and f ~x! 5 fYt, p ~x!+ Note that

*

T21

h s21

( f~Xl11 !khs ~ y 2 Xl11 !Wb ~x 2 Yl, p !

l5p

2 f~ y! fXt11 6Yt, p ~ y6x!

T21

#

1 fbZ ~x!

H

( Wb ~x 2 Yn, p !

n5p

6NT ~x; f! 2 N~x; f!6 1

6N~x; f!6 f ~x!

E

k s ~u!du

J

6 fbZ ~x! 2 f ~x!6 +

*

(5.10)

In the following discussion we show that supx[S 6NT ~x; f! 2 N~x; f!6 r 0 in probability+ Because supx[S 6 fbZ ~x! 2 f ~x!6 r 0 follows by the same arguments, the lemma is established using ~5+10! and ~A1!~iv!+ Consider 6NT ~x; f! 2 N~x; f!6+ For this term we have 6NT ~x; f! 2 N~x; f!6

* E f~z !k ~ y 2 z !W ~x 2 z ! f ~z , z !dz dz 2 f~ y! f ~ y, x! E k ~u!du * 1 1 h * ( f~X !k ~ y 2 X !W ~x 2 Y ! T

# h s21

s h

1

1

b

2

X t11 Yt, p

1

2

1

2

s

X t11 Yt, p T21

s21

l11

s h

l11

b

l, p

l5p

2

E

f~z 1 !khs ~ y 2 z 1 !Wb ~x 2 z 2 ! fXt11 Yt, p ~z 1 , z 2 !dz 1 dz 2

*

5 T1 ~x! 1 T2 ~x! with an obvious notation for T1~x! and T2 ~x!+ Consider first T1~x!+ Using the substitutions z 1 5 y 2 u 1 h and z 2 5 x 2 u 2 b we have T1 ~x! 5

* E~f~ y 2 u h! f 1

X t11 , Yt, p ~ y

2 u 1 h, x 2 u 2 b!

2 f~ y! fXt11 , Yt, p ~ y, x!!k s ~u 1 !W~u 2 !du 1 du 2

#

E*

*

f~ y 2 u 1 h! fXt11 , Yt, p ~ y 2 u 1 h, x 2 u 2 b! 2 f~ y! fXt11 , Yt, p ~ y, x!

*

3 k s ~u 1 !W~u 2 !du 1 du 2 5 O~max$h, b%! uniformly in x by ~A2!~ii!+ Consider next T2 ~x! and observe that because f is continuous and K is bounded and compactly supported, f~X l11 !k s ~~ y 2 X l11 !0h! is bounded+ Fur-

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

553

s thermore, instead of h s21 T 21 ( T21 l5p f~X l11 !kh ~ y 2 X l11 !Wb ~x 2 Yl, p ! we consider in the following discussion the asymptotically equivalent statistic R T ~ y, x! 5 s h s21 ~T 2 p!21 ( T21 l5p f~X l11 !kh ~ y 2 X l11 !Wb ~x 2 Yl, p !+ Observe that

E~R T ~ y, x!! 5 h s21

E

f~z 1 !khs ~ y 2 z 1 !Wb ~x 2 z 2 ! fXt11 Yt, p ~z 1 , z 2 !dz 1 dz 2

and that sup T2 ~x! # sup 6R T ~ y, x! 2 E~R T ~ y, x!!6 1 oP ~1!+ x[S

x[S

Thus we have to show that supx[S 6R T ~ y, x! 2 E~R T ~ y, x!!6 r 0 in probability+ To deal with this term divide S in a number of NT cubes denoted by I i, T with centers x i and length L T + We then get sup 6R T ~ y, x! 2 E~R T ~ y, x!!6 x[S

sup 6R T ~ y, x! 2 R T ~ y, x i !6

# max

1#i#NT x[SùI i, T

1 max 6R T ~ y, x i ! 2 E~R T ~ y, x i !!6 1#i#NT

1 max

sup 6E~R T ~ y, x i !! 2 E~R T ~ y, x!!6

1#i#NT x[SùI i, T

5 M1, T 1 M2, T 1 M3, T + By the Lipschitz continuity of K it is easily seen that M1, T 5 OP ~L T b 2~ p11! ! and M3, T 5 OP ~L T b 2~ p11! !+ Furthermore, P

S max 6R ~ y, x ! 2 E~R ~ y, x !!6 . L b 1#i#NT

NT

#

T

(P

i51

i

S

T

1 ~T 2 p!hb p

i

*(v

T

T21 l5p

l, T ~ y, x i !

2~ p11!

D

*.L b T

2~ p11!

D

,

where vl, T ~ y, x i ! 5 f~X l11 !k s

F

S

DS D S D S DG

y 2 X l11 x i 2 Yl, p W h b

2 E f~X l11 !k s

y 2 X l11 x i 2 Yl, p W h b

554

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

and E~vl,2 T ~ y, x!! # s 2 5 C 2 hb p with C . 0 a generic constant independent of T+ To bound this probability let first vJ l, T ~ y, x i ! 5 C 21 vl, T ~x i ! and LT 5

~log~T 2 p!! 3 !b 21p1e

!T 2 p !h 12e

for some 0 , e , 1+ Applying Proposition 1 of Doukhan ~1994, p+ 33!, with s 5 !hb p and log s 21 5 log!hb p 21 and using h0h 12e . 1 and b 21p1e0 b 21p . 1, we then have that constants a . 0 and b . 0 exist such that NT

(P i51

S

* ( vJ T21

1 ~T 2 p!hb p

l, T ~ y, x i !

l5p

S* ( S* (

*

T21

# O~NT !P

vJ l, T ~ y, x i ! .

l5p

*

T21

# O~NT !P

*. C L b

vJ l, T ~ y, x i ! .

l5p

1

T

2~ p11!

~log~T 2 p!! 3 log!hb p 21 1 C

D

!T 2 p !hb p log!hb p 21

D

~log~T 2 p!! 2 !T 2 p !hb p log!hb p 21

D

# O~NT !a exp $2b log~T 2 p!Y!C% 5 o~1!+

(5.11)

Thus for b 5 O~T T r `+

2d

! and d , 10p we get that L T 0b

~ p11!

r 0 and L T r 0 as

n

The following result deals with a well known property of the kernel estimator m[ h ~x; f!+ Its proof is omitted because it closely follows that of Lemma 5+3+ LEMMA 5+4+ Suppose that Assumptions (A1)–(A4) hold and let w be a realvalued continuous function. Then as T r ` sup 6 m[ h ~x; w! 2 m~x; w!6 r 0 x[S

in probability. Let f ~{6x! denote the conditional density fXt11 6Yt, p ~{6x! and let f Z ~ y6x! be the kernel estimator of f ~ y6x! given by T21

( kh ~ y 2 Xl11 !Wb ~x 2 Yl, p !

f Z ~ y6x! 5

l5p

T21

,

(5.12)

( Wb ~x 2 Yn, p !

n5p

where y [ S+ Note that in the notation of Lemma 5+3 and if we set f~{! [ 1 and s 5 1 we immediately get that supx[S 6 f Z ~ y6x! 2 f ~ y6x!6 r 0 in probability as T r `+

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

555

To simplify calculations and because the results obtained are asymptotic for T r `, we assume in the following that Yp,* p ; FT* , that is, we assume that the bootstrap series starts with its stationary distribution+ Now, for k [ N let x jk 5 ~ x jk , x jk11 , + + + , x jk1p21 ! [ X p and recall that $X t* , t $ 1% is a discrete Markov chain with state space X p + Using iteratively the relation T

P~Yt,* p 5 x j1 ! 5

* * P~X t* 5 x j 6Yt21, p 5 x j !P~Yt21, p 5 x j ! ( 5p11 1

2

2

jp11

and the transition probability ~2+3! we get the following useful expression for the stationary probability mass function of Yt,* p+ * * * P~X t* 5 x j 6 X t21 5 x j , X t22 5 x j , + + + , X t2p 5 xj ( ( {{{( j

P~Yt,* p 5 x j1 ! 5

1

jp11 jp12

2

3

p11

!

2p

* * * * 3 P~X t21 5 x j 2 6 X t22 5 x j3 , X t23 5 x j4 , + + + , X t2p21 5 x jp12 ! * * 5 x jp 6 X t2p 5 x jp11 , 3 {{{ 3 P~X t2p11 * * 5 x jp12 , + + + , X t22p11 5 x j 2p ! X t2p21 * * * 5 x jp11 , X t2p21 5 x jp12 , + + + , X t22p11 5 x j 2p ! 3 P~X t2p

E) E) p

5

* * * P~X t2i11 5 x ji 6Yt2i, p 5 x ji11 ! dFT ~x jp11 !

i51

Wb ~x ji11 2 Yji 21, p !

p

5

i51

T21

( Wb ~x ji11 2 Yn, p !

dFT* ~x jp11 !+

(5.13)

n5p

To proceed with the proof of Theorem 3+1 we first note that L*T ~x! can be expressed as L*T ~x! 5

1 5

! hT

p T21

1 fhZ * ~x!

1

! hT

p T21

1 fhZ * ~x!

fhZ * ~x!

* * ! 2 E * @f~X j11 !6Yj,*p # !Kh ~x 2 Yj,* p ! ( ~f~Xj11

j5p

* !6Yj,*p # 2 m * ~x; f!!Kh ~x 2 Yj,* p ! ( ~E * @f~Xj11

j5p

$L*1, T ~x! 1 L*2, T ~x!%,

* * where fhZ * ~x! 5 T 21 ( T21 j5p Kh ~x 2 Yj, p ! and an obvious notation for L 1, T ~x! * and L 2, T ~x!+ Using the preceding expression, the assertion of the theorem is established if we show that fhZ * ~x! r f ~x! weakly, that L*1, T ~x! converges to a Gaussian distribution with mean zero and the appropriate variance, and that L*2, T ~x!0fhZ * ~x! is a consistent estimator of the mean ~bias term! of L T ~x!+ This is established in the following three lemmas+

556

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

LEMMA 5+5+ Under Assumptions (A1)–(A5) we have conditionally on X 1 , X 2 , + + + , X T that for all x [ S, fhZ * ~x! r f ~x!

in probability.

Proof+ Let x 5 ~ x 1 , x 2 , + + + , x p !+ Using ~5+13! we have E * ~ fhZ * ~x!! 5 5

T2p T T2p T

(

Kh ~x 2 x j1 !P~Yt,* p 5 x j1 !

j1 , j 2 , + + + , jp

E

p

p

( ) ) j , j , + + + , j m51 i51 1

2

kh ~ x m 2 x jm !Wb ~x ji11 2 Yji 21, p ! T21

( Wb ~x j

p

i11

dFT* ~x jp11 !

2 Yn, p !

n5p

5

T2p T

E

p

p

( ) ) f Z ~ x 1 6x j ! j , + + + , j m52 i52

kh ~ x m 2 x jm !Wb ~x ji11 2 Yji 21, p ! T21

2

2

( Wb ~x j

p

i11

dFT* ~x jp11 !

2 Yn, p !

n5p

5

T2p T

E

p

p

( ) ) f ~ x 1 6x j ! j , + + + , j m52 i52

kh ~ x m 2 x jm !Wb ~x ji11 2 Yji 21, p ! T21

2

2

( Wb ~x j

p

i11

dFT* ~x jp11 !

2 Yn, p !

n5p

1 oP ~1!

5

T2p T

E

p

p

( ) )

(j f ~ x 1 6x j !kh ~ x 2 2 x j !Wb ~x j 2

2

3

2 Yj 221, p !

2

T21

( Wb ~x j

j 3 , + + + , jp m53 i53

3

2 Yn, p !

n5p

3

kh ~ x m 2 x jm !Wb ~x ji11 2 Yji 21, p ! T21

( Wb ~x j

i11

dFT* ~x jp11 ! 1 oP ~1!

2 Yn, p !

n5p

5

T2p T 3

E

p

p

( ) ) f ~ x16 x2 , xj , + + + , xj 3

p11

! f ~ x 2 6x j3 !

j 3 , + + + , jp m53 i53

kh ~ x m 2 x jm !Wb ~x ji11 2 Yji 21, p ! T21

( Wb ~x j

i11

dFT* ~x jp11 ! 1 oP ~1!,

2 Yn, p !

n5p

where the preceding oP ~1! terms are due to the uniform ~over x! convergence of f Z ~{6x! given in ~5+12! to f ~{6x! and Lemma 5+3+ Continuing in this way we end up with the expression

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

E * ~ fhZ * ~x!! 5

T2p T

E

557

f ~ x 1 6 x 2 , + + + , x p , x jp11 ! f ~ x 2 6 x 3 , + + + , x p , x jp11 , x jp12 !

3 {{{ 3 f ~ x p 6 x jp11 , + + + , x j 2p ! dFT* ~x jp11 ! 1 oP ~1! r

E

f ~ x 1 6 x 2 , + + + , x p , x jp11 ! f ~ x 2 6 x 3 , + + + , x p , x jp11 , x jp12 !

3 {{{ 3 f ~ x p 6 x jp11 , + + + , x j 2p ! dF~x jp11 ! 5 f ~x!,

(5.14)

where the last convergence is due to Lemma 5+1 and Helly–Bray’s theorem+ To evaluate the variance of fhZ * ~x!, let Z h,* j 5 Kh ~x 2 Yj,* p ! and note that * Var * @ fhZ * ~x!# 5 ~T 2 p!T 22 Var * ~Z h,1 ! T2p21

1 2T 21

(

t51

* * ~1 2 t0T !Cov * ~Z h,1 , Z h,11t !+

(5.15)

Now, h p Var * @Z h,* j # # h p (j1 , j 2 , + + + , jp Kh2 ~x 2 x j1 !P~Yt,* p 5 x j1 ! and by the same arguments as those leading to ~5+14! we get h p Var * @Z h,* j # r f ~x! *K 2 ~u!du, that is, T 21 Var * @Z h,* j # 5 OP ~T 21 h 2p !+ To handle the second term on the right hand side of ~5+15! note first that by Davydov’s lemma ~Hall and Heyde, 1980, App+! we have * * * n 20n 6Cov~Z h,1 , Z h,11t !6 # 8a * ~t!1220n ~E * 6 Z h,1 6 ! * n for some real n . 2+ Because E * 6 Z h,1 6 5 (j1 , j 2 , + + + , jp Khn ~x 2 x j1 !P~Yt,* p 5 ~12n!p x j1 ! 5 OP ~h ! by the same arguments as in establishing that ~5+14! is true, we get by Lemma 5+2 and arguments similar to those in Masry and Tjøst* * ~1 2 t0T !Cov * ~Z h,1 , Z h,11t !6 5 heim ~1995, pp+ 271–272! that 6T 21 ( T2p21 t51 21 2p oP ~T h !+ n

LEMMA 5+6+ Under Assumptions (A1)–(A5) we have conditionally on X 1 , X 2 , + + + , X T that, as T r `,

! hT

p T21

* ! 2 m b* ~Yj,* p ; f!!Kh ~ x 2 Yj,* p ! n N~0, t 2 ~x; f!! ( ~f~Xj11

in probability,

j5p

where t 2 ~x; f! 5 *K 2 ~u!du f ~x! Var @f~X t11 !6Yt, p 5 x#. Proof+ Define

! hT ~f~X p

* [ Dj11

* j11 !

2 m b* ~Yj,* p ; f!!Kh ~x 2 Yj* !

and denote by Fj * 5 s~X 1* , X 2* , + + + , X j* ! the s-algebra generated by $X 1* , * !6Yt,* p 5 x# we have X 2* , + + + , X j* %+ Because for x [ X p , m b* ~x; f! 5 E * @f~X t11

558

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

* by the Markov property that $Dj11 , Fj * % j5p, p11, + + + forms a martingale difference scheme+ Thus to establish the desired result, we apply Theorem 1 of Brown ~1971!+ By this theorem we have to show that T21

* 6Fj * # r ( E * @Dj11 j5p 2

E

K 2 ~u!du f ~x!Var @f~X t11 !6Yt, p 5 x#

in probability (5.16)

and that for every d . 0 the Linderberg condition T21

* * I ~Dj11 $ d!6Fj * # r 0 ( E * @Dj11 j5p 2

2

in probability

(5.17)

is satisfied+ To simplify notation and to stress the essentials, we focus in what follows on the case p 5 1+ The case of general p can be handled along the same lines+ Consider first ~5+16!+ Because 2

* E * @Dj11 6Fj * # 5

h T

T

* 5 X l 6 X j* ! ( ~f~Xl ! 2 m b*~Xj* ; f!! 2 Kh2 ~ x 2 Xj* !P~Xj11 l52

we have T21

( j51

2

* E * @Dj11 6 Fj * # 5

1 Th

T

( K2

j51

S D x 2 X j* h

v[ b* ~X j* ; f!,

where T

v[ b* ~z; f! 5

( ~f~Xl ! 2 m b* ~z; f!! 2 Wb ~z 2 Xl21 ! l52 +

T21

( Wb ~z 2 Xs !

s51

Let v~z; f! 5 E @~f~X t11 ! 2 m~z; f!! 2 6 X t 5 z# and note that as in the proof of Lemma 5+5, ~Th!21 ( Tj51 K 2 ~~ x 2 X j* !0h! 5 OP ~1!+ Then T

1

T

S D D ( S

* 6Fj * # 5 ( E * @Dj11 ( K2 Th j51 j51 2

1

1 Th

x 2 X j*

T

K2

j51

h

v~X j* ; f!

x 2 X j* h

~ v[ b* ~X j* ; f! 2 v~X j* ; f!!+

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

559

Now, because v[ b* ~z; f! 5 m b* ~z; f 2 ! 2 ~m * ~z; f!! 2 is a Nadaraya–Watson kernel estimator of v~z; f! 5 E @~f~Xt11 ! 2 m~z; f!! 2 6Xt 5 z# we have for Dh 5 @x 2 h, x 1 h# that sup 6 v[ b* ~z; f! 2 v~z; f!6 # sup 6m b* ~z; f 2 ! 2 m~z; f 2 !6 z[Dh

z[Dh

1 sup 6m b* ~z; f! 2 m~z; f!6 sup 6m b* ~z; f! 1 m~z; f!6 z[Dh

z[Dh

r0 in probability by Lemma 5+4+ Thus 1 Th

T21

( K2 j51

S D x 2 X j* h

6 v[ b* ~X j* ; f! 2 v~X j* ; f!6

# sup 6 v[ b* ~z; f! 2 v~z; f!6OP ~1! r 0+ z[Dh

To conclude the proof of ~5+16! we show that VT* ~ x; f! [

1 Th

T

( K2 j51

S D x 2 X j* h

v~X j* ; f! r t 2 ~ x; f!

(5.18)

in probability+ Taking expectation of this term with respect to the bootstrap distribution we get using the same arguments as in establishing ~5+14! that T

E * @VT* ~ x; f!# 5

E

h 21

( v~Xl ; f!~K 2 ~~ x 2 Xl !0h!!Wb ~Xm 2 Xl21 !

l52

T21

( Wb ~Xm 2 Xs !

dFT* ~X m !

s51

r

E

K 2 ~u!duf ~ x!Var @f~X t11 !6 X t 5 x# ,

where the last convergence follows using Lemma 5+3+ Furthermore, along the same lines as in the proof of ~5+15! we get that Var * @VT* ~ x; f!# 5 OP ~T 21 h 2p !+ To prove ~5+17! we use the inequality E @6 X6I ~6 X6 . C!# # C 21 E6 X6 2 where C . 0 is a constant provided the random variable X satisfies E6 X6 2 , `+ We then have for every d . 0

560

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

T21

* * I ~Dj11 $ d!6Fj * # ( E * @Dj11 2

2

j51

# 5

1 d

T21

* 6Fj * # ( E * @Dj11 j51 4

T

1 dh 2 T 2

F

* ! 2 m b* ~X j* ; f!% 4 K 4 ( E * $f~Xj11

j51

S D G x 2 X j* h

6Fj *

T

5

1 dh 2 T 2

T21

( K 4 ~~ x 2 Xj* !0h!

( ~f~Xl ! 2 m b* ~Xj* ; f!! 4 Wb ~Xj* 2 Xl21 ! l52 +

T21

(

j51

Wb ~X j*

2 Xs !

s51

Let T

R T ~z; f! 5

~f~X l ! 2 m b* ~z; f!! 4 Wb ~z 2 X l ! ( l52 ,

T21

( Wb ~z 2 Xs !

n51

which is a Nadaraya–Watson estimator of R~z; f! 5 E @~f~Xt11 ! 2 m~z; f!! 4 6Xt 5 x# , and note that supz[Dh 6R T ~z; f! 2 R~z; f!6 r 0 in probability+ Thus T21

(

2

2

* * E * @Dj11 I ~Dj11 $ d!6Fj * # #

j51

1 dT 2 h 2

T21

( K 4 ~~ x 2 Xj* !0h!R T ~Xj* ; f! 1 oP ~1!

j51

5 OP ~d 21 T 21 h 21 !, where the last equality follows by the same arguments as those used to prove n ~5+18!+ LEMMA 5+7+ Under Assumptions (A1)–(A6) and if T 102 h ~ p14!02 r Ch . 0, we have conditionally on X 1 , X 2 , + + + , X T that, as T r `,

! hT

p T21

* !6Yj,* p # 2 m b* ~x; f!!Kh ~x 2 Yj,* p ! ( ~E * @f~Xj11

j5p

r Ch

E

p

H

u 2 k~u!du ( m ~ x l ! ~x; f! fYt, pl ~x! 1 l51

~x !

1 ~x x ! m l l ~x; f! fYt, p ~x! 2

J (5.19)

in probability. * Proof+ Let m b* ~Yt,* p ; f! 5 E * @f~X t11 !6Yt,* p # and x 5 ~ x 1 , x 2 , + + + , x p !+ By a Taylor series expansion we get

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

! hT

561

p T21

( ~m b* ~Yj,* p ; f! 2 m b* ~x; f!!Kh ~x 2 Yj,* p !

j5p

p

5

( m b*

~ xl !

~x; f!

l51

1

1 2

p

!

p

((

l 151 l 251

m b*

hp T

T21

* 2 x l !Kh ~x 2 Yj,* p ! ( ~Xj2l11

j5p

~ x l1 x l2 !

~ x;I f!

!

hp T

T21

* * 11 2 x l !~X j2l 11 2 x l ! ( ~Xj2l 1

1

2

2

j5p

3 Kh ~x 2 Yj,* p ! for some xI between x and Yj,* p+ Because under the assumptions made !Th p ~Em[ h ~x; f! 2 m~x; f!! r B~x; f! in probability, the assertions of the lemma will be established if we show that, in probability, m b*

!T

hp

~ xl !

~x; f! r m ~ x l ! ~x; f!,

T21

* 2 x l !Kh ~x 2 Yj,* p ! r Ch ( ~Xj2l11

j5p

E

(5.20) ~x !

u 2 k~u!dufYt, pl ~x!,

(5.21)

! hT

p T21

* * * 11 2 x l !~X j2l 11 2 x l !Kh ~x 2 Yj, p ! r dl , l Ch ( ~Xj2l 1

1

2

2

1

j5p

2

E

u 2 k~u!dufYt, p ~x!,

(5.22) and sup

6m b*

~ x l1 x l2 !

~z; f! 2 mK ~ x l

1

x l2 !

~z; f!6 r 0,

(5.23)

z[@x2h, x1h#

where mK ~ x i x j ! ~z; f! 5

H

1 1 ~1 1 di, j ! fYt, p ~z! 2

E

~x x !

~x x !

f~ y! fXt11i Yj t, p ~ y, z!dy 2 m~z; f! fYt, pi j ~z! ~x !

J

~x !

2 m ~ x i ! ~z; f! fYt, pj ~z! 2 m ~ x j ! ~z; f! fYt, pi ~z! + The rest of the proof is devoted to showing that ~5+20!–~5+23! are true+ Consider ~5+20! and note that m b*

~ xl !

~x; f! 5

H

1 1 fbZ ~x! Tb p11

T21

( f~Xj11 !W ~ x ! ~~x 2 Yj, p !0b! l

j5p

2 m b* ~x; f!

1 Tb p11

T21

W ~ x ! ~~x 2 Yj, p !0b! ( j5p l

J

+

(5.24)

562

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

Recall that fbZ ~x! r fYt, p ~x! and m b* ~x; f! r m~x; f! in probability+ Furthermore, by standard arguments we have 1 Tb p11

T21

( f~Xj11 !W ~ x ! ~~x 2 Yj, p !0b! r j5p l

E

~x !

f~ y! fXt11l Yt, p ~ y, x! dy

(5.25)

and T21

1

~x ! W ~ x ! ~~x 2 Yj, p !0b! r fY ~x!+ ( j5p

Tb

(5.26)

l t, p

l

p11

~ xl !

Therefore, m b* ~x; f! r m ~ x l ! ~x; f! in probability+ The proof of ~5+21! and ~5+22! is very similar and makes explicit use of ~5+13! and arguments like those applied in the proof of Lemma 5+3+ To avoid tedious manipulations of formulas we focus on the case p 5 1+ Consider ~5+21! so we have E*

F!

h T

G

T21

~X j* 2 x!Kh ~ x 2 X j* ! ( j51

5 ~T 2 1!

!

h T

T

( ~Xl 2 x!Kh ~ x 2 Xl !P~Xt* 5 Xl !

l52

T

! E

h 5 ~T 2 1! T

( ~Xl 2 x!Kh ~ x 2 Xl !Wb ~Xs 2 Xl21 !

l52

dFT* ~X s !+

T

( Wb ~Xs 2 Xn !

n52

Now, to establish the desired result we have to show that

sup y[S

*

T

!Th ( ~Xl 2 x!Kh ~ x 2 Xl !Wb ~ y 2 Xl21 ! l52

~ x!

2 Ch

T21

( Wb ~ y 2 Xn21 !

fXt Xt21 ~ x, y! fXt ~ y!

E

u 2 K~u!du

n51

r 0+

*

(5.27)

By ~A1!~iv! and the uniform convergence of fbZ ~ y! to f ~ y!, it suffices to show that h * ! T ( ~X 2 x!K ~ x 2 X !W ~ y 2 X T

sup y[S

l

l52

r 0+

h

l

b

l21 !

~ x!

2 Ch fXt Xt21 ~ x, y!

E

u 2 K~u!du

*

(5.28)

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

For this note that E

F!

h T

563

G

T

( ~Xl 2 x!Kh ~ x 2 Xl !Wb ~ y 2 Xl21 !

l52

5 2~T 2 1! 5 ~T 2 1!

! E uK~u!W~r! f h3 T

! E u K~u!W~r! f h5 T

2

X t X t21 ~ x

~ x! I X t X t21 ~ x,

2 uh, y 2 rb!drdu y 2 rb!drdu,

where 6 xI 2 x6 , h+ Thus by the Lipschitz continuity of f ~ x! ~{,{! we get that

! hT E u K~u!W~r! f 5

~T 2 1!

2

~ x!

uniformly in y; that is, we have shown that

h * F ! T ( ~X 2 x!K ~ x 2 X !W ~ y 2 X T

sup E y[S

l

~ x!

E

u 2 K~u!du

~ x!

E

*

~ x,I y 2 rb!drdu r Ch fXt Xt21 ~ x, y!

h

l

b

G

l21 !

l52

r0

2 Ch fXt Xt21 ~ x, y!

u 2 K~u!du

in probability+ Thus to establish ~5+28! it remains to show that h * ! T ( $~X 2 x!K ~ x 2 X !W ~ y 2 X T

sup y[S

l

h

l

b

l21 !

l52

*

2 E~X l 2 x!Kh ~ x 2 X l !Wb ~ y 2 X l21 !% r 0 in probability+ To handle this term, we proceed as in the proof that M1, T r 0 in probability in Lemma 5+3+ In particular, let Z l, T ~ y! 5 ~X l 2 x!K

S

DS D S DS D

x 2 Xl W h

2 E~X l 2 x!K

y 2 X l21 b

x 2 Xl W h

y 2 X l21 b

and note that 6 Z l, T 6 # C1 h and that EZ 2 ~ y! # C22 h 3 b for some generic positive constants C1 and C2 + Using the splitting device applied in the proof of Lemma 5+3 we get by the Lipschitz continuity of W that sup y[S

*

1 !Thb

max * ( Z l, T ~ y! * # 1#i#N ( Z l, T ~ yi ! * 1 O~h 102 L T b22 !+ !Thb l52 l52 T

1

T

T

Along the same lines as in dealing with ~5+11! we get for the first term on the right hand side of the preceding inequality using the notation ZE l, T ~ yi ! 5 h 21 C221 Z l, T ~ yi ! and for 0 , e , 1 and L T 5 ~log~T !! 3 b 3021e h 1021e that

564

P

S

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

max 1#i#NT

*

NT

# #

*

T

1 !Thb

L T h 102 b2

( Z l, T ~ yi ! $

S S* (

l52

D

D

( P * l52 ( ZE l, T ~ yi ! * $ i51

L T T 102 C2 b

NT

~log~T !! 2 !T !hb log!hb 21 C2

T

(P i51

*

T

ZE l, T ~ yi ! $

l52

# O~NT !a exp $2b log~T !Y C2 %

D

!

for some a . 0 and b . 0+ Thus by the assumptions made on b and h we get that L T r 0, L T h 1020b 2 r 0, and O~NT !a exp $2b log~T !0C2 % r 0 as T r `+ The proof of ~5+22! follows exactly the same lines as the proof of ~5+21! and is therefore omitted+ To establish ~5+23! verify first that m b*

~ x l1 , x l2 !

5

~z; f!

H

1 1 fbZ ~z! Tb p12 2 m b* 2 m b*

T21

( f~Xj11 !W ~ x

l1 ,

x l2 !

~~z 2 Yj, p !0b!

j5p

~ x l2 !

~ x l1 !

~z; f!

1 Tb p12 ~ x l2 !

~z; f! fbZ

T21

( W ~x

l1 ,

x l2 !

~~z 2 Yj, p !0b!

j5p

~z! 2 m b*

~ x l2 !

~ x l1 !

~z; f! fbZ

J

~z! +

(5.29)

We then have to show that every element on the right hand side of ~5+29! converges uniformly to the corresponding elements on the right hand side of ~5+24!+ We show this for the first term on the right hand side of ~5+29!, the other terms being handled in exactly the same manner+ For this note that sup z[S

*

1 T

T21

( f~Xj11 !Wb~ x j5p

l1 x l2 !

~z 2 Yj, p ! 2

* T ( ~f~X !W T2p 1 sup * Ef~ y!W T

# sup

1

z[S

T21

~ x l1 x l2 ! ~z b

j11

1 ~1 1 dx l 2

1

x l2 !

E

~x 1 x 2!

~ x l1 x l2 !

2 Yj, p ! 2 Ef~X j11 !Wb

j5p

~ x l1 x l2 ! ~z b

z[S

2

1 1 dx l 2

1

x l2

E

* !! *

f~ y! fXt11l Ylt, p ~ y, z! dy ~z 2 Yj, p

2 r! fXt11 Yt, p ~ y, r! dydr

~x 1 x 2!

f~ y! fXt11l Ylt, p ~ y, z! dy

*

5 B1, T 1 B2, T , with an obvious notation for B1, T and B2, T + Now, by the same arguments used to handle the term T1~x! in the proof of Lemma 5+3 and by ~A6!, we get that B2, T 5 O~b!+ Furthermore, by the mean value theorem we have applying the

MARKOVIAN LOCAL RESAMPLING OF KERNEL ESTIMATORS

565

same splitting device as the one used in handling the term T2 ~x! in the lemma mentioned earlier, that sup z[S

* T ( ~f~X !W ~z 2 Y 1 # max * ( ~f~X !W T 1

T21

~ x l1 x l2 ! b

j11

j, p !

~ x l1 x l2 !

2 Ef~X j11 !Wb

~z 2 Yj, p !!

j5p

T21

j11

1#i#NT

~ x l1 x l2 ! ~z i b

*

~ x l1 x l2 !

2 Yj, p ! 2 Ef~X j11 !Wb

j5p

~z i 2 Yj, p !!

*

1 OP ~L T b 2~ p13! !+

To conclude the proof verify that Var~f~X j11 !W ~ x l x l ! ~~z i 2 Yj, p !0b!! # Cb p and let L T 5 ~log T ! 3 b ~ p02!111eY!T for some 0 , e , 1+ Using the notation 1

2

Cj, T ~z i ! 5 f~X j11 !W ~ x l1 x l2 ! ~~z i 2 Yj, p !0b! 2 E~f~X j11 !W ~ x l1 x l2 ! ~~z i 2 Yj, p !0b!!

and applying the exponential inequality for strongly mixing sequences used in the proof of Lemma 5+3 we then get that for some constants a . 0 and b . 0 NT

(P i51

S

1 Tb p12 NT

#

(P

i51

T21

( Cj, T ~zi ! $ L T b2~ p13!

j5p

S(

T21 j5p

C 21 Cj, T ~z i ! $

!T L T Cb

~ p02!11

# O~NT !a exp $2b log~T !Y!C% 5 o~1!+

D

log!b

p02 21

!T b p02 log!b p02 21

D n

REFERENCES Auestad, B+ & D+ Tjøstheim ~1990! Identification of nonlinear time series: First order characterization and order determination+ Biometrika 77, 669– 687+ Brown, B+M+ ~1971! Martingale central limit theorems+ Annals of Mathematical Statistics 42, 59– 66+ Doukhan, P+ ~1994! Mixing: Properties and Examples+ Lecture Notes in Statistics 85+ New York: Springer-Verlag+ Efron, B+ ~1979! Bootstrap methods: Another look at the jackknife+ Annals of Statistics 7, 1–26+ Franke, J+, J+-P+ Kreiss, & E+ Mammen ~1996! Bootstrap of Kernel Smoothing in Nonlinear Time Series+ Preprint+ Hall, P+ & C+C+ Heyde ~1980! Martingale Limit Theory and Its Applications+ San Diego: Academic Press+ Künsch, H+R+ ~1989! The jackknife and the bootstrap for general stationary observations+ Annals of Statistics 17, 1217–1241+ Liu, R+ & K+ Singh ~1992! Moving blocks jackknife and bootstrap capture weak dependence+ In R+ LePage & L+ Billard ~eds+!, Exploring the Limits of the Bootstrap, pp+ 225–248+ New York: Wiley+ Masry, E+ & D+ Tjøstheim ~1995! Nonparametric estimation and identification of nonlinear ARCH time series+ Econometric Theory 11, 258–289+ Neumann, M+ & J+-P+ Kreiss ~1998! Regression-type inference in nonparametric autoregression+ Annals of Statistics 26, 1570–1613+ Paparoditis, E+ & D+N+ Politis ~1997! The local bootstrap for Markov processes+ Journal of Statistical Planning and Inference, forthcoming+ Robinson, P+M+ ~1983! Nonparametric estimation for time series models+ Journal of Time Series Analysis 4, 185–208+

566

EFSTATHIOS PAPARODITIS AND DIMITRIS N . POLITIS

Roussas, G+ ~1990! Nonparametric regression estimation under mixing conditions+ Stochastic Processes and Their Applications 36, 107–116+ Tjøstheim, D+ & B+ Auestad ~1994! Nonparametric identification of nonlinear time series: Projections+ Journal of the American Statistical Association 89, 1398–1409+ Tong, H+ ~1990! Nonlinear Time Series: A Dynamical Approach+ New York: Oxford University Press+