NONPARAMETRIC HETEROGENEITY TESTING FOR MASSIVE DATA
arXiv:1601.06212v1 [math.ST] 23 Jan 2016
By Junwei Lu‡ , Guang Cheng∗,§ and Han Liu†,‡ Princeton University‡ and Purdue University§ A massive dataset often consists of a growing number of (potentially) heterogeneous sub-populations. This paper is concerned about testing various forms of heterogeneity in the framework of nonparametric varying-coefficient models. By taking advantage of parallel computing, we design a set of heterogeneity testing procedures with reasonable scalability. In theory, their null limiting distributions are derived as being nearly Chi-square with diverging degrees of freedom as long as the number of sub-populations, denoted as s, does not grow too fast. Moreover, we characterize a lower bound on s that is needed to gain sufficient power in testing, so-called “blessing of aggregation” phenomenon. As a by-product, homogeneity testing is also considered with a test statistic being aggregated over all sub-populations. Numerical results are presented to support our theory.
1. Introduction. Massive datasets are generally collected from multiple and distributed sources at different time points by different experimental methods. Hence, it is commonly believed that a massive dataset consists of a growing number of (potentially) heterogeneous sub-populations. Heterogeneity testing thus becomes a necessary pre-processing step. In this paper, we aim to develop computationally feasible and theoretically valid heterogeneity testing in presence of a growing number of sub-populations. Varying coefficient model (Hastie and Tibshirani, 1993) is known to provide ∗
Associate Professor, Department of Statistics, Purdue University, West Lafayette, IN 47906. E-mail:
[email protected]. Tel: +1 (765) 496-9549. Fax: +1 (765) 494-0558. Research Sponsored by NSF CAREER Award DMS-1151692, DMS-1418042, Simons Fellowship in Mathematics, Office of Naval Research (ONR N00014-15-1-2331) and a grant from Indiana Clinical and Translational Sciences Institute. Guang Cheng was on sabbatical at Princeton while this work was finalized; he would like to thank the Princeton ORFE department for its hospitality and support. † Assistant Professor. Research Sponsored by NSF IIS1408910, NSF IIS1332109, NIH R01MH102339, NIH R01GM083084, and NIH R01HG06841. MSC 2010 subject classifications: Primary 62G10, 62G08; secondary 62E20 Keywords and phrases: Heterogeneity, kernel ridge regression, massive data, minimax optimality, nonparametric testing
1
2
J. LU, G. CHENG AND H. LIU
a flexible framework for modelling heterogeneous data as follows: e T β0 (X) + , (1.1) Y =Z
e T = (1, Z T ) and β0 (·) includes both commonality and where X is a scalar, Z heterogeneity components as illustrated by the following example. Cleveland et al. (1993) were interested in exploring how ethanol exhaustion NOx for a engine depends on the fuel-air mixture equivalence ratio E and the compression ratio of the engine C. They found that NOx is linear to C when E is fixed and nonlinear to E, which can be modelled as NOx = β0 (E) + β1 (E)C + . Nowadays, it becomes common to collect a huge amount of data from various types of engines in distributed places. In this case, one can reasonably assume that the slope β1 (E) potentially varies with the engine type, while the intercept β0 (E) is a shared factor across all vehicles. In reality, it is of interest in checking whether the effect of compression ratio C is the same for a selection of engine types, and also estimating the common factor β0 by utilizing the data collected from all engines. In view of the above discussions, we assume the following varying coefficient model (as a special case of (1.1)) in the j-th sub-population: (1.2)
Y = g0 (X) + Zβ0j (X) + , for j = 1, . . . , s,
where s denotes the total number of sub-populations. We remark that the homogeneity function g0 and heterogeneity function β0j do not necessarily share the same smoothness, and also that a multivariate extension of Z (and thus β0j ) is possible (but omitted for simplicity). The random covariate Z represents the magnitude of heterogeneity, and is assumed to be independent of X and . Note that allowing s → ∞ makes our work substantially different from existing nonparametric literature, e.g., Pardo-Fern´andez et al. (2007); Munk and Dette (1998); Srihera and Stute (2010); Chen and Zhong (2010) and Zhu et al. (2012). A main purpose of this paper is to develop large-scale heterogeneity testing with computational feasibility and theoretical guarantee. As a first step, we estimate both homogeneity function g0 and heterogeneity functions β0j ’s through a two-step procedure in a general kernel ridge regression framework (Shawe-Taylor and Cristianini, 2004). This requires a careful design and study of a general product Hilbert space. In theory, these two estimators are shown to achieve the so-called “oracle efficiency” in the sense that their (joint) asymptotic behaviors are exactly the same as those for oracle estimators (defined in Remark 3.5). This oracle rule holds as long as the number of sub-populations does not grow too fast. The obtained estimators will be
NONPARAMETRIC HETEROGENEITY TESTING
3
used to construct heterogeneity and homogeneity testings, and their oracle efficiency plays a key role in establishing nice theoretical properties of those testing methods. In this paper, we consider the following hypotheses for detecting various forms of heterogeneity: (a) Simple test: H0 : β0j = β0 . (b) Pairwise test: H0 : β0j = β0j 0 , for some j 6= j 0 . (c) Simultaneous test: H0 : β01 = · · · = β0s .
A set of test statistics is proposed for (a) – (c) based on the (penalized) likelihood ratio principle (Shang and Cheng, 2013). These testing methods are particularly designed to accommodate a growing number of sub-populations in the sense that they are much more computationally affordable than those commonly used nonparametric multiple-sample tests such as Pardo-Fern´ andez et al. (2007). A relevant heterogeneity testing is considered in Zhao et al. (2016), but only applied to parametric components in partially linear models. The diverging s brings challenges in analyzing theoretical properties of heterogeneity testing. In particular, we are interested in knowing how fast s is allowed to grow such that these testing methods are still consistent. For this purpose, we provide sufficient conditions on s under which null limiting distributions for (a) – (c) are derived as being nearly Chi-square with diverging degrees of freedom. Furthermore, their power performances are examined under a sequence of local alternatives, measured by separation rates. To obtain minimal separation rates, we need to borrow sufficient information across all the sub-populations to reduce the estimation error on g0 , which is now treated as a nuisance parameter. Interestingly, this requires s to diverge fast enough, called as “blessing of aggregation” phenomenon. As a by-product, we also consider homogeneity testing on g0 such as H0 : g0 = g0∗ for some specified g0∗ . Since g0 is shared among all subpopulations, we certainly want to construct our test statistic based on the entire massive data. This leads to our construction of an aggregated test statistics over s sub-populations. Our power analysis further illustrates that this aggregation mechanism yields higher testing sensitivity to a sequence of local alternatives as s grows properly. This again reflects “blessing of aggregation,” but from another perspective. Under a different upper bound on s, we derive a similar null limiting distribution as in the case of heterogeneity testing. We note that the effects of aggregation have recently been examined from many other viewpoints such as Chen and Xie (2012); Li et al. (2013); Kleiner et al. (2014); Lee et al. (2015); Battey et al. (2015); Wang et al. (2015); Huang and Huo (2015). However, none of these work takes into account of data heterogenetiy, which is a salient feature for massive data.
4
J. LU, G. CHENG AND H. LIU
In the end, we would like to comment on the selection of regularization parameters applied to g0 and β0j ’s. No matter for the purpose of testing or estimation, it is not surprising that the former parameter should be chosen in the scale of entire sample size, while the latter in the scale specific to the sample to which the heterogeneity function belongs. The only exception to this general principle is homogeneity testing in which both regularization parameters need to be in the scale of entire sample size. This is because we need to under-regularize heterogeneity estimators such that their biases play an asymptotically ignorable role in the homogeneity testing statistic. The rest of this paper is organized as follows. Section 2 reviews basic knowledge on reproducing kernel Hilbert space and further defines a type of product Hilbert space needed in our theoretical analysis. Basic model assumptions and a preliminary two step estimation procedure (together with oracle rule) are introduced in Section 3. Our general theory on heterogeneity and homogeneity testing are presented in Section 4, while more specific examples are discussed in Section 5. Simulations are given in Section 6. Notations: N = {0, 1, 2, . . .} represents a set of all nonnegative integers. Denote δjk as the Kronecker delta, where δjk = 1 if j = k and δjk = 0 if j = 6 k. For positive sequences an and bn , we write an . bn if there exists a constant c > 0 independent to n such that an ≤ cbn for all n ∈ N and an & bn if an ≤ c0 bn for some universal constant c0 . We denote an bn if both an . bn and an & bn . We also denote an . bn as an = O(bn ). Write x ∨ y = max(x, y), x ∧ y = min(x, y), and denote supx∈X |f (x)| as kf ksup . 2. Background. In this section, we provide some necessary background on the reproducing kernel Hilbert space (RKHS). 2.1. Reproducing Kernel Hilbert Space. Let X be a subset of R and P be a probability measure defined on the X . Given an RKHS H ⊆ L2 (P) with an inner product h·, ·iH and the induced norm k · kH , there exists a symmetric and square integrable function K : X × X → R satisfying hf, K(x, ·)iH = f (x), for all f ∈ H,
where Kx (·) := K(x, ·). We call K as the reproducing kernel of H. If K is also a continuous function on X × X , Mercer’s theorem (Riesz and Sz.-Nagy, 1955) guarantees that the kernel has the following expansion K(x, t) =
∞ X
µk φk (x)φk (t),
k=0
where {µk }∞ k=0 is a non-negative descending sequence of eigenvalues and ∞ {φk }k=0 are eigen-functions forming an orthonormal basis for L2 (P). In
NONPARAMETRIC HETEROGENEITY TESTING
5
particular, the eigen-system satisfies for any j, k ∈ N,
hφj , φk iL2 (P) = δjk and hφj , φk iH = δjk /µk .
The underlying smoothness of f ∈ H can be characterized by the decaying rate of the eigenvalues {µk }∞ k=0 . We focus on three typical decaying rates in this paper. As will be demonstrated later, the nonparametric inference for kernel ridge regression crucially depends on the decaying rate. Finite rank: There exists some r ∈ N such that µk = 0 for k > r. This gives arise to the so-called finite rank kernel (with rank r). For example, the linear kernel K(z1 , z2 ) = hz1 , z2 iRd has rank d, and generates a d-dimensional linear function space. The eigen-functions are given by φ` (z) = z` for ` = 1, . . . , d. For one dimensional z, the polynomial kernel K(z1 , z2 ) = (1+z1 z2 )m has rank m + 1, and induces a space of polynomial functions with degree at most m. The eigen-functions are given by φ` = z `−1 for ` = 1, . . . , m + 1. Exponentially decaying: There exist some α, p > 0 such that µk exp(−αk p ). Exponentially decaying kernels include the Gaussian kernel K(x, t) = exp(−kx − tk22 /σ 2 ). The corresponding eigen-functions are √ √ √ 2 (2.1) φk (x) = ( 5/4)1/4 (2k−1 k!)−1/2 e−( 5−1)x /4 Hk ( 5/2)1/2 x , √ and the eigenvalues are µk = `2k+1 , for k ∈ N, where ` = ( 5 − 1)/2 and Hk (·) is the k-th Hermite polynomial (Sollich and Williams, 2005). Polynomially decaying: There exists some m > 0 such that µk k −2m . The polynomially decaying function class includes Sobolev space and Besov Space (Birman and Solomjak, 1967). We denote an m-th order Sobolev space as Hm ([0, 1])1 . For its homogeneous subspace H0m ([0, 1])2 , we have trigonometric eigen-functions as follows: 1, k=0 √ 2 cos(2π`x), k = 2` for ` = 1, 2, . . . , (2.2) φk (x) = √ 2 sin(2π`x), k = 2` − 1 for ` = 1, 2, . . . , and eigenvalues are µ2`−1 = µ2` = (2π`)−2m for ` ≥ 0 and µ0 = ∞. The corresponding Sobolev kernels can be found in Gu (2013).
2.2. Product Hilbert Space. In this section, we define a type of product Hilbert space that is needed in analyzing our main model (1.2). Suppose the true functions g0 ∈ H1 and β0j ∈ H2 for all 1 ≤ j ≤ s, where 1
Hm ([0, 1]) = {g : [0, 1] 7→ R| g (j) is abs. cont. for j = 0, . . . , m − 1, and g (m) ∈ L2 ([0, 1])}. H0m ([0, 1]) ⊆ Hm ([0, 1]) is a class of periodic functions satisfying a set of additional boundary conditions: g (j) (0) = g (j) (1) for j = 0, . . . , m − 1. 2
6
J. LU, G. CHENG AND H. LIU
H1 and H2 are two (possibly different) RKHS’s. Define a product space as n o H2 = f | f (u) = g(x)+zβ(x), for u = (x, z) ∈ X ×Z := U, g ∈ H1 , β ∈ H2 . Denote f = (g, β) if and only if f (u) = g(x) + zβ(x) for any u = (x, z). For any f1 = (g1 , β1 ), f2 = (g2 , β2 ) ∈ H2 , we define the inner product on H2 : (2.3) hf1 , f2 i = E f1 (U )f2 (U ) + λ1 hg1 , g2 iH1 + λ2 hβ1 , β2 iH2 ,
where the random variable U := (X, Z) takes values on U = X ×Z. We denote the induced norm kf k2 = hf, f i. Moreover, for any g1 , g2 ∈ H1 , β1 , β2 ∈ H2 , we simply denote hg1 , g2 i = h(g1 , 0), (g2 , 0)i, hβ1 , β2 i = h(0, β1 ), (0, β2 )i and similar for the norm k·k. The inner product is designed as (2.3) for facilitating the asymptotic analysis of our estimators defined later. For example, the second Fr´echet derivative of the objective function for estimation turns out to be identity under this inner product; see Appendix A.2 for more details. The following result shows that (H2 , h·, ·i) is indeed a RKHS. Proposition 2.1. There exists a kernel function K λ : U × U → R and an operator Wλ : H2 7→ H2 such that for any f1 = (g1 , β1 ), f2 = (g2 , β2 ) ∈ H2 and u = (x, z) ∈ U, we have hKuλ , f1 i = g1 (x) + zβ1 (x) and hWλ f1 , f2 i = λ1 hg1 , g2 iH1 + λ2 hβ1 , β2 iH2 ,
where Kuλ (·) = K λ (u, ·).
3. Two-Step Estimation. In this section, we estimate the nonparametric heterogenous model (1.2) in two steps. The obtained estimators for homogeneity and heterogeneity functions will be used in the main result of this paper: heterogeneity/homogeneity testing (Section 4). In theory, these two estimators are shown to achieve the so-called “oracle efficiency” in the sense that their (joint) asymptotic behaviors are exactly the same as those for oracle estimators; see Remark 3.5. This oracle rule holds as long as the number of sub-populations does not grow too fast and the smoothing parameters in the kernel ridge estimation are properly chosen. In the j-th sub-population, we observe {(Yi , Xi , Zi )}i∈Ij (with |Ij | = n) Yi = g0 (Xi ) + Zi β0j (Xi ) + i ,
where g0 ∈ H1 and β0j ∈ H2 . Denote n × s as the entire sample size N . With a bit abuse of notation, we write f0j = (g0 , β0j ). For simplicity, we assume (Xi , Zi , i )’s are independent and follow the same distribution across all sub-populations, denoted as (X, Z, ). 3.1. Model Assumptions. In this section, we introduce three regularity assumptions: Assumption 3.1 on the model specifications; Assumption 3.2
NONPARAMETRIC HETEROGENEITY TESTING
7
on the reproducing kernel K; Assumption 3.3 on the complexity of H1 , H2 . Assumption 3.1 (Covariate and Noise). The random variables (X, Z, ) are pairwise independent with E(Z) = E() = 0. We assume that |Z| ≤ cz almost surely and is a subgaussian random variable such that there exists a σ > 0 and E[exp(t)] ≤ exp(t2 σ2 /2) for any t ∈ R. Denote Var(Z) = σz2 and Var() = σ 2 . Assumption 3.2 (Kernel and Eigenfunction). The kernel functions of H1 , H2 , denoted by K1 (x, t), K2 (x, t), are bounded on the diagonal by some constant cb such that sup |K1 (x, x) ∨ K2 (x, x)| ≤ cb .
x∈X
∞ Their eigen-functions {φk }∞ k=0 and {ψk }k=0 are uniformly bounded on X , namely there exists a constant cφ such that
(3.1)
sup {kφj ksup ∨ kψj ksup } ≤ cφ . j∈N
√ We define two useful constants, i.e., cH = cb (1 + cz ) and cK = cφ 1 + cz , which will be used in stating our inference results later. Assumption 3.2 is commonly used in the literature and is satisfied for a variety of kernels (Shang and Cheng, 2013; Zhang et al., 2013; Lafferty and Lebanon, 2005; Guo, 2002). For example, the polynomial kernel K(x, t) = (1 + xt)d is uniformly bounded as long as X is a compact subset of R. The trigonometric basis (2.2) in the periodic Sobolev space H0m ([0, 1]) trivially satisfies this assumption. In fact, Proposition 2.2 in Shang and Cheng (2013) proved that the eigen-functions of the more general H m ([0, 1]) are bounded under a mild condition on covariate density. More generally, Lemma 3 in Minh et al. (2006) says that the eigen-functions of any kernel function expressed in the form K(x, t) = g(hx, ti), where g : [−1, 1] → R is any continuous function, satisfy supk∈N kφk ksup < ∞ if X = S 1 := {x ∈ R2 | kxk2 = 1}. Before stating Assumption 3.3, we need to introduce a few notations related to the covering number. Suppose F is a function space. An -net for (F, d) with d being a metric is a set {f1 , . . . , fM } ⊆ F such that for any f ∈ F, there exists an fj for some j = 1, . . . , M satisfying d(f, fj ) ≤ . The -covering number is defined as N (F; d, ) = min |N | : N is an -net for (F, d) . A subspace in H2 of particular interest is defined as (3.2) H0 = f = (g, β) ∈ H2 : ||f ||sup ≤ 1, kgkH1 ≤ 1, kβkH2 ≤ 1 ,
8
J. LU, G. CHENG AND H. LIU
whose entropy integral is defined as Z δq (3.3) ω(δ) = log(1 + N (H0 ; k · ksup , ))d. 0
Assumption 3.3. (Complexity of H0 ) For any fixed a > 0, ω(aδ)/δ is always upper bounded by some non-increasing function, denoted as ω ¯ a (δ), for δ ∈ (0, 1). Assumption 3.3 characterizes the complexity of H0 through its entropy integral ω(δ), and is satisfied for function classes considered in this paper. Suppose H1 = H2 = H. When the eigenvalues of H decay polynomially, the entropy integral can be written as ω(δ) = cδ q , where q is some constant on (0, 1). When H is finite rank or exponentially decaying, the entropy integral has the form ω(δ) = cδ(log(1/δ))q for some 0 < q < 1. Obviously, Assumption 3.3 holds in both cases. When two RKHS’s are different, ω(·) can be bounded by a summation of the above two forms, which still satisfies Assumption 3.3. 3.2. Oracle Rule. The two-step estimation procedure is described in this subsection. A nice “oracle rule” holds for both homogeneity and heterogeneity estimators once the number of sub-populations does not grow too fast. Our general theory is further applied to three specific RKHS families. Step 1. (Individual Estimation) In the j-th sub-population, we estimate (g0 , β0j ) as (j) (j) (j) (j) (3.4) gbn,λ , βbn,λ := fbn,λ = argmin Ln,λ (f ), f ∈H2
where the loss function is 2 λ1 1 X λ2 (j) Yi − g(Xi ) − Zi β(Xi ) + kgk2H1 + kβk2H2 . (3.5) Ln,λ (f ) := 2n 2 2 i∈Ij
Step 2. (Aggregation for Commonality) Average over all the estimators for homogeneity function: s 1 X (j) gbn,λ . (3.6) g¯N,λ = s j=1
Several quantities need to be defined before stating our oracle result. ∞ Denote {µ1,j }∞ j=0 and {µ2,j }j=0 as the eigenvalues of H1 and H2 , and γ1 = γ1 (λ1 ) :=
∞ X j=0
(1 + λ1 /µ1,j )−1 ,
γ2 = γ2 (λ2 ) :=
∞ X
(σz2 + λ2 /µ2,j )−1 ,
j=0
γ = γ1 (λ1 ) + γ2 (λ2 ), h1 = 1/γ1 , h2 = 1/γ2 , h = 1/γ.
NONPARAMETRIC HETEROGENEITY TESTING
9
Also, we define a quantity related to the statistical rate p p √ (3.7) rn := (log N )/ nh + λ1 + λ2 . and the measure on the capacity of RKHS:
ξλ = (c2K γλ)−1/2 ω (c2K γλ)1/2 ,
(3.8)
where λ = λ1 ∧ λ2 and cK is specified in Assumption 3.2. The conditions on (s, λ1 , λ2 ) required in Theorem 3.4 are determined by the above quantities. (j) The following theorem depicts an asymptotic normality of (¯ gN,λ , βbn,λ ), irrespective of their different convergence rates. Theorem 3.4. Suppose Assumptions 3.1 – 3.3 hold. Given any fixed x1 , x2 ∈ X , we assume λ = o(1) and 2 P φk (x1 ) = σg2 < ∞, limn→∞ σ 2 h1 ∞ k=0 1+λ1 /µ1,k 2 (3.9) P ψk (x2 ) limn→∞ σ 2 σz2 h2 ∞ = σβ2 < ∞. k=0 σ 2 +λ2 /µ z
2,k
If s, λ1 , λ2 satisfy (3.10) (3.11) (3.12)
log N ξλ /(hλ) = o(log2 N ), √ (nh)−1 ( n · ξλ + 1) rn log N = o(N −1/2 ), p rn−1 · ω rn λ/h (nhλ)−1/2 log N = o(1),
we have the following joint asymptotic normality result ! √ !! N h1 g¯N,λ (x1 ) − g0u (x1 ) σg2 0 , (3.13) N 0, √ (j) u (x ) 0 σβ2 nh2 βbn,λ (x2 ) − β0j 2
u) = f u where the re-centered functions (g0u , β0j 0j + Wλ f0j := f0j , for any j = 1, . . . , s. Recall that the bias operator Wλ is defined in Proposition 2.1.
Note that when H1 = H0m ([0, 1]), σg2 has an explicit expression given in Shang and Cheng (2013). The asymptotic independence between g¯N,λ and (j) βbn,λ is not surprising since the sub-sample used to estimate the heterogeneity is asymptotically ignorable comparing with the entire sample size used to estimate the homogeneity, i.e., n/N = s−1 → 0. We remark that (3.10) controls some tail probability such that the union bound argument can be applied to a diverging number of sub-populations, while (3.11) and (3.12) control the nonparametric estimation error in each sub-population. Remark 3.5. An important consequence of Theorem 3.4 is the following
10
J. LU, G. CHENG AND H. LIU
oracle rule. Define the oracle estimate for g0 and β0j as follows: 2 P P gbor := argming∈H1 N −1 sj=1 i∈Ij Yi − g(Xi ) − Zi β0j (Xi ) + λ1 kgk2H1 , 2 P βbor,j := argmin n−1 Yi − g0 (Xi ) − Zi β(Xi ) + λ2 kβk2 . β∈H2
H2
i∈Ij
These two estimators are “oracle” in the sense that we use true homogeneity/heterogeneity function in the estimation. The proof of Theorem 3.4 (j) implies that (b gor , βbor,j ) shares the same joint distribution as (¯ gN,λ , βbn,λ ) (after similar re-centering and re-scaling). This type of “oracle efficiency” indicates that our two-step estimation efficiency cannot be further improved. Corollary B.3 in Appendix B gives more explicit conditions in terms of (s, λ1 , λ2 ) for three RKHS families. From Corollary B.3, it is easy to see that λ1 should be chosen in the scale of N , while λ2 in the scale of n. In other words, we need to sub-regularize the homogeneity estimation in each sub-population. Without heterogeneity part, Zhang et al. (2013) discovered a similar phenomenon for estimating g0 optimally. We show that this phenomenon persists at an inferential level (in a more general context). 4. Penalized Likelihood Ratio Theory. In this section, we apply the penalized likelihood ratio principle advocated in (Shang and Cheng, 2013) to various types of heterogeneity/homogeneity testing. For simplicity, we only focus on the case that H1 = H2 = H in this section. 4.1. Heterogeneity Testing. In this section, we are interested in testing the following three hypotheses: (a) Simple test: H0 : β0j = β0 . (b) Pairwise test: H0 : β0j = β0j 0 , for some j 6= j 0 . (c) Simultaneous test: H0 : β01 = · · · = β0s .
In all these tests, the homogeneity function is treated as nuisance. The simultaneous test (c) is concerned about whether the entire dataset is homogenous or not. We are particularly interested in the largest number of sub-populations that can be tested in (c) with theoretical guarantee, e.g., minimax optimality. This is technically challenging given that s diverges. Let us begin with the simple test (a). A penalized likelihood ratio test (PLRT) statistic is constructed as follows: (j) (j) (j) (4.1) PLRTn,λ = Ln,λ g¯N,λ , βbn,λ − Ln,λ g¯N,λ , β0 (j)
by plugging in the aggregated estimate g¯N,λ . Recall that Ln,λ is defined in
NONPARAMETRIC HETEROGENEITY TESTING
11
(3.5). As for the pairwise test (b), we propose h i (j,j 0 ) (j) (j) (j) (j 0 ) PLRTn,λ = Ln,λ g¯N,λ , βbn,λ − Ln,λ g¯N,λ , βbn,λ h 0 i (j ) (j 0 ) (j 0 ) (j) + Ln,λ g¯N,λ , βbn,λ − Ln,λ g¯N,λ , βbn,λ := T1 + T2 .
In fact, either T1 or T2 is a valid test statistic. Here, we take their summation in order to increase its power in practice. To better understand T1 , we decompose it as T1 = T11 − T12 under the null β0j = β0j 0 , where (j) (j) (j) T11 = Ln,λ g¯N,λ , βbn,λ − Ln,λ g¯N,λ , β0j , (j) (j 0 ) (j) T12 = Ln,λ g¯N,λ , βbn,λ − Ln,λ g¯N,λ , β0j 0 .
Note that T11 and T12 are both PLRT statistic for some simple test (4.1). (j,j 0 ) Hence, the null limiting distribution of PLRTn,λ follows from that of PLRTn,λ . For the most challenging test (c), we can similarly construct (also see Remark 4.4) P (j) (j) (j) (j+1) (4.2) PLRTsn,λ = s−1 ¯N,λ , βbn,λ − Ln,λ g¯N,λ , βbn,λ . j=1 Ln,λ g
Alternatively, a maximum type test also works here (j) (j) (j) (j 0 ) max0 Ln,λ (¯ gN,λ , βbn,λ ) − Ln,λ (¯ gN,λ , βbn,λ ) , 1≤j6=j ≤s
but with slightly more computation as s grows. We want to point out that PLRTsn,λ is more computationally feasible than the commonly used nonparametric multiple-sample test (Pardo-Fern´andez et al., 2007), which requires to calculate an additional estimator based on the entire sample. This additional computation may not be affordable in the setting of massive data. In addition, the nonparametric multiple-sample tests in the literature (without nuisance g0 ) such as Pardo-Fern´andez et al. (2007), Munk and Dette (1998) and Srihera and Stute (2010) all require the number of tested functions to be fixed. However, our test method remains valid even when s diverges to infinity at a proper rate; see Theorems 4.1 and 4.2 below. We are ready to present null limiting distributions for tests (a) – (c). In a particular, a statistic Tn is said to be nearly χ2bn , denoted as Tn ∼ χ2bn , if (2bn )−1/2 (Tn − bn ) weakly converges to N (0, 1) for some sequence bn → ∞. Theorem 4.1. Suppose that Assumptions 3.1 – 3.3, and (3.10) – (3.12) hold, and E[4 |X] ≤ C, a.s., for some constant C. If nh2 λ2 = o(1) and
12
J. LU, G. CHENG AND H. LIU
(nh22 )−1 = o(1), we have 1 a (j,j 0 ) a −2nrT · PLRTn,λ ∼ χ2un and − nrT · PLRTn,λ ∼ χ2un , 2 −1 4 2 2 2 where un = h2 σT /ρT , rT = σT /ρT , and σT2 = σ 2 σz2 h2
∞ X
(σz2 + λ2 /µ2,k )−1 , ρ2T = σ 4 σz4 h
k=0
If we further assume
∞ X
(σz2 + λ2 /µ2,k )−2 .
k=0
(N h22 )−1
= o(1), then
4 a − nrT · PLRTsn,λ ∼ χ2ueN , 3 4 2 where u eN = (4s/3)h−1 2 σT /ρT . (4.3)
We next study the power performance of the above nonparametric testing in terms of their separation rates. Define a function space (4.4)
Fζ ≡ {β ∈ H| Var(β(X)2 ) ≤ ζE2 [β(X)2 ] and kβk2H ≤ ζ},
for some constant ζ > 0, and two separation rates as p 2 ηn2 = λ2 + 1/(n2 h2 ) + rN (4.5) , p 2 2 (4.6) , ηeN = λ2 + s/(n2 h2 ) + rN √ √ √ where rN = log N/ N h1 + λ1 + λ2 .
Theorem 4.2. Suppose that the same conditions as Theorem 4.1 hold. • Under the alternative sequence H1n : β0j = βn , we have for any cn → ∞, inf PH1n (2un )−1/2 − 2nrT · PLRTn,λ − un > zα → 1, inf g0 ∈Fζ β0 −βn ∈Fζ ∗ ≥c η 2 δ0j n n
∗ := kβ − β k2 . where δ0j 0 n • Under the alternative sequence H1n : β0j 6= β0j 0 , we have for any cn → ∞, 1 (j,j 0 ) −1/2 inf inf PH1n (2un ) − nrT · PLRTn,λ − un > zα → 1, g0 ∈Fζ β0j −β0j 0 ∈Fζ 2 2 ∆∗2 ≥cn ηn
where ∆∗2 := kβ0j − β0j 0 k2 . • Under the alternative sequence H1n : β0j = 6 β0j 0 for at least one pair
NONPARAMETRIC HETEROGENEITY TESTING
13
(j, j 0 ), we have for any cn → ∞ 4 inf PH1n (2e uN )−1/2 − nrT ·PLRTsn,λ −e uN > zα → 1, 3 {(β0j −β0(j+1) )}s−1 j=1 ⊆Fζ 2 g0 ∈Fζ ,∆∗s ≥cn ηeN
P s−1 2 where ∆∗s := s−1 j=1 kβ0j − β0(j+1) k and “{(β0j − β0(j+1) )}j=1 ⊆ Fζ ” means that each (β0j − β0(j+1) ) ∈ Fζ for j = 1, . . . , s − 1.
When g0 is known, we can show that Theorem 4.2 p still holds for hypotheses (a) and (b) by simply replacing (4.5) by ηn2 & λ2 + 1/(n2 h2 ). In other words, 2 in (4.5) is due to the estimation of the nuisance parameter g based on the rN 0 entire sample. However, this term can be dominated by the first two terms in the RHS of (4.5) if we can borrow sufficient information across all the sub-populations to reduce the estimation error, i.e., s diverges fast enough. This explains why we need to set a lower bound on s in order to obtain the minimal possible separation rate. We call this lower bound phenomenon as “blessing of aggregation”. See below for more details. Following Theorem 4.2, we derive the minimal possible values of ηn for tests (a) and (b), denoted as ηn∗ , in three RKHS families. • Finite rank kernel (with rank r)
√ ηn∗ = O(r1/4 / n) √ √ given that λ1 r/N , λ2 r/n and r . s . r−5/2 N 1/2 (log N )−4 ; p
• Exponentially decaying kernel (with µk e−αk ) √ ηn∗ = O((log n)1/(4p) / n) given that λ1 (log N )1/p /N , λ2 (log n)1/(2p) /n and (log n)1/(2p) . s . N 1/2 (log N )
− 7p+1 2p
;
• Polynomially decaying kernel (with µk k −2m ) ηn∗ = O(n−2m/(4m+1) )
given that λ1 N −2m/(2m+1) , λ2 n−4m/(4m+1) and s nd for (4m + 1)−1 < d < (8m2 + 4m)/(12m2 − 2m + 1).
We remark that in the polynomial case, the minimal separation rate n−2m/(4m+1) turns out to be the minimax lower bound established by Ingster (1993) in the special case that s = 1, Z ≡ 1 and g0 ≡ 0. The upper bounds of s in the above three cases guarantee that the higher order terms of test statistics are asymptotically ignorable under the same conditions of Corollary B.3 in Appendix B. Rather, the lower bounds make sure that the nuisance estimator
14
J. LU, G. CHENG AND H. LIU
g¯N,λ is close enough to the truth. As for the test (c), we obtain similar conclusions as follows. • Finite rank kernel (with rank r)
√ ∗ ηeN = O((sr)1/4 / n) √ √ given that λ1 r/N, λ2 r/n, and r . s . r−5/2 N 1/2 (log N )−4 ; p
• Exponentially decaying kernel (with µk e−αk ) √ ∗ ηeN = O(s1/4 (log n)1/(4p) / n)
given that λ1 (log N )1/p /N, λ2 (log n)1/(2p) /n and (log n)1/(2p) .
s . N 1/2 (log N )
− 7p+1 2p
;
• Polynomially decaying kernel (with µk k −2m ) ∗ ηeN = O(sm/(4m+1) n−2m/(4m+1) )
given that λ1 N −2m/(2m+1) , λ2 (s/n2 )2m/(4m+1) and s nd for (2m + 1)−1 < d < (8m2 + 4m)/(12m2 − 2m + 1). We point out that only the λ2 in the polynomial case is related to s. Also, the lower bound of s in this case is larger than those in tests (a) and (b) due to the fact that a faster convergent g¯N,λ is needed here. Remark 4.3. Another test of interest is H0 : β0j = c for some possibly unknown constant c. Under H0 , our main model (1.2) becomes a partially linear model estimated through X 1 (j) (j) 2 2 (b gc , b c ) = argmin (Yi − g(Xi ) − cZi ) + λ1 kgkH . n g,c i∈Ij
(j)
Note that the joint asymptotic behavior of (b gc , b c(j) ) for each fixed j can be found in Cheng and Shang (2015). Similarly, our test statistic follows as (j) (j) (j) PLRTconst ¯N,λ , βbn,λ − Ln,λ g¯N,λ , b c(j) , n,λ = Ln,λ g (j)
where βbn,λ and g¯N,λ are defined in (3.4) and (3.6), respectively. By slightly modifying the proof of Theorem 4.1 (see Supplementary S.3.3), we have a
2 0 2 2 2 −2nrT0 · PLRTconst n,λ ∼ χu0n , where rT = σT /(2ρP + ρT ), P∞ 4 2 2 2 4 −2 u0n = h−1 2 σT /(2ρP + ρT ) and ρP = σ h1 k=0 (1 + λ1 /µ1,k ) . Similar arguments also apply to composite hypotheses such as H0 : β0j belongs a class of polynomial functions (with a specified degree). (j,j 0 )
Remark 4.4. To be consistent with the form of PLRTn,λ , we can modify
NONPARAMETRIC HETEROGENEITY TESTING
15
(s) (s) (s) (1) PLRTsn,λ by adding an additional term Ln,λ g¯N,λ , βbn,λ − Ln,λ (¯ gN,λ , βbn,λ ) to the RHS of (4.2). Given this modification, all the above theoretical results remain valid after some minor changes, e.g., the degrees of freedom in (4.9).
4.2. Homogeneity Testing. In this section, we turn our attention to homogeneity testing H0 : g0 = g0∗ ,
(4.7)
where g0∗ is given. Since g0 is shared among all sub-populations, we define an averaged penalized likelihood ratio test (APLRT) statistic as s o 1 X n (j) (j) (j) (j) Ln,λ g¯N,λ , βbn,λ − Ln,λ g0∗ , βbn,λ . (4.8) APLRTN,λ = s j=1
(j) (j) Note that the use of βbn,λ in the second Ln,λ of (4.8) greatly saves computation cost than that using the restricted estimate of β0j under H0 , especially for a large s. We prove in Theorem 4.6 that the aggregated homogeneity test across s sub-populations is more sensitive to local alternative sequences (in terms of a smaller separation rate) than that based on a single sub-population as s → ∞. Also see more discussions after Theorem 4.6. And if we replace the (j) aggregated estimate g¯N,λ by gbn,λ in each summand of (4.8), the separation rate will also be slowed down; see Remark 4.7. These discussions justify the form of (4.8), and again echo with the “blessing of aggregation” phenomenon. The following theorem gives the null limiting distribution of APLRTN,λ .
Theorem 4.5. Suppose that Assumptions 3.1 – 3.3, and (3.10) – (3.12) hold, and E[4 |X] ≤ C, a.s., for some constant C. If nh1 λ1 = o(1) and (N h21 )−1 = o(1), then we have a
− 2N rP · APLRTN,λ ∼ χ2uN
(4.9)
4 2 2 2 where uN = h−1 1 σP /ρP , rP = σP /ρP , and
σP2 = σ 2 h1
∞ X
(1 + λ1 /µ1,k )−1 , ρ2P = σ 4 h1
k=0
∞ X
(1 + λ1 /µ1,k )−2 .
k=0
Our next theorem derives a minimal separation rate between null and the alternative sequence under which the proposed homogeneity test statistic APLRTN,λ can still detect the latter. Define a separation rate as (4.10)
1/2
2 ηN = λ1 + λ2 + (N h1 )−1 .
Theorem 4.6. Suppose the same conditions as Theorem 4.5. Under the
16
J. LU, G. CHENG AND H. LIU
alternative sequence H1N : g0 = gN , if rn = o(1), we have for any cN → ∞, (4.11) | − 2N rP APLRTN,λ − uN | √ > zα → 1, inf inf PH1N {β0j }sj=1 ⊆Fζ ∆gN ∈Fζ 2uN 2 k∆gN k2 ≥cN ηN
where ∆gN = gN − g0∗ . Here, “{β0j }sj=1 ⊆ Fζ ” means that each β0j ∈ Fζ for j = 1, . . . , s. In what follows, we derive explicit conditions of (λ1 , λ2 , s) for achieving ∗ , for three specific RKHS families. the minimal separation rate, denoted as ηN • Finite rank kernel (with rank r) given that λ1 , λ2
√
√ ∗ ηN = O r1/4 / N
r/N and s . r−5/2 N 1/2 (log N )−4 ; p
• Exponentially decaying kernel (with µk e−αk ) ∗ ηN = O [(log N )1/(2p) /N ]1/2
given that λ1 , λ2 (log N )1/(2p) /N and s . N 1/2 (log N )
− 7p+1 2p
;
• Polynomially decaying kernel (with µk k −2m ) ∗ ηN = O N −2m/(4m+1) given that λ1 , λ2 N −4m/(4m+1) and s nd for d
0, which is non-increasing. Solving (3.11), we have ! N 1/2 . (B.10) s=o 1/2 (log N )3 r5/2 (log(hλ−1 1 )) p Since λ1 = o( r/N ) and log(1/λ1 ) = o(log2 N ), we have log(N ξλ /(hλ)) = o(log2 N ) such that (3.10) is satisfied. We can verify (3.12) by −1/2 rn−1 ω (λ−1 rn (λ1 h)−1/2 n−1/2 log N = o(1). 1 h) • Exponentially Decaying For the exponentially decaying kernel, following Lemma S.2 and (B.8) again, we bound ω(δ) by Z δ p+1 − p+1 ω(δ) . [log(1/)] 2p d . δ [log(1/δ)] 2p . 0
We therefore construct the upper bound required in Assumption 3.3 as − p+1 ω ¯ a (δ) = Ca [log(1/(aδ))] 2p . It is easy to check ω ¯ a (δ) is non-increasing. We also have λ1 = λ1 ∧ λ2 . We next bound ξλ as −1 1/2 −1 −1/2 ω((c−2 ) . [log(hλ−1 ξλ = (c−2 1 )] K hλ1 ) K hλ1 )
p+1 2p
.
Following the proof of Lemma S.9 in the Supplementary Material, we have ∞ X 1 1/h1 = γ1 (λ1 ) = (log(1/λ1 ))1/p , 1 + λ1 /µ1,k k=0
NONPARAMETRIC HETEROGENEITY TESTING
27
and the same for h2 . We now verify (3.10) - (3.12). We obtain the upper bound of s by solving (3.11): ! N 1/2 h3/2 (B.11) s=o . (p+1)/2p (log N )3 [log(hλ−1 1 )] p p Since λ1 = o( 1/N ) and λ2 = o( 1/n), it is easy to check log(N ξλ (hλ1 )−1 ) = o(log2 N ) such that (3.10) is satisfied. We can easily check (3.12) that −1/2 rn−1 ω (λ−1 rn (λ1 h)−1/2 n−1/2 log N = o(1). 1 h) • Polynomially Decaying For the polynomially decaying kernel, similar to the above two cases, we apply Lemma S.2 and (B.8) and get Z δ ω(δ) . (1/)1/(2m) d . δ 1−1/(2m) . 0
We then construct a non-increasing function ω ¯ a (δ) = Ca1−1/2m δ −1/(2m) as the upper bound required in Assumption 3.3. Also ξλ can be bounded as −1 1/2 −1 −1/2 1/(4m) ξλ = (c−2 ω((c−2 ) . (hλ−1 1 ) K hλ1 ) K hλ1 )
Ps Moreover, from the proof of Lemma S.9, we have γ(λ1 ) = k=0 (1 + −1/(2m) −1 λ1 /µ1,k ) λ1 and the same for γ(λ2 ). Solving (3.11), we have 8m−1 2 (B.12) s = o N 1/2 λ18m , 4m2
− 8m−1 and λ−1 = o(1) as s is always larger than 1. 1 N −1/2 As λ1 = o(N ), we have log(N ξλ (hλ1 )−1 ) = o(log2 N ) such that (3.10) is satisfied. We then verify (3.12): −1/2 rn (λ1 h)−1/2 n−1/2 log N = o(1) rn−1 ω (λ−1 1 h) 1/(2m)
gives rn
1/(8m2 ) −1/2 n
. λ1
and therefore we have − 2m−1 −1/(m−2m2 ) 2 n & λ1 (log N )−2/(2m−1) ∨ λ1 4m . 2m−1 2
This gives the upper bound s . N 1/2 λ18m , which is weaker than (B.12). APPENDIX C: PROOFS FOR HYPOTHESIS TESTS In this section, we present partial proofs for Theorems 4.1 and 4.2, in (j,j 0 ) which only PLRTn,λ and PLRTsn,λ are considered. The remaining proofs for PLRTn,λ and PLRTconst n,λ can be found in the Supplementary Material Section S.3.3. In the end, we prove Theorems 4.5 and 4.6.
28
J. LU, G. CHENG AND H. LIU
C.1. Partial Proofs of Theorems 4.1 and 4.2. (j,j 0 )
C.1.1. Analysis of PLRTn,λ . Without loss of generality, we assume (1,2)
j = 1, j 0 = 2. We can decompose PLRTn,λ = L(1) + L0(1) , where (1) (1) (1) (2) L(1) := Ln,λ (¯ gN,λ , βbn,λ ) − Ln,λ (¯ gN,λ , βbn,λ ) , (2) (2) (2) (1) L0(1) := Ln,λ (¯ gN,λ , βbn,λ ) − Ln,λ (¯ gN,λ , βbn,λ ) .
The high level idea of our proof is to use Taylor expansion to decompose L(1) and L0(1) into the leading asymptotic normal terms and higher order terms. To implement this idea, we first study the derivative of operators in the (1) above two terms. Denote the second coordinate of the derivative ∇Ln,λ ((g0 , β01 )) in (A.2) as 1X (1) i Zi TXλ i + W2,λ β01 . ∇L2 (β01 ) := − n i∈I1
(2)
Similarly, we define the second coordinate of ∇Ln,λ ((g0 , β02 )) as 1X (2) ∇L2 (β02 ) := − i Zi TXλ i + W2,λ β02 . n i∈I2
Define β1,2 =
(1) βbn,λ
−
(2) βbn,λ .
We have the following Taylor expansion
1 (1) (2) (1) (2) gN,λ , βbn,λ )[β1,2 , β1,2 ] L(1) = h∇Ln,λ (¯ gN,λ , βbn,λ ), β1,2 i + ∇2 Ln,λ (¯ 2 1 (1) (1) (2) (C.1) = h∇Ln,λ (g0 , βbn,λ ), β1,2 i + E{∇2 Ln,λ (g0 , β01 )}[β1,2 , β1,2 ] + ∆1 /2, 2 (1) (1) 2 where ∆1 = ∇ Ln,λ (¯ gN,λ , β02 ) − E{∇2 Ln,λ (g0 , β01 )} [β1,2 , β1,2 ]. Following a similar proof as Lemma B.1, we have ∆1 = oP ((n2 h2 )−1/2 ). We refer the detailed derivation to the proof of (S.34) in the Supplementary Material. (1) Combining (C.1) with the fact that E{∇2 Ln,λ (g0 , β01 )} is an identity as shown in Proposition A.2, we have 1 (1) (2) (C.2) L(1) = h∇Ln,λ (g0 , βbn,λ ), β1,2 i + kβ1,2 k2 + oP ((n2 h2 )−1/2 , 2 (2)
(1)
(1)
(2)
By (B.1), we have β1,2 = ∇L2 (β02 )−∇L2 (β01 )+(Hn −Hn ). Applying (1) (2) Lemma B.1 and (3.11), we have kHn − Hn k = oP (N −1/2 ) and (C.3)
(2)
(1)
kβ1,2 k = k∇L2 (β02 ) − ∇L2 (β01 )k + oP (N −1/2 ).
29
NONPARAMETRIC HETEROGENEITY TESTING
Applying the Taylor expansion again, since β01 = β02 , we get (1) (2) (1) (1) (2) Ln,λ (g0 , βbn,λ ) = h∇Ln,λ (g0 , β01 ), β1,2 i + ∇2 Ln,λ (g0 , β01 )[βbn,λ − β02 , β1,2 ] (1) (2) = h∇L2 (β01 ), β1,2 i + hβbn,λ − β02 , β1,2 i + oP ((n2 h2 )−1/2 ) (1)
(2)
= −k∇L2 (β01 ) − ∇L2 (β02 )k2 + oP ((n2 h2 )−1/2 ),
(C.4)
where the second equality follows a similar argument as (C.2) and the third equality is similar to (C.3). The detailed proof follows a similar method in proving Lemma C.1 in the Supplementary Material. Combining (C.2) – (C.4), we have (1)
−1/2
(2)
− 2nL(1) = nk∇L2 (β01 ) − ∇L2 (β02 )k2 + oP (h2
(C.5)
),
where the bias term W2,λ β01 is ignorable due to the same derivation of (S.36) in the Supplementary Material. (1) (2) −1/2 Similarly, we also have −2nL0(1) = nk∇L2 (β01 )−∇L2 (β02 )k2 +oP (h2 ). Therefore, we have (1,2)
(1)
(2)
−1/2
− 2nPLRTn,λ = 2nk∇L2 (β01 ) − ∇L2 (β02 )k2 + oP (h2 ). P d (1) (2) Notice that ∇L2 (β01 ) − ∇L2 (β02 ) = n−1 i∈I1 ∪I2 i Zi TXλ i . We expand P k i∈I1 ∪I2 i Zi TXλ i k22 = W3 (2n) + ∆2 , where the cross term can be written as X (C.7) W3 (2n) = 2i j Zi Zj T λ (Xi , Xj ).
(C.6)
i c0 + λ1 kg0 k2H1 + λ2 kβ0 k2H2 ) ≤ 2 exp(−c0 n).
Now, we define the remainder term on the j-th sub-population 1X (j) (j) (j) u i KUλi . Rn(j) = fbn,λ − f0j − ∇Ln,λ (f0j ) = fbn,λ − f0j − n i∈Ij (j) (j) (j) We further denote Rn = Gn , Hn . According to the construction of g¯N,λ , we have GN := g¯N,λ − g0u −
i=1
s
1 XX i PXλ i s s j=1 j=1 i∈Ij s s 1 X (j) 1X 1 X (j) u λ = gbn,λ − g0 − i PXi = Gn . s n s
=
(S.9)
s 1X
N 1 X i PXλ i N
(j)
gbn,λ − g0u −
j=1
1/2
i∈Ij
1/2
j=1
(j)
(j)
Let rn = (nh)−1/2 T +λ1 +λ2 and define the event Ej = {fn,λ ∈ H, kfn,λ k ≤ c1 rn } and E = sj=1 Ej . We first study the remainder term in the jth sub-population. For simplicity,
43
NONPARAMETRIC HETEROGENEITY TESTING (j)
we omit the superscript (j) for Rn as well. Rn = fn,λ − ∇Ln,λ (f0j )
= E∇2 Ln,λ (f0j )[fn,λ ] − ∇Ln,λ (f0j )
= E∇Ln,λ (fn,λ + f0j ) − E∇Ln,λ (f0j ) − ∇Ln,λ (f0j )
= (∇Ln,λ (f0j ) − E∇Ln,λ (f0j )) (S.10)
− (∇Ln,λ (fn,λ + f0j ) − E∇Ln,λ (fn,λ + f0j )) n 1 X =− fn,λ (Ui )KUλi − E[fn,λ (U )KUλ ] . n i=1
The first equality is due to the definition of remainder term, the second and forth equalities are by Proposition A.2 and the third equality is by the taylor expansion of least square. In order to bound the rate of Rn , we turn to study the empirical process in (S.10). We separate Rn into two parts: n 1 X fn,λ (Ui )1Ej KUλi − E[fn,λ (U )1Ej KUλ ] , Rn2 = Rn − Rn1 . Rn1 = n i=1
We first bound Rn1 . Let dn = c1 cK rn h−1/2 and fen,λ = (e gn,λ , βen,λ ) = (j) d−1 n fn,λ 1Ej . On the event E = {kfn,λ k ≤ c1 rn for all j}, it follows by Lemma S.1 that kfen,λ ksup ≤ 1, ke gn,λ k2 ≤ d−2 λ−1 r2 ≤ c−1 c−1 hλ−1 and kβen,λ k2 ≤ −1 −1 c−1 1 cK hλ2 . Define an operator
(S.11)
H1
n
1
n
1
K
1
H2
1/2 Ψ(f, U ) = c−1 f KUλ , K h
and we consider the following empirical process: n 1 X e e Gn (fn,λ ) = √ Ψ(fn,λ , U ) − E[Ψ(fen,λ , U )] . n i=1
We now apply Lemma S.6 for Gn (fen,λ ) with d(·) = k · ksup and F = H0 = −1 −2 −1 2 {f = (g, β) ∈ H2 : ||f ||sup ≤ 1, kgk2H1 ≤ c−2 K hλ1 , kβkH2 ≤ cK hλ2 }. To 2 check the Lipschitz property, we have for any f1 , f2 ∈ H , 1/2 (S.12) kΨ(f1 , U ) − Ψ(f2 , U )k ≤ c−1 kKUλ kkf1 − f2 ksup ≤ kf1 − f2 ksup , K h
where the last inequality follows (S.5). According to (S.46), we have for all
44
J. LU, G. CHENG AND H. LIU
j = 1, . . . , s,
(j) n Rn1 P ≥t √ dn cK h−1/2 ( nξλ + 1) (j) (j) P || i∈Ij [fn,λ (Ui )1Ej KUλi − E(fn,λ (U )1Ej KUλ )]|| ≥t =P √ dn cK h−1/2 ( nξλ + 1) √ √ (j) nkGn (fen,λ )k nkGn (f )k √ =P > t ≤ P sup √ ≥t nξλ + 1 n ωd (1; H) + 1 f ∈H √ ≤ 2 log( nξλ ) + 2 exp(−t2 /4),
(S.13)
where ξλ is defined in (3.8). The first inequality follows from Lemma S.3 that ξλ ≥ ωd (1; H0 ), and the last inequality is due to (S.47) in Lemma S.6. Next, we study the second term n 1 X Rn2 = fn,λ (Ui )1Ejc KUλi − E[fn,λ (U )1Ejc KUλ ] . n i=1
(j) (j) Recall that fn,λ = fbn,λ − f0j , and fn,λ is its simplified version. To study Rn2 , we need the following lemma on the tail probability of kfn,λ k.
Lemma S.4. For a sufficiently large n, there exist constants C, c0 such that log N 1/2 1/2 + λ1 + λ2 ≤ 4n exp(−c0 log2 N ). P kfn,λ k > C √ nh We defer the proof of the lemma to Section S.2.2. Using Markov inequality, we have for all j, ! n X (j) λ λ P nkRn2 k ≥ t = P fn,λ (Ui )1Ejc KUi − E[fn,λ (U )1Ejc KU ] > t i=1
2n
≤ E fn,λ (U )1Ejc KUλ t q ≤ 2t−1 h−1/2 n
(S.14)
Ekfn,λ k2H2 P(Ejc )
≤ Ct−1 n3/2 (λ1 ∧ λ2 )−1 h−1/2 exp(−c0 log2 N ).
The first and second inequalities are simply Markov inequality and CauchySchwartz inequality, respectively, while the last one is by (S.7), Lemma S.1 and Lemma S.4. √ √ Let an = n−1 dn cK h−1/2 ( nξλ + 1) = c1 c2K (nh)−1 rn ( nξλ + 1). For every j ∈ 1, . . . , s. Combining (S.13) and (S.14), we can conclude that kRn k = oP (an log n) and accordingly we have kHn k = oP (an log n) and kGn k = oP (an log n). This completes the first part of the proof.
45
NONPARAMETRIC HETEROGENEITY TESTING
P (j) Next, we study GN = s−1 sj=1 Gn . According to (S.13) and (S.14) again, we have
(j) √ n Gn1 nξ ) + 2 exp(−t2 /4), (S.15) P ≥ t ≤ 2 log( √ λ dn cK h−1/2 ( nξλ + 1) and (j) (S.16) P nkGn2 k ≥ t ≤ Ct−1 n3/2 (λ1 ∧ λ2 )−1 h−1/2 exp(−c0 log2 N ). For any 1 ≤ j ≤ s, define the event (j) Aj = kGn1 k ≤ an log N and A = ∩sj=1 Aj .
By (S.15) and (S.16), we have (j) n Gn1 1 1 (j) c P(Aj ) ≤ P ≥ log N + P nkGn2 k ≥ an log N an 2 2 √ 2 ≤ 2 log( nξλ ) + 2 exp(− log N/4) (S.17)
+ 2C(an log N )−1 n3/2 (λ1 ∧ λ2 )−1 h−1/2 exp(−c0 log2 N ). (j)
According to the definition of Aj , we have kGn 1Aj k ≤ an log N for all j. (j)
(j)
(j)
(j)
We again separate Gn into two parts: Gn = Gn 1Aj + Gn 1Acj for all j. Similar to (S.16), it follows from (S.7), Lemma S.1 and Lemma S.4 that
(j) λ EkG(j) n 1Acj k ≤ 2E gn,λ (U )1Acj KU q (j) ≤ 2h−1/2 Ekgn,λ k2H1 P(Acj ) ≤ C(λ1 h)−1/2 P(Acj ). (S.18) (j)
(j)
Combining (S.13) and (S.14), we have E[kGn 1Aj k] ≤ EkGn k ≤ 4an log N (j) s (j) for sufficiently large N . We next apply Lemma S.8 to Gn 1Aj −E[Gn 1Aj ] j=1 (j)
since they are i.i.d. mean zero bounded variables. Note that kGn 1Aj − (j)
E[Gn 1Aj ]k ≤ 2an log N by (3.10) with (S.17), (S.18), and for sufficiently
46
J. LU, G. CHENG AND H. LIU (j)
large N , EkGn 1Acj k ≤ s−1/2 an log2 N . Therefore, we have s
1 X
P G(j) 1 > 6a log N
n Aj n s j=1
s
1 X
(j) (j) =P G(j) n 1Aj − E[Gn 1Aj ] + E[Gn 1Aj ] > 6an log N s j=1
s
1 X
(j) −1/2 2 ≤P G(j) 1 − E[G 1 ] > 2s a log N .
n Aj Aj n n s j=1
(S.19)
≤ 2 exp(− log2 N/2)
Therefore, we can bound the tail probability of GN : X s
1 (j) Gn > 12an log N P s j=1 X X s s
1
1 (j) (j) Gn 1Aj > 6an log N + P Gn 1Acj > 6an log N ≤P s s j=1
j=1
(S.20) ≤ 2 exp(− log2 N/2) + (6an log N )−1 EkG(j) n 1Acj k = o(1)
where the second inequality follows from (S.19) and (S.18), and the last equality follows by applying (3.10) to (S.17) and (S.18). Therefore, we complete the proof of the lemma. S.2.2. Proof of Lemma S.4. We now prove a sharper rate on fn,λ . Using the zero-order optimality condition (Luenberger, 1969), Ln,λ (f0j + fn,λ ) − Ln,λ (f0j ) (S.21) 1 = DLn,λ (f0j )[fn,λ ] + D2 Ln,λ (f0j )[fn,λ , fn,λ ] ≤ 0. 2 We next bound the above two terms separately. For the first term in the RHS of (S.21), Proposition A.2 gives the representation n 1X ∇Ln,λ (f0j ) = i KUλi − Wλ f0j . n i=1
For the regularization term Wλ f0j , according to Proposition A.1, we have 2
kWλ f0j k =
∞ X k=0
λ21 /µ21,k 1 + λ1 /µ1,k
hg0 , φk i2L2 (P)
+
∞ X k=0
λ22 /µ22,k σz2 + λ2 /µ2,k
hβ0j , φk i2L2 (P) .
NONPARAMETRIC HETEROGENEITY TESTING
It follows from g0 ∈ H1 that
P∞
−1 2 k=0 µ1,k hg0 , φk iL2 (P)
47
< ∞. Since
λ1 /µ2 1,k 2 hg0 , φk i2L2 (P) ≤ |µ−1 1,k hg0 , φk iL2 (P) | 1 + λ1 /µ1,k
for any k ∈ N, by dominate convergence theorem, we have ∞ X λ1 /µ21,k lim hg0 , φk i2L2 (P) = 0. N →∞ 1 + λ1 /µ1,k k=0
Similar result can be obtained for β0j . Hence, (S.22) Similarly, we can prove (S.23)
kWλ f0j k2 = o(λ1 + λ2 ).
kWλ,1 g0 k2 = o(λ1 ) and kWλ,2 β0j k2 = o(λ2 ). 1/2
Define two events B1 = {kfn,λ kH2 ≤ c0 (λ1 ∧ λ2 )−1/2 } and X B2 = {|| i KUλi || ≤ cK (n/h)1/2 log N }. i∈Ij
According to (S.8), we have P(B1c ) ≤ 2 exp(−c0 n) for some constant c0 . (1) (2) We denote i = i 1{i ≤log N } and i = i 1{i >log N } . By (3.10), we have √ n2 hN −c = o(1) for some sufficiently large c > 0. On the other hand, since exp(− log2 N ) = N − log N , we have for sufficiently large N , σn1/2 1 exp(− log2 N ) < cK (nh)−1/2 log N. cK log N 4 Denoting pn = cK (nh)−1/2 log N , we consider the decomposition 1 X 1 X
1
1 (1) (2) P(B2c ) ≤ P i KUλi > pn + P i KUλi > pn . n 2 n 2 i∈Ij i∈Ij | {z } | {z } P1
P2
We first control P2 by
2Ek(2) KUλ k σn1/2 ≤2 exp(− log2 N ), pn cK log N where the first inequality is by Chebyshev’s inequality and the second is due to the fact that kKUλ k ≤ cK h−1/2 in (S.5) and is subgaussian. We next P2 ≤
48
J. LU, G. CHENG AND H. LIU
control P1 . By Lemma S.8, there exists some constants C0 such that 1 X
1 (1) P1 ≤ P i KUλi − E[(1) KUλ ] > pn − Ek(1) KUλ k n 2 i∈Ij 1 X
1 (1) ≤P i KUλi − E[(1) KUλ ] > pn n 4 i∈Ij ≤ 2 exp −C0 log2 N .
Combining the upper bounds of P1 and P2 , we know that there exists some constant C 0 σn1/2 P(B2c ) ≤ 2 exp −C0 log2 N + 2 exp(− log2 N ) cK log N (S.24) ≤ C 0 n1/2 exp −C0 log2 N , P (1) where the first inequality follows kn−1 i∈Ij i KUλi −E[(1) KUλ ]k ≤ cK log nh−1/2 when maxj j ≤ log n and is subgaussian. Therefore combining (S.22) and (S.24), we derive that on B2 , there exists some constant C1 , (S.25)
|DLn,λ (f0j )[fn,λ ]| ≤ ||∇Ln,λ (f0j )||kfn,λ k
1/2
≤ C1 (log N (nh)−1/2 + λ1
For the second Fr´echet derivative, we have
1/2
+ λ2 )kfn,λ k.
D2 Ln,λ (f0j )[fn,λ , fn,λ ]
= h∇2 Ln,λ (f0j )[fn,λ ], fn,λ i 1X fn,λ (Ui )hKUλi , fn,λ i + hWλ fn,λ , fn,λ i = n i∈Ij
1X = fn,λ (Ui )hKUλi , fn,λ i − E[fn,λ (U )hKUλ , f i] n i∈Ij
(S.26)
+ E[fn,λ (U )hKUλ , f i] + hWλ fn,λ , f i 1X hfn,λ (Ui )KUλi − E[fn,λ (U )KUλ ], fn,λ i + kfn,λ k2 . = n i∈Ij
The empirical process in (S.26) is in the same form as the one in (S.10) with the only difference that fn,λ is in a different Hilbert space. Here we focus on bounding the supremum of the empirical process indexed by H0 = {f = (g, β) ∈ H : ||f ||sup ≤ 1, kgk2H ≤ 1, kβk2H ≤ 1}. We denote qn = (cH ∧ 1/2 c0 )(λ1 ∧λ2 )−1/2 , where cH is defined in Lemma S.1. Let f˚n,λ = qn−1 fn,λ , and it follows from Lemma S.1 again that under the event B1 = {λkfn,λ kH2 ≤ c0 }, we have kf˚n,λ ksup ≤ 1 and kf˚n,λ kH2 ≤ 1 and therefore f˚n,λ ∈ H0 . By the
NONPARAMETRIC HETEROGENEITY TESTING
49
definition of Ψ(f, X) given in (S.11), we have Ψ(f˚n,λ , X) = c−1 h1/2 f˚n,λ K λ (x) K
U
whose Lipchitz property is the same as (S.12). By applying Lemma S.6 to 1 X ˚ Gn (f˚n,λ ) := √ Ψ(fn,λ , U ) − E[Ψ(f˚n,λ , U )] . n i∈Ij
with d(·) = k · ksup and F = H0 , we have P || i∈Ij [fn,λ (Ui )KUλi − E(fn,λ (U )KUλ )]|| ≥t P B1 ∩ √ qn cK h−1/2 ( n ω(kqn−1 fn,λ ksup ) + 1) √ (j) nkGn (f˚n,λ )k >t = P B1 ∩ √ n ω(kf˚n,λ ksup ) + 1 √ nkGn (f )k ≤ P sup √ ≥t n ω(kf ksup ) + 1 f ∈H0 √ (S.27) ≤ 2 log( n ω(1)) + 2 exp(−t2 /4),
where the last inequality comes from (S.47). −1/2 log N (√n ω(kq −1 f For qn0 := n n,λ ksup )) + 1), we define the event PncK qn h B3 = || i=1 [fn,λ (Ui )KUλi − E(fn,λ (U )KUλ )]|| ≤ qn0 . By (S.27), we have √ P(B1 ∩ B3c ) ≤ 2 (log( n ω(1)) + 2) exp(− log2 N/4). Therefore, under the event B3 , we can bound the first term in (S.26) as n 1 X hfn,λ (Ui )KUλi − E[fn,λ (U )KUλ ], fn,λ i n i=1 n
1 X
fn,λ (Ui )KUλi − E[fn,λ (U )KUλ ] · fn,λ ≤ n i=1 √ ≤ cK n−1 qn h−1/2 log N ( n ω(kqn−1 fn,λ ksup ) + 1)kfn,λ k √ (S.28) ≤ cK n−1 qn h−1/2 log N ( n ω(qn−1 h−1/2 kfn,λ k) + 1)kfn,λ k,
where the last inequality follows from Lemma S.1 and the fact that ω(·) is non-decreasing. Combining (S.21),(S.25) and (S.26), it yields that under the events B2
50
J. LU, G. CHENG AND H. LIU
and B3 there exists a constant C 00 , 1 X kfn,λ k2 ≤ |DLn,λ (f0j )[fn,λ ]| + hfn,λ (Ui )KUλi − E[fn,λ (U )KUλ ], fn,λ i . n i∈Ij
−1/2
≤ C1 (log N (nh)
1/2
)kfn,λ k √ + cK n−1 qn h−1/2 log N ( n ω(qn−1 h−1/2 kfn,λ k) + 1)kfn,λ k. +λ
1/2
The inequality above implies that either kfn,λ k ≤ 2C1 (log N (nh)−1/2 + λ1 + 1/2 λ2 ) or (S.29)
kfn,λ k ≤ 2cK qn h−1/2 n−1/2 ω(qn−1 h−1/2 kfn,λ k) log N
According to (3.12), there exists a constant cr and rn = cr (log N (nh)−1/2 + 1/2 + λ2 ) such that
1/2 λ1
rn /ω(qn h−1/2 rn ) ≥ 2cK qn n−1/2 h−1/2 n1/2 log N.
It derives from (S.29) that rn /ω(qn h−1/2 rn ) ≥ kfn,λ k/ω(qn−1 h−1/2 kfn,λ k). Since ω ¯ a in Assumption 3.3 is non-increasing, we have kfn k ≤ rn . We conclude that on the events B2 and B3 , for the constant C = cr ∨ 2C1 , 1/2 1/2 log N √ + λ1 + λ2 . Therefore, for sufficiently large n, we have kfn,λ k ≤ C nh there exists a constant c0 , log N 1/2 1/2 √ P kfn,λ k > C + λ1 + λ2 nh ≤ P(B2c ) + P(B3c ) ≤ P(B1c ) + P(B1 ∩ B3c ) + P(B2c ) √ ≤ 2 exp(−c0 n) + 2 log( n ω(1)) + 2 exp(− log2 N/4) + C 0 n1/2 exp −C0 log2 N (S.30)
≤ 4n exp(−c0 log2 N ).
We therefore obtain the tail bound in Lemma S.4. APPENDIX S.3: RESULTS ON LIKELIHOOD RATIO TEST We first prove Lemma C.1 on the rate of remainder term of APLRTN,λ in subsection S.3.1 and then prove Lemma C.2 on the power of APLRTN,λ in subsection S.3.2. In the last subsection S.3.3, we complete the remaining proofs of Theorems 4.1 and 4.2, i.e., on PLRTn,λ and PLRTconst n,λ . We start from a preliminary lemma. Lemma S.5. Suppose (3.10), (3.11) and (3.12) hold and λ = λ1 ∧ λ2 = o(1). Define the full-sample estimate gbN,λ as g¯N,λ in the case s = 1. We have k¯ gN,λ − gbN,λ k = OP (N −1/2 ) and k¯ gN,λ − g0 k = OP (rN ), where recall that 1/2 1/2 −1/2 rN = log N (N h1 ) + λ1 + λ2 .
NONPARAMETRIC HETEROGENEITY TESTING
51
gN,λ − gbN,λ k = oP (N −1/2 ). Denote g0u = Proof. We first prove that k¯ g0 + W1,λ g0 . According to Lemma S.4, we have k¯ gN,λ − g0u − ∇LN,λ (g0 )k = −1/2 oP (N ). By the definition of gbN,λ (noting that (3.10), (3.11) and (3.12) satisfy for s = 1), it holds that kb gN,λ − g0u − ∇LN,λ (g0 )k = oP (N −1/2 ). Therefore, we have k¯ gN,λ − gbN,λ k ≤ k¯ gN,λ − g0u − ∇LN,λ (g0 )k + kb gN,λ − g0u − ∇LN,λ (g0 )k
(S.31)
= oP (N −1/2 ).
Actually, by the proof of Lemma S.4, we can bound the tail probability that for some constant C, C 0 , P k¯ gN,λ − gbN,λ k ≥ CN −1/2 −1/2 ≤ P k¯ gN,λ − gbN,λ k ≥ 2s an log N ≤ P k¯ gN,λ − g0u − ∇Ln,λ (g0 )k ≥ s−1/2 an log n + P kb gN,λ − g0u − ∇Ln,λ (g0 )k ≥ s−1/2 an log n = o(1), where the last equality follows from Lemma B.1. Therefore, we have
k¯ gN,λ − g0 k ≤ kb gN,λ − g0 k + k¯ gN,λ − gbN,λ k = OP (rN + N −1/2 ),
where the last equality is due to Lemma S.4 for s = 1. Since N −1/2 = O(rN ), the lemma is proven. (j) (j) S.3.1. Proof of Lemma C.1. Denote f0j = (g0 , β0j ), fb0 = (g0 , βbn,λ ), (j) (j) gN,λ = g¯N,λ − g0 , fˇ = (gN,λ , 0), f0 = (0, βbn,λ − β0j ). By applying the Taylor (j) (j) expansion to L at fb , we have n,λ
(S.32) APLRTN,λ =
n,λ
s 1 X (j) b(j) (j) (j) Ln,λ (fn,λ ) − Ln,λ (fb0 ) = I1 + I2 + I3 , s j=1
where
s
1X I1 := h∇L(j) (f0j ), fˇi, s I2 := I3 :=
1 s 1 s
j=1 s X j=1 s X j=1
(j) ∇2 L(j) (f0j )[f0 , fˇ],
1 2 (j) b(j) ˇ ˇ ∇ Ln,λ (f0 )[f , f ]. 2
52
J. LU, G. CHENG AND H. LIU
We first decompose I3 into two terms as follows s 1 X 1 2 (j) b(j) ˇ ˇ (j) (j) I31 := ∇ Ln,λ (f0 )[f , f ] − E{∇2 Ln,λ (fb0 )}[fˇ, fˇ] , s 2 I32 :=
1 s
j=1 s X j=1
1 (j) E{∇2 Ln,λ (f0j )}[fˇ, fˇ]. 2
According to Proposition A.2, we have s n
1 X −1
X ˇ λ λ ˇ |I31 | ≤ n f (Ui )KUi − E[f (U )KU ] kgN,λ k. s j=1
i=1
Lemma S.5 gives the rate of kgN,λ k = OP (N −1/2 ) and (S.20) gives the P P rate of s−1 sj=1 n−1 k ni=1 fˇ(Ui )KUλi − E[fˇ(U )KUλ ]k = OP (12an rN log N ). We can therefore control |I31 | as P (|I31 | > 12an rN log N ) = o(1).
(S.33)
Therefore, |I31 | = OP (an rN log N ). Applying Proposition A.2 again, the second term I32 = kfˇk2 /2 = kgN,λ k2 /2. Since an log N = o(N −1/2 ) and √ N h = o(1) yields N an rN log N = o(h−1/2 ), we derive that (S.34) Recall that GN
2N · I3 = N kgN,λ k2 + oP (h−1/2 ). P (j) (j) = gN,λ + s−1 sj=1 ∇L1 (f0j ), where ∇L1 (f0j ) is the first (j)
coordinate of ∇Ln,λ (f0j ) ∈ H2 , and satisfies 1X (j) i PXλ i + W1,λ g0 ∈ H1 . ∇L1 (f0j ) = − n i∈Ij
By (S.20), we have kGN k = OP (an log N ) = oP (N −1/2 ) and s
1 X
2
(j) 2 −1/2 (S.35) 2N I3 = N kgN,λ k +oP (h ) = N ∇L1 (f0j ) +oP (h−1/2 ). s j=1
Ps −1
(j)
2 Next we study the leading term N ks j=1 ∇L1 (fj,0 )k in the above equation. By Proposition A.2, P P (j) N ks−1 sj=1 ∇L1 (fj,0 )k2 = N −1 k N P λ k2 PNi=1 i Xi −2 i=1 i (W1,λ g0 )(Xi ) + N kW1,λ g0 k2 .
We bound the second term on the RHS above. It follows by the Fourier
NONPARAMETRIC HETEROGENEITY TESTING
53
expansions of W1,λ g0 in Proposition A.1 that ( N ) X E | i (W1,λ g0 )(Xi )|2 = N E{2 |W1,λ g0 (X)|2 } i=1
(S.36)
2
=σ N
X ν∈Z
2
|hg0 , φν iL2 P |
λ1 /µν 1 + λ1 /µν
2
= o(N λ1 ),
where the last equality is in the P derived based on the similar arguments 1/2 ) = o (h−1/2 ). derivation of (S.22). So N (W g )(X ) = o ((N λ ) i 1 P P 1,λ 0 i=1 i According to (S.22) again, kW1,λ g0 k2 = o(λ1 ). Consequently, N ks−1
s X j=1
(j)
∇L1 (fj,0 )k2 = N −1 k
N X i=1
i PXλ i k2 + o(N λ) + oP (h−1/2 ).
P λ 2 In what follows, we study the statistical rate of N −1 k N i=1 i PXi k . We write
N
2 N
1 1 X 2 λ
X λ P = i P (Xi , Xi ) + W (N ),
i Xi
N N i=1 i=1 P λ where W (N ) = i6=j i j P (Xi , Xj ). Here the W (N ) term is the leading term of (C.13). P 2 λ Finally, we bound N −1 N i=1 i P (Xi , Xi ) by studying the moment 2 X N 1 2 λ i P (Xi , Xi ) E N i=1 X 2 N 1 2 λ 2 λ =E i P (Xi , Xi ) − E[ P (X, X)] + (E[2 P λ (X, X)])2 N ≤ 2N
−1
i=1 4
E[ P λ (X, X)2 ] + (E[2 P λ (X, X)])2 = h−1 σP2 + o(1),
where the last equality is due to E[4 P λ (X, X)2 ] ≤ O(h−2 ) and the definition of σP2 . Then, as N −1 h−2 = o(1), we can derive that (S.37)
N 1 X 2 λ i P (Xi , Xi ) = h−1 σP2 + oP (1). N i=1
According to (S.36) and (S.37), we have s
2 X
(j) (S.38) N s−1 ∇L1 (fj,0 ) = OP (h−1 + N λ + h−1/2 ) = OP (h−1 ). j=1
54
J. LU, G. CHENG AND H. LIU
Now we turn to bound I1 . By the definition of fˇ = (gN,λ , 0), we have s s
1 X
2 1 X
(j) (j) I1 = − ∇L1 (f0j ) + h∇L1 (f0j ), GN i s s j=1
(S.39)
j=1
s
1 X
2
(j) = − ∇L1 (f0j ) + oP ((N 2 h)−1/2 ), s j=1
where the last inequality follows from (S.38) and (S.20). Finally, we calculate I2 by s s 1 X (j) ˇ 1 X bj (S.40) I2 = hf0 , f i = (0, βn,λ − β0j ), (¯ gN,λ − g0 , 0) = 0. s s j=1
j=1
Combining (S.35), (S.39) and (S.40), it follows that s
2
1 X
(j) ∇L1 (f0j ) + oP (h−1/2 ). −2N · APLRTN,λ = N (S.41) s j=1
Moreover, using (S.38), (S.37) and (S.36), we have
−2N · APLRTN,λ = W (N ) + h−1 σP2 + N kW1,λ g0 k2 + oP (h−1/2 )
Therefore, we complete the proof of the lemma.
S.3.2. Proof of Lemma C.2. We separate the APLRTN,λ into two parts: APLRTN,λ =: T1 − T2 , where s o 1 X n (j) (j) (j) (j) Ln,λ (¯ T1 := gN,λ , βbn,λ ) − Ln,λ (gN , βbn,λ ) , s T2 :=
1 s
j=1 s n X j=1
o (j) (j) (j) (j) Ln,λ (g0∗ , βbn,λ ) − Ln,λ (gN , βbn,λ ) .
We next bound T1 and T2 . Note that T1 can be viewed as APLRT statistic for testing H0 : g0 = gN v.s. H1 : g0 6= gN . We next check the proof of Theorem 4.5 to show that (2uN )−1/2 (rP T1 − N kW1,λ gN k2 − uN ) = OP (1) uniformly over β01 , . . . , β0s , gN ∈ Fζ . In the proofs of Lemma S.13 and Theorem 4.5, the results are only related to the magnitudes of kgN kH , kW1,λ gN k and kβ0j kH , kW2,λ β0j k for 1 ≤ j ≤ s. Other constants are only dependent on the kernel function, eigen-functions ord . By the definition of Fζ , we can uniformly bound kgN kH ≤ ζ and kW1,λ gN k2 ≤ λkgN k2 ≤ ζ 2 by (S.22) and similarly for β0j for all j = 1, . . . , s. (j) We next give a uniform bound of T2 . Denoting fN = (gN , βbn,λ ), ∆g =
NONPARAMETRIC HETEROGENEITY TESTING
55
gN − g0∗ and fN 0 = (gN − g0∗ , 0), we compute the expectation (j) (j) (j) ∗ b(j) b E Ln,λ (g0 , βn,λ ) − Ln,λ (gN , βn,λ ) 1 (j) (j) = E ∇Ln,λ (fN )[fN 0 ] + ∇2 Ln,λ (fN )[fN 0 , fN 0 ] 2 1 (j) (S.42) = Eh∇Ln,λ (fN ), fN 0 i + λ1 h∆g, gN iH + k∆gk2 . 2 P (j) (j) Recall that Hn = βb − β u + n−1 i Zi T λ . By Cauchy-Schwarz n,λ
j0
i∈Ij
inequality, Lemma S.1 and Lemma S.4, we have (S.43) (j) Eh∇Ln,λ (fN ), fN 0 i
Xi
(j) (j) (j) = Eh∇Ln,λ ((gN , βj0 )), fN 0 i + hE∇2 Ln,λ ((gN , βj0 ))[βbn,λ − βj0 ], ∆gi p ≤ hWλ,2 βj0 , ∆gi + EkHn(j) kk∆gk ≤ o( λ2 )k∆gk, √ where the last inequality is due to (S.22), Lemma S.4 and an log N = oP ( λ2 ). Since ∆g ∈ Fζ , we also have h∆g, gN iH ≤ |hg0∗ , ∆giH | + h∆g, ∆giH ≤ kg0∗ kH ζ 1/2 + ζ. (j) (j) Denote ∆βn,λ = βbn,λ − β0j , the variance of T2 is 2 s 1 X (j) (j) (j) 2 ∗ b(j) b E |T2 − E[T2 ]| ≤ 2 E Ln,λ (g0 , βn,λ ) − Ln,λ (gN , βn,λ ) s j=1
s i n X h (j) E |21 ∆g(X1 ) + 2Z1 ∆βn,λ (X1 )∆g(X1 ) + Z12 ∆g(X1 )2 |2 ≤ 2 N j=1
(S.44) ≤ O(k∆gk2 + rn2 k∆gk2 + k∆gk4 ),
where the last inequality holds uniformly according to the definition of Fζ 2 ]. Combining (S.42), (S.43) and (S.44), we have that Var(∆g 2 ) ≤ ζE2 [gN √ −2N T2 = −N k∆gk2 + OP (N λ1 + N λ2 k∆gk +N 1/2 k∆gk + N 1/2 rn k∆gk + N 1/2 k∆gk2 ), uniformly for β01 , . . . , β0s , ∆g ∈ Fζ .
56
J. LU, G. CHENG AND H. LIU
With the two parts ready, the normalized averaged statistic (2uN )−1/2 (−2N rP · APLRTN,λ − uN )
≥ N (2uN )−1/2 k∆gk2 (1 + OP (λ1 k∆gk−2 +
p λ2 k∆gk−1
+ N −1/2 + N −1/2 k∆gk−1 )) + (2uN )−1/2 N kW1,λ g0 k2 + OP (1) (S.22) p ≥ N (2uN )−1/2 k∆gk2 (1 + OP (λ1 k∆gk−2 + λ2 k∆gk−1 + N −1/2 + N −1/2 k∆gk−1 )) + OP (1),
uniformly for β01 , . . . , β0s , ∆g ∈ Fζ .
S.3.3. Remaining Proofs of Theorems 4.1 and 4.2. To complete the proofs of Theorems 4.1 and 4.2, we prove the properties of PLRTn,λ . Also, we prove the properties of PLRTconst n,λ defined in Remark 4.3. • Analysis of PLRTn,λ . We follow a similar proof strategy as Theorem 4.5 and only outline their difference here. Recall that the first coordinate of (1) the derivative ∇Ln,λ ((g0 , β0j )) is denoted as 1X (1) i Zi TXλ i + W2,λ β0j ∈ H. ∇L2 (f0j ) = − n i∈Ij
By the Taylor expansion, we have (1)
−1/2
−2n · PLRTn,λ = nk∇L2 (f0j )k2 + oP (h2
).
(1)
Expanding k∇L2 (f0j )k2 yields the U-statistic X (j) 2u v Zu Zv T λ (Xu , Xv ) W3 (n) = u t ≤ 2 log nωd (1; F) + 2 exp(−t2 /4). n ωd (1; F) + 1 f ∈F
Remark S.7. The above concentration inequality is similar as the tail inequality of continuity moduli of empirical processes, which was initiated by Gin´e et al. (2003); Gin´e and Koltchinskii (2006) in a measurable function class. Our result here focuses on Hilbert spaces, and fits into the RKHS framework in consideration. Proof. For simplicity, we will write ωd (δ; F) as ω(δ) in the proof. Since for any f, g ∈ F, kΨ(f, X) − Ψ(g, X)k ≤ d(f − g), it follows Theorem 3.5 of Pinelis (1994), for any t > 0, 2 kGn (f ) − Gn (g)k t ≥ t ≤ 2 exp − . P d(f − g) 8
Therefore, Gn (f ) is a subgaussian empirical process such that by Lemma 8.1 in Kosorok (2008), we have kkGn (g) − Gn (f )kkφ2 ≤ 8d(f − g), where k · kφ2 is the Orlicz norm associated with φ2 (s) ≡ exp(s2 ) − 1. Define Fδ = {f ∈ F | d(f ) ≤ δ}. Theorem 8.4 of Kosorok (2008) implies that there exists a constant C1 , for an arbitrary δ > 0,
Z δp
sup kGn (f ) − Gn (f )k ≤ C1 log(1 + N (F; d, ))d = C1 ω(δ).
f −g∈Fδ
φ2
0
We apply Lemma 8.1 in Kosorok (2008) again and obtain, t2 (S.48) P sup kGn (g)k ≥ t ≤ 2 exp − 2 . C1 ω(δ)2 f ∈F d(f )≤δ
We next apply the peeling argument. Define the set V0 := {f ∈ F : nω(d(f )) ≤ 2} and √ Vm = {f ∈ F : 2m−1 ≤ nω(d(f )) ≤ 2m } for 1 ≤ m ≤ M, √ where the index m ranges up to M := dlog nω(1)e as ω(·) is increasing and √
NONPARAMETRIC HETEROGENEITY TESTING
d(f ) ≤ 1 for all f ∈ F. Using union bound, we have ! √ n||Gn (f )|| P sup √ >t nω(d(f )) + 1 f ∈F ! √ M X n||Gn (f )|| ≥t ≤ P sup √ nω(d(f )) + 1 f ∈Vm m=0 ! √ n||Gn (f )|| ≥ t ≤P sup √ d(f )≤ω −1 (2/ n)
+
M X
m=1
P
sup
√ d(f )≤ω −1 (2m / n)
√
n||Gn (f )|| ≥ (1 + 2m−1 )t
59
!
√ 2 ! X √ 2 ! M t/ n (1 + 2m−1 )t/ n √ √ ≤ 2 exp − + 2 exp − m/ n 2C1 / n C 2 1 m=1 t2 ≤ 2(M + 1) exp − 2 . 4C1
The second last inequality follows from (S.48). We can therefore derive (S.47) by the fact that d(f ) ≤ 1 for all f ∈ F and ω(·) is an increasing function. Lemma S.8. (Pinelis, 1994) If X1 , . . . , Xn are zero mean independent random variables in a separable Hilbert space and kXi k ≤ C for i = 1, . . . , n, then
n
r !
1 X 2t
P < 2e−t . Xi > C
n
n i=1 √ Lemma S.9. Define ` = ( 5 − 1)/2, I1 = π/(2m sin(π/(2m))) and I2 = (2m − 1)π/(4m2 sin(π/(2m))). Under H0m [0, 1], we choose λi = h2m for i = i −1/p 1, 2, while under the Gaussian RKHS HG we choose hi = (log(1/λi )) for i = 1, 2. We obtain the following constants needed in the inference results: Sobolev space : σg2 ∼
σ 2 I2 2 σ 2 I2 σ 4 I2 2 σ 2 I1 , σβ ∼ 2−1/m , ρ2P ∼ , σP ∼ , π π π σz π 1/m
1/m
σ 4 σ z I2 2 σ 2 σ z I1 , σT ∼ ; π π σg2 ≤ 1.7178σ 2 , σβ2 ≤ 1.7178(σ/σz )2 , ρ2P ∼ σ 4 log(1/`), ρ2T ∼
Gaussian RKHS :
σP2 ∼ σ 2 log(1/`), ρ2T ∼ σ 4 log(1/`) and σT2 ∼ σ 2 log(1/`).
60
J. LU, G. CHENG AND H. LIU
Proof. For the Sobolev space, by definitions, we have ∞ X σg2 = lim σ 2 h1 1 + 2 (1 + λ1 (2πk)2m )−2 , N →∞
k=0
ρ2P = lim σ 4 h1 1 + 2 N →∞
σP2
2
= lim σ h1 1 + 2 N →∞
∞ X k=0
∞ X
(1 + λ1 (2πk)2m )−2 , (1 + λ1 (2πk)
2m −1
)
k=0
.
We calculate the infinite summation Z ∞ ∞ Z 2πh1 (k+1) ∞ X X 1 1 2πh1 ≤ dx ≤ dx 2m )l (1 + (2πh1 k)2m )l (1 + x (1 + x2m )l 0 2πh1 k k=0
k=1
and similarly, we have ∞ X k=0
2πh1 ≥ (1 + (2πh1 k)2m )l
Z
∞
0
1 dx. (1 + x2m )l
Therefore we have the asymptotic variance Z σ2 ∞ 1 σ 2 I2 2 σg ∼ dx = . π 0 (1 + x2m )2 π
Similar arguments apply to ρ2P and σP2 . Recall the definitions of the remaining three constants: ∞ X (σz2 + λ2 (2πk)2m )−2 , σβ2 = lim σ 2 σz2 h2 1 + 2 N →∞
k=0
ρ2T = lim σ 4 σz4 h2 1 + 2 N →∞
∞ X k=0
(σz2 + λ2 (2πk)2m )−2 ,
∞ X σT2 = lim σ 2 σz2 h2 1 + 2 (σz2 + λ2 (2πk)2m )−1 . N →∞
k=0
The computation of the above constants can be reduced to those of the previous ones by the change of variable. For example, we rewrite σβ2 by Z ∞ ∞ X 2πh2 σ 2 σz−2 σ2 dx σ 2 I2 2 σβ ∼ ∼ = 2−1/m πσz2 0 (1 + σz−2 x2m )2 (1 + σz−2 (2πh2 k)2m )l πσz k=1
We can evaluate σT2 and ρ2T in the same way.
61
NONPARAMETRIC HETEROGENEITY TESTING
As for the Gaussian RKHS, we have ∞ X −2 σg2 = lim σ 2 h1 1 + λ1 ` exp(−2 ln ` · k) , N →∞
k=0
ρ2P = lim σ 4 h1
∞ X
N →∞
k=0
σP2 = lim σ 2 h1 N →∞
∞ X k=0
1 + λ1 ` exp(−2 ln ` · k) 1 + λ1 ` exp(−2 ln ` · k)
−2
−1
, .
To be more general, we set h1 = (log(1/λ1 ))−1/p and will show that ∞ X
h1 ∼ α−1/p . (1 + λ1 β exp(αk p ))l
k=0
For p = 1, it gives the result for Gaussian RKHS. On one hand, according to the convexity, we have Z ∞ ∞ X dx h1 ≤ p l (1 + λ1 β exp(αk )) (1 + λ1 β exp(αxp ))l 0 k=1
We can further study the integration Z ∞ dx (1 + λ1 β exp(αxp ))l 0 Z (α−1 log(1/λ1 ))1/p Z ∞ dx = + p l 0 (α−1 log(1/λ1 ))1/p (1 + λ1 β exp(αx )) Z ∞ ≤ (α−1 log(1/λ1 ))1/p + (λ1 β)−l exp(−αlxp )dx ≤ (α
−1
1/p
log(1/λ1 ))
(α−1 log(1/λ1 ))1/p −1 −l −1
+ αp
β p(α
log(1/λ1 ))−(p−1)/p .
On the other hand, for any t ∈ (0, 1), Z ((t/α) log(1/λ1 ))1/p ∞ X dx h1 ≥ p l (1 + λ1 β exp(αk )) (1 + λ1 β exp(αxp ))l 1 k=1
≥ Therefore ,
1 1/p −1 . 1−t l ((t/α) log(1/λ1 )) (1 + λ1 β)
1/p 1/p ∞ X t h1 1 ≤ lim ≤ , p l N →∞ α α (1 + λ1 β exp(αx )) k=0
and letting t → 1, we achieve the desired result.
62
J. LU, G. CHENG AND H. LIU
The eigen-functions of the Gaussian RKHS are (Zhu et al., 1998): √ √ √ 2 φk (x) = ( 5/4)1/4 (2k−1 k!)−1/2 e−( 5−1)x /4 Hk ( 5/2)1/2 x ,
where Hk (·) is the k-th Hermite polynomial, for k = 0, 1, 2, · · · . The eigen√ 2k+1 values are µk = ` , where ` = ( 5 − 1)/2, can also be re-written as an exponential form: µk = ` exp(2 ln ` · k). According to Krasikov (2004), we can get that √ 2e15/8 ( 5/4)1/4 √ cK = sup kφk ksup ≤ ≤ 1.336. 3 2π21/6 k∈N Therefore, we have ∞ X 2 2 2 σg ≤ lim σ cK h1 (1+λ` exp(−2 ln `·k))−2 ≤ c2K ·2σ 2 log(1/`) ≤ 1.7178σ 2 , N →∞
The values of
k=0
σβ2 , ρ2T
and σT2 can be obtained by the rescaling as well.
