Law of large numbers

1

Law of large numbers Sayan Mukherjee We revisit the law of large numbers and study in some detail two types of law of large numbers   Sn 0 = lim P | − p| ≥ ε ∀ε > 0, Weak law of larrge numbers n→∞ n   Sn 1 = P ω : lim =p , Strong law of large numbers n→∞ n Weak law of large numbers We study the weak law of large numbers by examining less and less restrictive conditions under which it holds. We start with a few prelimary concepts that are useful. 1. Truncation: We replace the random sequence {Xn } with a truncated version {Xn I[|Xn |≤n] }. 1. Tail equivalence: A common property of the truncated sequence that we will try to exploit is tail equivalence. Two sequence {Xn } and {Xn0 } are tail equivalent if X P(Xn 6= Xn0 ) < ∞. n

We will prove something nice about the truncated sequence {Xn I[|Xn |≤n] } anmd then prove it is tail equivalent to {Xn }. Proposition 0.0.1 Suppose {Xn } and {Xn0 } are tail equivalent. Then − Xn0 ) converges a.s. P P P P 0 0 2) n Xn and n Xn converges or diverges a.s. or n Xn converges a.s. iff n Xn converges a.s. 1)

P

n (Xn

3) If there exists a sequence {an } such that an ↑ ∞ and there exists a random variable X such that if n n 1 X 0 a.s. 1 X a.s. Xj → X ⇒ X → X. an j=1 an j=1 j

Proof. For (1) we use Borel-Cantelli P([Xn 6= Xn0 ] i.o. )

=

0

Xn0 ])

=

1

P(lim inf [Xn 6= n→∞

so if we set ω ∈ {lim inf n→∞ [Xn 6= Xn0 ]} this implies Xn (ω) = Xn0 (ω) or n ≥ N (ω). For (2)

∞ X n=N

For (3)

Xn (ω) =

∞ X

Xn0 (ω).

n=N

n 1 X a.s (Xj − Xj0 ) → 0.  an j=1

The following theorem provides necessary and sufficient conditions for weak law of large numbers. These are the weakest conditions required.

2 Theorem Pn 0.0.1 (General law of large numbers) Suppose {Xn , n ≥ 1} are independent random variables and Sn = j=1 Xj . If i) ii)

Pn

P(|Xj | > n) → 0 Pn 2 j=1 E(Xj I[|Xj |≤n] ) = 0

j=1

1 n2

then for an =

n X

Sn − an P → 0. n

E(Xj2 I[|Xj |≤n] ),

j=1

Proof.

We prove sufficiency.

Define 0 Xnj = Xj I[|Xj |≤n] ,

Sn0 =

n X

0 Xnj .

j=1

Observe

n X

0 P[Xnj 6= Xj ] =

j=1

n X

P(|Xj | > n) → 0,

j=1

so P(|Sn − Sn0 | ≥ ε) ≤ P[Sn 6= Sn0 ] ≤ P(∪[Xnj 6= Xj ]) n X 0 ≤ P(Xnj 6= Xj ) → 0, j=1

so

P

Sn − Sn0 → 0. Since Var(X) = E(X 2 ) − (EX)2 ≤ E(X 2 ) so   0 Sn − ESn0 ≤ P >ε n ≤

≤ Set an = ESn =

Pn

j=1

Var(Sn0 ) n2 ε2 n 1 X n2 ε2 1 n2 ε2

02 E(Xnj )

j=1 n X

E(Xj2 I[|Xj |≤n] ) → 0.

j=1

E(Xj2 I[|Xj |≤n] ) so Sn − an n Sn − Sn0 + Sn0 − an n Sn − an n

P

→ P

→ P

→

0, 0, 0. 

The following example illustrates that one can have a law of large numbers even if the first moment is not bounded. Example 0.0.1 F is a symmetric distribution function such that 1 − F (x) = and F (x) =

e , 2x log(x)

e , −2x log(−x)

x ≥ e,

x ≤ −e.

3 First observe that EX + = EX − = ∞ so the first moment does not exist since Z ∞ Z e e ∞ dy EX + = dx = = ∞. 2 log(x) 2 1 y e Set τ (x) = XP(|X| > x) =

e log x

→ 0. Also set an = 0 since F is symmetric so Sn P → 0, n

the weak law of large numbers holds, the strong law does not. In the following we weaken conditions under which the law of large numbers hold and show that each of these conditions satisfy the above theorem. Example 0.0.2 (Bounded second moment) If {Xn , n ≥ 1} are iid random variables with E(Xn ) = µ and E(Xn2 ) < ∞ then 1X P Xn → µ. n i) nP(|X1 | > n) ≤ ii)

nE(X12 ) n2

1 2 n2 nE(X1 I[|X1 |≤n] )

≤

→0

1 2 n E(X1 )

→0

Example 0.0.3 (Khintchin’s WLLN) If {Xn , n ≥ 1} are iid random variables with E(Xn ) = µ and E(|Xn |) < ∞ then 1X P Xn → µ. n i) nP(|X1 | > n) = E(nI[|X1 |>n] ) ≤ E(|X1 |I[|X1 |>n] ) → 0 ii) 1 E(X12 I[|X1 |≤n] ) ≤ n




1 E X12 I[|X1 |≤ε√n] + E X12 I[|X1 |ε√nkX1 |≤n] n 1 ε2 n + E(n|X1 |I[ε√n≤|X1 |≤n] ) ≤ n n ≤ ε2 + E(|X1 |I[ε√n≤|X1 |] )

≤ ε2 → 0. So

Sn − E(X1 I[|X1 |≤n] P →0 n nE(X1 I[|X1 |≤n] − EX 1 ≤ E(|X1 |I[|X1 |>n] ) → 0. n

Example 0.0.4 (Feller’s WLLN) If {Xn , n ≥ 1} are iid random variables with lim xP(|X1 | > x) = 0,

x→∞

then

Sn P − E(X1 I[|X1 |≤n] ) → 0. n

Strong law of large numbers We want to understand the conditions under which Sn − E(Sn ) a.s. → 0. bn Start with a few results that we will need in proving SSLNs.

4 Theorem 0.0.2 P(L´evy) If {Xn , n ≥ 1} is an independent sequence of random variables then probability iff Xn converges almost surely and for Sn the following are equivalent

P

Xn converges in

1) {Sn } is Cauchy in probability 2) {Sn } converges in probability 3) {Sn } converges in almost surely 3) {Sn } is almost surely Cauchy. The following convergence criterion will be used. Theorem 0.0.3 (Kolmogorov) Suppose {Xn , n ≥ 1} is an independent sequence of random variables. If ∞ X

V ar(Xj ) < ∞

j=1

then

∞ X (Xj − E(Xj )) j=1

converges almost surely. Proof. Without loss of generality set E(Xj ) = 0, so Cauchy so kSn −

Sm k2L2

P∞

j=1

E(Xj2 ) < ∞. This implies that {Sn } is L2 ∞ X

= Var(Sn − Sm ) =

EXj2 → 0.

j=m+1 2

{Sn } is L Cauchy so {Sn } is Cauchy in probability and so converges almost surely.



Lemma 0.0.1 (Kronecker’s lemma) Given sequences {xk } and {an } such that xk R and 0 < an ↑ ∞. If P ∞ xk k=1 ak converges then n X lim a−1 xk = 0. n n→∞

k=1

Kronecker’s lemma with the Kolmogorov convergence criteria immediately provides a SLLN. Corollary 0.0.1 {Xn , n ≥ 1} is an independent sequence of random variables such that E(Xn2 ) < ∞ . Givena monotone sequence bn ↑ ∞. If X Xk Var( ) 0

∞ X

P(|X1 | > εn) < ∞.

n=1

Theorem 0.0.4 (Kolmogorov’s SLLN) If {Xn , n ≥ 1} is an iid sequence of random variables and Sn = There exists c ∈ R such that Sn a.s. → c n iff E(|X1 |) < ∞ and c = E(X1 )

P

Xn .

Proof. We show

Sn a.s. n →

c ⇒ E(|X1 |) < ∞, Sn − Sn−1 Sn n − 1 Sn−1 Xn = = − → c − c = 0. n n n n n−1

Since

Xn a.s. n →

0 this implies E(|X1 |) < ∞.



P∞Almost sure convergence can be proven when the Kolmogorov convergence criterion does not hold, j=1 Var(Xj ) < ∞. This is given by the three series theorem of Kolmogorov. Theorem P 0.0.5 (Kolmogorov) Let {Xn , n ≥ 1} be a sequence of independent random variables. In order for Sn = Xn to converge almost surely it is necessary and sufficient for there to exist a c > 0 such that 1)

P

P(|Xn | > c) < ∞

2)

P

Var(Xn I[|Xn |≤c] ) < ∞

3)

P

E(Xn I[|Xn |≤c] ) converges.

n

n

n

Proof. We prove sufficiency. Define Xn0 = Xn I[] |Xn | ≤ c. To prove (1) X n

P(Xn0 6= Xn ) =

X

P(|Xn | > c) < ∞

n

P P so {Xn } and {X,0 } are tail equivalent and Xn converges a.s. iff Xn0 conveges a.s. P P To prove (2) observe n Var(Xn0 )∞ so (Xj0 − E(Xj0 ) converges a.s. P To prove (3) we see that n E(Xn0 ) converges.