Eigentime identity for asymmetric finite Markov chains

Front. Math. China 2010, 5(4): 623–634 DOI 10.1007/s11464-010-0067-8

Eigentime identity for asymmetric finite Markov chains∗ Hao CUI, Yong-Hua MAO School of Mathematical Sciences; Key Laboratory of Mathematics and Complex Systems, Ministry of Education, China, Beijing Normal University, Beijing 100875, China

c Higher Education Press and Springer-Verlag Berlin Heidelberg 2010 

Abstract Two kinds of eigentime identity for asymmetric finite Markov chains are proved both in the ergodic case and the transient case. Keywords Asymmetric Markov chain, eigenvalue, hitting time, Jordan decomposition MSC 60J10, 60J25, 15A51, 15A18

1 Introduction Let (Xt )t0 be an ergodic continuous-time Markov chain on a denumerable state space E with transition probability matrix P (t) = (pij (t)) and Q-matrix Q = (qij ). Refer to [3,5,6,9] for more knowledge about Markov chains. Let π = (πj > 0 : j ∈ E) be the stationary distribution, and let τj = inf{t  0 : Xt = j} be the hitting time of j. Define  πi πj Ei τj . (1.1) T = ij

T is called the average hitting time [1, Chap. 3], which has found its applications in bounding mixing time for Markov chains. In [13], T is also used via shift coupling to get the upper bound the convergence rates of averages as t → ∞,    1  t    p (t) − π ij j . t 0 j∈E

See also [2]. From [1, Chap. 3], we know that for any i ∈ E, ∗

Received December 15, 2009; accepted May 12, 2010

Corresponding author: Yong-Hua MAO, E-mail: [email protected]

624

Hao CUI, Yong-Hua MAO

T =



πj Ei τj .

(1.2)

j

When the chain is reversible, that is, πi pij (t) = πj pji (t) or πi qij = πj qji ,

i, j ∈ E,

then Q, viewed as an operator from L2 (π) to L2 (π), is self-adjoint. All eigenvalues of −Q are real and non-negative, and denoted by λ0 , λ1 , λ2 , . . . . Since Q is conservative ( j∈E qij = 0) and irreducible, λ0 = 0 and its multiplicity is one. The following so-called eigentime identity and its analogue in the discrete-time case are first proved in [1, Chap. 3] in the case where the state space E is finite:  λ−1 (1.3) T = n . n1

The eigentime identity (1.3) is then generalized to continuous-time reversible Markov chains on countable state spaces, both in the ergodic case ([10]) and in the transient case ([11]). In the ergodic case, T is also connected to the strong ergodicity and the spectral property of Q in Hilbert space L2 (π). For reversible Markov chains on a countable state space, it was proved in [10] that T < ∞ implies that the essential spectrum of Q in L2 (π) is empty. For the birth and death process on Z+ , T < ∞ is equivalent to the strong ergodicity. In the transient case, it was proved in [11] that a new form of the eigentime identity holds for symmetric and transient continuous-time Markov chains, see (1.5) below. Also, this identity is connected to the uniform decay of the transient chains. In this paper, we extend the above two kinds of eigentime identity to asymmetric finite Markov chains. When the chain is asymmetric, the situation is more complicated. First of all, the eigenvalues of −Q may be complex numbers, so we have to study the problem over complex number field C. Second, since −Q is not assumed to be symmetric, it may not be diagonalizable; we need to find a new formula in place of Kendall’s spectral representation for the symmetric Markov chains, cf. [3, Sect. 1.6]. Kendall’s spectral representation is essentially derived from the classical spectral representation theorem for a self-adjoint operator in Hilbert space. Instead, we shall utilize the subtle and refined spectral theory for the general finite matrix (Q-matrix), such as Jordan decomposition, matrix function. Theorem 1.1 For an ergodic continuous-time Markov chain on a finitestate space E = {0, 1, . . . , N } with Q-matrix Q = (qij ), let λ1 , λ2 , . . . , λs be the distinct non-zero eigenvalues of −Q such that the multiplicity of λk is mk . Let T be defined in (1.1). Then the eigentime identity holds : T =

s  mk k=1

λk

.

(1.4)

Eigentime identity for asymmetric finite Markov chains

625

Remark 1.2 Note that if λ is an eigenvalue of −Q, then its conjugate number λ is also an eigenvalue, and 1 1 λ+λ + = , λ λ |λ|2 so the right-hand side of (1.4) is real and positive. t or P(t) = ( Now, we turn to the transient case. Let X pij (t)) be an irreducible but non-conservative continuous-time Markov chain on {1, 2, . . . , N  = ( N }. Its Q-matrix is Q qij ), and there exists an i, such that j=1 qij < 0. 1 , λ 2 , . . . , λ N , are non is non-conservative, all N eigenvalues of −Q,  λ Since Q zero. Actually, from the spectral representation (see Proposition 2.3 below), 2 , . . . , λ N are positive. 1 , λ we know that the real parts of λ Theorem 1.3 For an irreducible, transient continuous-time Markov chain 2 , . . . , λ s be the distinct eigenvalues of −Q 1 , λ  such  = {1, 2, . . . , N }, let λ on E   k. that the multiplicity of λk is m + t = j} be the (first) return time (1) Let τj = inf{t > the first jump : X to j, with the convention inf ∅ = ∞. Then the eigentime identity holds : s  m k k=1

k λ

=

N 

1

q P [ τ+ j=1 j j j

= ∞]

,

(1.5)

qjj > 0. where qj = −  be the lifetime of X t . Then t ∈  E} (2) Let ζ = inf{t  0 : X sup Ej ζ 

1jN

s  m k k=1

k λ



N 

Ej ζ  N sup Ej ζ.

j=1

(1.6)

1jN

The eigentime identities (1.4) and (1.5) hold also for general discrete-time finite Markov chains. For discrete-time Markov chains, refer to [9]. Let (Xn )n0 be an ergodic discrete-time Markov chain on a finite state space E = {0, 1, . . . , N } with transition probability matrix P = (Pij ), such that its stationary distribution is (πi )i∈E . Let τj = inf{n  0 : Xn = j} be the hitting time of the state j. Define as before the average hitting time T as  T = πi πj Ei τj . (1.7) i,j∈E

Theorem 1.4 For an ergodic discrete-time Markov chain (Xn )n0 on state space E with transition probability matrix P = (Pij ), let λ1 , λ2 , . . . , λs be the distinct eigenvalues (other than 1) of P such that the multiplicity of λk is mk . Let T be defined in (1.7). Then the eigentime identity holds : T =

s  k=1

mk . 1 − λk

(1.8)

626

Hao CUI, Yong-Hua MAO

n )n0 be an irreducible and transient discrete-time Markov chain Let (X  = {1, . . . , N } with sub-stochastic matrix P = (Pij ), and there exists on E  an i, such that j∈E Pij < 1. Since P is sub-stochastic, all its N eigenvalues 1 , λ 2 , . . . , λ s are different from 1. λ  = {1, 2, . . . , N }, let Theorem 1.5 For the above sub-Markov chain on E 1 , λ 2 , . . . , λ s such that the multiplicity of  be λ the distinct eigenvalues of −Q k is m λ  k. n = j} be the (first) return time to j, with the (1) Let τj+ = inf{n  1 : X convention inf ∅ = ∞. Then the eigentime identity holds : s  k=1

N

 m k 1 . = +  Pj [ τj = ∞] 1 − λk

(1.9)

j=1

n ∈  be the lifetime of X n . Then (2) Let ζ = inf{n  0 : X  E} sup Ej ζ 

1jN

s  k=1

N

 m k  Ej ζ  N sup Ej ζ. k 1jN 1−λ

(1.10)

j=1

The rest of the paper is organized as follows. In Section 2, we prove Theorems 1.1 and 1.3 for the continuous-time case, and in Section 3, we prove Theorems 1.4 and 1.5 for the discrete-time case. Finally, in Section 4, we present several examples to illustrate these theorems.

2 Continuous-time Markov chain Let us first recall some classical results concerning spectral theory of matrices. See, for example, [4,12]. Let MK (C) be the collection of K × K matrices over the complex number C. For A ∈ MN (C) with eigenvalues λ1 , . . . , λs such that the multiplicity (index) of λk is mk , and for a function f : C → C such that f (λk ), f  (λk ), . . . , f (mk −1) (λk ) exist for each λk , the value of f (A) is f (A) =

s m k −1 

f (j) (λk )Ekj ,

(2.1)

k=0 j=0 j

d f where f (j) (λ) = dλ j , and the component matrices or the constituent matrices Ekj have the following properties: s 1. k=0 Ek0 = I (identity matrix); 2. Eki Elj = 0 for k = l; 2 3. Eki = Eki if and only if i = 0; 4. Eki Ek0 = Eki , ∀ k, i.

Eigentime identity for asymmetric finite Markov chains

Indeed, for k = 0, 1, . . . , s,



j : j=k Ek0 = 

627

(A − λj I)

j : j=k (λk

− λj )

,

(2.2)

and for i = 0, 1, . . . , mk − 1, 1 (A − λk I)i Ek0 . (2.3) i! is nilpotent (cf. [12, Chap. 7]), then tr (Eki ) = 0 for i  1. So Eki =

For i  1, Eki we have

Lemma 2.1 tr (Ek0 ) = mk and tr (Eki ) = 0 for i  1.   Proof Since tr (A) = sl=0 ml λl , we have tr (An ) = sl=0 ml λnl for n  1 by spectral mapping theorem. Noting that    (A − λj I) = As−1 − As−2 λj + · · · + (−1)s−1 I λj , j : j=k

j : j=k

j : j=k

it follows by linearity that

 (A − λj ) tr =

=

j : j=k s  l=0 s  l=0

= mk

ml λs−1 − l 

ml

s  l=0

ml λs−2 l



λj + · · · + (−1)s−1

j : j=k

s  l=0

ml



λj

j : j=k

(λl − λj )

j : j=k



(λk − λj ),

j : j=k



so that tr (Ek0 ) = mk by (2.2). Now, we apply the spectral theory above to the finite Markov chain.

Proposition 2.2 For the regular Q-matrix Q on E = {0, 1, . . . , N } as in Theorem 1.1, such that the component matrices are Ekj , k = 0, 1, . . . , s, j = 0, 1, . . . , mk , we have P (t) = Π +

s m k −1 

ti e−λk t Eki ,

(2.4)

k=1 i=0

where Π = (Πij ) is such that Πij = πj and the real part of λk is positive for k  1. Proof Since λ0 = 0 and its multiplicity is one by irreducibility of Q, it follows from (2.1) that P (t) = etQ =

s m k −1  k=0 i=0

ti e−λk t Eki = E00 +

s m k −1  k=1 i=0

ti e−λk t Eki .

628

Hao CUI, Yong-Hua MAO

If λk = ak + ibk , ak , bk ∈ R, then e−λk t = e−ak t (cos bk t − sin bk t), so that limt→∞ ti e−λk t = 0 if and only if ak > 0. Since P (t) is ergodic, limt→∞ P (t) = Π and the real part of λk is positive for k  1. Letting t → ∞ in the above equality, we see that E00 = Π.  For the transient Q-matrix, we can get the following proposition directly from (2.1).  on E = {1, 2, . . . , N } as Proposition 2.3 For the transient Q-matrix Q kj , k = 1, . . . , s, given in Theorem 1.3, such that the component matrices are E j = 0, 1, . . . , m  k , we have  s m k −1 

P (t) =



ki , ti e−λk t E

(2.5)

k=1 i=0

k is positive for k  1. and the real part of λ k is positive Proof Since P (t) → 0 as t → ∞ by transience, the real part of λ for k  1.  Proof of Theorem 1.1 Let 

∞

djj =

(pjj (t) − πj )dt.

0

Since for the finite Markov chain, pjj (t) − πj converges to zero exponentially and uniformly in j, pjj (t) − πj is absolutely integrable on [0, ∞), and djj is  finite. It follows from the fact T = N j=0 djj (cf. [1, Chap. 3]) that T =

N   j=0

=

=

(pjj (t) − πj )dt

0

N   j=0

∞

∞

0

dt

s m k −1  k=1 i=0

N  s m k −1  j=0 k=1 i=0

= = =

(−1)i i! (Eki )jj λi+1 k

s m k −1  (−1)i i! k=1 i=0 s  k=1 s  k=1

λi+1 k

tr (Ek0 ) λk mk , λk

ti e−λk t (Eki )jj

tr (Eki )

Eigentime identity for asymmetric finite Markov chains

629



where (Eki )jj is the (j, j)-entry of the matrix Eki .

To prove Theorem 1.3, we will utilize the Green matrix as a bridge  = ( connecting the eigenvalues and the hitting times. Let G gij ) be the  Green matrix of P (t), where  ∞ pij (t)dt. gij = 0

Proof of Theorem 1.3 a) It follows from Proposition 2.3 that = G



∞

P (t)dt =



∞

dt

0

0

 s m k −1 



ki , ti e−λk t E

k=1 i=0

so by the uniform convergence of P (t) → 0 as t → ∞, we have  = tr (G)



∞

dt 0



∞

dt

= 0

=

 s m k −1  k=1 i=0 s  kt −λ

e

 ki ) ti e−λk t tr (E

m k

k=1

s   λk . m k

(2.6)

k=1

It was proved in [11] that gjj =

1 qj Pj [ τj+

= ∞]

and for i = j,

gij = gjj Pi [  τj+ < ∞].

Combining (2.6) and (2.7) gives (1.5). b) Since Pi [ τj+ < ∞] < 1, it follows from (2.7) that for any i ∈ E, 

gij 

j∈E



 gjj = tr (G).

j∈E

As for ζ, we have 

 gij =

∞

0

j∈E



pij (t)

j∈E ∞

= 0



 

t = j, ζ > t] Pi [X

j∈E ∞

= 0

= Ei ζ.

Pi [ζ > t]dt

(2.7)

630

Hao CUI, Yong-Hua MAO

Thus, for any i, Ei ζ =



 gij  tr (G).

(2.8)

j∈E

On the other hand,  ∞   gjj =  Pj [Xt = j, ζ > t]dt  0

∞

Pj [ζ > t]dt = Ej ζ,

(2.9)

0

so that   tr (G)

N 

Ej ζ  N sup Ej ζ. 1jN

j=1

Combining (2.6), (2.8) and (2.9) gives (1.6).



3 Discrete-time Markov chain The proofs of Theorems 1.4 and 1.5 are similar to the proofs in the last section with some significant modifications. First, we apply the spectral theory to the (sub-)stochastic matrices. Proposition 3.1 (1) For the stochastic matrix given in Theorem 1.4 such that the component matrices are Ekj , k = 0, 1, . . . , s, j = 0, 1, . . . , mk , we have P 0 = I, and for n  1, n

P = Π+

−1)∧n s (mk  k=1

j=0

n! λn−j Ekj , (n − j)! k

(3.1)

where Π = (Πij ) is such that Πij = πj and the real part of λk is less than 1 for k  1. (2) For the sub-stochastic matrix P given in Theorem 1.4 such that the kj , k = 1, . . . , s, j = 0, 1, . . . , m component matrices are E  k , we have P 0 = I, and for n  1,  k −1)∧n s (m   n! n−j E kj , λ P n = (3.2) k (n − j)! j=0 k=1

k is less than 1 for k  1. where the real part of λ Proof (a) Since P is stochastic and irreducible, we have λ0 = 1 with the multiplicity one, it follows from (2.1) that P n = E00 +

−1)∧n s (mk  k=1

j=0

n! λn−j Ekj . (n − j)! k

If λk = rk (cos θk + i sin θk ),

rk  0, θk ∈ [0, 2π),

(3.3)

Eigentime identity for asymmetric finite Markov chains

then

631

λn−j = rkn−j (cos(n − j)θk + i sin(n − j)θk ), k

so that

n! λn−j = 0 if and only if rk < 1. (n − j)! k Since P is ergodic, limn→∞ P n = Π and the real part of λk is less than 1 for k  1. Letting n → ∞ in (3.3), we see that E00 = Π. (b) The proof of (3.2) is similar by noting now that limn→∞ P n = 0.  ∞ n Proof of Theorem 1.4 Let D = n=0 (P − Π). Then it is known that T = tr (D) (cf. [1, Chap. 3]). It follows from (3.1) and Lemma 2.1 that lim

n→∞

T = tr (D) = tr (I − Π) +

−1)∧n ∞  s (mk  n=1 k=1

=N+ =

s  k=1

=

s 

k=1

∞  s 

j=0

n! λn−j tr (Ekj ) (n − j)! k

λnk mk

n=1 k=1 s 

mk +

k=1

λk mk 1 − λk

mk . 1 − λk



Proof of Theorem 1.5 We use the Green matrix for discrete-time Markov  be the Green matrix for P , chain. Let G = G

∞ 

Pn .

n=0

Then it is well known for the transient Markov chain that gjj = Pj [ τj+ = ∞]−1 ,

and for i = j, gij = gjj Pi [ τj+ < ∞].

Therefore, the proof can follow from that of Theorem 1.3 by obvious modifications.  4 Examples In this section, we will give some examples to illustrate the above results. For the calculus of the hitting time Ei τj and Pi [ τi+ = ∞], refer to [8, Chap. 9]. Example 4.1 Let the Q-matrix be ⎛ ⎞ −1/2 1/2 0 −1 1 ⎠ . Q=⎝ 0 1 0 −1

632

Hao CUI, Yong-Hua MAO

It is easy to see that the eigenvalues of −Q are 0, 

λ−1 n =

n1

5 4

±

√ 7 4 i.

Then we have

5 . 4

On the other hand, let mij = Ei τj . Then we can calculate (mij ) by [8, Theorem 9.3.3] as ⎛ ⎞ 0 2 3 ⎝ 2 0 1 ⎠, 1 3 0 and the stationary distribution is (1/2, 1/4, 1/4). Thus, T =



πi πj Ei τj =

ij

If we consider the symmetrization ⎛ −1/2 Q = ⎝ 1/2 1/2

5 . 4

Q of Q as follows: ⎞ 1/4 1/4 −1 1/2 ⎠ , 1/2 −1

then −Q have eigenvalues 0, 1, 3/2, and T =



λ−1 n =

n1

5 . 3

Example 4.2 Let Q be qii = −1, qi,i+1 = 1 on the circle state space ZN . All eigenvalues of −Q are θk = 2kπ/N, k = 0, 1, . . . , N − 1,

1 − (cos θk + i sin θk ), which gives



λ−1 n =

n1

N −1 . 2

The stationary distribution is πi = 1/N, 0  i  N − 1. Since the chain goes from i to j step by step, we have E0 τj = j, 0  j  N − 1, so that by (1.2), T =



πj E0 τj =

j

N −1 . 2

Now, consider the symmetrization Q of Q, q ii+1 = 1/2, q ii−1 = 1/2, q ij = 0 for other i = j. It is known that all eigenvalues of −Q are λk = 1 − cos

2kπ , N

k = 0, 1, . . . , N − 1.

Eigentime identity for asymmetric finite Markov chains

633

N −1 −1 It seems difficult to compute k=1 λk directly. However, we can compute that E0 τ j = j(N − j) for 0  j  N − 1, so that by (1.2), N −1 

T =

N2 − 1 . 6

πj E0 τ j =

j=0

Example 4.3 Let Q be qii = −ai , qi,i+1 = ai , 0  i  N − 1 on the circle state space ZN . All eigenvalues {λn : n = 0, 1, . . . , N − 1} (with λ0 = 0) of −Q satisfy N 

(λ − an ) =

n=0

N 

(−an ).

n=0

j We can compute out that E0 τj = k=0 a−1 k and the stationary distribution as a−1 j πj = N −1 −1 , 0  j  N − 1. k=0 ak Thus, we have

N −1  n=1

1 λ−1 n = N −1



−1 k=0 ak j>k

(aj ak )−1 .

Example 4.4 Let the transition probability matrix be ⎛ ⎞ 0 1/2 0 P = ⎝ 0 0 1 ⎠. 1 0 0 All the eigenvalues of P are  3 1 (cos θk + i sin θk ), λk = 2

θk =

2kπ , k = 0, 1, 2. 3

Also, Pi [τi+ = ∞] = 1/2. Then 2  k=0

 1 1 = 6. = + 1 − λk P [τ = ∞] i=0 i i 2

The following example appeared in [7], a discrete analogue of Example 4.1. Example 4.5 Consider the walk on a cycle of length n that steps in the counterclockwise direction with probability one. The eigenvalues are λk = e2πki/n (0  k  n − 1). Then n−1  j=0

E0 τj =

n−1  k=0

λk =

n−1 . 2

634

Acknowledgements

Hao CUI, Yong-Hua MAO

This work was supported in part by the Program for New

Century Excellent Talents in University, the 973 Project (Grant No. 2006CB805901) and the National Natural Science Foundation of China (Grant No. 10721091). The authors would like to thank Prof. Mu-Fa Chen for valuable suggestions.

References 1. Aldous D, Fill J. Reversible Markov Chains and Random Walks on Graphs. 2003. URL www.berkeley.edu/users/aldous/book.html 2. Aldous D, Thorisson H. Shift coupling. Stoch Proc Appl, 1993, 44: 1–14 3. Anderson W. Continuous-time Markov Chains. New York: Springer-Verlag, 1991 4. Chen G N. Matrix Theory and Applications. 2nd Ed. Beijing: Science Press, 2007 (in Chinese) 5. Chen M -F. From Markov Chains to Non-equilibrium Particle Systems. 2nd Ed. Singapore: World Scientific, 2004 6. Chen M -F. Eigenvalues, Inequalities and Ergodic Theory. New York: Springer, 2004 7. Diaconis P, Saloff-Coste L. Nash inequalities for finite Markov chains. J Theor Prob, 1996, 9: 459–510 8. Hou Z T, Guo Q F. Homogeneous Denumerable Markov Processes. Berlin: Springer, 1988 9. Kemeny J G, Snell J L, Knapp A W. Denumerable Markov Chains. New York: Springer-Verlag, 1976 10. Mao Y H. Eigentime identity for ergodic Markov chains. J Appl Probab, 2004, 41: 1071–1080 11. Mao Y H. Eigentime identity for transient Markov chains. J Math Anal Appl, 2006, 315: 415–424 12. Meyer C. Matrix Analysis and Applied Linear Algebra. Philadelphia: SIAM, 2000 13. Roberts G, Rosenthal J. Shift coupling and convergence rates of ergodic averages. Communications in Statistics-Stochastic Models, 1997, 13: 147–165