Transient Solutions of Markov Processes by Krylov

INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE

Transient Solutions of Markov Processes by Krylov Subspaces Bernard PHILIPPE and Roger B. SIDJE

N˚ 1989 Aoˆut 1993

PROGRAMME 6

Calcul scientifique, modélisation et logiciels numériques

ISSN 0249-6399

apport de recherche

1993

Transient Solutions of Markov Processes by Krylov Subspaces  Bernard PHILIPPE and Roger B. SIDJE

Programme 6 | Calcul scienti que, modelisation et logiciel numerique Projet ALADIN Rapport de recherche n1989 | Ao^ut 1993 | 33 pages

Abstract: In this note we exploit the knowledge embodied in in nitesi-

mal generators of Markov processes to compute eciently and economically the transient solution of continuous time Markov processes. We consider the Krylov subspace approximation method which has been analysed by Y. Saad for solving linear dierential equations. We place special emphasis on error bounds and stepsize control. We discuss the computation of the exponential of the Hessenberg matrix involved in the approximation and an economic evaluation of the Pade method is presented. We illustrate the usefulness of the approach by providing some application examples.

Key-words: Markov chains, matrix exponential, Krylov subspaces, Arnoldi.

(Resume : tsvp)  This work was supported by an NSF-INRIA agreement.

Unit´e de recherche INRIA Rennes IRISA, Campus universitaire de Beaulieu, 35042 RENNES Cedex (France) T´el´ephone : (33) 99 84 71 00 – T´el´ecopie : (33) 99 38 38 32

Approximation des solutions transitoires d'un processus de Markov dans des sous-espaces de Krylov

Resume : Nous nous interessons dans cet article au calcul du regime transitoire d'un processus de Markov. Un tel probleme necessite le calcul d'une exponentielle de matrice. En prenant en consideration certaines proprietes inherentes aux modeles markoviens, nous examinons l'approximation dans un espace de Krylov proposee par Y. Saad. Cela nous permet de preciser certaines caracteristiques theoriques et de de nir un algorithme able et ef cace. Nous presentons des resultats numeriques attestant de la validite de la methode. Mots-cle : Cha^nes de Markov, exponentielle d'une matrice, espaces de Krylov, methode d'Arnoldi.

1

Transient Solutions of Markov Processes by Krylov Subspaces

1 Introduction Problem De nition. Continuous-time Markov Chains (CTMC) are widely used in the

analysis of the behavior of many physical systems. Given a system whose evolution can be modeled by a CTMC, X = fX (t); t  0g having a nite discrete state space = f1; 2; :::; N g, the transient analysis involves the computation of the state probability vector (t) = (1(t); 2(t); :::; N (t)) where i (t) = ProbfX (t) = ig is the probability that the Markov process X is in state i at time t. Interesting measures (availability, reliability, performability) of the behavior of the system being modeled can then be evaluated as weighted sums of state probabilities (see, e.g, [18] and references therein). Let Q = [qij ] be the in nitesimal generator of X . Q is also called the transition rate X matrix and qij  0; i 6= j , is the rate of transition from state i to state j and qii = ? qij . i6=j

The evolution of the state probability vector  (t) can be mathematically represented by the Chapman-Kolmogorov system of dierential equations. For convenience of notation we set A = QT ; v =  T (0); w(t) =  T (t) and hence, the dierential system under concern may be written as 8 dw(t) < = Aw(t); t 2 [0; T ]

dt : w(0) = v;

initial state probability vector

and has the analytic solution

w(t) = etA v:

(1) (2)

Largeness, stiness and accuracy are the three major diculties related to the practical computation of (2) [6, 11, 18]. Intrinsic Properties. There are a number of inherent characteristics of the model (see, e.g., [4, 21] for a general overview). We give below a list of those which are relevant for our presentation. P1 T A = 0 where = (1; 1; :::; 1)T . 1

1

P2 0  wi(t)  1 and kw(t)k1 = T w(t) = 1: 1

P3 0 is a simple eigenvalue of A and for any non-null eigenvalue  of A, 0 such that for any  , 0 <    ?; pm; (A)  0; where

pm; (z) denotes the Hermite interpolation polynomial of ez at (Hm ) [ f0g. Proof. From Lemma 2 in [23], pm;0(?x) is completely monotonic in (?1; +1], and therefore, in any smaller subinterval (?a~; +~a) neighborhood of 0. The uniform convergence of pm; to pm;0 insures that there exists ~ > 0 such that for any  < ~, pm; (?x) is completely monotonic in (?a~; +~a). Let us take  ?, 0 <  ?  ~, suciently small to make sure that it will always be possible to take s 2 (?a~; +~a) such that P  A + sI is nonnegative for any  , 0 <    ? . Now de ne f (x)  pm; (?x) in the interval (?a~; +~a). Applying Theorem 5.1 yields f (sI ? P ) = pm; (A)  0. 2

19

Transient Solutions of Markov Processes by Krylov Subspaces

This theorem shows that it is always possible to select a stepsize small enough such that the approximation vector of the integration process (4) is nonnegative. In Example 5.1 for illustration, if instead of computing we (1)  eA v directly, we were to compute for the start we( 161 )  e 161 A v, the problem would not have been encountered. In practice, this drawback is rarely observed. If we end up with a nonpositive approximation vector, the components under concern are, in magnitude, less than the error tolerance for otherwise, the approximate solution would have been rejected. Therefore a simple strategy to overcome any diculty is to set the negative components to zero and aftewards, perform a normalization. Another strategy is of course to reduce the stepsize. We point out that, theoretically, even if not mentioned in literature, this inconsistency may well appear with general purpose ODE solvers such as RK, BDF and related extensions commonly used for numerical transient analysis.

6 Numerical Issues

6.1 Accuracy and Roundo Errors Analysis We will now derive an upper bound for the global truncation error and an upper bound for roundo errors when we use an integration scheme of the form (4). Given a vector wi and a stepsize i

 wi+1 denotes the exact local solution, i.e. the value of eiAwi.  wei+1 denotes the exact solution of the approximation method, i.e. the value of the

proposed Krylov method in exact arithmetic.  wi+1 is the computed value of wei+1. We start with the following preliminary result.

Proposition 6.1 For any matrix A satisfying P1 , P2 , P3 and P4 , 8  0; 8w 2 N , R

if w  0 then keA wk1 = kwk1. More generally keA wk1  kwk1 and therefore keA k1 = 1:

Proof. The rst part is a well known result. For any vector w 2 w = w+ ? w?

where

w+ ; w?

N

we can decompose  0 and for all 1  i  N ; i i = 0. Thus,

w+ w?

R

keAwk1  keAw+ k1 + keAw? k1 = kw+ k1 + kw? k1 = kwk1: 2

20

B. PHILIPPE AND R. B. SIDJE

It is well known in Ordinary Dierential Equations theory that it is hard to estimate the global error from local errors. The following result overcomes this diculty for the CTMC case.

Theorem 6.1 For 1  j  i, let "ej be the local truncation error vectors i.e. "ej = ej?1 Awej?1 ? wej and let "j = ej?1 Awj?1 ?wj then, at step i

kwi ? weik1 =  kwi ? wik1 = 



i

X (ti?tj )A e

"j

e

1 j =1 i X k"ej k1 j =1

i

X e(ti?tj )A"

j

1 j =1 i X k"j k1

j =1

(40) (41) (42) (43)

Proof. We will ignore the norms in (40) and prove the equality by induction. For i = 0 it

is clear that w1 = et1 A w0 = we1 + "e1 . Let us suppose that the equality holds up to a given i, i.e. Xi wi ? wei = e(ti?tj )A "ej : j =1

Then,

wi+1 = eiA wi

X = ei A wei + e((i +ti )?tj )A "ej i

j =1

= wei+1 + "ei+1 + = wei+1 +

i+1 X j =1

Xi j =1

e(ti+1 ?tj )A"ej

e(ti+1?tj )A "ej :

The inequality (41) follows immediately from Proposition 6.1. The proof of the remaining part of the theorem is similar. 2 As a consequence of this theorem, we see that there is no ampli cation factor neither on the local truncation errors nor on roundo errors. Thus the integration process is con dently stable. We point out that this theorem is valid for any integration scheme used for the transient solution of a CTMC.

Transient Solutions of Markov Processes by Krylov Subspaces

21

6.2 A Posteriori Error Estimates The results in the preceding sections supply a theoretical fundation but can not be plainly incorporated in production codes in their present forms. Recall that we are systematically using the corrected scheme, we will give some practical estimates for the errors. We rst consider roundo errors for which we have

k k1kwei ? wik1  j T (wei ?wi )j 1

and thus using (37)

1

kwei ?wi k1  j1 ?n wij : 1

T

(44)

Hence if the quantity on the right hand side of (44) is far from machine epsilon then, the computed approximation wi is too contaminated and the process should be stopped. This is a sucient condition to indicate a high level of roundo error. The indicator is de cient when the vector wi ? wei is orthogonal to . This may happen but we expect it to be rare. For local truncation errors the starting point is relation (16) in which we now insert the stepsize  to obtain 1

eA v = VmeHm e1 + hm+1;m

1 X

j =1

eTmj (Hm)e1 (A)j?1 vm+1 :

As we have already mentioned, the j are positive increasing functions with j +1 (x)  1  (x) in [0; 1). For a small stepsize the next term in this expansion series is an approj j priate estimate of the error. Thus we can use the local error estimate

err loc = jhm+1;m eTm2 (Hm )e1j kAvm+1 k2:

(45)

We remind the reader that a simple way to cheaply compute hm+1;m eTm j (Hm )e1 has been shown in Proposition 3.2. A few other estimates can be found in [19]. We have prefered this one because in most of the cases it has the more regular behavior.

6.3 Stepsize Control In this section we give some practical guidelines on how to handle the stepsize  and m, the size of the Krylov basis. In fact at each step, for a prescribed tolerance there exists optimal values min  opt  max and mmin  mopt  mmax which minimize rounding errors and execution cost with respect to the whole integration interval. Unfortunately those values can not be obtained straightforwardly and we must be satis ed with some heuristics as is usual in ODE solvers. The following presentation is inspired by [10].

22

B. PHILIPPE AND R. B. SIDJE

The control depends on the choice between the two measures

i =

(

k"ik; error per step (EPS) k"ik=i?1; error per unit step (EPUS):

Here k"i k has the same meaning as in Theorem 6.1 and is to be approximated by err loc given by (45) or any other suitable estimate. If tol denotes the prescribed tolerance to be achieved, the rst stepsize is chosen to satisfy the theoretical bound (25) and within the integration process, the stepsize is selected by means of the formula

 1=r i = tol i?1 i

(46)

rounded to 2 signi cants digits to prevent numerical noise. The value of r is m or m + 1 if the control is based on EPUS or EPS respectively. The scalar is a safety factor which is intended to reduce the risk of rejection of the step (  1). A step is rejected if i+1 >  tol. Classical values of and  are 0:9 and 1:2 respectively. The control based on EPUS is more attractive because it yields the same hhaccumulatedii global error for a xed integration time regardless of the number of steps used. In this instance we always have

X j

k"j k  T  tol

and according to Theorem 6.1 we infer that

kwi ?wik  T  tol:

(47)

When a step is rejected the stepsize may be reduced by using formula (46) once more but there is also the opportunity of adjusting the dimension of the Krylov subspace by the way of the relation  tol= )  i : (48) mi = mi?1 + log( log i?1 Let us note that (48) allows us to increase or to decrease the size of the basis.

6.4 Description of the Algorithm Given t; A; v; `; m; k, the following algorithm computes an approximation of the solution vector w(t) = etA v by incorporating in an Arnoldi basis ` schur vectors and by using the k-corrected scheme. If ` = 0 and k = 0 (resp. ` = 0 and k 6= 0) then the basic (resp. k-corrected) scheme is recovered.

0. set w := v; tc := 0; H := zeros[` + m + k; ` + m + k] 1. obtain ` Schur vectors V` := [v1; :::; v`] and compute H` := V`T AV`

Transient Solutions of Markov Processes by Krylov Subspaces

23

2. while tc < t

vc := w for i := 1 : ` i := viT vc ; vc := vc ? i vi

end

:= kvc k2; v`+1 := vc = for j := ` + 1 : ` + m p := Avj for i := 1 : j hij := viT p ; p := p ? hij vi

end

hj+1;j := kpk2 if hj+1;j  tolkAk breakdown fan invariant subspace is foundg vj+1 := p=hj+1;j

end

set 1's for the k-corrected scheme choose stepsize  from the local error estimate compute F`+m+k := exp(H`+m+k ) w := V`+m F`+m ( 1; :::; `; ; 0; :::; 0)T + Pkj=1 f`+m+j;`+1 (A)j?1 v`+m+1 tc := tc + 

end

The transition rate matrix is accessed only through matrix-vector products. Hence the algorithm is independent of the matrix data structure and this allows the analyst to handle compact sparse storage and to design high ecient matrix-vector routines exploiting structure, sparsity and any knowledge embodied in the matrix.

7 Numerical Examples We illustrate in this section the practical value of the method using three problems. The initial probability vector is (1; 0; :::; 0)T and the prescribed tolerance is tol = 10?10 . For each problem the distribution of eigenvalues and the sparsity pattern of its in nitesimal generator are plotted (Figures 3, 4 and 5). We denote by nz the number of nonzero elements in the matrix. Problem 1. The system is an open queueing network consisting of two stations in tandem. Each station consists of a nite queue and a single server. In station 1, the server takes

24

B. PHILIPPE AND R. B. SIDJE

vacations when the number of customers in station 2 becomes greater than a threshold S . The arrival stream to the rst queue is Poisson with parameter . The services are exponential with parameter 1 , 01 (vacation) for the rst queue and 2 for the second. Such queueing systems arise in computer systems modelling, production and telecommunication systems. The transition matrix is of order 800x800 with nz = 3081. 01 = 0:9

N1 = 19

 = 0:8

N2 = 19 1 = 0:7

S=5

2 = 1

Figure 2: Illustration of Problem 1. Problem 2. A presentation of the system yielding this example can be found in [6]. Two matrices were generated for this model. The matrix (a) was chosen to be of order 1024x1024 with nz = 11264. The matrix (b) is of order 8192x8192 with nz = 114688. The spectrum and the sparsity pattern in Figure 4 corresponds to case (a). Problem 3. The Markov process here is an alternate PH-renewal process [13] for which the two phases follow a Coxian law. The resulting matrix is of order 1000x1000 with nz = 2998. As one can see on their spectra, Problem 1 and Problem 2 are relatively smooth while Problem 3 is very sti (maxi j