Approximating stationary measures of structured continuous-time Markov models using matrix diagrams1 Gianfranco Ciardo† Andrew S. Miner‡ Min Wan† Andy Jinqing Yu† † Department of Computer Science and Engineering, University of California at Riverside ‡ Department of Computer Science, Iowa State University {ciardo, mwan, jqyu}@cs.ucr.edu
[email protected] 1 Introduction We consider the stationary solution of large ergodic continuous-time Markov chains (CTMCs) with a finite state space S, i.e., thePcomputation of π as solution of π · Q = 0 subject to i∈S π[i] = 1, where Q coincides with transition rate Pmatrix R except in its diagonal elements, Q[i, i] = − j∈S R[i, j ]. In practice, storing R and computing π may overwhelm even the largest computers. Approximation techniques have been proposed, where portions of the model are solved in isolation and the results somehow combined. With the overall model is decomposed into submodels, the approach of [7] uses a multi-valued decision diagram (MDD) [8] to encode S and aggregates the states of the CTMC based on the MDD representation, to compute an approximation of the stationary probabilities. As these aggregated states are relatively few, storage and computational costs are enormously reduced. This method requires a decomposition of the model in Kronecker form, i.e., R must be expressed as the sum (over all events) of the Kronecker product (over all submodels) of local matrices [1, 2, 5]. While in principle this Kronecker form always exists, in practice it might be achieved only by splitting model events, resulting in too many events, or by merging submodels, resulting in submodels with too large state spaces. To overcome this restriction, we propose to encode R with matrix diagrams (MxDs) [4, 6], which can store arbitrary transition rate matrices while still retaining the memory and time efficiency of a Kronecker encoding. In particular, the MxD-based approach results in the same approximation when the model is Kronecker-consistent, but is applicable to arbitrary models. Before presenting our contribution, we review some background material, see Fig. 1 for a summary of symbols used in the paper. Multi-valued decision diagrams. A multi-valued decision diagram (MDD) [8] is a directed acyclic graph 1 Work supported in part by the National Science Foundation under Grants CNS-0546041, CNS-0501747 and ATM-0428880.
Symbol S/Sk i, j/ik , jk Q R π Lk /Mk root/Root 0, 1/Ω p, q/M p[ik ] M [ik , jk ] (p, α) (e, α, α0 ) e.d/e.v A(p) B(p) hp, ik i Pr{X } Pr{p} Pr{p, γ} Pr{γ|p}
Definition or Meaning state space/local state space of submodel k global state/local state for submodel k infinitesimal generator matrix transition rate matrix stationary probability vector set of MDD/MxD nodes at level k MDD/MxD root MDD/MxD terminal node MDD/MxD non-terminal node node pointed by node p’s ik th edge edge in M with index [ik , jk ] MDD path started from node p MxD path started from edge e edge or path e’s destination node/value {α|(root, α).d = p} {β|(p, β).d = 1} {i|i P ∈ A(p) × {ik } × B(p[ik ]} i∈X π[i] Pr{A(p) × B(p)} Pr{A(p) × {γ} × B((p, γ).d)} Pr{p, γ}/ Pr{p}
Figure 1: Symbols used in the paper.
whose nodes are assigned to a level. Level 0 consists of two terminal nodes, L0 = {0, 1}, while level k, for L ≥ k ≥ 1, consists of non-terminal nodes p, each with nk edges pointing to nodes at lower levels. Given a decomposition of a model into L submodels, and identifying the set Sk of local states for submodel k with the integer interval {0, ..., nk − 1}, we can canonically encode the (global) state space S in an MDD with L non-terminal levels, where a path from root to terminal node 1 corresponds to a state in S. i.e. B(root) = S. Matrix diagrams. An (edge–valued) matrix diagram (MxD) [4, 6] is directed acyclic graphs whose nodes are assigned to a level. There is a single terminal node at level 0, M0 = {Ω}, carrying no value, while each nonterminal node at level k, for L ≥ k ≥ 1, is an nk × nk matrix of edges, each with an associated real value and p. 1
pointing to a lower-level node. We define the value of a path from edge e, following sequences α and α0 of local states as the product of all edge values along the path. Similarly to MDD encoding S, we can canonically encode R in an MxD with L non-terminal levels, where a path from Root to Ω corresponds to a transition rate in R. i.e., ∀i, j ∈ S, R[i, j ] = (Root, i, j).v.
Solve() 1 while not converged do 2 Set all B matrices to 0; 3 ComputeBelow (Root.d, root, root); 4 for k ← L to 1 do 5 ComputeAMatrices(k); 6 SolveLevel (k);
Figure 2: MxD-based approximation algorithm. and, further,
Our new approximation algorithm AM 0 [p0 , q 0 ] = MDD–based exact aggregation. If we partition S into n disjoint sets of states {C1 , . . . , Cn } and define an aggregated CTMC with state space Sagg = {1, . . . , n} and transition rate matrix Ragg given by Ragg [i, j] =
X
π[i|Ci ] ·
i∈Ci
X
R[i, j ],
(1)
j∈Cj
where π[i|Ci ] is the stationary conditional probability of being in state i given that the CTMC is in class Ci , it is well known that this aggregated CTMC is alsoPergodic if the original one is ergodic, and that π agg [i] = i∈Ci π[i]. We assume S is encoded as an L-level MDD and R as an L-level MxD. For each p ∈ Lk and ik ∈ Sk , we can define hp, ik i as in Fig. 1 the aggregated state and define L aggregated CTMCs as in [7]. Simplifying assumption. The assumption in [7]: Pr{ik | α} ≈ Pr{ik | p}, ∀α ∈ A(p), is also adopted in our algorithm to simplify the computation of Eq. (1) and thus make the solution of large CTMC possible (see the notation for path and node probability in Fig. 1). A immediate implication from [7] under this assumption is, for i = (α, ik , β) ∈ hp, ik i, π[i|hp, ik i] ≈ Pr{α | p} · Pr{β | p[ik ]}.
where AM [p, q] =
X
Pr{α | p} · (Root, α, α0 ).v,
∀α,α0 :(root,α,α0 ).d=M
BM [p, q] =
X ∀β∈B(p),β 0 ∈B(q)
Pr{β | p} · (M, β, β 0 ).v,
AM [p, q] · Pr{p, ik |p0 } · M [ik , jk ].v,
∀M,p,q
BM [p, q] =
X
(3) Pr{ik | p} · M [ik , jk ].v · BM 0 [ p , q ] . (4) 0
0
∀ik ,jk
Thus, the AM matrices, for M ∈ Mk , are built based on π agg,l , for levels L ≥ l ≥ k, and terminated at level L with ARoot.d [root, root] = Root.v, while the BM matrices, for M ∈ Mk , are built based on π agg,l for levels k ≥ l ≥ 1 and terminated at level 0 with BΩ [1, 1] = 1. As such, the level–k aggregated CTMC depends on the solutions of the aggregated CTMCs at all other levels. To break this cyclic dependency, we use a fixpoint computation, as done in [7]. Algorithm. Fig. 2 shows the pseudocode of our approximation. Procedure Solve performs the overall fixpoint computation. Procedure ComputeAMatrices(k) builds the matrices AM by applying Eq. (3). Procedure ComputeBelow is used to recursively compute an element of BM , based on Eq. (4). Procedure SolveLevel (k) is used to build and solve the level–k aggregated CTMC using the Gauss-Seidel method [9]. A stopping criterion must be used to determine when the overall computation has reached a fixpoint (cf. line 1). As in [7], we stop the iterations when, at every level k, only one iteration is required to update vector π agg,k in the numerical solution.
(2)
Approximate aggregation. Substituting Eq. (2) into Eq. (1), and after some manipulations, we can break the computation of the rhs of Eq. (1) into two separate recursive computations as shown below (assume that M , M 0 , p, p0 , q, q 0 , ik , and jk satisfy M 0 = M [ik , jk ].d, p0 = p[ik ], and q 0 = q[jk ] in all following equations) X Ragg,k [hp, ik i, hq, jk i] ≈ AM [p, q]·M [ik , jk ].v·BM 0 [p0 , q 0 ] M ∈Mk
X
Computing measures. Once we have obtained an estimate for π agg , we can compute one or more measures of interest defined via a reward function r : S → R, using a bottom-up recursive computation.
2 Experimental results We compare four different solution approaches on a set of benchmarks described by stochastic Petri nets. The Exact solution stores R explicitly and computes π using Gauss–Seidel for the numerical solution, with a relative stopping criterion set to 10−4 . Discrete event Simulation computes the given measure to within an interval of relative width 2% with 99% confidence. The Kronecker solution uses the approximate technique in [7] and is applicable only when the decomposition of the p. 2
Model fms fms kanban kanban cyclesteal cyclesteal polling polling slottedring slottedring robin robin
N 3 20 5 15 4 5 5 20 5 60 10 60
|S| 2.06 × 104 5.91 × 1011 1.27 × 106 6.62 × 109 5.82 × 105 2.05 × 107 7.12 × 104 1.08 × 1028 5.38 × 104 7.29 × 1062 2.30 × 104 1.55 × 1020
Time 0.50 nr 33.94 nr 26.74 nr 0.62 nr 0.12 nr 0.08 nr
Exact Mem 1.17 nr 106.29 nr 39.37 nr 3.62 nr 2.62 nr 1.24 nr
Measure 0.756866 nr 0.459167 nr 0.003087 nr 0.888889 nr 0.928206 nr 0.102385 nr
Simulation Time Measure 5.06 0.756427 5.73 0.746341 4.82 0.459662 41.07 0.269436 115.02 0.003094 1576.55 0.003045 3.74 0.889246 5.09 0.972512 0.73 0.927799 71.45 0.934519 7.69 0.102102 137.76 0.017102
Kronecker Time Mem nkc nkc nkc nkc nkc nkc nkc nkc nkc nkc nkc nkc 0.01 0.01 0.15 0.10 0.01 0.01 16.12 2.11 0.09 0.25 12.09 7.21
Time 0.02 5.31 0.07 37.29 0.05 0.17 0.01 0.08 0.01 55.50 0.07 1.77
MxD Mem Measure 0.08 0.755264 8.20 0.750000 0.92 0.437037 46.11 0.249982 0.14 0.003145 0.91 0.003125 0.02 0.888892 0.22 0.972222 0.02 0.924915 1.63 0.930323 0.11 0.102360 0.70 0.017063
Figure 3: Results (Time in sec, Mem in MB): nr means Time > 3600sec or Mem > 1GB, nkc means not Kronecker-consistent. model is Kronecker-consistent. When solving the level– k aggregate CTMC, we use Gauss–Seidel with a relative precision of 10−4 , and the global fixpoint iterations stop when every aggregate CTMC solution converges within one Gauss–Seidel iteration. Our MxD-based approximation method, which is always applicable, uses the same stopping criteria as for the Kronecker-based solution. All algorithms are implemented in SmArT [3] and all experiments are run on a Pentium 4 3.0GHz PC with 2.0GB memory, with CentoOS Linux 2.6.9. Each benchmark is parameterized by an integer N . The performance measure is the probability of a given place being empty.
References [1] V. Amoia, G. De Micheli, and M. Santomauro. Computer-oriented formulation of transition-rate matrices via Kronecker algebra. IEEE Trans. Rel., 30:123–132, June 1981.
Fig. 3 shows the numerical results as well as the time and memory requirements to solve the models and the size of the exact state space. The exact solution is feasible only when the number of states is in the millions. Even then, our MxD-based technique requires much less time and memory and its results are quite accurate, most of them well within 2%. As N increases, we can only compare with simulation, and the conclusion remains that our approximation is quite accurate, if we assume that the simulation results are reliable. Finally, comparing with our previous Kronecker-based approximation when possible, we observe that the runtime and memory are usually even better in our more general setting, due to the inherent efficiency of decision diagram techniques (the Kronecker-based solution has no “Measure” column since, when it can be run, it produces the same numerical results as our new method that generalizes it).
[4] G. Ciardo and A. S. Miner. A data structure for the efficient Kronecker solution of GSPNs. Proc. 8th Int. Workshop on Petri Nets and Performance Models (PNPM’99), pages 22–31, 1999. IEEE Comp. Soc. Press.
[2] P. Buchholz, G. Ciardo, S. Donatelli, and P. Kemper. Complexity of memory-efficient Kronecker operations with applications to the solution of Markov models. INFORMS J. Comp., 12(3):203–222, 2000. [3] G. Ciardo, R. L. Jones, A. S. Miner, and R. Siminiceanu. Logical and stochastic modeling with SMART. Perf. Eval., 63:578–608, 2006.
[5] P. Fernandes, B. Plateau, and W. J. Stewart. Efficient descriptor-vector multiplication in stochastic automata networks. J. ACM, 45(3):381–414, 1998. [6] A. S. Miner. Efficient solution of GSPNs using canonical matrix diagrams. Proc. 9th Int. Workshop on Petri Nets and Performance Models (PNPM’01), pages 101–110, 2001. IEEE Comp. Soc. Press. [7] A. S. Miner, G. Ciardo, and S. Donatelli. Using the exact state space of a Markov model to compute approximate stationary measures. Proc. ACM SIGMETRICS, pages 207–216, 2000. ACM Press. [8] A. Srinivasan, T. Kam, S. Malik, and R. K. Brayton. Algorithms for discrete function manipulation. In Int. Conference on CAD, pages 92–95. IEEE Comp. Soc. Press, 1990. [9] W. J. Stewart. Introduction to the Numerical Solution of Markov Chains. Princeton University Press, 1994.
p. 3
