Perfect Simulation and Non-monotone Markovian Systems - HAL-Inria

Perfect Simulation and Non-monotone Markovian Systems Ana Busic, Bruno Gaujal, Jean-Marc Vincent

To cite this version: Ana Busic, Bruno Gaujal, Jean-Marc Vincent. Perfect Simulation and Non-monotone Markovian Systems. 3rd International Conference Valuetools’08, 2008, Athens, Greece. ICST, 2008.

HAL Id: hal-00953624 https://hal.inria.fr/hal-00953624 Submitted on 28 Feb 2014

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destin´ee au d´epˆot et `a la diffusion de documents scientifiques de niveau recherche, publi´es ou non, ´emanant des ´etablissements d’enseignement et de recherche fran¸cais ou ´etrangers, des laboratoires publics ou priv´es.

Perfect simulation and non-monotone Markovian systems

∗

Ana Buši´c

Bruno Gaujal

Jean-Marc Vincent

INRIA Grenoble - Rhône-Alpes 51, Av. J. Kuntzmann, 38330 Montbonnot, France

INRIA Grenoble - Rhône-Alpes 51, Av. J. Kuntzmann, 38330 Montbonnot, France

Université Joseph Fourier 51, Av. J. Kuntzmann, 38330 Montbonnot, France

[email protected]

[email protected]

[email protected]

ABSTRACT Perfect simulation, or coupling from the past, is an efficient technique for sampling the steady state of monotone discrete time Markov chains. Indeed, one only needs to consider two trajectories corresponding to minimal and maximal state in the system. We show here that even for non-monotone systems one only needs to compute two trajectories: an infimum and supremum envelope. Since the sequence of states obtained by taking infimum (resp. supremum) at each time step does not correspond to a feasible trajectory of the system, envelopes and not feasible trajectories. We show that the envelope approach is efficient for some classes of nonmonotone queuing networks, such as networks of queues with batch arrivals, queues with fork and join nodes and/or with negative customers.

Categories and Subject Descriptors I.6 [Computing Methodologies]: Simulation and modeling; G.3 [Probability and Statistics]: Markov processes

Keywords Markov chains, perfect simulation, queuing networks

1. INTRODUCTION Simulation approaches are alternative methods to estimate the stationary behavior of stochastic systems by providing samples distributed according to the stationary distribution, even when it is impossible to compute this distribution numerically. Propp and Wilson used a backward coupling [6] to derive a simulation algorithm (PSA: Perfect Simulation Algorithm) providing perfect sampling (i.e. which distribution is stationary) of the state of a discrete time Markov chain over a finite state space X . The main idea is to start many simulated trajectories from all x ∈ X at some time in the past until time t = 0. If the output is the same for all trajectories, then the chain has coupled and the common state has the stationary distribution of the Markov chain. Otherwise, the simulations are started further in the past. ∗ This work was partially supported by ANR-05-BLAN-000902 SMS and ANR-06-SETIN-002 Checkbound.

ValueTools 2008, October 21 - 23, 2008, Athens, GREECE.

Although the Perfect Simulation Algorithm provides exact samples for all recurrent finite Markov chains, its complexity can be high. It suffers from two drawbacks that jeopardize its applicability for very large chains. The first one is the fact that the coupling time can be very large. Some work focused on the computation of the coupling time for certain classes of Markov chains. It was shown in [3] that Markov chains modelling a class of networks of queues with finite capacities have a very small coupling time with respect to the size of the state space. The second factor in the complexity of PSA is the fact that one needs to run one simulation per state in X . When the state space is ordered and the Markov chain is monotone Propp and Wilson have shown that it is enough to run two simulations only: one starting from the largest state and one from the smallest state. Since then, several extensions were proposed when the state space is partially ordered. In that case one needs to run one simulation per extremal state. This is the case for stochastic event graphs, a sub-class of stochastic Petri nets [2], as well as for heaps of pieces [1]. However, when the Markov chain is not monotone, nothing has been proposed so far to avoid the construction of one trajectory per state in X . Meanwhile, non-monotone chains are very common. In the context of networks of queues, introducing fork and join nodes or negative customers or batch arrivals and services destroys the monotonicity of the underlying Markov chain. The goal of this paper is to provide a new perfect simulation algorithm (EPSA: Envelope Perfect Simulation Algorithm) that only uses two simulated trajectories for general nonmonotone Markov chains. This is done by computing from the past, a lower and an upper envelope of all trajectories of the Markov chain. Whenever the two envelopes couple, the coupling state is distributed according to the stationary distribution of the chain. This new simulation technique raises new questions. Do envelopes couple with probability one? Actually, using the construction of envelopes given in section 3, they do not couple in some cases. However, we show in §5.2 that it is always possible to change the constructive definition of the Markov chain (i.e. the function Φ) so that the corresponding envelopes will couple. However, the complexity of this construction is quadratic in the size of the state space. If envelopes couple then the EPSA provides perfect samples for the Markov chain but its complexity can make it impossible to use in practice. This question is discussed in details in the paper. We show that in some cases, the coupling

time of the envelopes is similar to the coupling time of the chain (in § 4.1 and §4.2) while in other cases, it becomes embarrassingly larger (in §4.3). When envelopes do not couple, it is still possible to exploit them because they provide bounds on the stationary distribution (see § 5) or because they can serve as an initial step in the perfect simulation: use envelopes as long as they contract the state space and continue with an exhaustive simulation from that point on (this is called splitting in the paper, see § 5).

M4

M3

M2

M1

E4

E1 E2′

E3 E2

Figure 1: Markov chains and perfect simulation. All this extends the limit of applicability of perfect simulation beyond the monotone case, as illustrated in Figure 1. The set M1 is the set of monotone Markov chains, where perfect simulation was known to be very efficient. The set M2 is the set of non-monotone Markov chains, where envelope perfect simulation can be used efficiently (the coupling time is similar to the coupling time of the chain). The set M3 is the set of all Markov chains where the envelopes do couple but take a much larger time to couple than the Markov chain, while the set M4 is the set where the envelopes do not couple so that splitting must be used to get samples (or a new set of events must be constructed to guarantee that envelopes will couple). Another contribution of this paper is to provide concrete examples in the context of queuing networks falling in each of these sets. The example E1 is any network of finite queues with monotone routing. Example E2 is a network as E1 with negative customers (the arrival of a negative customer in a queue destroys one regular customer). Example E2′ is a network as E1 with fork and join nodes (a fork splits a customer into two independent customers while a join replaces two customers by a single one). Example E3 is a network of queues with individual customers as well as batches (customers may arrive by batches and may also be served and routed alone). Finally, example E4 is a network of queues with only batches larger than two. All these experiments have been done with the perfect simulation software Ψ2 [10], freely available at http://psi.gforge.inria.fr.

2. PERFECT SIMULATION Let {Xn }n∈N be an irreducible and aperiodic discrete time Markov chain with a finite state space X and a transition matrix P = (pi,j ). Let π denote the steady state distribution of the chain: π = πP . The evolution of the Markov chain can always be described by a stochastic recurrence sequence Xn+1 = Φ (Xn , en+1 ), with {en }n∈Z an independent and identically distributed sequence of events en ∈ E, n ∈ N. The transition function Φ : X × E → X verifies the property that P (Φ(i, e) = j) = pi,j for every pair of states (i, j) ∈ X × X and a random event e ∈ E.

def

X . That is, Φn (x, e1→n ) = Φ (. . . Φ (Φ (x, e1 ) , e2 ) , . . . , en ). This notation can be extended to sets of states: for A ⊂ X , def Φn (A, e1→n ) = {Φn (x, e1→n ) , x ∈ A} . In the following, |A| denotes the cardinality of set A. Theorem 1 ([6]). There exists ℓ ∈ N such that ˛ ˛ lim ˛Φn (X , e−n+1→0 ) ˛ = ℓ almost surely. n→∞

The system couples if ℓ = 1. In that case, the value of Φn (X , e−n+1→0 ) is steady state distributed.

Algorithm 1: Perfect Simulation Algorithm (PSA) Data: A function Φ and i.i.d. events {e−n }n∈N Result: One state in X generated according to the stationary distribution of the system begin n := 1; foreach state x ∈ X do y(x) := x; repeat foreach state x ∈ X do z(x) := y(Φ(x, e−n )); foreach state x ∈ X do y(x) := z(x); n := n + 1; until |{y(x), x ∈ X }| = 1 ; return {y(x), x ∈ X }; end Theorem 1 has an algorithmic counterpart (PSA), given in Algorithm 1. The main drawback of PSA algorithm is the fact that one needs to simulate the MC starting with all states in X , which could be too large for Algorithm 1 to be used in practice. Several approaches have been used to overcome this problem. The main one is already present in [6]. When the state space X is partially ordered and when the function Φ(·, e) is monotone for all e, then the Markov chain is monotone and it is possible to generate a steady state by starting Algorithm 1 with maximal and minimal states only. This technique has been successfully used in [7] to construct PSA for network of queues. When Φ(·, e) is not monotone, one can still use monotone bounds, as in [4].

3. ENVELOPES The approach proposed here generalizes what has been done in [5] to simulate non-monotone Markov chains. Its main advantage is that it does not need any preliminary assumption on the structure of the Markov chain. Assume that the state space X is equipped with an lattice order relation ≤ and consider a new transition function Γ, called the envelope, defined on the Cartesian product of the state space : Γ : X × X × E → X × X, for all m, M in X and e in E, def

Γ1 (M, m, e) = Φ(M, m, e)

def

=

sup

Φ(x, e)

m≤x≤M n

n

Let Φ : X ×E → X denote the function whose output is the state of the chain after n iterations and starting in state x ∈

def

Γ2 (M, m, e) = Φ(M, m, e)

def

=

inf

m≤x≤M

Φ(x, e).

def

def

Let us call T = sup X (resp . B = inf X ) the top (resp. bottom) element of X . The process def

(Mn , mn ) = Γn (T, B, e1→n ) is a Markov chain over the state space X × X . However, the upper envelope alone is not a Markov chain (it depends on the lower enveloppe), neither is the lower one. Theorem 2. Assume that (Mn , mn ) hits the diagonal D (i.e. states of the form (x, x)) in finite time:  ff def τe = min n : Γn (T, B, e−n+1→0 ) ∈ D , then τe is a backward coupling time of the initial Markov chain Xn . The state defined by Γτe (T, B, e−τe +1→0 ) has the steady state distribution of Xn .

(P1 ) The assumption that (Mn , mn ) hits the diagonal in Theorem 2 may not hold. (P2 ) Even if Theorem 2 applies, the complexity of the algorithm may be prohibitive. The average complexity of EPSA is O(C(Φ, Φ) · Eτe ) where τe is the number of iterations of the main loop of EPSA (called the coupling time in the following) and C(Φ, Φ) is the complexity (number of elementary operations) to compute Φ and Φ. Meanwhile, the average complexity of the classical PSA is O(C(Φ) · |X | · Eτ ), where τ is the coupling time of PSA, |X | the size of the state space and C(Φ) the complexity of the computation of Φ. If C(Φ, Φ) is not too high w.r.t C(Φ) (this is usually the case in the following examples), then the comparison of the two methods is essentially a comparison between the coupling time of EPSA (Eτe ) and (|X | · Eτ ).

def

Proof. The proof simply uses the fact that if (M0 , m0 ) = Γ (T, B, e−n+1→0 ), then the set Φn (X , e−n+1→0 ) is included in the set {x : m0 ≤ x ≤ M0 }, consequently if the latter is reduced to one point, so is the set Φn (X , e−n+1→0 ) and the backward scheme has coupled. n

Now, Algorithm 1 can be adapted to the envelope simulation: start the simulation with only two states: T and B and iterate using the bi-dimensional function (Φ, Φ). Algorithm 2: Backward simulation algorithm using envelopes (EPSA) Data: - Φ , {e−n }n∈N - Γ the pre-computed envelope function Result: A state x∗ ∈ X generated according to the stationary distribution of the system begin n = 1; M := T ; m := B; repeat for i = n downto 1 do (M, m) := Γ(M, m, e−i+1 ); n := 2n; until M = m ; return M ; end

In the following we will focus on problems P1 and P2 in the context of networks of queues. We show that networks of queues with negative customers are not monotone but envelopes couple with probability one (problem P1 ) and the coupling of the envelope algorithm (P2 ) remains reasonable (see §4.1). The same holds when fork and join nodes are introduced (see § 4.2). In these cases, there is a clear gain to use EPSA over PSA (the complexity is divided by the size of the state space). However, if customers arrive by batches, things change dramatically, as shown in § 4.3. First, envelopes never couple in some cases. We provide a characterization of the set of networks of queues with batches that do couple with probability one. When coupling occurs, the complexity C(Φ, Φ) of computing the envelopes remains small but the coupling time Eτe can grow exponentially with the parameters of the network. We can bypass this by using a split (see § 5) where coupling occurs w.p.1 and the coupling time remains similar to the coupling time of PSA. Again, the complexity is divided by the size of the state space in most cases.

4. EPSA FOR NETWORKS OF QUEUES We consider an open network N consisting of N queues Q1 , . . . , QN . Each queue Qi has a finite capacity, denoted by Ci , i = 1, . . . N and a finite number of servers. The state space of a single queue Qi is Xi = {0, . . . Ci } and the state def

This new algorithm (Algorithm 2) calls for several comments. 1- Everything works the same if Φ (resp. Φ) is replaced by an upper (resp. lower) bound on the supremum (res. infimum). 2- The definition of the envelopes is based on the constructive definition Φ of the Markov chain. If one comes up with a new construction Φ′ of the Markov chain, then the envelopes are modified accordingly. 3- If the function Φ(., e) is non-decreasing for all event e, then for any m ≤ M , Φ(M, m, e) = Φ(M, e) and Φ(M, m, e) = Φ(m, e), so that Algorithm EPSA coincides with the classical monotone perfect simulation algorithm for monotone Markov chains. 4- The envelope approach may not gain over the general PSA because of two problems:

space of the network is X = (X1 × · · · × XN ). X is a lattice for the usual product order, denoted ≤. Its Bottom (resp. Top) element is (0, . . . , 0) (resp. (C1 . . . , CN )). In the following we will denote by 1i the vector with null components except the ith one which is equal to one. Since the network is open, customers are able to enter and depart the network. The customers who enter from outside the network arrive according to a Poisson process. Also, the service time at any server is exponentially distributed. After finishing his service at a server, a customer is either directed to another queue or leaves the network according to a routing policy. Here, we consider a quite general class of routing policies, based on indexes (it includes the classical routing policies such as deterministic, join the shortest queue, priority routing and others, see [7]). The set of output queues of a queue Qi is denoted ∆i . It may include the dummy queue QN +1 corresponding to the exit of the network. Also, each queue Qi is equipped with a family I i of index functions,

one for each queue in ∆i : Iki : {0, . . . , Ck } → R. The routing works as follows: a customer ending its service at Qi in state (x1 , . . . , xN ) is sent to queue Qj with

Mean coupling time 1600

j = arg mink∈∆i Iki (xk ).

PSA EPSA

1400

Under these assumptions the network evolves as a finite discrete time Markov chain, after uniformization. It can be constructed using events ri that correspond to the transfer of a customer after a service at queue Qi (for i ≤ N ) or to exogenous arrivals (for i = N + 1). The transition function of event ri , 1 ≤ i ≤ N , is given by:

1200 1000 800 600 400

Φ(x, ri ) = x − 1i δxi >0 + 1j δxi >0 δxj 0 , m− j∈∆j

As for EPSA, the envelopes correspond to the trajectories of the Markov chain starting with the top (resp. bottom) element of the state space because of monotonicity of the chain. This shows that the complexity of computing the envelopes is the same as the complexity of computing Φ: C(Φ, Φ) = 2 C(Φ). Also the coupling time of EPSA and the coupling time of PSA have the same distribution: τe =d τ . Therefore, the complexity of EPSA is O(N · Eτ ) and the complexity of PSA is O(N · |X | · Eτ ), with a clear gain for EPSA. Actually, in this case, EPSA corresponds to the well known simulation of monotone Markov chains.

4.1

Negative customers λ2 λ1 = 0.8

µ2 = 1.5

µ1 = 1 C1 = 15

C2 = 15 neg. customer with prob. 0.8

Figure 2: A network with negative customers. Here, we consider a network with additional events (gi ): negative customers. The arrival of a negative customer in a queue kills one customer in that queue, if any. The evolution of the state under the transfer of a negative customer from Qi is Φ(x, gi ) = x − 1i δxi >0 − 1j δxi >0 δxj >0 where j = arg mink∈∆i Iki (xk ). In that case, the Markov chain is no longer monotone. Indeed, consider two queues in tandem Q1 , Q2 with a negative customer leaving Q1 to join Q2 under two states x = (0, 1) < x′ = (1, 1). Then Φ(x, g1 ) = (0, 1) > Φ(x′ , g1 ) = (0, 0).

otherwise: Φ(M, m, gi ) = Φ(M, gi ),

Φ(M, m, gi ) = Φ(m, gi ).

As for the coupling time, it is easy to show that the bounds on the envelopes computed above always couple (the state where both bounds are equal to 0 can be reached with positive probability). However, the coupling time of these bounds is larger than the coupling time of PSA: τ ≤st τe . However, the numerical experiments reported here (as well as many others) suggest that τ and τe have the same order of magnitude. We ran simulations over the small network displayed in Figure 2. The coupling time of PSA and EPSA are reported in Figure 3 (in the following all data are given with 95 % confidence intervals). The coupling time of EPSA is around 3 times larger than PSA. This means that EPSA is about 40 times faster than PSA in this case. We also ran simulations of the same network without the negative customers. We compared the coupling time for the same average number of customers in queue 2 (Figure 4). They remain in the same order of magnitude. This shows that EPSA is not only much faster than PSA but also very efficient in absolute terms: the simulation time is similar to that of a similar monotone system.

4.2

Fork and join

Here we consider a network with additional types of queues: forks and joins. A fork queue is such that at service completion, several customers are sent simultaneously to several disjoint output queues if none is full, otherwise they are all lost. A join is a queue with several buffers. The server is only active when no buffer is empty. A service completion

if batch arrivals or departures of size one happen with positive probability in each queue.

Mean coupling time 3000

Proof. Let p be the probability that batch arrivals or departures of size one happen in each queue. Then, under this type of events only, the network has no batches and is monotone so that envelopes couple. Conversely, if at least one queue Qi has no arrival and no departure of single customers, then the gap between the two envelopes in Qi cannot become smaller than the smallest batch size (see Proposition 3 in Appendix).

pos neg

2500 2000 1500 1000 500 0

Figure 4: Coupling times of EPSA algorithm for a network with (“neg”) and without (“pos”) negative customers.

If individual customers cannot enter or leave one queue, the regular PSA using function Φ converges almost surely as soon as the batches sizes in each queue have no common divisor and the capacity of the queue is large enough (see Theorem 3 in Appendix). So this is a case where PSA can be applied and EPSA cannot. Section 5 presents a patch for EPSA in this case.

removes one customer in each buffer. Here again, the addition of a fork node (or a join node) in the queuing network makes it non-monotone.

Proposition 2. The envelopes Φ(x, y, e) and Φ(x, y, e) can be constructed for any couple of states (x, y), x ≥ y in time O(N log(C)).

Unlike for negative customers the computation of the exact envelopes is easy and is in O(N ) because fork and join nodes no do interfere with routing (not detailed). EPSA couples in finite time w.p.1 and experimental results show that the coupling times τe and τ are very close to each other (even closer than for negative customers).

The proof is given in Appendix B. According to this result, the supremum and the infimum computing cost can be considered negligible (logarithmic in the state space size).

0

2

4

6

8

10

12

14

16

18

20

E[x2]

Mean coupling time 1e+07

4.3

EPSA (+1, -1) EPSA (+2, -1) EPSA (+3, -1) EPSA (+5, -1)

Batch arrivals

Here, we consider a system of N queues with finite capacities Ci , 1 ≤ i ≤ N , with batch arrivals (made of several customers) and atomic rejections: if all the customers forming a batch cannot be accepted all together into the destination queue, then the whole batch is rejected. We have the following events in the system: An arrival of batch of size k, 1 ≤ k ≤ K, where K is the maximal size of a batch and such that, in state x, the destination queue is given by j = arg mini Ii (xi ). The new batch is accepted to queue j if and only if xj + k ≤ Cj . Otherwise the whole batch is rejected. Similarly, a departure in queue i, 1 ≤ i ≤ n of a batch of size ℓ can only happen if xi ≥ ℓ. The arrival and the departure of a batch in a queue are not monotone events. Consider the simple example of a single queue of size C and two states x = C − 1 and x′ = C − 2 respectively. With an event e corresponding to an arrival of a batch of size 2, one gets Φ(x, e) = x and Φ(x′ , e) = C, so that x > x′ and φ(x, e) < φ(x′ , e), contradicting the monotonicity property. Once the transition function for the Markov chains describing the behavior of a network with batches is defined, the next question is whether EPSA converges with probability one. The case of a single queue is studied in full details in Appendix. Based on this, the following can be shown for networks. Proposition 1. EPSA couples almost surely if and only

1e+06 100000 10000 1000 100 10 0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Figure 5: Coupling time in a (+k, −1) queue with batch arrivals of size k = 1, 2, 3 and 5 respectively, as a function of the arrival rate λ. The coupling time is a more difficult point. Here, the coupling time of EPSA can either be comparable or be much larger than the coupling time of PSA, depending on the load of the queues. This is illustrated by the set of experiments displayed in Figure 5. EPSA is used to simulate a single queue with capacity 50 with batch arrivals with rate λ, all with the same size k, and individual departures (denoted a (+k, −1) queue). In that case coupling of EPSA is guaranteed. When the queue is monotone (k = 1). the coupling time is always small and is maximum when the arrival rate λ equals the service rate µ. When the queue is not monotone (k > 1) the coupling time of EPSA is similar to the monotone case (even marginally smaller) for small values of λ, but grows extremely fast (even on a logarithmic scale) when kλ/µ becomes larger than 1. It is straightforward to explain why this happens. Envelopes can only couple when the

T

Upper envelope Mean coupling time 1e+06

EPSA PSA split

X S

100000 10000

lower envelope

1000

B −n

splitting

0

Figure 6: Splitting.

queue becomes empty (see the proof of Proposition 1).When the load grows, this happens with a very small probability. Actually, the classical PSA has similar coupling times, for the same reasons. Again, using EPSA is a clear gain over PSA, the complexity being roughly divided by |X |. Nevertheless, its simulation time may be too large to be used in practice in some cases.

5. BEYOND ENVELOPES As mentioned before, there are cases where EPSA does not couple or where its coupling time is too large. In that case envelopes can still be useful.

5.1

Splitting and bounds

Even if the two envelopes Φ, Φ cannot couple, they still may get close. Let L be the smallest possible cardinality of the n set {x ∈ X : Φn (T, B, e0→n )  x  Φ (T, B, e0→n }. One way to exploit the envelopes in that case is to continue the simulation once the gap between the envelopes reaches L using a classical PSA, simulating the chain starting with all states between the upper and lower trajectories. This is called splitting in the following. Algorithm 3: EPSAWS (EPSA With Splitting). Data: - Φ, {e−n }n∈N , Γ - L size of the set that triggers the splitting Result: A state x∗ ∈ X generated according to the stationary distribution of the system begin n = 1; M := T , m := B ; S = {x ∈ X , m ≤ x ≤ M }; repeat i := n; repeat (M, m) := Γ(M, m, e−i ); i := i − 1; S = {x ∈ X , m ≤ x ≤ M }; until |S| = L or i = 0 ; for j = i downto 1 do S := Φ(S, e−i ); n := 2n; until |S| = 1 ; return S; end Figure 6 illustrates a successful run of EPSA with splitting (called EPSAWS).

100 10 0

0.2

0.4 0.6 arrival rate

0.8

1

Figure 7: Coupling times for PSA, EPSA and EPSA with splitting for a (+2, +3, −1) queue. The main use of EPSAWS is when the state space is too big to use PSA directly but envelopes can reduce the size of the space very fast. Figure 7 reports the coupling times of EPSA, PSA and EPSAWS for a queue with capacity 20 and arrival batches +2 and +3 with probabilities 0.49 and 0.51 and departures of size one (with rate 1). In this queue, Theorem 3 says that all three algorithms converge. In the figure, the coupling time of PSA and EPSAWS are the same while EPSA’s coupling time grows very fast when λ/µ is larger than one. In this case EPSAWS has a clear advantage over the other two algorithms. Another possibility is to continue to compute the envelopes after the spliiting point instead of all trajectories. When time zero is reached, we get M0 and m0 that satisfy P(M0  x) ≤ Π(x) ≤ P(m0  x), where Π(x) is the cumulative stationary Pdistribution function (cdf) of the Markov chain: Π(x) = yx πy . Note that the gap between M0 and m0 may be larger than L. Hopefully, it is still small on most trajectories.

Mean coupling time (+4, -3) 1e+06

PSA bounds

100000 10000 1000 100 10 1 0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

arrival rate

Figure 8: Exact values versus bounds. Figure 8 reports the coupling times of EPSAWS and the number of steps needed for envelopes to get closer than 4

by Ψ) is given by: Distribution of number of jobs (+4, -3) 1

Event e1 : For each x ∈ X , Ψ(x, e1 ) = y only if the arc (x, y) is a part of one shortest path to 0. (If there is more than one shortest path from x to 0, then chose one of them.) The rate of event e1 is set to λ.

lower bound upper bound

0.8

0.6

Event e2 : For each x ∈ X , Ψ(x, e2 ) = Φ(x, d)δΨ(x,e1 )=Φ(x,a) + Φ(x, a)δΨ(x,e1 )=Φ(x,s) . The rate of event e2 is λ.

0.4

Event e3 : For each x ∈ X , Ψ(x, e3 ) = Φ(x, d). The rate of event e3 is µ − λ. (If λ = µ, there is no event e3 .)

0.2

0 0

20

40

60

80

100

Figure 9: Bounds for the c.d.f. for a (+4, −3) queue. in a (+4, −3) queue with capacity 15. The theory says that EPSA does not couple here but PSA and EPSAWS will terminate. The Figure shows that the coupling time of EPSAWSis too large to be useful for large queues. On the other hand, the time needed for both envelopes to get close is very small. It allows one to get rather tight upper and lower bounds on the c.d.f. (given for a (+4,-3) queues with capacity 100 and λ/µ = 1 in Figure 9).

5.2

Note that we might have more than one choice for event e1 . However, once we have fixed e1 , then e2 and e3 are completely defined. In Table 1 we report simulation times for example with k = 3, l = 2 and different values of capacity C. We can notice that alternative event description decreases significantly the coupling time. However, for this new event description computing of envelopes becomes more difficult. C 5 10 20

p=1/4 nat. alt. 150.46 9.54 244.64 16.94 4113.89 31.12

p=1/3 nat. alt. 92.29 13.06 142.51 25.15 696.30 54.92

p=1/2 nat. alt. 62.28 44.28 187.45 81.56 1304.00 585.19

Table 1: Coupling times for natural and alternative event description for case: k = 3, l = 2.

Alternative to splitting: change the events

Until now, we supposed that the constructive definition of the Markov chain Xn+1 = Φ(Xn , e) is fixed. Taking an alternative description of the chain can reduce the coupling time significantly, or make envelopes converge if this was not the case. If the Markov chain Xn is irreducible and aperiodic, it is always possible to transform slightly the chain preserving the stationary distribution such that there exists one state 0 for which the transition matrix verifies P00 > 0. In that case, choose a spanning tree A on the transition graph of the chain rooted in 0 [8]. By defining the order on X as the distance on that spanning tree A to state 0 and defining an event e which occurs with probability min(i,j) ∈A Pij such that Φ(x, e) is the next vertex on A after x, and other events in an arbitrary way such that Xn+1 = Φ(Xn , e), then EPSA converges with probability one. The reason is that under event e, the chain is monotone and both envelopes converge to 0 in that case. The main drawback of this approach is that the construction of the spanning tree has a complexity O(|X |2 ) which may be too large in practical cases. Modifying the event can also be used to improve the coupling time of both EPSA and PSA. We illustrate this on the example of a single queue with batch arrivals and services. In the appendix, we consider the natural events to construct the function Φ: batch arrivals of size k (event a) and batch services of size ℓ (event d). If gcd(k, ℓ) = 1 and k + ℓ ≤ C + 1, then Lemma 1 implies that the underlying Markov chain is irreducible and a spanning tree rooted in state 0 can be constructed. Suppose that arrival rate λ and service rate µ verify λ ≤ µ. Furthermore, without loss of generality, suppose that λ + µ = 1. An alternative event description (we denote the corresponding transition function

5.3

Computing envelopes

There is a compromise between simulation time and the complexity due to the number of trajectories that need to be considered. As seen in the previous paragraph, an alternative event description can decrease the coupling time of the PSA. However, in this case envelopes become more difficult to compute. One can relax further the notion of envelopes and, instead of taking infimum and supremum states for all the trajectories, one might prefer taking a state that is even smaller (resp. greater) than infimum (resp. supremum) state, but that is much easier to compute, as in the negative customer case. The question of finding an event representation which is better than the natural event representation, but for which envelopes can be computed efficiently (without generating the state space) remains open for the general case of networks of queues.

6. REFERENCES [1] A. Bouillard and B. Gaujal. Backward coupling for perfect simulation of free-choice nets. Journal of Discrete Event Dynamics Systems, theory and applications, 2006. Special issue of selected papers from the Valuetools conference. [2] A. Bouillard and B. Gaujal. Backward coupling in petri nets. In Valuetools, Pisa, Italy, 2006. [3] J. Dopper, B. Gaujal, and J.-M. Vincent. Bounds for the coupling time in queueing networks perfect simulation. In Numerical Solutions for Markov Chain (NSMC06), pages 117–136, Charleston, June 2006. Celebration of the 100th anniversary of Markov. [4] J.-M. Fourneau, I. Y. Kadi, N. Pekergin, J. Vienne, and J.-M. Vincent. Perfect Simulation and Monotone

[5]

[6]

[7]

[8]

[9]

[10]

Stochastic Bounds. In 2nd International Conference Valuetools’07, Nantes, France, October 2007. B. Gaujal and F. Perronnin. Coupling from the past in hybrid models for file sharing peer to peer systems. In Proceedings of the 10th International Conference HSCC’07, Pisa, Italy, April 2007. J. G. Propp and D. B. Wilson. Exact sampling with coupled markov chains and applications to statistical mechanics. Random Structures and Algorithms, 9(1-2):223–252, 1996. J.-M. Vincent. Perfect simulation of monotone systems for rare event probability estimation. In Winter Simulation Conference, Orlando, Dec. 2005. J.-M. Vincent and C. Marchand. On the exact simulation of functionals of stationary markov chains. Linear Algebra and its Applications, 386:285–310, 2004. J.-M. Vincent and J. Vienne. Perfect simulation of index based routing queueing networks. Performance Evaluation Review, 34(2):24–25, 2006. J.-M. Vincent and J. Vienne. Psi2 a software tool for the perfect simulation of finite queueing networks. In QEST, Edinburgh, Sept. 2007.

APPENDIX A. A SINGLE QUEUE WITH BATCHES This appendix is dedicated to the perfect simulation of a single queue with batch arrival and services. This appendix actually contains the main technical difficulties of the paper. Consider a single queue with finite capacity C, batch arrivals and batch services. Suppose that the arrival batch size is always equal to k and service batch size is equal to ℓ. Function Φ is based on two types of events: arrivals of k customers simultaneously (ak ) and departures of ℓ customers (dℓ ).   x, x > C −k x, xC −k  m + k, M ≤C −k Φ(M, m, ak ) = (m + k) ∧ ((C + k − 1) ∨ m), M > C − k  (m − ℓ) ∨ 0, M ≥ ℓ Φ(M, m, dℓ ) = m, M C. Then mt = mt−1 < Mt−1 = Mt , which is also not possible. • Mt−1 + k > C and mt−1 + k ≤ C. Then consider states x = Mt−1 and y= x − 1 ≥ mt−1 . Then Φ(x, ak ) = x x − 1, x>C −k+1 and Φ(y, ak ) = 6= x, so x − 1 + k, x ≤ C − k + 1 we obtain again that mt < Mt . Therefore, the envelopes do not couple. Lemma 1. The Markov chain is irreducible if and only if gcd(k, ℓ) = 1 and k + ℓ ≤ C + 1. Proof. We show first that both conditions are necessary. If k + ℓ > C + 1, then state ℓ − 1 is absorbing. Indeed, Φ(ℓ−1, ak ) = ℓ−1 since ℓ−1+k > C, and Φ(ℓ−1, dl ) = ℓ−1. If k + ℓ ≤ C + 1, then either an arrival or a departure modify the state of the system. Suppose that gcd(k, ℓ) = d. Then starting from a state x we can only go to states y ∈ [0, C] of form ak + b(−ℓ). Let k = sd and ℓ = rd. Then y = ak + b(−ℓ) = (as − br)d. If d > 1, then for instance from state 0 we can never reach state 1. In order to show that these constraints are also sufficient, we will show that if gcd(k, ℓ) = d and k + ℓ ≤ C + 1, then from each state 0 ≤ x ≤ C − d we can reach state x + d and from each state d ≤ x ≤ C we can reach state x − d. Then if d = 1 the chain is clearly irreducible. From B´ezout’s lemma, there exist α, β ∈ N such that αk − βℓ = d. Then state x + d can be reached from state x by a sequence of α arrivals and β services as follows: While (α + β > 0) repeat: • if (α > 0 and x + k ≤ C) then {α ← α − 1, x ← x + k} • else {β ← β − 1, x ← x − ℓ} Note that if α + β > 0, then α = 0 or x + k > C imply β > 0 and x − ℓ ≥ 0. At each iteration α + β is thus decreased by 1, so the above algorithm terminates in finite time. State x − d can be reached similarly by a sequence of α′ arrivals and β ′ departures such that α′ k − β ′ ℓ = −d. Lemma 1 implies that it is possible to construct a new function Φ such that Algorithm 1 converges. The following result shows that this is also the case with the original function Φ based on arrivals and departures.

Proposition 3. If min{k, ℓ} > 1, then the envelopes do not couple (EPSA never converges). Proof. Suppose that envelopes couple in finite time t. Then at time t − 1 we have mt−1 < Mt−1 and mt = Mt , where mt = Φ(Mt−1 , mt−1 , ak ) and Mt = Φ(Mt−1 , mt−1 , ak ). By symmetry of the problem, without loss of generality we can suppose that the coupling occurs after an arrival. We have three different cases: • Mt−1 + k ≤ C. Then mt = mt−1 + k < Mt−1 + k = Mt , which is not possible since we supposed that t is a coupling time.

Lemma 2. PSA converges almost surely if and only if gcd(k, ℓ) = 1 and k + ℓ ≤ C + 1. Proof. The proof that the constraints are necessary follows from Lemma 1. We will show that they are also sufficient. Consider two trajectories starting in points at distance d > 0. It is sufficient to show that we can decrease their distance by 1. Indeed, applying the same reasoning for any two points we can construct a sequence of events for which the trajectories starting at that two points couple, decreasing thus the total number of trajectories by 1. Thus

we can eventually construct a sequence of events for which all trajectories couple. Suppose that ℓ ≤ k. (The other case is symmetrical.) In order to show that we can decrease a distance of two trajectories by 1 we will use the following facts:

operator A): 1- Apply δ times operator B ◦ D. After each step make sure that we still have d < ℓ. If not, apply operator A as many times as needed to have d < ℓ. This gives d′ = δ(k − ℓ) − ζℓ + d. 2- Remark that ζ = γ − δ. Thus d′ = d − 1.

1- If d ≥ ℓ, then we can decrease distance by ℓ. We will call this operator A: A(d) = d − ℓ, d ≥ ℓ. Suppose that initial positions of two considered trajectories are points x and x + d, where 0 ≤ x ≤ C − d. Then˚ anˇ event sequence corresponding to operator A is given by xℓ successive departures.

Consider now a general case with arrival batch sizes that belong to a set K = {k1 , k2 , . . . , kK } and service batch sizes that belong to a set L = {ℓ1 , ℓ2 , . . . , ℓL }. Without loss of generality we can assume that k1 < . . . < kK and ℓ1 < . . . < ℓL . Denote by M = (L ∪ K)\{ℓ1 , k1 } = {m1 , m2 , . . . , mM }, with m1 < . . . < mM . Note that M ≤ K + L − 2. Let g0 = gcd(k1 , ℓ1 ) and

2- If d < ℓ, then there exists a sequence of events corresponding to operator B:

gi = gcd(gi−1 , mi ), 1 ≤ i ≤ M.

B(d) = ℓ − d, d < ℓ.

Furthermore, we suppose that gi > 1, for all i < M . If for i < M we have gi = 1, then we can consider M′ = {m1 , . . . , mi } instead of M.

˝ an initial configuration (x, x + d) start first by applying ¨For x successive departures. This gives a new position (y, y + ℓ d) such that 0 ≤ y < ℓ. If y + d ≥ ℓ, then apply one more departure which modifies distance to ℓ − d. Thus, an ¨ ˝event sequence corresponding to operator b consists of xℓ + 1 departures. If y + d < ℓ, then we will show that there is an event sequence that leads to a new configuration (z, z + d) ` such that one of the following is true: z < ℓ and z + d ≥ ´ ` ℓ or z = y + 1). By repeating this a finite number of times we will end up in a configuration (w, w + d) such that w < ℓ and w+d ≥ ℓ from which a departure leads to a configuration with a distance equal to ℓ − d. We proceed as follows: • i = 0; y0 = y. • Until z is not found do: – i ← i + 1. – Use one arrival followed by a finite number of departures in order to reach a new configuration (yi , yi + d) such that yi < ℓ. Note that the arrival does not modify d since yi−1 + d < ℓ and ℓ + k ≤ C + 1. Similarly, for a configuration (u, u + d) such that u ≥ ℓ a departure does not modify d. – If yi + d ≥ l, then z = yi and we are in case (a). – Else if yi = y+1, then z = yi and we are in case (b). We will show that the above algorithm terminates after a finite number of steps. From B´ezout’s lemma it follows that there exist α, β ≥ 0 such that α(−ℓ) + βk = 1. Consider the configuration (yβ , yβ + d). We have applied exactly β arrivals and a finite number η of departures. Thus yβ = y + η(−ℓ) + βk. Now 0 ≤ yβ < ℓ and α(−ℓ) + βk = 1 imply η = α. 3- If d < ℓ, then similarly to the above case, there exists a sequence of events corresponding to operator D: D(d) = k − d, d < ℓ. (We only need d < k to get the existence of a corresponding sequence, but we will always apply operator a when d ≥ ℓ.) From B´ezout’s lemma it follows that there exist γ, δ ≥ 0 such that γ(−ℓ) + δk = −1. A sequence of events allows to decrease the distance by 1 can be then obtained as follows. We suppose that initially d < ℓ (otherwise we can apply

Proposition 4. The Markov chain is irreducible if gcd(L ∪ K) = 1 and n o M max k1 + ℓ1 , max(gi−1 + mi ) ≤ C + 1. i=1

(1)

Proof. The proof that the above conditions are sufficient is similar to the proof of Lemma 1. Indeed, only by arrivals of size k1 and services of size ℓ1 we can reach all the states of the same equivalence class modulo g0 . Suppose now that mi corresponds to an arrival (the other case is symmetrical). Then condition gi−1 + mi ≤ C + 1 implies that from each state x we can go either to a state x − gi−1 ≥ 0 or to a state x + mi ≤ C. Then similarly to the proof of Lemma 1, it can be shown that from a state x we can reach all the states of the same equivalence class modulo gi = gcd{gi−1 , mi }. Since gM = 1, the chain is irreducible. Similarly to the proof of Lemma 1, the constraint gcd(L ∪ K) = 1 is also necessary, since otherwise starting from state x we can only visit states of type y = x+λd, where λ ∈ Z and d = gcd(L ∪ K). Condition k1 + ℓ1 ≤ C + 1 is also necessary (otherwise state ℓ1 − 1 is absorbing). The other conditions are not always necessary. Take for instance K = {4, 11}, L = {8, 10} and C = 12. Then it can be easily verified that the chain is irreducible. On the other hand, g0 = 4, m1 = 10, but g0 + m1 = 14 > 13 = C + 1. Theorem 3. PSA converges almost surely if gcd(L∪K) = 1 and C ≥ 3 max(L ∪ K). Proof. Suppose that ℓ1 ≤ k1 (the other case is symmetrical). Then we can find a sequence of events corresponding to operators A and Bm , m ∈ {ℓ1 , k1 } ∪ M: A(d) = Bm (d) =

d − ℓ1 , d ≥ ℓ1 , m − d, d < ℓ1 , m ∈ {ℓ1 , k1 } ∪ M.

Indeed, for two trajectories starting in points x and x + d, an event sequence corresponding to operator a is given by ⌈ ℓx1 ⌉ events dℓ1 . For operators Bm the event sequence is slightly more complex, but follows similar steps as in the proof of Lemma 2. Under condition C ≥ 3 maxm∈L∪K m,

it is always possible to translate the two points by 1 to the right (resp. left) using B´ezout’s lemma, without hitting the boundaries (i.e. without modifying distance d). Thus for any ℓ ∈ L (resp. k ∈ K) and any two initial states x and x + d we can always reach a state (y, y + d) such that y < ℓ and y + d ≥ ℓ (resp. a state (z, z + d) such that z ≤ C − k and z + d > C − k) with a finite sequence of events. Then applying an additional event dℓ (resp. ak ) yields a sequence of events corresponding to operator bℓ (resp. bk ). B´ezout’s lemma implies that there are α ≥ 0 and βx ∈ Z, x ∈ M ∪ {k1 } such that: X α(−ℓ1 ) + βm m = −1. m∈M∪{k1 }

Suppose that we have two trajectories at distance d < ℓ1 (otherwise use operator a). Then a sequence of events that decreases distance by 1 can be obtained as follows:

Iℓ (m) > IdM (M ) which implies Φℓ (z, ak ) = zℓ , m ≤ z ≤ M, where Φ = (Φ1 , . . . , ΦN ). Thus, m′ℓ = mℓ ,

Case 2: Iℓ (m) ≤ IdM (M ). We consider first Mℓ′ . We have two cases: • If ℓ = dM , then Md′ M = H (dM ) (MdM , mdM ). Indeed, for state z = (M1 . . . MdM −1 , a, MdM +1 . . . Mn ) we clearly have arg mini Ii (z) = dM . • If ℓ 6= dM , then we consider state: z = (M1 , . . . , Mℓ−1 , mℓ + b, Mℓ+1 , . . . , MN ), where b ≥ 0 is defined as follows: Iℓ (m + 1ℓ ) ≤ .. . Iℓ (m + b1ℓ ) ≤

For each m ∈ M ∪ {k1 } apply |βm | times: • a sequence of events corresponding to operator Bℓ1 ◦Bm if βm > 0, and Bm ◦ Bℓ1 if βm < 0, • a sequence corresponding to a finite number of successive applications of operator A, such that after each iteration we have a new distance d < ℓ1 .

m

P

m

Iℓ (m + (b + 1)1ℓ ) > IdM (M ). Then ∀α 6= ℓ, Iℓ (z) = Iℓ (m + b1ℓ ) ≤ IdM (M ) ≤ Iα (M ) = Iα (z),

As for the lower envelope, we have the following cases:

B. PROOF OF PROPOSITION 2 ′

Here is the construction of M = Φ(M, m, ak ) and m = Φ(M, m, ak ) when a batch of k customers are routed into the system according to the index functions I. It should be clear looking at the construction of M ′ and m′ below that it can be done in O(N log C), where C is the maximal capacity of all queues by using a dichotomic search. For each queue i we first introduce for notational convenience, two functions h(i) and H (i) , describing the infimum and supremum values for the number of customers in queue i after a batch arrival of size k in queue i, given that the number of customers in that queue just before the new arrival is between u and v:  (v + k) ∧ C, u ≤ C − k H (i) (v, u) = v, u>C −k  u + k, v ≤C −k h(i) (v, u) = (u + k) ∧ ((C + k − 1) ∨ u), v > C − k.

• If ℓ 6= dm , then m′ℓ = mℓ . • If ℓ = dm and Idm (M ) ≤ mini6=dm Ii (m) (note that this implies dm = dM ), then: m′dm = h(dm ) (Mdm , mdm ). • If ℓ = dm and Idm (M ) > mini6=dm Ii (m) then we consider state z = (m1 , . . . , mdm −1 , mdm + c, mdm +1 , . . . , mN ), where c ≥ 0 is defined as follows: Idm (m + 1dm ) ≤

Let dm and dM be defined as follows: dM = arg mini Ii (M ).

We consider all the queues ℓ = 1..N . For each of them, two different cases have to be considered. Case 1: Iℓ (m) > IdM (M ). Then obviously ℓ 6= dM , since m ≤ M and I is increasing. For all z, m ≤ z ≤ M , Iℓ (z) ≥

min Ii (m),

i6=dm

.. . Idm (m + c1dm ) ≤ Idm (m + (c + 1)1dm ) >

dm = arg mini Ii (m),

IdM (M ),

Mℓ′ = max{Ml , H (ℓ) (mℓ + b, mℓ )}.

βm + γ. Thus, d′ = d − 1.

′

IdM (M ),

and arg mini Ii (z) = ℓ. Furthermore, state z is a state with the highest value of the component ℓ among all the states in which the new batch is sent to queue ℓ. Thus for the upper envelope, we have:

This gives a new distance X βm (m − ℓ1 ) + γ(−ℓ1 ) < ℓ1 . d′ = d + Finally, remark that α =

Mℓ′ = Mℓ .

min Ii (m),

i6=dm

min Ii (m).

i6=dm

Note that mdm + c is the maximal value of component dm for which we still always send the new batch to queue dm . We have: m′dm = min{mdm + c + 1, h(dm ) (mdm + c, mdm )}.