dynamic admission control for loss systems with batch ... - CiteSeerX

Applied Probability Trust (19 November 2004)

DYNAMIC ADMISSION CONTROL FOR LOSS SYSTEMS WITH BATCH ARRIVALS

¨ ˙ ∗ Ko¸c University E. L. ORMEC I,

A. BURNETAS,∗∗ University of Athens

Abstract We consider the problem of dynamic admission control in a Markovian loss system with two classes. Jobs arrive at the system in batches, where each admitted job requires different service rates and brings different revenues depending on its class. We introduce the definition of “preferred class” for systems receiving mixed and one-class batches separately, and derive sufficient conditions for each system to have a preferred class.

We also establish

a monotonicity property of the optimal value functions, which reduces the number of possibly optimal actions. Keywords: admission control; loss systems; batch arrivals AMS 2000 Subject Classification: Primary 93E20 Secondary 90B99

1. Introduction In this paper, we consider a two class loss system with c identical parallel servers, and no waiting room. Arrivals occur according to a Poisson process with rate λ. Each arrival consists of a random batch of jobs from one or both classes. Specifically, the probability that an arriving batch consists of j1 jobs of class 1 and j2 jobs of class 2 is equal to pj1 j2 . Whenever a class-i job ∗

Postal address: Department of Industrial Engineering

˙ Ko¸c University Rumeli Feneri Yolu 34450 Sarıyer, Istanbul, Turkey email: [email protected] ∗∗ Postal address: Department of Mathematics, University of Athens Panepistimiopolis 15784, Athens, Greece, email: [email protected] The second author’s research was partially supported by the Greek Ministry of Education and the European Union Research Program “Pythagoras”, Grant no. 70/3/7388.

1

¨ Ormeci & Burnetas

2

is admitted to the system, it brings a reward of ri > 0 upon its arrival and requires a service time exponentially distributed with rate µi . We are interested in dynamic admission policies that maximize the total expected discounted reward with a continuous discount rate β over an infinite horizon as well as the long-run average net profit. The system may employ batch acceptance, in which it can either accept or reject the entire batch, or partial acceptance, where some of the jobs in a batch can be admitted and the remaining ones rejected. For the case of batch acceptance, ¨ Ormeci and Burnetas [12] consider a system with equal service rates and equal rewards for all jobs, so that the only difference between the jobs is the size of the batches they belong to. They show that this system does not possess any monotonicity property. Hence, our paper concentrates only on systems following a partial acceptance policy. Systems of this kind may arise in various applications such as production systems and rental businesses. Controlling the workload in a production system via dynamic policies of admission has received increasing recognition in recent years. The types of items to produce may differ in service requirements and the profit they generate. Although the system owner may charge a fixed entrance fee for jobs of a specific type, the actual cost of serving each job may vary due to the changing utilization of the system resources. In such situations, the system can benefit from basing its acceptance decisions on the current state of the system. This calls for dynamic rather than static admission control rules. Moreover, in most production systems orders arrive in batches, and the system owner, generally, has the option of accepting some of the jobs in a batch while rejecting the rest. All these features are represented in the system described above. Our model can also be applied to revenue management problems in rental businesses. Capacity control in revenue management addresses the optimal allocation of a fixed amount of resources to different demand segments. The underlying assumption is the perishability of the resources at a certain time. The effect of revenue management on the performance of car rental firms are discussed by Carrol and Grimes [4] and Geraghty and Johnson [5]. However, for many rental businesses, in particular for car rental, the perishability assumption is not appropriate, since the resources are rented for a random amount of time, after which they are available again for future customers. Savin, Cohen, Gans and Katalan [20] are the first to formulate such systems as multiple-server loss models with uncertain customer arrivals and uncertain rental durations. Geraghty and Johnson [5] indicate that the major car rental companies depend largely on corporate customers, who demand more than one resource at a time. The system under consideration models the uncertain demand size generated by these customers using random

Admission control with batch arrivals

3

batches. We first develop sets of sufficient conditions which ensure that a job class is “preferred”, in the sense that its jobs are always accepted whenever there are free servers, regardless of the system congestion level. We also show that the characterization of a preferred class when all batches consist of jobs from the same class is analogous to that in a system with single job arrivals. In both cases, accepting a job (single or from the batch) depends on the tradeoff between the burden this job will bring to the system if accepted and the forgone profit if rejected. On the other hand, when a batch includes jobs from both classes, in order for a job of one class to be accepted, its net benefit must not only be positive, but also higher than the benefit of a job from the other class. Therefore, the characterization of preferred classes in this case is more complicated. We note that the conditions developed in this paper are sufficient but not necessary, since there are combinations of parameter values for which they do not ensure the existence of a preferred class. We analyze such cases numerically and make several observations and conjectures. The existence of a preferred class specifies a static property of the optimal admission policy, in the sense that if class i is preferred, jobs of this class are accepted whenever there is an available server, regardless of the current state of the system. However, the optimal policy is generally state dependent. For this general question, we prove submodularity of the value function, a desirable property which guarantees the existence of optimal thresholds in single-arrival loss systems (see Altman, Jim´enez and Koole [2]). However, in systems with batch arrivals, submodularity can characterize the structure of optimal policy only partially, although its proof is mathematically challenging and interesting due to the boundary effects, multiple servers and batch arrivals. Markov decision processes are widely used in the control of queueing systems: Altman [1] surveys the theoretical tools developed to model and to solve such problems, as well as their application areas in communication networks. In particular, admission control problems in stochastic knapsacks have been studied by many authors, see Chapter 4 of Ross [18] for a comprehensive review. A stochastic knapsack is defined as a system which consists of c identical parallel servers, no waiting room and K job classes. Each class is distinguished by its size bi , its arrival rate and its mean service time. If a class-i job is accepted to the system, it seizes bi servers, and occupies all of them till the end of its service time, upon which it releases all bi servers at the same time. This is substantially different from the system with batch arrivals for various reasons: In a system with batch arrivals, each admitted job of an arriving batch behaves independently, i.e., each job has its own service time and it occupies and releases only one server. Moreover,

¨ Ormeci & Burnetas

4

these systems, unlike stochastic knapsacks, can receive mixed batches. For further comparison of stochastic knapsacks and systems with batch arrivals, we refer to [12] which includes examples that compare the structure of the corresponding optimal policies. There have been earlier studies which investigate the structural properties of optimal admission policies for certain stochastic knapsacks: Miller [10] has showed that there exists an optimal trunk reservation policy for a stochastic knapsack with K different job classes, each of which has the same mean service time and a unit size, so bi = 1. Lippman and Ross [7] analyze the optimal admission rule for a system with one server and no waiting room which receives offers from jobs according to a joint service time and reward probability distribution. They also consider this system when it receives batch arrivals. Several authors study the structural properties of optimal dynamic admission policies in Markovian stochastic knapsacks with two classes of jobs: ¨ Altman et al. [2] show that these policies are of threshold type, whereas Ormeci, Burnetas and van der Wal [14] establish the monotonicity of thresholds under certain conditions while assuming ¨ bi = 1. Moreover, Ormeci, et al. [14] and Savin et al. [20] analyze the issue of preferred classes ¨ in these systems. Ormeci, Burnetas and Emmons [13] address the issues of optimal thresholds and preferred jobs when the rewards are random. Carrizosa, Conde and Munoz-Marquez [3] analyze an optimal static control policy for a stochastic knapsack with K classes of jobs, where the service times follow a general distribution. A loss system with batch arrivals is considered by Puhalskii and Reiman [15], where they restrict the domain of admission policies to the set of trunk reservation policies. However, optimal policies are not necessarily trunk reservation ¨ policies, as illustrated in section 4. Ormeci and Burnetas [12] also consider the structure of optimal policies in loss systems with batch arrivals, where they concentrate on systems with K classes of jobs which have different rewards, but the same service rate. This paper is organized as follows: In the next section, we present the corresponding Markov decision process (MDP) model of the system described above. The third section presents the conditions under which a preferred class exists for systems with one-class and mixed batches. The fourth section proves submodularity of the optimal value functions. We conclude and point out possible extensions of this work in the last section.

Admission control with batch arrivals

5

2. Model Description 2.1. Discrete Time Equivalent In this section, we build a discrete time MDP for systems employing a partial acceptance policy with the objective of maximizing total expected discounted returns over a finite time horizon with β as the discount rate. We first define the state as x = (x1 , x2 ), where xi is the number of class-i jobs. We interpret discounting as exponential failures, i.e., the system closes down in an exponentially distributed time with rate β (for the equivalence of the process with discounting and the process without discounting but with an exponential deadline, see e.g., Walrand [21]). We also assume, without loss of generality, that µ1 ≤ µ2 . Arrivals occur according to a Poisson process with rate λ. Then, the maximum possible rate out of any state is λ + cµ2 + β. Since the maximum rate of transitions is finite, we can use uniformization (introduced by Lippman (1975)) and normalization to build a discrete time equivalent of the original system. Specifically, we assume that the time between two successive transitions is exponentially distributed with rate λ + cµ2 + β, and using the appropriate time scale, that λ + cµ2 + β = 1. At each transition epoch there is either an arrival of a batch of jobs with probability λ, a service completion of a class-i job with probability xi µi , a fictitious service completion with probability cµ2 − x1 µ1 − x2 µ2 , due to uniformization, or a transition to the terminal state with probability β due to discounting. Furthermore, we will refer to the instantaneous states at the arrival epochs as (x, j) = (x1 , x2 ; j1 , j2 ) showing that a batch of (j1 , j2 ) jobs has arrived at the system to find xi classi jobs already present, i = 1, 2. Note that admission and rejection decisions are assumed to be made upon an arrival so that these states are observed only at arrival epochs. Immediately after those epochs the system moves to another state according to the decision made. A batch (j1 , j2 ) which consists of two classes, i.e., j1 > 0 and j2 > 0, is called a mixed batch, whereas batches with j1 = 0 or j2 = 0 have only one class of jobs, and they are referred to as one-class batches. We will consider two kinds of systems: systems with mixed batches receive at least one mixed batch with a positive probability, and systems with one-class batches never receive mixed batches. 2.2. A Markov Decision Model for Finite Horizon We denote the maximal expected β-discounted n-horizon net benefit of a system starting in state x and in state (x; j) by un (x) and v n (x; j), respectively. Let S be the state space, so S = {x : x1 + x2 ≤ c}. Arrivals occur according to a Poisson process with rate λ, and at each

¨ Ormeci & Burnetas

6

arrival epoch the batch of arriving jobs consists of ji class-i jobs, i = 1, 2 with probability pj1 j2 ,   where cj1 =0 cj2 =0 pj1 j2 = 1 and p00 = 0. We occasionally denote a batch (j1 , j2 ) by j and the corresponding probability by pj for brevity. Define y n (x; j) = (y1n (x; j), y2n (x; j)) as the optimal state to move into when a batch j arrives at the system in state x with n transitions remaining. Then let S(x; j) be the action space for state (x; j), i.e., the set of states that can be reached from state x when a batch j arrives: +  S(x; j) = y ∈ S : xi ≤ yi ≤ xi + ji , i = 1, 2 .

Note that S(x; j) = {x} for any (x; j) such that x1 + x2 = c.

We present the optimality equations. Let ei be the ith unit vector. For x1 + x2 ≤ c: +3  v n (x; j) = max ri (yi − xi ) + un (y) : y ∈ S(x; j) un+1 (x) = λ

3 j

(1)

i=1,2

pj v n (x; j) + x1 µ1 un (x − e1 ) + x2 µ2 un (x − e2 )

+(cµ2 − x1 µ1 − x2 µ2 )un (x),

(2)

where we define un (−1, x2 ) = un (0, x2 ) and un (x1 , −1) = un (x1 , 0). Let y ∗ = y n (x; j) be the optimal decision if a batch j arrives in state x when there are n more transitions. If the nth event is an arrival of batch j, then yi∗ − xi of ji class-i jobs are accepted so that the system moves  to state y ∗ with total reward equal to i=1,2 ri (yi∗ − xi ). If a class-i job completes service with

probability xi µi , the system state changes to x − ei . The “fictitious” service completions, which

occur with probability cµ2 − x1 µ1 − x2 µ2 , affect neither the state nor the total reward. Finally, if the terminal state is reached (with probability β), no more reward is received. Remark 1. In this model we assume class-dependent rewards, ri , which are collected in the beginning of service, immediately upon admission of a class-i job. Indeed, these rewards can model more general reward/cost functions, as our results do not change if holding or rejection costs are incurred: A system with holding cost hi , rejection cost bi , and reward ri is equivalent to a system which collects a reward of ri = ri +bi −hi /(µi +β). Similarly, a system collecting a reward of Ri at the end of service is equivalent to a system which collects a reward of ri = Ri µi /(µi + β) upon admission. 2.3. Infinite Horizon Models The main aim of this paper is to describe the structure of optimal dynamic admission policies for systems which operate over an infinite horizon. For this purpose, we first prove our results

Admission control with batch arrivals

7

with the objective of maximizing the total expected β-discounted reward for a finite number of transitions, n, including the “fictitious” transitions due to the “fictitious” service completions (last term in (2)). Thus, “finite” horizon problems are pseudo finite problems, which allow us to use the powerful tool of induction to prove our results for all finite n. This subsection shows that the results for finite n extend to infinite horizon problems. To start the induction, we specify the initial function u0 as u0 (x) = 0 for all x ∈ S in all our results. Of course, this makes no difference in the optimal policy for infinite horizon problems. We first analyze the model when the total expected β-discounted reward over an infinite horizon is maximized. A maximizer policy is sought in the set of all history dependent policies. However, for this problem, there is always an optimal deterministic stationary policy, due to the finiteness of the state space and the action spaces of all states and the bounded rewards (Theorem 6.2.10 of Puterman [16]). Moreover, this policy can be computed by the value iteration algorithm. Then, all our results for finite horizon extend to infinite horizon with discounting. Specifically, let v(x; j) and u(x) denote the value functions for the infinite horizon expected discounted reward. Thus, for β > 0: v(x; j) = lim v n (x; j), n→∞

u(x) = lim un (x), n→∞

and y(x; j) = (y1 (x; j), y2 (x; j)) is an optimal action in state (x; j) so that it is optimal to accept yi (x; j) − xi class-i jobs when a batch of j = (j1 , j2 ) arrives at a system with xi class-i jobs. Next we consider the criterion of maximizing the expected long-run average reward. In this case we need to define the relative value functions, v (x; j) and u (x), and the gain, g, in the usual MDP formulation. Specifically: v (x; j) = max g + u (x) = λ

+3

3 j

i=1,2

ri (yi − xi ) + u (y ) : y ∈ S(x; j)



pj v (x; j) + x1 µ1 u (x − e1 ) + x2 µ2 u (x − e2 )

+(cµ2 − x1 µ1 − x2 µ2 )u (x), where we define y (x; j) = (y1 (x; j), y2 (x; j)) as an optimal action in state (x; j). We first observe that the resulting MDP is unichain, since under all possible policies state (0, 0) is reachable. Moreover, it is aperiodic due to the fictitious service completions. Then, Theorem 8.4.5 of Puterman [16] guarantees the existence of an optimal deterministic stationary policy in the set of all history dependent policies for the long-run average criterion, as well as the validity of the

¨ Ormeci & Burnetas

8

value iteration algorithm to find this policy. Moreover, as β → ∞ in the infinite-horizon problem with discount rate β, we obtain the long-run average problem (Theorem 6.18 and Theorem 6.19 of Ross [19]). Hence, all our results for the infinite-horizon problem with discounting hold for the long-run average case. 2.4. Coupling and policy perturbations In our proofs, we mostly use induction and coupling together. Coupling is a widely-used method in Markov decision models, especially to show the stochastic optimality of a specified policy; see Liu, Nain and Towsley [9] for some examples in the control of queueing systems, and Righter [17] for some examples in stochastic scheduling. In the context of our model, a question that often arises is that of the relative value of one state versus another one. To answer this question, we define a number of systems, which have the same cost structure and dynamics as the model described in section 2.2, and start in different states. Furthermore, they are coupled in the realizations of arrivals and service completions, and follow policies related in a way that will be specified below. We describe the coupling process and the policies for two systems A and B only, and the extension to more than two is straightforward. First, we explicitly couple all the random variables for A and B as in an algorithm for simulation, or construction of realizations. Specifically, both systems will have the same arrival stream. Moreover, the service times of jobs are coupled as follows: If the coupled jobs are of the same class, they depart at the same time, otherwise we use the assumption that µ1 ≤ µ2 (from which it follows that class-1 jobs are “slow”): We let ξ be a uniformly distributed random variable in (0, 1), and we generate the service time of the class-1 job, say job d1 , and the class-2 job, say job d2 , using the same ξ, so job d2 has a shorter service time than job d1 with probability 1. In terms of discrete time, this translates to the following: Both jobs leave the system with probability µ1 , and a class-2 job departs from the system with probability µ2 − µ1 leaving the coupled class-1 job in the system. Thus, coupling does not allow a coupled class-1 job to leave the system while the coupled class-2 job is still there. The introduction of coupling brings certain restrictions on the dynamics of the two systems implying relationships between the generated profits that facilitate the comparison of the different states. In terms of the policies followed, we let π be an optimal policy which is obtained as a solution to equations (1) and (2). Hence, π = (π n )n≥1 specifies a rule to follow in each state (x; j) for every period n. Moreover, we can compute πn using dynamic programming for any given n.

Admission control with batch arrivals

9

We now let system A always follow the optimal policy π, whereas system B follows π except for m period m when it follows a feasible rule, πB , which is a modification of π m . In order to ensure

the feasibility, this rule depends on the initial state, xB , of system B. In other words, in period m m system B follows a rule which depends only on xB and π m . In particular, πB depends on m πm (xA ), where xA is the initial state of system A. However, there is no dependence of πB on the

sample path of system A, or on any realizations of the coupling process ξ. In this framework, the policy that system B follows belongs to the class of history dependent policies for system B. Therefore if unB (x) is the value function of system B, we always have unB (x) ≤ un (x) for all x and for all n, an inequality we use in our proofs. 2.5. The Effect of an Additional Job In our analysis below, the effect of admitting one class-i job is an important quantity. To that end let Dn (ki)(x) = un (x+ek )−un (x+ei ) denote the difference in the total expected discounted rewards between two systems A and B, where system A starts in state x ‘plus’ one class-k job and system B starts in x plus a class-i job. Note that k = 0 means that system A is in state x, i.e., there is no additional job. We occasionally omit the arguments x and n when there is no danger of confusion. The four functions of interest are Dn (01), Dn (02), Dn (21) and Dn (12). We can interpret the difference Dn (0i)(x) as the loss in future rewards because of the increased load if a class-i job is accepted, or equivalently, the expected burden that an additional class-i job brings to the system in state x when there are n remaining transitions. Similarly, the difference Dn (21)(x) represents the expected additional burden, when in state x + e2 , of changing a class-2 job that is already in the system to a class-1 job. If the arrivals were single, the admission policy would compare the effects of two actions in state x upon a class-i arrival, namely accepting a class-i job and moving to state x + ei , or rejecting it and remaining in state x. Thus, it would be better to accept a class-i job if and only if un (x) ≤ ri + un (x + ei ), or equivalently Dn (0i)(x) ≤ ri .

(3)

When equation (3) holds for all x ∈ S for some class i, it specifies class i as a preferred class in a system with one-class batches, since we easily argue that in such a system it is optimal to

¨ Ormeci & Burnetas

10

accept as many jobs as possible from a class-i batch: Assume by contradiction that there exists an optimal policy under which the system moves to state y with yi < xi + ji and y1 + y2 < c, so that this optimal policy rejects at least one job from the arriving class-i batch while the system has at least one idle server. Then, from optimality equations (1), un (y) > ri + un (y + ei ), which contradicts to our assumption that equation (3) is true for all x ∈ S for class i. On the other hand, when mixed-class batches are possible, equation (3) is not sufficient to characterize a class as preferred. Indeed, assume that only one server is empty and a mixed batch has arrived. Then only one job can be accepted, which calls for a comparison of the actions of accepting a class-i or a class-k job. Hence, it is better to accept a job of class i rather than of class k in state x if rk + un (x + ek ) ≤ ri + un (x + ei ), or in terms of Dn (ki)(x): Dn (ki)(x) ≤ ri − rk .

(4)

Therefore, a sufficient condition for class i to be preferred is that both equations (3) and (4) hold for all x for i. When class i is preferred, free servers are filled with class i jobs first, and only then admission of jobs of the other class is considered.

3. Existence of a Preferred Class We define a preferred class as the class whose jobs are always admitted to the system whenever there are available servers. As discussed earlier, this definition leads to different characterizations of preferred class(es) with one-class and mixed batches, which are considered separately in this section. In fact, the results for one-class batches, as well as their proofs, are very similar to those ¨ for systems with single arrivals (see Ormeci et al. [14]). Hence, we will only state the results, and the corresponding intuitions for this case. In determining preferred class(es), two different criteria can be considered, lump rewards, and average rewards. That is, we would expect that whenever the corresponding reward of class i is higher than that of the other class, class i is preferred. We will discuss the validity of both criteria in conjunction with the related results. Before analyzing the systems with mixed and one-class batches separately, we prove the following result, which applies to both systems: Lemma 1. For all x ∈ S and for all n ≥ 0:

Admission control with batch arrivals

11

(i) Dn (0i)(x) ≥ 0 for i = 1, 2. (ii) Dn (21)(x) = −Dn (12)(x) ≥ 0. Proof. We prove the statements by a sample path analysis. (i) Assume that system A is in state x and system B in x + ei in period n. We let system B move to state y ∗ = y n (x + ei ; j) following the optimal policy, and system A move to y ∗ − ei . In other words, system A and B always accept the same number of jobs from each class. We couple the two systems via the service and interarrival times, so that except for the additional job in system B, all the departure and arrival times are the same in both systems. Then, all future rewards of system A and B are the same: Dn (0i)(x) = un (x) − un (x + ei ) ≥ unA (y ∗ − ei ) − un (y ∗ ) = 0, where unA is the expected discounted return of system A. (ii) Assume that system A starts in state x + e2 and system B starts in x + e1 , where we now couple the additional class-2 job, say job d2 , in system A with the additional class-1 job, say job d1 , in system B, as well as all other service and interarrival times, so that, as discussed earlier, if d1 leaves the system, d2 also leaves. Then, we can let system B follow the optimal policy moving to state y ∗ = y n (x + e1 ; j) and system A accept exactly the same jobs, so that it moves to y ∗ − e1 + e2 . Now, again, all future rewards of both systems are equal: Dn (21)(x) = un (x + e2 ) − un (x + e1 ) ≥ unA (y ∗ − e1 + e2 ) − un (y ∗ ) = 0, with unA the expected discounted return of system A. This lemma shows that it is always preferable to be in a state where there are fewer or faster jobs: Recall that un (x) is equal to the expected discounted total reward of the system under an optimal policy when there are n more transitions. Since rewards are collected in the beginning of service, jobs that are already in the system do not contibute to un (x). Therefore, the jobs initially in the system bring only more burden by blocking acceptance of future jobs. 3.1. Mixed Batches In this section, we assume that there exists a batch j = (j1 , j2 ) with j1 > 0, j2 > 0 and pj1 j2 > 0, so we have batches consisting of both classes. Whenever a mixed batch arrives at the system, i.e., when a batch of j arrives with j1 > 0 and j2 > 0, we need to compare at least three actions, having an empty server or having a server work on class-1 or class-2 job. As discussed

¨ Ormeci & Burnetas

12

earlier in section 2.5, there can be at most one preferred class for these systems. Moreover, class i is preferred if Dn (ki)(x) ≤ ri − rk and Dn (0i)(x) ≤ ri , i = k, for all x and for all n. Therefore,

the effect of changing a class-i job to class k, Dn (ki)(x), is important as well as the effect of an additional class-i job, Dn (0i)(x), in determining the preferred class. We first present the sufficient conditions for class 2 to be preferred: Proposition 1. If r2 ≥ r1 , then class 2 is the single preferred class. Proof. Assume r2 ≥ r1 . It is enough to show that Dn (12)(x) ≤ r2 − r1 and Dn (02)(x) ≤ r2 for all x and for all n. For the former statement, we use Lemma 1 and the assumption r2 ≥ r1 so that: Dn (12)(x) ≤ 0 ≤ r2 − r1

∀x ∈ S

∀ n.

Hence, we only need to show that Dn (02)(x) ≤ r2 for all x and for all n. D0 (02)(x) ≤ r2 for

u0 (x) = 0 for all x ∈ S. Thus, assume that Dn (02)(x) ≤ r2 for all x and for some n, and consider n + 1: Let system A be in state x and system B be in state x + e2 in period n + 1. We let system A take optimal actions y ∗ = y n+1 (x; j), and system B imitate these actions whenever possible: If system A accepts at least one class-2 job(s) so that y2∗ > x2 , then both systems end up in the same state with a difference of r2 in reward, since system B rejects one of the class-2 jobs that system A accepts. If y2∗ = x2 and y1∗ + y2∗ < c, then system B goes to state y ∗ + e2 so that system B accepts exactly the same jobs that system A accepts, resulting in the same amount of rewards. If y2∗ = x2 and y1∗ + y2∗ = c so that y1∗ > x1 , then system B moves to state y ∗ + e2 − e1 , accepting one less class-1 job than system A, so the difference in the rewards of the two systems is r1 . If the extra job in system B leaves, which happens with probability µ2 , the two systems couple with no difference in reward. If there is any other service completion, then the difference between two systems remains the same due to the extra class-2 job in system B. If un+1 B (x + e2 ) is the expected discounted return of system B, then: Dn+1 (02)(x) = un+1 (x) − un+1 (x + e2 ) ≤ un+1 (x) − un+1 B (x + e2 ) ‚ \ ≤ λ max r2 , Dn (02)(z), r1 + Dn (12) (z) z∈S

+(c − 1)µ2 max{Dn (02)(z)} z∈S \ ‚ ≤ λ max r2 , r2 , r1 + r2 − r1 + (c − 1)µ2 r2 < r2

where the first inequality is true because the policy followed by system B is not necessarily optimal, the second is due to the coupling described above, and the third follows by the induction

Admission control with batch arrivals

13

hypothesis Dn (02)(x) ≤ r2 and the fact that Dn (12)(x) ≤ r2 − r1 under the assumption r2 ≥ r1 .

Hence, Dn (02)(x) ≤ r2 and Dn (12)(x) ≤ r2 − r1 for all x and for all n, implying that class-2 jobs are preferred. We can state Proposition 1 in terms of the lump reward criterion to determine preferred class(es): If the lump reward of class 2 (fast class) is higher, then class 2 is preferred. Obviously,

class 1 (slow class) is not necessarily preferred if r1 > r2 , as we may have µ1 x1 , system B also moves to state y ∗ rejecting one of the class-1 jobs system A accepts; so that the two systems couple with a difference of r1 in reward. If y1∗ = x1 and y1∗ + y2∗ < c, then system B goes to state y ∗ + e1 . In this case, system B accepts exactly the same jobs that system A accepts, resulting in the same amount of rewards. If y1∗ = x1 and y1∗ + y2∗ = c so that y2∗ > x2 , then system B goes to state y ∗ + e1 − e2 , accepting one less class-2 job than system A, so the difference in the rewards of the two systems is r2 . With the departure of the additional class-1 job in system B, both systems enter the same state with no difference in reward, whereas all other service completions keep the

¨ Ormeci & Burnetas

14

difference between the two systems the same, which is due to the extra class-1 job. Then: Dn+1 (01)(x) ≤ un+1 (x) − un+1 B (x + e1 ) ≤ λ max{r1 , Dn (01)(y), r2 + Dn (21)(y)} y∈S

+(cµ2 − µ1 ) max{Dn (01)(y)} y∈S

≤ λ max{r1 , r2 + r1 − r2 } + (cµ2 − µ1 ) = λr1 + (1 − λ − µ1 − β)

λr1 λ + µ1 + β

λr1 λ + µ1 + β

λr1 λ + µ1 + β where the first and second inequalities are due to the policy followed by system B, the third =

due to the induction hypotheses Dn (01)(x) = λr1 /(λ + µ1 + β) < r1 and Dn (21)(x) ≤ r1 − r2 , whereas the equalities are due to uniformization, i.e., λ + cµ2 + β = 1, and some algebra. Thus, the first statement is true for all x ∈ S and for all n ≥ 0. (ii) Now consider the second pair of systems: Let system A be in state x + e2 and system B I

in x + e1 in period n + 1. System A takes the optimal actions, where we set y ∗ = y n (x + e2 ; j), and we let system B accept exactly the same jobs that system A accepts so that system B I

goes to state y ∗ + e1 − e2 in this period, incurring no difference between the rewards of the two systems. We, as in Lemma 1, couple the additional class-2 job, say job d2 , in system B with the additional class-1 job, say job d1 , as well as all other service and interarrival times. Then, if d1 leaves the system, which happens with probability µ1 , d2 also leaves. The departure of d1 leads the two systems to couple with no difference in reward, the departure of d2 alone, which happens with probability µ2 − µ1 , takes the systems to two different states, x + e1 and x, again with no difference in reward and whenever there is any other transition, both systems continue to have their additional jobs so that the difference between the two systems is only due to changing a class-1 job to class 2: I

Dn+1 (21)(x) ≤ λDn (21)(y ∗ − e2 ) + (µ2 − µ1 )(Dn (01)(x)) +(c − 1)µ2 max{Dn (21)(y)} y∈S

≤ (λ + (c − 1)µ2 ) max{Dn (21)(y)} + (µ2 − µ1 ) max{Dn (01)(y)} y∈S

λr1 ≤ (1 − µ2 − β)(r1 − r2 ) + (µ2 − µ1 ) λ + µ1 + β λ + µ2 + β = r1 − r2 + r2 (µ2 + β) − r1 (µ1 + β) λ + µ1 + β ≤ r1 − r2

y∈S

Admission control with batch arrivals

15

where the first inequality is due to the coupling, the third follows by uniformization and the induction hypotheses for Dn (01)(x) and Dn (21)(x), the last one is due to the assumption (λ + µ1 + β)r2 (µ2 + β) ≤ (λ + µ2 + β)r1 (µ1 + β) and the rest is some algebra. This proves the second part of the lemma. Now we can conclude that class-1 jobs are preferred under the following condition: Proposition 2. If (λ + µ1 + β)r2 (µ2 + β) ≤ (λ + µ2 + β)r1 (µ1 + β), then class 1 is the single preferred class. The condition specified in this proposition compares the average rewards of the two classes: A class-i job stays in the system for an exponential amount of time with rate µi + β, since it either finishes its service with rate µi or the system closes down with rate β. Thus, a class-i job with a lump reward of ri brings an average reward of ri (µi + β). Proposition 2 shows that when the average reward of class 1 (slow-service class) is higher than that of class 2, class 1 is preferred. In a queueing environment with two classes of jobs, this criterion would determine a preferred class, as shown in Harrison [6], but in a loss system it favors slower jobs: If both classes bring the same average reward per unit time, then the system will prefer the one who occupies a server for a longer time to guarantee that this server will generate a reward for a longer period. The difference between loss and queueing systems is due to the fact that a queueing system is not concerned with the “occupation” time of a server since all jobs can wait, i.e., no loss of work. On the other hand, class 2 (fast-service class) is not necessarily preferred if r1 (µ1 + β) < r2 (µ2 + β): Consider a firm which has to choose one from two jobs, one of which brings $1,000 profit each month for 12 months and the other with a profit of $1,200 per month for only 3 months. The possibility that the firm will have no job after 3 months works in favor of the longer duration job. Hence, class-2 jobs with higher average rewards can be rejected if µ1 ri , D(0k)(x) ≤ rk , D(0i)(x ) ≤ ri and D(0k)(x ) > rk . This shows a major change in the preference of the system with respect to the state, as an idle server is preferred over each class in different states, a situation which we never observe in our examples. In addition, a system receiving mixed batches, which satisfies D(0i)(x) ≤ ri for all x, does not have a preferred class if it contains two states x and x with D(ik)(x) > rk − ri and D(ik)(x ) ≤ rk − ri . Such a system accepts as many class-i jobs as possible when the incoming batch consists of only class i, while it rejects at least one class-i job(s) in the presence

¨ Ormeci & Burnetas

18

of jobs of the other class. In fact, all the systems we have observed as having no preferred class have satisfied D(0i)(x) ≤ ri for all x for both i = 1 and i = 2 while violating the condition D(ki)(x) ≤ ri − rk for some state(s) x. In other words, these systems accept as many jobs as possible from all one-class batches, whereas they prefer different classes in different states when the batches consist of both classes. This suggests that systems with “no preferred class” have a “very mild” change in their preferences of classes with respect to the state of the system. This subsection presents examples of systems receiving mixed batches, which numerically explore the optimal strategy in a large set of systems for which our results are inconclusive, i.e., when the parameters satisfy inequality (5). However, as the above discussion suggests, they also give insight for systems with one-class batches. Example 1 We consider a system with 5 servers over an infinite horizon, for which we set β = 1, µ1 = 1, r2 = 1, p10 = 1/6, p02 = 2.9/6, p13 = 1/3, p55 = 0.1/6, and let r1 change between 1.1 to 6.1, λ change between 1 to 31, and µ2 between 1.1 to 6.1, all of them with the increments of 0.1. This way, a total of 750,000 examples is created, 129,686 of which satisfy inequality (5). In all of these 129,686 examples, when a batch of j = (0, 2) arrives, optimal policy accepts as many class-2 jobs as possible. In 79,031 of them class 1 is rejected in at least one state. However, this does not mean that class 2 is preferred in all these systems, since there are two different kinds of optimal policies which give higher or “equal” priority to class 1: In the first case, which have been observed 6,094 times, class 1 is the preferred class, i.e., when a mixed batch arrives, the system first accepts as many class-1 jobs as possible, and only then, depending on the availability of the servers, accepts class-2 jobs. The second case provides examples with no preferred class: In 11,215 examples, an empty system receiving a batch of (5,5) chooses to move to states (4,1), (3,2), (2,3) or (1,4), rather than (5,0) or (0,5), which obviously violates D(ki)(x) ≤ ri − rk for all x for both i = 1 and i = 2. Note that all these systems satisfy D(0i)(x) ≤ ri for all x for both i = 1 and i = 2. Hence, when the arriving batch consists of only one class, namely when j = (1, 0) or j = (0, 2), optimal policies accept as many jobs as possible in all states. In the light of this example, the optimal policies of the inconclusive region can be grouped into three: One with class 1 as the preferred class, one with no preferred class, and finally one with class 2 as the preferred class. Let L1 be the first and L2 be the third term in (5). Optimal policies of systems generated in Example 1 lead to the following crude observations: Class 1 is preferred when the ratio r2 (µ2 + β)/(r1 (µ1 + β)) is close to L1 . As this ratio increases, we first

Admission control with batch arrivals

19

observe no preferred class, and then class 2 becomes preferred. When the ratio becomes large, class-1 jobs can be rejected in some states. However, the cut-off points for the changes in the optimal policies cannot be pinpointed. Our next example shows the relation between the optimal policies and the ratio r2 (µ2 + β)/(r1 (µ1 + β)) more precisely for a two-server system: Example 2 Consider a system with 2 servers over an infinite horizon, where we set parameters as β = 1, λ = 30, µ1 = 2, µ2 = 3, r1 = 1, p10 = 1/6, p01 = 1/3, and p22 = 1/2. If L1 is the first and L2 the third term in (5), then we have L1 = 34/33 and L2 = 4/3, so that if r2 varies between 17/22 and 1, the whole “inconclusive” region given in Corollary 1 will be explored. Figure 1 shows the optimal policies as r2 changes with increments of 0.1/22, with several exceptions which correspond to the changes in the structure of optimal policies. We plot whether it is optimal to admit an incoming class-i job or not in all states, which corresponds to the optimal number of jobs to be accepted when a batch of (1,0) or (0,1) arrives. Actions indicated as “accept none” refers to rejection of all jobs due to a full system. The plots also show the optimal action for state (0,0) when a batch of (2, 2) arrives, which can be interpreted as the “best” state for the empty system to move into when it has the option. Figure 1 does not include optimal actions in states (1,0) and (0,1) when a batch of (2,2) arrives, which can be described as follows: All of the actions try to reach a state that is closer to the “best” state, so that in case (a), it is optimal to accept one of the class-1 jobs from batch (2,2) in both states (1,0) and (0,1), and in cases (c), (d), (e) and (f), it is optimal to accept one of the class-2 jobs in these states, whereas case (b) accepts a class-1 job in (0,1), and a class-2 job in (1,0). For r2 = 17 class 1 is known to be preferred, which is also observed in the figure. Class 1 is still preferred when r2 = 17.02/22. However, the system with r2 = 17.03/22 does not have a preferred class due to the optimal actions in all states when a batch of j = (2, 2) arrives, e.g., it is optimal to accept one job of each class while rejecting one of each in state (0,0;2,2). Class 2 becomes preferred even within an increment of 0.01/22 = 4, 55 × 10−4 in r2 , since the system with r2 = 17.04/22 prefers class 2. Notice that jobs of both classes are always accepted in cases (a), (b) and (c) when the batches consist of only one class. Class 2 is preferred in all other cases, and the rejection region for class 1 grows as r2 increases. All examples of systems with no preferred class admit all jobs in one-class batches of either class. We have discussed that each class has its own inferiorities and superiorities, and apparently under certain conditions a combination of the two classes creates a good mix, yielding higher revenues in the long run. In other words, for systems with no preferred class, the two classes

¨ Ormeci & Burnetas

20

do not have a strict priority over each other, rather their overall qualities, determined by their service rates, µi ’s, and rewards, ri ’s, are compatible. This also provides a strong support for expecting that systems receiving mixed batches with no preferred class would have two preferred classes if the system were receiving only one-class batches. Therefore, we conjecture that systems receiving one-class batches always have at least one preferred class. In fact, systems with one-class batches are very similar to single-arrival systems in terms of the existence of a preferred class, so ¨ we refer to Ormeci et al. (2001) for a detailed discussion on the issue in single-arrival systems. We also observe that the structure of the optimal policies changes within a small increment in the parameters, which to some extent justifies the gap in our results regarding the existence of preferred class(es). Finally, we have three remarks on the relation between the number of servers and the arrival and service rates of jobs with preferred class(es). These remarks apply to both one-class and mixed batch systems. Remark 2. In all results of this section, the inequalities and the conditions on the parameters never include the number of servers. At first sight, this is surprising. However, it is not really so remarkable as all these results provide sufficient conditions for always accepting jobs of a class so that the main question is on the decisions when almost all servers, or more boldly all but one server, are busy. ¨ Remark 3. “Being preferred” also depends on the arrival stream of jobs: Ormeci et al. [14] present an example of a system with single arrivals, in which different classes are preferred with different sets of arrival rates for each class even though the service rates, µi ’s, and the rewards, ri ’s, are kept fixed. Remark 4. The two criteria we have considered, i.e., lump rewards and average rewards, fail to determine a preferred class correctly when µ1 c. Lemma 4 summarizes all useful, and also obvious, relations among these sets, which is presented without a proof since all the relations are very easy to verify.

Admission control with batch arrivals

29

Lemma 4. (i)

S 0 = S(x + e1 ; j) ∩ S(x + e2 ; j)

(ii)

S x = S(x; j)\S(x + e1 + e2 ; j)

(iii)

S x¯ = S(x + e1 + e2 ; j)\S(x; j)

(iv)

Sxx1 ⊆ S(x + e1 ; j) ∩ S(x; j)

(v)

Sxx1 ∩ S(x + e2 ; j) = Sxx1 ∩ S(x + e1 + e2 ; j) = ∅

(vi)

Sxx2 ⊆ S(x + e2 ; j) ∩ S(x; j)

(vii)

Sxx2 ∩ S(x + e1 ; j) = Sxx2 ∩ S(x + e1 + e2 ; j) = ∅

(viii)

Sxx¯1 ⊆ S(x + e1 ; j) ∩ S(x + e1 + e2 ; j)

(ix)

Sxx¯1 ∩ S(x + e2 ; j) = Sxx¯1 ∩ S(x; j) = ∅

(x)

Sxx¯2 ⊆ S(x + e2 ; j) ∩ S(x + e1 + e2 ; j)

(xi)

Sxx¯2 ∩ S(x + e1 ; j) = Sxx¯2 ∩ S(x; j) = ∅

Now we can present the proof of Lemma 3:

Proof of Lemma 3. u0 (x) = 0 for all x satisfies inequality (6). Assume that the statement is true for n. Then inequality (9) is satisfied for n:

un (x) − un (x + b1 e1 ) ≤ un (x + b2 e2 ) − un (x + b1 e1 + b2 e2 ),

(9)

as derived in section 4. We first show that v n ’s satisfy inequality (6). So let δ n be: δ n = v n (x; j) − v n (x + e2 ; j) − v n (x + e1 ; j) + v n (x + e1 + e2 ; j), and we will show that δ n ≤ 0 for all possible actions. I

I

I

I

Let y ∗ = (y1∗ , y2∗ ) = y n (x; j) and y ∗ = (y1∗ , y2∗ ) = y n (x + e1 + e2 ; j) so that y ∗ and y ∗ are the optimal states to go from states (x; j) and (x + e1 + e2 ; j), respectively. Now we differentiate cases due to possible actions: I

Case I: y ∗ ∈ S 0 and y ∗ ∈ S 0

¨ Ormeci & Burnetas

30 I

In this case, both y ∗ and y ∗ are reachable from states x + e1 and x + e2 since S 0 = S(x + e1 ; j) ∩ S(x + e2 ; j) by Lemma 4 part (i). Thus we let one of them go to state y ∗ and the other I

one to y ∗ . Then, the difference of immediate rewards in δn is 0, and we have: I

I

δ n ≤ un (y ∗ ) − un (y ∗ ) − un (y ∗ ) + un (y ∗ ) = 0, since v n ’s are optimal. I

Case II: y ∗ ∈ S 0 and y ∗ ∈ S x¯ I

Case II.1: y ∗ ∈ Sxx¯1 Since S 0 = S(x+e1 ; j)∩S(x+e2 ; j) by Lemma 4 part (i), y ∗ is a feasible action in state (x+e2 ; j), I

and because Sxx¯1 ⊆ S(x + e1 ; j) by Lemma 4 part (viii), y ∗ is reachable from state (x + e1 ; j). Then: I

I

δ n ≤ un (y ∗ ) − un (y ∗ ) − un (y ∗ ) + un (y ∗ ) = 0 I

Case II.2: y ∗ ∈ Sxx¯2 I

This is very similar to Case II.1. We only need to observe that y ∗ is feasible for state (x + e2 ; j) since Sxx¯2 ⊆ S(x + e2 ; j) by Lemma 4 part (x), and y ∗ is reachable from (x + e1 ; j) since S 0 ⊆ S(x + e1 ; j) by Lemma 4 part (i). I

Case II.3: y ∗ ∈ S•2 I

Since y ∗ = (x1 + j1 + 1, x2 + j2 + 1), it cannot be reached from either (x + e2 ; j) or (x + e1 ; j). I

Therefore we cannot use the same technique as above. We first observe that the state (y1∗ , y2∗ ) is I

reachable from state (x + e2 ; j) and (y1∗ , y2∗ ) is reachable from (x + e1 ; j). Also notice that both I

I

I

I

(y1∗ , y2∗ ) ∈ S and (y1∗ , y2∗ ) ∈ S, since y1∗ < y1∗ and y2∗ < y2∗ due to the case assumptions. Then: I

I

I

I

δ n ≤ un (y1∗ , y2∗ ) − un (y1∗ , y2∗ ) − un (y1∗ , y2∗ ) + un (y1∗ , y2∗ ) ≤ 0 I

where the first inequality is due to the optimality of v n ’s and feasibility of actions (y1∗ , y2∗ ) and I

I

I

(y1∗ , y2∗ ), and the second is due to inequality (9) with x = y ∗ , b1 = y1∗ − y1∗ and b2 = y2∗ − y2∗ . I

Case III: y ∗ ∈ S x and y ∗ ∈ S 0 We need to consider three cases; y ∗ ∈ Sxx1 , y ∗ ∈ Sxx2 and y ∗ = x, each of which is similar to the corresponding subcase of Case II, and so details are omitted. I

Case IV: y ∗ ∈ S x and y ∗ ∈ S x¯

Admission control with batch arrivals

31

For this case, we have 9 subcases, most of which are proven similarly to each other. We consider two cases in detail and mention the similarities of the others: I

I

Let y ∗ ∈ Sxx1 and y ∗ ∈ Sxx¯1 . Then y ∗ and y ∗ are both reachable from (x + e1 ; j) and not I

I

reachable from (x + e2 ; j). Now observe that (y1∗ , y2∗ ) is feasible for (x + e2 ; j) and (y1∗ , y2∗ ) is I

I

feasible for x +e1 . Moreover, (y1∗ , y2∗ ) ∈ S and (y1∗ , y2∗ ) ∈ S: Since y ∗ ∈ Sxx1 , x1 < y1∗ ≤ min{x1 + I

I

I

j1 , c − x1 } and y2∗ = x2 , and because y ∗ ∈ Sxx¯1 , y1∗ = x1 + j1 + 1 and x2 + 1 ≤ y2∗ < x2 + j2 + 1, I

so that yk∗ ≤ yk∗ for k = 1, 2. Then:

I

I

I

I

δ n ≤ un (y1∗ , y2∗ ) − un (y1∗ , y2∗ ) − un (y1∗ , y2∗ ) + un (y1∗ , y2∗ ) ≤ 0 I

I

by inequality (9) with x = y ∗ , b1 = y1∗ − y1∗ and b2 = y2∗ − y2∗ as in Case II.3. I

I

We will let (x + e2 ; j) move to state (y1∗ , y2∗ ) and x + e1 to (y1∗ , y2∗ ) in the following cases: I

I

I

I

y ∗ ∈ Sxx1 and y ∗ ∈ S•x¯ , y ∗ ∈ Sxx2 and y ∗ ∈ Sxx¯2 , y ∗ ∈ Sxx2 and y ∗ ∈ S•x¯ , y ∗ ∈ S•x and y ∗ ∈ Sxx¯1 , I

I

I

y ∗ ∈ S•x and y ∗ ∈ Sxx¯2 , y ∗ ∈ S•x and y ∗ ∈ S•x¯ . Notice that in all these cases either y ∗ or y ∗ is not reachable from both (x + e1 ; j) and (x + e2 ; j) or both states can be reached from only one

of them. Thus, we cannot cancel out the terms, instead we have to let one of these states move I

I

I

to (y1∗ , y2∗ ) and the other one to (y1∗ , y2∗ ). Moreover, it can be easily verified that yk∗ ≤ yk∗ for I

I

k = 1, 2 in all these cases so that (y1∗ , y2∗ ) ∈ S and (y1∗ , y2∗ ) ∈ S. I

I

If y ∗ ∈ Sxx1 and y ∗ ∈ Sxx¯2 , then y ∗ is reachable from (x + e1 ; j) and y ∗ is reachable from

(x + e2 ; j). Then by optimality of v n ’s:

I

I

I

I

δ n ≤ un (y1∗ , y2∗ ) − un (y1∗ , y2∗ ) − un (y1∗ , y2∗ ) + un (y1∗ , y2∗ ) = 0. I

I

Similarly, when y ∗ ∈ Sxx2 and y ∗ ∈ Sxx¯1 , y ∗ is reachable from (x + e2 ; j) and y ∗ is reachable from (x + e1 ; j). Thus for all possible cases, δ n ≤ 0. Now we can consider un+1 : There are four systems starting from four different states, x, x + e2 , x + e1 and x + e1 + e2 . We couple all these systems so that, except for the additional customers, they all behave in the same way. Moreover, we couple the class-1 customers in systems starting in states x + e1 and x + e1 + e2 , and we couple the class-2 customers in systems starting in states x + e2 and x + e1 + e2 . Then if the additional class-i job departs from the system starting in state x + ei , the additional class-i job in x + e1 + e2 also

¨ Ormeci & Burnetas

32

departs. Thus: un+1 (x) − un+1 (x + e2 ) − un+1 (x + e1 ) + un+1 (x + e1 + e2 )  = λ pj1 ,j2 [v n (x; j) − v n (x + e2 ; j) − v n (x + e1 ; j) + v n (x + e1 + e2 ; j)] j1 ,j2

+x1 µ1 +µ1 +x2 µ2 +µ2 +α

[un (x − e1 ) − un (x − e1 + e2 ) − un (x) + un (x + e2 )] [un (x) − un (x + e2 ) − un (x) + un (x + e2 )]

[un (x − e2 ) − un (x) − un (x + e1 − e2 ) + un (x + e1 )] [un (x) − un (x) − un (x + e1 ) + un (x + e1 )]

[un (x) − un (x + e2 ) − un (x + e1 ) + un (x + e1 + e2 )]

≤ 0 where α = cµ2 − (x1 + 1)µ1 − (x2 + 1)µ2 . The terms in the summation are less than or equal to

0 since v n ’s are shown to satisfy the inequality, the µ1 and µ2 terms are 0 and all other terms are non-positive by the induction hypothesis. Thus, the value functions, un , satisfy inequality (6) for all n whenever u0 does.

2

Admission control with batch arrivals

33

x1 0

1

2

3

4

0 30|01|13|05 30|11|13|14 30|21|23|23 30|31|32|32 40|41|41|41 1 21|02|04|05 21|12|14|14 21|22|23|23 31|32|32|32 x2 2 12|03|05|05 12|13|14|14 22|23|23|23 3 13|04|05|05 13|14|14|14 4 04|05|05|05 Table 1: Optimal admission policy for Example 3

34

¨ Ormeci & Burnetas

Figure 1: Illustration of optimal policies for the system in Example 1 with (a) r2 = 17/22 and r2 = 17.02/22, (b) r2 = 17.03/22, (c) r2 = 17.04/22, r2 = 17.1/22, ..., 19.2/22, and r2 = 19.21/22, (d) r2 = 19.22/22, ..., 19.24/22, (e) r2 = 19.25/22, and r2 = 19.3/22, ..., 20.0/22, and (f) r2 = 20.1/22, ..., 22.0/22.