An Ergodic Control Problem for Constrained

An Ergodic Control Problem for Constrained Diffusion Processes: Existence of Optimal Markov Control. Amarjit Budhiraja Department of Statistics University of North Carolina at Chapel Hill Chapel Hill, NC 27599 April 5, 2001 Abstract An ergodic control problem for a class of constrained diffusion processes is considered. The goal is the almost sure minimization of long term cost per unit time. The main result of the paper is that there exists an optimal Markov control for the considered problem. It is shown that under the assumption of regularity of the Skorohod map and appropriate conditions on the drift coefficient the class of controlled diffusion processes considered have strong, uniform in control, stability properties. The stability results on the class of controlled constrained diffusion processes considered in this work are non-trivial in that the domains are unbounded and the corresponding unconstrained diffusions are typically transient. These stability properties are key in obtaining appropriate tightness estimates. Once these estimates are available the remaining work lies in identifying weak limits of a certain family of occupation measures. In this regard an extension to the Echeverria-Weiss-Kurtz characterization of invariant measures of Markov processes, to the case of constrained-controlled processes considered in this paper, is proved. This characterization result is also crucially used in proving the compactness of the family of invariant measures of Markov processes corresponding to all possible Markov controls. Keywords: Ergodic control, Optimal Markov control, Controlled reflected diffusions, Constrained processes, Control of queuing networks, Patchwork martingale problem, Controlled martingale problem, Echeverria’s criterion.

1

1

Introduction

Constrained diffusion processes arise in a natural fashion in the heavy traffic analysis of queuing networks coming from problems in computer, communications and manufacturing systems. The problem of control of such queuing systems is of great current interest (cf. [24], [13], [17], [12], [25]). Excepting a few special cases, the control problem for queuing networks is quite difficult to analyze directly and thus one tries to find more tractable approximations. In that respect, diffusion approximations obtained via appropriate scaling limits become very attractive since in the limit many fine details are eliminated and usually the only parameters remaining are the means, variances of the various processes and the mean routing structure. Because of the simple structure, the limit problem is considerably easier to solve. Once the optimal solution to the control problem for the constrained diffusions is obtained, one can then approximate the properties and suggest good policies for the actual physical system. Thus the study of constrained controlled diffusion processes is one of the central objectives in the optimal control of queuing networks. In this work we consider a control problem for a class of diffusion processes which are constrained to lie in a polyhedral cone G with a vertex at origin. The domain G ⊂ IRk is given as an intersection of N half spaces Gi ; i = 1, · · · N . Associated with each Gi are two vectors: the first vector, denoted as ni , represents the inward normal to Gi while the second, denoted as di , gives the ”direction of constraint”. Roughly speaking, the constrained version of a given unrestricted trajectory in IRk is obtained by pushing back the trajectory, whenever it is about to exit the domain, in a prespecified direction of constraint using the minimal force required to keep the trajectory within the domain. Precise definitions will be given in Section 2. Constraining mechanism is described via the Skorohod map, denoted as Γ(·) which takes an unrestricted trajectory ψ(·) and maps it . to a trajectory φ(·) = Γ(ψ)(·) such that φ(t) ∈ G for all t ∈ (0, ∞). Under appropriate conditions on (di , ni )N i=1 it follows from the results in [7] that the Skorohod map is well defined and it enjoys a rather strong regularity property (See Theorem 2.3.). The controlled constrained diffusion process that we consider in this paper are obtained as a solution to the equation:   Z · Z · X(t) = Γ X(0) + b(X(s), u(s))ds + σ(X(s))dW (s) (t); t ∈ [0, ∞), 0

0

(1.1) where W (·) is a standard Wiener process, b : G × U → IRk ; σ : G → IRk×k are suitable coefficients, U is a given control set and u(·) is a U valued ”admissible” control process. The control problem that we study is concerned with the ergodic cost criterion: Z 1 T lim sup k(X(s), u(s))ds, (1.2) T →∞ T 0 1

where the limit above is taken almost surely and k : G × U → IR is a suitable map. The two key objectives of the controller are: first, to choose a control, in a non-anticipative fashion , which minimizes the cost in (1.2) and second, to obtain a control which is ”easy” to implement. As regards the second goal, one of the most desirable feature of a good control is that the control depend only on the current value of the state and not on the whole history of the state and/or the control process. In other words, we are seeking controls u(·) such that there exists some measurable map v : G → U satisfying u(t) = v(X(t)), a.s. for all t ∈ [0, ∞). Under such a control the solution to (1.1) becomes a Markov process and for this reason the map v(·) is referred to as a ”Markov control”. The objective of this work is to show that, under appropriate conditions on the model, (cf. Conditions 2.2, 2.4, 2.5, 3.1) there is a Markov control which minimizes the cost in (1.2). The ergodic control problem for unconstrained diffusions is one of the classical problems in stochastic control. The problem has been studied extensively in [32, 35, 23, 4, 3, 33, 15, 22]. The approach taken in the present paper has been inspired by the techniques and results in [4]. For ergodic control results on constrained jump-diffusion processes in bounded domains and expected long term cost per unit time criterion, we refer the reader to [2, 26, 24]. For the case of constrained diffusions in unbounded domains, we are not aware of any results which give the existence of optimal Markov control under the ergodic cost criterion considered in this paper. As is classical in an ergodic control problem of the above form (cf. [3], [33], [24], [22]) the problem of existence of optimal Markov controls under the cost criterion in (1.2) is closely related to certain stability properties of the solutions to (1.1). In a recent work [1], which dealt with the case of uncontrolled constrained diffusions, various stability properties of the solution to   Z · Z · ξ(t) = Γ ξ(0) + β(ξ(s))ds + σ(ξ(s))dW (s) (t); t ∈ [0, ∞), (1.3) 0

0

where β : IRk → IRk is an appropriate drift vector, were obtained. In particular it was shown that if for all x ∈ G, β(x) lies in a certain cone C (see (3.2)) and its distance from the boundary of the cone is uniformly bounded below by a positive constant then the constrained diffusion is positive recurrent and has a unique invariant measure. The above results identify an important, non-trivial class of ergodic constrained diffusions in unbounded domains, in the sense that the corresponding unconstrained version of these processes would typically be transient. To see this one only needs to consider the case where b(x, u) ≡ b, where b is some fixed vector in C 0 . For this latter case (b(x, u) ≡ b), the cone, in fact, provides a necessary and sufficient condition for positive recurrence of constrained diffusions, i.e. if b 6∈ C then the corresponding constrained diffusion is transient[5]. In the context of the constant drift case, this necessary and sufficient condition for positive recurrence, for constrained diffusions which correspond to single class networks, was first proved in [14]. 2

The estimates used in the study of the stability properties of the uncontrolled system in (1.3) can be used for the controlled problem studied in this paper as well. Using these estimates, we will obtain rather strong (uniform in control) stability properties for the processes obtained as solutions to (1.1) (See Section 6). These stability results are then used in various tightness arguments in this paper. As another consequence of results in [1] we have that with appropriate assumptions on the drift vector b(·, ·), under any Markov control v(·), the solution to (1.1) is positive recurrent and has a unique invariant measure, denoted as ηv . We abbreviate this statement by saying that ”all Markov controls are stable”. Using this fact and an ergodic theorem of Khasminskii[18] it will follow that under a Markov control v(·) the limit in (1.2) is almost surely equal to: Z k(x, v(x))ηv (dx). (1.4) G

The next key step in the program is to show that for any admissible control u(·) the limit in (1.2) can be expressed in the form (1.4) for some measurable v : G → U which may depend on ω (the parameter of randomness), the control u(·) and other data. This is done in Proposition 7.5. As an immediate consequence of this step it follows that Z Z 1 T k(X(s), u(s))ds ≥ inf k(x, v(x))ηv (dx), (1.5) lim sup v T →∞ T 0 G a.s., where the infimum on the right side above is taken over all measurable maps v : G → U . In order to prove the above step we need a characterization result for the invariant measures of solutions of (1.1) (with u(·) = v(X(·)) for some measurable v(·)). The characterization result, which extends similar results due to Echeverria [10], Weiss [36], Kurtz [21], to the case of controlled constrained diffusions considered in this paper is proved in Section 5. The proof uses the ideas of patchwork and constrained martingale problems introduced by Kurtz [20, 21]. The key idea, as in [21], is to show that if (5.2) holds then there is a sequence of solutions to certain patchwork controlled martingale problems (see Section 5 for definitions) which converge to a stationary solution of (1.1). We then use the results on existence of optimal Markov controls (for unconstrained processes) obtained in [22] and the Feller properties of solutions of (1.1) (using a Markov control) to conclude that for this stationary solution u(·) = v(X(·)) for some v : G → U . Once this characterization result is available we can identify the limit in (1.2) as an integral with respect to the invariant measure ηv for some v : G → U . As a final step we need to show that the infimum on the right side of (1.5) is attained for some Markov control v : G → U . This step, (Proposition 7.4) combined with the ergodic theorem in [18] yields our main theorem: Theorem 3.4. The fact that the infimum is attained, is a consequence of the fact that the family {ηv : v is a Markov control} is compact. The proof of the compactness of this family once more uses the characterization result for the invariant 3

measures studied in Section 5 and various tightness estimates which follow from the stability properties of our controlled diffusions. This is done in Lemma 7.3. Observe that since (1.5) holds almost surely, we can replace the left side of the expression by: Z 1 T k(X(s), u(s))ds. ess inf lim sup T →∞ T 0 Thus our main result says that there is a Markov control for which the cost, for almost every realization, is no worse than that for the (essentially) best possible realization corresponding to any other control. In this paper we do not address the problem of construction of the optimal Markov control or the approximation of optimal control for physical queuing systems using the optimal Markov control for the limit diffusion. These questions will be addressed in a future work. The paper is organized as follows. In Section 2 we present the basic definitions and properties of the Skorohod map. We also introduce the main assumptions on our problem data which assure the regularity of the Skorohod map. We then define the class of constrained controlled diffusions considered in the work. The study of Markov controls forces us to consider the solution of (1.1) in a weak sense. This weak formulation is also introduced in the section. Section 3 introduces the ergodic cost problem that interests us in this work. We give our key condition on the drift vector which assures that all Markov controls are stable. Finally in this section we state our main result: Theorem 3.4. Section 4 is an assortment of some background results used in the proof of our main theorem. We state the ergodic theorem of Khasminskii which permits us to write the limit in (1.2) corresponding to a Markov control v(·) as the integral in (1.4). We also present a very useful result from [6] which enables us to control the growth and obtain the tightness of the reflection terms in our constrained diffusions. We then show that we can assume without loss of generality that all our control processes u(·) are adapted to the filtration generated by the state process and the reflection terms. This result is then used to show that certain conditional laws of the constrained diffusions in (1.1) are almost surely the same as the law of some other constrained diffusion of same form as (1.1) but with possibly a different control process. This result is key to many of the uniform estimates and tightness results in Sections 6 and 7. The last two results, for the case of unconstrained diffusions, are proved in [3], Chapter I and the proofs for the constrained case are similar. However, for the sake of completeness we provide the proofs of these results. In Section 5 we present our extension of the Echeverria-Weiss-Kurtz criterion for the invariant measures. We define the patchwork and constrained controlled martingale problems of Kurtz and study some of their basic properties. The main result of this section is Theorem 5.7. Section 6 is devoted to obtaining stability properties of our constrained diffusions. This section crucially uses some estimates derived in [1] (cf. Lemmas 6.1 and 6.2). As consequences of these results we prove strong, uniform in control,

4

tightness properties of the solutions of (1.1). Finally, in Section 7 we present the proof of Theorem 3.4. In the Appendix of this paper we provide an alternate proof of Lemma 4.3 of [1] which, unlike the proof in [1], does not appeal to the Markov property of ξ(·). This lemma is needed for the proof of Theorem 4.4 of [1] (See Remark 6.3). and thus in the proof of Lemma 6.2 of the present paper.

2

Skorohod Map and Controlled-Constrained Diffusions

Let G ⊂ IRk be a polyhedral cone in IRk with the vertex at origin given as the intersection of half spaces Gi , i = 1, · · · , N . Each half space Gi is associated with a unit vector ni via the relation Gi = {x ∈ IRk : hx, ni i ≥ 0}, where h·, ·i denotes the usual inner product in IRk . Denote the boundary of a set B ⊂ IRk by ∂B. We will denote the set {x ∈ ∂G : hx, ni i = 0} by Fi . For x ∈ ∂G, define the set, n(x), of inward normals to G at x by . n(x) = {r : |r| = 1, hr, x − yi ≤ 0, ∀y ∈ G}. With each face Fi we associate a unit vector di such that hdi , ni i > 0. This vector defines the direction of constraint associated with the face Fi . For x ∈ ∂G define     X . d(x) = d ∈ IRk : d = αi di ; αi ≥ 0; ||d|| = 1 ,   i∈In(x) where

. In(x) = {i ∈ {1, 2, · · · N } : hx, ni i = 0}.

We will denote the collection of all subsets of {1, · · · N } by Λ. Also for λ ∈ Λ . we will define Fλ = ∩i∈λ Fi . As a convention we will take F∅ as G. Let D([0, ∞) : IRk ) denote the set of functions mapping [0, ∞) to IRk that are right continuous and have limits from the left. We endow D([0, ∞) : IRk ) with the usual Skorohod topology. Let . DG ([0, ∞) : IRk ) = {ψ ∈ D([0, ∞) : IRk ) : ψ(0) ∈ G}. For η ∈ D([0, ∞) : IRk ) let |η|(T ) denote the total variation of η on [0, T ] with respect to the Euclidean norm on IRk . Definition 2.1 Let ψ ∈ DG ([0, ∞) : IRk ) be given. Then (φ, η) ∈ D([0, ∞) : IRk ) × D([0, ∞) : IRk ) solves the Skorohod problem (SP) for ψ with respect to G and d if and only if φ(0) = ψ(0), and for all t ∈ [0, ∞) 5

1. φ(t) = ψ(t) + η(t); 2. φ(t) ∈ G; 3. |η|(t) < ∞; Z 4. |η|(t) = I{φ(s)∈∂G} d|η|(s); [0,t]

5. There exists (Borel) measurable γ : [0, ∞) → IRk such that γ(t) ∈ d(φ(t)) (d|η|-almost everywhere) and Z η(t) = γ(s)d|η|(s). [0,t]

On the domain D ⊂ DG ([0, ∞) : IRk ) on which there is a unique solution to the . Skorohod problem we define the Skorohod map (SM) Γ as Γ(ψ) = φ, if (φ, ψ −φ) is the unique solution of the Skorohod problem posed by ψ. We will make the following assumptions on the data defining the Skorohod problem above. Condition 2.2 (a) There exists a compact, convex set B ∈ IRk with 0 ∈ B 0 , such that if v(z) denotes the set of inward normals to B at z ∈ ∂B, then for i = 1, 2, · · · , N , z ∈ ∂B and |hz, ni i| < 1 implies that hv, di i = 0 for all v ∈ v(z) . (b) There exists a map π : IRk → G such that if y ∈ G, then π(y) = y, and if y 6∈ G, then π(y) ∈ ∂G, and y − π(y) = αγ for some α ≤ 0 and γ ∈ d(π(y)). (c) For every x ∈ ∂G, there is n ∈ n(x) such that hd, ni > 0 for all d ∈ d(x). The above assumptions can be verified for a rich class of problems arising from queuing networks. For example, in the seminal work [11], it was shown that the above properties hold for Skorohod problems associated with open, single class queuing networks (cf. [8]). Other classes of network examples for which the above properties hold are in [7], [8], [9], [28], [29]. Condition (c) above is equivalent to the assumption that the N × N matrix with the (i, j)-th entry hdi , nj i is completely - S (cf. [30], [7]). The following result is taken from [7]. Theorem 2.3 ([7]) Under Condition 2.2 the Skorohod map is well defined on all of DG ([0, ∞) : IRk ), i.e., D = DG ([0, ∞) : IRk ) and the SM is Lipschitz continuous in the following sense. There exists a K < ∞ such that for all φ1 , φ2 ∈ DG ([0, ∞) : IRk ): sup |Γ(φ1 )(t) − Γ(φ2 )(t)| < K sup |φ1 (t) − φ2 (t)|. 0≤t 0 and 0 < t0 < t1 < ∞ there exists an  > 0 such that for all A ∈ B(G) with λ(A) ≤ , where λ denotes the Lebesgue measure on G, P (t, x, A) ≤ δ, x ∈ G, t ∈ [t0 , t1 ]. Finally for any  > 0, 0 < t0 < t1 < ∞ and compact set K0 ⊂ G there is a δ > 0 such that for all x ∈ K0 , t ∈ [t0 , t1 ] and A ∈ B(G) with λ(A ∩ K0 ) ≥  we have that P (t, x, A) ≥ δ.

3

The Ergodic Cost Problem

In this work we are interested in a control problem with an ergodic cost criterion. Namely, we are interested in minimizing, over the class of all admissible controls, the cost: Z 1 t lim sup k(X(s), u(s))ds, (3.1) t→∞ t 0 where X(·) is given as a solution of (2.2) on some filtered probability space with an admissible pair (u(·), W (·)), the limit above is taken almost surely on the corresponding probability space and k : G × U → IR is a map defined as follows. For (x, u) ∈ G × U Z . k(x, α)u(dα), k(x, u) = S

where k is in Cb (G × S). 9

We will call a Markov control v a stable Markov control (SMC) if the corresponding controlled Markov process {Pxv }x∈G is positive recurrent and has a unique invariant measure. We are interested in obtaining conditions under which there is an optimal SMC for the cost criterion (3.1). The following stability assumption on the underlying model will be assumed throughout this paper. Define ( N ) X . C= − αi di : αi ≥ 0; i ∈ {1, · · · , N } . (3.2) i=1

The cone C was used to characterize stability of a certain class of constrained diffusion processes in [5, 1]. Let δ ∈ (0, ∞) be fixed. Define the set . C(δ) = {v ∈ C : dist(v, ∂C) ≥ δ}. Our next assumption, which also will be assumed throughout this paper, on the diffusion model stipulates the permissible drifts in the underlying diffusion. Condition 3.1 There exist a δ ∈ (0, ∞) such that for all (x, u) ∈ G × U , b(x, u) ∈ C(δ). Under the assumptions made above the results of [1] show that all Markov controls are SMC, more precisely: Theorem 3.2 The Markov family {Pxv }x∈G of Theorem 2.9 is positive recurrent and admits a unique invariant measure, denoted as ηv . Remark 3.3 In [1] the proof of positive recurrence assumes that the drift coefficient in the constrained diffusion process satisfies a Lipschitz condition, however, as is pointed out in Remark 4.6 of that paper, the same proof continues to hold with the assumptions on the coefficients made in this paper. Now we are able to state the main result of this paper. Theorem 3.4 There exists a Markov control v(·) such that if for some µ ∈ P(G), X(·) is the corresponding process solving (2.4), on some filtered probability space, with the probability law of X(0) being µ then: lim sup T →∞

1 T

Z

T

k(X(s), v(X(s)))ds = inf ess inf lim sup T →∞

0

1 T

Z

T

k(X(s), u(s))ds, 0

(3.3) a.s., where the outside infimum on the right side above is taken over all controlled processes X(·) with an arbitrary initial distribution and solving (2.2) over some filtered probability space with some admissible pair (W (·), u(·)). The proof of the above theorem will be given in Section 7.

10

4

Some Background Results:

In this section we collect some background results which will be used in the proof of Theorem 3.4. We begin with an ergodic theorem of Khasminskii [18] which is applicable to the Markov family {Pxv } considered in this paper because of Theorem 2.9. As a consequence of this result the limiting time averages on the left side of (3.3) can be replaced with expectations with respect to the measure ηv . Lemma 4.1 [Khasminskii [18] Theorem 3.1] For a given µ ∈ P(G) and a Markov control v let X(·) be the process on some filtered probability space solving (2.4) with the distribution of X(0) being µ. Then for all ηv integrable functions RT R g on G: T1 0 g(X(s))ds converges almost surely to G g(x)ηv (dx). The following lemma has been proved in [6], however the domain G there is different from our problem. Thus for the sake of completeness we sketch the proof below. This lemma will be used several times in this paper in controlling the reflection term Y (·) in our constrained diffusion processes. Lemma 4.2 There exists a g ∈ Cb2 (G) such that h∇g(x), di i ≥ 1; ∀ x ∈ Fi ; i ∈ {1, · · · N }.

(4.1)

Proof: We begin by observing that the geometry of the space G implies that there exists C ∈ (1, ∞) such that ∀x ∈ G and λ ∈ Λ d(x, Fλ ) ≤ C maxhx, ni i. i∈λ

Next, from Condition 2.2(c), itPfollows that ∀λ ∈ Λ, there exist positive con. stants {cλi }i∈λ such that η λ = i∈λ cλi ni satisfies hη λ , di i > 0; ∀ i ∈ λ. . λ Define λ∈Λ hη , di i. Furthermore as a convenient normalization we Pc˜ = λinf i∈λ; 1 take i∈λ ci = 2 . Next define constants (γk , βk ); k = 0, 1, · · · N inductively as follows: 1 . . γN = ; βN = c˜γN 2(C + 1) and for k = 1, · · · N . βN −k+1 . γN −k = ; βN −k = c˜γN −k , C Let φ, ψ be maps from IR+ to IR+ defined as follows: 1 . φ(x) = x − 1; x ∈ [0, ] 2 . = 0; x ≥ 1 11

and 1 . ψ(x) = 0; x ∈ [0, ] 2 . = 1; x ≥ 1. Now define for λ ∈ Λ; fλ : G → IR as follows:     λ hnj , xi hη , xi Y . ψ , fλ (x) = aλ φ β|λ| γ|λ| j6∈λ

where |λ| denotes the cardinality of the set λ, aλ are suitable positive chosen inductively as follows. For all λ with |λ| = 1 we choose aλ so that aλ hηλ , di i ≥ β1 ; i ∈ λ. Having chosen aλ for all λ with 1 ≤ |λ| < k, we choose aλ , for a λ ∈ Λ with |λ| = k, such that aλ hηλ , di i ≥ βk (Mk−1 + 1); i ∈ λ, where  k−1 . X X = −

Mk−1

j=1 λ:|λ|=j

+ h∇fλ (x), di i .

inf

x∈Fi ;i∈λ

Finally define g : G → IR as: . X g(x) = fλ (x). λ∈Λ

It can be verified as in [6] that g defined as above satisfies (4.1). The following lemma essentially says that in considering admissible controls, we can without loss of generality restrict ourselves to controls that are adapted with respect to the filtration generated by (X(·), Y (·)). Lemma 4.3 Let (Ω, F, {Ft }, P ) be a filtered probability space on which is given an admissible pair (u(·), W (·)). Let X(·) be a solution to (2.2) with the corresponding boundary processes {Yi (·)}N i=1 . Then there exists a enlargement (Ω, F, {F t }, P ) of the above probability space on which is given a {F t } Wiener ˜ (·) and P(S) valued measurable stochastic process u process W ˜(·) such that for X,Y X,Y a.e. t ∈ [0, ∞) u ˜(t) is Ft measurable, where Ft denotes the P completion of σ{X(s); {Yi (s)}N i=1 ; 0 ≤ s ≤ t}, and X(·) solves Z X(t) = X(0) +

t

Z b(X(s), u ˜(s))ds +

0

0

12

t

˜ (s) + σ(X(s))dW

N X i=1

di Yi (t).

Proof: Let {fi (·)} be a countable dense set in Cb (S). Then there exists a P(S) valued measurable stochastic process u ˜(·) (cf. Theorem 2.7.1 [16]) satisfying: Z  Z fi (α)˜ u(t)(dα) = E fi (α)˜ u(t)(dα) | FtX,Y , a.e. t ∈ [0, ∞); a.s., S

S

i = 1, 2, · · ·. Noting that X(·) − X(0) −

N X

di Yi (·)

i=1

is an Itˆ o process given as: Z · Z · b(X(s), u(s))ds + σ(X(s))dW (s) 0

0

we have from Theorem 4.2 of Wong [37] that there is a IRk valued measurable ˆ and a k-dimensional Brownian motion on some augmented space process φ(·) such that the following representation holds a.s. Z X(t) = X(0) + 0

t

ˆ φ(s)ds +

Z

t

0

˜ (s) + σ(X(s))dW

N X

di Yi (t),

i=1

and a.e. ω
ˆ ω) = E b(X(s), u(s)) | F X,Y ; a.e. s ∈ [0, ∞). φ(s, s The result now follows on observing that for a.e. s ∈ [0, ∞)
E b(X(s), u(s)) | FsX,Y = b(X(s), u ˜(s)), a.s. The following lemma will be used in some conditioning arguments in the proofs of Lemma 6.7 and Proposition 7.5. For a Polish space K, denote by C([0, ∞) : K) the space of continuous functions from [0, ∞) to K, endowed with the topology of uniform convergence on compacts. Lemma 4.4 Let (Ω, F, {Ft }, P ) be a filtered probability space. Let (u(·), W (·)) be an admissible pair on this probability space. Let X(·) be given as a solution of (2.2) and let τ be an a.s. finite {Ft } stopping time. Denote the conditional distribution of X(τ + ·) given Fτ by π(ω)(·), i.e. for A ∈ B(C([0, ∞) : G)) and a.e. ω P (X(τ + ·) ∈ A | Fτ ) (ω) = π(ω)(A). Then, for a.e. ω, π(ω) equals the probability law of X ω (·), where X ω (·) solves an equation of the form (2.2) with (u(·), W (·)) replaced by some other admissible pair: (uω (·), W ω (·)) given on some filtered probability space (Ωω , F ω , {Ftω }, P ω ) and X ω (0) = X(τ (ω)). 13

. Proof: Let Y (·) = (Y1 (·), · · · , YN (·)), where {Yi (·)}N i=1 is the boundary process for X(·). By virtue of Lemma 4.3 we can assume without loss of generality that u(·) is {FtX,Y } adapted. Thus there is a measurable map (cf. Theorem 2.7.2 [16]): f : [0, ∞) × Ω1 × Ω2 → U, . . where Ω1 = C([0, ∞) : G) and Ω2 = C([0, ∞) : [0, ∞)N ) such that for a.e. t ∈ [0, ∞), [P ] a.e. ω u(t, ω) = f (t, X t (·, ω), Y t (·, ω)). where . X t (s, ω) = X(s, ω); 0 ≤ s ≤ t . = X(t, ω); s ≥ t . and Y t (·, ω) is defined in a similar manner. Let Ω3 = C([0, ∞) : IRk ). Endow . Ω = Ω1 × Ω2 × Ω3 with the usual product Borel σ - field, denoted as F and let {F t }t≥0 denote the canonical filtration. Consider the probability measure P ω on (Ω, F) which is the conditional probability law of (X(τ + ·), Y (τ + ·), W (τ + ·) − W (τ )) given Fτ . Complete the filtration {F t } with respect to P ω and denote the completed filtration by the same symbol. Let (X 0 (·), Y 0 (·), W 0 (·)) be the canonical coordinate process on Ω and let X ω (·) be the process defined on the filtered probability space (Ω, F, {F t }, P ω ) as a solution of:   Z · Z · 0 X ω (t) = Γ X ω (0) + b(X ω (s), uω (s))ds + σ(X ω (s))dW (s) (t) 0

X ω (0)

0

= X(τ (ω), ω),

where for t ≥ 0 and [P ω ]-a.e ω (ω denotes a typical element of Ω) . ˜ ω (·, ω), Y˜ω (·, ω)), uω (t, ω) = f (τ (ω) + t, X ˜ ω (·) and Y˜ω (·) in the above control process are and for ω ∈ Ω, the processes X defined as: ˜ ω (t, ω) X

= X(t, ω); t ∈ [0, τ (ω)] = X 0 (t − τ (ω), ω); t ≥ τ (ω)

Y˜ω (t, ω)

= Y (t, ω); t ∈ [0, τ (ω)] = Y 0 (t − τ (ω), ω); t ≥ τ (ω).

and

Then it can be shown following the proof of Theorem 6.1.2 of [34] that for [P ]a.e. ω the probability law of X ω (·) equals Π(ω). This completes the proof of the lemma on identifying (Ωω , F ω , {Ftω }, W ω (·)) with (Ω, F, {F t }, W 0 (·)). 14

5

Characterization of the Invariant Measure:

One of the key ingredients of the proof of Theorem 3.4 is an extension of Echeverria-Weiss characterization of invariant measures (cf. [10], [36]) to the class of constrained controlled Markov processes considered in this paper. The proof of this characterization uses a clever idea presented in the proof of a similar characterization result for constrained (uncontrolled) Markov processes in Kurtz [21]. It also uses ideas from [22]. We begin with the following definitions. For f ∈ Cb2 (G) let Lf : G × U → IR be defined as: k k X ∂2f ∂f . 1 X ai,j (x) (x) + bi (x, u) (x); (x, u) ∈ G × U, (Lf )(x, u) = 2 i,j=1 ∂xi ∂xj ∂x i i=1

. where aij (x) = σ(x)σ T (x). With an abuse of notation we will write for α ∈ S, (Lf )(x, δ{α} ), merely as (Lf )(x, α), where δ{α} denotes the probability measure concentrated at the point α. Thus with this notation, for (x, u) ∈ G × U , Z (Lf )(x, u) = (Lf )(x, α)u(dα). S

For i = 1, 2, · · · N and f ∈

Cb2 (G)

let Di f : G → IR be defined as: . (Di f )(x) = hdi , ∇f (x)i, x ∈ G.

Definition 5.1 (Constrained-Controlled Martingale Problem (CCMP)) For µ ∈ P(G) a solution to the (µ, L, G, (Di , Fi )N i=1 ) CCMP is a pair of {Ft } adapted processes (Z(·), Φ(·)) on some filtered probability space (Ω, F, {Ft }, P ) such that the following hold. (i) Z(·) is a G valued process with, almost surely, continuous trajectories. (ii) Z(0) has probability law µ. (iii) Φ(·) is a U valued, measurable and {Ft } -adapted process. (iv) There is an {Ft }-adapted, N -dimensional ”boundary ” process Y (·) = (Y1 (·), · · · YN (·)) such that for each i ∈ {1, 2, · · · N }, P − almost surely (a) Yi (0) = 0. (b) Yi (·) is continuous and non-decreasing. Rt (c) for all t ∈ (0, ∞), 0 IFi (Z(s))dYi (s) = Yi (t). (v) For all f ∈ C0∞ (G) Z tZ f (Z(t)) −

(Lf )(Z(s), α)Φ(s)(dα)ds − 0

S

N Z X i=1

is a Ft martingale. 15

0

t

(Di f )(Z(s))dYi (s)

The proof of the following result is standard and thus is omitted (cf. Theorem 4.5.2 [34]). Theorem 5.2 Let (Z(·), Φ(·)) be a solution of the (µ, L, G, (Di , Fi )N i=1 ) CCMP on some filtered probability space (Ω, F, {Ft }, P ). Then there exists an enlargement (Ω, F, {F t }, P ) of the above space such that: (i) there is a F t − Wiener process W (·) defined on the enlarged space, (ii) the processes (Z(·), Φ(·)) are measurable and F t adapted, (iii) for all t ≥ 0, a.s.   Z tZ Z t Z(t) = Γ Z(0) + b(Z(s), α)Φ(s)(dα)ds + σ(Z(s))dW (s) . 0

S

0

(5.1) Conversely, if there is a pair of processes (Z(·), Φ(·)) solving (5.1) on some filtered probability space (Ω, F, {F t }, P ) satisfying (i), (ii) and (iii) above then the pair is a solution of the (µ, L, G, (Di , Fi )N i=1 ) CCMP where µ is the probability law of Z(0). A solution to the CCMP is closely related to the following Patchwork Controlled Martingale Problem (PCMP) introduced in the context of uncontrolled constrained processes by Kurtz in [20]. Definition 5.3 For µ ∈ P(G) a solution to the (µ, L, G, (Di , Fi )N i=1 ) PCMP is an {Ft } adapted vector stochastic process (ξ(·), Λ(·), λ0 (·), · · · , λN (·)), with N +1 values in G × U × IR+ on some filtered probability space (Ω, F, {Ft }, P ) such that the following hold. (i) ξ(·) has continuous trajectories almost surely. (ii) ξ(0) has probability law µ. (iii) Λ(·) is a U valued, measurable, {Ft } adapted process. (iv) For all i ∈ {0, 1, 2, · · · N }, P − almost surely (a) λi (0) = 0. (b) λi (·) is continuous and non-decreasing. Rt . (c) for all t ∈ [0, ∞), 0 IFi (ξ(s))dλi (s) = λi (t), where we define F0 = G. Pm (iv) For all t ≥ 0, i=0 λi (t) = t, a.s. (v) For all f ∈ C0∞ (G) Z tZ N Z t X f (ξ(t)) − (Lf )(ξ(s), α)Λ(s)(dα)dλ0 (s) − (Di f )(Z(s))dλi (s) 0

S

i=1

is a Ft martingale. 16

0

The proof of the following result is similar to the proof of Lemma 3.1 of [6] except that instead of condition (S.a) and (S.b) of [6] we use Condition 2.2(c). Lemma 5.4 Suppose that (ξ(·), Λ(·), λ0 (·), · · · , λN (·)) is a solution of the (µ, L, G, (Di , Fi )N i=1 ) PCMP on some filtered probability space (Ω, F, {Ft }, P ). Then, almost surely, λ0 (·) is a strictly increasing process such that λ0 (t) → ∞ as t → ∞. The following proposition establishes the connection between a solution of a CCMP and a solution of the PCMP. For the proof of the proposition we refer the reader to Theorem 3.4 of [6]. Proposition 5.5 Suppose that (ξ(·), Λ(·), λ0 (·), · · · , λN (·)) is a solution of the (µ, L, G, (Di , Fi )N i=1 ) PCMP on some filtered probability space . . (Ω, F, {Ft }, P ). Then if for t ≥ 0 τ (t) = inf{s ≥ 0 : λ0 (s) ≥ t}, Gt = Fτ (t) , . . . Z(t) = ξ(τ (t)), Φ(t) = Λ(τ (t)) and for i = 1, · · · N Yi (t) = λi (τ (t)) then (Z(·), Φ(·)) is the solution of the (µ, L, G, (Di , Fi )N i=1 ) CCMP on (Ω, F, {Gt }, P ) with the corresponding boundary processes {Yi (·)}N i=1 . The following lemma is the first step in the characterization of the invariant measure for the family {Pxv } of Theorem 2.9. For a measurable space (Ω, F) we denote by MF (Ω) the space of all finite, possibly identically zero, measures on (Ω, F). Also, denoting by 0 the identically . 0 measure, we let M(Ω) = MF (K)\0. For µ ∈ M(Ω) we denote its normalized . µ(·) version by µ ˆ, i.e. µ ˆ(·) = µ(Ω) . For a measurable map v : G → U , x ∈ G and B ∈ B(S) we will sometimes write v(x)(B) as v(x, B). Lemma 5.6 Let v : G → U be a measurable map. Let ηv be as in Theorem 3.2. Then there exist measures µi ∈ MF (Fi ) such that for all f ∈ C0∞ (G): Z N Z X (Lf )(x, α)µ0 (dx, dα) + (Di f )(x)µi (dx) = 0, (5.2) G×S

i=1

Fi

where µ0 ∈ P(G × S) is given as Z . µ0 (A × B) = v(x, B)ηv (dx), A ∈ B(G), B ∈ B(S). A

Proof: Let X(·) be a solution of (2.4) with X(0) ∼ ηv on some filtered probability space. Then X(·) is a stationary process. From Remark 2.7 there exist continuous increasing adapted processes Yi (·); i = 1, · · · N such that (2.3) holds with u(·) replaced by v(X(·)) . Let g ∈ Cb2 (G) be as in Lemma 4.2. Then via an application of Ito’s formula we have that Z t N Z t X g(X(t)) = g(X(0)) + (Lg)(X(s), v(X(s))ds + (Di g)(X(s))dYi (s) 0

Z

i=1

t

h∇g(X(s)), σ(X(s))dW (s)i.

+ 0

17

0

Taking expectations in the above equality, using the stationarity of X(·) and recalling the properties of the function g(·) we have that for all t ≥ 0 N X

E(Yi (t)) ≤

i=1

N X

Z E

 (Di g)(X(s))dYi (s)

0

i=1 Z t

≤

t

E|Lg(X(s), v(X(s)))|ds 0

≤ Ct, where

. C=

|Lg(x, u)|.

sup

(5.3)

x∈G,u∈U

Thus if we define for A ∈ B(Fi ); i = 1, · · · N , . µi (A) = E

Z

1

 IA (X(s))dYi (s)

0

then µi ∈ MF (Fi ) since Z E

1

 IFi (X(s))dYi (s) = E(Yi (1))

≤ C.

(5.4)

0

Now let f ∈ C0∞ (G) be arbitrary. Then another application of Ito’s formula gives Z f (X(1))

= f (X(0)) +

1

(Lf )(X(s), v(X(s))ds + 0

N Z X i=1

1

(Di f )(X(s))dYi (s)

0

1

Z

h∇f (X(s)), σ(X(s))dW (s)i.

+ 0

Taking expectations and using the stationarity of X(·) we have that Z (Lf )(x, v(x))ηv (dx) + G

N Z X i=1

(Di f )(x)µi (dx) = 0.

Fi

The proof now follows on recalling the definition of µ0 and observing that for (x, u) ∈ G × U Z (Lf )(x, u) = (Lf )(x, α)u(dα). S

The following extension of Echeverria-Weiss-Kurtz criterion (cf. [36] , [21]) is an essential step in our proof of Theorem 3.4. 18

Theorem 5.7 Suppose that there exist measures µ0 ∈ M(G×S), µi ∈ MF (Fi ) i = 1, · · · N such that for all f ∈ C0∞ (G) (5.2) holds. Decompose µ ˆ0 as: µ ˆ0 (dx, dα) = v(x, dα)η(dx),

(5.5)

where η ∈ P(G) is given as η(A) = µ ˆ0 (A × S); A ∈ B(G) and v(x, dα) is the appropriate regular conditional distribution. Then there exists a solution (Z(·), Φ(·)) to the CCMP (η, L, G, (Di , Fi )N i=1 ) on some filtered probability space (Ω, F, {Ft }, P ) such that (i) Z(·) is a stationary process with the invariant measure η. (ii) Φ(s)(·) = v(Z(s), ·) for all s ∈ [0, ∞), a.s. (iii) Z(·) is a positive recurrent, strongly Feller Markov process with transition probability family {Pxv }x∈G and η = ηv is its unique invariant measure (cf. Theorem 3.2). Proof: Assume without loss of generality that S ∩ {1, · · · N } = ∅. Define a new . ˜ ·) be a distance on it defined as follows. control set S˜ = S ∪ {1, · · · N }. Let d(·, ˜ For x, y ∈ S: . ˜ y) = d(x, d(x, y) if x ∈ S and y ∈ S . = 0 if x = y . = 1 otherwise, ˜ ·)) is a compact metric ˜ d(·, where d(·, ·) is the given metric on S. Clearly (S, space. For n ∈ IN (where IN is the set of all positive integers), define the ˜ as follows. For f ∈ C ∞ (G) and linear operator Cn : C0∞ (G) → Cb (G × S) 0 ˜ (x, α ˜) ∈ G × S . (Cn f )(x, α ˜ ) = (Lf )(x, α ˜ ); if α ˜∈S . = n(Di f )(x); if α ˜ ∈ {1, · · · N }. ˜ as follows. For h ∈ Cb (G × S) ˜ Define ν˜n ∈ P(G × S) Z h(x, α ˜ )˜ νn (dx, dα ˜) ˜ G×S

. 1 = Kn

N

Z

1X h(x, α)µ0 (dx, dα) + n i=1 G×S

. where Kn = µ0 (G × S) +

1 n

PN

i=1

µi (Fi ). 19

Z

! h(x, i)µi (dx) ,

Fi

(5.6)

From the assumption that (5.2) holds it follows now that for all f ∈ C0∞ (G) Z (Cn f )(x, α ˜ )˜ νn (dx, dα ˜ ) = 0. (5.7) ˜ G×S

˜ Disintegrate ν˜n as follows. For A ∈ B(G) and B ∈ B(S) Z ν˜n (A × B) =

v˜n (x, B ∩ S)η n (dx) +

A

N Z X i=1

v˜i,n (x)δ{i} (B)η n (dx),

A

where for B ∈ B(S) the maps v˜n (·, B) and v˜i,n (·) are measurable; for all x ∈ G PN v˜n (x, ·) ∈ MF (S), v˜i,n (x) ≥ 0, i = 1, · · · N ; v˜n (x, S) + i=1 v˜i,n (x) = 1 and η n ∈ P(G) is given as follows. For A ∈ B(G) ! N 1X . 1 n η (A) = µ0 (A × S) + µi (A ∩ Fi ) . (5.8) Kn n i=1 . Also define ν˜ ∈ P(G × S) as the normalization of µ0 , i.e. ν˜ = µ ˆ0 . Recall that from (5.5): ν˜(dx, dα) = v(x, dα)η(dx). For fixed x ∈ G define a probability measure vn (x, d˜ α) on S˜ as follows. For ˜ A ∈ B(S) . vn (x, A) = v(x, A ∩ S); if x ∈ G0 N

X . = v˜n (x, A ∩ S) + v˜i,n (x)δ{i} (A); otherwise.

(5.9)

i=1

˜ It is easy to check that for all h ∈ Cb (G × S) Z Z h(x, α ˜ )˜ νn (dx, dα ˜) = h(x, α ˜ )vn (x, d˜ α)η n (dx). ˜ G×S

(5.10)

˜ G×S

Using (5.7), (5.10) and Theorem 2.4 of [22] we now have that there exists a filtered probability space (For the sake of simplicity we suppress the dependence of the filtered probability space on n in our notation.) (Ω, F, P, (Ft )) on which is given an adapted G valued stationary process Xn (·) with continuous paths such that the probability law of Xn (0) is η n and for all f ∈ C0∞ (G)  Z t Z f (Xn (t)) − (Cn f )(Xn (s), α ˜ )vn (Xn (s), dα ˜ ) ds (5.11) 0

˜ S

is an Ft martingale.

20

For x ∈ G let vn0 (x, dα) ∈ MF (S) be defined as follows. For A ∈ B(S) . vn0 (x, A) = v(x, A); if x ∈ G0 . = v˜n (x, A); if x ∈ ∂G.

(5.12)

0 Also, define for i ∈ {1, · · · N } vi,n : G → [0, 1] as:

. 0 vi,n (x) = v˜i,n (x)I∂G (x).

(5.13)

Note that for all x ∈ G = vn0 (x, A); if A ∈ B(S);

vn (x, A)

N X

=

0 vi,n (x)I{i} (A); if A ⊂ {1, · · · , N }.

(5.14)

i=1

Rewriting (5.11) using (5.14) we have that for all f ∈ C0∞ (G)  Z t Z 0 f (Xn (t)) − (Lf )(Xn (s), α)vn (Xn (s), dα) ds 0

S

Z t N X 0 − n (Di f )(Xn (s))vi,n (Xn (s))ds 0

i=1

is an Ft martingale. Now define for t ∈ [0, ∞), i ∈ {1, · · · , N } . =

λn0 (t)

Z

t

vn0 (Xn (s), S)ds

0

. =

λni (t)

Z

t 0 vi,n (Xn (s))ds.

0

Also, for x ∈ G define Λn (x, ·) ∈ P(S) as follows. For A ∈ B(S) vn0 (x, A) ; if vn0 (x, S) 6= 0 vn0 (x, S) . = π(A); otherwise,

. Λn (x, A) =

(5.15)

where π is an arbitrary probability measure on S. Then in this new notation we have that for all f ∈ C0∞ (G)  Z t Z f (Xn (t)) − (Lf )(Xn (s), α)Λn (Xn (s), dα) dλn0 (s) 0

S

Z t N X − n (Di f )(Xn (s))dλni (s) i=1

0

21

PN n is a Ft martingale. Clearly, for all t ≥ 0, i=0 λi (t) = t. Furthermore, for i = 1, · · · , N Z t λni (t) = IFi (Xn (s))dλni (s); ∀t ≥ 0. (5.16) 0

To prove (5.16) note that, it suffices to show that ∀t ≥ 0 0 0 E(vi,n (Xn (t))) = E(IFi (Xn (t))vi,n (Xn (t))).

Also, 0 E(vi,n (Xn (t)) =

Z

0 vi,n (x)η n (dx)

G

= ν˜n (G × {i}) = ν˜n (Fi × {i}) 0 = E(IFi (Xn (t))vi,n (Xn (t))), where the next to last equality follows from (5.6). This proves (5.16). Thus it follows that (Xn (·), Λn (Xn (·)), λn0 (·), · · · , λnN (·)) solves the (η n , L, G, (nDi , Fi )N i=1 ) PCMP on (Ω, F, P, (Ft )). From Lemma 5.4 it follows that λn0 is a.s. strictly increasing. Now define τn : [0, ∞) → [0, ∞) as: . τn (t) = inf{s ≥ 0 : λn0 (s) ≥ t}; t ∈ [0, ∞). . . . Also for t ≥ 0 let, Gtn = Fτn (t) , Zn (t) = Xn (τn (t)), Φn (t) = Λn (Zn (t)) and . for i = 1, · · · , N Yi,n (t) = λi,n (τn (t)). Then from Proposition 5.5 (Zn (·), Φn (·)) solve the CCMP on (Ω, F, P, (Gtn )) with the corresponding boundary processes n {Yi,n (·)}N i=1 . Next note that since λ0 (0) = 0 and for 0 ≤ s ≤ t < ∞ |λn0 (t) − λn0 (s)| = |

Z

t

vn0 (Xn (r), S)dr|

s

≤

|t − s|; a.s.

we have that the family {λn0 (·)} is tight in C([0, ∞); [0, ∞)). Next observe that from Theorem 5.2 there exists an enlargement (Ω, F, {F t }, P ) of the above space such that: there is a F t − Wiener process W (·) defined on the enlarged space and   Z ·Z Z · Zn (t) = Γ Zn (0) + b(Zn (s), α)Φn (s)(dα)ds + σ(Zn (s))dW (s) (t), 0

S

0

(5.17) where the dependence of the Wiener process and the space on n is again suppressed in the notation. Since the probability law of Zn (0) is same as that of Xn (0), i.e. η n and from (5.8) η n (A) → η(A) as n → ∞ for all A ∈ B(G), we

22

have that the family {Zn (0)} is tight. Furthermore using the Lipschitz property of the Skorohod map we have that for 0 ≤ s ≤ t < ∞ Z t |Zn (t) − Zn (s)| ≤ Kr|t − s| + K| σ(Zn (s))dW (s)|. s

Recalling that σ(·) is bounded we have as a result of the above observations that the family {Zn (·)} is tight in C([0, ∞) : G). Let (Z(·), λ0 (·)) be a weak limit point of the sequence (Zn (·), λn0 (·)) and re-label the convergent subsequence as (Zn (·), λn0 (·)). Observing that λn0 (t) ≤ t, a.s. for all t ≥ 0 and n ∈ IN and E(λn0 (t))

= →

t µ0 (G × S) Kn t

as n → ∞, we have that λ0 (t) = t for all t ≥ 0, a.s. Next observe that from the weak convergence of Zn (·) and λn0 (·) we have that as n → ∞ Xn (·) = Zn (λn0 (·)) converges weakly to Z(λ0 (·)) ≡ Z(·). Since for each n ∈ IN Xn (·) is stationary, we must have that the limit Z(·) is a stationary process too. Also, since the law of Z(0) is η we have that the stationary distribution is η. Next note that, from (5.12) and (5.15), for all x ∈ G0 Λn (x, dα) = v(x, dα). Also from Theorem 4.2.1 [24] we have that for all n ∈ IN Z ∞  E I∂G (Zn (s))ds = 0. 0

From these two observations, (5.17) and the Lipschitz property of the Skorohod map it follows that   Z ·Z Z · Zn (t) = Γ Zn (0) + b(Zn (s), α)v(Zn (s), dα)ds + σ(Zn (s))dW (s) (t), 0

S

0

for all t ≥ 0, a.s. The Feller property of the family {Pxv } (See Theorem 2.9) now gives that Zn (·) converges weakly to the solution of   Z ·Z Z · ˜ = Γ Z(0) ˜ ˜ ˜ ˜ Z(t) + b(Z(s), α)v(Z(s), dα)ds + σ(Z(s))dW (s) (t). 0

S

0

˜ have Since Zn (·) also converges weakly to Z(·) we must have that Z(·) and Z(·) the same distribution, in particular Z(·) is a stationary Markov process with the stationary distribution η. This proves (i) and (ii) of the theorem. Finally part (iii) follows from Theorem 3.2.

23

6

Stability Properties of the Constrained Controlled Diffusions:

We will now like to obtain some stability properties of the class of processes obtained as a solution of an equation of the form (2.2). We will begin with the following definitions. Let β : [0, ∞) → IRk be a measurable map such that Z

t

|β(s)|ds < ∞; for all t ∈ [0, ∞).

(6.1)

0

Let x ∈ G. Define the trajectory z : [0, ∞) → IRk as   Z · . z(t) = Γ x + β(s)ds (t); t ∈ [0, ∞).

(6.2)

0

For x ∈ G let A(x) be the collection of all absolutely continuous functions z : [0, ∞) → IRk defined via (6.2) for some β : [0, ∞) → C(δ) which satisfies (6.1). For a fixed x ∈ G, we now define the “hitting time to the origin” function as follows: . T (x) = sup inf{t ∈ [0, ∞) : z(t) = 0}. (6.3) z∈A(x)

The following properties of the function T (·) have been proved in [1]. Lemma 6.1 (Lemma 3.3 [1]) There exist constants c, C1 ∈ (0, ∞) depending only on K and δ such that the following holds. (i) For all x, y ∈ G |T (x) − T (y)| ≤ C1 |x − y|. (ii) For all x ∈ G T (x) ≥ c|x|. For x ∈ G let X x (·) be the solution of (2.2) with X x (0) = x on some filtered probability space (Ω, F, {Ft }, P ) on which is given an admissible pair . (u(·), W (·)). For a compact set B ⊂ G, define τB (x) = inf{t > 0 : X x (t) ∈ B}. . Also, for a ∈ (0, ∞) define B a = {y ∈ G : T (y) ≤ a}. Let α0 , ∆ ∈ (0, ∞) be chosen such that: (kα0 C1 Kr)2 − α0 +

log 8 . = −θ < 0. 2∆

The proof of the following lemma is contained in the proof of Theorem 4.4 of [1].

24

Lemma 6.2 (cf. Theorem 4.4 [1]) For all x ∈ G and t ∈ (0, ∞) sup P (τB ∆ (x) > t) ≤

eαT (x) −θt e , e(α−θ)∆

where the supremum on the left side above is taken over all possible solutions X(·) of (2.2) with X(0) = x given on some filtered probability space (Ω, F, {Ft }, P ) with some admissible pair (u(·), W (·)). Remark 6.3 Theorem 4.4 of [1] is stated for uncontrolled constrained diffusion processes, however the result (and most of the proof ) holds in the generality considered here. The only place where the proof in [1] needs to be modified is as follows. The proof of Theorem 4.4 of [1] relies on Lemma 4.3 of the same paper. However, the proof of Lemma 4.3, presented in [1], at one place uses the Markov property of X x (·). Thus we provide, in the appendix of this work an alternate proof of this lemma which does not appeal to the Markov property and holds for the class of processes X x (·) considered here. Define

. ∆ r1 = , c where c is as in Lemma 6.1. For X x (·) as above, let . τ1 (x) = inf{t > 0 : |X x (t)| = r1 }.

(6.4)

From Lemma 6.1 (ii) it follows that for all x ∈ G τ1 (x) ≤ τB ∆ (x); a.s.

(6.5)

Proposition 6.4 Let r2 ∈ (r1 , ∞) be fixed. Then there exist κ, θ ∈ (0, ∞) such that for all t ≥ 0 sup

sup

P (τ1 (x) > t) < κe−θt ,

x∈G:|x|=r2 u(·),W (·)

where the inside supremum on the left side is taken over all possible solutions X(·) of (2.2) with X(0) = x given on some filtered probability space (Ω, F, {Ft }, P ) with some admissible pair (u(·), W (·)). Proof: Observe that, for all x ∈ G with |x| = r2 P (τ1 (x) > t) ≤ P (τB ∆ (x) > t) 1 ≤ eα0 T (x) e−θt (α 0 e −θ)∆ eα0 C1 r2 −θt e , ≤ e(α−θ)∆ where the first inequality follows from (6.5), the second from Lemma 6.2 while the third one follows from Lemma 6.1(i). This proves the lemma. 25

Lemma 6.5 Let r1 be as defined in (6.4) and let r2 ∈ (r1 , ∞) be arbitrary. For x ∈ G let X x,(u,W ) (·) denote the solution of (2.2), with X x,(u,W ) (0) = x, given on some filtered probability space (Ω, F, {Ft }, P ) and with an admissible pair (u(·), W (·)). Let . τ (x) = inf{t ≥ 0 : |X x,(u,W ) (t)| = r1 and |X x,(u,W ) (s)| = r2 for some s ∈ [0, t]}, where we have suppressed the dependence of τ (x) on (u(·), W (·)) in the notation. Then there exists a δ0 ∈ (0, ∞) such that inf

inf

x∈G:|x|=r1 u(·),W (·)

E(τ (x)) > δ0

(6.6)

and there exist κ0 , θ0 ∈ (0, ∞) such that for all t ∈ [0, ∞) sup

P (τ (x) > t) < κ0 e−θ0 t .

sup

(6.7)

x∈G:|x|=r1 u(·),W (·)

Proof: For notational simplicity we denote X x,(u,W ) (·) by X x (·). We first prove (6.7). Given X x (·) as in the statement of the lemma, define . τ0 (x) = inf{t ≥ 0 : |X x (t)| = r2 }. In view of Proposition 6.4 and Lemma 4.4 it suffices to show that there exist κ0 , θ00 ∈ (0, ∞) such that for all t ≥ 0 sup

sup

0

P (τ0 (x) > t) < κ0 e−θ0 t .

x∈G:|x|=r1 u(·),W (·)

This will follow, if we show that for all k ∈ IN sup

sup

0

P (τ0 (x) > k) < e−θ0 k .

(6.8)

u(·),W (·) x∈G:|x|=r1

Noting that

P (τ0 (x) > k) = E E I[τ0 (x)>k] | Fk−1 I[τ0 (x)>k−1] we have from Lemma 4.4 that in order to show (6.8) it suffices to show that there exists an 0 ∈ (0, 1) such that sup

sup

P (τ0 (x) > 1) < 0 .

(6.9)

x∈G:|x|≤r2 u(·),W (·)

We will prove this by the method of contradiction. Suppose that (6.9) does not hold for any 0 ∈ (0, 1). Then there exist {xn , un (·), W n (·), X n (·), Y n (·)}n≥1

26

such that for each n ∈ IN xn ∈ {x ∈ G : |x| ≤ r2 }, (un (·), W n (·)) is an admissi. ble pair on some filtered probability space, X n (·) and Y n (·) = (Y1n (·), · · · YNn (·)) are obtained as a solution of (2.2) with (u(·), W (·)) replaced by (un (·), W n (·)), X n (0) = xn and lim P (τ0,n > 1) = 1, n→∞ . where τ0,n = inf{t ≥ 0 : |X n (t)| = r2 }. Let {fi }∞ i=1 be a countable dense set in the unit ball of C(S). Define, for t ≥ 0 and j ≥ 1, Z . βjn (t) = fj (α)un (t, dα). S

Let B denote the closed unit ball of L∞ [0, ∞) endowed with the metric d(·, ·) defined as follows. For x, y ∈ B ∞ ∞ M . X X hx − y, ej iM d(x, y) = , 2M 2j j=1 M =1

∞ 2 where for each M ∈ IN {eM j (·)}j=1 is a CONS in L [0, M ] and h·, ·iM denotes 2 the usual inner product in L [0, M ]. Clearly (B, d(·, ·)) is a compact metric space. Let E denote the countable product of B endowed with the product topology. Then E is a compact Polish space and . β n (·) = (β1n (·), · · ·)

is an E valued random variable. Recalling that X n (0) = xn and |xn | ≤ r2 we see that the family X n (0) is tight. Furthermore, using the Lipschitz property of the Skorohod map and Condition 2.4 (ii) we see that there exists C˜ < ∞ such that for all 0 ≤ s ≤ t < ∞ Z t ˜ − s| + | |X n (t) − X n (s)| ≤ C[|t σ(X n (q))dW n (q)|]. s

Using the boundedness of σ(·) we now have that {X n (·)} is tight in C([0, ∞) : G). Next choosing g ∈ Cb2 (G) as in Lemma 4.2 we see that for 0 ≤ s ≤ t < ∞ N X

|Yin (t) − Yin (s)|

≤

i=1

N Z X

(Di g)(X n (q))dYin (q)

s

i=1 t

Z

t

|Lg(X n (q), un (q))|dq

≤ s

Z

t

h∇g(X n (q)), σ(X n (q))dW n (q)i| Z t ≤ C|t − s| + | h∇g(X n (q)), σ(X n (q))dW n (q)i|, + |

s

s

27

where C is as defined in (5.3). Combining this with the fact that Y n (0) = 0 we have that {Y n (·)} is tight in C([0, ∞) : [0, ∞)N ). Thus (β n (·), X n (·), Y n (·)) is a tight family of random variables with values in . E = E × C([0, ∞) : G) × C([0, ∞) : [0, ∞)N ). Pick a weakly convergent subsequence of the above sequence and re-label it as the original sequence. By going to the Skorohod representation space: (Ω, F, P ), however keeping the same notation for random variables for convenience, we have that there exists a E valued random element (α(·), X(·), Y (·)) such that (αn (·), X n (·), Y n (·)) converges almost surely to (α(·), X(·), Y (·)) as n → ∞. Next note that for f ∈ C0∞ (G) and 0 ≤ t1 ≤ t2 < ∞   Z t2 Z n n n n E f (X (t2 )) − f (X (t1 )) − (Lf )(X (s), α)u (s, dα) ds t1

−

N Z X i=1

S

!

t2

(Di f )(X

n

(s))dYin (s)

! n

n

n

n

ψ(X (s1 ), Y (s1 ), · · · X (sm ), Y (sm ))

= 0,

t1

(6.10) where m ∈ IN ; 0 ≤ s1 ≤ · · · ≤ sm ≤ t1 and ψ is an arbitrary continuous and bounded function defined on the obvious domain. From Lemma 2.4 of [6] we have that Z t2 Z t2 (Di f )(X n (s))dYin (s) → (Di f )(X(s))dYi (s) t1

t1

almost surely as n → ∞. Also from the Lipschitz property of the coefficients b(·) and σ (cf. Condition 2.4 (i), (iii)) and Lemma II.1.3 of [3] we have that   Z t2 Z Z t2 Z n n (Lf )(X (s), α)u (s, dα) ds → (Lf )(X(s), α)u(s, dα) ds t1

S

t1

S

almost surely, as n → ∞, where u(·) is a U valued measurable process satisfying Z fi (α)u(t, dα) = αi (t), S

for all i ∈ IN . Thus taking limit as n → ∞ in (6.10) we have that   Z t2 Z E f (X(t2 )) − f (X(t1 )) − (Lf )(X(s), α)u(s, dα) ds t1

−

N Z t2 X i=1

S

!

!

(Di f )(X(s))dYi (s) ψ(X(s1 ), Y (s1 ), · · · X(sm ), Y (sm ))

t1

28

= 0.

Furthermore, as in Lemma 4.3 we can take u(·) to be FtX,Y adapted. Also note that for any f ∈ Cb (G) such that f = 0 on Fi we have that Z ∞ f (X n (s))dYin (s) = 0, 0

it follows that for such an f Z

∞

f (X(s))dYi (s) = 0. 0

Thus for all i ∈ {1, · · · N } and t ≥ 0 Z

t

IFi (X(s))dYi (s),

Yi (t) = 0

almost surely. Thus (X(·), u(·)) solves the CCMP for (δ{x} , L, G, (Di , Fi )N i=1 ) . X,Y on the filtered probability space (Ω, F, Ft ). Finally, defining τ = inf{t : |X(t)| = r2 } we have that τ0,n → τ almost surely and thus P (τ ≥ 1) = 1. But this is clearly impossible in view of Condition 2.5. Thus we have arrived at a contradiction. This proves (6.7). The proof of (6.6) follows via a similar argument via contradiction, i.e if (6.6) does not hold then it follows as above that there exist {X(·), u(·), W (·), X(·), Y (·)} such that (u(·), W (·)) is an admissible . pair on some filtered probability space, X(·) and Y (·) = (Y1 (·), · · · YN (·)) are obtained as a solution of (2.2) with X(0) = x, where x ∈ G; |x| = r1 , and E(τ ∗ ) = 0, where . τ ∗ = inf{t ≥ 0 : |X(t)| = r1 and |X(s)| = r2 for some s ∈ [0, t]}, which is clearly impossible. This proves the lemma. The proof of the following lemma follows exactly as the proof of Lemma 4.5 of [1]. Lemma 6.6 ( cf. Lemma 4.5 [1]) For x ∈ G and an admissible pair (u(·), W (·)) on some filtered probability space, denote by X x,(u,W ) (·) the solution of (2.2) with X x,(u,w) (0) = x. Then for all M ∈ (0, ∞) the family {X x,(u,W ) (t), t ≥ 0; |x| ≤ M ; (u(·), W (·))admissible} is tight. As an immediate consequence of above lemmas we have the following result. Lemma 6.7 For π ∈ P(G) with support contained in L and admissible pair (u(·), W (·)) given on some filtered probability space (Ω, F, {Ft }, P ), let X π,(u,W ) (·), denote the solution of (2.2), with X π,(u,W ) (0) having the probability law π. Let . τ (π) = inf{t ≥ 0 : |X π,(u,W ) (t)| = r1 and |X π,(u,W ) (s)| = r2 forsome s ∈ [0, t]}.

29

Define η π,(u,W ) ∈ P(G) as follows. For f ∈ Cb (G) R  τ (π) π,(u,W ) Z E f (X (t))dt 0 . f (y)η π,(u,W ) (dy) = . E(τ (π)) G Then the family {η π,(u,W ) : π ∈ P(G); supp(π) ⊂ L; (u(·), W (·)) admissible} is tight. Proof: Let  > 0 be arbitrary. Let θ0 , κ0 , δ0 be as in Lemma 6.5. Then from Lemmas 4.4 and 6.6 there exists a compact set K  in G such that P (X π,(u,W ) (t) 6∈ K  ) ≤

(θ0 δ0 )2 , 4κ0

for all t ∈ (0, ∞), π ∈ P(G), supp(π) ⊂ L and (u(·), W (·)) admissible. Hence
R∞ E I{τ (π)>t} I{X π,(u,W ) (t)∈K /  } dt 0 π,(u,W )  c η ((K ) ) = E(τ (π)) R∞p P (τ (π) > t)dt θ0 δ0 0 ≤ √ 2 κ0 E(τ (π)) Z θ0 ∞ − θ0 t e 2 dt ≤ 2 0 ≤ , where the next to last inequality follows from Lemmas 4.4 and 6.5. This proves the lemma.

7

Proof of Theorem 3.4:

Let X x (·) solve (2.2) with X x (0) = x on some filtered probability space with an admissible pair (u(·), W (·)). Define stopping times τi , ξi , i ∈ IN as follows:

For n ≥ 1 and

. τ1 = inf{t ≥ 0 : |X x (t)| = r1 }.

(7.1)

. ξn = inf{t ≥ τn : |X x (t)| = r2 }

(7.2)

. τn+1 = inf{t ≥ ξn : |X x (t)| = r1 }.

(7.3)

From Proposition 6.4 and Lemma 6.5 it follows that for all i ∈ IN E(ξi ) < ∞ and E(τi ) < ∞. 30

Now suppose that there is a v : G → U such that for all t ≥ 0, v(X x (t)) = u(t), a.s. From the strong Markov property of the solution of (2.4) it follows that {X x (τi )}i≥1 is a . L = {x ∈ G : |x| = r1 } valued Markov chain. Furthermore from Condition 2.5 it follows that for i ∈ IN the probability law of X(τi ) has a density, with respect to the surface measure on L, which is bounded away from 0. Using the above property of the Markov chain {X(τi )}i≥1 we have the following result the proof of which is identical to that of Lemma IV.4.1 and Theorem IV.4.1 of [19]. Lemma 7.1 (Lemma IV.4.1, Theorem IV.4.1 [19]) Let X x (·) and v : G → U be as above. For n ∈ IN let P (n, y, A) denote the n step transition probability function for the Markov chain {X(τi )}i≥1 , where y ∈ L and A ∈ B(L). Then there exists a unique invariant measure, ρ, for this chain. Extend the measure . ρ to ρ˜ ∈ P(G) by setting ρ˜(A) = ρ˜(A ∩ L) for A ∈ B(G). Let X(·) be given as a solution of (2.4) with X(0) having the probability law ρ˜. Define η ∈ P(G) as follows. For f ∈ Cb (G)
R τ2 Z . E 0 f (X(t))dt . f (x)η(dx) = E (τ2 ) G Then η is the unique invariant measure for the Markov process X(·), i.e. in the notation of Theorem 3.2, η = ηv . An immediate consequence of Lemmas 7.1 and 6.7 is the following result. Lemma 7.2 The family {ηv |v : G → U ; v is measurable} is relatively compact in P(G). The following result shows that in fact a stronger result is true. Lemma 7.3 The family {ηv |v : G → U ; v is measurable} is a compact set in P(G). Proof: Let {vn } be a sequence of measurable maps from G to U . Suppose that ηvn converges to η ∈ P(G). We will like to show that there exists a measurable v : G → U such that η = ηv . Define the sequence {νn } of elements of P(G × S) as follows. For A ∈ B(G), B ∈ B(S) Z . νn (A × B) = vn (x, B)ηvn (dx). A

From Lemma 5.6 there exist measures µni ∈ MF (Fi ); i = 1, · · · N ; n ∈ IN such that for all f ∈ C0∞ (G) Z N Z X (Lf )(x, α)νn (dx, dα) + (Di f )(x)µni (dx) = 0. (7.4) G×S

i=1

31

Fi

From the compactness of S and Lemma 7.2 we have that {νn }n≥1 is a tight family. Denote by F i the one point compactification of Fi . Extend, for each i = 1, · · · N , µni to an element of MF (F i ) in a natural way, denoting the extension as µni . Also note that from (5.4) we have that the measures µni can be chosen such that µni (Fi ) = µni (F i ) ≤ C, where C is the constant defined in (5.3). Thus, by going to a subsequence if necessary, we have that there exist ν ∈ P(G × S) and µi ∈ MF (F i ) such that for all h ∈ Cb (G × S) and hi ∈ Cb (F i ); i = 1, · · · N Z Z h(x, α)νn (dxdα) → h(x, α)ν(dxdα) G×S

and

G×S

Z

hi (x)µin (dx)

Z

hi (x)µi (dx),

→

Fi

Fi

as n → ∞. Also note that ν(dx × S) = η(dx). Let v : G → U be a measurable map such that ν(dxdα) = v(x, dα)η(dx). For i = 1, · · · N , let µi be the restriction of µi to Fi . Then from (7.4) we have that Z (Lf )(x, α)ν(dx, dα) + G

N Z X i=1

(Di f )(x)µi (dx) = 0.

Fi

From Theorem 5.7 it now follows that η = ηv . This proves the Lemma. Let Z . β ∗ = inf k(x, v(x))ηv (dx), v

(7.5)

G

where the infimum on the right side is taken over all Markov controls v. In rest of the section we will show that the infimum above is attained by some Markov control v and furthermore for any admissible pair (u(·), W (·)) given on some filtered probability space 1 lim sup T →∞ T

Z

T

k(X(s), u(s))ds ≥ β ∗ ,

0

a.s., where X(·) solves (2.2) with an arbitrary initial distribution. This fact along with Lemma 4.1 will prove Theorem 3.4. Proposition 7.4 Let β ∗ be as defined in (7.5). Then, there exists a measurable map v : G → U such that Z k(x, v(x))ηv (dx) = β ∗ . G

32

Proof: Let vn : G → U be a sequence of maps such that Z k(x, vn (x))ηvn (dx) → β ∗ , G

as n → ∞. For n ∈ IN , define νn ∈ P(G × S) as follows. For f ∈ Cb (G × S) Z Z . f (x, α)νn (dxdα) = f (x, α)vn (x, dα)ηvn (dx). G×S

G×S

From Lemma 7.2 we have that {νn }n≥1 is a tight sequence of probability measures. By going to a subsequence if necessary, we have that there exists a ν ∈ P(G × S) and η ∈ P(G) such that νn → ν and ηvn → η as n → ∞. Thus recalling that Z Z k(x, α)νn (dxdα) k(x, vn (x))ηvn (dx) = G

we have that

G×S

Z

k(x, α)ν(dxdα) = β ∗ .

G×S

Clearly, η(dx) = ν(dx × S). Furthermore, as in the proof of Lemma 7.3 we have via an application of Theorem 5.7 that if v : G → U is a measurable map such that ν(dxdα) = v(x, dα)η(dx) then η = ηv . Hence Z k(x, α)ν(dxdα) β∗ = G×S  Z Z k(x, α)v(x, dα) ηv (dx) = S ZG = k(x, v(x))ηv (dx). G

Proposition 7.5 Let X(·) be the solution of (2.2) on some filtered probability space (Ω, F, {Ft }, P ) on which is given an admissible pair (u(·), W (·)). Then Z 1 t lim sup k(X(s), u(s))ds ≥ β ∗ , t→∞ t 0 a.s., where β ∗ is as defined in (7.5). Proof: In view of Proposition 7.4 it suffices to show that for a.e. ω and for every sequence tk → ∞ (as k → ∞), there exists a further subsequence (denoted again as the original sequence) such that Z Z 1 tk lim k(X(s), u(s))ds = k(x, v(x))ηv (dx), (7.6) k→∞ tk 0 G 33

for some measurable v : G → U , possibly depending on ω and the subsequence. Define a family of probability measures {νt }t≥0 on G × S as follows. For f ∈ Cb (G × S) Z tZ . 1 νt (f ) = f (X(s), α)u(s, dα)ds. t 0 S We first claim that the family {νtk } is a tight family for any sequence tk → ∞. Since S is compact, in order to prove the claim, it suffices to show that the family {˜ νtk }k≥1 of probability measures on G defined as: . 1 ν˜tk (f ) = tk

tk

Z

f (X(s)ds; f ∈ Cb (G) 0

is tight. For n ∈ IN , let fn be a non-negative smooth map defined on G as follows. fn (x)

= 0; |x| < n; = 1; |x| ≥ n + 1.

Also let f0 : G → IR+ be defined as f0 (x) = 1 for all x ∈ G. In order to prove the tightness of the family {˜ νtk }k≥1 it suffices to show that for each  > 0 there exists N () ∈ IN such that lim sup t→∞

1 t

Z

t

fn (X(s))ds < ;

(7.7)

0

for all n ≥ N (). Now fix  > 0. Define stopping times τi , ξi i ∈ IN as in (7.1), (7.2), (7.3) with X x (·) replaced by X(·). From Lemmas 6.5 (cf. 6.6), 6.7, 4.4 we have that there exists N () ∈ IN such that for all n ≥ N () and i ∈ IN Z τi+1  E fn (X(s))ds | Fτi < δ0 , (7.8) τi

a.s., where δ0 is as in Lemma 6.5. Once more from Lemma 6.5(see (6.7)) we have that sup E(τi+1 − τi )2 < ∞. i∈IN

Thus from Proposition IV.6.1, [27] it follows that for all n ∈ {0, 1, · · ·} Z τi+1 ! m Z τi+1 1 X lim fn (X(s))ds − E fn (X(s))ds | Fτi = 0, m→∞ m τi τi i=1 (7.9)

34

a.s. Next observe that on using (7.9) for n = 0 and Lemma 6.5 (6.6), we have that τm → ∞, a.s. as m → ∞. Furthermore, since r2 in Proposition 6.4 is arbitrary, it follows that τ0 < ∞ a.s. This implies that for all n ≥ N () lim sup t→∞

1 t

Z

t

fn (X(s))ds 0

m Z 1 X τi+1 fn (X(s))ds m→∞ τm i=1 τi Pm R τi+1 1 fn (X(s))ds i=1 τi m = lim sup Pm−1 1 m→∞ i=1 (τi+1 − τi ) m−1 R  P τi+1 m 1 E f (X(s))ds | F n τ i i=1 m τi = lim sup Pm−1 1 m→∞ i=1 E ((τi+1 − τi ) | Fτi ) m−1

≤ lim sup

δ0 δ0 = , ≤

where the inequality above follows from (7.8) and Lemma 6.5 (6.6). This proves the a.s. tightness of {νtk }k≥1 . Now define measures µi,k ∈ MF (Fi ) as: Z tk . 1 µi,k (f ) = f (X(s))dYi (s); f ∈ Cb (Fi ), tk 0 i = 1, · · · N. Observe that if g ∈ Cb2 (G) is defined as in the proof of Lemma 5.4, then µi,k (Fi )

Yi (tk ) tk Z Z tk 1 tk 1 ≤ |Lg(X(s), u(s))|ds + | h∇g(X(s)), σ(X(s))dW (s)i| + o(1) tk 0 tk 0 Z tk 1 ≤ C+ | h∇g(X(s)), σ(X(s))dW (s)i| + o(1), tk 0 =

where C is defined via (5.3). Noting that the second term on the right side of above display converges to 0, a.s., as k → ∞ we have that almost surely supk µi,k (Fi ) < ∞; i = 1, · · ·. Denote the extension of µi,k to F i , the one point compactification of Fi , by µi,k . Now fix an ω outside a suitable null set. Then, by going to a subsequence if necessary, we have that, for the given ω, there exist ν ∈ P(G × S), µi ∈ MF (F i ) such that for all f ∈ Cb (G × S) and fi ∈ Cb (F i ); i = 1, · · · N Z Z f (x, α)νtk (dxdα) → f (x, α)ν(dxdα) G×S

and

G×S

Z

Z fi (x)µi,k (dx) →

Fi

fi (x)µi (dx), Fi

35

as k → ∞. In particular, for all f ∈ C0∞ (G) Z Z (Lf )(x, α)νtk (dxdα) → G×S

and

(Lf )(x, α)ν(dxdα)

(7.10)

G×S

Z

Z (Di f )(x)µi,k (dx) →

(Di f )(x)µi (dx),

Fi

(7.11)

Fi

as k → ∞, where µi is the restriction of µi to Fi . Next via an application of Ito’s formula we have that for all f ∈ C0∞ (G) and k ∈ IN Z f (X(tk )) − f (X(0))

tk

=

(Lf )(X(s), u(s))ds + 0

Z

N Z X i=1

tk

(Di f )(X(s))dYi (s)

0

tk

h∇f (X(s), σ(X(s))dW (s)i.

+ 0

Then dividing by tk in the above expression and taking limit as k → ∞ in the above expression, we have from (7.10) and (7.11) that Z (Lf )(x, α)ν(dxdα) + G×S

N Z X i=1

(Di f )(x)µi (dx) = 0.

Fi

. Hence from Theorem 5.7 we have that if η(dx) = ν(dx×S) and ν is disintegrated as: ν(dxdα) = v(x, dα)η(dx) then η = ηv . Thus 1 lim k→∞ tk

Z

tk

k(X(s), u(s))ds

=

0

= = = =

Z 1 tk lim k(X(s), α)u(s, dα)ds k→∞ tk 0 Z lim k(x, α)νtk (dxdα) k→∞ G×S Z k(x, α)ν(dxdα) G×S Z k(x, α)v(x, dα)ηv (dx) G×S Z k(x, v)ηv (dx). G

This proves the proposition. We are now ready to prove our main result. Proof of Theorem 3.4: From Proposition 7.4 there exists a measurable map v : G → U such that Z β∗ = k(x, v(x))ηv (dx). (7.12) G

36

Let X(·) solve (2.4), with v replaced by v on some filtered probability space with probability law of X(0) equal to µ. Then, from Lemma 4.1 Z Z 1 T lim sup k(X(s), v(X(s))ds = k(x, v(x))ηv (dx), (7.13) T →∞ T 0 G a.s. Combining (7.12) and (7.13) we have that Z 1 T β ∗ = lim sup k(X(s), v(X(s)))ds. T →∞ T 0 The result now follows from Proposition 7.5.

8

APPENDIX

In this Appendix we provide an alternate proof of Lemma 4.3 of [1] which does not appeal to the Markov property of X x (·). This lemma is needed for the proof of Theorem 4.4 of [1] and therefore the proof of Lemma 6.2 of the present paper. Lemma 8.1 For x ∈ G and an admissible pair (u(·), W (·)) given on some filtered probability space, (Ω, F, {Ft }, P ). Let X x (·) denote the solution of (2.2) with X x (0) = x. Let ∆ > 0 be fixed. For n ∈ IN let νn be defined as follows. Z s . x νn = σ(X (s))dW (s) . sup (n−1)∆≤s≤n∆ (n−1)∆ Then for any ρ ∈ (0, ∞) and m, n ∈ IN ; m ≤ n  Pn   √ 2 2 2 (n−m+1) E eρ i=m νi ≤ 2 2ek ρ r ∆ . Proof: Let

Z

·

σ(X x (s))dW (s) ≡ (M1 (·), · · · , Mk (·)).

0

Then for i = 1, · · · k, Mi (·) is a square integrable {Ft } martingale. An application of Cauchy-Schwarz inequality gives that  

E eρ

Pn i=m

νi



≤

k Y

 k1 

E ekρ

Pn i=m

 i

Mj

 ,

j=1

where for j = 1, · · · k; i = m, m + 1, · · · n . Mji = sup |Mj (s) − Mj ((i − 1)∆)|. (i−1)∆≤s≤i∆

37

Now fix a j ∈ {1, · · · , k}. In view of Condition 2.5 | < Mj > (t)| → ∞, a.s. as t → ∞. Thus from Theorem V.1.6 [31], Mj (·) has the same probability law . as B(τ (·)), where τ (·) =< Mj > (·), B(t) is a Gt - standard Brownian motion; . Gt = FS(t) and . S(t) = inf{s ≥ 0 :< Mj > (s) > t}. Next observe that   Pn i E ekρ i=m Mj

where

  Pn = E ekρ i=m sup0≤s≤∆ |B(τ ((i−1)∆+s))−B(τ ((i−1)∆))|   = E Hn−1 ekρ sup0≤s≤∆ |B(τ ((n−1)∆+s))−B(τ ((n−1)∆))| ,

. Hn−1 = ekρ

Pn−1 i=m

sup0≤s≤∆ |B(τ ((i−1)∆+s))−B(τ ((i−1)∆))|

.

Note that Hn−1 is Gτ ((n−1)∆) measurable. Furthermore, from Condition 2.4(iv) we have that for 0 < s < t < ∞ |τ (t) − τ (s)| ≤ r2 |t − s| and therefore sup |B(τ ((n − 1)∆ + s)) − B(τ ((n − 1)∆))| 0≤s≤∆

≤

sup

|B(τ ((n − 1)∆) + t) − B(τ ((n − 1)∆))|.

0≤t≤r 2 ∆

Finally note that   E Hn−1 ekρ sup0≤s≤∆ |B(τ ((n−1)∆+s))−B(τ ((n−1)∆))|   ≤ E Hn−1 ekρ sup0≤t≤r2 ∆ |B(τ ((n−1)∆)+t)−B(τ ((n−1)∆))|    = E Hn−1 E ekρ sup0≤t≤r2 ∆ |B(τ ((n−1)∆)+t)−B(τ ((n−1)∆))| | Gτ ((n−1)∆)    12  2 ≤ E Hn−1 2 E(e2kρ|B(r ∆)| )  √ 2 2 2  ≤ E Hn−1 2 2ek ρ r ∆ √ 2 2 2 = 2 2ek ρ r ∆ E (Hn−1 ) , where the second inequality above follows from Doob’s inequality for submartingales. Iterating the above inequalities we have the result.

References [1] R. Atar, A. Budhiraja, and P. Dupuis. On positive recurrence of constrained diffusion processes. Submitted for Publication., 2000. [2] A. Bensoussan. Perturbation Methods in Optimal Control. Wiley, NY, 1988. 38

[3] V. Borkar. Optimal Control of Diffusion Processes. Longman Scientific and Technical, 1989. [4] V. S. Borkar and M. K. Ghosh. Ergodic control of multidimensional diffusions I:The existence results. SIAM J. of Control and Optimization, 26:112– 126, 1988. [5] A. Budhiraja and P. Dupuis. Simple necessary and sufficient conditions for the stability of constrained processes. SIAM J. Applied Math., 59:1686– 1700, 1999. [6] J. G. Dai and R.J. Williams. Existence and uniqueness of semimartingale reflecting Brownian motions in convex polyhedrons. Theory of Probability and its Applications, 40:1–40, 1995. [7] P. Dupuis and H. Ishii. On Lipschitz continuity of the solution mapping to the Skorokhod problem, with applications. Stochastics, 35:31–62, 1991. [8] P. Dupuis and K. Ramanan. Convex duality and the Skorokhod Problem. I, II. Probability Theory and Related Fields, 2:153–195, 197–236, 1999. [9] P. Dupuis and K. Ramanan. A multiclass feedback queueing network with a regular Skorokhod problem. LCDS Report 99-5, 1999. [10] P. E. Echeverria. A criterion for invariant measures of Markov processes. SIAM J. Control Optim., 61:1–16, 1982. [11] J. M. Harrison and M. I. Reiman. Reflected Brownian motion on an orthant. The Annals of Probab., 9:302–308, 1981. [12] J. M. Harrison and J. A. Van Miegham. Dynamic control of Brownian networks: state space collapse and equivalent workload formulation. Annals of Applied Probability, 7:747–771, 1997. [13] J. M. Harrison and L. Wein. Scheduling networks of queues: heavy trafic analysis of a simple open network. Queuing Systems, 5:265–280, 1989. [14] J. M. Harrison and R. J. Williams. Brownian models of open queueing networks with homogeneous customer populations. Stochastics, 22:77–115, 1987. [15] U. G. Haussmann and J. P. Lepeltier. On the existence of optimal controls. SIAM J. of Control and Optimization, 28:851–902, 1990. [16] G. Kallianpur. Stochastic Filtering Theory. Springer-Verlag, 1980. [17] F. P. Kelly and C. N. Laws. Dynamic routing in open queuing networks: Brownian models, cut constraints and resource pooling. Queuing Systems, 13:47–86, 1993. 39

[18] R. Z. Khasminskii. Ergodic properties of recurrent diffusion processes and stabilization of the solution to the Cauchy problem for parabolic equations. Theory of Probability and its Applications, 5:179–196, 1960. [19] R. Z. Khasminskii. Stochastic Stability of Differential Equations. Sijthoff and Noordhoff, 1980. [20] T. G. Kurtz. Martingale problems for constrained Markov processes. In J. S. Baras and V. Mirelli, editors, Advances in Stochastic Calculus, pages 151–168. Springer–Verlag, New York, 1990. [21] T. G. Kurtz. A control formulation for constrained Markov processes. Lectures in Applied Mathematics, 27:139–150, 1991. [22] T. G. Kurtz and R. H. Stockbridge. Existence of Markov controls and characterization of optimal Markov controls. Z. Wahrsch. verw. Gebiete, 35:609–653, 1998. [23] H. J. Kushner. Optimality conditions for the average cost per unit time problem with a diffusion model. SIAM J. of Control and Optimization, 16:330–346, 1978. [24] H. J. Kushner. Heavy Traffic Analysis of Controlled and Uncontrolled Queueing and Communication Networks. Springer-Verlag, To Appear. [25] H. J. Kushner and L. F. Martins. Routing and singular control for queuing networks in heavy traffic. SIAM J. Control Optim, 28:1209–1233, 1990. [26] J.L. Menaldi and M. Robin. Ergodic control of reflected diffusions with jumps. Applied Math. and Opt., pages 117–138, 1997. [27] J. Neveu. Mathematical foundations of the calculus of probability. HoldenDay Inc., 1965. [28] V. Nguyen. Processing networks with parallel and sequential tasks: Heavy traffic analysis and Brownian limits. Annals of Applied Probability, 3:28–55, 1993. [29] W. Peterson. A heavy traffic limit theorem for networks of queues with multiple customer types. Math. of Operations Research, 16:90–118, 1991. [30] M. Reiman and R. J. Williams. A boundary property for semimartingale reflecting Brownian motions. Probability Theory and Related Fields, 77:87– 97, 1988. [31] D. Revuz and M. Yor. Continuous Martingales and Brownian Motion. Springer-Verlag, 1991.

40

[32] M. Robin. Long-term average cost control problems for continuous time markov processes-a survey. Acta Appl. Math., 1:281–300, 1983. [33] R. H. Stockbridge. Time average control of martingale problems. Ph. D Thesis, University of Wisconsin, 1987. [34] D. Stroock and S. Varadhan. Springer-Verlag, 1979.

Multidimensional Diffusion Processes.

[35] R. Tarres. Asymptotic evolution of a stochastic control problem. SIAM J. of Control and Optimization, 23:614–631, 1985. [36] A. Weiss. Invariant measures of diffusions in bounded domains. Ph. D Thesis, New York University, 1981. [37] E. Wong. Representations of martingales, quadratic variation and application. SIAM J. Control, 9(4):621–633, 1971.

41