multi-period mean variance optimal control of markov jump with

Dedicated to Dr. TOADER MOROZAN on the occasion of his 70th birthday

MULTI-PERIOD MEAN VARIANCE OPTIMAL CONTROL OF MARKOV JUMP WITH MULTIPLICATIVE NOISE SYSTEMS OSWALDO L.V. COSTA and RODRIGO T. OKIMURA

We consider the multi-period mean variance stochastic optimal control problem of discrete-time Markov jump with multiplicative noise linear systems. First, we consider the performance criterion to be a linear combination of the final variance and expected value of the output of the system. We analytically derive an optimal control policy for this problem. By using this solution, we consider next the cases in which the performance criterion is to minimize the final variance subject to a restriction on the final expected value of the output, and to maximize the final expected value subject to a restriction on the final variance of the output of the system. The optimal control strategies are obtained from a set of interconnected Riccati difference equations. AMS 2000 Subject Classification: 49N10, 60J10, 91B28, 93E20. Key words: mean variance control, Markov jump system, multiplicative noise, stochastic optimal control. 1. INTRODUCTION

The uni-period mean-variance optimization is a classical financial problem introduced by [10] which paved the foundation for the modern portfolio theory. Using a stochastic linear quadratic theory developed in [1], the continuous-time version of Markowitz’s problem was studied in [15], with closed-form efficient policies derived, along with an explicit expression of the efficient frontier. In [5] the authors extended the mean-variance allocation problem to the discrete-time multi-period case while in [16] they considered a multi-period generalized mean-variance formulation for the risk control over bankruptcy. A geometric approach to these problems was presented in [4], considering assets as well as liabilities in the portfolios. In [13] the authors considered the discrete-time multi-period mean-variance allocation problem in the case where the parameters are subject to Markovian jumps, following an approach closely related to that in [5], while in [14] they studied a similar problem under a different point of view. As pointed out in [16], one of MATH. REPORTS 9(59), 1 (2007), 21–34

22

Oswaldo L.V. Costa and Rodrigo T. Okimura

2

the key difficulties in solving the multi-period mean variance problem is the non-separability in the associated stochastic control problem in the sense of dynamic programming. Due to that reason, a tractable (from the dynamic programming point of view) auxiliary problem is introduced. In this paper we consider the multi-period mean variance stochastic optimal control problem of discrete-time Markov jump with multiplicative noise linear systems. As in [5] we introduce a tractable auxiliary stochastic quadratic optimal control problem, where the performance criterion consists of a linear part and a quadratic cost of the state variable at the final time T . There is no penalty on the control variable, so that the standard techniques for the LQG problems cannot be used. It should be pointed out that problems with indefinite weighting matrices have been intensively studied lately as can be seen, for instance, in [8], [11], and for the case with Markov jumps and multiplicative noise in [3], [6], [7], [9]. The paper is organized as follows. In Section 2 we present the notation and some preliminary results that will be required for the solution of the auxiliary stochastic quadratic optimal control problem. In Section 3 we present the three problems that we will consider. In Problem P1 it is asked to maximize the final expected value subject to a restriction on the final variance of the output of the system while in Problem P2 it is asked to minimize the final variance subject to a restriction on the final expected value of the output. In Problem P3 it is asked to minimize a performance criterion which is a linear combination of the final variance and expected value of the output of the system. An analytical optimal control policy for these problems can be obtained through an auxiliary stochastic quadratic optimal control problem, solved in Section 4 in terms of a set of interconnected Riccati difference equations. The paper is concluded in Section 5 with a solution for the 3 problems stated in Section 3, expressed in terms of some key parameters. These parameters are explicitly written as functions of the parameters of the system and in terms of the set of interconnected Riccati difference equations. A possible application of the results in this paper would be in an asset liabilities management (ALM) model for defined-benefit (BD) pension funds with regime switching. We could assume that the market parameters depend on the market mode that switches according to a Markov chain among a finite number of states. The ALM for DB pension funds problem can then be written as a Markov jump with multiplicative noise LQ optimal control problem with linear and quadratic costs, so that the results presented here can be applied to solve the problem.

3

Multi-period mean variance optimal control

23

2. PRELIMINARIES

We denote by Rn the n-dimensional real Euclidean space and by the normed bounded linear space of all m × n real matrices, with B(Rn ) := B(Rn , Rn ). For a matrix A ∈ B(Rn , Rm ), N(A) denotes the null space of A, R(A) the range of A and A
the transpose of A. As usual, for A ∈ B(Rn ), A ≥ 0 (A > 0 respectively) means that the matrix A is positive semi-definite (positive definite), and tr(A) denotes the trace of A. The operator expected value will be denoted by E(.). Set Hn,m for the linear space made up of all N -sequences of real matrices V = (V1 , . . . , VN ) with Vi ∈ B(Rn , Rm ), i = 1, . . . , N , and, for simplicity, set Hn := Hn,n . We say
n ) if V ∈ Hn and, for each i = 1, . . . , N , Vi that V = (V1 , . . . , VN ) ∈ Hn+ (H
n, is a positive-semidefinite (symmetric) matrix. For V = (V1 , . . . , VN ) ∈ H n
, we write V ≥ R if Vi − Ri ≥ 0 for each i = 1, . . . , N . R = (R1 , . . . , RN ) ∈ H We denote by B(Hn , Hm ) the space of all bounded linear operators from Hn to Hm and, in particular, B(Hn ) := B(Hn , Hn ). We say that T ∈ B(Hn+ , Hm+ ) if T ∈ B(Hn , Hm ) and is such that T (V ) ∈ Hm+ whenever V ∈ Hn+ . For a sequence of n  dimensional square matrices A(0), . . . , A(T ), we use the following notation: t
=s A(
) = A(t) . . . A(s) for t ≥ s, I for t < s. We define 1S as the usual indicator function, that is, 1S (ω) = 1 if ω ∈ S, zero elsewhere. We need the following definition (see [12], pages 12–13)). B(Rn , Rm )

Definition 1. For a matrix A ∈ B(Rn , Rm ), the generalized inverse of A (or Moore-Penrose inverse of A) is defined to be the unique matrix A† ∈ B(Rm , Rn ) such that i) AA† A = A, ii) A† AA† = A† , iii) (AA† )
= AA† , and iv) (A† A)
= A† A. We recall the result below (see [12], pages 12–13). Proposition 1 (Schur’s complement). The following assertion are equivalent.   Q11 Q12 ≥ 0. a) Q = Q
12 Q22 b) Q22 ≥ 0, Q12 = Q12 Q†22 Q22 and Q11 − Q12 Q†22 Q
12 ≥ 0. c) Q11 ≥ 0, Q12 = Q11 Q†11 Q12 and Q22 − Q
12 Q†11 Q12 ≥ 0. The following result will be useful in the sequel. Proposition 2. Consider Y ∈ B(Rn ) and M ∈ B(Rm ) with Y ≥ 0. Let A and B be stochastic matrices (that is, each entry of them is a random variable) in B(Rn ) and B(Rm , Rn ), respectively. Then  † (1) E(A
Y A) − E(A
Y B) E(B
Y B) E(B
Y A) ≥ 0

24

and (2)

(3)

Oswaldo L.V. Costa and Rodrigo T. Okimura

4

 †   E(A
Y B) = E(A
Y B) E(B
Y B) + M E(B
Y B) . Proof. Consider the stochastic matrix    Y Q11 Q12 = Q= B
Y Q
12 Q22

YB R

 ,

where R = B
Y B. Clearly we have Q11 = Y ≥ 0, and Q12 = Y B = Q11 Q†11 Q12 = Y Y † Y B since Y Y † Y = Y from Definition 1. Furthermore, using again Y Y † Y = Y , we have Q22 − Q
12 Q†11 Q12 = R − B
Y Y † Y B = R − B
Y B = 0. Thus, by Schur’s complement (Proposition 1), Q ≥ 0, hence   
   
A 0 A 0 A Y A A
Y B Y YB = ≥ 0. R 0 I R 0 I B
Y A B
Y Taking the expected value of the above equation, we get    
11 Q
12 Q E(A
Y A) E(A
Y B) ≥ 0. =


E(R) E(B
Y A) Q 12 Q22 By Schur’s complement again, we have

†

12 Q
† Q


11 − Q 0≤Q 22 12 = E(A Y A) − E(A Y B)E(R) E(B Y A)

and


†
12 Q
† Q

12 = E(A
Y B) = Q Q 22 22 = E(A Y B)E(R) E(R), showing the result stated.


3. PROBLEM FORMULATION

On a probabilistic space (Ω, P, F) consider the Markov Jump Linear System with multiplicative noise (4)

εx  
θ(k),s (k)wsx (k) x(k)+ A x(k + 1) = A¯θ(k) (k) + s=1 εu  
θ(k),s (k)wsu (k) u(k), ¯ B + Bθ(k) (k) + s=1

x(0) = x0 , θ(0) = θ0 , where θ(k) is a time-varying Markov chain taking on values in {1, . . . , N } with transition probability matrix P(k) = [pij (k)], {wsx (k); s = 1, . . . εx , k = 0, 1, . . . , T − 1} are zero-mean random variables independent of the Markov chain {θ(k)}, with variance equal to 1 and E(wix (t)wjx (k)) = 0, for all t = k

5

Multi-period mean variance optimal control

25

and i = j. Similarly, {wsu (k); s = 1, . . . εu , k = 0, 1, . . . , T − 1} are zeromean random variables independent of the Markov chain {θ(k)}, with variance equal to 1 and E(wiu (t)wju (k)) = 0, for all t = k and i = j. The initial conditions θ0 and x0 are assumed to be independent of {wsx (k)} and {wsu (k)}, with x0 an n-dimensional random vector with finite second moments. Set µi (0) = E(x0 1{θ0 =i} ), µ(0) ∈ RN n as µ(0)
= (µ1 (0) · · · µN (0)), and Qi (0) = E(x(0)x(0)
1{θ0 =i} ), Q(0) = (Q1 (0), . . . , QN (0)) ∈ Hn+ . The correlation of wsx1 (k) and wsu2 (k) is denoted by E(wsx1 (k)wsu2 (k)) = ρs1 ,s2 (k). Without loss of generality, assume that ε = εx = εu . For each k = 0, 1, . . . , T − 1, we also have ¯ A(k) = (A¯1 (k), . . . , A¯N (k)) ∈ Hn ,
1,s (k), . . . , A
N,s (k)) ∈ Hn , s = 1, . . . , ε,
s (k) = (A A ¯N (k)) ∈ Hm,n , ¯ ¯1 (k), . . . , B B(k) = (B
1,s (k), . . . , B
N,s (k)) ∈ Hm,n , s = 1, . . . , ε.
s (k) = (B B Set πi (k) = P (θ(k) = i), let Ft be the σ-field generated by {(θ(s), x(s)); s = 0, . . . , τ }, and write U(τ ) = {uτ = (u(τ ), . . . , u(T − 1)); u(k) is an m-dimensional random vector with finite second moments that is Fk -measurable for each k = τ, . . . , T − 1}. Consider the scalar output (5)

y(t) = Lx(t)

of system (4), where L ∈ B(Rn , R). The multi-period mean-variance problem aims at selecting u ∈ U(0) which yields the greatest expected terminal value of the output E(y u (T )) given a maximal terminal output σ 2 for the variance Var(y u (T )), or which produces the lesser terminal output variance Var(y u (T )) given a maximal exu pected terminal value for  the output E(y (T )). Formally, these problems, 2 caled respectively P 1 σ and P 2 ( ), can be stated as   (6) min −E (y u (T )) subject to Var (y u (T )) ≤ σ 2 , P 1 σ2 : u∈U(0)

(7)

P 2 ( ) :

min Var (y u (T )) subject to E (y u (T )) ≥ .

u∈U(0)

Alternatively, an unconstrained form would be (8)

P 3 (ν) :

min νVar (y (T )) − E (y (T )) ,

u∈U(0)

where ν ∈ [0, ∞) is a risk aversion coefficient, giving a trade-off preference between the expected terminal wealth and the associated risk level. Since problem P3(ν) involves a non-linear function of the expectation in Var(V (t)) =

26

Oswaldo L.V. Costa and Rodrigo T. Okimura

6

E(V (t)2 ) − E(V (t))2 , it cannot be directly solved by dynamic programming. A solution procedure to seek an optimal dynamic control policy for problem P3(ν) based on a tractable auxiliary problem is proposed in [16]. We will adopt the same procedure in this paper, and consider the auxiliary problem (9)

A (λ, ν) :

 min E νy (T )2 − λy (T ) .

u∈U(0)

4. SOLUTION OF THE AUXILIARY PROBLEM

Let us consider the following intermediate problems for problem (9). At each time k ∈ {0, . . . , T − 1} define

 J (x (k) , θ (k) , k) = min E νy(T )2 − λy(T ) | Fk . uk ∈U(k)

Define next for k = 0, . . . , T −1 the operators E(k, .) ∈ B(Hn ), A(k, .) ∈ B(Hn ), G(k, .) ∈ B(Hn , Hn,m ), R(k, .) ∈ B(Hn , Hm ), P(k, .) ∈ B(Hn ), V(k, ., .) ∈ B(Hn × Hn,1 , Hn,1 ), D(k, ., ., .) ∈ B(Hn × Hn,1 × H1 , H1 ), and H(k, .) ∈ B(Hn,1 , Hn,m ). For X ∈ Hn , V ∈ Hn,1 , γ ∈ H1 , and i = 1, . . . , N , set Ei (k, X) =

N 

pij (k)Xj ,

j=1

Ai (k, X) = A¯i (k)
Ei (k, X)A¯i (k) +  Gi (k, X) =

ε 


i,s (k)
Ei (k, X)A
i,s (k), A

s=1

¯i (k)+ A¯i (k)
Ei (k, X)B +

ε  ε 





ρs1 ,s2 (k)Ai,s1 (k) Ei (k, X)Bi,s2 (k) ,

s1 =1 s2 =1

¯i (k)
Ei (k, X)B ¯i (k) + Ri (k, X) = B

ε 


i,s (k),
i,s (k)
Ei (k, X)B B

s=1


Pi (k, X) = Ai (k, X) − Gi (k, X) Ri (k, X)† Gi (k, X), ¯i (k)Ri (k, X)† Gi (k, X)), Vi (k, X, V ) = Ei (k, V )(A¯i (k) − B 1 ¯i (k)Ri (k, X)† B ¯i (k)
Ei (k, V )
, Di (k, X, V, γ) = Ei (k, γ) − Ei (k, V )B 4 ¯i (k)
Ei (k, V )
. Hi (k, V ) = B

7

Multi-period mean variance optimal control

27

It is easy to see that E(k, .) ∈ B(Hn+ ), A(k, .) ∈ B(Hn+ ) and R(k, .) ∈ B(Hn+ , Hm+ ). The next proposition will be useful in the sequel. It justifies the definition of the operators above. Notice that this result is closely related to the optimality Bellman equation. Proposition 3. Let P = (P1 , . . . , PN ) ∈ Hn+ , V = (V1 , . . . , VN ) ∈ Hn,1 and γ ∈ H1 . Then P(k, P ) ∈ Hn+ and Gi (k, P )
= Gi (k, P )
Ri (k, P )† Ri (k, P ).

(10)

Moreover, for any uk ∈ U(k), u(k) = u, x(k) = x and θ(k) = i we have  λ2 E νx(k + 1)
Pθ(k+1) x(k + 1) − λVθ(k+1) x(k + 1) + γθ(k+1) |Fk = ν



= ν[x Ai (k, P )x + 2x Gi (k, P ) u + u Ri (k, P )u]− 2   ¯i (k)u ] + λ Ei (k, γ), −λ[Ei (k, V ) A¯i (k)x + B ν

(11)

and, if for each i, ¯i (k) ∈ R(Ri (k, P )), Ei (k, V )B

(12)

then (11) can be rewritten as ν[x
Ai (k, P )x + 2x
Gi (k, P )
u + u
Ri (k, P )u]− 2   ¯i (k)u ] + λ Ei (k, γ) = −λ[Ei (k, V ) A¯i (k)x + B ν

= ν[x Pi (k, P )x + (u + a(x)) Ri (k, P )(u + a(x))]−

(13)

−λVi (k, P, V )x + where (14)

λ2 Di (k, P, V, γ), ν

 λ ¯

(k) E (k, V ) . a(x) = Ri (k, P )† Gi (k, P )x − B i i 2ν

Proof. Setting A = A¯i (k)+

ε ε ¯i (k)+ B
i,s (k)wsx (k), B = B
i,s (k)wsu (k), A

s=1

s=1

Y = Ei (P ) in Proposition 2, from inequation (1), and the hypothesis on {wsx (k)} and {wsu (k)} we have Pi (k, P ) = A¯i (k)
Ei (P )A¯i (k) +


−Gi (k, P )

ε 


i,s (k)−
i,s (k)
Ei (P )A A

s=1 † Ri (k, P ) Gi (k, P )

≥ 0,

28

Oswaldo L.V. Costa and Rodrigo T. Okimura

8

hence Pi (k, P ) ≥ 0. It also follows from Proposition 2 and equation (2) that (10) holds. We then have  (15) E x(k + 1)
Pθ(k+1) x(k + 1)|Fk = ε  
i,s (k)
Ei (k, P )A
i,s (k) x+ A = x
A¯i (k)
Ei (k, P )A¯i (k) + s=1



¯i (k) + +2x
A¯i (k)
Ei (k, P )B 

ε ε  


i,s (k)
Ei (k, P )B
i,s (k) u+ ρs1 ,s2 (k)A 1 2

s1 =1 s2 =1

¯i (k) + ¯i (k)
Ei (k, P )B +u
B

ε 


i,s (k) u
i,s (k)
Ei (k, P )B B

s=1

and (16) (17)

  ¯i (k)u , E Vθ(k+1) x(k + 1)|Fk = Ei (k, V ) A¯i (k)x + B  E γθ(k+1) |Fk = Ei (k, γ).

Equations (15), (16) and (17) yield (11). Considering now on the right hand side of (11) only the terms dependent on u and calling them f (u), we have  λ ¯i (k) u. (18) f (u) = νu
Ri (k, P )u + 2 νx
Gi (k, P )
− Ei (k, V )B 2 It follows from (10) and (12) that (18) can be written as (19)

f (u) = νu
Ri (k, P )u+  λ ¯i (k) Ri (k, P )† Ri (k, P )u. +2ν x
Gi (k, P )
− Ei (k, V )B 2ν

Writing a(x) as in (14), equation (4) can be rewritten as f (u) = ν[u
Ri (k, P )u + 2a(x)
Ri (k, P )u] = ν[(u + a(x))
Ri (k, P )(u + a(x)) − a(x)
Ri (k, P )a(x)]. Notice now that (20)

 −a(x)
Ri (k, P )a(x) = − x
Gi (k, P )
Ri (k, P )† Gi (k, P )x− λ ¯i (k)Ri (k, P )† Gi (k, P )x+ − Ei (k, V )B ν

+

λ2 †
¯ ¯ E (k, V ) B (k)R (k, P ) (E (k, V ) B (k)) , i i i i i 4ν 2

9

Multi-period mean variance optimal control

29

where we have used the fact that Ri (k, P )† Ri (k, P )Ri (k, P )† = Ri (k, P )† . Thus we have (21)

ν[x
Ai (k, P )x + 2x
Gi (k, P )
u + u
Ri (k, P )u] +

λ2 Ei (k, γ)− ν

¯i (k)u] = −λEi (k, V )[A¯i (k)x + B λ2 = νx
Ai (k, P )x − λEi (k, V )A¯i (k)x + Ei (k, γ) + f (u) = ν

= ν[x (Ai (k, P ) − Gi (k, P ) Ri (k, P )† Gi (k, P ))]x− ¯i (k)Ri (k, P )† Gi (k, P )]x+ −λ[Ei (k, V )A¯i (k) − Ei (k, V )B +ν(u + a(x))
Ri (k, P )(u + a(x))+

+

1 λ2  ¯i (k)Ri (k, P )† (Ei (k, V )B ¯i (k))
= Ei (k, γ) − Ei (k, V )B ν 4 λ2 = νx
Pi (k, P )x − λVi (k, P, V )x + Di (k, P, V, γ)+ ν +ν(u + a(x))
Ri (k, P )(u + a(x)),

showing (13) and completing the proof of the proposition.




For k = T, T − 1, . . . , 0 define (22)

P (k) = P(k, P (k + 1)),

P (T ) = (L
L, . . . , L
L),

(23)

V (k) = V(k, P (k + 1), V (k + 1)),

V (T ) = (L, . . . , L),

(24)

γ(k) = D(k, P (k + 1), V (k + 1), γ(k + 1)),

γ(T ) = 0.

Theorem 4. If (25)

¯i (k) ∈ R(Ri (k, P (k + 1))) Ei (k, V (k + 1))B

for each k = 0, 1, . . . , T − 1, then the value function J(x(k), θ(k), k) is given by  λ2  (26) J(x(k), θ(k), k) = E νx(k)
Pθ(k) (k)x(k) − λVθ(k) (k)x(k) + γθ(k) (k), ν and an optimal control law is given by (27)

 u(k) = −Rθ(k) (k, P (k + 1))† Gθ(k) (k, P (k + 1))x(k)− λ − Hθ(k) (k, V (k + 1)) . 2ν

Proof. For k = T there is no control to take, and it follows that J(x(T ), θ(T ), T ) = νE(x(T )
LL
x(T ) − λLx(T )), showing (26) from the definition of P (T ), V (T ) and γ(T ) = 0. Suppose from the induction hypothesis

30

Oswaldo L.V. Costa and Rodrigo T. Okimura

10

that (26)-(27) hold for k + 1. From the Bellman equation, (11) and (13), for x(k) = x, θ(k) = i, we have

  (28) = J(x, i, k) = infm E J(x(k + 1), θ(k + 1), k + 1)|Fk u∈R

 = infm νE x(k + 1)
Pθ(k+1) (k + 1)x(k + 1)− u∈R

 λ2 = −λVθ(k+1) (k + 1)x(k + 1) + γθ(k+1) (k + 1)|Fk ν λ2 = νx
Pi (k, P (k + 1))x − λVi (k, P (k + 1), V (k + 1))x + Di (k, P, V, γ), ν

with a minimum value reached at u(k) as in (27). Now, (22), (23), (24) and (28) complete the proof.
5. SOLUTION OF PROBLEMS

In this section we solve the three mean-variance problems stated   in  Section 3. We assume throughout this section that (25) holds. Let Π P 1 σ 2 , Π (P 2 ( )), Π (P 3 (ν)) and Π  (A (λ, ν)) denote, respectively, the set of optimal solutions for problems P 1 σ 2 , P 2 ( ), P 3 (ν) and A (λ, ν). We recall the following results proved in [5]. Proposition 5. If u ∈ Π (P 3 (ν)) and λ = 1 + 2νE (y u (T )), then u ∈ Π (A (λ, ν)). On the other hand, if u ∈ Π (A (λ, ν)) then a necessary condition for u ∈ Π (P 3 (ν)) is that λ = 1 + 2νE (y u (T )). Proposition 6. Suppose that ν ≥0 and  u∈ Π (P 3 (ν)). a) If Var (y u (T )) = σ 2 then u ∈ Π P 1 σ 2 . b) If E (y u (T )) = then u ∈ Π (P 2 ( )). We shall next derive some expressions for λ and ν such that the conditions   of Propositions 5 and 6 will be verified, yielding a solution of problems P 1 σ 2 , P 2 ( ) and P 3 (ν). For i = 1, . . . , N define (29) Ki (k) = Ri (k, P (k + 1))† Gi (k, P (k + 1)), ¯i (k)Ei (k, V (k + 1)), Ui (k) = Ri (k, P (k + 1))† B ¯ ¯ ¯i (k)Ui (k), Ci (k) = π(k)B Acl i (k) = Ai (k) − Bi (k)Ki (k),   cl p11 (k)Acl 1 (k) . . . pN 1 (k)AN (k)   .. .. .. A(k) =  , . . . cl p1N (k)Acl 1 (k) . . . pN N (k)AN (k)

11

Multi-period mean variance optimal control





N 

    ,   

pi1 (k)Ci (k)    i=1  .. V(k) =  .  N    piN (k)Ci (k)

I=



i=1

  T −1 A(
) µ(0), a=L I
=0

c=

N 

31

tr(Pi (0)Qi (0)),

i=1

1 b= L 2 d=

N 

I ... I



,

T −1  k−1     I A(
) V(t) , t=0


=t+1

Vi (0)µi (0),

e=

i=1

N 

πi (0)γi (0).

i=1

We present next an explicit formula for E(y u (T )) and Var(y u (T )) in terms of λ, a, b, c, d, e when the optimal control strategy (27) is applied to system (4). Proposition 7. Suppose that the optimal control strategy (27) is applied to system (4). Then λ b ν

E(y u (T )) = a +

(30) (31) Var(y u (T )) = c−a2 −

       b 2(d − a) λ e λ +4a− . 2 1− −b ν 2 b ν b

Proof. Using the control law (27) in (4), we get ε    cl
θ(k),s (k)Kθ(k) (k)wu (k) xu (k)+
θ(k),s (k)wx (k)− B A x (k+1) = Aθ(k) (k)+ s s u

s=1

(32)

  ε  λ ¯ u
B (k) + Bθ(k),s (k)ws (k) Uθ(k) (k). + 2ν θ(k) s=1

ziu (k)

xu (k)1{θ(t)=i} ,

= Defining µN (k)), from [2] we have µj (k + 1) =

N  i=1

and µi (k) = E(ziu (k)), µ(k)
= (µ1 (k) · · ·

pij (k)Acl i (k)µi (k) +

N λ  ¯i (k)Ui (k), pij (k)πi (k)B 2ν i=1

or, in other words, (33)

µ(k + 1) = A(k)µ(k) +

λ V(k). 2ν

32

Oswaldo L.V. Costa and Rodrigo T. Okimura

12

Iterating (33) we get (34)

µ(k) =

 k−1 
=0

  k−1  k−1  λ  A(
) µ(0) + A(
) V(t). 2ν t=0


=t+1

From (34) we have u

(35)

E(x (T )) = E

 N

 ziu (T )

i=1

=

N  i=1

=

N 

E(ziu (T )) =

i=1

 T   −1 T −1  k−1  λ  µi (T ) = Iµ(T ) = I A(
) µ(0) + I A(
) V(t). 2ν t=0
=0

y u (T )

it follows that T −1  k−1  T −1       λ λ u A(
) µ(0)+ L I A(
) V(t) = a+ b, E(y (T )) = L I 2ν ν

Since

=


=t+1

Lxu (t),

t=0


=0


=t+1

showing (30). To show (31), notice that from (26) we have (36)

  λ2 E(νy u (T )2 −λy u (T )) = E νx(0)
Pθ(0) (0)x(0)−λVθ(0) (0)x(0)+ γθ(0) (0) = ν =ν

N 

tr(Pi (0)Qi (0))−λ

i=1

N  i=1

Vi (0)µi (0)+

N λ2  λ2 πi (0)γi (0) = νc−λd+ e. ν ν i=1

Therefore, it follows from (30) and (36) that   νVar(y u (T )) = ν E(y u (T )2 ) − E(y u (T ))2 = = νE(y u (T )2 ) − λE(y u (T )) + λE(y u (T )) − νE(y u (T ))2 =      λ λ λ2 a+ b , = νc − λd + e + λ − ν a + b ν ν ν thus showing (31).




Next, we obtain the values of λ and ν such that the conditions in Propositions 5 and 6 hold in order to obtain a solution of problems P 3 (ν),   P 1 σ 2 and P 2 ( ). First, we determine the value of λ satisfying the equation λ = 1 + 2νE(y u (T )). From (30) we have   λ u λ = 1 + 2νE(y (T )) = 1 + 2ν a + b , ν

13

Multi-period mean variance optimal control

33

hence an optimal strategy u for problem P 3 (ν) is given by (27) with 1 + 2νa . (37) λ= 1 − 2b Next, we determine the value of ν such that E (y u (T )) = . From (30) and (37) we have = E(y u (T )) = a +

1+2νa 1−2b

ν

b,

hence (38)

ν=

b . (1 − 2b) − a

Finally, we determine of ν such that Var (y u (T )) = σ 2 . Define  the value     + 4a , h = b 1 − eb − b and υ = λν . It follows f = c − a2 , g = 2b 2 (d−a) b from (31) that hυ 2 − gυ + (f − σ 2 ) = 0,  1+2νa  g 2  f −σ2 g ± − . But υ = λν = 1−2b so that υ = 2h 2h h ν , so we have 1 υ(1 − 2b) − 2a with the signal in υ chosen such that ν > 0. (39)

ν=

Acknowledgements. This work was partially supported by CNPq (Brazilian National Research Council), Grant 304866/03-2, CAPES (Brazilian Ministry of Education Agency), FAPESP (Research Council of the State of S˜ ao Paulo), Grant 03/067367, IM-AGIMB, and PRONEX, Grant 015/98.

REFERENCES [1] S. Chen, X. Li and X.Y. Zhou, Stochastic linear quadratic regulators with indefinite control weight costs. SIAM J. Control Optim. 36 (1998), 1685–1702. [2] O.L.V. Costa, M.D. Fragoso and R.P. Marques, Discrete-Time Markov Jump Linear Systems. Springer-Verlag, 2005. [3] V. Dr˘ agan and T. Morozan, The linear quadratic optimization problems for a class of linear stochastic systems with multiplicative white noise and markovian jumping. IEEE Trans. Automat. Control 49 (2004), 665–675. [4] Markus Leippold, Fabio Trojani and Paolo Vanini, A geometric approach to multiperiod mean variance optimization of assets and liabilities. J. Econom. Dynamics Control 28 (2004), 1079–1113. [5] Duan Li and Wan-Lung Ng, Optimal dynamic portfolio selection: Multiperiod meanvariance formulation. Math. Finance 10 (2000), 387–406. [6] X. Li and X.Y. Zhou, Indefinite stochastic LQ controls with Markovian jumps in a finite time horizon. Comm. Inform. and Systems 2 (2002), 265–282.

34

Oswaldo L.V. Costa and Rodrigo T. Okimura

14

[7] X. Li, X.Y. Zhou and M. Ait Rami, Indefinite stochastic linear quadratic control with markovian jumps in infinite time horizon. J. Global Optim. 27 (2003), 149–175. [8] A. Lim and X.Y. Zhou, Stochastic optimal LQR control with integral quadratic constraints and indefinite control weights. IEEE Trans. Automat. Control 44 (1999), 1359– 1369. [9] Y. Liu, G. Yin, and X.Y. Zhou, Near-optimal controls of random-switching LQ problems with indefinite control weight costs. Automatica 41 (2005), 1063–1070. [10] H. Markowitz. Portfolio Selection: Efficient Diversification of Investments. Wiley, New York, 1959. [11] A.C.M. Ran and H.L. Trentelman, Linear quadratic problems with indefinite cost for discrete time systems. SIAM J. Matrix Anal. Appl. 14 (1993), 776–797. [12] A. Saberi, P. Sannuti and B.M. Chen, H2 -Optimal Control. Prentice Hall, 1995. ¨ [13] U. C ¸ akmak and S. Ozeckici, Portfolio optimization in stochastic markets. Math. Methods Oper. Res. 63 (2006), 151–168. [14] G. Yin and X.Y. Zhou, Markowitz’s mean-variance portfolio selection with regime switching: From discrete-time models to their continuous-time limits. IEEE Trans. Automat. Control 49 (2004), 349–360. [15] X.Y. Zhou and D. Li, Continuous-time mean-variance portfolio selection: A stochastic LQ framework. Appl. Math. Optim. 42 (2000), 19–33. [16] Shu-Shang Zhu, Duan Li and Shou-Yang Wang, Risk control over bankruptcy in dynamic portfolio selection: A generalized mean-variance formulation. IEEE Trans. Automat. Control 49 (2004), 447–457. Received 30 September 2006

Escola Polit´ecnica da Universidade de S˜ aoPaulo Departamento de Engenharia de Telecomunica¸co˜es e Controle 05508-900 So Paulo SP, Brazil [email protected] [email protected]