2010 IEEE International Symposium on Computer-Aided Control System Design Part of 2010 IEEE Multi-Conference on Systems and Control Yokohama, Japan, September 8-10, 2010
Markov Chain Monte Carlo method exploiting barrier functions with applications to control and optimization B.T. Polyak and E.N.Gryazina Abstract— In previous works the authors proposed to use Hit-and-Run (H&R) versions of Markov Chain Monte Carlo (MCMC) algorithms for various problems of control and optimization. However the results are unsatisfactory for ”bad” sets, such as level sets of ill-posed functions. The idea of the present paper is to exploit the technique developed for interior-point methods of convex optimization, and to combine it with MCMC algorithms. We present a new modification of H&R method exploiting barrier functions and its validation. Such approach is well tailored for sets defined by linear matrix inequalities (LMI), which are widely used in control and optimization. The results of numerical simulation are promising.
I. I NTRODUCTION Randomized methods for control and optimization become highly popular [16], [19]. They often use modern versions of Monte Carlo technique, based on Markov Chain Monte Carlo (MCMC) approach [5], [6]. The examples of such MCMC methods as Hit-and-Run and Shake-and-Bake applied to various control and optimization problems are provided in [13], [14]. However the results are unsatisfactory for ”bad” sets, such as level sets of ill-posed functions. The idea of the present paper is to exploit the technique, developed for interior-point methods of convex optimization [12], and to combine it with MCMC algorithms. Suppose a convex set Q and a corresponding selfconcordant barrier F (x) are given. The problem is to generate points xi approximately uniformly distributed in Q. The method uses H&R and local geometry of Q at the point xi provided by Hessian of F (xi ). We construct Dikin ellipsoid centered at xi and choose random direction uniformly distributed in this ellipsoid. The stepsize in this direction is made according to boundary oracle [13], [14] and Metropolis-like rule. Similar ideas are used in recent paper [9] for polyhedral sets Q; the detailed comparison with algorithm [9] is given below. The proposed approach is well tailored for sets Q defined by linear matrix inequalities (LMI) [3]. Some applications to control and optimization are provided. II. H IT- AND -RUN We start with presenting the idea of H&R method in general setting. Suppose there is a bounded closed set Q ⊂ Rn and a point x0 ∈ Q. At every step we choose a random vector d uniformly distributed on the unit sphere in Rn . We call boundary oracle (BO) an algorithm that provides the B.T. Polyak (
[email protected]) and E.N. Gryazina (
[email protected]) are with Institute for Control Sciences RAS, Moscow, Russia
978-1-4244-5355-9/10/$26.00 ©2010 IEEE
intersection of the straight line x0 + td, −∞ < t < +∞ with Q, i.e., the set T = {t ∈ R : x0 + td ∈ Q}.
In the simplest case, when Q is convex, this set is the interval [t, t], where t = max{t : x0 +td ∈ Q}, t = min{t : x0 +td ∈ t>0
t 1010 2 ln , r ε where constants R, r and M are characteristics of the set, n is the dimension. Of course this estimate looks hopeless even for moderate n and R/r; indeed, the examples below demonstrate that standard H&R method generates points that strongly differ from uniformly distributed ones. The attempts to improve the method are far from successful [18]. III. E XPLOITING
METHOD
An ellipsoid specified by Hx = ∇ F (x) and centered in x ∈ Int(Q) 2
Ex = {y : (Hx (y − x), y − x) ≤ 1}
(1)
is called Dikin ellipsoid, it can be proved that Ex ⊂ Q [11]; see Fig. 2. In some cases Dikin ellipsoid is the restriction of ellipsoid (1) on other linear constraints (subspace L in Example 3).
BARRIER FUNCTIONS
The H&R routine is based on two random variables: step direction d uniformly distributed on the unit sphere and step size t uniformly distributed in T . We propose a specific local choice of the direction distribution and discuss the properties of the obtained distribution of the sampled points. Suppose Q is a convex set. F (x) is called a barrier function for Q if it is convex, defined for all x ∈ Int Q and F (x) → ∞ as x → ∂Q. We assume additionally that F (x) is self-concordant [12], in particular F (x) is three times differentiable. Such barrier functions are widely used in interior-point methods for convex optimization, their explicit expressions are well known for numerous sets Q. Below we provide typical convex sets Q and their barriers F (x) as well as their second derivatives, which are required for the calculations. Example 1. For polyhedral sets in Rn Q = {x ∈ Rn : (ai , x) ≤ bi , the barrier function is F (x) = −
i = 1, . . . , m}
m X
log(bi − (ai , x)), its X ai gradient and Hessian are ∇F (x) = and 1 − (ai , x) T X ai ai ∇2 F (x) = . (1 − (ai , x))2 Example 2. LMI in standard format, Ai are symmetric n × n matrices: Ai ∈ Sn×n , i=1
Q = {x ∈ R` : A(x) = A0 +
` X i=1
xi Ai 0},
F (x) = −log detA(x),
∇2 F (x)ij = trA(x)−1 Ai A(x)−1 Aj . Example 3. LMI with matrix variables; typical example is Q = {X 0,
IV. BARRIER M ONTE C ARLO
X ∈ L ⊂ Sn×n }.
F (X) = −log det X; here L is a linear subspace of the space of symmetric matrices Sn×n equipped with Frobenius norm. Then ∇F (X) = −X −1 , h∇2 F (X)Y, Y i = hX −1 Y X −1 , Y i.
Fig. 2.
Dikin ellipsoid for a point of a convex set.
Let us choose direction uniformly distributed in Dikin ellipsoid, then apply BO and choose a step size as in the H&R method. Since the direction distribution varies from point to point one can’t guarantee asymptotical uniformity of the points. To obtain the desired property we add Metropolislike rule: we generate a candidate point y, calculate Ey and either accept the point as the new one or stay at x and generate a new candidate point. Finally, Barrier Monte Carlo (BMC) method operates as follows: 1. Starting point x0 ∈ Int Q is given; i = 0, x = x0 . 2. Pick random direction d uniformly in Ex : d = Hx−1/2 ξ,
ξ is uniform random in the unit ball.
3. Pick y = x + td, where t ∈ [t, t] is defined via Boundary Oracle as in the original version of H&R. ||d|| 4. Calculate rx = p , Hy = ∇2 F (y) and ry = (Hx d, d) ||d|| p . (Hy d, d) 5. Accept y with probability ! n r ry det Hy , i = i + 1, xi = y. α = min 1, rx det Hx Go to 2. Theorem 1: The distribution of sampled points xi tends to uniform on Q. Proof. In view of Theorem 1 in [17] it suffices to prove that p(y|x) > 0 for all x, y ∈ IntQ and p(y|x) = p(x|y), where p(y|x) is probability density of transition to y from x.
1554
the long axis of it. Fig. 4 demonstrates this ”bad” behavior of H&R for R2 − stick. There is no movement along xn for H&R. In contrast, the distribution of BMC points is close to uniform. It is not hard to prove that BMC is affine invariant, i.e., its sampling for the stick and the cube is the same.
Validation of BMC method.
The first condition is obvious; to prove the second consider small balls Bx , By centered at x and y with radii ε (Fig. VolSx 3). The probability P (x → By ) = 2 , where Sx is VolEx the intersection of the cone {x + λ(Bx − x), λ ≥ 0} with ellipsoid Ex . The volume of Sx is proportional to rxn , rx ||d|| being the height of Sx : rx = . Thus (Hx d, d)1/2 √ ||d||n detHx P (x → By ) = c · αxy = ϕx αxy , (Hx d, d)n/2 where αxy is the probability to accept y as the next iteration. Similarly, p ||d||n detHy P (y → Bx ) = c · αyx = ϕy αyx , (Hy d, d)n/2 where αyx is the probability accept x starting to fromy. ϕy ϕx Choosing αxy = min 1, , αyx = min 1, we ϕx ϕy get P (x → By ) = P (y → Bx ). V. C OMPARISON
Fig. 4. ”Stick” in R2 : magenta – H&R method, blue points – BMC method.
Method of Kannan and Narayanan [9] similarly to BMC exploits directions uniformly distributed in Dikin ellipsoids (for polyhedral sets Q only). However step size strategy is completely different. New point y is always close enough 3 to x (in Ex ), thus step sizes are small and it leads to 40 strong serial correlation (correlation between xi and xi+1 is large). For BMC method stepsizes are much larger and serial correlation is weaker. Fig. 5 draws the trajectory of sampled points in the simplest case when Q is 1D interval [0, 1]. Small step sizes for the method [9] cause strong correlation – in average it requires more than 1000 iterations for xi to escape half-interval [0, 0.5] starting close to zero (upper trajectory). In contrast, xi+1 and xi are not correlated for BMC in this example (lower trajectory).
0.6 0.4
WITH KNOWN METHODS
0.2 0
The BMC method is much more flexible than H&R with respect to local geometry of Q at x. Indeed, the direction d in H&R is uniformly distributed in the ball independently of x. For ”thin and long” sets such as a ”stick” n
Q = {x ∈ R : |xi | ≤ ε, i = 1, . . . , n−1, |xn | ≤ 1},
Scalar case Q=[0,1]
1 0.8
x
Fig. 3.
500
1000
1500
2000
2500 i
3000
3500
4000
4500
5000
0
500
1000
1500
2000
2500 i
3000
3500
4000
4500
5000
0.8
x
0.6 0.4 0.2 0
ε 1
d is directed mostly perpendicular to the long axis (en = (0, . . . , 0, 1) for the stick). Thus the step size will be small; the same jamming situation holds in the corners of polytopic Q. The behavior of BMC is strongly different. The ellipsoid Ex copies the local shape of Q at x, and most d are along
0
1
Fig. 5.
Iterations for 1D interval [0, 1].
However the authors of [9] provide theoretical estimates for the rate of convergence to uniform distribution while for our method such results are lacking.
1555
VI. I MPLEMENTATION ISSUES Let us estimate complexity of each iteration. Calculation of Hx = ∇2 F (x) is not hard, see examples in Section III. Generation of points uniformly in Ex in most cases can be done avoiding calculation of Hx−1/2 . Indeed, for Example 1 1 T 2 Hx = A D A, D = diag . bi − (ai , x)
Let ξ ∼ N (0, I) = randn(n, 1). Solve Hx η = DAξ, get η as a desired direction. Thus it suffices to d= (Hx η, η)1/2 solve one linear equation at each iteration. Similar approach is proposed in [9]. Other tricks can be used in Example 2; we omit their description. Example 3 demonstrates a bit different situation, when Dikin ellipsoid is an intersection of the ellipsoid Ex = {y : (Hx (y − x), y − x) ≤ 1} with the subspace L. In this case we can transform Ex into a ball, generate η uniformly in the ball and find its projection on the subspace; coming back to the original space we obtain (after normalization) the point uniformly distributed in Dikin ellipsoid. Important question relates to the number of candidate points y to generate before we accept one of them as the next iteration. The theoretic estimate of the average number of attempts N can be done for simple sets such as a ball or a cube (and their affine images – ellipsoids and boxes). These particular examples supported by simulation for various sets conjecture the general estimate O(1) N= √ , n where O(1) is close to 1 for most cases. For instance, if n = 100, calculations at steps 2–5 of the algorithm should be repeated approximately 10 times. Boundary Oracle calculations in most cases are simple enough, see [13], [14]. For instance, LMI sets (Example 2) require to compute a generalized eigenvalue of a matrix pair. Finally, all other calculations at each iteration are “cheap”. VII. A PPLICATION
min(∇f (xi ), x)
x∈Q
and can be solved fast. There exist the huge literature on deterministic methods for concave optimization; see, e.g., [8]; we conjecture that the proposed randomized methods could be good competitors with them. Moreover it is possible to estimate the complexity of such methods similarly to estimates in [4] for linear optimization. VIII. A PPLICATIONS
IN CONTROL
There are many situations in control theory when sampling of systems with prescribed properties is of interest. For instance, in the hard problem of static output feedback control the set of stabilizing controllers can be characterized with some conditions, and natural method requires generation of state feedback stabilizing controllers and checking these conditions. The latter controllers can be described by LMIs, thus we need random points in LMI set. Such approach demonstrated its efficiency in [2]. There are numerous problems of robust stability analysis and design which can be solved by generation of admissible uncertainties. The examples can be found in [7], [13], [14]. Finally, semidefinite programming is the standard tool in control [3], and we have seen that random methods can be effective for its solution. In all mentioned above cases (as well as in many others) the efficient generation of random points based on barrier functions will be helpful. There exist also hard design problems which can be converted to Bilinear Matrix Inequalities (BMI), see [21]. Some of them can be written as concave optimization problems as described in [1]. Thus the technique from the previous Section can be effectively applied. IX. C ONCLUSIONS
TO OPTIMIZATION
In our previous works we developed different cutting plane methods for convex optimization, exploiting Monte Carlo techniques [4], [13], [14], [15]. They were based on H&R algorithm for generating points in Q. Of course they can be strongly improved if we replace standard H&R with more advanced MBC method proposed above. Numerical simulation confirms this conclusion. However we’ll focus on applications to nonconvex problems because it is hard to compete with powerful interior-point deterministic method [12]. The general approach to nonconvex optimization min f (x),
technique, for instance, we can choose the most perspective points xi (with top values of f (xi )) and apply local search just for them. Such algorithms are well tailored for concave f (x) and polyhedral Q. The local search problems are linear programming ones:
We introduce the new approach to Markov Chain MonteCarlo methods for generation of random points in a convex set. It links these methods with barrier functions, used in convex optimization. The random points produced by the new method are preferable in comparison with standard methods such as H&R. For optimization purposes the method can be considered as a random version of the interior-point method applied for nonconvex problems. We don’t provide here numerous examples of simulation for applications in optimization and control; our goal was to clarify the idea of the approach. R EFERENCES
x∈Q
where f (x) is a nonconvex function while Q is a convex set with known Boundary Oracle and barrier function looks as follows. We apply sampling in Q as above and local descent from these points. There are many versions of such
1556
[1] P. Apkarian and H.D. Tuan. Concave programming in control theory. Journal of Global Optimization, 15:343–370, 1999. [2] D. Arzelier, E. Gryazina, D. Peaucelle, and B. Polyak. Mixed lmi/randomized methods for static output feedback control design. In Proceedings of the IEEE American Control Conference, Baltimore, 2010.
[3] S. Boyd, L. El Ghaoui, E. Ferron, and V. Balakrishnan. Linear Matrix Inequalities in System and Control Theory. SIAM, Philadelphia, 1994. [4] F. Dabbene, P.S. Shcherbakov, and B.T. Polyak. A randomized convex optimization method with probabilistic geometric convergence. In Proceedings of the IEEE Conference on Decision and Control, Cancun, Mexico, 2008. [5] P. Diaconis. The markov chain monte carlo revolution. Bull. of the AMS, 46(2):175–205, 2009. [6] W. Gilks, S. Richardson, and D. Spiegelhalter. Markov Chain Monte Carlo. Chapmen and Hall, 1996. [7] E. Gryazina and B. Polyak. Robust stabilization via hit-and-run techniques. In Proceedings of the 3rd IEEE Multi-conference on Systems and Control, St-Peterburg, Russia, 2009. [8] R. Horst and H. Tuy. Global Optimization: Deterministic Approach. Springer, Berlin, 1996. [9] Kannan and Narayanan. Random walks on polytopes and an affine interior point method for linear programming. In Proceedings of the ACM Theory of Computing, Taipei, Taiwan, 2009. [10] L. Lovasz and S. Vempala. Hit-and-run from a corner. In Proceedings of the 36th annual ACM symposium on Theory of computing, pages 310–314, Chicago, IL, USA, 2004. [11] Y. Nesterov. Introductory Lectures on Convex Optimization: a Basic Course. Kluwer Academic Publishers, 2004. [12] Y. Nesterov and A. Nemorivsky. Interior Point Polynomial Methods in Convex Programming. SIAM, Philadelphia, 1994. [13] B. Polyak and E.N. Gryazina. Randomized methods based on new monte carlo schemes for control and optimization. Annals of Operational Research, page in press, 2009. [14] B.T. Polyak and E.N. Gryazina. Hit-and-Run: New design technique for stabilization, robustness and optimization of linear systems. In Proceedings of the IFAC World Congress, pages 376–380, 2008. [15] B.T. Polyak and P. Shcherbakov. Randomized algorithm for solving semidefinite programs. In Stochastic optimization in informatics, pages 3–37, Saint-Peterburg, Russia, 2006. [16] R.Y. Rubinstein and D.P. Kroese. Simulation and the Monte Carlo Method. Wiley, NJ, 2008. [17] R.L. Smith. Efficient monte carlo procedures for generating points uniformly distributed over bounded regions. Operations Research, 32(6):1296–1308, 1984. [18] R.L. Smith. Direction choice for accelerated convergence in hit-andrun sampling. Operations Research, 46(1):84–95, 1998. [19] R. Tempo, G. Calafiore, and F. Dabbene. Randomized Algorithms for Analysis and Control of Uncertain Systems. Communications and Control Engineering Series. Springer-Verlag, London, 2004. [20] V. F. Turchin. On the computation of multidimensional integrals by the monte carlo method. Theory of Probability and its Applications, 16(4):720–724, 1971. [21] J.G. VanAntwerp and R.D. Braatz. A tutorial on linear and bylinear matrix inequalities. Journal on Process Control, 10(4):363–385, 2000.
1557
