A nonconvex weighted potential function for ... - Semantic Scholar

A nonconvex weighted potential function for polynomial target following methods Report 95-127

E. de Klerk C. Roos T. Terlaky

Faculteit der Technische Wiskunde en Informatica Faculty of Technical Mathematics and Informatics Technische Universiteit Delft Delft University of Technology

ISSN 0922-5641

Copyright c 1995 by the Faculty of Technical Mathematics and Informatics, Delft, The Netherlands. No part of this Journal may be reproduced in any form, by print, photoprint, microfilm, or any other means without permission from the Faculty of Technical Mathematics and Informatics, Delft University of Technology, The Netherlands. Copies of these reports may be obtained from the bureau of the Faculty of Technical Mathematics and Informatics, Julianalaan 132, 2628 BL Delft, phone +31152784568. A selection of these reports is available in PostScript form at the Faculty’s anonymous ftp-site. They are located in the directory /pub/publications/tech-reports at ftp.twi.tudelft.nl

DELFT UNIVERSITY OF TECHNOLOGY

REPORT Nr. 95{127 A nonconvex weighted potential function for polynomial target following methods

E. de Klerk, C. Roos, T. Terlaky

ISSN 0922{5641 Reports of the Faculty of Technical Mathematics and Informatics Nr. 95{127 Delft, November 13, 1995 i

E. de Klerk, C. Roos and T. Terlaky, Faculty of Technical Mathematics and Informatics, Delft University of Technology, P.O. Box 5031, 2600 GA Delft, The Netherlands. e{mail: [email protected], [email protected], [email protected]

c 1995 by Faculty of Technical Mathematics and Copyright Informatics, Delft, The Netherlands. No part of this Journal may be reproduced in any form, by print, photoprint, micro lm or any other means without written permission from Faculty of Technical Mathematics and Informatics, Delft University of Technology, The Netherlands.

ii

Abstract Long step interior point methods in linear programming are some of the most ecient algorithms from a computational point of view. We prove polynomial complexity of a class of long step target following methods in a novel way, by introducing a new non-convex potential function and adapting the analysis framework of Jansen et al. [6, 7, 4]. The main advantage is that the new potential function has an obvious extension to semi-de nite programming, whereas the potential used in abovementioned papers does not. Key words: interior{point method, primal{dual method, target{following, Dikin steps. Running title: A nonconvex weighted potential function for polynomial target following methods.

iii

1 Introduction Medium and long step primal{dual interior{point methods in linear programming are of signi cant practical importance. Introduced by Kojima et al. [9], these methods have proven ecient in computational studies [11]. The worst-case complexity of long step algorithms with O(1) step sizes is O(n lnp1=) iterp ations, and for medium step sizes of O(1= n) one has a worst{case bound of O( n ln 1=) iterations [3, 2, 8]. Although the long step methods have a worse complexity bound than the short and medium step variants, the number of iterations performed in practice are often lower as becomes clear from the cited references. Jansen et al. [6, 7, 4] provided a unifying framework of analysis for these important algorithms. Their 'target-following' approach involves choosing a series of targets to be approximated in the primal-dual space. The underlying principle is that for each target in the positive orthant, say v 2 IRn+, there exists a unique primal dual feasible pair (x; s), i.e. Ax = b; x 2 IRn ; x  0 AT y + s = c; s 2 IRn; s  0; y 2 IRm such that1 xs = v2. Since all optimal pairs satisfy xs = 0, it is natural to choose a sequence of targets fv(j)g in the positive orthant which converges to zero, and to compute a pair (x(j); s(j)) such that x(j)s(j)  (v(j))2 for each target in the sequence v(0); v(1); : : : Denoting v2 = xs for any primal-dual pair (x; s), we can make the approximation relation `' more precise by using the proximity measure introduced by Jansen et al. in [6]:

2 2

1 (v; v) = 2 min(v)

v ?v v

: We say (v(j))2  (v(j))2 if (v(j); v(j))   for some tolerance  < 1. The pair (x(j); s(j)) is obtained by (approximately) solving the nonlinear system 8 > = b; x0 > < Ax (I) > AT y + s = c; s0 > : xs = (v(j))2 This is done iteratively by a damped Newton method, i.e. by taking damped Newton steps until the approximation condition is satis ed. The pairs (x(j); s(j)) are called outer iterates and the points generated in the Newton process will be termed inner iterates.

1 We use componentwise notation: xs indicates the vector obtained by multiplying the corresponding components of x and s, v2 is the vector obtained by squaring the components of v, etc.

1

The Newton step (
x;
s) is obtained by solving the linearized system A
x = 0 T A
y +
s = 0 x
s + s
x = (v(j))2 ? xs where the pair (x; s) is the last pair of inner iterates. A damped Newton step (
x; 
s) with   1 is used (as opposed to a full Newton step) and some care is required in choosing the step length  to ensure convergence of Newton's method. To this end, a potential function is used in the analysis of the Newton process. The idea is that a sucient reduction in the potential ensures proximity of the Newton iterates to the target v(j). The analysis therefore reduces to analysing the eect of the damped Newton steps on the potential. (In practice the potential function may be used in line searches to do larger steps than allowed for by the analysis.) It is shown that a step length   1 may be found at each step which ensures a decrease of the potential by an absolute constant. The target v(j) is updated as soon as the proximity condition is satis ed, i.e. as soon as the potential has been suciently decreased. The result is the conceptually appealing target following framework:

Target following algorithm Initialization

Given an initial feasible pair (x(0); s(0)); Let  > 0 be an accuracy parameter and  < 1 a proximity parameter; Choose an initial target v(0) such that (v(0); v(0))   . Set counter j = 0, x = x(0), and s = s(0).

While (x(j))T s(j) >  do 1. 2. 3. 4.

Solve the Newton equations (1) to obtain
x and
s. Choose a suitable damping parameter (step length)   1. p Set x = x + 
x; s = s + 
s; v = xs; If (v; v(j))   then  Let (x(j+1); s(j+1)) = (x; s);  Choose a new target v(j+1);  Set j = j + 1. 2

Enddo. The primal{dual potential function used in the papers [4, 6, 7] to determine the step length  is a strictly convex function. A new potential function is proposed in this paper which is non-convex but still suitable for the complexity analysis of long step algorithms. The advantage of the new function is that it has an obvious analogy in the semi-de nite programming case, whereas the potential used in [4, 6, 7] does not.

2 A new potential function The new potential function used in this paper is n   X f (x; s; v) = xisivi?2 ? 1 ? ln xisivi?2 ; i=1

(1)

p

de ned on the primal-dual feasible region. Using v = xs we can write (1) as n v2 2! X v i i (v; v) = 2 ? 1 ? ln vi 2 : i=1 vi

(2)

Note that (v; v)  (v; v) = 0. The proposed potential function diers from the potential used by Jansen et al. in [6, 7, 4], # "X n   1 ? 2 (3) xisi ? 1 ? vi ln xisi ; f~(x; s; v) = max(v2) i=1 in that the `weights' vi are introduced in the duality gap term in stead of in the barrier term. The corresponding potential to (3) in terms of v is n 2# 2 " v2 X v v  i i i (4) ~(v; v) = max(v2) v 2 ? 1 ? ln v 2 : i i i=1 vi 2 appear in (4) which are absent from (2). Although Notice that weighting factors max( v2 ) the new formulation seems more natural it suers from the apparent drawback that it is nonconvex, whereas f~ in (3) is a strictly convex function of x and s for xed v. Surprisingly, convexity is not a crucial issue here as the two potentials (1) and (3) have the same rst order optimality conditions: Ax = b T A y+s = c xs = v2 x; s  0 3

which is simply the relaxed LP optimality conditions (I) and known to have a unique solution (see e.g. [6]). Moreover it has already been indicated that the new potential attains its lower bound if xs = v2, proving existence of a unique minimizer of f . In other words, both potential functions have the solution of (1) as unique minimizer. 2 A xed v therefore represents a target which is approached by reducing the potential (1) using Newton's method. Once the potential has been suciently reduced, the target can be updated.

3 Reducing the potential It remains to be shown that (1) can be successfully minimized by Newton's method. The next theorem shows that a damping parameter   1 can always be found so that the damped Newton step reduces (1) by an absolute constant, determined by the current point (x; s) and target v only.

Notation: The potential reduction will be given in terms of a function  of the distance (v; v) (where no confusion is possible we will use  := (v; v)):

p

() =  + 1 + 2: We will also borrow the following notation from Jansen [4]: px = v
x x ps = v
s s 2 2 pv = px + ps = v ?v v

px ps

r =

[ v ; v ]

:

Theorem 3.1 A damped Newton step (
x; 
s) with damping parameter v )2  = 1r ? 1 kp kmax( 2 v )2  1 2 v + r max(

(5)

gives a reduction of the potential function (1), bounded by 2 4 f (x; s; v) ? f (x + 
x; s + 
s; v)  22 () + !2()! 2 2 It is interesting to note that the new function does allow a convex reformulation in terms of variables ti = P xi si =v i2 , and can be written as (t) = ni=1 i (ti ); with i (ti) = ti ? 1 ? ln ti . 4

v where ! = max min v .

Proof:

By de nition, the reduction of f is given by
f ()  f (x; s; v) ? f (x + 
x; s + 
s; v) n   X  xi  1 + 
si  = ?eT 2
x
sv?2 + x
sv?2 + 
xsv?2 + ln 1 + 
xi si i=1 ! !  pxps  n X = ?eT 2 v2 +  vv2 (px + ps ) + ln 1 + (vpx )i 1 + (vps )i i i i=1 ! ! !   n 2 2 X  ( p 1  ( p v p x )i s )i v 2 1 2 px + ps T 2 = ?e 2  v ? 2  v2 +  v2 pv + ln 1 + v 1+ v i i i=1 where e denotes the all{one vector and we have used pv = px + ps . The last term can be bounded applying the inequality n n X X ln(1 + hi)  hi + khk + ln(1 ? khk) if khk < 1 i=1 i=1 h px ps i to the combined vector h =  v ;  v . Noting that khk = r in this case, we obtain !  p v 2 1 p 2 + p 2 1 v p v T 2 2 x s
f ()  e ? 2  v + 2  v2 ?  v2 pv +  v + r + ln(1 ? r)  p v 2 1 p 2 + p 2  p v 2 ! 1 = eT ? 2 2 v + 2 2 x v2 s +  v + r + ln(1 ? r)  1   pv 2 1 p2 + p2 ! T = e  ? 2 2 v + 2 2 x v2 s + r + ln(1 ? r) kpv k2 + r + ln(1 ? r)  12  (max v)2 where the last inequality uses   1 and discards a nonnegative term. The nal expression is maximized by v )2 (6)  = 1r ? 1 kp kmax( 2 + r max( v )2 which corresponds to

2

v

!

2 2
f ()  kpv k 2 ? ln 1 + kpv k 2 : 2r(max v) 2r(max v)

(7)

The lower bound (7) on
f () is obviously nonnegative but must be bounded away from kpv k2 . zero. To accomplish this, note that expression (7) increases monotonically with 2r(max v )2 We can therefore replace this quantity by a smaller value. Jansen [4] shows that kpv k2  2! 2 ; r(max v)2 () 5

v where ! = max min v . It follows that


f ()  Using the inequality

! ! 2 ? ln 1 + ! 2 : () ()

2 x ? ln(1 ? x)  2(xx+ 1) we arrive at the bound in the theorem statement.

2

To x our ideas, we choose a threshold value to decide when the current iterate v is `close enough' to the current target v. Following Jansen, we use  = 41 as the threshold value. As long as (v; v) > 41 we perform damped Newton steps with respect to v with the following guaranteed reduction of f each time: ! 4 . Corollary 3.1 If (v; v)  41 then
f  53+11 !2

The actual reduction obtained from a linesearch is of course much larger in general. Once the proximity condition is satis ed, an upper bound on the potential is also known:

Lemma 3.1 If (v; v)  14 then (v; v)  52 . Proof: The potential  in (2) can be written as ! X n n v2 X vi2 i ln ? 1 ? (v; v) = 2 2 i=1 vi i=1 vi n n X X hi ? ln(1 + hi); = i=1

i=1

vi2 vi2

where hi = ? 1. Since



13 < 1 khk =

vv2 ve (v2 ? v2)



vv2

1 kpv k  ()

v?1

1 2 min(v) = 2() < 20 if   14 , one can use the inequality n n X X hi ? ln(1 + hi)  ?khk ? ln(1 ? khk) if khk < 1 to obtain

i=1

i=1

(v; v)  ?2() ? ln(1 ? 2()): 13 , we have (v; v)  2 . Since (v; v)  41 and consequently () < 40 5 6

2

All the tools necesserary to control the Newton process have now been developed, and we turn to the analysis of target updates.

4 Analysis of a general target update Once the current iterate v is close enough to the target v, i.e. (v; v)  41 , the target can be updated to v+. The new potential (v; v+) can be bounded from above as follows:

Lemma 4.1 Given a current iterate v, current target v, and target update v+, it holds that

(v; v+)  (v; v+) +

2 # pn max v  i  ( v; v  ) + 4  (  ) max + 2 1in 1in ( vi )

"

! 2 vi ? 1 (vi+)2

Proof: The potential after the target update can be written as n v 2 v2 ! X n n X vi2 ? X vi2 ? n i i + (v; v ) = ln ln ? 2 v + )2 vi+)2 i=1 vi2 i i=1 vi ( i=1 !( n v2 n X vi2 vi2 ? X i ?1 ln = (v; v+) + 2 (vi+)2 i=1 vi2 i=1 v"i # ! 2 n 2 2 X v  v  vi i i +  (v; v ) + 1max (v; v) + ? 1 ln + 2 in (vi+ )2 vi ) vi2 i=1 ( In [4] (Theorem 4.3.6) it is proved that the last term is bounded by ! ! n X vi2 ? 1 ln vi2  4()pn max vi2 ? 1 : 1in ( vi+)2 vi2 vi+)2 i=1 ( Substitution of this bound completes the proof.

2

By combining Corollary 3.1 and Lemma 4.1 the following result is obtained.

Lemma 4.2 If the current iterate v(j) satis es (v(j); v)  14 , and the target v is updated +

to v , then fewer than !# " " # 53 + 11!2 (v; v+) + 2 max vi2 + 13 pn max vi2 ? 1 ! 4 5 1in (vi+)2 10 1in (vi+)2 damped Newton steps with respect to v+ are required to obtain an iterate v (j+1) satisfying (v(j+1); v+)  41 . 7

5 Complexity analysis for Dikin{type target updates All that remains is to choose a target updating scheme. Consider for example the Dikintype updates introduced by Jansen et al. [5, 6, 7]: !  + 2  v = v e ? max(v)2 v (8) with 0 <  < 1=(2 + 1). Note that  = 0 corresponds to weighted path following methods. Furthermore an initial choice v(0) = e for some xed  > 0 leads to a central path following algorithm. We can bound the number of Newton steps necessary to approximate a new target v+ given (v; v)  14 by providing bounds for each of the terms in Lemma 4.2.

Lemma 5.1 Let v+ be a new target obtained by updating the old target v via (8). We then have the bounds ! 2 " 2 # 2 3 n v  1 v   (2 ? ) : i i + (v; v )  (1 ? )2 ; 1max  ; max ? 1 + + in (vi )2 (1 ? )2 1in (vi )2 (1 ? )2

Proof: We denote the largest component of v by vmax. The last two inequalities now follow from the observation vi2 = 1 : 1  + 2  2 2 2  (v ) (1 ? v =v ) (1 ? )2

The bound on

(v; v+)

i

max

i

is obtained as follows ! n n v2 2 X X v  i i + (v; v )  vi+ )2 ? i=1 ln (vi+)2 ? 1 i=1 ( 1 0 n n X XB 1 1 C ln = ? @ 2  2  2  v i2 2 ? 1A (1 ?  v  = v  ) i max (1 ? vmax i=1 i=1 2 ) 2 0 13 ! n 2 !2 2 ! 2 2 X 1  v   v   v  i 4 @ln 1 ? 2i ? 1 ? 2i A5 = vi2 2 1 + 2 1 ? v2 vmax vmax max i=1 (1 ? vmax 2 ) Using ln(1 ? x)  ?x if x < 1 and simplifying, we have 3 2 !2 !2 n 2  2  2 4  2  X ?vi + 2 vi ?  vi 5 1 vi 4 (v; v+)  2 2 2 4 vi2 2 2 1 ? vmax vmax vmax vmax i=1 (1 ? vmax 2 ) n 2 v4 3 v6 ! 2 X 1   i  (1 ? )2 3 v4 ? 2 v6i  (13?n)2 : max max i=1 8

2

We now have a bound on how many damped Newton steps are required to reach a new target:

Corollary 5.1 Assume that (v; v)  14 and that the target is updated to v+ using the target updating scheme (8). At most

!

  O (!1+ )4 n2 + pn

+

v ) damped Newton steps are needed to approximate v+ , where ! + = max( min(v+ ) .

The last question is how many target updates are required to obtain an {approximate solution. It is simple to prove the following (see [4]):

Lemma 5.2 Given a primal-dual starting pair (x(0); s(0)). Choose the rst target as v(0) = (0)

v . After at most

(0))T s(0) ! 1 ( x O !2 ln  0 target updates using (8) the algorithm terminates with a primal dual pair (x; s ) such that (x)T s  .

Combining these results, we obtain the complexity bound for the complete algorithm:

Theorem 5.1 The target following algorithm requires at most pn (x(0))T s(0) ! n + O !2+4 ln  0

damped Newton steps for convergence.

A large target update with  = O(1) therefore requires fewer than O(n=!02+4p) Newton p steps, whereas medium step methods with  = O(1= n) require fewer than O( n=!02+4 ) steps. These complexity bounds are the same as those obtained using the standard convex potential function. We conclude that the nonconvex potential (2) is a proper alternative to the usual convex logarithmic barrier potential.

6 Further work It has already been mentioned that the new potential function (1) has an extension to the semi-de nite programming (SDP) case. The recent revival of interest in SDP started 9

more or less with the work of Alizadeh [1], and an excellent review on developments and applications up to 1995 is given by Vanderberghe in [13]. One reason for the recent interest in SDP is that most interior point methods for LP can be extended to SDP. This is presently an active research area, as can be seen by the number of recent publications (see e.g. [12, 14, 10]). The general semi-de nite problem can be formulated as minTr(CX ) Tr(AiX ) = bi i = 1; : : : ; m X0 where the Ai's and C are symmetric matrices and `' denotes positive semi-de niteness. The optimality conditions for the semi-de nite problem are Tr(AiX ) = bi i = 1; : : : ; m Pm y A + S = C i=1 i i XS = 0 X; S  0: Note that the potential function   f (X; S; V ) = Tr(XS V ?2) ? ln det XS V ?2 ? n: is a natural extension of the new LP potential (1) to the semi-de nite case, where V is a symmetric positive de nite `target matrix'. To the best of our knowledge this is the rst weighted potential function for semi{de nite programming. Extension of the analysis in the previous sections would therefore broaden the target following framework to semi-de nite programming. This is the subject of further research.

References [1] F. Alizadeh. Combinatorial optimization with interior point methods and semi{ de nite matrices. PhD thesis, University of Minnesota, Minneapolis, USA, 1991. [2] C.C. Gonzaga. Large steps path{following methods for linear programming, Part I: Barrier function method. SIAM Journal on Optimization, 1:268{279, 1991. [3] D. den Hertog. Interior point approach to linear, quadratic and convex programming, Algorithms and complexity. Kluwer Publishers, Dordrecht, The Netherlands, 1994. [4] B. Jansen. Interior Point Techniques in Optimization. PhD thesis, Delft University of Technology, Delft, The Netherlands, 1995. (To be published). 10

[5] B. Jansen, C. Roos, and T. Terlaky. A polynomial primal{dual Dikin{type algorithm for linear programming. Technical Report 93{36, Faculty of Technical Mathematics and Computer Science, Delft University of Technology, Delft, The Netherlands, 1993. (To appear in Mathematics of Operations Research). [6] B. Jansen, C. Roos, T. Terlaky, and J.-Ph. Vial. Primal{dual target{following algorithms for linear programming. Technical Report 93{107, Faculty of Technical Mathematics and Computer Science, Delft University of Technology, Delft, The Netherlands, 1993. [7] B. Jansen, C. Roos, T. Terlaky, and J.-Ph. Vial. Long{step primal{dual target{ following algorithms for linear programming. Technical Report 94{46, Faculty of Technical Mathematics and Computer Science, Delft University of Technology, Delft, The Netherlands, 1994. [8] B. Jansen, C. Roos, T. Terlaky, and J.-Ph. Vial. Primal{dual algorithms for linear programming based on the logarithmic barrier method. Journal of Optimization Theory and Applications, 83:1{26, 1994. [9] M. Kojima, S. Mizuno, and A. Yoshise. A primal{dual interior point algorithm for linear programming. In N. Megiddo, editor, Progress in Mathematical Programming: Interior Point and Related Methods, pages 29{47. Springer Verlag, New York, 1989. [10] M. Kojima, M. Shida, and S. Shindoh. Global and local convergence of predictorcorrector infeasible{interior{point algorithms for semide nite programs. Technical Report B-305, Department of Mathematical and Computing Sciences, Tokyo Institute of Technology, Tokyo, Japan, 1995. [11] I.J. Lustig, R.E. Marsten, and D.F. Shanno. Interior point methods : Computational state of the art. ORSA Journal on Computing, 6:1{15, 1994. [12] R.D.C. Monteiro. Primal-dual algorithms for semide nite programming. Working Paper, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, USA, 1995. [13] L. Vanderberghe. Semide nite programming. Technical Report, Information Systems Laboratory, Electrical Engineering Department, Stanford University, Stanford CA 94305, California, 1994. (To be published in SIAM Review). [14] Y. Zhang. On extending primal-dual interior point algorithms from linear programming to semide nite programming. Technical Report, Department of Mathematics and Statistics, University of Maryland at Baltimore County, Baltimore, USA, 1995.

11