On primal-dual path-following algorithms in semidefinite programming Report 96-102
E. de Klerk C. Roos T. Terlaky
Faculteit der Technische Wiskunde en Informatica Faculty of Technical Mathematics and Informatics Technische Universiteit Delft Delft University of Technology
ISSN 0922-5641
Copyright c 1996 by the Faculty of Technical Mathematics and Informatics, Delft, The Netherlands. No part of this Journal may be reproduced in any form, by print, photoprint, microfilm, or any other means without permission from the Faculty of Technical Mathematics and Informatics, Delft University of Technology, The Netherlands. Copies of these reports may be obtained from the bureau of the Faculty of Technical Mathematics and Informatics, Julianalaan 132, 2628 BL Delft, phone +31152784568. A selection of these reports is available in PostScript form at the Faculty’s anonymous ftp-site. They are located in the directory /pub/publications/tech-reports at ftp.twi.tudelft.nl
DELFT UNIVERSITY OF TECHNOLOGY
REPORT Nr. 96{102 On Primal{Dual Path{Following Algorithms for Semidefinite Programming
E. de Klerk, C. Roos, T. Terlaky
ISSN 0922{5641 Reports of the Faculty of Technical Mathematics and Informatics Nr. 96{102 Delft, October 9, 1996 i
E. de Klerk, C. Roos and T. Terlaky, Faculty of Technical Mathematics and Informatics, Delft University of Technology, P.O. Box 5031, 2600 GA Delft, The Netherlands. e{mail:
[email protected],
[email protected],
[email protected]
c 1996 by Faculty of Technical Mathematics and Copyright Informatics, Delft, The Netherlands. No part of this Journal may be reproduced in any form, by print, photoprint, micro lm or any other means without written permission from Faculty of Technical Mathematics and Informatics, Delft University of Technology, The Netherlands.
ii
Abstract Interior point methods for semide nite programming have recently been studied intensively, due to their polynomial complexity and practical eciency. Most of these methods are extensions of linear programming algorithms. The primaldual central path following method for linear programming by Jansen et al. [8] has recently been extended to semide nite programming by Jiang [9], utilizing the Nesterov-Todd direction and introducing a new distance measure. In this note we re ne and extend this analysis: A weaker condition for a feasible full Newton step is established, and quadratic convergence to target points on the central path is shown. Moreover, we show how to compute large dynamic target updates which still allow full Newton steps. Key words: interior{point method, primal{dual method, path{following, semide nite programming. Running title: Path{Following Methods for SDP.
iii
1 Introduction Primal-dual path following methods have emerged as the most successful interior point algorithms for linear programming (LP). Predictor-corrector methods are particularly in favour, following successful implementations [13]. The extension of algorithms from linear to semide nite programming (SDP) has followed the same trends, and recent attention has focused on path following algorithms. Primal path following algorithms are studied in [5] (general analysis), [2] (long step method), and [7] (full Newton step method). The primal-dual path following methods employ di erent search directions which arise from di erent linearization strategies of the relaxed optimality conditions. A comparison of the best known search directions can be found in [1] and [18]. One of the popular primal-dual directions is the so-called Nesterov-Todd (NT) direction, introduced in [15]. (Recently Kojima et al [10] showed the NT direction to be a special case of the primal{dual directions for monotone semide nite complementarity problems introduced in [11].) A long step primal-dual path following method using the NT direction was recently presented by Jiang [9]. This paper is an extension of the LP analysis by Jansen et al. [8]. This method targets a speci c point on the primal-dual central path, which is then updated if the current iterates are `close enough' to it. The novelty of this analysis lies in the use of a new centrality measure, which is analogous to the LP measure in [8]. In this paper we extend and re ne this analysis { a weaker sucient condition for full Newton steps is derived, and quadratic convergence to target points is proved. Moreover, a dynamic target updating schemes is discussed. The aim is to obtain a method which uses only full Newton steps, and uses large target updates to reduce the iteration count. A worst-case iteration complexity bound of O(pn) of the resulting algorithm is proved.
Preliminaries
The semide nite programming problem is studied in the standard form. Thus the primal problem (P) is given by: minTr(CX ) subject to Tr(AiX ) = bi; i = 1; : : : ; m X 0 and its dual problem (D) is: max bT y subject to m X i=1
yiAi + Z = C 1
Z 0 where C and the Ai's are symmetric n n matrices, b; y 2 IRm and `X 0' means X is symmetric positive semide nite. The matrices Ai are further assumed to be linearly independent. We will assume that a strictly feasible pair (X 0; Z 0) exists. This ensures the existence of an optimal primal-dual pair (X ; Z ) and a corresponding zero duality gap (Tr(X Z ) = 0). The optimality conditions for the pair of dual problems are Tr(AiX ) = bi; i = 1; : : : ; m m X y i Ai + Z = C i=1 XZ = 0 X; Z 0: If these conditions are relaxed to Tr(AiX ) = bi; i = 1; : : : ; m m X y i Ai + Z = C i=1 XZ = I X; Z 0 for some > 0, then a unique solution of the relaxed system exists, denoted by fX (); y(); Z ()g. This solution gives a parametric representation of the central path as a function of .1 The existence and uniqueness of fX (); y(); Z ()g follow from the fact that fX (); y(); Z ()g corresponds to the unique minimum of the strictly convex primal{dual barrier function f (X; Z; ) = 1 Tr(XZ ) ? ln det(XZ ) (1) de ned on the primal{dual feasible region. Because of the two di erent associations, the parameter is called either the barrier parameter, or the centering parameter. We will also refer to as a target parameter, since successive points on the central path will be `targeted'.
The Nesterov-Todd search directions For a primal feasible X 0 and dual feasible Z 0, the scaling matrix 1
D := Z ? 21 Z 12 XZ 12 2 Z ? 12 ;
(2)
1 The central path is an analytic curve which converges to the analytic center of the optimal face as ! 0; for
details, see [6].
2
satis es D?1 X = ZD, or
L?1 XL?T = LT ZL := V; where D = LLT is a factorization of D. In other words, the matrix D may be used to scale the variables X and Z to the same symmetric positive de nite matrix. The matrix V depends on the factorization of D. This scaling was introduced by Nesterov and Todd [15], and its properties were further investigated by Sturm and Zhang [17] who used it to provide a framework for a number of central path following methods. The scaling seems to be a natural extension of that used in linear programming, and was used by De Klerk et al. [4] to generalize primal{dual ane scaling methods from LP to SDP. Letting `' denote the similarity relation, one has V 2 = L?1 XZL XZ; i.e. V 2 has the same eigenvalues as XZ and is symmetric positive de nite. As a consequence the duality gap is given by X Tr(XZ ) = Tr(V 2) = kV k2 = 2i (V ); i
where the norm is the Frobenius norm. Any pair of primal{dual search directions X and Z must satisfy Tr(AiX ) = 0; i = 1; : : :; m Z 2 span fA1; : : :; Amg : One can scale X and Z in same way as X and Z to obtain: DX := L?1XL?T and DZ := LT ZL: The scaled directions DX and DZ are orthogonal by the orthogonality of X and Z , i.e. Tr(DX DZ ) = 0. The scaled Newton step is de ned by DV := DX + DZ . After a feasible primal{dual step X , Z the duality gap becomes Tr ((X + X )(Z + Z )) = Tr ((V + DX )(V + DZ )) = Tr(V 2 + V DV ): The Newton equations in the V-space may be derived by requiring: 1 h(V + D )(V + D ) + ((V + D )(V + D ))T i = I; X Z X Z 2 3
which is linearized by neglecting the cross terms DX DZ and DZ DX to obtain 1 ((D + D )V + V (D + D )) = I ? V 2: (3) X Z 2 X Z Equation (3) (called a Sylvester equation) has a unique symmetric solution [17], given by DV = V ?1 ? V: Pre and postmultiplying with L and LT respectively yields the Nesterov-Todd (or NT) equations: DZD + X = Z ?1 ? X subject to Tr(AiX ) = 0; i = 1; : : :; m Z 2 span fA1; : : :; Amg :
A measure of centrality
In a recent paper Jiang [9] uses (up to the constant 21 ) the measure
(X; Z; ) = 21 p1 kDV k = 12
pV ?1 ? p1 V
: This measure generalizes the LP measure of Jansen et al [8] to semide nite programming, and will be used in this paper. It is shown in [9] that (X; Z; ) is the absolute value of the directional derivative of the primal-dual barrier (1) along the NT direction. In this sense it is a natural centrality measure associated with the NT direction.
The target following approach
For a given value of , the point I is regarded as a target point with associated target duality gap n. In other words if the unique pair (X (); Z ()) is computed then the duality gap is equal to n. Target following algorithms iteratively compute (X (); Z ()) approximately, followed by a decrease in the value of . The algorithms presented here all t in the following framework:
4
Generic Primal-Dual Logarithmic Barrier Algorithm Input: a strictly feasible pair (X0; Z0); parameters < 1 and 0 > 0 such that (X0; Z0; 0) ;
an accuracy parameter > 0.
begin
X := X0 ; Z := Z0 ; = 0 while Tr(XZ ) > do
begin
compute the NT directions X , Z with respect to ; X := X + X ; Z := Z + Z ; compute an update parameter (default := 2p1 n ); := (1 ? );
end end
2 Feasibility of the Newton step One can now prove the following two results which are analogous to the LP case: If < 1 then the Newton step is feasible, and the duality gap after the step attains its target value. To this end we need the following three results from De Klerk et al. [4].
Lemma 2.1 (General condition for a feasible primal-dual step) Let X ( ) := X + X; Z ( ) := Z + Z: If one has
det (X ( )Z ( )) > 0 8 0 ; then X ( ) 0 and Z ( ) 0.
5
2
Lemma 2.2 The spectral norm of DXZ := 21 (DX DZ + DZ DX ) is bounded by kDXZ k2 41 kDX + DZ k2:
2
Lemma 2.3 Let Q be an n n real symmetric matrix, S an n n real skew{symmetric matrix. One has det(Q + S ) > 0 if Q 0. 2 We now prove the feasibility of the Newton step in the following lemma. The condition (X; Z; ) < 1 in the lemma is a signi cant improvement over the corresponding condition (X; Z; ) < 2p1 2 derived by Jiang [9].
Lemma 2.4 If := (X; Z; ) < 1 then the full Newton step is strictly feasible. Proof: We show that the determinant of X ( )Z ( ) remains positive for all 1. One then has X (1); Z (1) 0 by Lemma 2.1. To this end note that X ( )Z ( ) (V + DX )(V + DZ ) = V 2 + DX V + V DZ + 2DX DZ = V 2+ (I ? V 2) + 2 (DXZ ) + 21 2(DX DZ ? DZ DX ) + 21 (DX V + V DZ ? V DX ? DZ V ) ;
using the Newton equation (3). The matrix in square brackets in the last equation is skew{symmetric. Lemma 2.3 therefore implies that the determinant of [X ( )Z ( )] will be positive if the matrix M ( ) := V 2 + (I ? V 2) + 2DXZ is positive de nite. Since we can rewrite the expression for M ( ) as " # 2 M ( ) = (1 ? )V + I + DXZ ; one will have M ( ) 0 if 1 and k(DXZ =)k2 < 1. The last condition is easily shown to hold by using Lemma 2.2 and < 1 successively: kDXZ =k2 = 1 kDXZ k2 41 kDV k2 = 2 < 1: 6
2
This completes the proof.
The following result is analogous to the LP-case, and is useful in constructing target updating schemes.
Corollary 2.1 The target duality gap is attained after one full Newton step. Proof: Since
1 1 X (1)Z (1) I + DXZ + 2 (DX DZ ? DZ DX ) + 2 (DX V + V DZ ? V DX ? DZ V ) (4) one has Tr (X (1)Z (1)) = Tr(I ) by using the orthogonality of DX and DZ and the skew symmetry of the matrix in square brackets. 2
Notation:
In what follows we denote the skew-symmetric matrix in (4) by S . As we will only work with full Newton steps, i.e. = 1, it will also be convenient to write X + := X (1), Z + := Z (1), etc.
3 Quadratic convergence to the target We proceed to prove quadratic convergence to the target I . To this end we need three technical results which give information concerning2 the spectrum of X + Z + . We denote the symmetrical transformation of X + Z + by (V +) .
Lemma 3.1 One has
min
2 + (1 ? 2); V
where min denotes the smallest eigenvalue.
Proof:
From (4) it follows that has
min V +
2
= min (I + DXZ + S ) : 7
The skew-symmetry of S implies
min V +
2
min (I + DXZ ) ? kDXZ k2 :
Substitution of the bound for kDXZ k2 from Lemma 2.2 now yields: 2 + ? 41 kDV k2 = 1 ? 2 ; min V which completes the proof.
2
Lemma 2.2 gave a bound on the spectral norm of DXZ . We now derive a similar bound on its Frobenius norm:
Lemma 3.2 One has
kDXZ k2 81 kDV k4 :
Proof: It is trivial to verify that i h DX DZ + DZ DX = 21 (DX + DZ )2 ? (DX ? DZ )2
Since DX and DZ are orthogonal the matrices DV = DX + DZ and QV := DX ? DZ have the same norm. Consequently
2 kDXZ k2 =
41 DV2 ? Q2V
1 Tr D4 + Q4 ? 2D2 Q2 = 16 V V V V
161
DV2
2 +
Q2V
2 161 kDV k4 + kQV k4 = 81 kDV k4 :
2
The quadratic convergence result may now be proved.
8
Lemma 3.3 After a feasible Newton step the distance measure satis es 4 2 2 + := (X +; Z + ; ) 2(1 ? 2) : Proof: The distance measure after the full Newton step is given by
2
?1 2
1 + + +
? V
= 4 V
2
2
+ ?1 1
= 4 V I ? V +
2
2 ?1
1 + : + 2 4 max V
I ? V
We now substitute the bound from Lemma 3.1 to obtain
2
2 2
1 +
42 (1 ? 2) I ? V +
:
To complete the proof we show that:
I
?
In order to prove (5), note that
I
?
2
2 + V
= =
2
2 + V kD
n X i=1
n X i=1
XZ k
2:
(i (I + DXZ + S ) ? i (I ))2 (i (DXZ + S ))2
= Tr (DXZ + S )2 : Using the skew-symmetry of S one obtains
I
?
(5)
2
2 + V
= Tr (DXZ )2 ? SS T Tr(DXZ )2 = kDXZ k2: The nal result now follows from Lemma 3.2.
2
The quadratic convergence result may be stated concisely as:
Corollary 3.1 If
