introduction to dc programming & dca and recent

Outline DC programming & DCA Exact penalty techniques in DC Programming Recent advances in DC programming Exact penalty and Error Bounds in DC programming

INTRODUCTION TO DC PROGRAMMING & DCA AND RECENT ADVANCES Pham Dinh Tao National Institute for Applied Sciences-Rouen, France

Joint work with Le Thi Hoai An

23 octobre 2013

Pham Dinh Tao

INTRODUCTION TO DC National PROGRAMMING Institute&for DCA Applied AND RECENT Sciences-Rou ADVANCES


Plan 1

2 2

3

Outline DC programming & DCA Convex Programming Nonconvex Programming : Nonlinear Optimization − Global Optimization Exact penalty techniques in DC Programming Recent advances in DC programming Improved Convergence of DCA for DC programming with subanalytic data Extended DC Algorithms to general DC programs with DC constraints Exact penalty and Error Bounds in DC programming Exact penalty in concave programming Exact penalty in DC programming via error bounds Error bounds for DC inequality systems Error bounds for concave inequality systems over polyhedral

Pham Dinh Tao



Convex Programming Nonconvex Programming : Nonlinear Optimization − Global Optimization

Convex Programming −→ Nonconvex Programming DC Programming & DCA

Pham Dinh Tao




Convex Programming

Γ0(IRn ) := {f : IRn −→ IR ∪ {+∞}, lower semicontinuous proper (domf 6= ∅) convex}, a convex cone dom θ := {x ∈ IRn : θ(x) < +∞}. Convex program : f ∈ Γ0(IRn ) α := inf{f (x) : x ∈ IRn } (Pc )

Pham Dinh Tao




Constrained convex program : C ⊂ IRn nonempty closed convex set ⇔ χC ∈ Γ0(IRn ) χC indicator function of C χC (x) := 0 if x ∈ C, + ∞ otherwise Convex constrained program α := inf{f (x) : x ∈ C} (Pc ) can be rewritten as (f + χC ∈ Γ0(IRn )) α := inf{(f + χC )(x) : x ∈ IRn }

Pham Dinh Tao




In practice solvable (Pc ) takes the form : f , fi ∈ Γ0(IRn ), i = 1, ..., m α := inf{f (x) : fi (x) ≤ 0, i = 1, ..., m} Convex programming is featured by : Local solution = global solution. Existence of optimality conditions verifiable −→ Iterative solution algorithms. No Lagrangian duality gap. Extensively investigated from both a theoretical and an algorithmic point of views from the period 1960-1985 and after, with the development of Interior Point Methods and SDP.

Pham Dinh Tao




Nonconvex Programming : Nonlinear Optimization − Global Optimization (1980 −→)

(Hoang Tuy, R. Horst, H. Benson, H. Konno, P. Pardalos, Le Dung Muu, Le Thi Hoai An, Nguyen Van Thoai, Phan Thien Thach and Pham Dinh Tao) Complexity (NP hard in general) : Finding Global solution the Holy Grail quest for mathematicians (optimizers) Local solution 6= global solution Non existence of practically verifiable global optimality conditions : no iterative global solution algorithm

Pham Dinh Tao




Two approaches (two schools) in Nonconvex Programming : Local Approaches (Nonlinear Optimization, for smooth nonconvex programming) : Iterative algorithms based on local optimality conditions. Global Approaches (Global Optimization) : consisting, generally, of Bounding procedure : Computing a sequence of increasing lower bounds αk for the optimal value α by solving relaxed convex programs and that of decreasing upper bound βk .

Pham Dinh Tao




Branching procedure : Choosing a subdomain corresponding to the smallest lower bound to subdivide (usually bisect) and delecting those providing lower bounds greater than the current record (the current smallest upper bound) Repeating until the gap βk − αk ≤ to carry out an − global optimal solution Other methods : outer approximation using cuttings plane or sphere,... Global algorithms are generally too expensive to handle large-scale nonconvex programs.

Pham Dinh Tao




DC programming and DCA (1985 −→) Le Thi Hoai An, Pham Dinh Tao

Local Approaches based on Convex Programming for Smooth/Nonsmooth Nonconvex Programming http ://lita.sciences.univ-metz.fr/~lethi/ DC Programming and DCA : the backbone of Nonconvex Programming and Global Optimization. Key idea ( Philosophy/ Principle) : Extension of Convex Analysis/ Programming to Nonconvex Analysis/Programming, broad enough to cover most real-world nonconvex programs , but not too much to continue to use the powerful arsenal of convex analysis/ programming. Unlike the sum, the difference of convex functions destroys convexity. The feature DC will be advantageously explored and exploited within DC Programming and DCA. Pham Dinh Tao




DC(IRn ) := Γ0(IRn ) − Γ0(IRn ), vector subsapce spanned by Γ0(IRn ) DC program (Pdc ) α = inf{f (x) := g(x) − h(x) : x ∈ IRn } DC Programming investigates : structure of the vector space DC(IRn ) : it contains almost all realistic objective functions and is closed under all the operations usually considered in optimization. Real-world nonconvex programs are DC programs.

Pham Dinh Tao




DC duality : g ∈ Γ0(IRn ) g ∗(y ) := sup{hx, y i − g(x) : x ∈ IRn } g ∗ ∈ Γ0(IRn ), g ∗∗ = g α = inf{h∗(y ) − g ∗(y ) : y ∈ IRn } (Ddc )

Perfect symmetry : the dual of (Ddc ) is (Pdc ) itself.

Pham Dinh Tao




Local optimality : For primal DC Program (for dual DC program by similarity) ∂h(x ∗) ∩ ∂g(x ∗) 6= ∅ (such a point x ∗ is called critical point of g − h or generalized KKT point for (Pdc )), and ∅= 6 ∂h(x ∗) ⊂ ∂g(x ∗). (1) The condition (1) is also sufficient (for local optimality) in many important classes of DC programs. In particular it is sufficient for the next cases quite often encountered in practice :

Pham Dinh Tao




In polyhedral DC programs with h being a polyhedral convex function. In this case, if h is differentiable at a critical point x ∗, then x ∗ is actually a local minimizer for (Pdc ). Since a convex function is differentiable everywhere except for a set of measure zero, one can say that a critical point x ∗ is almost always a local minimizer for (Pdc ). In case the function f is locally convex at x ∗. In particular, if f is convex then x ∗ is a (global) optimal solution. Note that, if h is polyhedral convex, then f = g − h is locally convex everywhere h is differentiable. The transportation of global solutions between (Pdc ) and (Ddc ) is expressed by : P and D denote the solution sets of (Pdc ) and (Ddc ) respectively [ [ [ ∂g ∗(y ∗)] ⊆ P , [ ∂h(x ∗)] ⊆ D (2) y ∗∈D

x ∗∈P

equalities holds under technical assumptions. Pham Dinh Tao




Under technical assumptions,this transportation also holds for local solutions. Solving a DC program amounts solving its dual and vice -versa. The complexity of DC programs resides, of course, in the lack of verifiable conditions for global optimality. Subdifferential θ ∈ Γ0(IRn ) and x0 ∈ dom θ, ∂θ(x0) denotes the subdifferential of θ at x0, i.e., ∂θ(x0) := {y ∈ IRn : θ(x) ≥ θ(x0) + hx − x0, y i, ∀x ∈ IRn } (3) The subdifferential ∂θ(x0) is a closed convex set in IRn . It generalizes the derivative in the sense that θ is differentiable at x0 if and only if ∂θ(x0) is a singleton which is exactly {∇θ(x0)}. Pham Dinh Tao




Description of DCA Two sequences {x k } and {y k } of trial solutions of the primal and dual programs respectively, 1) {g(x k ) − h(x k )} and {h∗(y k ) − g ∗(y k )} are decreasing, 2) {x k } (resp. {y k }) converges to a primal feasible solution xe (resp. a dual feasible solution ye) satisfying local optimality conditions and xe ∈ ∂g ∗(ye), ye ∈ ∂h(xe). (4) The sequences {x k } and {y k } are determined in the way that x k +1 (resp. y k +1) is a solution to the convex program (Pk ) (resp. (Dk +1)) defined by (x 0 ∈ dom ∂h being a given initial point and y 0 ∈ ∂h(x 0) being chosen) (Pk ) (Dk +1) Pham Dinh Tao

inf{(g − h)k (x) := g(x) − [h(x k ) + hx − x k , y k i] : x ∈ Rn }, inf{(h∗ − g ∗)k (y ) := h∗(y ) − [g ∗(y k ) + hy − y k , x k +1i] : y ∈ Rn }. (5)




DCA deals with the convex DC components g and h but not with the DC function f itself. DCA is one of the rare algorithms for nonconvex nonsmooth programming. DC function f has infinitely many DC decompositions which have crucial implications for the qualities (speed of convergence, robustness, efficiency, globality of computed solutions,...) of DCA. We have fk (x) = f (x) + h(x) − [h(x k ) + hx − x k , y k i] ≥ f (x) ∀x (6) (h∗ − g ∗)k (y ) = (h∗ − g ∗)(y ) + g ∗(y ) − [g ∗(y k ) + hy − y k , x k +1i] ≥ (h∗ − g ∗)(y ) ∀y It is clear that the more tightly the convex function (g − h)k approximates f the more efficient the resulting DCA will be. Consider now another DC decomposition of f with ϕ ∈ Γ0(Rn ) Pham Dinh Tao




f := (g + ϕ) − (h + ϕ), and compare the two convex approximation (g − h)k (x) and [(g + ϕ) − (h + ϕ)]k .We have (y k ∈ ∂g(x k ), z k ∈ ∂ϕ(x k ))

(7)

ki [(g + ϕ) − (h + ϕ)]k (x) = (g + ϕ)(x) − [(g + ϕ)(x k ) + hx − x k , y k + z(8) = g(x) − [g(x k ) + hx − x k , y k i] + ϕ(x) −[ϕ(x k ) + hx − x k , z k i] = (g − h)k (x) + ϕ(x) − [ϕ(x k ) + hx − x k , z k i] ≥ (g − h)k (x).

Shift procedure for constructing new DC decompositions is not relevant. (see later the proximal DC decomposition).

Pham Dinh Tao




The choice of optimal DC decompositions is still open. This depends strongly on the very specific structure of the problem being considered. In order to tackle the large-scale setting, one tries in practice to choose g and h such that sequences {x k } and {y k } can be easily calculated, i.e., either they are in an explicit form or their computations are inexpensive.

Pham Dinh Tao




DCA’s Convergence {x k } ⊂ C convex, {y k } ⊂ D convex, ρ(g, C) (or ρ(g) if C = IRn ) the modulus of strong convexity of g on C given by : ρ(g, C) = sup{ρ ≥ 0 : g − (ρ/2)k · k2 be convex on C}. DCA’s convergence properties : DCA is a descent method without linesearch (linesearch is actually dealt within DC decomposition) which enjoys the following properties : i) The sequences {g(x k ) − h(x k )} and {h∗(y k ) − g ∗(y k )} are decreasing and • g(x k +1) − h(x k +1) = g(x k ) − h(x k ) iff y k ∈ ∂g(x k ) ∩ ∂h(x k ), y k ∈ ∂g(x k +1) ∩ ∂h(x k +1) and [ρ(g, C) + ρ(h, C)]kx k +1 − x k k = 0. Moreover if g or h are strictly convex on C then x k = x k +1. In such a case DCA terminates at the k th iteration (finite convergence of DCA) Pham Dinh Tao




• h∗(y k +1) − g ∗(y k +1) = h∗(y k ) − g ∗(y k ) iff x k +1 ∈ ∂g ∗(y k ) ∩ ∂h∗(y k ), x k +1 ∈ ∂g ∗(y k +1) ∩ ∂h∗(y k +1) and [ρ(g ∗, D) + ρ(h∗, D)]ky k +1 − y k k = 0. Moreover if g ∗ or h∗ are strictly convex on D, then y k +1 = y k . In such a case DCA terminates at the k th iteration (finite convergence of DCA). ii) If ρ(g, C) + ρ(h, C) > 0 (resp. ρ(g ∗, D) + ρ(h∗, D) > 0) then the series {kx k +1 − x k k2} (resp. {ky k +1 − y k k2} converges.)

iii) If the optimal value α of problem (Pdc ) is finite and the infinite sequences {x k } and {y k } are bounded then every limit point xe (resp. ye) of the sequence {x k } (resp. {y k }) is a critical point of g − h (resp. h∗ − g ∗). iv) DCA has a linear convergence for general DC programs. v) DCA has a finite convergence for polyhedral DC programs. For suitable DC decompositions DCA generates almost all standard algorithms in Convex and Nonconvex Programming. Pham Dinh Tao




An illustrative example General nonconvex quadratic program 1 T (9) min f (x) := x Qx + q T x : x ∈ K , 2 where Q is an n × n symmetric matrix, q ∈ IRn , and K ⊂ IRn is a nonempty polyhedral convex set. f is a DC function with infinitely many DC decompositions. The following is particularly interesting because the resulting DC component are quadratic functions too : f (x) = ϕ(x) − ψ(x), (10) 1 T 1 T T ϕ(x) := x Q1x + q x, ψ(x) := x Q2x, 2 2 where Q1 and Q2 are two n × n positive semidefinite symmetric matrices such that Q = Q1 − Q2. Pham Dinh Tao

(11)




This leads to the equivalent DC program inf{g(x) − h(x) : x ∈ IRn },

(12)

where g(x) := ϕ + χK , h := ψ The matrices Q1 and Q2 can have one of the following forms : (i) Spectral DC decomposition : Q1 := PD +P T , Q2 := PD −P T where P is an orthogonal matrix composed of eigenvectors of Q related to its eigenvalues λ1 ≤ ... ≤ λn such that Q = PDP T , with D := diag(λ1, ..., λn ) being the diagonal matrix with diagonal entries λi . The matrices D + and D − are given by : +), D − := diag(λ−, ..., λ−) D + := diag(λ+ , ..., λ n n 1 1

− = max{0, −λ }). (here λ+ = max{0, λ }, λ i i i i Pham Dinh Tao




(ii) Proximal DC decomposition : Q1 := Q + ρI, Q2 := ρI , with ρ ≥ 0 such that Q + ρI is positive semidefinite, i.e., ρ ≥ −λ1(Q). (iii) Projection DC decomposition : Q1 := ρI, Q2 := ρI − Q, with ρ ≥ 0 such that ρI − Q is positive semidefinite, i.e., ρ ≥ λn (Q). It is clear that the spectral DC decomposition is expensive, especially in the large-scale setting, while there are inexpensive algorithms for computing the extreme eigenvalues λ1(Q) and λn (Q).

Pham Dinh Tao




Let us describe now the DCA applied to the DC program (12) (x 0 being a given initial point) : y k = Q2x k ;

x k +1 ∈ arg min

1 T x Q1x + q T x − x T Q2x k : x ∈ K 2

. (13)

Note that the convex quadratic program 1 T inf x Q1x + q T x − x T Q2x k : x ∈ K 2 is equivalent to (Pk ) above and can also be written as 1 T 1 T min x Qx + q x + (x − x k )T Q2(x − x k ) : x ∈ K . 2 2

Pham Dinh Tao

(14)



We recognize here that, with the quadratic DC decomposition (10), the resulting DCA is an extended proximal-like method for nonconvex quadratic programming (applied to (9)). First result on proximal point algorithm for nonconvex programming. See below for extended result to non quadratic DC functions. The best choice of ρ ≥ −λ1(Q) will be the smallest value −λ1(Q), accordingly with (8). Consider the general DC program α = inf{f0(x) := g0(x)−h0(x) : x ∈ C, p(x) := g(x)−h(x) ≤ 0}

(Pdcg ) (15) where C is a nonempty closed convex set in Rn , g, h, g0, h0 ∈ Γ0(Rn ) and its feasible set S = {x ∈ C : g(x) − h(x) ≤ 0} assumed to be nonempty. The usual penalized DC program is the DC program defined by

Pham Dinh Tao



α(τ ) = inf{f0(x) + τ p+(x) : x ∈ C}

τ ) (Pdcg

(16)

We will summarize main exact penalty results with/without error bounds of nonconvex constraint sets S, which establish the equivalence of τ ). These results are based on the (Pdcg ) and its penalized (Pdcg concavity of objective/constraint functions and local error bounds [Le Thi, Pham Dinh and Huynh Van, 2012]. More precisely, by using the DC decomposition (ii) with ρ > 0, (14) becomes 1 T ρ T min x Qx + q x + kx − xk k2 : x ∈ K , (17) 2 2

Pham Dinh Tao



and then the corresponding DCA is the extended standard proximal algorithm for nonconvex programming. Lastly, the DC decomposition (iii) leads to the following DCA : ( )

2

1 1 k k k k +1

:x ∈K . y = (ρI − Q)x ; x ∈ arg min x − (y − q)

2 ρ (18)

Pham Dinh Tao



In other words, the primal sequence {x k } generated by this algorithm is given by 1 k +1 k x := PK x − (Qx k + q) , (19) ρ where PK denotes the projection mapping on K . One recognizes, in (19), the gradient projection algorithm of Goldstein-Levitin-Polyak in convex programming, i.e., when the objective function f is convex. It is worth noting that the above DC decompositions can be extended to functions f twice continuously differentiable on an open convex set O containing K under the condition : There is ρ > 0 such that 2ρ k.k2 + f (resp. 2ρ k.k2 − f ) is convex on K . Clearly, these conditions are satisfied when K is bounded. Note that, the DCA via (iii) can be viewed as a proximal regularization of gradient method for nonconvex programming. Pham Dinh Tao



Improved Convergence of DCA for DC programming with subanalytic data Extended DC Algorithms to general DC programs with DC constraints

Improved Convergence of DCA for DC programming with subanalytic data

Definition 1 (i) A subset C of Rn is said to be semianalytic if each point of Rn admits a neighborhood V such that C ∩ V is of the following form : C∩V =

p \ q [

{x ∈ V : fij (x) = 0, gij (x) > 0},

i=1 j=1

where fij , gij : V → R (1 ≤ i ≤ p, 1 ≤ j ≤ q ) are real-analytic functions. (ii) A subset C of Rn is called subanalytic if each point of Rn admits a neighborhood V such that C ∩ V = {x ∈ Rn : ∃y ∈ Rm , (x, y ) ∈ B}, where B is a bounded seminalytic subset of Rn × Rm with m ≥ 1. (iii) A function f : Rn → R ∪ {+∞} is said to be subanalytic if its graph gph f is a subanalytic subset of Rn × R. Pham Dinh Tao




The class of subanalytic sets (resp. functions) contains all analytic sets (resp. functions). The elementary properties of subanalytic sets and subanalytic functions (see, e.g., Bierstone and Milman ; Lojasiewicz ; Shiota) : - Subanalytic sets are closed under locally finite union and intersection (A collection of sets C is locally finite if any compact set intersects only finitely many sets in C). The complement of a subanalytic set is subanalytic. - The closure, the interior, the boundary of a subanalytic set are subanalytic. -A closed C ⊆ Rn is subanalytic iff its indicator function χC , defined by χC (x) = 0 if x ∈ C and +∞ otherwise, is subanalytic.

Pham Dinh Tao




- Given a subanalytic set C, the distance function dC (x) := infz∈C kx − zk is a subanalytic function. - Let f , g : X → R be continuous subanalytic functions, where X ⊆ Rn is a subanalytic set. Then the sum f + g is subanalytic if f maps bounded sets into bounded sets, or if both functions f , g are bounded from below. - Let X ⊆ Rn , T ⊆ Rm be subanalytic sets, where T is compact. If f : X × T → R is a continuous subanalytic function, then g(x) := mint∈T f (x, t) is continuous subanalytic.

Pham Dinh Tao




Proposition If f : Rn → R ∪ {+∞} is a lower semicontinuous subanalytic strongly convex function then its conjugate f ∗ is a C 1,1 (the class of functions whose derivative is Lipschitz) subanalytic convex function. S(Rn ) denotes the set of lower semicontinuous functions f : Rn → R ∪ {+∞}.

Pham Dinh Tao




For a function f ∈ S(Rn ), the Fréchet subdifferential of f at x ∈ dom f is defined by ∗ f (x + h) − f (x) − hx , hi F ∗ n ∂ f (x) = x ∈ R : lim inf ≥0 . khk h→0 For x ∈ / dom f , we set ∂f (x) = ∅. A point x0 ∈ Rn is called a (Fréchet) critical point for the function f , if 0 ∈ ∂ F f (x0).

Pham Dinh Tao




When f is a convex function, then ∂ F coincides with the subdifferential in the sense of Convex Analysis. Moreover, if f is a DC function, i.e., f := g − h, where g, h is convex functions, then ∂ F f (x) ⊆ ∂g(x) − ∂h(x) wherever h is continuous at x. Moreover, if h is differentiable at x, then one has the equality : ∂ F f (x) = ∂g(x) − ∇h(x). Let us recall the nonsmooth version of the Lojasiewicz inequality established by Bolte-Daniliidis-Lewis, which is needed in the convergence analysis of DCA in the sequel.

Pham Dinh Tao




Theorem 2 (Theorem 3.1, Bolte et al.) Let f : Rn → R ∪ {+∞} be a subanalytic function such that its domain Dom f is closed and f |dom f is continuous and let x0 is a Fréchet critical point of f . Then there exist θ ∈ [0, 1), L > 0 and a neighborhood V of x0 such that the following inequality holds. |f (x) − f (x0)|θ ≤ Lkx ∗k

for all x ∈ V , x ∗ ∈ ∂ F f (x),

where a convention 00 = 1 is used. The number θ in the theorem is called a Lojasiewicz exponent of the critical point x0.

Pham Dinh Tao




Theorem 3 Let us consider DC program (Pdc ) with α ∈ R. Suppose that the sequences {x k }, {y k } are defined by the DCA. (i) Suppose that the DC function f := g − h is subanalytic such that dom f is closed ; f |dom f is continuous and that g or h is differentiable on dom h or dom g, respectively with locally Lipschitz derivative. Assume that ρ(g) + ρ(h) > 0, where ρ(g), ρ(h) are modulus of the strong convexity of g, h, respectively. If either the sequence {x k } or {y k } is bounded then {x k } and {y k } are convergent to critical points of (Pdc ) and (Ddc ), respectively. (ii) Similarly, for the dual DC program (Ddc ), suppose that h∗ − g ∗ is subanalytic such that dom (h∗ − g ∗) is closed ; (h∗ − g ∗) |dom (h∗−g ∗) is continuous and that g ∗ or h∗ is differentiable on dom g ∗ or dom h∗, respectively with locally Lipschitz derivative. If ρ(g ∗) + ρ(h∗) > 0 and either the sequence {x k } or {y k } is bounded then {x k } and {y k } are convergent to critical points of (Pdc ) and (Ddc ), respectively.

Pham Dinh Tao




Corollary 4 Suppose that g − h and h∗ − g ∗ are subanalytic functions with closed domain such that (g − h) |dom (g−h)and (h∗ − g ∗) |dom (h∗−g ∗) are continuous. Assume that ρ(g) + ρ(h) > 0 as well as ρ(g ∗) + ρ(h∗) > 0. If either the sequence {x k } or {y k } is bounded then these sequences converge to critical points of (P) and (D), respectively.

Pham Dinh Tao




Theorem 5

Suppose that the assumptions of Theorem 3 (i) are satisfied. Let x ∞ be the limit point of {x k } with a Lojasiewicz exponent θ ∈ [0, 1). Then there exists constants τ1, τ2 > 0 such that ∞ X k ∞ j j+1 k k −1 k k −1 1−θ kx − x k ≤ kx − x k ≤ τ1kx − x k + τ2kx − x k θ , k = 1, 2, ... (20) j=k

As a result, one has k

∞

1−θ 1−2θ

If θ ∈ (1/2, 1) then kx − x k ≤ ck for some c > 0. If θ ∈ (0, 1/2] then kx k − x ∞k ≤ cq k for some c > 0; q ∈ (0, 1). If θ = 0 then {x k } is convergent in a finite number of steps.

Pham Dinh Tao




In general, the Lojasiewicz exponent is unknown. But in several special cases, one can determine the Lojasiewicz exponent. Let us consider the case of trust-region subproblems, that is, the problems of minimizing a (nonconvex) quadratic function over an Euclidean ball in Rn . 1 T min{ x Qx + hb, xi : x ∈ Rn , kxk ≤ R}, 2 or equivalently, 1 T min{f (x) := x Qx + hb, xi + χC (x) : x ∈ Rn }, (21) 2 where, Q is a n × n real symmetric matrix, b ∈ Rn , R is a positive scalar and C := {x ∈ Rn : kxk ≤ R}; χC (x) stands for the indicator function of the set C. The following theorem gives a Lojasiewicz exponent of some critical point of the objective function f under a suitable assumption. Pham Dinh Tao




Lemma 6 Let A be a n × n real symmetric matrix. Then there exists a constant M > 0 such that the following inequality holds |x T Ax|1/2 ≤ MkAxk

for all x ∈ Rn .

Theorem 7 Suppose x 0 ∈ C is a critical point of problem (21),i.e., there exists λ0 ≥ 0 such that Qx 0 + b + λ0x 0 = 0; λ0(kx 0k − R) = 0.

(22)

In addition, assume that either kx 0k < R or kx 0k = R and hv , Qv + λ0v i > 0 for all v ∈ Rn with hx 0, v i = 0, kv k = 1.

(23)

The Lojasiewicz exponent of the critical point x 0 is 1/2.

Pham Dinh Tao




Extended DC Algorithms to general DC programs with DC constraints

Introduction Consider the following nonconvex optimization problem fi (x) ≤ 0, (GPdc ) min f0(x) subject to x ∈ C,

i = 1, ..., m;

where C ⊆ Rn is a nonempty closed convex set ; f , fi : Rn → R (i = 0, 1, ..., s) are DC functions. Problem (Pdc ) is a general DC program with DC constraints. To apply the DCA to (GPdc ), a penalization is used for DC constraints to equivalently transform this problem to a standard DC program without DC constraint. A penalty function largely used is l∞−penalty function of the form p+(x) := max{0, fi (x) : i = 1, ..., m}, Pham Dinh Tao

x ∈ Rn .




However, in the computational point of view, an inconvenience of this exact penalty method is that the penalty parameter is generally unknown. Moreover, there are practical optimization problems for which the exact penalization is not satisfied. The purpose of this paper is to develop the DCA for solving general DC program (GPdc ) by using a penalty technique with updated parameter. The global convergence of the method proposed is investigated.

Pham Dinh Tao




Let f : Rn → R ∪ {+∞} be a locally Lipschitz function at a given x ∈ Rn . The Clarke directional derivative and the Clarke subdifferential of f at x is given by the following formulas. f (y + tv ) − f (y ) ↑ f (x, v ) := lim sup . t (t,y )→(0+,x) n o ∂ ↑f (x) := x ∗ ∈ Rn : hx ∗, v i ≤ f ↑(x, v ) ∀v ∈ Rn . If f is continuously differentiable at x then ∂ ↑f (x) = ∇f (x) (the Fréchet derivative of f at x). When f is a convex function, then ∂f (x) coincides with the subdifferential in the sense of Convex Analysis, i.e., n n ∂f (x) = y ∈ R : hy , di ≤ f (x + d) − f (x), ∀d ∈ R .

Pham Dinh Tao




We list the following calculus rules for the Clarke subdifferential which are needed thereafter. 1. For given two locally Lipschitz functions f , g at a given x ∈ Rn , one has ∂(−f )(x) = −∂f (x); ∂(f + g)(x) ⊆ ∂f (x) + ∂g(x), (24) and, the equality in the latter inclusion holds if f is continuously differentiable at x. 2. For given functions fi : Rn → R (i = 1, ..., m), let f : Rn → R be the max-type function defined by f (x) := max {fi (x) : i = 1, ..., m} , x ∈ Rn .

Pham Dinh Tao



Then one has  m X ∂f (x) ⊆ λi ∂fi (x) :  i=1

λi ≥ 0,


m X i=1

λi = 1; λi = 0 if fi (x) < f (x)

 

,

 (25)

and, the equality holds if fi : Rn → R (i = 1, ..., m) are continuously differentiable at x ∈ Rn . Let C be a nonempty closed subset of Rn . Denote by χC (x) the indicator function of C, that is, χC (x) = 0 if x ∈ C, otherwise χC (x) = +∞. For a closed subset C of Rn , the normal cone of C at x ∈ C, denoted by N(C, x), is defined by n N(C, x) := ∂χC (x) = u ∈ R : hu, y − xi ≤ 0 ∀y ∈ C .

Pham Dinh Tao




DC Algorithm using l∞−penalty function with updated parameter

(GPdc )

min

f0(x)

subject to

fi (x) ≤ 0, x ∈ C,

i = 1, ..., s;

where C ⊆ Rn is a nonempty closed convex set fi : Rn → R ∩ {+∞}, i = 0, 1, ..., m are DC functions. D = {x ∈ C : fi (x) ≤ 0, i = 1, ..., s} : the feasible set of (GP dc ) p(x) := max{f1(x), ..., fm (x)};

I(x) := {i ∈ {1, ..., m} : fi (x) = p(x)} ;

p+(x) = max{p(x), 0}. We consider the following penalty problems   minimize ϕk (x) := f (x) + βk p+(x) 

(26)

subject to x ∈ C,

where, βk is penalty parameters. Pham Dinh Tao




Note that p+ is a DC function whenever fi0s are DC function and that there are available DC decompositions for p+. For example, if DC decompositions of fi , (i=1,...,m) are given fi = gi − hi , i = 1, ..., m, then a well-known DC decomposition of p+ is defined by   m m   X X p+(x) = max 0, gi (x) + hj (x) − hj (x).  i=1,...,m  j=1,j6=i

(27)

j=1

So the objective function ϕk (x) of problem (26) is a DC function. Let DC decompositions of f0 and p+ be given by f0(x) = g0(x) − h0(x);

(28)

p+(x) = p1(x) − p2(x), (29) where g0, h0 p1, p2 are convex functions defined on the whole space. Pham Dinh Tao




Then, we have the following DC decomposition for ϕk ϕk (x) = gk (x) − hk (x),

x ∈ Rn ,

(30)

where gk (x) := g0(x) + βk p1(x);

hk (x) := h0(x) + βk p2(x).

(31)

We make the following assumption that will be used in the sequel. Assumption 1. fi0s (i = 0, ..., m) are locally Lipschitz functions at every point of C. Assumption 2. Either gk or hk is differentiable on C, and ρ(g0) + ρ(h0) + ρ(p1) + ρ(p2) > 0.

Pham Dinh Tao




DC Algorithm with updated parameter ALGORITHM 1. Initialization: Take an initial point x 1 ∈ C; δ > 0; an initial penalty parameter β1 > 0 and set k := 1. 1 Compute y k ∈ ∂h (x ). k k ∗ k k +1 is a solution of the 2 Compute x k +1 ∈ ∂(g k + χC ) (y ), i.e., x convex program min{gk (x) − hx, y k i : x ∈ C}. (32) 3 Stopping test. Stop if x k +1 = x k and p(x k ) ≤ 0. 4 Penalty parameter update. Compute rk := min{p(x k ), p(x k +1)} and set ( βk if either βk ≥ kx k +1 − x k k−1 or rk ≤ 0, βk +1 = βk + δ if βk < kx k +1 − x k k−1 and rk > 0, 5

Set k := k + 1 and go to Step 1.

Pham Dinh Tao




Global convergence Recall that a point x ∗ ∈ D (the feasible set of (GPdc )) is a Karush-Kuhn-Tucker (KKT) point for the problem (GPdc ) if there exist nonnegative scalars λi , i = 1, ..., s such that Pm ∗ 0 ∈ ∂f0(x ) + i=1 λi ∂fi (x ∗) + N(C, x ∗), (33) λi fi (x ∗) = 0, i = 1, ..., m. We say that the extended Mangasarian-Fromowitz constraint qualification (EMFCQ) is satisfied at x ∗ ∈ D with I(x ∗) 6= ∅ if (MFCQ) there is a vector d ∈ cone(C − {x ∗}) (the cone hull ↑ of C − {x ∗}) such that fi (x ∗, d) < 0 for all i ∈ I(x ∗). ↑

When fi0s are continuously differentiable, then fi (x ∗, d) = h∇f (x ∗), di. Therefore, (EMFCQ) becomes the well-known Mangasarian-Fromowitz constraint qualification. Pham Dinh Tao




It is well known that if the (extended) Mangasarian-Fromowitz constraint qualification is satisfied at a local minimizer x ∗ of problem (GPdc ) then the KKT first order necessary condition (33) holds (see Mangasarian ; Mangasarian and Fromovitz). Assumption 3. The (extended) Mangasarian-Fromowitz constraint qualification (EMFCQ) is satisfied at any x ∈ Rn with p(x) ≥ 0. When fi , i = 1, ..., s are all convex functions, then it is obvious that this assumption is satisfied under the Slater regular condition, i.e., there exists x ∈ C such that fi (x) < 0

Pham Dinh Tao

for all i = 1, ..., s.




Theorem 8 Suppose that C ⊆ Rn is a nonempty closed convex set and fi , i = 1, ..., m are DC functions on C. Suppose further that Asumptions 1-3 are verified. Let δ > 0, β1 > 0 be given. Let {xk } be a sequence generated by Algorithm 1. Then Algorithm 1 either stops finitely at a KKT point x k for problem (P) or generates an infinite sequence {x k } of iterates such that limk →∞ kx k +1 − x k k = 0 and every limit point x ∞ of the sequence {x k } is a KKT point of problem (P).

Instead of using the penalty function p+, we could also use the following one m X ψ(x) := fi+(x), x ∈ Rn . i=1

It is easy to see that the convergence result also holds for this penalty function. Pham Dinh Tao




DC Algorithm by linearizing convex parts of DC constraints By using the main idea of the DCA that linearizes the nonconvex part of the DC structure, we can devise the extended DCA to DC constraints which consists of the solution of a sequence of convex programs of the form (Le Thi Hoai An, Pham Dinh Tao, The extended DCA to general DC programs with DC constraints, Research Report, 2007, National Insitute for Applied Sciences, Rouen) : min g0(x) −

hy0k , xi

s.t.

gi (x) − hi (x k ) − hyik , x − x k i ≤ 0, i = 1, ..., m; x ∈ C,

(34)

where, x k ∈ Rn is the current iterate, yik ∈ ∂hi (x k ) for i = 0, ..., m.

Pham Dinh Tao




However, this linearization can lead to infeasibility of convex subproblem (34). We propose a relaxation technique to deal with the feasibility of subproblems. Instead of (34), we consider the subproblem min g0(x) −

hy0k , xi

+ βk t s.t.

gi (x) − hi (x k ) − hyik , x − x k i ≤ t, i = 1, ..., m x ∈ C, t ≥ 0;

(35)

where βk > 0 is a penalty parameter. Obviously, (35) is a convex problem that is always feasible.

Pham Dinh Tao




Furthermore, the Slater constraint qualification is satisfied for the constraints of (35), thus the Karush-Kuhn-Tucker optimality condition holds for some solution (x k +1, t k +1) of (35) : there exist some λki +1 ∈ R, i = 1, ..., m, and µk +1 ∈ R such that  Pm k +1 k +1 k +1 k k +1 k  + N(C, x ), ∂g (x ) − y 0 ∈ ∂g (x ) − y + λ i 0  i=1 i 0 i  Pm k +1    βk − i=1 λi − µk +1 = 0, gi (x k +1) − hi (x k ) − hyik , x k +1 − x k i ≤ t k +1, λik +1≥ 0 i = 1, ..., m, x k +1 ∈ C,    λki +1 gi (x k +1) − hi (x k ) − hyik , x k +1 − x k i − t k +1 = 0, i = 1, ..., m,    t k +1 ≥ 0, µk +1 ≥ 0, t k +1µk +1 = 0. i

Pham Dinh Tao

(36)




ALGORITHM 2. Initialization: Take an initial point x 1 ∈ C; δ1, δ2 > 0; an initial penalty parameter β1 > 0 and set k := 1. 1 Compute y k ∈ ∂h (x k ), i = 0, ..., m. i i

4

Compute (x k +1, t k +1) as the solution of (35), and the associated Lagrange multipliers (λk +1, µk +1). Stopping test. Stop if x k +1 = x k and t k +1 = 0. Penalty parameter update. Compute rk := min{kx k +1 − x k k−1, kλk +1k1 + δ1}, where Pm k +1 kλ k1 = i=1 |λki +1|, and set βk if βk ≥ rk , βk +1 = βk + δ2 if βk < rk .

5

Set k := k + 1 and go to Step 1.

2

3

Pham Dinh Tao




Note that the updated penalty parameter rule is inspired by Solodov in [Solodov M. V., On the sequential quadratically constrained quadratic programming methods, Mathematics of Oper. Research, 29, 64-79 (2004)]. That is to ensure that the unboundedness of {β k } leads to the unboundedness of {kλk k1} and kx k +1 − x k k → 0. As in the preceding section, denote by ϕk (x) := f0(x) + βk p+(x), x ∈ Rn . Lemma 9 The sequence (x k , t k ) generated by Algorithm 2 satisfies the following inequality ρ k +1 k k +1 − x k k2, for all k = 1, 2, ... (37 ) ϕk (x ) − ϕk (x ) ≥ kx 2 where, ρ := ρ(g0) + ρ(h0) + min{ρ(gi ) : i = 1, ...m}.

Pham Dinh Tao




Theorem 10 Suppose that C ⊆ Rn is a nonempty closed convex set and fi , i = 1, ..., m are DC functions on C such that assumptions 1 and 3 are verified. Suppose further that for each i = 0, ..., m, either gi or hi is differentiable on C and that ρ := ρ(g0) + ρ(h0) + min{ρ(gi ) : i = 1, ...m} > 0. Let δ1, δ2 > 0, β1 > 0 be given. Let {xk } be a sequence generated by Algorithm 2. Then Algorithm 2 either stops finitely at a KKT point x k for problem (GPdc ) or generates an infinite sequence {x k } of iterates such that limk →∞ kx k +1 − x k k = 0 and every limit point x ∞ of the sequence {x k } is a KKT point of problem (GPdc ).

Pham Dinh Tao




Note that, as shown in Theorem 8 and Theorem 10, the penalty parameter βk is constant when k is sufficiently large. Observing from the proof of these convergence theorems, the sequence {ϕ(x k )} of values of the function ϕ(x) = f0(x) + βk p+(x) along with the sequence {x k } generated by Algorithms 1 and Algorithm 2 is decreasing. Note that these results remain valide if we replace, Pm in (35), the variable t by ti for i = 1, , m and the function βk t by βk i=1 ti .

Pham Dinh Tao



Exact penalty in concave programming Exact penalty in DC programming via error bounds Error bounds for DC inequality systems Error bounds for concave inequality systems over polyhedral convex sets Exact penalty in Mixed Integer DC Programming

Consider the general DC program α = inf{f0(x) := g0(x)−h0(x) : x ∈ C, p(x) := g(x)−h(x) ≤ 0}

(Pdcg ) (38) where C is a nonempty closed convex set in Rn , g, h, g0, h0 ∈ Γ0(Rn ) and its feasible set S = {x ∈ C : g(x) − h(x) ≤ 0} assumed to be nonempty. The usual penalized DC program is the DC program defined by α(τ ) = inf{f0(x) + τ p+(x) : x ∈ C}

τ ) (Pdcg

(39)

We will summarize main exact penalty results with/without error bounds of nonconvex constraint sets S, which establish the equivalence of τ ). These results are based on the (Pdcg ) and its penalized (Pdcg concavity of objective/constraint functions and local error bounds [Le Thi, Pham Dinh and Huynh Van, 2012]. Pham Dinh Tao




We first deal with exact penalty for nonconvex programs having concave objective functions and bounded polyhedral convex constraint sets with additional concave constraint functions. These results are not derived from error bounds of feasible solution sets because the question concerning their error bounds is still open. We get back to it later. Let K be a nonempty bounded polyhedral convex subset of R n and let f , g be finite functions on K . Recall that the function f is said to be continuous relative to K , if its restriction to K is a continuous function. The notions of relative upper semicontinuity and relative lower semicontinuity are defined similarly. In this section, we are concerned with the following nonconvex programs :

Pham Dinh Tao

α := min{f (x) : x ∈ K , g(x) = 0} (P)

(40)

min{f (x) : x ∈ K , g(x) ≤ 0} (P 0)

(41)




In [Le Thi, Pham Dinh, Le Dung, 1999] it has been established an exact penalty result for the problem (P) under the hypothesis that g is nonnegative on K . We consider (P) here without the nonnegativity assumption of g. Problem (P) could be rewritten equivalently α := min{f (x) : x ∈ K , g(x) ≤ 0, g(x) ≥ 0} (P) Its penalized reads α(τ ) := min{f (x) + τ g(x) : x ∈ K , g(x) ≥ 0} (Pτ ).

(42) (43)

The following result constitutes a substantial extension of the key theorem in ([Le Thi, Pham Dinh, Le Dung, 1999]- Theorem 2) concerning the special case of (P)− (42) when the concave function g is assumed to be nonnegative on K . Let us denote by V (K ) the vertex set of K and by P and Pτ the optimal solution sets of (P)− (42) and (Pτ )− (43), respectively. Here, the convention min∅ g(x) = +∞ is used. Pham Dinh Tao




Theorem 11 Let K be a nonempty bounded polyhedral convex set in Rn and let f , g be finite concave functions continuous realive to K . Suppose that the feasible set of (P) is nonempty. Then there exists τ0 ≥ 0 such that for all τ > τ0, the problems (P)− (42) and (Pτ ) − (43) have the same optimal value and the same optimal solution set. Furthermore, we can take f (x )−α(0) τ0 = 0 m with m = min{g(x) : x ∈ V (K ), g(x) > 0} and any x0 ∈ K , g(x0) = 0.

Pham Dinh Tao




Obviously, (P 0) is equivalent to the following one : min{f (x) : (x, t) ∈ K × [0, β], g(x) + t = 0} (P 0)

(44)

where, β ≥ max{−g(x) : x ∈ K } and such the equivalence is in the following usual sense : If x is a solution of (P 0) then (x, −g(x)) is a solution of (P 0). Conversely, if (x, t) is a solution of (P 0) then x is a solution of (P 0). In virtue of Theorem 11, there is τ0 ≥ 0 such that for all τ > τ0, the problem (P 0) is equivalent to the following one : min{f (x) + τ (t + g(x)) : (x, t) ∈ K × [0, β], g(x) + t ≥ 0} (Pτ0 )

Pham Dinh Tao

(45)




Thus we obtain the following corollary. Corollary 12 Under the assumptions of Theorem 11, the problems (P 0) − (??) and (Pτ0 )−(45) are equivalent in the sense given in Theorem 11. It is obvious that (Pτ0 )−(45) is a DC program. As shown by the corollary, it is worth noting that if f , g are finite concave functions on a nonempty bounded polyhedral convex set K , then Problem (P 0) − (??) is equivalent to (Pτ0 )−(45). By applying Corollary 12, we list below the nonconvex programs (with DC objective functions and DC constraint functions) frequently encountered in practice that we can advantageously recast into more suitable DC programs equivalent (in the sense given Theorem 11) :

Pham Dinh Tao




(a) Polyhedral convex objective function and concave constraint function Consider the problem with a nonempty feasible set : min{f (x) : x ∈ K , g(x) ≤ 0}, here, K is a polyhedral convex set, f ∈ Γ0(Rn ) is a polyhedral convex function, i.e., it is a sum of a pointwise supremum of a finite collection of affine functions and an indicator funtion of a nonempty polyhedral convex set D f (x) : = max{hx, yi i − αi : i = 1, ..., r } + χD (x) = f (x) + χD (x),

(46)

such that D ∩ K is bounded and g is a finite concave function continuous relative to D∩ K . Pham Dinh Tao




Obviously, this problem is equivalent to min{t : f (x) ≤ t, (x, t) ∈ [D ∩ K ] × [a, b], g(x) ≤ 0},

(47)

where, a ≤ min{f (x) : x ∈ [D ∩ K ]} and b ≥ max{f (x) : x ∈ [D ∩ K ]}. According to Corollary 2, if the feasible set of there is τ0 such that for all τ > τ0, Problem (47) is equivalent to following DC program (with β ≥ maxx∈[D∩K ](−g(x))) :

min{t+τ (s+g(x)) : f (x) ≤ t, (x, t, s) ∈ [D∩K ]×[a, b]×[0, β], g(x)+s ≥ 0}, (48) that can be reduced to min{f (x) + τ (s + g(x)) : (x, s) ∈ [D ∩ K ] × [0, β], g(x) + s ≥ 0} (49)

Pham Dinh Tao




(b) Concave objective function and DC (whose first DC component is polyhedral convex function) constraint function Let f be a finite concave function on a polyhedral convex set K . Let g be a polyhedral convex function defined as in (46) with Dom g := {x ∈ Rn : g(x) < +∞} = D such that D ∩ K is bounded nonempty, and h be a finite convex function on D ∩ K . Let us now consider the following problem with a polyhedral DC constraint : min{f (x) : x ∈ K , g(x) − h(x) ≤ 0}.

(50)

This problem is equivalently transformed into min{f (x) : (x, t) ∈ [D ∩ K ] × [a, b], g(x) − t ≤ 0, t − h(x) ≤ 0},

(51)

where, a ≤ minx∈[D∩K ] g(x), b ≥ maxx∈[D∩K ] g(x). In virtue of Corollary 2, if the feasible set of (50) is nonempty then there is τ0 ≥ 0 such that Pham Dinh Tao




for all τ > τ0 Problem (51) is equivalent to the DC program (β ≥ maxx∈[D∩K ] h(x) − a) : min{f (x) + τ (s + t − h(x)) : (x, t, s) ∈ [D ∩ K ] × [a, b] × [0, β],(52) g(x) − t ≤ 0, s + t − h(x) ≥ 0}.

Pham Dinh Tao




(c) DC objective function and DC constraint function (whose first DC component is polyhedral convex function) Let K be a nonempty polyhedral convex set in Rn . Consider the nonconvex program with a polyhedral DC objective function f = g0 − h0 and DC constraint function : min{g0(x) − h0(x) : x ∈ K , g1(x) − h1(x) ≤ 0} where g0, g1 are polyhedral convex functions on Rn , i.e., g0(x) : = max{hx, yi i − αi : i = 1, ..., q} + χD (x) = g0(x) + χD (x),

(53) (54)

g1(x) : = max{hx, zi i − αi : i = 1, ..., r } + χR (x) = g1(x) + χR (x), Pham Dinh Tao




with D, R being nonempty polyhedral convex sets and h0, h1 are a finite convex function continuous relative to D ∩ K ∩ R. Assume that the polyhedral convex set D ∩ K ∩ R is bounded nonempty. Problem (53) is equivalent to min{s − h0(x) : (x, s, t) ∈ D ∩ K ∩ R × [a0 , b0] × [a1 , b1], g0(x) ≤ s, g1(x) t − h1(x) ≤ 0} where a0 , b0, a1 , b1 are constant numbers verifying a0 ≤ min{ g0(x) : x ∈ D ∩ K ∩ R} ≤ max{ g0(x) : x ∈ D ∩ K ∩ R} ≤ b0 a1 ≤ min{ g1(x) : x ∈ D ∩ K ∩ R} ≤ max{ g1(x) : x ∈ D ∩ K ∩ R} ≤ b1 Hence, it follows from Corollary 12 that if the feasible set of Problem (55) is nonempty then there is τ0 ≥ 0 such that for all τ > τ0 this problem is equivalent to Pham Dinh Tao




min{s − h0(x) + τ [u + t − h1(x)] : (x, s, t, u) ∈ D ∩ K ∩ R × [a0 , b0] × [a1 , g0(x) ≤ s, g1(x) ≤ t, u + t − h1(x) ≥ 0} with c being a constant number satisfying c ≥ max{h1(x) : x ∈ D ∩ K ∩ R} − a1. Finally, Problem (53) is equivalent to the DC program

min{f (x, u, t) = g0(x) − h0(x) + τ [u + t − h1(x)] : (57) (x, t, u) ∈ D ∩ K ∩ R × [a1 , b1] × [0, c], g1(x) ≤ t, u + t − h1(x) ≥ 0}

Pham Dinh Tao




To complement the results concerning exact penalty in concave programming of Subsection 1, we will use the error bounds results in Subsection 3 to establish below some exact penalty properties in DC programming. Recall that, throughout this section, two problems are said to be equivalent if they have the same optimal value and the same optimal solution set. First, we give a general exact penalty result which is a refinement of that mentioned by Clarke in [?]. Recall that a real-valued function f defined on a set C in Rn is said to be Lipschitz on C, if there exists a nonnegative scalar L such that |f (x) − f (y )| ≤ Lkx − y k for all x, y ∈ C. Also f is said to be locally Lipschitz relative to C at some x ∈ C if for some > 0, f is Lipschitz on B(x, ) ∩ C. Let θ be a finite DC function on a closed convex set C, i.e. Pham Dinh Tao




θ(x) = ϕ(x) − ψ(x) ∀x ∈ C,

(58)

where ϕ and ψ belong to Γ0(Rn ) such that C ⊂ Dom ϕ := {x ∈ Rn : ϕ(x) < +∞} ⊂ Dom

(59)

Note that if (i) Dom ϕ and Dom ψ have the same dimension and (ii) C is bounded and contained in the relative interior of Dom ϕ, then the DC function θ is Lipschitz on C.

Pham Dinh Tao




Let f , h be real-valued functions defined on C. Consider the minimization problem whose optimal solution set is denote by P α = inf{f (x) : x ∈ C, h(x) ≤ 0}

(P),

(60)

that we can write as α = inf{f (x) : x ∈ S},

(61)

where S := {x ∈ C : h(x) ≤ 0} (62) Let g : C → R be a nonnegative function such that S can be expressed by S := {x ∈ C : g(x) ≤ 0} (63) Such a function g must verify g(x) = 0 if and only if x ∈ S Pham Dinh Tao

(64)




Exact penalty in mathematical programming usually deals with (Section ) + α g(x) := h (x) (65) For τ ≥ 0, we define the problems (Pτ ) by α(τ ) = inf{f (x) + τ g(x) : x ∈ C}

(Pτ ),

(66)

whose optimal solution set is denoted by Pτ .

Pham Dinh Tao




Proposition 13 Let f be a Lipschitz function on C with constant L and let g be a nonnegative finite function on C such that S := {x ∈ C : h(x) ≤ 0} = {x ∈ C : g(x) ≤ 0}. If S is nonempty and there exists some ` > 0 such that d(x, S) ≤ `g(x) for all x ∈ C, then one has : (i) α(τ ) = α and P ⊂ Pτ for all τ ≥ L` (ii) Pτ = P for all τ > L` The above proof is quite standard and follows the line in [?]. However, it is worth noting that our proof [?], unlike that of [?], needs neither the nonemptiness of the optimal solution set of (P) nor the closedness of its feasible set S. Pham Dinh Tao




Throughout the paper k.k denotes the Euclidean norm on Rn unless otherwise specified. Let C be a nonempty closed subset in Rn and let h be a finite function on C. We consider the solution set of the following inequality system S = {x ∈ C : h(x) ≤ 0} (67)

Pham Dinh Tao




Recall that an error bound of S is an inequality of the form d(x, S) ≤ τ h+(x) for all x ∈ C,

(68)

where, d(x, S) = infz∈S kx − zk, and τ is a positive number. For x0 ∈ S, if the inequality in (68) holds for all x in a neighborhood of x0, then we say that S has an error bound around x0. Instead of (68), an inequality of the form d(x, S) ≤ τ h+(x)γ for all x ∈ C, (69) with τ > 0 and 0 < γ < 1 will be called an error bound for (67) with the exponent γ. Note that if an error bound of S holds for a norm it holds for every norm because all norms on Rn are equivalent. Error bounds have important applications in many areas of mathematical programming, e.g., in sensitivity analysis, in convergence analysis of some algorithms, and in exact penalty, etc [Le Thi, Pham Dinh, Huynh Van, 2012]. Pham Dinh Tao




In our work [?], we derive several new results on error bounds for the inequality system (67) when C is a suitable closed convex set and h is a concave function on C. These systems play an important role in global optimization. In general, a system which has local error bounds at all points in its solution set does not necessarily attain the global error bound. However, under the compactness hypothesis, such an implication holds as will show the following proposition that will be needed thereafter.

Pham Dinh Tao




Theorem 14 Let C be a nonempty compact set in Rn . Let h be a finite function lower semicontinuous relative to C and S := {x ∈ C : h(x) ≤ 0} = 6 ∅. Suppose that for each z ∈ S, there exist τ (z), γ(z) > 0 and (z) > 0 such that d(x, S) ≤ τ (z)h+(x)γ(z) for all x ∈ B(z, (z)) ∩ C,

(70)

where, as usual, B(z, (z)) denotes the closed ball in Rn with center z and radius (z). Then there exist τ, γ > 0 such that d(x, S) ≤ τ h+(x)γ for all x ∈ C.

(71)

Remark 15 We see from the proof above that if (70) is satisfied for a common exponent α which does not depend on z, then (71) holds also with this exponential parameter. Pham Dinh Tao




As a consequence we can derive the following error bound Corollary 16 Let C be the Euclidean ball {x ∈ Rn : kx − ck ≤ r } where c ∈ Rn and r > 0 and S := {x ∈ Rn : kx − ck = r }.Then we have d(x, S) ≤ h(x) for all x ∈ C,

(72)

with h(x) := r − kx − ck,and 1 d(x, S) ≤ k (x) for all x ∈ C, r where k (x) := r 2 − kx − ck2.

Pham Dinh Tao

(73)




It is well known, see e.g. Proposition 13 in Section ?? , that error bounds of feasible solution sets of nonconvex programs provide, in an elegant and deep way, exact penalty for those problems with Lipschitz objective functions. This subsection is concerned with error bounds for DC inequality systems. Let f : Rn → R ∪ {+∞} be a lower semicontinuous function on Rn . The lower directional derivative (or contingent derivative or Hadamard derivative) of f at x ∈ Dom f := {x ∈ Rn : f (x) < +∞} is given by f 0(x; u) =

Pham Dinh Tao

f (x + tv ) − f (x) lim inf . t (t,v )→(0+,u)




If f is differentiable at x with the derivative ∇h(x) then f 0(x; u) = h∇h(x), ui. When f is convex or concave then the lower directional derivative coincides with the usual directional derivative.

We present first, in the context of finite dimension, a general sufficient condition for the error bound in terms of the lower directional derivative [Le Thi, Pham Dinh, Huynh Van, 2012].

Pham Dinh Tao




Proposition 17 Let C be a nonempty closed convex set in Rn , let h be a finite function lower semicontinuous relative to C and S := {x ∈ C : h(x) ≤ 0} = 6 ∅. (i) Let x0 ∈ S. If there exists µ > 0, > 0 such that for each y ∈ [B(x0, ) ∩ C] \ S there exists z ∈ C, z 6= y such that h0(y ; z − y ) < −µkz − y k then one has 1 + d(x, S) ≤ h (x) for all x ∈ B(x0, /2) ∩ C. µ

Pham Dinh Tao




Proposition 18 (ii) In addition, if C is bounded, h is continuous relative to C and for all x ∈ S with h(x) = 0, the set {(µ, ) > (0, 0) : ∀y ∈ [B(x, )∩C]\S

∃zy ∈ C such that h0(y ; zy −y ) < −µk

is nonempty, then there is τ > 0 such that d(x, S) ≤ τ h+(x) for all x ∈ C.

Pham Dinh Tao




Remark 19 Proposition 17 remains valid if we replace the conditions (i) (resp. (ii) by the following conditions (iii) (resp. (iv)) : (iii) Let x0 ∈ S. If there exits z ∈ C such that the function x → h0(x; z − x) is upper semicontinuous at x0 and h0(x0; z − x0) < 0. (iv) In addition, if C is bounded, h is continuous relative to C and for all x0 ∈ S with h(x0) = 0,there exits z ∈ C such that the function x → h0(x; z − x) is upper semicontinuous at x0 and h0(x0; z − x0) < 0.

Pham Dinh Tao




Next, we give a condition ensuring the error bound for DC inequality systems which is a slight extension the well known Slater one for convex systems (when h = 0). The systems of that type play an important role in DC programming. Let C be a nonempty closed convex set in Rn and let g, h be finite convex functions on C. Consider the following set : S = {x ∈ C : g(x) − h(x) ≤ 0}.

Pham Dinh Tao




Theorem 20 Let C ⊂ Rn be a nonempty compact convex set, g be a finite convex function continuous relative to C and h be a differentiable convex function on C. If for each x0 ∈ S with g(x0) − h(x0) = 0 the set {z ∈ C : g(z) − h∇h(x0), z − x0i − h(x0) < 0} is nonempty, then there exists τ > 0 such that d(x, S) ≤ τ [g(x) − h(x)]+ for all x ∈ C. In the case of systems consisting of quadratic inequalities, the conditions in the preceding proposition can be rewritten more simply as follows.

Pham Dinh Tao




Corollary 21 Let C be a nonempty compact convex subset of Rn and 1 fi (x) := hx, Qi xi + hqi , xi + ri , i = 1, ..., m 2 where Qi are symmetric n × n matrices, qi ∈ Rn , ri ∈ R. Let f (x) := max{fi (x) : i = 1, ..., m}. If for each x ∈ S := {x ∈ C : f (x) ≤ 0} with f (x) = 0, there holds λ {z ∈ C : f (z) + kz − xk2 < 0} = 6 ∅, 2 where λ := max{ρ(Qi ) : i = 1, ..., m} and ρ(Qi ) is the spectral radius of Qi , then there exists τ > 0 such that d(x, S) ≤ τ f +(x) for all Pham Dinh Tao

x ∈ C.




Next, let hi : Rn → R, (i = 1, ..., m) be m concave functions. Consider now the following system of the concave inequaties : S = {x ∈ Rn : hi (x) ≤ 0, i = 1, ..., m}. Setting (= := {1, ..., m}) h(x) := max{h1(x), ..., hm (x)}, =(x) := {i ∈ {1, ..., m} : hi (x) = h(x)} Proposition 22 Let hi , (i = 1, ..., m), h and S be defined as above. If S and the set {z ∈ Rn : hi (z) > 0 for i = 1, ..., m} are nonempty, then for every ρ > 0, there exists τ > 0 such that

(74)

d(x, S) ≤ τ h+(x) for all x ∈ Rn , kxk ≤ ρ. Moreover, if Rn \ S is bounded then the inequality holds for all x ∈ Rn . Pham Dinh Tao




In the case of m = 1 concave inequality, if the assumption (74) doesn’t hold then S = Rn and it is trivial that d(x, S) ≤ τ h+(x) for all x ∈ Rn . Hence one has Corollary 23 Let h be a finite concave function on Rn and let S := {x ∈ Rn : h(x) ≤ 0} be nonempty. Then for every ρ > 0, there exists τ > 0 such that d(x, S) ≤ τ h+(x) for all x ∈ Rn , kxk ≤ ρ. Moreover if Rn \ S is bounded, then d(x, S) ≤ τ h+(x) for all x ∈ Rn .

Pham Dinh Tao




Remark 24 Let E := {x ∈ Rn : kxk = r } that can be written as E := {x ∈ C : h1(x) = r − kxk ≤ 0} where C := {x ∈ Rn : kxk ≤ r }. According to Corollary 24 there is τ1 > 0 such that d(x, E) ≤ τ1h1(x) for all x ∈ C.

(75)

Similarly using the concave function h2(x) := r 2 − kxk2 ≥ 0 to define E := {x ∈ C : h2(x) ≤ 0} yields the error bound with τ2 > 0 d(x, E) ≤ τ2h2(x) for all x ∈ C

Pham Dinh Tao

(76)




Remark 25 Since 2rh1(x) ≥ h2(x) := r 2 − kxk2 = [r + kxk][r − kxk] = [r + kxk]h1(x) ≥ rh1(x) ∀x ∈ C,

(77 )

there hold τ (i) If τ1 > 0 verifies (75) then τ2 = r1 verifies (76) (ii) Conversely if τ2 > 0 verfies (76) then τ1 = 2r τ2 verifies (75)

Pham Dinh Tao




Remark 26 Consider now the feasible set F := {x ∈ B(c, r ) : h3(x) := r − kx − ck ≥ 0} in Corollary 16. Since B(c, r ) = c + C, F = c + S and the distance is invariant with respect to translations, it follows from (75) that there τ3 > 0 such that d(x, F ) ≤ τ3h3(x) for all x ∈ B(c, r )

(78)

Likewise, the concave function h4(x) := r 2 − kx − ck2 leads to the error bound with τ4 > 0 d(x, F ) ≤ τ4h4(x) for all x ∈ B(c, r )

(79)

and the parameters τ3, τ4 satisfy the properties (i) and (ii) as τ1, τ2. Note at last that the error bound of F given by (72) and (73) are practically advantageous insofar as the parameters τ3, τ4 are known. Pham Dinh Tao




Let now K be a nonempty bounded polyhedral convex set and let h be a finite concave function on K . Consider the following set : S = {x ∈ K : h(x) ≤ 0}. The constraint ”h(x) ≤ 0” is often called reverse convex constraint. Of course, the preceding results on exact penalty in Subsection 1 can be applicable for this system. However, under additional assumptions, we can establish [?] some results on error bounds which are easy to verify in many applications. It is worth noting that the sets of that type figure in many practical problems (see, e.g., DC Programming and DCA : http ://lita.sciences.univ-metz.fr/~lethi/DCA.html).

Pham Dinh Tao




Theorem 27 Let K be a nonempty bounded polyhedral convex set in Rn and let h be a differentiable concave function on K . If S is nonempty, then there exists τ > 0 such that d(x, S) ≤ τ h+(x) for all x ∈ K .

(80)

As before, let K be a given nonempty bounded polyhedral convex set. We now consider the function h defined by h(x) =

m X

min{hij (x) : i ∈ =j }

(81)

j=1

where, =1, ..., =m are finite index sets and hij are differentiable concave functions on K . The functions of this type are not differentiable. Pham Dinh Tao




By applying Theorem 27, we next show that the conclusion of Theorem 27 remains valid for the function h given by (81). Theorem 28 Let K be a nonempty bounded polyhedral convex set and let h be defined as above. Suppose that the set S := {x ∈ K : h(x) ≤ 0} is nonempty. Then there exists τ > 0 such that d(x, S) ≤ τ h+(x) for all x ∈ K .

Pham Dinh Tao




Let us end this subsection with two other error bounds resulting from Theorem ??. The first one involves the convex quadratic constraints 1 gi (x) := hQi x, xi + hqi , xi + ri , i = 1, ..., m (82) 2 where qi ∈ Rn , ri ∈ R and Qi are positive semidefinite symmetric n × n matrices. Let K be a nonempty bounded polyhedral convex set in Rn . Consider the following inequality system : S = x ∈ K : gi (x) ≤ 0, hp (x) ≤ 0, (i, p) ∈ {1, ..., m} × {1, ..., `} , where, hp are functions of the same type as in Theorem 28, i.e., X hp (x) := min{hij (x) : i ∈ =j,p }, j∈=p

Pham Dinh Tao




where, =p , =j,p are finite index sets and hij are differentiable concave functions on K . Set S1 = {x ∈ K : gi (x) ≤ 0, i = 1, ..., m}. Theorem 28 yields the following result :

(83)

Corollary 29 Let gi , hp be defined as above. In addition, suppose that S is nonempty, gi , (i = 1, ..., m), are nonnegative on K and hp , (p = 1, ..., ` ), are nonnegative on S1. Then there exists τ > 0 such that d(x, S) ≤ τ

l X

hp (x) for all x ∈ S1.

p=1

Pham Dinh Tao




Finally the second error bound derived from Theorem 28 is related to finite feasible sets Corollary 30 Let K be a nonempty bounded polyhedral convex set and let h be as in Theorem 28. Assume that the set S := {x ∈ K : h(x) ≤ 0} has finitely many elements. Then for all closed set C ⊂ K such that 0 S = {x ∈ C : h(x) ≤ 0} is nonempty, there exists τ > 0 such that d(x, S 0) ≤ τ h+(x)

Pham Dinh Tao

for all

x ∈ C.




Remark 31 It is easy to see that the assumption on the finiteness of the set S is satisfied if h is strictly concave and nonnegative on K . For example, consider the feasible set in 0 − 1 programming : S := {x ∈ C : x ∈ {0, 1}n } where C is a closed subset of Rn . One can rewrite S = {x ∈ C ∩ [0, 1]n : h(x) ≤ 0} where h(x) := he, xi − kxk2, and e ∈ Rn is the vector of ones. Obviously, the assumptions of Corollary 30 are satisfied. Note that the concave (but not strictly) penalty function h(x) =

n X

min{xi , 1 − xi }

i=1

also verifies the assumptions of Corollary 30. Pham Dinh Tao




Let us end the paper with new key resutls on exact penalty in mixed integer DC programming DC [Le Thi, Pham Dinh, 2012] Consider the class of mixed integer DC programs α := inf{f (x, y ) = g(x, y ) − h(x, y ) : (x, y ) ∈ K , x ∈ [l, u] ∩ Zn }

(84)

where K is a bounded polyhedral convex set in Rn+p : K := {(x, y ) ∈ Rn+p : C(x, y ) := Ax + By ≤ b} with C ∈ n Y Rm×(n+p), A ∈ Rm×n , B ∈ Rm×p , [l, u] := [li , ui ] ⊂ Rn , with li , ui ∈ Z, li < ui for i = 1, ..., n.

Pham Dinh Tao

i=1




This challenging class of nonconvex programs encompasses most combinatorial optimization problems, in particular usual mixed integer linear/quadratic programming problems. To our best knowledge, our exact penalty results concerning (84) are the first ones to date. Let us first summarize the penalty functions p(x, y ) used to penalize the integer variables x. n P 1) p1(x, y ) := sin2(πxj ) has been already used in some earlier works. j=1 2) p2(x, y ) := d22(x, [l, u] ∩ Zn ) = min{kx − zk22 : z ∈ [l, u] ∩ Zn } n P = min{(xj − zj )2 : zj ∈ [lj , uj ] ∩ Z} is piecewise convex. j=1 n P 3) p3(x, y ) := sin πxj is piecewise concave j=1 4) p4(x, y ) := d1(x, [l, u] ∩ Zn ) = min{kx − zk1 : z ∈ [l, u] ∩ Zn }

Pham Dinh Tao



=

n P j=1


min{ xj − zj : zj ∈ [lj , uj ] ∩ Z} is piecewise concave.

n P j 5) p5(x, y ) = p5(x) := p5(xj ), with j=1

j

p5(xj ) := max{[xj − (lj + k )][(lj + k + 1) − xj ] : k = 0, ..., (uj − lj ) − 1} is piecewise concave. The five penalty functions are DC functions with explicit DC decompositions. By using Theorem 28 , we have proved the exact penalty for the last three penalty functions [Le Thi, Pham Dinh, 2012]. Hence there is equivalence between (84) and its penalized by either of them.

Pham Dinh Tao




Thank you for your attention !

Pham Dinh Tao
