A New Augmented Lagrangian Method for Equality Constrained ...

Hindawi Discrete Dynamics in Nature and Society Volume 2017, Article ID 6406514, 9 pages https://doi.org/10.1155/2017/6406514

Research Article A New Augmented Lagrangian Method for Equality Constrained Optimization with Simple Unconstrained Subproblem Hao Zhang1,2 and Qin Ni1 1

College of Science, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China College of Engineering, Nanjing Agricultural University, Nanjing 210031, China

2

Correspondence should be addressed to Hao Zhang; [email protected] Received 3 May 2017; Accepted 16 August 2017; Published 19 October 2017 Academic Editor: Seenith Sivasundaram Copyright © 2017 Hao Zhang and Qin Ni. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. We propose a new method for equality constrained optimization based on augmented Lagrangian method. We construct an unconstrained subproblem by adding an adaptive quadratic term to the quadratic model of augmented Lagrangian function. In each iteration, we solve this unconstrained subproblem to obtain the trial step. The main feature of this work is that the subproblem can be more easily solved. Numerical results show that this method is effective.

1. Introduction

For (1), we define the Lagrangian function

In this paper, we consider the following equality constrained optimization:

𝑙 (𝑥, 𝜆) = 𝑓 (𝑥) − 𝜆𝑇 𝑐 (𝑥)

(2)

and the augmented Lagrangian function min 𝑓 (𝑥) s.t.

𝑐𝑖 (𝑥) = 0,

𝑖 = 1, . . . , 𝑚,

(1)

where 𝑥 ∈ 𝑅𝑛 , 𝑓(𝑥) : 𝑅𝑛 → 𝑅, and 𝑐𝑖 (𝑥) : 𝑅𝑛 → 𝑅 (𝑖 = 1, . . . , 𝑚) are twice continuously differentiable. The method presented in this paper is a variant of the augmented Lagrangian method (denoted by AL). In the late 1960s, AL method was proposed by Hestenes [1] and Powell [2]. Later Conn et al. [3, 4] presented a practical AL method and proved the global convergence under the LICQ condition. Since then, AL method attracted the attentions of many scholars and many variants were presented (see [5–11]). Up to now, there are many computer packages based on AL method, such as LANCELOT [4] and ALGENCAN [5, 6]. In the past decades, AL method was fully developed. Attracted by its well performance, there are still many scholars devoted to research AL method and its applications in recent years (see [7, 8, 11–15]).

𝐿 (𝑥, 𝜆, 𝜎) = 𝑙 (𝑥, 𝜆) +

𝜎 ‖𝑐 (𝑥)‖2 , 2

(3)

where 𝜆 is called the Lagrangian multiplier and 𝜎 is called the penalty parameter. In this paper, ‖ ⋅ ‖ refers to the Euclidean norm. In a typical AL method, at the 𝑘th step, for given multiplier 𝜆 𝑘 and penalty parameter 𝜎𝑘 , an unconstrained subproblem min𝑛 𝐿 (𝑥, 𝜆 𝑘 , 𝜎𝑘 )

(4)

𝑥∈𝑅

is solved to find the next iteration point. Then, the multiplier and penalty parameter are updated by some rules. For convenience, for given 𝜆 𝑘 and 𝜎𝑘 , we define Φ𝑘 (𝑥) = 𝐿 (𝑥, 𝜆 𝑘 , 𝜎𝑘 ) = 𝑙 (𝑥, 𝜆 𝑘 ) +

𝜎𝑘 ‖𝑐 (𝑥)‖2 . 2

(5)

Motivated by the regularized Newton method for unconstrained optimization (see [16–19]), we construct a new

2

Discrete Dynamics in Nature and Society

subproblem of (1). At the 𝑘th iteration point 𝑥𝑘 , Φ𝑘 (𝑥𝑘 + 𝑠) is approximated by the following quadratic model: 𝑚𝑘 (𝑠) 1 𝑇 = 𝑙 (𝑥𝑘 , 𝜆 𝑘 ) + (∇𝑥 𝑙 (𝑥𝑘 , 𝜆 𝑘 )) 𝑠 + 𝑠𝑇 𝐵𝑘 𝑠 2 𝜎 󵄩 󵄩2 + 𝑘 󵄩󵄩󵄩𝑐 (𝑥𝑘 ) + 𝐴 (𝑥𝑘 ) 𝑠󵄩󵄩󵄩 2 𝑇

𝑇

= 𝑙 (𝑥𝑘 , 𝜆 𝑘 ) + (∇𝑥 𝑓 (𝑥𝑘 ) − 𝐴 (𝑥𝑘 ) 𝜆 𝑘 ) 𝑠 (6)

𝜎 󵄩 1 󵄩2 + 𝑠𝑇 𝐵𝑘 𝑠 + 𝑘 󵄩󵄩󵄩𝑐 (𝑥𝑘 ) + 𝐴 (𝑥𝑘 ) 𝑠󵄩󵄩󵄩 2 2 = Φ𝑘 (𝑥𝑘 ) 𝑇

𝑇

𝑔̃𝑘 = ∇Φ𝑘 (𝑥𝑘 ) 𝑇

= ∇𝑥 𝑓 (𝑥𝑘 ) − 𝐴 (𝑥𝑘 ) 𝜆 𝑘 + 𝜎𝑘 𝐴 (𝑥𝑘 ) 𝑐 (𝑥𝑘 ) ;

(7)

𝑇 𝐵̃𝑘 = 𝐵𝑘 + 𝜎𝑘 𝐴 (𝑥𝑘 ) 𝐴 (𝑥𝑘 ) ,

thus we have 1 𝑚𝑘 (𝑠) = Φ𝑘 (𝑥𝑘 ) + 𝑔̃𝑘𝑇 𝑠 + 𝑠𝑇 𝐵̃𝑘 𝑠. 2

(8)

In [14, 15], 𝑚𝑘 (𝑠) is minimized within a trust region to find the next iteration point. Motivated by the regularized Newton method, we add a regularization term to the quadratic model 𝑚𝑘 (𝑠) and define 1 𝑚𝑘 (𝑠) = 𝑚𝑘 (𝑠) + 𝜇𝑘 ‖𝑠‖2 2 1 1 + 𝑠𝑇 𝐵̃𝑘 𝑠 + 𝜇𝑘 ‖𝑠‖2 , 2 2

(9)

where 𝜇𝑘 is called regularized parameter. At the 𝑘th step of our algorithm, we solve the following convex unconstrained quadratic subproblem: 1 1 min𝑛 𝑚𝑘 (𝑠) = Φ𝑘 (𝑥𝑘 ) + 𝑔̃𝑘𝑇 𝑠 + 𝑠𝑇 𝐵̃𝑘 𝑠 + 𝜇𝑘 ‖𝑠‖2 𝑠∈𝑅 2 2

(10)

for finding the trial step 𝑠𝑘 . Then, we compute the ratio between the actual reduction and predicted reduction 𝜌𝑘 =

(13)

󵄩 󵄩 𝜇𝑘 (󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 − Δ 𝑘 ) = 0,

where 𝐴(𝑥𝑘 ) = [∇𝑥 𝑐1 (𝑥𝑘 ), . . . , ∇𝑥 𝑐𝑚 (𝑥𝑘 )]𝑇 and 𝐵𝑘 is a positive 2 𝑙(𝑥𝑘 , 𝜆 𝑘 ). Let semidefinite approximation of ∇𝑥𝑥

= Φ𝑘 (𝑥𝑘 ) +

(12)

The exact solution 𝑠𝑘 of (12) satisfies the first-order critical conditions if there exists some 𝜇𝑘 ≥ 0 such that 𝐵̃𝑘 + 𝜇𝑘 𝐼 is positive semidefinite and 󵄩󵄩 󵄩󵄩 󵄩󵄩𝑠𝑘 󵄩󵄩 ≤ Δ 𝑘

1 𝑇 + 𝑠𝑇 [𝐵𝑘 + 𝜎𝑘 𝐴 (𝑥𝑘 ) 𝐴 (𝑥𝑘 )] 𝑠, 2

𝑔̃𝑘𝑇 𝑠

1 min 𝑚𝑘 (𝑠) = Φ (𝑥𝑘 ) + 𝑔̃𝑘𝑇 𝑠 + 𝑠𝑇 𝐵̃𝑘 𝑠. 𝑠∈Δ 𝑘 2

𝑔̃𝑘 + (𝐵̃𝑘 + 𝜇𝑘 𝐼) 𝑠𝑘 = 0

𝑇

+ [∇𝑥 𝑓 (𝑥𝑘 ) − 𝐴 (𝑥𝑘 ) 𝜆 𝑘 + 𝜎𝑘 𝐴 (𝑥𝑘 ) 𝑐 (𝑥𝑘 )] 𝑠

𝑇

a sufficiently “good” approximation of Φ𝑘 (𝑥𝑘 + 𝑠) and reduce the value of 𝜇𝑘 . Conversely, when 𝜌𝑘 is close to zero, we set 𝑥𝑘+1 = 𝑥𝑘 and increase the value of 𝜇𝑘 , by which we wish to reduce the length of the next trial step. This technique is similar to the update rule of trust region radius. Actually, sufficiently large 𝜇𝑘 indeed reduces the length of the trial step 𝑠𝑘 . However, the regularized parameter is different from trust region radius. In [14, 15], the authors construct a trust region subproblem

Ared𝑘 Φ𝑘 (𝑥𝑘 ) − Φ𝑘 (𝑥𝑘 + 𝑠𝑘 ) = . Pred𝑘 Φ𝑘 (𝑥𝑘 ) − 𝑚𝑘 (𝑥𝑘 + 𝑠𝑘 )

(11)

When 𝜌𝑘 is close to 1, we accept 𝑥𝑘 + 𝑠𝑘 as the next iteration point. At the same time, we think the quadratic model 𝑚𝑘 (𝑠) is

while the first-order critical condition of (10) is 𝑔̃𝑘 + (𝐵̃𝑘 + 𝜇𝑘 ) 𝑠𝑘 = 0.

(14)

Equations (13) and (15) can show the similarities and differences between regularized subproblem (10) and trust region subproblem (12). It seems that the parameter 𝜇𝑘 plays a role similar to the multiplier 𝜇𝑘 in the trust region subproblem. But, actually, the update rule of 𝜇𝑘 (see (26)) shows that 𝜇𝑘 is not the approximation of 𝜇𝑘 . The update of 𝜇𝑘 depends on the quality of last trial step 𝑠𝑘−1 and has no direct relation with system (13). To establish the global convergence of an algorithm, some kind of constraint qualification is required. There are many well-known constraint qualifications, such as LICQ, MFCQ, CRCQ, RCR, CPLD, and RCPLD. In case there are only equality constraints, LICQ is equivalent to MFCQ in which {∇𝑐𝑖 (𝑥) | 𝑖 = 1, . . . , 𝑚} has full rank; CRCQ is equivalent to CPLD in which any subset of {∇𝑐𝑖 (𝑥) | 𝑖 = 1, . . . , 𝑚} maintains constant rank in a neighborhood of 𝑥; RCR is equivalent to RCPLD in which {∇𝑐𝑖 (𝑥) | 𝑖 = 1, . . . , 𝑚} maintains constant rank in a neighborhood of 𝑥. RCPLD is weaker than CRCQ, and CRCQ is weaker than LICQ. In this paper, we use RCPLD which is defined in the following. Definition 1. One says that RCPLD holds at a feasible point 𝑥∗ of (1), if there exists a neighborhood 𝑈(𝑥∗ ) of 𝑥∗ such that {∇𝑐𝑖 (𝑥) | 𝑖 = 1, . . . , 𝑚} maintains constant rank for all 𝑥 ∈ 𝑈(𝑥∗ ). The rest of this paper is organized as follows. In Section 2, we give a detailed description of the presented algorithm. The global convergence is proved in Section 3. In Section 4, we present the numerical experiments. Some conclusions are given in Section 5. Notations. For convenience, we abbreviate ∇𝑥 𝑓(𝑥𝑘 ) to 𝑔𝑘 , 𝑓(𝑥𝑘 ) to 𝑓𝑘 , 𝑐(𝑥𝑘 ) to 𝑐𝑘 , and 𝐴(𝑥𝑘 ) to 𝐴 𝑘 . In this paper, 𝜆(𝑖) denotes the 𝑖th component of the vector 𝜆.


3

2. Algorithm In this section, we give a detailed description of the proposed algorithm. As mentioned in Section 1, we solve the unconstrained subproblem (10) to obtain the trial step 𝑠𝑘 . Since 𝐵𝑘 is at least positive semidefinite and 𝐵̃𝑘 = 𝐵𝑘 + 𝜎𝑘 𝐴𝑇𝑘 𝐴 𝑘 , 𝐵̃𝑘 + 𝜇𝑘 𝐼 is positive definite as 𝜇𝑘 > 0. Therefore, (10) is a strictly convex quadratic unconstrained optimization. 𝑠𝑘 solves (10) if and only if 𝑔̃𝑘 + (𝐵̃𝑘 + 𝜇𝑘 𝐼) 𝑠𝑘 = 0

(15)

holds. Global convergence does not depend on the exact solution of (15), although the linear system (15) is easy to be solved. For minimizer of (10) along the direction −𝑔̃𝑘 , specifically, we consider the following subproblem: 󵄩 󵄩2 1 min 𝑚𝑘 (−𝛼𝑔̃𝑘 ) = Φ𝑘 (𝑥𝑘 ) − 𝛼 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 + 𝛼2 𝑔̃𝑘𝑇 𝐵̃𝑘 𝑔̃𝑘 𝛼≥0 2 1 󵄩 󵄩2 + 𝛼2 𝜇𝑘 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 . 2

(16)

Pred𝑘 ≥ Φ𝑘 (𝑥𝑘 ) − 𝑚𝑘 (−𝛼𝑘∗ 𝑔̃𝑘 ) (17)

(18)

In Section 3, we always suppose that (18) holds. In a typical AL algorithm, the update rule of 𝜎𝑘 depends on the improvement of constraint violation. A commonly used update rule is that if ‖𝑐𝑘+1 ‖ < 𝜏𝑘 ‖𝑐𝑘 ‖, where 0 < 𝜏𝑘 < 1, one may think that the constraint violation is reduced sufficiently and thus 𝜎𝑘+1 = 𝜎𝑘 is a good choice. Otherwise, if ‖𝑐𝑘+1 ‖ ≥ 𝜏𝑘 ‖𝑐𝑘 ‖, one thinks that current penalty parameter can not sufficiently reduce the constraint violation and increase it in the next iteration. In [20], Yuan proposed a different update rule of 𝜎𝑘 for trust region algorithm. Specifically, if 󵄩 󵄩 󵄩 󵄩2 Pred𝑘 < 𝛿𝑘 𝜎𝑘 min {Δ 𝑘 󵄩󵄩󵄩𝑐𝑘 󵄩󵄩󵄩 , 󵄩󵄩󵄩𝑐𝑘 󵄩󵄩󵄩 } ,

𝜎𝑘 is increased.

when 𝜀𝑘 is sufficiently small, 𝜆 𝑘 − 𝜎𝑘 𝑐𝑘 is a good estimate of the next multiplier 𝜆 𝑘+1 . As we obtain 𝑥𝑥+1 by minimizing 𝑚𝑘 (𝑠), the critical point of 𝑚𝑘 (𝑠) has no direct relation to ‖∇𝑥 𝐿(𝑥𝑘+1 , 𝜆 𝑘 , 𝜎𝑘 )‖. Therefore, the update rule 𝜆 𝑘+1 = 𝜆 𝑘 − 𝜎𝑘 𝑐𝑘 does not suit our algorithm. We obtain 𝜆 𝑘+1 by approximately solving the following least squares problem: min

𝜆∈𝑅𝑚

1 󵄩󵄩 󵄩2 󵄩󵄩𝑔 − 𝐴𝑇𝑘+1 𝜆󵄩󵄩󵄩 . 󵄩 2 󵄩 𝑘+1

(22)

Most AL algorithms require that {𝜆 𝑘 } is bounded to ensure the global convergence. Hence, all components of 𝜆 𝑘 are restricted to certain interval [𝜆, 𝜆]. This technique is also used in our algorithm. Now, we give the detailed algorithm in the following.

Step 0 (initialization). Choose the parameters 0 < 𝛾1 < 𝛾2 < 1, 0 < 𝜃1 < 1 ≤ 𝜃2 < 𝜃3 , 𝜆 > 𝜆. Determine 𝑥0 ∈ 𝑅𝑛 , 𝜆 0 ∈ 𝑅𝑚 , 𝜎0 > 0, 𝛿0 > 0, 𝐵0 ∈ 𝑅𝑛×𝑛 . Let 𝑅0 = max{‖𝑐(𝑥0 )‖, 1}. Set 𝑘 fl 0. Step 1 (termination test). If ‖𝑔̃𝑘 ‖ = 0 and ‖𝑐𝑘 ‖ = 0, return 𝑥𝑘 as a KKT point. If ‖𝑔̃𝑘 ‖ = 0, ‖𝑐𝑘 ‖ > 0, and ‖𝐴𝑇𝑘 𝑐𝑘 ‖ = 0, return 𝑥𝑘 as an infeasible KKT point.

1 1 min𝑛 𝑚𝑘 (𝑠) = Φ𝑘 (𝑥𝑘 ) + 𝑔̃𝑘𝑇 𝑠 + 𝑠𝑇 𝐵̃𝑘 𝑠 + 𝜇𝑘 ‖𝑠‖2 𝑠∈𝑅 2 2

(20)

(23)

such that (18) holds. Compute the ratio between the actual reduction to the predicted reduction 𝜌𝑘 =

Ared𝑘 , Pred𝑘

(24)

where Ared𝑘 = Φ𝑘 (𝑥𝑘 )−Φ𝑘 (𝑥𝑘 +𝑠𝑘 ), Pred𝑘 = Φ𝑘 (𝑥𝑘 )−𝑚𝑘 (𝑥𝑘 + 𝑠𝑘 ). Set {𝑥𝑘 + 𝑠𝑘 , 𝜌𝑘 ≥ 𝛾1 , 𝑥𝑘+1 = { 𝑥 , 𝜌𝑘 < 𝛾1 , { 𝑘

(19)

𝜎𝑘 is increased. In (19), 𝛿𝑘 is an auxiliary parameter such that 𝛿𝑘 𝜎𝑘 tends to zero. We slightly modify (19) in our algorithm. Specifically, if 󵄩󵄩 󵄩󵄩 󵄩 󵄩 󵄩𝑐𝑘 󵄩󵄩 max {1, 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩} 󵄩󵄩 󵄩󵄩2 󵄩 < 𝛿 𝜎 min { , 󵄩󵄩𝑐𝑘 󵄩󵄩 } , Pred𝑘 𝑘 𝑘 𝜇𝑘

(21)

Step 2 (determine the trial step). Evaluate the trial step 𝑠𝑘 by solving

By direct calculation, we have that 󵄩󵄩 ̃ 󵄩󵄩2 󵄩𝑔 󵄩 1 1 Pred𝑘 ≥ 󵄩 𝑘 󵄩 min { 󵄩󵄩 󵄩󵄩 , } . 4 󵄩󵄩𝐵̃𝑘 󵄩󵄩 𝜇𝑘 󵄩 󵄩

∇𝑥 𝐿 (𝑥𝑘+1 , 𝜆 𝑘 , 𝜎𝑘 ) = 𝑔𝑘+1 − 𝐴𝑇𝑘+1 (𝜆 𝑘 − 𝜎𝑘 𝑐𝑘 ) ,

Algorithm 2.

If ‖𝑔̃𝑘 ‖ > 0, then the minimizer of (16) is 𝛼𝑘∗ = ‖𝑔̃𝑘 ‖2 /(𝑔̃𝑘𝑇 𝐵̃𝑘 𝑔̃𝑘 +𝜇𝑘 ‖𝑔̃𝑘 ‖2 ). Therefore, at the 𝑘th step, it follows that 󵄩󵄩 ̃ 󵄩󵄩4 󵄩󵄩𝑔𝑘 󵄩󵄩 = 󵄩 󵄩2 . 𝑇 2 (𝑔̃𝑘 𝐵̃𝑘 𝑔̃𝑘 + 𝜇𝑘 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 )

In typical AL method, next iteration point 𝑥𝑘+1 is obtained by minimizing 𝐿(𝑥, 𝜆 𝑘 , 𝜎𝑘 ). In most AL methods, 𝑥𝑘+1 satisfies that ‖∇𝑥 𝐿(𝑥𝑘+1 , 𝜆 𝑘 , 𝜎𝑘 )‖ < 𝜀𝑘 , where 𝜀𝑘 is controlling parameter which tends to zero. As

𝜇𝑘+1

𝜃1 𝜇𝑘 , 𝜌𝑘 ≥ 𝛾2 , { { { { = {𝜃2 𝜇𝑘 , 𝛾1 ≤ 𝜌𝑘 < 𝛾2 , { { { {𝜃3 𝜇𝑘 , 𝜌𝑘 < 𝛾1 .

Step 3 (update the penalty parameter). If 󵄩 󵄩 󵄩󵄩 󵄩󵄩 󵄩𝑐 󵄩 max {1, 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩} 󵄩󵄩 󵄩󵄩2 Pred𝑘 < 𝛿𝑘 𝜎𝑘 min { 󵄩 𝑘 󵄩 , 󵄩󵄩𝑐𝑘 󵄩󵄩 } , 𝜇𝑘

(25)

(26)

(27)

4


set 𝜎𝑘+1 = 2𝜎𝑘 , 1 𝛿𝑘+1 = 𝛿𝑘 . 4

(28)

Otherwise, set 𝜎𝑘+1 = 𝜎𝑘 , 𝛿𝑘+1 = 𝛿𝑘 .

(29)

Now, we discuss convergence properties in two cases. One is that the penalty parameter 𝜎𝑘 tends to ∞ and the other is that {𝜎𝑘 } is bounded. 3.1. The Case of 𝜎𝑘 → ∞

Step 4 (update the multiplier). If ‖𝑐𝑘+1 ‖ ≤ 𝑅𝑘 , set 𝑅𝑘+1 = ̂ (1/2)𝑅𝑘 . Evaluate 𝜆 𝑘+1 by 󵄩 󵄩 min 󵄩󵄩󵄩󵄩𝑔𝑘 − 𝐴𝑇𝑘 𝜆󵄩󵄩󵄩󵄩 , (30) 𝜆∈𝑅𝑚 and let ̂ (𝑖) 𝜆(𝑖) 𝑘+1 = min {max {𝜆 𝑘+1 , 𝜆} , 𝜆} , 𝑖 = 1, . . . , 𝑚.

If ‖𝑠𝑘 ‖ = 0, then (32) holds. If ‖𝑠𝑘 ‖ > 0, (‖𝑔̃𝑘 ‖ − (1/ 4)𝜇𝑘 ‖𝑠𝑘 ‖)‖𝑠𝑘 ‖ + (1/2)(‖𝐵̃𝑘 ‖ − (1/2)𝜇𝑘 )‖𝑠𝑘 ‖2 ≥ 0 implies that ‖𝑔̃𝑘 ‖ − (1/4)𝜇𝑘 ‖𝑠𝑘 ‖ ≥ 0 or ‖𝐵̃𝑘 ‖ − (1/2)𝜇𝑘 ≥ 0. Thus we can obtain (32).

(31)

If ‖𝑐𝑘+1 ‖ > 𝑅𝑘 , set 𝑅𝑘+1 = 𝑅𝑘 and 𝜆 𝑘+1 = 𝜆 𝑘 . Set 𝑘 fl 𝑘 + 1 and go to Step 1.

Lemma 5. Suppose that (A1)-(A2) hold and 𝜎𝑘 → ∞; then there exists a constant 𝑐∗ such that ‖𝑐𝑘 ‖ → 𝑐∗ . Proof. See Lemma 3.1 in Wang and Yuan [15]. In Lemma 5, if 𝑐∗ > 0, then any accumulation point of {𝑥𝑘 } is infeasible. Sometimes (1) is naturally infeasible; in other words, the feasible set {𝑥 | 𝑐(𝑥) = 0} is empty. In this case, we wish to find a minimizer of constraint violation. Specifically, we wish to solve ‖𝑐 (𝑥)‖2 .

min𝑛

(34)

Remark 3. In practical calculation, it is not required to ̂ . In our implementation of solve (30) exactly to find 𝜆 𝑘+1 Algorithm 2, we use the Matlab subroutine minres to find an approximate solution of the linear system 𝐴 𝑘 𝐴𝑇𝑘 𝜆 = 𝐴 𝑘 𝑔𝑘 ̂ . and take it as an approximation of 𝜆 𝑘+1

The solution of this problem is characterized by

3. Global Convergence

In the next theorem, we show that if {𝑐𝑘 } is not convergent to zero, at least one of the accumulation points of {𝑥𝑘 } satisfies (35).

In this section, we discuss the global convergence of Algorithm 2. We assume that Algorithm 2 can find an infinite set {𝑥𝑘 } and give some assumptions in the following. Assumptions 1. (A1) 𝑓(𝑥) and 𝑐(𝑥) are twice continuously differentiable. (A2) {𝑥𝑘 } and {𝐵𝑘 } are bounded, where 𝐵𝑘 is positive 2 𝑙(𝑥𝑘 , 𝜆 𝑘 ). semidefinite approximation of ∇𝑥𝑥 Firstly, we give a result on the upper bound of the trial step. Lemma 4. If 𝑠𝑘 solves subproblem (23), then one has 󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔𝑘 󵄩 󵄩󵄩 󵄩󵄩 󵄩󵄩𝑠𝑘 󵄩󵄩 ≤ 4 󵄩 󵄩 𝜇𝑘 (32) 󵄩 ̃ 󵄩󵄩 󵄩 or 𝜇𝑘 ≤ 2 󵄩󵄩󵄩𝐵𝑘 󵄩󵄩󵄩 hold ∀𝑘 ≥ 0. Proof. Any approximate solution 𝑠𝑘 of (23) satisfies Pred𝑘 = Φ𝑘 (𝑥𝑘 ) − 𝑚𝑘 (𝑠𝑘 ) ≥ 0. Clearly, 0 ≤ Φ𝑘 (𝑥𝑘 ) − 𝑚𝑘 (𝑠𝑘 )

󵄩 󵄩 1 󵄩 󵄩 󵄩 󵄩 1 󵄩 󵄩 1 󵄩 󵄩2 = (󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 − 𝜇𝑘 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩) 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 + (󵄩󵄩󵄩󵄩𝐵̃𝑘 󵄩󵄩󵄩󵄩 − 𝜇𝑘 ) 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 . 4 2 2

𝑇

𝐴 (𝑥∗ ) 𝑐 (𝑥∗ ) = 0.

(35)

Theorem 6. Suppose that (A1)-(A2) hold and 𝜎𝑘 → ∞. If ‖𝑐𝑘 ‖ → 𝑐 > 0, then 󵄩 󵄩 lim inf 󵄩󵄩󵄩󵄩𝐴𝑇𝑘 𝑐𝑘 󵄩󵄩󵄩󵄩 = 0. 𝑘→∞

(36)

Proof. We prove this result by contradiction. Suppose that there exists some 𝜏 > 0 such that 󵄩󵄩 𝑇 󵄩󵄩 󵄩󵄩𝐴 𝑘 𝑐𝑘 󵄩󵄩 > 2𝜏, ∀𝑘 ≥ 0. (37) 󵄩 󵄩 By the definition of 𝑔̃𝑘 in (7), we know that 󵄩 𝑇 󵄩 󵄩 󵄩󵄩 ̃ 󵄩󵄩 𝑇 󵄩 󵄩󵄩𝑔𝑘 󵄩󵄩 ≥ 𝜎𝑘 󵄩󵄩󵄩󵄩𝐴 𝑘 𝑐𝑘 󵄩󵄩󵄩󵄩 − 󵄩󵄩󵄩󵄩𝑔𝑘 − 𝐴 𝑘 𝜆 𝑘 󵄩󵄩󵄩󵄩 .

(38)

As {𝑥𝑘 } and {𝜆 𝑘 } are bounded, we can deduce the boundedness of ‖𝑔𝑘 − 𝐴𝑇𝑘 𝜆 𝑘 ‖ by (A2); that is, there exists some 𝜏 > 0 such that 󵄩󵄩 󵄩 󵄩󵄩𝑔𝑘 − 𝐴𝑇𝑘 𝜆 𝑘 󵄩󵄩󵄩 < 𝜏. (39) 󵄩 󵄩 By (37), (38), and (39), we can conclude that

1 1 󵄩 󵄩2 = −𝑔̃𝑘𝑇 𝑠𝑘 − 𝑠𝑘𝑇 𝐵̃𝑘 𝑠𝑘 − 𝜇𝑘 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 2 2 󵄩 󵄩 󵄩 󵄩 1 󵄩 󵄩 󵄩 󵄩2 1 󵄩 󵄩2 ≤ 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 + 󵄩󵄩󵄩󵄩𝐵̃𝑘 󵄩󵄩󵄩󵄩 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 − 𝜇𝑘 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 2 2

𝑥∈𝑅

(33)

󵄩󵄩 ̃ 󵄩󵄩 󵄩󵄩𝑔𝑘 󵄩󵄩 ≥ 2𝜏𝜎𝑘 − 𝜏 > 𝜏𝜎𝑘

(40)

holds for all sufficiently large 𝑘. By the boundedness of 𝐵𝑘 and 𝐴 𝑘 , we can conclude that there exists 𝑀 > 0, such that 󵄩󵄩 ̃ 󵄩󵄩 󵄩󵄩𝐵𝑘 󵄩󵄩 < 𝑀𝜎𝑘 󵄩 󵄩

(41)


5

holds for all sufficiently large 𝑘, where 𝐵̃𝑘 is defined by (7). By (18), (40), and (41), 󵄩󵄩 ̃ 󵄩󵄩2 󵄩𝑔 󵄩 1 1 Pred𝑘 ≥ 󵄩 𝑘 󵄩 min { 󵄩󵄩 󵄩󵄩 , } 4 󵄩󵄩𝐵̃𝑘 󵄩󵄩 𝜇𝑘 󵄩 󵄩 󵄩󵄩 ̃ 󵄩󵄩 󵄩󵄩 ̃ 󵄩󵄩 󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔 󵄩 󵄩𝑔 󵄩 󵄩𝑔 󵄩 = 󵄩 𝑘 󵄩 min { 󵄩󵄩󵄩 𝑘 󵄩󵄩󵄩 , 󵄩 𝑘 󵄩 } 4 󵄩󵄩𝐵̃𝑘 󵄩󵄩 𝜇𝑘 󵄩 󵄩 󵄩 󵄩 𝜏𝜎 𝜏 󵄩󵄩𝑔̃ 󵄩󵄩 ≥ 𝑘 min { , 󵄩 𝑘 󵄩 } 4 𝑀 𝜇𝑘

(42)

holds for all sufficiently large 𝑘. By the update rule of 𝜎𝑘 and the fact that 𝜎𝑘 → ∞, we have that 󵄩 󵄩 󵄩󵄩 󵄩󵄩 󵄩󵄩𝑐𝑘 󵄩󵄩 max {1, 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩} 󵄩󵄩 󵄩󵄩2 (43) , 󵄩󵄩𝑐𝑘 󵄩󵄩 } Pred𝑘 < 𝛿𝑘 𝜎𝑘 min { 𝜇𝑘 holds for infinitely many 𝑘. As ‖𝑔̃𝑘 ‖ > 1 holds for all sufficiently 𝑘 by (40), it is easy to see that (42) contradicts to (43) as 𝛿𝑘 𝜎𝑘 → 0 and {‖𝑐𝑘 ‖} is convergent. Thus we can prove the desired result. Lemma 7. Suppose that (A1)-(A2) hold, 𝜎𝑘 → ∞, and ‖𝑐𝑘 ‖ → 0; then 󵄩 󵄩 lim inf 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 = 0. (44) 𝑘→∞ Proof. Assume that there exists 𝜏 > 0 such that 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 > 𝜏 > 0, ∀𝑘 ≥ 0. 󵄩 󵄩

(45)

Then, by (18) and (41), we know that, for all sufficiently large 𝑘, 󵄩 󵄩 𝜏 𝜏 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 (46) , }. Pred𝑘 ≥ min { 4 𝑀𝜎𝑘 𝜇𝑘 By the update rule of 𝜎𝑘 and 𝜎𝑘 → ∞, 󵄩 󵄩 󵄩󵄩 󵄩󵄩 󵄩𝑐 󵄩 max {1, 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩} 󵄩󵄩 󵄩󵄩2 , 󵄩󵄩𝑐𝑘 󵄩󵄩 } Pred𝑘 < 𝛿𝑘 𝜎𝑘 min { 󵄩 𝑘 󵄩 𝜇𝑘

(47)

holds for infinitely many 𝑘. We will prove that (47) contradicts to (46). Let 𝐾 be the index set containing all 𝑘 such that (47) holds. Therefore, for all 𝑘 ∈ 𝐾, 󵄩󵄩󵄩𝑐 󵄩󵄩󵄩 max {1, 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩} 󵄩 󵄩 , (48) Pred𝑘 < 𝛿𝑘 𝜎𝑘 󵄩 𝑘 󵄩 𝜇𝑘 󵄩 󵄩2 Pred𝑘 < 𝛿𝑘 𝜎𝑘 󵄩󵄩󵄩𝑐𝑘 󵄩󵄩󵄩 .

(49)

If there exists an infinite subset 𝐾1 ⊆ 𝐾 such that ‖𝑔̃𝑘 ‖ ≤ 1 holds for all 𝑘 ∈ 𝐾1 , then, by (48), it holds that 󵄩󵄩 󵄩󵄩 󵄩𝑐 󵄩 (50) Pred𝑘 < 𝛿𝑘 𝜎𝑘 󵄩 𝑘 󵄩 , ∀𝑘 ∈ 𝐾1 . 𝜇𝑘 As 𝛿𝑘 𝜎𝑘 → 0, ‖𝑐𝑘 ‖ → 0, and ‖𝑔̃𝑘 ‖ > 𝜏, (50) implies that 󵄩 󵄩 𝜏 󵄩󵄩𝑔̃ 󵄩󵄩 (51) Pred𝑘 < 󵄩 𝑘 󵄩 4 𝜇𝑘

holds for all sufficiently large 𝑘 ∈ 𝐾1 . If there exists an infinite subset 𝐾2 ⊆ 𝐾 such that ‖𝑔̃𝑘 ‖ > 1 holds for all 𝑘 ∈ 𝐾2 , then by (48) we have that 󵄩󵄩󵄩𝑐 󵄩󵄩󵄩 󵄩󵄩󵄩𝑔̃ 󵄩󵄩󵄩 (52) Pred𝑘 < 𝛿𝑘 𝜎𝑘 󵄩 𝑘 󵄩 󵄩 𝑘 󵄩 𝜇𝑘 for all 𝑘 ∈ 𝐾2 . Equation (52) also implies (51) as 𝛿𝑘 𝜎𝑘 → 0 and ‖𝑐𝑘 ‖ → 0. From (28) and (29), it follows that 𝛿𝑘 𝜎𝑘2 = 𝛿0 𝜎02 holds for all 𝑘 ≥ 0. Therefore, by (49) we know that, for all 𝑘 ∈ 𝐾, 󵄩 󵄩2 󵄩 󵄩2 1 󵄩 󵄩2 1 Pred𝑘 < 𝛿𝑘 𝜎𝑘 󵄩󵄩󵄩𝑐𝑘 󵄩󵄩󵄩 = 𝛿𝑘 𝜎𝑘2 󵄩󵄩󵄩𝑐𝑘 󵄩󵄩󵄩 = 𝛿0 𝜎02 󵄩󵄩󵄩𝑐𝑘 󵄩󵄩󵄩 . (53) 𝜎𝑘 𝜎𝑘 As ‖𝑐𝑘 ‖ → 0, (53) implies that Pred𝑘
0 such that 󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔 󵄩 (58) Pred𝑘 > 𝑐 󵄩 𝑘 󵄩 holds ∀sufficiently large 𝑘, 𝜇𝑘 then ∑𝑘∈𝑆 (‖𝑔̃𝑘 ‖/𝜇𝑘 ) is divergent.

6


Proof. We prove this lemma by contradiction. We will show that if ∑𝑘∈𝑆 (‖𝑔̃𝑘 ‖/𝜇𝑘 ) is convergent, then 𝜇𝑘+1 < 𝜇𝑘 holds for all sufficiently large 𝑘 which contradicts to the fact that 1/𝜇𝑘 → 0, as 𝑘 → ∞. Suppose that 󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔 󵄩 ∑ 󵄩 𝑘 󵄩 = 𝑠. 𝜇𝑘 𝑘∈𝑆

where 𝑅𝑘 is defined by Steps 0 and 4 in Algorithm 2. From Step 4 of Algorithm 2, we know that if 𝑘 ∉ 𝐾, then ‖𝑐𝑘+1 ‖ > 𝑅𝑘 and 𝜆 𝑘+1 = 𝜆 𝑘 . Hence we have ∞

∞

𝑘=0

𝑘=0

(59)

𝑇

= −𝜆𝑇0 𝑐0 + ∑ (𝜆 𝑘 − 𝜆 𝑘+1 ) 𝑐𝑘+1 𝑘∈𝐾

lim𝑘→∞ (1/𝜇𝑘 ) = 0 and (32) imply that

≤

󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔𝑘 󵄩 󵄩󵄩 󵄩󵄩 󵄩󵄩𝑠𝑘 󵄩󵄩 ≤ 4 󵄩 󵄩 𝜇𝑘

(60)

∞

󵄩 󵄩 󵄩 󵄩 ∑ 󵄩󵄩󵄩𝑥𝑘+1 − 𝑥𝑘 󵄩󵄩󵄩 = ∑ 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 .

󵄩 󵄩 󵄩 󵄩 + 2 󵄩󵄩󵄩𝜆 max 󵄩󵄩󵄩 ∑ 󵄩󵄩󵄩𝑐𝑘+1 󵄩󵄩󵄩

(68)

𝑘∈𝐾

󵄩 󵄩 ≤ −𝜆𝑇0 𝑐0 + 2 󵄩󵄩󵄩𝜆 max 󵄩󵄩󵄩 ∑ 𝑅𝑘 ,

(61)

where ‖𝜆 max ‖ is the upper bound of {𝜆 𝑘 }. From Step 4 and (67), we have

𝑘∈𝑆

∑ 𝑅𝑘 ≤ 𝑅0 (1 +

𝑘∈𝐾

Equations (59)–(61) imply that {𝑥𝑘 } is convergent. Let

1 1 + + ⋅ ⋅ ⋅) = 2𝑅0 , 2 4

(69)

which implies

𝑟𝑘 = (𝜌𝑘 − 𝛾2 ) Pred𝑘 = (1 − 𝛾2 ) Pred𝑘 + (Ared𝑘 − Pred𝑘 ) .

∞

(62)

󵄩 󵄩 ∑ (−𝜆𝑇𝑘 𝑐𝑘 + 𝜆𝑇𝑘 𝑐𝑘+1 ) ≤ −𝜆𝑇0 𝑐0 + 4𝑅0 󵄩󵄩󵄩𝜆 max 󵄩󵄩󵄩 .

(70)

𝑘=0

It is clear that 𝑟𝑘 > 0 ⇔ 𝜌𝑘 > 𝛾2 . By Taylor’s theorem, it holds that

Then, we have ∞

∞

𝑘=0

𝑘=0

∑ Ared𝑘 = ∑ [Φ𝑘 (𝑥𝑘 ) − Φ𝑘 (𝑥𝑘+1 )]

Ared𝑘 − Pred𝑘 = 𝑚𝑘 (𝑠𝑘 ) − Φ𝑘 (𝑥𝑘 + 𝑠𝑘 ) 𝑇

= − [∇Φ𝑘 (𝑥𝑘 + 𝜉𝑘 ) − 𝑔̃𝑘 ] 𝑠𝑘 1 1 󵄩 󵄩2 + 𝑠𝑘𝑇 𝐵̃𝑘 𝑠𝑘 + 𝜇𝑘 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 2 2 󵄩󵄩 󵄩 󵄩 ≥ − 󵄩󵄩󵄩∇Φ𝑘 (𝑥𝑘 + 𝜉𝑘 ) − 𝑔̃𝑘 󵄩󵄩󵄩 󵄩󵄩󵄩𝑠𝑘 󵄩󵄩󵄩 ,

󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔 󵄩 󵄩 󵄩󵄩 󵄩 Ared𝑘 − Pred𝑘 ≥ −4 󵄩󵄩∇Φ𝑘 (𝑥𝑘 + 𝜉𝑘 ) − 𝑔̃𝑘 󵄩󵄩 󵄩 𝑘 󵄩 𝜇

∞

𝑘=0

𝑘=0

+

𝜎0 ∞ 󵄩󵄩 󵄩󵄩2 󵄩󵄩 󵄩󵄩2 ∑ (󵄩𝑐 󵄩 − 󵄩𝑐 󵄩 ) 2 𝑘=0 󵄩 𝑘 󵄩 󵄩 𝑘+1 󵄩

(71)

𝜎 󵄩 󵄩2 󵄩 󵄩 ≤ 𝑓0 + (−𝜆𝑇0 𝑐0 + 4𝑅0 󵄩󵄩󵄩𝜆 max 󵄩󵄩󵄩) + 0 󵄩󵄩󵄩𝑐0 󵄩󵄩󵄩 2 ≜ 𝐴,

(64)

𝑘

where Φ𝑘 (𝑥) is defined by (5). Secondly, we prove

holds for all sufficiently large 𝑘 and thus 󵄩 󵄩 󵄩󵄩 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩 󵄩󵄩 ̃ . 𝑟𝑘 ≥ [𝑐 (1 − 𝛾2 ) − 4 󵄩󵄩∇Φ𝑘 (𝑥𝑘 + 𝜉𝑘 ) − 𝑔𝑘 󵄩󵄩] 𝜇

∞

= ∑ (𝑓𝑘 − 𝑓𝑘+1 ) + ∑ (−𝜆𝑇𝑘 𝑐𝑘 + 𝜆𝑇𝑘 𝑐𝑘+1 )

(63)

where 𝜉𝑘 is a convex combination of 𝑥𝑘 and 𝑥𝑘 +𝑠𝑘 . According to (60), we have that

󵄩 󵄩 lim inf 󵄩󵄩󵄩𝑐𝑘 󵄩󵄩󵄩 = 0 𝑘→∞

(65)

𝑘

The convergence of {𝑥𝑘 } and the boundedness of {𝜆 𝑘 } imply that ‖∇Φ𝑘 (𝑥𝑘 + 𝜉k ) − 𝑔̃𝑘 ‖ → 0. Therefore, for all sufficiently large 𝑘, 𝑟𝑘 > 0. This implies that 𝜌𝑘 > 𝛾2 and 𝜇𝑘+1 < 𝜇𝑘 . Lemma 10. Suppose that (A1)-(A2) hold and 𝜎𝑘 = 𝜎0 for all 𝑘 ≥ 0; then we have that 󵄩 󵄩 lim 󵄩󵄩𝑐𝑘 󵄩󵄩 = 0. (66) 𝑘→∞ 󵄩 󵄩 Proof. Firstly, we prove that the sum of Ared𝑘 is bounded. Define the indices set 󵄩 󵄩 𝐾 = {𝑘 | 󵄩󵄩󵄩𝑐𝑘+1 󵄩󵄩󵄩 ≤ 𝑅𝑘 } ,

−𝜆𝑇0 𝑐0

𝑘∈𝐾

holds for all sufficiently large 𝑘. By the definition of 𝑆,

𝑘=0

𝑇

∑ (−𝜆𝑇𝑘 𝑐𝑘 + 𝜆𝑇𝑘 𝑐𝑘+1 ) = −𝜆𝑇0 𝑐0 + ∑ (𝜆 𝑘 − 𝜆 𝑘+1 ) 𝑐𝑘+1

(67)

(72)

by contradiction. Suppose that there exists some 𝜏 > 0 such that 󵄩󵄩 󵄩󵄩 (73) 󵄩󵄩𝑐𝑘 󵄩󵄩 > 𝜏 > 0, ∀𝑘 ≥ 0. Equations (56) and (73) imply that Pred𝑘 ≥ 𝛿0 𝜎0 min {

󵄩 󵄩 max {1, 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩} 2 𝜏, 𝜏 } . 𝜇𝑘

(74)

Considering the sum of Pred𝑘 on the index set 𝑆 (see (57)), we have by (71) that ∑ Pred𝑘 ≤

𝑘∈𝑆

1 1 ∞ 1 ∑ Ared𝑘 = ∑ Ared𝑘 ≤ 𝐴. 𝛾1 𝑘∈𝑆 𝛾1 𝑘=0 𝛾1

(75)

Discrete Dynamics in Nature and Society It can be deduced by (74) and (75) that 󵄩 󵄩 max {1, 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩} < +∞ ∑ 𝜇𝑘 𝑘∈𝑆 and thus

󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔 󵄩 ∑ 󵄩 𝑘 󵄩 < +∞, 𝜇𝑘 𝑘∈𝑆 1 ∑ < +∞. 𝜇 𝑘∈𝑆 𝑘

7 Thus by Lemma 4, we have that, for all sufficiently large 𝑘 ∈ 𝐾,

(77)

holds for all sufficiently large 𝑘. Hence it can be deduced by Lemma 9 that ∑𝑘∈𝑆 (‖𝑔̃𝑘 ‖/𝜇𝑘 ) is divergent which contradicts the first part in (77). Finally, we prove (66). If 𝑆 is a finite set, then {𝑥𝑘 } is convergent. Thus, (72) implies (66). From now on we assume that 𝑆 is an infinite set. Suppose that (66) does not hold; then ̃ = {𝑘𝑖 } (𝐾 ̃ ⊂ 𝑆) and a there exist an infinite index set 𝐾 constant 𝑐 > 0 such that 󵄩󵄩 󵄩󵄩 ̃ 󵄩󵄩𝑐𝑘𝑖 󵄩󵄩 > 2𝑐 holds ∀𝑘𝑖 ∈ 𝐾. (79) 󵄩 󵄩 ̂ = {𝑙𝑖 } (𝐾 ̂ ⊂ 𝑆) By (72), there also exists an infinite index set 𝐾 such that 𝑘𝑖 < 𝑙𝑖 , 󵄩󵄩󵄩𝑐𝑘 󵄩󵄩󵄩 ≥ 𝑐 holds, ∀𝑘 such that 𝑘𝑖 ≤ 𝑘 < 𝑙𝑖 , (80) 󵄩 󵄩 󵄩󵄩󵄩𝑐 󵄩󵄩󵄩 < 𝑐 holds ∀𝑙 ∈ 𝐾. ̂ (81) 󵄩󵄩 𝑙𝑖 󵄩󵄩 𝑖 ̃ ⊂ 𝐾 and 𝐾 is an infinite Let 𝐾 = {𝑘 | 𝑘 ∈ 𝑆, ‖𝑐𝑘 ‖ ≥ 𝑐}; then 𝐾 index set, ∑ Pred𝑘 ≤ ∑ Pred𝑘 . 𝑘∈𝑆

(82)

Therefore, by (75), we have that ∑ Pred𝑘 ≤ 𝑘∈𝐾

1 𝐴. 𝛾1

With the help of (56), (80), and (83), we obtain that 󵄩 󵄩 max {1, 󵄩󵄩󵄩𝑔̃𝑘 󵄩󵄩󵄩} < +∞. ∑ 𝜇𝑘

(83)

󵄩󵄩 󵄩 󵄩󵄩𝑥𝑙𝑖 − 𝑥𝑘𝑖 󵄩󵄩󵄩 ≤ 󵄩 󵄩

∑ 𝑘𝑖 ≤𝑘 𝜏

holds, ∀𝑘.

(89)

By (18), we have that Pred𝑘 ≥

1 𝜏2 1 min { 󵄩󵄩 󵄩󵄩 , } 4 󵄩󵄩𝐵̃𝑘 󵄩󵄩 𝜇𝑘 󵄩 󵄩

holds ∀𝑘.

(90)

As {𝐵̃𝑘 } is bounded above, similar to the second part in the proof of Lemma 10, we can conclude that 1 󳨀→ 0, 𝜇𝑘

as 𝑘 󳨀→ ∞,

(91)

holds for all sufficiently large 𝑘.

(92)

and thus Pred𝑘 ≥

𝜏2 1 4 𝜇𝑘

By (75) and (92), we have that ∑𝑘∈𝑆 (1/𝜇𝑘 ) is convergent and thus ∑𝑘∈𝑆 (‖𝑔̃𝑘 ‖/𝜇𝑘 ) is also convergent as {𝑔̃𝑘 } is bounded. However, Lemma 9, (91), (92), and the boundedness of {𝑔̃𝑘 } deduce the divergence of ∑𝑘∈𝑆 (‖𝑔̃𝑘 ‖/𝜇𝑘 ). This contradiction completes the proof. With the help of Lemmas 10 and 11, we can easily obtain the following result.

(84)

Theorem 12. Suppose that (A1)-(A2) hold and 𝜎𝑘 = 𝜎0 for all 𝑘 ≥ 0; there exists an accumulation point of {𝑥𝑘 } at which the KKT condition holds.

(85)

Note that, in Theorem 12, we do not suppose that RCPLD holds.

𝑘∈𝐾

A direct conclusion which can be drawn by (84) is 1 󳨀→ 0, as 𝑘 ∈ 𝐾, 𝑘 󳨀→ ∞. 𝜇𝑘

(86)

Therefore, for sufficiently large 𝑗,

If 𝑆 is a finite set, then it follows from (57) and Step 2 that 𝜌𝑘 ≥ 𝛾1 and 𝜇𝑘+1 = 𝜃3 𝜇𝑘 (𝜃3 > 1) hold for all sufficiently large 𝑘. Therefore, 1/𝜇𝑘 → 0, as 𝑘 → ∞. If 𝑆 is an infinite set, the second inequality in (77) implies that 1/𝜇𝑘 → 0, as 𝑘 → ∞ and 𝑘 ∈ 𝑆. From Step 2, we know that if 𝑘 ∉ 𝑆, then 𝜇𝑘+1 ≥ 𝜇𝑘 . Hence, we have 1/𝜇𝑘 → 0, as 𝑘 → ∞. The fact that 1/𝜇𝑘 → 0 and (74) imply that 󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔 󵄩 (78) Pred𝑘 ≥ 𝛿0 𝜎0 𝜏 󵄩 𝑘 󵄩 𝜇𝑘

𝑘∈𝐾

󵄩󵄩 ̃ 󵄩󵄩 󵄩𝑔𝑘 󵄩 󵄩󵄩 󵄩󵄩 󵄩󵄩𝑠𝑘 󵄩󵄩 ≤ 4 󵄩 󵄩 . 𝜇𝑘

(76)

8

Discrete Dynamics in Nature and Society Table 1: Results of Algorithm 2 and ALGENCAN.

Name AIRCRFTA ARGTRIG BOOTH BROWNALE BROYDN3D BYRDSPHR BT1 BT2 BT4 BT5 BT7 BT8 BT9 BT10 BT11 BT12 CHNRSNBE CLUSTER CUBENE DECONVNE EIGENB GOTTFR HATFLDF HEART6 HEART8 HS39 HS48 HYDCAR6 INTEGREQ MARATOS MWRIGHT ORTHREGB RECIPE RSNBRNE SINVALNE TRIGGER YFITNE ZANGWIL3

n

m

8 200 2 200 5000 3 2 3 3 3 5 5 4 2 5 5 50 2 2 63 7 2 3 6 8 4 5 29 502 2 5 27 3 2 2 7 3 3

5 200 2 200 5000 2 1 1 2 1 3 2 2 2 3 3 98 2 2 40 7 2 3 6 8 2 2 29 500 1 3 6 3 2 2 6 17 3

4. Numerical Experiment In this Section, we investigate the performance of Algorithm 2. We compare Algorithm 2 with the famous Fortran package ALGENCAN. In our computer program, the parameters in Algorithm 2 are chosen as follows: 𝜃1 = 0.1, 𝜃2 = 1, 𝜃3 = 4, 𝛾1 = 0.1, 𝛾2 = 0.9, 𝜎0 = 1,

Algorithm 2 𝑛𝑓 5 69 6 8 23 42 17 40 17 12 5 19 18 15 15 10 100 11 8 6 2550 6 5 122 29 65 46 267 6 13 14 6 6 6 6 481 11 6

ALGENCAN 𝑛𝑔 5 69 6 8 23 21 17 37 17 11 5 15 15 15 15 10 37 11 6 6 2550 6 5 61 10 61 19 85 6 11 14 6 6 6 6 290 11 6

𝑛𝑓 20 15 5 21 22 80 67 74 34 28 97 76 106 95 75 20 19 26 17 3982 2451 24 55 387 87 106 6 212 14 42 29 23 39 16 8 757 805 5

𝑛𝑔 21 16 6 22 23 78 45 71 34 28 98 51 100 95 75 21 20 27 18 1195 608 24 45 216 43 100 7 154 15 43 29 24 40 17 8 270 520 6

𝛿0 = 1, 𝜆 = 1020 , 𝜆 = 10−20 . (93) We set 𝐵𝑘 to be the exact Hessian of the Lagrangian 𝑓(𝑥) − 𝜆𝑇 𝑐(𝑥) at the point 𝑥𝑘 . The Matlab subroutine minres is used to solve (15). All algorithms are terminated when one of the following conditions holds: (1) ‖𝑔𝑘 +𝐴𝑇𝑘 𝜆 𝑘 ‖ ≤ 10−8 and ‖𝑐𝑘 ‖ ≤ 10−8 ; (2) ‖𝑔𝑘 + 𝐴𝑇𝑘 𝜆 𝑘 ‖ ≤ 10−8 and ‖𝐴𝑇𝑘 𝑐𝑘 ‖ ≤ 10−8 ; (3) ‖𝑠𝑘 ‖ ≤ 10−8 . All test problems are chosen from CUTEst collection [22]. The numerical results are listed in Table 1 where the name of problem is denoted by 𝑁𝑎𝑚𝑒, the number of its variables

Discrete Dynamics in Nature and Society is denoted by 𝑛, the number of constraints is denoted by 𝑚, the number of function evaluations is denoted by 𝑛𝑓 , and the number of gradient evaluations is denoted by 𝑛𝑔 . In Table 1, we list the results of 38 test problems. Considering the numbers of function evaluations (𝑛𝑓 ), Algorithm 2 is better than ALGENCAN for 30 cases (78.9%). Considering the numbers of gradient evaluations (𝑛𝑔 ), Algorithm 2 is better than ALGENCAN for 31 cases (81.6%).

5. Conclusions In this paper, we present a new algorithm for equality constrained optimization. We add an adaptive quadratic term to the quadratic model of the augmented Lagrangian function. In each iteration, we solve a simple unconstrained subproblem to obtain the trail step. The global convergence is established under reasonable assumptions. From the numerical results and the theoretical analysis, we believe that the new algorithm can efficiently solve equality constrained optimization problems.

9

[9]

[10]

[11]

[12]

[13]

Conflicts of Interest

[14]

The authors declare that there are no conflicts of interest regarding the publication of this paper.

[15]

Acknowledgments This work is supported by NSFC (11771210, 11471159, 11571169, and 61661136001) and the Natural Science Foundation of Jiangsu Province (BK20141409).

[16]

[17]

References [1] M. R. Hestenes, “Multiplier and gradient methods,” Journal of Optimization Theory and Applications, vol. 4, pp. 303–320, 1969. [2] M. J. D. Powell, “A method for nonlinear constraints in minimization problems,” in Optimization, R. Fletcher, Ed., pp. 283– 298, Academic Press, New York, NY, USA, 1969. [3] A. R. Conn, N. I. Gould, and P. L. Toint, “A globally convergent augmented Lagrangian algorithm for optimization with general constraints and simple bounds,” SIAM Journal on Numerical Analysis, vol. 28, no. 2, pp. 545–572, 1991. [4] A. R. Conn, N. I. Gould, and P. L. Toint, ,LANCELOT: A Fortran Package for Large-scale Nonlinear Optimization(Release A), vol. 17 of Springer, New York, USA, 1992. [5] R. Andreani, E. G. Birgin, J. M. Martnez, and M. L. Schuverdt, “Augmented Lagrangian methods under the constant positive linear dependence constraint qualification,” Mathematical Programming, vol. 111, no. 1-2, Ser. B, pp. 5–32, 2008. [6] R. Andreani, E. G. Birgin, J. M. Martnez, and M. L. Schuverdt, “On augmented Lagrangian methods with general lower-level constraints,” SIAM Journal on Optimization, vol. 18, no. 4, pp. 1286–1309, 2007. [7] E. G. Birgin and J. M. Martnez, “Augmented Lagrangian method with nonmonotone penalty parameters for constrained optimization,” Computational Optimization and Applications, vol. 51, no. 3, pp. 941–965, 2012. [8] E. G. Birgin and J. M. Martnez, “On the application of an augmented Lagrangian algorithm to some portfolio problems,”

[18]

[19]

[20] [21]

[22]

EURO Journal on Computational Optimization, vol. 4, no. 1, pp. 79–92, 2016. Z. Dost´al, “Semi-monotonic inexact augmented Lagrangians for quadratic programming with equality constraints,” Optimization Methods & Software, vol. 20, no. 6, pp. 715–727, 2005. Z. Dost´al, A. Friedlander, and S. A. Santos, “Augmented Lagrangians with adaptive precision control for quadratic programming with simple bounds and equality constraints,” SIAM Journal on Optimization, vol. 13, no. 4, pp. 1120–1140, 2003. F. E. Curtis, H. Jiang, and D. P. Robinson, “An adaptive augmented Lagrangian method for large-scale constrained optimization,” Mathematical Programming, vol. 152, no. 1-2, Ser. A, pp. 201–245, 2015. F. E. Curtis, N. I. Gould, H. Jiang, and D. P. Robinson, “Adaptive augmented Lagrangian methods: algorithms and practical numerical experience,” Optimization Methods & Software, vol. 31, no. 1, pp. 157–186, 2016. A. F. Izmailov, M. V. Solodov, and E. I. Uskov, “Global convergence of augmented Lagrangian methods applied to optimization problems with degenerate constraints, including problems with complementarity constraints,” SIAM Journal on Optimization, vol. 22, no. 4, pp. 1579–1606, 2012. L. Niu and Y. Yuan, “A new trust-region algorithm for nonlinear constrained optimization,” Journal of Computational Mathematics, vol. 28, no. 1, pp. 72–86, 2010. X. Wang and Y. Yuan, “An augmented Lagrangian trust region method for equality constrained optimization,” Optimization Methods & Software, vol. 30, no. 3, pp. 559–582, 2015. C. Cartis, N. I. Gould, and L. Toint, “Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results,” Mathematical Programming, vol. 127, no. 2, Ser. A, pp. 245–295, 2011. C. Cartis, N. I. Gould, and L. Toint, “Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity,” Mathematical Programming, vol. 130, no. 2, Ser. A, pp. 295–319, 2011. K. Ueda and N. Yamashita, “A regularized Newton method without line search for unconstrained optimization,” Computational Optimization and Applications, vol. 59, no. 1-2, pp. 321– 351, 2014. H. Zhang and Q. Ni, “A new regularized quasi-Newton algorithm for unconstrained optimization,” Applied Mathematics and Computation, vol. 259, pp. 460–469, 2015. Y. X. Yuan, “On the convergence of a new trust region algorithm,” Numerische Mathematik, vol. 70, no. 4, pp. 515–539, 1995. R. Andreani, G. Haeser, M. a. Schuverdt, and P. J. Silva, “A relaxed constant positive linear dependence constraint qualification and applications,” Mathematical Programming, vol. 135, no. 1-2, Ser. A, pp. 255–273, 2012. N. I. M. Gould, D. Orban, and P. L. Toint, “CUTEst: a Constrained and Unconstrained Testing Environment with safe threads for mathematical optimization,” Computational Optimization and Applications, vol. 60, no. 3, pp. 545–557, 2015.

Advances in

Operations Research Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

Advances in

Decision Sciences Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

Journal of

Applied Mathematics

Algebra

Hindawi Publishing Corporation http://www.hindawi.com


Volume 2014

Journal of

Probability and Statistics Volume 2014

The Scientific World Journal Hindawi Publishing Corporation http://www.hindawi.com


Volume 2014

International Journal of

Differential Equations Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

Volume 2014

Submit your manuscripts at https://www.hindawi.com International Journal of

Advances in

Combinatorics Hindawi Publishing Corporation http://www.hindawi.com

Mathematical Physics Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

Journal of

Complex Analysis Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

International Journal of Mathematics and Mathematical Sciences

Mathematical Problems in Engineering

Journal of

Mathematics Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014


Volume 2014

Volume 2014


Volume 2014

#HRBQDSDĮ,@SGDL@SHBR

Journal of

Volume 201



Journal of

Function Spaces Hindawi Publishing Corporation http://www.hindawi.com

Abstract and Applied Analysis

Volume 2014


Volume 2014


Volume 2014

International Journal of

Journal of

Stochastic Analysis

Optimization



Volume 2014

Volume 2014