Descent direction of two algorithms conjugate gradient Jafar Fathali Department of Mathematics, Shahrood University of Technology, Shahrood, Iran.
[email protected] Mehrdad.Moshtagh Faculty of Mathematical Sciences, Sharif University of Technology, Tehran, Iran
Abstract In this paper we consider the secant equations and their improvements. We show that search directions which are obtained from two hybrid conjugate gradient methods are descent. Two approaches for computing the initial value of the steplength proposed by Babaie et al. and Andrei are used for accelerating the performance of the line search. Using the performance profile of Dolan and Moré on the comparative results, the effectiveness of the use of Andrei’s line search method on the hybrid conjugate gradient methods is shown.
Keywords. Unconstrained optimization. Hybrid conjugate gradient algorithm. Modified BFGS method. Descent direction.
1 Introduction Conjugate gradient method is one of the useful methods for solving nonlinear large scale sets. We can use this method for solving the following unconstrained optimization problem, min 𝑓(𝑥),
(1)
𝑥∈ℝ𝑛
In which f is a smooth function. The similarity of iterative in this method causes to using less memory. The effectiveness of conjugate gradient as one of the best exist methods result from global and local convergence [1]. Notations For a sufficiently smooth function f, we use the following notations 𝑓𝑘 = 𝑓(𝑥𝑘 ),
𝑔𝑘 = ∇𝑓(𝑥𝑘 ),
𝐺𝑘 = ∇2 𝑓(𝑥𝑘 ),
𝑦𝑘 = 𝑔𝑘+1 − 𝑔𝑘 ,
𝑠𝑘 = 𝑥𝑘+1 − 𝑥𝑘 ,
‖. ‖ Denote the Euclidean norm. The iterative formula of a conjugate gradient method is 𝑥𝑘+1 = 𝑥𝑘 + 𝑠𝑘 ,
𝑠𝑘 = 𝛼𝑘 𝑑𝑘 ,
Where 𝛼𝑘 > 0 is a steplength and 𝑑𝑘 is search direction defined by
(2)
−∇𝑓(𝑥0 ) 𝑑𝑘+1 = { −∇𝑓(𝑥𝑘+1 ) + 𝛽𝑘 𝑑𝑘
𝑘=0 𝑘 >0,
(3)
Where 𝛽𝑘 is a scaler and namely conjugate gradient parameter. The steplength of 𝛼𝑘 can be computed by exact line search as follow; 𝛼𝑘 = 𝑎𝑟𝑔 min 𝑓(𝑥𝑘 + 𝛼𝑑𝑘 ).
(4)
𝛼≥0
Because of the time cost of computing steplength in exact line search, one of usual terminate criteria such as Wolf line search can be used, i.e. 𝑓(𝑥𝑘 + 𝛼𝑘 𝑑𝑘 ) − 𝑓(𝑥𝑘 ) ≤ 𝛿𝛼𝑘 ∇𝑓(𝑥𝑘 )𝑇 𝑑𝑘 ∇𝑓(𝑥𝑘 + 𝛼𝑘 𝑑𝑘 )𝑇 𝑑𝑘 ≥ 𝜎∇𝑓(𝑥𝑘 )𝑇 𝑑𝑘
(5)
Where𝑑𝑘 is a descent direction, (i.e.∇𝑓(𝑥𝑘 )𝑇 𝑑𝑘 < 0) and 0 < 𝛿 < 𝜎 < 1. The conjugate gradient method was introduced by Hestense and Steifel [2] for solving linear set in 1952 for first time. Fletcher and Reeves [3] extended conjugate gradient method for nonlinear set.Al-Baali [4] explained the global convergence of Fletcher-Reeves method with approaches line search. Under proper assumptions usually used Zoutendijk conditions for proving global convergence of nonlinear conjugate gradient (CG) methods [5].Touati et al. [6] with assuming strong wolf conditions proving global convergence of CG method.Polak and Ribiére [7] proposed an algorithm with defined a new conjugate gradient parameter. Gilbert and Nocedal [8] shown global convergence of Fletcher-Reeves and PolyakRibiére methods. In [9], Andrei classified conjugate gradient algorithms in five category: classical, Hybrid, scaled, modified and parametric. In this category, classical and Hybrid conjugate gradient methods have been used more in compare with methods of other researchers. Hestense- Steifel (HS) and Dai-Yuan (DY) hybrid conjugate gradient methods proposed by Dai and Yuan [10], and Andrei [11,12]. The HS method has nice properties to adjust conjugacy conditions 𝑦𝑘 𝑇 𝑑𝑘+1 = 0, ∀𝑘 ≥ 0, that independent from the linear search conditions and objective function convexity. The DY method has remarkable strong convergence properties in compare with other CG methods [13]. This led to that the researchers trying to combine these methods. Hybrid conjugate gradient parameter of HS and DY algorithms is as follow [12] 𝛽𝑘𝐶 = (1 − 𝜃𝑘 )𝛽𝑘𝐻𝑆 + 𝜃𝑘 𝛽𝑘𝐷𝑌 = (1 − 𝜃𝑘 )
𝑇 𝑇 𝑔𝑘+1 𝑦𝑘 𝑔𝑘+1 𝑔𝑘+1 + 𝜃 𝑘 𝑇 𝑇 𝑦𝑘 𝑑𝑘 𝑦𝑘 𝑑𝑘
Where 0 ≤ 𝜃𝑘 ≤ 1 is scaler and namely hybridization parameter of CG method. Form (3) search direction can be computed as follow 𝑑𝑘+1 = −𝑔𝑘+1 + (1 − 𝜃𝑘 )
𝑇 𝑔𝑘+1 𝑦𝑘
𝑦𝑘
𝑇𝑑
𝑘
𝑑𝑘 + 𝜃𝑘
𝑇 𝑔𝑘+1 𝑔𝑘+1
𝑦𝑘 𝑇 𝑑𝑘
𝑑𝑘
(7)
We know, if the point 𝑥𝑘+1 sufficiently close to a local minimizer 𝑥 ∗ , then Newton direction used as −1 usual direction which 𝑑𝑘+1 = −∇2 𝑓𝑘+1 𝑔𝑘+1 .Andrei in [11,12] used Newton direction in (7) and computed𝜃𝑘 as follow
𝑇
𝜃𝑘 =
𝑔 𝑦𝑘 𝑇 2 𝑠𝑘𝑇 ∇2 𝑓𝑘+1 𝑔𝑘+1 −𝑠𝑘𝑇 𝑔𝑘+1 − 𝑘+1 𝑠𝑘 ∇ 𝑓𝑘+1 𝑠𝑘 𝑇 𝑦𝑘 𝑠𝑘
𝑔𝑇 𝑘+1 𝑔𝑘 𝑠𝑇 ∇2 𝑓 𝑘+1 𝑠𝑘 𝑦𝑘 𝑇 𝑠𝑘 𝑘
.
(8)
2 Formulas for the hybridization parameter 𝜽𝒌 In themid-50 decade Dividon proposed Quasi-Newton method for the first time. In this method for obtaining new point in iterations, only the objective function gradient information is needed. The quasiNewton methods were extended based on Newton method. In these methods, search direction computes from following set, 𝐵𝑘 𝑑𝑘 = −𝑔𝑘 ,
(9)
Where matrix 𝐵𝑘 is an approach form Hessian f in point 𝑥𝑘 . The matrix 𝐵𝑘 is computed such that satisfy in the secant equation 𝐵𝑘+1 𝑠𝑘 = 𝑦𝑘 ,
(10)
Now, with using secant equation (10) and equation (8) we should compute the following parameter [12] 𝑠𝑇 𝑔𝑘+1 . 𝑘+1 𝑔𝑘
𝜃𝑘 = − 𝑔𝑘𝑇
(11)
In [11], with using Li, Tang and Wei [14] modified secant equation as follow, ∗ 𝐵𝑘 𝑠𝑘−1 = 𝑦𝑘−1 ,
(12)
Where ∗ 𝑦𝑘−1 = 𝑦𝑘−1 +
𝑣𝑘
‖𝑠𝑘−1 ‖2
𝑠𝑘−1 ,
𝑣𝑘 = 2(𝑓𝑘 − 𝑓𝑘+1 ) + (𝑔𝑘 + 𝑔𝑘+1 )𝑇 𝑠𝑘 .
(13)
𝜃𝑘 Can be obtained as follow (
𝜃𝑘 =
𝑔𝑇 𝑇 𝑘+1 𝑦𝑘 2 −1)𝑠𝑘 𝑔𝑘+1 − 𝑦 𝑇 𝑠 𝑣𝑘 ‖𝑠𝑘 ‖ 𝑘 𝑘 𝑔𝑇 𝑔𝑘 𝑇 𝑘+1 𝑔𝑘+1 𝑔𝑘 + 𝑇 𝑣𝑘 𝑦𝑘 𝑠 𝑘 𝑣𝑘
.
(14)
One of the usual quasi-Newton methods is BFGS method, in which named from Broyden, Fletcher, Goldfarb and Shanno. In this method matrix 𝐵𝑘 updates as follow 𝐵𝑘+1 = 𝐵𝑘 −
𝐵𝑘 𝑠𝑘 𝑠𝑘𝑇 𝐵𝑘 𝑠𝑘𝑇 𝐵𝑘 𝑠𝑘
𝑦 𝑦𝑇
+ 𝑠𝑇𝑘𝑦𝑘 . 𝑘 𝑘
(15)
If 𝐵𝑘 be symmetric and positive definite and 𝑠𝑘𝑇 𝑦𝑘 > 0, then matrix 𝐵𝑘+1would also is positive definite and therefore the direction that obtained from (10) is a descent direction for f from 𝑥𝑘 [15]. If f be convex, then BFGS method would be global convergence and we have superlinear convergency by using exact linear search [16, 17]. Mascarenhas shown if f is nonconvex and exact line search is used in this method, then BFGS method would not be global convergence [18]. Liu in [19, 20], Li and Fukushima in [21] proposed global convergency of this method for nonconvex functions by modified BFGS algorithm. Yuan[22] with using interpolation conditions modified this method.In [23], two formulas proposed for hybridization parameter conjugate gradient based on modified BFGS methods of Yuan[22] and Li and Fukushima[21].Yuan’s modified BFGS method, is defined as follow 𝐵𝑘+1 = 𝐵𝑘 − 𝑡𝑘 =
𝐵𝑘 𝑠𝑘 𝑠𝑘𝑇 𝐵𝑘 𝑠𝑘𝑇 𝐵𝑘 𝑠𝑘
+ 𝑡𝑘
𝑦𝑘 𝑦𝑘𝑇 𝑠𝑘𝑇 𝑦𝑘
,
(16)
2(𝑓𝑘 −𝑓𝑘+1 +𝑠𝑘𝑇 𝑔𝑘+1 )
,
𝑠𝑘𝑇 𝑦𝑘
(17)
Form (17) we have 𝐵𝑘+1 𝑠𝑘 = 𝑦̃𝑘 ,
𝑦̃𝑘 = 𝑡𝑘 𝑦𝑘 = 𝑦𝑘 +
𝑣𝑘 𝑦 𝑠𝑘𝑇 𝑦𝑘 𝑘
(18)
Which 𝑣𝑘 defined as (13). Following Theorem shows that, when ‖𝑠𝑘 ‖ is sufficient small, then (18) obtain better approach from (9) for Hessian f [14]. Theorem 1 if f is sufficiently smooth and ‖𝑠𝑘 ‖ is sufficient small, then 1
𝑇 𝑇 𝑇 (𝑇 4 𝑠𝑘−1 ∇2 𝑓𝑘+1 𝑠𝑘−1 − 𝑠𝑘−1 𝑦̃𝑘−1 = 3 𝑠𝑘−1 𝑘 𝑠𝑘−1 )𝑠𝑘−1 + 𝑂(‖𝑠𝑘−1 ‖ ) 1 2
𝑇 𝑇 𝑇 (𝑇 4 𝑠𝑘−1 ∇2 𝑓𝑘+1 𝑠𝑘−1 − 𝑠𝑘−1 𝑦𝑘−1 = 𝑠𝑘−1 𝑘 𝑠𝑘−1 )𝑠𝑘−1 + 𝑂(‖𝑠𝑘−1 ‖ )
(19) (20)
Where, 𝜕3 𝑓(𝑥 )
𝑗
𝑘 𝑛 𝑇 (𝑇 𝑖 𝑙 𝑠𝑘−1 𝑘 𝑠𝑘−1 )𝑠𝑘−1 = ∑𝑖,𝑗,𝑙=1 𝜕𝑥 𝑖 𝜕𝑥 𝑗 𝜕𝑥 𝑘 𝑠𝑘 𝑠𝑘 𝑠𝑘 .
(21)
In [23], have been assumed that𝐵𝑘+1 (which is satisfying (18) conditions) is an approach from Hessian and 𝜃𝑘 is computed as follow 𝜃𝑘 = −
𝑠𝑘𝑇 𝑔𝑘+1 𝑇
𝑔 𝑔 𝑔𝑘𝑇 𝑔𝑘+1 + 𝑘𝑇 𝑘+1𝑣𝑘
.
(22)
𝑠𝑘 𝑦𝑘
Hence, the search direction computes as follow 𝑦𝑘𝑇 𝑔𝑘+1 𝑠𝑘 𝑇 𝑦𝑘
𝑑𝑘+1 = −𝑔𝑘+1 + (
−
𝑠𝑘𝑇 𝑔𝑘+1 ) 𝑠𝑘 , 𝑠𝑘 𝑇 𝑦𝑘 +𝑣𝑘
(23)
In the following Theorem we show the direction in (23) is descent. Theorem 2 Assume that f is convex function and 𝜃𝑘 computed from (22), 𝛼𝑘 steplength from Wolf line search condition (5). If 0 < 𝜃𝑘 < 1 and (𝑦𝑘𝑇 𝑔𝑘+1 )(𝑠𝑘𝑇 𝑔𝑘+1 ) 𝑠𝑘𝑇 𝑦𝑘
≤ ‖𝑔𝑘+1 ‖2 ,
(24)
Then, the search direction 𝑑𝑘+1 which computed from (23) is a descent direction. Prove we have, 𝑇 𝑔𝑘+1 𝑑𝑘+1 = −‖𝑔𝑘+1 ‖2 +
(𝑦𝑘𝑇 𝑔𝑘+1 )(𝑠𝑘𝑇 𝑔𝑘+1 ) 𝑠𝑘𝑇 𝑦𝑘
2
−
(𝑠𝑘𝑇 𝑔𝑘+1 )
𝑠𝑘𝑇 𝑦𝑘 +𝑣𝑘
,
(25)
Because f is convex function, so 𝑠𝑘𝑇 𝑦𝑘 + 𝑣𝑘 ≥ 0, and with using theorem assumption, we have 𝑇 𝑔𝑘+1 𝑑𝑘+1 ≤ −
2
(𝑠𝑘𝑇 𝑔𝑘+1 )
𝑠𝑘𝑇 𝑦𝑘 +𝑞𝑘
≤ 0,
(26)
Therefore, the search direction 𝑑𝑘+1 obtained from (23), is a descent direction. Li and Fukushima modified BFGS method [21] which is defined as follow 𝐵𝑘+1 𝑠𝑘 = 𝑦̅𝑘 , 𝑦̅𝑘 = 𝑦𝑘 + ℎ𝑘 ‖𝑔𝑘 ‖𝑟 𝑠𝑘 ,
(27)
Where, 𝑟 > 0 and ℎ𝑘 > 0 defining as follow 𝑠𝑇 𝑦𝑘 2 𝑘‖
ℎ𝑘 = 𝐶 + max {− ‖𝑠𝑘
, 0} ‖𝑔𝑘 ‖−𝑟 ,
(28)
For some 𝐶 > 0 [24].So strong wolf condition (5), we have 𝑠𝑘𝑇 𝑦𝑘 = 𝑠𝑘𝑇 𝑔𝑘+1 − 𝑠𝑘𝑇 𝑔𝑘 ≥ 𝜎𝑠𝑘𝑇 𝑔𝑘 − 𝑠𝑘𝑇 𝑔𝑘 = −(1 − 𝜎)𝑠𝑘𝑇 𝑔𝑘 > 0 ,
(29)
Therefore, from (29), rewriting (27) as follow 𝐵𝑘+1 𝑠𝑘 = 𝑦̅𝑘 , 𝑦̅𝑘 = 𝑦𝑘 + 𝐶‖𝑔𝑘 ‖𝑟 𝑠𝑘 .
(30)
In [23] 𝐵𝑘+1 (which is satisfying (30) conditions), have been assumed as an approach from Hessian and 𝜃𝑘 computing as follow
𝑇
𝜃𝑘 =
𝑦 𝑔 (𝐶‖𝑔𝑘 ‖𝑟 −1)𝑠𝑘𝑇 𝑔𝑘+1 − 𝑘𝑇 𝑘+1 𝐶‖𝑔𝑘 ‖𝑟 ‖𝑠𝑘 ‖2 𝑠𝑘 𝑦𝑘 𝑇 𝑔 𝑔 𝑔𝑘𝑇 𝑔𝑘+1 + 𝑘𝑇 𝑘+1 𝐶‖𝑔𝑘 ‖𝑟 ‖𝑠𝑘 ‖2 𝑠𝑘 𝑦𝑘
,
(31)
And, the search direction as follow 𝑑𝑘+1 = −𝑔𝑘+1 +
𝑔𝑘+1 𝑇 (𝑦𝑘 +(𝐶‖𝑔𝑘 ‖𝑟 −1)𝑠𝑘𝑇 ) 𝑠𝑘 . 𝑠𝑘𝑇 𝑦𝑘 +𝐶‖𝑔𝑘 ‖𝑟 ‖𝑠𝑘 ‖2
(32)
In following Theorem3 we show the descent of (32). Theorem 3 Assume that 𝜃𝑘 computed from (31) and 𝛼𝑘 steplength obtained from strong wolf line search condition (5) then, the search direction 𝑑𝑘+1 obtained from (32) is a descent direction. Prove we have, 𝑇 𝑔𝑘+1 𝑑𝑘+1 = −‖𝑔𝑘+1 ‖2 + ‖𝑔𝑘+1 ‖2 𝑠𝑘𝑇 𝑦𝑘 +(𝐶‖𝑔𝑘 ‖𝑟 −1)‖𝑠𝑘 ‖2 𝑠𝑘𝑇 𝑦𝑘 +𝐶‖𝑔𝑘 ‖𝑟 ‖𝑠𝑘 ‖2
Because
𝑠𝑘𝑇 𝑦𝑘 +(𝐶‖𝑔𝑘 ‖𝑟 −1)‖𝑠𝑘 ‖2 𝑠𝑘𝑇 𝑦𝑘 +𝐶‖𝑔𝑘 ‖𝑟 ‖𝑠𝑘 ‖2
,
(33)
𝑇 ≤ 1, so𝑔𝑘+1 𝑑𝑘+1 ≤ 0. Therefore, the search direction 𝑑𝑘+1 obtained from
(32) is a descent direction.
2 The line search procedures In conjugate gradient algorithm, 𝛼𝑘 steplength maybe have unexpected condition and influenced by problem scaling, that could be larger or smaller from one [25]. Numerical camper between conjugate gradient method and less memory Quasi-Newton method (proposed by Liu and Nocedal [26]), show that, because it accepts one as steplength in more it iterations, the second method is more efficient [9]. In the algorithm procedure choosing initial steplength (𝛼𝑘0 ), has great effect on decrease or increase the number of iterations. we study the line search procedure which proposed in [11] and We show that, if we using the Andrie’s line search procedure [11] in the methods which proposed in [23] then we have better results. ‖𝑑𝑘 ‖
0∗ For effectiveness of 𝛼𝑘 initial steplength, Andrei proposed this choice with first guess (𝛼𝑘+1 = 𝛼𝑘 ‖𝑑
)
𝑘+1 ‖
in iterations. This steplength using in strong wolf line search for obtaining suitable steplength. To enhance the performance of the algorithm (efficiently decrease in the function value in iterations), used follow method for estimation a new point 𝑥𝑘+1 = 𝑥𝑘 + 𝜆𝑘 𝛼𝑘 𝑑𝑘 , 𝜆𝑘 = −
𝑎𝑘 , 𝑏𝑘
(34)
Which 𝑎𝑘 = 𝛼𝑘 𝑔𝑘𝑇 𝑑𝑘 , 𝑏𝑘 = −𝛼𝑘 (𝑔𝑘 − 𝑔𝑧 ) and 𝑔𝑧 = ∇𝑓(𝑥𝑘 + 𝛼𝑘 𝑑𝑘 ).If 𝑏𝑘 ≠ 0, then by using (34) the new point is updating, otherwise we set 𝑥𝑘+1 = 𝑥𝑘 + 𝛼𝑘 𝑑𝑘 . 3 Numerical results
We used MATLAB 7.10.0.499 (R2010a) for obtaining numerical results. Test problems contain 57 problem with different domain in [2, 10000] which selected from problem of CUTEr collection [27, 28]. Comparative conjugate gradient hybrid methods contain follow methods: HA and HB methods proposed by Babaie, Fatemi and Mahdavi-Amiri in [23] (below) 1. Hybrid CG method HA:𝜃𝑘 is computed by (22). 2. Hybrid CG method HB:𝜃𝑘 is computed by (31). In HA+ and HB+ methods, we using Andrei’s line search procedure (34) in the methods proposed in [23], also using Powell’s restart criterion [29] and when 𝑔𝑘𝑇 𝑑𝑘 > −10−4 ‖𝑔𝑘 ‖‖𝑑𝑘 ‖we restart the search direction 𝑑𝑘 = −
𝑠𝑘𝑇 𝑠𝑘 𝑔 𝑠𝑘𝑇 𝑦𝑘 𝑘
[23], as result we always have descent direction. We used the strong wolf line
search condition (5) with 𝛿 = 0.01and𝜎 = 0.1, and used 𝐶 = 10−3 and 𝑟 = 0.1for‖𝑔𝑘 ‖ ≥ 0.01, Otherwise we set 𝑟 = 3, in HB (and HB+) algorithm. The stopping criterions are achieving a solution with ‖𝑔𝑘 ‖ < 10−6 or reaching maximum of 10000 iterations. We compare the algorithm performance based on efficient assessment criterion function 𝑒𝑓𝑒 =
𝑛𝑓 𝑛
+ 𝑛𝑔 ,
(35)
Where nf is the number of function evaluations, n is problem domain and ng is the number of function’s gradient evaluations. Based on Dolan and Moré [30] efficient assessment criterion function performance profile, we compare the performance of HA+, HB+ algorithms with the HA, HB CG algorithms (Fig 1). All the four algorithms successfully solved 50 out of 57 problems, and failures occurred for all the algorithms on a set of 7 test problems including ARGLINA, EDENSH, HELIX, NONDQUAR, INDEF, OSBORNEB, SCOSINE. Also failures occurred for HA, HB algorithm in BDQRTIC and DQRTIC.
Fig. 1Efficient assessment Criterion function performance Profiles for the four algorithms
1 0.9 0.8 0.7
𝝆(𝝉)
0.6 0.5 0.4 HA HB HA+ HB+
0.3 0.2
1
2
3
4
5
𝝉
6
7
8
9
10
5 conclusions We show that Babaie’s group search directions are descent and by using Powell’s restart criterion and Andrei’s line search procedure, we try to improve HA, HB algorithms. Reference 1. Y.H. Dai, J.Y. Han, G.H. Liu, D.F. Sun, H.X. Yin and Y.X. Yuan, Convergence properties of nonlinear conjugate gradient methods, SIAM J. Optim. 10(2) (1999) 348–358. 2. M.R. Hestenes and E.L. Stiefel, Methods of conjugate gradients for solving linear systems, J. Res.Natl. Bur. Stand. 49(5) (1952) 409–436. 3. R. Fletcher and C.M. Revees, Function minimization by conjugate gradients, Comput. J. 7 (1964) 149–154. 4. A. Al-Baali, Descent property and global convergence of the Fletcher–Reeves method with inexact line search, IMA J. Numer. Anal. 5 (1985) 121–124. 5. G. Zoutendijk, Nonlinear programming computational methods, In: Abadie, J. (ed.) Integer and Nonlinear Programming, North-Holland, Amsterdam (1970) 37–86. 6. D. Touati-Ahmed and C. Storey, Efficient hybrid conjugate gradient techniques, J. Optim. Theory Appl. 64 (1990) 379–397. 7. E. Polak and G. Ribière, Note sur la convergence de méthodes directions conjugées, Rev. FrancaiseêInform. Rech. Opér. 16 (1969) 35–43. 8. J.C. Gilbert and J. Nocedal, Global convergence properties of conjugate gradient methods for optimization, SIAM J. Optim. 2 (1992) 21–42. 9. N. Andrei, Numerical comparison of conjugate gradient algorithms for unconstrained optimization, Stud. Inform. Control 16 (2007) 333–352. 10. Y.H. Dai and Y.X. Yuan, An efficient hybrid conjugate gradient method for unconstrained optimization, Ann. Oper. Res. 103 (2001) 33–47.
11. N. Andrei, Accelerated hybrid conjugate gradient algorithm with modified secant condition for unconstrained optimization, Numer. Algorithms 54(1) (2010) 23–46. 12. N. Andrei, Another hybrid conjugate gradient algorithm for unconstrained optimization, Numer. Algorithms 47 (2008) 143–156. 13. Y.H. Dai, New properties of a nonlinear conjugate gradient method, Numer. Math. 89 (2001) 83–98. 14. G. Li, C. Tang and Z. Wei, New conjugacy condition and related new conjugate gradient methods for unconstrained optimization, J. Comput. Appl. Math. 202(2) (2007) 523–539. 15. J. Nocedal and S.J. Wright, Numerical Optimization, 2nd ed. Springer (2006). 16. L.C.W. Dixon, Variable metric algorithms: necessary and sufficient conditions for identical behavior on nonquadratic functions, J. Optim. Theory Appl. 10 (1972) 34–40. 17. M.J.D. Powell, On the convergence of the variable metric algorithm, J. Inst. Math. Appl. 7 (1971) 21–36. 18. W.F. Mascarenhas, The BFGS method with exact line searches fails for non-convex objective functions, Math. Program. 99(2004) 49–61. 19. J.G.Liu, Q. Guo, Global convergence properties of the modified BFGS method, J. Appl. Math. Comput. 16 (2004) 195–205. 20. J.G. Liu, Z.Q. Xia, R.D. Ge, Q. Guo, An modified BFGS method for non-convex minimization problems, Oper. Res. Manag. 13 (2004) 62–65. 21. D.H. Li and M. Fukushima, A modified BFGS method and its global convergence in nonconvex minimization, J. Comput. Appl. Math. 129 (2001) 15–35. 22. Y.X. Yuan, A modified BFGS algorithm for unconstrained optimization. IMA J. Numer. Anal. 11 (1991) 325–332. 23. S. Babaie-kafaki, M. Fatemi and N. Mahdavi-Amiri, Two effective hybrid conjugate gradient method algorithms based on modified BFGS updates, Numer. Algorithms 58 (2011) 315-331. 24. W. Zhou and L. Zhang, A nonlinear conjugate gradient method based on the MBFGS secant condition, Optim. Methods Softw. 21(5) (2006) 707–714. 25. J. Nocedal, Conjugate gradient methods and nonlinear optimization, In: Adams, L., Nazareth, J.L. (eds.) Linear and Nonlinear Conjugate Gradient—Related Methods, SIAM, Philadelphia (1996). 26. D.C. Liu, J. Nocedal, On the limited memory BFGS method for large scale optimization, Math. Program. 45 (1989) 503–528. 27. N.I.M. Gould, D. Orban and Ph.L. Toint, CUTEr, a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. 29 (2003) 373–394. 28. Y. Xiao, Q. Wang and D. Wang, Notes on the Dai-Yuan-Yuan modified spectral gradient method, J. Comput. Appl. Math. 234(10) (2010) 2986–2992. 29. M.J.D. Powell, Restart procedures for the conjugate gradient method. Math. Program. 12, (1977) 241–254. 30. E.D. Dolan and J.J. Moré, Benchmarking optimization software with performance profiles, Math. Program. 91 (2002) 201–213.
