A Modified Super-linear QN-Algorithm for Unconstrained Optimization

Iraqi Journal of Statistical Science (18) 2010 p.p. [1-34]

A Modified Super-linear QN-Algorithm for Unconstrained Optimization Abbas Y. AL-Bayati*

Ivan Subhi Latif**

ABSTRACT In this paper, we have proposed a modified QN-algorithm for solving a self-scaling large scale unconstrained optimization problems based on a new QN-update. The performance of the proposed algorithm is better than that used by Wei, Li, Yuan algorithm. Our numerical tests show that the new proposed algorithm seems to converge faster as compared with a standard similar algorithm in many situations . ‫ﺨﻭﺍﺭﺯﻤﻴﺔ ﻤﻁﻭﺭﺓ ﻷﺸﺒـﺎﻩ ﻨﻴﻭﺘﻥ ﻓﻭﻕ ﺍﻟﺨﻁﻴﺔ ﻓﻲ ﺍﻻﻤﺜﻠﻴﺔ ﻏﻴﺭ ﺍﻟﻤﻘﻴﺩﺓ‬ ‫ﺍﻟﻤﻠﺨﺹ‬ ‫ﻓﻲ ﻫﺫﺍ ﺍﻟﺒﺤـﺙ ﺘﻡ ﺍﺴﺘﺤﺩﺍﺙ ﺨﻭﺍﺭﺯﻤـﻴﺔ ﻤﻁﻭﺭﺓ ﺇﻟﻰ ﺃﺸـﺒﺎﻩ ﻨﻴﻭﺘﻥ ﻓﻭﻕ‬ ‫ ﺍﻟﻔـﻜﺭﺓ ﺍﻷﺴـﺎﺴـــﻴﺔ ﺘﻌﺘـﻤﺩ‬.‫ﺍﻟﺨﻁـﻴﺔ ﻓـﻲ ﻤﺠـﺎل ﺍﻷﻤﺜـﻠﻴﺔ ﻏـﻴﺭ ﺍﻟﻤـﻘﻴﺩﺓ‬ ‫ ﺍﻟﺨﻭﺍﺭﺯﻤـﻴﺔ ﺍﻟﺠـﺩﻴﺩﺓ ﺫﺍﺕ ﻜﻔـﺎﺀﺓ‬. ‫ﻋﻠﻰ ﺃﺤـﺩ ﻨﻤـﺎﺫﺝ ﺘﺤـﻭﻴل ﺃﺸﺒﺎﻩ ﻨﻴـﻭﺘﻥ‬ ‫ ﻭﺍﻟﻨﺘـﺎﺌﺞ ﺍﻟـﻌﻤﻠﻴﺔ‬Wei, Li, Yuan. ‫ﺍﻋﻠﻰ ﻤـﻥ ﺍﻟﺨـﻭﺍﺭﺯﻤـﻴﺔ ﺍﻟﺘﻲ ﺃﺴﺘـﺨﺩﻤﻬﺎ‬

*

Prof. Dr/ Dept. of Mathematics/College of Computers Sciences and Mathematics/ University

of Mosul.

E-mail :[email protected]

**

Dept. of Mathematics, College of Scientific Education, University of Salahaddin, Erbil-Iraq.

E-mail :[email protected]

Received:28/8 /2006 ____________________Accepted: 17 /12 / 2006

[2]

____________ A Modified Super-linear QN-Algorithm …

‫ﺘﻭﻀـﺢ ﺒﺄﻥ ﺍﻟﻨﻤـﻭﺫﺝ ﺍﻟﺠﺩﻴﺩ ﺍﻟﻤـﺴﺘﺨﺩﻡ ﺫﺍﺕ ﻜﻔـﺎﺀﺓ ﻋﺎﻟـﻴﺔ ﺘﻘـﺎﺭﺏ ﺃﻓﻀـل‬ .‫ﻤﻘـﺎﺭﻨﺔ ﺒﺎﻟﺨﻭﺍﺭﺯﻤﻴـﺎﺕ ﺍﻟﻤﻤﺎﺜﻠﺔ ﻓﻲ ﻤﺠﺎﻻﺕ ﻋـﺩﺓ‬ KEYWORDS: Unconstrained Optimization, Quasi-Newton updates, Superliner convergence.

1. Introduction This paper analyzes the convergence properties of self-scaling QN-methods for solving the unconstrained optimization problem Min f ( x)

x ∈ ℜn ,

(1)

where f is twice continuously differentiable function. The convergence of QN-methods for unconstrained Optimization has been the subject of much analysis. The Broyden-Fletcher-GoldfarbShanno (BFGS) method is generally considered to be the most effective among other variable metric methods for unconstrained Optimization problem. One interesting property of BFGS method is its self correcting mechanism (A detailed explanation for example, in Nocedal (see [11]) with this self correcting property, Powell see[13] shows that the BFGS method with an inexact line search satisfies Wolfe conditions is globally super-linearly convergent for convex problem, and Byrd, Nocedal and Yuan (see [5]) extend Powell's analysis to the restricted Broyden class excluding the DFP method. AL-Bayati's (see [1]) presented a new self-scaling variablemetric algorithm which was based on a known two-parameter family of rank-two updating formulae. The best of these algorithm are also modified to employ inexact line searches with marginal effect thus Wei, Li and Qi (see [15]) have proposed some modified BFGS that the average performance of their algorithm was better than standard BFGS algorithm. Wei, et al. (see [14]) proved the super-linear convergence of Wei,

Iraqi Journal of Statistical Science (18) 2010 ___________

Li and Qi (see [15]) algorithm under some suitable conditions. In this paper, a new modified QN-algorithm is proposed. The Basic idea is based on the new QN-equation Vk = H k y k∗ where y k∗ is the sum of y k and AkVk and Ak is some matrix. This paper is organized as follows in the next section; we represent some basic properties of the modified BFGS algorithm. In section3 we prove the super-linear convergence for the modified QN-algorithm under some reasonable conditions. The search direction in a VM-method is the solution of the system of equations (2)

d k = −H k g k

where the matrix H k is an approximate to Gk−1 the new approximation H k +1 is chosen to take account of this new curvature information which is done by satisfying the condition H k +1 y k = ζ kVk

(called QN-like condition)

(3)

where ζ k is a scalar, generally for the QN-methods ζ k = 1 and hence equation (3) reduces to H k +1 y k = Vk

(called the QN-condition)

(4)

since information has been gained about f only in one dimension (along d k ), H k +1 is allowed to differ from H k by a correction matrix C k of at most rank two, i.e. H k +1 = H k + C k

(5)

the matrix C k is therefore the update to H k there are an infinite number of possible rank-two updates which satisfy the QNcondition but our main interest is in updates which form the Broyden one-parametric class (see [3]). The matrix H k +1 is defined by:

[3]

____________ A Modified Super-linear QN-Algorithm …

[4]

H k +1 = H k −

H k y k y kΤ H k VkVkΤ Τ W W + θ + k k k y kΤ H k y k VkΤ y k

(6)

with H y ⎤ 1 ⎡ V Wk = ( y kΤ H k y k ) 2 ⎢ Τ k − Τ k k ⎥ ⎣Vk y k y k H k y k ⎦

(7)

where θ k is a scalar chosen such that θ k ∈ [0,1] different choice of θ k then defined different updates. the Davidon-FletcherPowell(DFP)update (see [7]) is defined as equation(4) with θ k = 0 where the Broyden-Fletcher-Goldfarb-Shanno(BFGS) update corresponds to θ k = 1 (see[6] and [9]). Oren (see[13]) found that a proper scaling of the objective function improve the performance of algorithms that use Broyden family of updated. Hence Oren'sُ family of self-scaling VM-updates can be expressed as: ⎡ ⎤ H y yΤH V VΤ H k +1 = ⎢ H k − k Τ k k k + θ kWkWkΤ ⎥η k + kΤ k yk H k yk Vk y k ⎣ ⎦

(8)

where ηk =

y kΤVk y kΤ H k y k

(9)

This choice for the scalar parameter η k was made primarily because in this case η k requires the quotient of two quantities which are already computed in the updating formula. Al-Bayati (see [2]) found another interesting family of VM-updates by further scaling of Oren'sُ family of updates with a scalar δk =

1

ηk

So that the updating formulas becomes

(10)

Iraqi Journal of Statistical Science (18) 2010 ___________

H k +1 = H k −

H k y k y kΤ H k VkVkΤ Τ W W + + σ ( ) k k k y kΤ H k y k VkΤ y k

[5]

(11)

For more details see[8]

2. Modified BFGS Algorithm: Wei, Li and Qi proposed a new QN-equation (see[15]) (12)

Bk +1Vk = y k∗

where Bk = H k−1 where yk∗ = yk + AkVk and Ak is some matrix. By using equation(12)they gave BFGS type updates Bk +1 = Bk −

BkVkVkΤ Bk y k∗ y k∗Τ + Τ ∗ VkΤ BkVk Vk y k

where y = y k + AkVk and Ak = ∗ k

(13) 2( f ( xk ) − f ( xk +1 )) + ( g ( xk +1 ) − g ( xk )) Τ Vk

2

I using

equation(13) and the following Wolf Powell step-size rule

f ( xk + α k d k ) ≤ f ( xk ) + δα k g ( xk ) Τ d k

(14)

where δ ∈ (0, 1 2 ) and σ ∈ (δ ,1) and g ( xk + α k d k ) Τ d k ≥ σg ( xk ) Τ d k

(15)

2.1. Outline of the Modified BFGS Algorithm (MBFGS): Corresponding (MBFGS) the outliers of MBFGS algorithms may be listed Step1: choose an initial point x0 ∈ ℜ n and an initial positive definite matrix B1 set K = 1 .

____________ A Modified Super-linear QN-Algorithm …

[6]

Step2: if g k = 0 then stop! Go to step2. Step3: solve H k d k + g k = 0 to obtain a search direction d k Step4: find d k by Wolf-Powell step-size rule f ( xk + α k d k ) ≤ f ( xk ) + δα k g ( xk ) Τ d k and g ( xk + α k d k ) Τ d k ≥ σg ( xk ) Τ d k where

δ ∈ (0, 1 2 ) and σ ∈ (δ ,1) and

Step5: set xk +1 = xk + α k d k .calculate Ak and update Bk +1 by formula Bk +1 = Bk −

BkVkVkΤ Bk y k∗ y k∗Τ + Τ ∗ VkΤ BkVk Vk y k

where y k∗ = y k + AkVk and Ak =

2( f ( xk ) − f ( xk +1 )) + ( g ( xk +1 ) − g ( xk )) Τ Vk

2

I

Step6:set k = k + 1 , go to step1 2.2. Some Properties of the MBFGS Algorithm: The global convergence of the MBFGS algorithm needs the following three assumptions Assumption2.2.1:The level set Ω = {x f ( x) ≤ f ( x0 )} is contained in a bounded convex set D

Assumption2.2.2:The function f is continuously differentiable on D and there exists constant L ≥ 0 such that g ( x) − g ( y ) ≤ L x − y ,for all x, y ∈ D . Assumption2.2.3: The function f is uniformly convex that is there are positive constants m1 and m2 such that 2

m1 z ≤ z Τ G ( x) z ≤ m2 z

2

(16)

Iraqi Journal of Statistical Science (18) 2010 ___________

[7]

for all x, z ∈ ℜ n ,where G denotes the Hessian matrix of f . Theorem2.1: Let {xk } be generated by MBFGS algorithm then we have (see[11]) lim inf g k = 0 .

(17)

k →0

The super-linear convergence analysis of the MBFGS algorithm needs the following assumptions: Assumption2.2.4: xk → x ∗ at which g ( x ∗ ) = 0 and G ( x ∗ ) is positive definite. Assumption2.2.5: G is holder continuous at x ∗ that is, there exists constant r ∈ (0,1) and M > 0 such that G ( x) − G ( x ∗ ) ≤ M x − x ∗

r

(18)

for all x in neighborhood of x ∗ since { f ( xk )} is a decreasing sequence also the sequence {xk } generated by MBFGS is contained in Ω and there exists a constant f ∗ such that lim f ( xk ) = f ∗ k →∞

(19)

3. A new Modified QN- Algorithm: In this section we propose a new QN-method based on the following QN-condition Vk = H k y k∗

(20)

where y k∗ = y k + AkVk and Ak is some matrix defined by Ak =

y kΤVk I y kΤ H k y k

using equation(20)and taking H k +1 as Al-Bayati update (see[1]).

____________ A Modified Super-linear QN-Algorithm …

[8]

H k +1 = H k −

H k y k y kΤ H k VkVkΤ Τ W W + + σ ( ) k k k y kΤ H k y k VkΤ y k

where δk =

y kΤ H k y k H y ⎤ 1 ⎡ V , Wk = ( y kΤ H k y k ) 2 ⎢ Τ k − Τ k k ⎥ Τ y k Vk ⎣ Vk y k y k H k y k ⎦

and using also the following Armijo condition f k − f k +1 ≥ c1 VkΤ y k , VkΤ y k ≥ (1 − c2 )(VkΤ y k )

(21)

where c1 ∈ (0, 1 2 ) , c 2 ∈ (c1 ,1) in (MBFGS) yields a new QN-algorithm given as below: 3.1. Outline of the Modified QN- Algorithm (NEW): The outliers of the new algorithm may be given as: Step1: choose an initial point x0 ∈ ℜ n and use-update positive definite matrix H 1 set K = 1 . Step2: if g k = 0 then stop! Go to step2. Step3: solve H k d k + g k = 0 to obtain a search direction d k Step4: find d k by Armijo line search step-size rule f k − f k +1 ≥ c1 VkΤ y k , VkΤ y k ≥ (1 − c2 )(VkΤ y k )

where c1 ∈(0, 12) , c2 ∈(c1,1) Step5: set xk +1 = xk + α k d k .calculate a new Ak and update H k +1 by the following formula

Iraqi Journal of Statistical Science (18) 2010 ___________

H k +1 = H k −

H k y k∗ y k∗Τ H k VkVkΤ Τ + + σ ( ) W W k k k y k∗Τ H k y k∗ VkΤ y k∗

[9]

(22)

where δk =

y k∗Τ H k y k∗ H k y k∗ ⎤ ∗Τ ∗ 1 2 ⎡ Vk , = − ( ) W y H y k k k k ⎢ Τ ∗ ∗Τ ∗⎥ y k∗ΤVk ⎣Vk y k y k H k y k ⎦

and yk∗ = y k + AkVk and Ak =

y kΤVk I y kΤ H k y k

(newly defined) (23)

Step6:set k = k + 1 , go to step1. 3.2. Some Theoretical Properties of the New Algorithm: To show the global and super-linear convergence of the new algorithm use the same assumption 2.2.1,2.2.2,2.2.3 and 2.2.4 where used for all x in neighborhood of x ∗ since { f ( xk )}is a decreasing sequence also the sequence { xk }generated by new algorithm is contained in Ω and that there exists a constant f ∗ such that lim f ( xk ) = f ∗ k →∞

(24)

lemma3.2.1: let (α k , xk +1 , g k +1 , d k +1 ) be generated by the new algorithm then H k−+11 is positive definite for all k provided that VkΤ yk∗ > 0 . Proof:The new algorithm has the following QN-condition H k +1Vk = y k

(25)

and preserve positive definiteness of the matrices { H k } if α k chosen to satisfy the Armijo condition

is

____________ A Modified Super-linear QN-Algorithm …

[10]

f k − f k +1 ≥ c1 VkΤ y k ,

VkΤ y k ≥ (1 − c2 )(VkΤ y k )

where f k denoted f ( xk ) , c1 ∈(0, 12) , c2 ∈(c1,1) .Note that the second condition in (21) guarantees that VkΤ y k∗ > 0 whenever g k ≠ 0 . lemma3.2.2: let { xk } be generated by the new algorithm then we have 2

m1 Vk

≤ VkΤ y k∗ ≤ m2 Vk

2

, k = 1,2,...

(26)

and y k∗ ≤ (2 L + m2 ) Vk , k = 1,2,...

Proof:Using assumption2.2.2 and equation (16) 2

2

m1 z ≤ z Τ G ( x) z ≤ m2 z

and tailors formula we have y k∗ΤVk = ( g k +1 − g k ) Τ Vk = VkΤ G (ζ 1 )Vk

where ζ 1 ∈ ( xk , xk +1 ) thus (26) holds m1 Vk

2

2

≤ VkΤ y k∗ ≤ m2 Vk .

To prove (27) using the equation (26) VkΤ y k∗ ≤ m2 Vk

therefore

2

(27)

Iraqi Journal of Statistical Science (18) 2010 ___________

y k∗ ≤ m2 Vk

2

[11]

(28)

also use assumption 2.2.2 g ( x) − g ( y ) ≤ L x − y ,for all x, y ∈ D .

Thus y k∗ ≤ L Vk

(29)

from equation (28),(29) we get y k∗ ≤ (2 L + m2 ) Vk .

Theorem3.1 let { xk } be generated by the new algorithm then xk tends to x super-linearly.(see[2])

lemma3.2.3: suppose that (α k , xk +1 , g k +1 , d k +1 ) be generated by the new algorithm and that G is continuous at x ∗ then we have lim A = 0 k →∞

Proof:By using Taylor'sُ formula, we have y k∗ΤVk = ( g k +1 − g k ) Τ Vk

= VkΤ G (ζ 1 )Vk

and

(30)

____________ A Modified Super-linear QN-Algorithm …

[12]

f k − f k +1 = ( g k +1 − g k ) Τ Vk

= VkΤ G (ζ 2 )Vk

and f k − f k +1 = − g kΤ+1Vk + 12 VkΤ G (ζ 2 )Vk

where ζ 1 = xk + θ1k ( xk +1 − xk ) ζ 2 = xk + θ 2 k ( xk +1 − xk )

and θ1k ,θ 2 k ∈ (0,1) .from the definition of A and lemma3.2.1 we get A=

VkΤ H k−+11Vk − VkΤ G (ζ 1 )Vk y kΤ H k y k

and VkΤ H k−+11Vk = VkΤ G (ζ 1 )Vk

hence Ak ≤ G (ζ 1 ) − G (ζ 2 ) Therefore (30) holds.

lemma3.2.4: let{ xk } be generated by the new algorithm denoted Q = G ( x ∗ ) then there are positive constants bi , i = 1,2,3,4 and η ∈ (0,1) such that for all large k −1

2

H k−+11G ( x ∗ ) −1 ≤

( 1 − ρW

2 k

)

+ b1τ k + b2 A H k − G ( x ∗ ) −1 + b3τ k + b4 Ak (31)

where A = Q Τ AQ

as

F

, . F is the forbenius norm of a matrix and Wk is defined

Iraqi Journal of Statistical Science (18) 2010 ___________

Wk =

Q −1 ( H k − G ( x ∗ ) −1 y k∗

[13]

(32)

H k − G ( x ∗ ) −1 Qy k∗

In particular H k , H k−1 are bounded Proof:To prove (32) H k +1 = H k +

(Vk − H k y k∗ )VkΤ + Vk (Vk − H k y k∗ ) Τ ( y k∗Τ )(Vk − H k y k∗ )VkVkΤ ) + y k∗ΤVk ( y k∗ΤVk ) 2

⎛ V y ∗Τ ⎞ ⎛ y ∗V Τ = ⎜⎜ I − k∗Τ k ⎟⎟ H k ⎜⎜ I − ∗kΤ k y k Vk ⎠ ⎝ y k Vk ⎝

⎞ VkVkΤ ⎟⎟ + ∗Τ ⎠ y k Vk

It is the dual form of the DFP type algorithm in the sense that H k +1 → H k−+11 and Vk → y k we also have Qy k∗ − Q −1Vk ≤ Q . y k∗ − Q −2Vk

[

= Q . y k∗ − G ( x ∗ )Vk

]

⎞ ⎛ 1 ≤ Q ⎜ ∫ G ( xk + ηVk )Vk dη − G ( x ∗ )Vk + AVk ⎟ ⎟ ⎜ ⎠ ⎝ 0

⎛ 1 ⎞ ≤ Q . Vk ⎜ ∫ G ( xk + ηVk )Vk − G ( x ∗ ) dη + A ⎟ ⎜ ⎟ ⎠ ⎝ 0 1 ⎛ ⎞ ⎜ ≤ Q . Vk ⎜ M 2 ∫ G ( xk + ηVk )Vk − G ( x ∗ )dη + A ⎟⎟ 0 ⎝ ⎠ 1 ⎛ ⎞ r ⎜ ≤ Q . Vk ⎜ M 2 ∫ ( xk − x ∗ + ηVk )dη + A ⎟⎟ 0 ⎝ ⎠

____________ A Modified Super-linear QN-Algorithm …

[14]

(

)

1 ⎛ ⎞ r ⎜ ≤ Q . Vk ⎜ M 2 ∫ η xk − x ∗ + (1 − +η ) xk − x ∗ dη + A ⎟⎟ 0 ⎝ ⎠

≤ Q . Vk (M 2τ k + A )

since τ k → 0 and A → 0 it is clear that when k is large enough Qyk∗ − Q −1Vk ≤ β QVk ,

for some constant β ∈ (0, 1 2 ) ,therefore from lemma3.2.1 (with , y → Vk , H −1 → H k , A → G ( x ∗ ) −1 and identification V → yk M → Q −1 there are constants p ∈ (0,1) and b5 , b6 > 0 such that ∗ −1

H k +1 − G ( x )

⎛ Q −1Vk − Qy k∗ 2 ⎜ ≤ 1 − ρWk + b5 ⎜ Qy k∗ ⎝

∗ −1 ∗ ⎞ ⎟ H − G ( x ∗ ) −1 + b Vk − G ( x ) y k 6 ⎟ k Qy k∗ ⎠

where Wk is defined as more over ,there exists a constant b7 such that for all k large enough y k∗ = Qg k +1 − g k + AVk ≥ Qg k +1 − g k − A QVk ≥ b7 xk +1 − xk − A Q Vk = (b7 − A Q ) Vk

using A → 0 the above inequality implies that there is a constant C such that when k is sufficiently large Qy k∗ ≥ C Vk

so we may obtain that

Iraqi Journal of Statistical Science (18) 2010 ___________

Qy k∗ − Q −1Vk Qy

∗ k

≤ C −1 Q ( M 2τ k + A )

[15]

(33)

and Vk − G ( x ∗ ) −1 y k∗ Qy k∗ =

≤

=

Vk − Q 2 y k∗ Qy k∗

Q(Qy k∗ − Q −1Vk Qy k∗

Q Qy k∗ − Q ∗Vk C Vk 2

≤ C −1 Q ( M 2τ k + A )

from which and (33),we get (31) lemma3.2.5: Let { xk }be generated by the new algorithm then the following Dennis More condition holds for the new technique lim

H k−1 − G ( x ∗ )Vk

k →∞

Vk

(34)

=0

Proof:Using τ k → 0 , inequality

A → 0 and

H k is bounded and following

1 − τ ≤ 1 − 1 2 τ , ∀τ ∈ (0,1)

we can deduce that there are positive constants M 1 and M 2 such that for all large k ] H k − G ( x ∗ ) −1 ≤ (1 − 12 ρWk2 ) H k − G ( x ∗ ) −1 + M 1τ k + M 2 Ak

[16]

____________ A Modified Super-linear QN-Algorithm …

that is 1 2

ρWk2 H k − G ( x ∗ ) −1 ≤ H k − G ( x ∗ ) −1 − H k +1 − G ( x ∗ ) −1 + M 1τ k + M 2 Ak

summing the above inequality over k ,we get ∞

1 2

ρ ∑Wk2 H k − G ( x ∗ ) −1 < +∞ k = k0

where k 0 is sufficiently large index such that (31) holds for all k ≥ k 0 .In particular, we have lim Wk2 H k − G ( x ∗ ) −1 = 0 k →∞

that is lim

k →∞

Q −1 ( H k − G ( x ∗ ) −1 y k∗ Qy k∗

=0

(35)

Moreover we have Q −1 ( H k − G ( x ∗ ) −1 y k∗ = Q −1 ( H k (G ( x ∗ ) − H k−1 ) G ( x ∗ ) −1 y k∗

≥ Q −1 ( H k (G ( x ∗ ) − H k−1 )Vk − Q −1 ( H k (G ( x ∗ ) − H k−1 )(Vk − G ( x ∗ )) −1 y k∗

≥ Q −1 ( H k (G ( x ∗ ) − H k−1 )Vk − Q −1 ( H k (G ( x ∗ ) − H k−1 )(Vk − G ( x ∗ )) −1 y k∗ − A Q −1 ( H k (G ( x ∗ ) − H k−1 )G ( x ∗ ) −1Vk

using the fact that H k and H k +1 are bounded and that G ( x) is continuous , we have

Iraqi Journal of Statistical Science (18) 2010 ___________

[17]

Q −1 ( H k (G ( x ∗ ) − H k−1 )Vk − G ( x ∗ ) −1 y k∗ = Q −1 ( H k (G ( x ∗ ) − H k−1 )G ( x ∗ ) −1 (G ( x ∗ )Vk − y k∗

= Q −1 ( H k (G ( x ∗ ) − H k−1 )G ( x ∗ ) −1[(G ( x ∗ ) − G ( xk ))Vk + (G ( xk )Vk − y k )] ≤ Q −1 ( H k (G ( x ∗ ) − H k−1 )G ( x ∗ ) −1 (G ( x ∗ ) − G ( xk ) Vk G ( xk )Vk − y k = O Vk

and A Q −1 ( H k (G ( x ∗ ) − H k−1 )Vk

≤ A Q −1 ( H k (G ( x ∗ ) − H k−1 )G ( x ∗ ) −1 Vk = O ( Vk )

(36)

therefore ,there exists a positive constant k > 0 such that Q −1 ( H k − G ( x ∗ ) −1 y k∗ ≥ K G ( x ∗ ) − H k−1 Vk − O( Vk )

on the other hand, from equation (36)and lemma 3.2 we have Qy k∗ ≤ Q y k∗ ≤ (2 L + m2 ) Q Vk

from the above inequality (35) and (36) we conclude that the Dennis-More condition holds. □ Theorem3.2 Let { xk } be a sequence generated by the new algorithm then the xk tends to x super-linearly. Proof:We will verify that α k = 1 for large k .since the sequence H k is bounded we have

____________ A Modified Super-linear QN-Algorithm …

[18]

dk = H k gk ≤ H k gk → 0

by Taylor's expansion, we get f k +1 − f k − δg kΤ d k = (1 − δ ) g kΤ d k + 12 d kΤ G ( xk + θd k )d k = −(1 − δ )d kΤ H −1d k + 12 d kΤ G ( xk + θd k )d k

= −( 12 − δ )d kΤ H −1d k − 12 d kΤ ( H k−1 − G ( xk + θd k ))d k 2

= −( 12 − δ )d kΤ G ( x ∗ )d k + O( d k )

where θ ∈ (0,1) and the last equality follows the Dennis More condition(34)thus f k +1 − f k − δg kΤ d k ≤ 0

for all large k .In other words α k = 1 the firs inequality of the Armijo equation(21) for all k sufficiently large on the other hand g kΤ d k − δg kΤ d k = ( g k +1 − g k ) Τ d k + (1 − σ ) g kΤ d k = d kΤ G ( xk + θd k )d k − (1 − δ ) g kΤ H k−1d k 2

= d kΤ G ( xk + θd k )d k − (1 − δ )d kΤ G ( xk )d k + O ( d k ) 2

= δd kΤ G ( xk )d k + O( d k )

where θ ∈ (0,1) so we have g kΤ+1d k ≥ δg kΤ d k

which means that α k = 1 satisfies the Armijo equation(21) for all sufficiently large .Therefore we assert that α k = 1 for large k . Consequently, we can deduce that xk converges super-linearly.

Iraqi Journal of Statistical Science (18) 2010 ___________

[19]

4. Numerical Results: In this section, we compare the numerical behavior of the new algorithm with the MBFGS algorithm for different dimensions of test functions. Comparative tests were performed with (41)(specified in the Appendices 1 and 2) well-known test functions (see [10]). All the results are show in Table (1), (2) while Table (3) give the percentage of NOI and NOF. All the results are obtained with newly-programmed FORTRAN routines which employ double precision. The comparative performances of the algorithms taken in the usual way by considering both the total number of function evolutions (NOF) and the total number of iterations required to solve the problem (NOI) .In each case the convergence criterion is that the value of g k < 1 × 10 −5 the Armijo fitting by Frandsen (see [2]) and Powell line search (see [3]) used as the common linear search subprogram. Each of the function was solved using the following algorithms (1) MBFGS Algorithm : (2)The new algorithm The important thing is that the new algorithm needs less iteration, fewer evaluations of f ( x) and g ( x) than MBFGS. We can see that other algorithm may fail in some cases while the new algorithm always converges. Moreover numerical experiments also show that the new algorithm always convergence stabiles. Namely

[20]

____________ A Modified Super-linear QN-Algorithm …

there are about (60-87) % improvements of NOI for all dimensions Also there are (30 -78) % improvements of NOF for all test functions. Table (1):Comparison between the New algorithm and MBFGS algorithms using different value of 12< N