CONVERGENCE AND COMPLEXITY OF NEWTON

CONVERGENCE AND COMPLEXITY OF NEWTON ITERATION FOR OPERATOR EQUATIONS J. F. Traub Department of Computer Science Carnegie-Mellon University Pittsburgh, Pennsylvania 15213 H. Wozniakowski Department of Computer Science Carnegie-Melion University Pittsburgh, Pennsylvania 15213 (Visiting from the University of Warsaw)

March 1977 Revised December 1977

This research was supported in part by the National Science Foundation under Grant MCS75-222-55 and the Office of Naval Research under Contract N0014-76C-0370, NR 044-422,

ABSTRACT

An optimal convergence condition for Newton iteration in a Banach space is established.

We show there exist problems for which the iteration

converges but the complexity is unbounded.

Thus for actual computation

convergence is not enough. We show what stronger condition must be imposed 11

to also assure "good complexity •

1.

INTRODUCTION Numerous papers have analyzed sufficient conditions for the convergence

of algorithms for the solution of non-linear problems.

In addition to con-

vergence, we consider another fundamental question. What stronger conditions 11

must be imposed to assure "good complexity ?

This is clearly one of the

crucial issues (in addition to stability) if one is interested in actual computation. We believe it is also a most interesting theoretical question. We consider Newton iteration for a simple zero of a non-linear operator in a Banach space of finite or infinite dimension. We establish the optimal radius of the ball of convergence with respect to a certain functional. There exist problems where the iteration converges but the complexity increases logarithmically to infinity as the initial iterate approaches the boundary of the ball of convergence.

(This phenomenon does not occur in the Kantorovich

theory of operator equations; see Section 3.) We establish the optimal radius of the ball of good complexity. In this paper we limit ourselves to the important case of Newton iteration. In other papers (Traub and Wozniakowski [76b] and [77]) we study optimal convergence and complexity for classes of iterations. We summarize the results of this paper. Definitions and theorems concerning the optimal ball of convergence are given in Section 2. We conclude this Section by giving conditions under which the radius of the ball of convergence is a constant fraction of the radius of the ball of analyticity of the operator. Complexity of Newton iteration is studied in Section 3. We show that Newton iteration may converge but have arbitrarily high complexity and conjecture this is a general phenomenon. We establish the radius of the ball of good complexity as well as a lower bound on the complexity of Newton iteration.

-2-

2.

CONVERGENCE OF NEWTON ITERATION We consider the solution of the non-linear equation.

(2.1) F(x) - 0 where F: D_c

-+ B2 and B^, B2 are Banach spaces over the real or complex

fields of dimension N, N = dim(B ) = dim(B ) , 1 £ N ^ -H». We solve (2.1) ]L

2

by Newton iteration which constructs the sequence [x^3 as follows :

x

where x

i+l

Q

=

x

,

1

i " F (x )" F(x ) i

i

is an initial guess and i = 0,1,... . Suppose that F has a simple

zero a, i.e., F(a> - 0 and F (or)" exists and is bounded. f

that if F

f

1

It is well-known

is a Lipschitz operator in a neighborhood of a and x^ is suffic-

iently close to a then {x.^3 converges to a and the order of convergence is two, see, e.g., Gragg and Tapia [74], Kantorovich [48], Ortega and Rheinboldt [70] and Rail [69]. In this paper we are interested in sharp bounds on how close x^ has to be to a to get quadratic convergence of Newton iteration and how close x^ has to be to a to guarantee good complexity. We begin with a theorem which describes the character of convergence of Newton iteration. Let F > 0 and J = {x:

^ V).

From now on we assume that

a is a simple zero of F, (2.2)

J F (x) exists for all x € J, 1

Note that if F

n

is continuous for x 6 J then

-3-

m d

r

- sup

1

w

^

!

!

which shows that in this case k^ is a bound on the "normalized second derivative. 11

Let q be any number such that 0 < q < 1

Theorem 2.1 If F satisfies (2.2) and

2

3

< - >

V * l f e '

(2.4) x

Q

6 J,

then Newton iteration is well-defined and

lim x

(2.6)

e. - £ C. e . i+l 1 1

i

=

a

(2.5)

e

9

£ qe , Vi,

1 + 1

±

2

v

where C ~ k^ (1 - 2A e ) . ±

2

i

If F is continuously twice-differentiable at a then (2-7) x .

- c-|F'( )"V(c )(x -a) + 2

+ 1

a

y

od^-alf).

i

Proof The proof is by induction. Let x^ € J and let

(2.8) R(x;x ) = J^CF'Cx. + t(x-x.)) - F'(x.)}(x-x.)dt. 0

1

.

1

1

Define w » w(x;x^) as the polynomial of first degree which interpolates F(x^) 1

l

and F (x^) . Then w(x;x^ * F(x ) 4- F (x )(x-x ) and the next approximation x is the zero of w whenever F (x^) is invertible. From Rail [69, p. 124] i

1

i + 1

we get the error formula

i

i

(2.9) F(x) - w(x;x ) - R ^ X j ) ±

for x € J while due to (2.2), ,

(2.10)

1

|^ (oi)" R(x;x )|| * A |( -x |f. i

2

From the definition of

x

i

we have

| | i - F ' < a ) " V < x ) | | * 2 A | M I * 2A r S-jfg^ < i . 2

2

!

This means that F (x) is invertible for any x € J and

is well-defined.

Furthermore ||F'(xrV(a)|| * 1/(1 - 2A |(x-a|| ) , 2

1

see Rail [69, p.43]. Set x

a in (2.9).

23

We can rewrite the polynomial w as w(x;x^) - F (x^) (x-x^ ^) +

1

x

i+

-1

lh

i +

HI

=

x

Ifr' < i>

due to (2.3).

c

Then F (x^ ( j" *) ~ R(»;x > which yields

F

i

A

1

e

A

9 5

9

r

Wx.)II * i ^ a ^ : * i r ^ f ± * i

F

e

' '

q e

This proves (2.5) and (2.6).

If F (x) is continuous at ot then fl

R(of;x ) = \ F»(a)( -a) + o d ^ - a f p ) , 2

±

Xi

F'(x.) = F'(or) + Od^-crll ) which implies (2.7).

•

Remark 2.1

e

Theorem 2.1 states quadratic convergence of Newton iteration since 2 l ^i i i ^ 2 ^ ^ ~ 2 0^ * " known ( S Rheinboldt C

i +

e

a n d

C

A

2 A

e

I(

i s

see

0 r t e

a

a n d

-5-

[70, p.319]) that if F

1

is not a Lipschitz operator then Newton iteration

does not possess, in general, quadratic convergence. For instance, applying Newton iteration to F(x) - x + x* where a = 5/3 we get

e

a

i

e

i+i • ( - > i /
#

•

Remark 2.2 Theorem 2.1 states how fast Newton iteration converges. The speed of convergence depends on the value of q. This technique based on q seems to be general and can be applied for any iteration.

See Traub and Wozniakowski [77]

where the class of interpolatory iterations I , n ^ 3, is considered.

•

We show that if we want (2.5) to hold then (2.3) is sharp for any value of q, 0 < q < 1. Theorem 2.2 There exists a polynomial F satisfying (2.2) for which the following is true.

For any q 6 (0,1) such that A ( O r > q/(l+2q) there exists an x^ so 2

that for the Newton iterate Xj^ we have e.^ > q e . Q

Proof 2 Let F(x) « x + x , x 6 K. Then a = 0 and A ( D = 1 . 2

A T » r > q/(l .4- 2q) and let x 2

Q

- -F.

by Newton iteration is equal to x^ - x Hence e^ > qe^.

Suppose that

The next approximation x^ constructed - F (x^) * F ( * q ) ~ ^I(1-2H f

Q

> q r - -q^Q-

•

From theorem 2 . 1 with q -» l" it follows that Newton iteration converges whenever

-6-

A rr = ! 2

then there exists an x^ 6 J for which Newton iteration is not convergent.

Proof Define fx + x

(2.11)

F(x) =
c ^-cy||
0.

A

•

Conjecture 3.1 states that starting from any point of J we get convergence; however, comp(Xg;e) can be arbitrarily high when x^ approaches the boundary of J. We prove Conjecture 3.1 for Newton iteration.

Theorem 3.1 If Newton iteration is applied to 2 CYL '+ x (3.8) F(x) =< ix - x

for x ^ 0

2

for x ^ 0

then for e fixed comp(Xg,e) tends logarithmically to infinity as A2F approaches l/3 and

XQ

*

T.

Proof Note that a

88

0 and A ( 0 = 1, see (2.11). 2

Applying Newton iteration to

(3.8) we get 2 -sign(x )x , \ and e l-2sign(x )x i+l j[

x

s

i+1 i ( 1

c

i

which is equivalent to (3.5) with A we get

2 e

i

i

2

- 1. Let x

±

l-2e

Q

i

- T =» | - 6. From Lemma (3.2)

-14-

comp(x.,e) T(l+o(l)) * T — * 4

1 + o ( 1 )

a s

6

"* °'

clg(J)

This proves theorem 3.1.

"

This phenomenon aoes not occur in the Kantorovich theory.

If h * ^ (in

the usual notation; see Rail [74]), the character of the convergence depends on the multiplicity of or. If a is a simple zero, the convergence is quadratic and the complexity g^es as lg lg(l/e).

If a is a multiple zero, the conver-

gence is linear and the complexity goes as lg(l/e). Note that F defined in (3.8) is not twice-differentiable at a. "'It seems to us that comp can tend to infinity as A^T approaches l/3 for twicedifferentiable problems. Therefore we propose

Conjecture 3.2 There exists a twice-differentiable function F with a simple zero a such that the comp « c o m p ^ D of Newton iteration satisfies lim comp(A D = + A r-i/3~ 9

•

0 0

1

2

Theorem 3.1 states that if k^V is close to l/3 then complexity can be arbitrarily large. We seek a bound on k^T (stronger than A r < l/3) to be 2

sure that complexity is not too large. Define

(3.9)

t - max(l, lg l/e).

Theorem 3.2 If A T £ l/4 then the complexity of Newton iteration satisfies 2

(3.10) comp £ c lg(l+t).

-15-

Proof Let h Vi.

i+1 - K h with K = A /(l - 2A e ) and h0

£ h 4> ^. 2 then In Traub and Wozniakowski [76a] we proved that if w » (Keg)"1 ^ ±

2

2

Q

83

e0. ._

From (3.1), e /-rr _

\

r\

1

±

±

. i

the complexity comp(h) of the sequence (hu) is bounded by

comp(h) £ c lg(l+t) •

Since comp ^comp(h), (3.10) follows. Remark 3.1 We have chosen the endpoint w = 2 in the inequality w ^ 2 only for convenience and definiteness. Any value w bounded away from unity can be used 1/v to get a good bound on complexity. For instance, w ^ 2 comp £ c log(l+vt). See Traub and Wozniakowski [76a].

implies g

Theorem 3.2 motivates the definition of the optimal complexity number for a superlinear iteration. Let Y