Nonlinear successive over-relaxation - Springer Link

Numer. Math. 44, 309-315 (1984)

Numerische Mal emaUk

9 Springer-Verlag 1984

Nonlinear Successive Over-Relaxation* M.E. Brewster and R. Kannan The University of Texas at Arlington, Department of Mathematics Arlington, Texas 76019, USA

Summary. We study the convergence of Gauss-Seidel and nonlinear successive overrelaxation methods for finding the minimum of a strictly convex functional defined on R".

Subject Classifications: AMS(M OS): 65 H 10; CR: 5.15. Nonlinear Successive Over-Relaxation 1. In this paper we consider the convergence of successive overrelaxation methods (SOR) for a system of nonlinear equations F ( x ) = 0 in R". The system of nonlinear equations is assumed to be derived from the problem of minimizing a strictly convex (not necessarily quadratic) functional and our discussions follow the lines of [9, 10]. We obtain here a convergence theorem which parallels the results of [9, 10] but our method of proof gives more general sufficient conditions for convergence and also allows us to study a more general class of functionals whose Hessian matrix may be singular. This investigation of nonlinear SOR was motivated by two considerations: i) the recent results on Newton's method at singular points ([3, 5, 7, 8]) and ii) the discussions in [1] concerning the range of the relaxation parameter for the minimal surface problem. In [3, 5, 7, 8] sufficient conditions for the convergence of the classical Newton's method for a system of nonlinear equations in R" whose Jacobian is singular at the solution are derived. We propose to study the general problem of convergence criteria for variants of Newton's method at singular points. In this paper we restrict ourselves to the discussion of convergence of SOR for such problems. In [-1] it is observed that for the minimal surface problem the optimum rate of convergence is obtained for the choice of relaxation parameter close to 2. However in [10] the relaxation parameters are required to be bounded away from 2. We obtain here sufficient Conditions for convergence of nonlinear SOR when the parameters approach 2. *

Partially supported by U.S. Army Grant 0r DAAG29-80-C-0060

310

M.E. Brewster and R. Kannan

It must be remarked here that the conjugate gradient method has also been utilized in [2] to study the minimal surface problem numerically and a comparative study of various techniques has been presented. Nonlinear SOR has also been utilized in [4, 6] to study numerical solutions of elliptic problems and the divergence phenomenon observed in [4] (cf. also [1]) also provided some of the motivation to study the theoretical aspects of nonlinear SOR. We conclude by remarking that a detailed account of the various aspects of linear SOR may be seen in [12]. 2. The following assumptions will be made throughout this paper: i) D is a convex domain in Rn; ii) 49: D ~ R is a strictly convex functional i.e., for all x,y in D and for every

,~(0,1), ;~ 49(x) + ( 1 - ;~) r

- 49(;~x + ( 1 - ,~) y) >=o,

with equality holding only when x = y; iii) 49 is twice continuously differentiable on D. In addition to these assumptions on 49 we make the following two important hypotheses: iv) there exists ~eR such that S~={xeD: 49(x)49(x*) and Sr is compact9 By ii) the minimum point x* is unique9 Finally a point x*~D is the minimum point of 49 if and only if grad 49(x*)=0. 3. We first derive, as in I-9], the nonlinear analogue of the Gauss-Seidel method for positive definite matrices. Thus we have: Theorem 1. From any x~176 follows:

~.... ,x ~ in S~ we generate a sequence {x k} as

:*ik and

(1) xR.+X is the solution of |k

F~k(x],xk2.... ,X~k_l,Xi~,X~k+l ..... xk,)=0

where Fi denotes the component of grad 49 in the ith coordinate direction and ik is any one of the integers 1, 2, ..., n. Such a sequence {x k} is uniquely defined and converges to x*, the unique global minimum of 49, provided that in the above iterative process every coordinate direction i is chosen an infinite number of times. Proof By the hypotheses on 49, for a given xkeS~ and ika(1,2 ..... n) there exists 9 a umque point x k+l satisfying (1). Also x k + l sS~ and hence {xk} is uniquely

Nonlinear Successive Over-Relaxation

311

defined for a given x~ and sequence {JR}. F r o m the hypotheses it follows that q5(x k) > dp(x k + 1) > q5(x*) for all k = 0, 1, 2 ..... and thus the sequence {th (xk)} is non-increasing, b o u n d e d below and hence convergent. By the application of Taylor's theorem, we now obtain: ~(x k)_ (o(xk+l )=$F~,~(y 1 k)

k

Xik--

X.k+l~2 ~-k J '

k = 0 , 1, 2, " ' "

(2)

for some yk~(xk, xk+l) (where (xk, x k§ denotes the open line segment joining x k and xk+I). Let m=glb{F~klk(yk), k = 0 , 1 , 2 , . . . } . Since the Hessian of ~b is positive semidefinite, we have m=>0. If m = 0 , then by v) there exists some subsequence of {y*} converging to x*. Since dp(yk)>=~(xk+l), k = 0 , 1 , 2 ..... it follows that ck(xk)~(x *) and thus xR~x *. We now consider the case when m > 0 . Since the sequence {~b(xk)} is convergent, we have from (2) that Ix~ - x l kk+l I-~0. Because {xk}~sr (a bounded, closed set) there exists a limit point ~ of the sequence {XR}. Let I ={i:i~{1,2,...,n} such that F/(~)=0} and J = { 1 , 2 ..... n } - I . We define the sets H~, i = 1 , 2 , ... n by Hi={u~Sr: F~(u)=0}. It can be seen that the sets H~ are closed and nonempty. Then I is a n o n e m p t y set because ~eH~ for some i. If J is empty, clearly ~ = x * . Let J = (Jl, ...,JL} and I = {il, ..., i,-L}. Finally let H s j~J

Since H j is closed, there exists z > 0 such that

y~Hs

implies

I~-y[__>T.

(1)

Recall x k - x k+ 1__,0. Thus there exists an integer N such that k>=N implies

Ixk - x k+ 11< z/2.

(2)

Let B a c R" be the set of all x such that: I(xj,, ..., x j ~ ) - (~,, .... xi~)l < 6

(3)

~b(x) < ~ + a = ~b(~) + 6.

(4)

and We first observe that for any e > 0 we m a y choose 6(e)>0 such that if xeBa then I x - ~ l < g . F o r if the assertion is not true, then for any sequence {6~} decreasing to zero there exists {z 1} such that zleBa and Iz 1 - ~ l > e . All such z 1 are contained in S~+a, which is c o m p a c t and thus there exists a limit p o i n t z ~ o .... z o~ ) - ( ~ , , ..., ~j~)l=0 and of {z 1} such that ] z ~ Also from (3), I(z j,, from (4), qS(z~ q~=~b(~). Thus ( g r a d ~b(~), ~ - z~ = ~b(~) - ~b(z~ = 0 which contradicts the strict convexity of ~b. Thus for every e, 6 as above we m a y choose k(a)>N such that xt~Bo. If the next coordinate directions ik+,,ik+2,.., are in I, then xk+*,X ~+2.... remain in Ba. If i~ is the first direction chosen from J for q ~ k then xq~B and xq+leHs. Thus

z/2 >tx~-xq+ ll >lxq + ~ - ~ l - l x ~ - X l > z - e > ~ / 2

312


which is a contradiction. H e n c e xqeB6 for q>k(3). But 6 can be m a d e arbitrarily small by choosing e small. Hence ~ is the unique limit point of {xk}. Because every coordinate direction is chosen an infinite n u m b e r of times in generating the sequence {xk}, then for each i = 1.... , n there exists a subsequence of {x k} in H i. Thus ~ is in H i for all i = l ..... n. Since x* is the unique intersection point of all H i, ~ = x*. 4. In the spirit of the above proof, we now consider the convergence of nonlinear SOR. Thus we consider, under the hypotheses of the above theorem, the iterative process which is generated by implementing one step of Newton's m e t h o d instead of exact minimization in some coordinate direction as in the previous theorem. Theorem 2. From any x ~ in S~, let {x k} be the sequence generated by

x jk + l _-- Xj,k

j~ik

and

(3) X~+1

=xkk--(.O k Fik (Xk)

'~

F~ ,k(xk) "

Further let {Ik} be defined by

Ik={x: ch(x) 0 and if m=0, then x t ~ x *. If m>0, two cases arise.

Case 1. 5 0 such that lY~-yt~- by the continuity of Fj. As before there exists e' > 0 such that

yeB~,

implies

[~-y[ N a such that xPeB~,. Thus ipeI and hence x~+*~B~,. By induction we have that x~B~, for q>p. This implies iqq~J for q>p, hence J is empty which is a contradiction. Thus xk ~ x *.

Case 2: g(F(xk)) < cok < 26 k -- g (F(xk)). In this case either F(x k) is bounded away from zero and then case 1 is applicable or a subsequence of F(x k) converges to zero which implies xk-ox *.

Remarks a) The range of the parameter cok in Theorem 2 of our paper is different from the corresponding range of cok in [10]. We first recall the result of [10] and use

314


the notation ]~k to distinguish from

~k"

Thus, in [10], ~'k is defined to be

F~i~ (xk) where max {Fi~ik(v)} reDk

lFi~(xk)l) and Dk = {Y: lY-Xkl