Recursive projection and interpolation algorithms with

Recursive projection and interpolation algorithms with applications Khalide Jbilou

Abstract: In the present paper, we will consider the recursive projection algorithm (RPA) and the recursive interpolation algorithm (RIA). Using the RIA, we will derive two direct methods for solving linear systems of equations and give the connections with some known methods. We will show that the Huang and the Gram-Schmidt algorithms are particular cases of the RIA. Finally, we will derive an iterative method for solving linear systems and show that the resulting algorithm is mathematically equivalent to GCR. Keywords: Linear systems, recursif algorithm, projection, interpolation. AMS(MOS) subject classi cations. 65F10, 65F25.

1 Introduction Solving systems of linear equations by direct or iterative methods can be always derived from a more general interpolation and projection problem. In [1], Brezinski proposed two algorithms, called the recursive projection algorithm (RPA) and the recursive interpolation algorithm (RIA). These algorithms could be applied for implementing some vector extrapolation methods for solving linear and nonlinear systems, see [1], [10] and [11]. They are also connected to other methods used in numerical analysis [1]. The aim of this paper is rst to study theoretically the RPA and the RIA and give some extensions. Secondly, we will establish that many existing iterative or direct methods for solving linear systems could be derived from these algorithms. In particular, we will show that some known methods such as the bordering method [6], the Purcell's method [13], the Huang algorithm [9], the classical and modi ed Gram-Schmidt algorithms [8], and the generalized conjugate residual (GCR)  Laboratoire d'Analyse Num erique et d'Optimisation, UFR IEEA - M3, Universite des Sciences et Technologies de Lille, F-59655 Villeneuve d'Ascq Cedex, France.

1

method [5] could be obtained from the RIA. Notations: Let G be a symmetric positive de nite matrix of order N . We denote by (:; :)G the inner product with p respect to G de ned by (x; y)G = (x; Gy) and the corresponding norm k x kG = (x; Gx) for x, y 2 RN where (:; :) denotes the usual Euclidean scalar product in RN . The transpose of any matrix B with respect to the G-inner product will be denoted B t and de ned by (x; By )G = (B t x; y )G. Finally B T will denote the transpose of B when G = I .

2 Recursive projection and interpolation algorithms

Let y be any vector in RN . Let k be an integer such that k  N and let U~k and Z~k be two subspaces of RN of dimension k spanned respectively by fu1 ; : : :; uk g and fz1 ; : : :; zk g. Consider the following generalized interpolation problem: Find a vector pk such that pk = a1u1 + : : : + ak uk (2:1) with the interpolation condition (zi ; pk )G = (zi ; y )G = !i ; i = 1; : : :; k: (2:2) The relation (2.1) and (2.2) are respectively equivalent to pk 2 U~k ; (2:3) and y ? pk 2 Z~k ? ; (2:4) where Z~k ? denotes the orthogonal, with respect to the G-inner product, of the subspace Z~k . Now, if Uk and Zk denote the matrices of dimension N  k, whose columns are respectively u1; : : :; uk and z1 ; : : :; zk , we get from (2.3) pk = Uk a (2:5) where a = (a1 ; : : :; ak )T is a vector of Rk to be determined. In fact, the interpolation condition (2.4) is written as ZkT G (y ? pk ) = 0: (2:6) 2

Replacing (2.5) in (2.6), it follows that

ZkT G y ? (ZkT GUk ) a = 0:

(2:7)

If the k  k matrix ZkT GUk is nonsingular, the vector a is uniquely determined by

a = (ZkT GUk )?1 ZkT G y:

(2:8)

Finally, the unique solution of the generalized interpolation problem considered before is given by pk = Uk (ZkT GUk )?1 ZkT G y: (2:9) If we set Qk = Uk (ZkT GUk )?1 ZkT G, we have

pk = Qk y:

(2:10)

As Q2k = Qk , the operator Qk is an oblique projector onto U~k along the orthogonal Z~k ? of the subspace Z~k . Let Ek be the vector of RN de ned by Ek = y ? pk . Hence from (2.10), Ek is given by Ek = (I ? Qk ) y: (2:11) Remark that, in the case where the two subspaces U~k and Z~k are identical, we obtain an orthogonal projection with respect to the G-inner product. In this case, the vector pk given by (2.9) is the unique solution of the minimization problem min k y ? (a1 u1 + : : : + ak uk ) kG :

ai2R

(2:12)

Notice that Ek = y ? pk is the corresponding error. If G = I , pk is the best approximation to y from among the linear combinations of u1; : : :; uk and Ek is the remaider error; see [4] for some applications. Invoking the expressions (2.10) and (2.11) of pk and Ek , we can easily establish the following results

Theorem 1 Ek and pk are uniquely de ned if and only if the matrix ZkT GUk is nonsingular.

Theorem 2 If k = N , then pN = y and EN = 0.

3

Now, applying Schur's formula [2] to the expression (2.9), the vectors Ek and pk can be expressed as ratios of two determinants as follows

u ::: uk y Ek = (z ; y)G (z ; u )G : : : (z ; uk )G .. .. ... . . (zk ; y )G (zk ; u )G : : : (zk ; uk )G 1

1

1

1

1

1

(z ; u )G : : : (z ; uk )G = .. .. . . (zk ; u )G : : : (zk ; uk )G 1

u ::: uk 0 ( z ; y ) ( z ; u ) : : : ( z ; uk )G G G pk = ? .. .. ... (z ; y) (z ; .u ) : : : (z ; .u ) k G k G k k G 1

1

1

1

1

and

1

1

1

1

(2:12)

(z ; u )G : : : (z ; uk )G = .. .. . . (zk ; u )G : : : (zk ; uk )G 1

1

1

1

:

Notice that the determinant appearing in the numerator is a vector obtained by expanding the deteminant with respect to its rst row. In [1], Brezinski proposed two algorithms for computing recursively the Ek 's and the pk 's. These algorithms has been called respectively the recursive projection algorithm (RPA) and the recursive interpolation algorithm (RIA). They are de ned as follows

8 E = y; g ;i = ui ; i  1; > > > > > < Ek = Ek? ? (zk ; Ek? )G gk? ;k ; k > 0; (RPA) > (zk ; gk? ;k )G > > > : gk;i = gk? ;i ? ((zzk;; ggk? ;i))G gk? ;k ; i > k > 0: 0

0

1

1

1

1

1

1

We notice that the vector determinants (see [1]):

k k?1;k G gk;i , i > k,

1

can also be expressed as a ratio of two

u ::: uk ui (z ; ui)G (z ; u )G : : : (z ; uk)G gk;i = : : : : : : : : (z ; u ) (z ; u ) : : : (z ; :u ) k iG k G k kG 1

1

1

1

1

1

4

(z ; u )G : : : (z ; uk )G : : = : : : : (zk ; u )G : : : (zk ; uk)G 1

1

1

1

:

Remark that gk;i is obtained from Ek by replacing y by ui for i > k. Then using (2.11) we obtain gk;i = (I ? Qk ) ui (2:13) where the oblique projector Qk is de ned in (2.10). The RIA is derived from the RPA by taking Ek = y ? pk and then we obtain the following algorithm

8 > > < p = 0; (RIA) > (z ; p ) ? (z ; y ) > : pk = pk? ? k (kz?; gG ) k G gk? ;k ; k > 0: 0

1

1

1

k k?1;k G Let us notice that if for some k, Ek or pk cannot be computed by the preceding algorithms, i.e. (zk ; gk?1;k )G = 0, then particular rules are necessary to jump over

the singularity and to avoid the breakdown.

Let us consider now the case where the two subspaces U~k and Z~k are identical. Then the projector I ? Qk is an orthogonal projector with respect to the G-inner product. Taking Uk = Zk , we shall give another formulation of the RPA and the RIA. Let us rst give the following result

Theorem 3 If Uk = Zk , the vectors generated by the RPA satisfy the relations a) (zk ; Ek?1 )G = (gk?1;k ; Ek?1)G , k > 0 b) (zk ; gk?1;i)G = (gk?1;k ; gk?1;i)G , i  k.

Proof a) We showed that for k > 0, we have

Ek?1 = (I ? Qk?1 ) y and gk?1;i = (I ? Qk?1 ) ui for i  k > 0: (2:14) As zk = uk , it follows that (zk ; Ek?1)G = (uk ; (I ? Qk?1 ) y )G : (2:15) On the other hand, since I ? Qk?1 is an orthogonal projector, with respect to the G-inner product, we have

(I ? Qk?1 )2 = I ? Qk?1 and (I ? Qk?1 )t = I ? Qk?1 : Replacing in (2.15), we obtain (zk ; Ek?1)G = (uk ; (I ? Qk?1 )2 y )G = ((I ? Qk?1 ) uk ; (I ? Qk?1 ) y )G: 5

(2:16)

Now, since gk?1;k = (I ? Qk?1 ) uk , it follows that (zk ; Ek?1)G = (gk?1;k ; Ek?1)G :

b) Similarly, by using (2.14) and (2.16), we get (zk ; gk?1;i)G = (uk ; (I ? Qk?1 )2ui )G = ((I ? Qk?1 ) uk ; (I ? Qk?1 ) ui)G : Invoking (2.14), we nally obtain (zk ; gk?1;i)G = (gk?1;k ; gk?1;i)G ; i  k: Using the results of the preceding theorem, we obtain new formulations of the RPA and the RIA. These algorithms will be called the orthogonal recursive projection algorithm (ORPA) and the orthogonal recursive interpolation algorithm (ORIA). The ORPA is de ned as follows

8 E = y; g ;i = ui; i  1; > > > > > (gk? ;k ; Ek? )G < (ORPA) > Ek = Ek? ? (gk? ;k ; gk? ;k )G gk? ;k ; k > 0; > > > > : gk;i = gk? ;i ? ((ggk? ;k;; ggk? ;i))G gk? ;k ; i > k > 0: k? ;k k? ;k G Setting Ek = y ? pk , the vector pk is computed as follows 8 p = 0; g ;i = ui ; i  1; > > > > < (ORIA) > pk = pk? ? (gk? ;k(;gpk? );Gg? (gk)? ;k ; y )G gk? ;k ; k > 0: k? ;k k? ;k G > > ( g k ? ;k > : gk;i = gk? ;i ? (g ;; ggk? ;i))G gk? ;k ; i > k > 0: 0

0

1

1

1

1

0

1

1

1

1

1

1

1

1

0

1

1

1

1

1

1

1

1

1

k?1;k k?1;k G

1

1

Some applications of the ORIA for solving systems of linear equations will be given in the next sections. Let us rst give some properties of the preceding algorithms.

Theorem 4 The vectors pk and Ek generated by the ORIA and the ORPA exist

and are unique if and only if the Gram determinant: det[(ui ; uj )G ]; i; j = 1; : : :; k, is dierent from zero. 6

Proof: pk and Ek are uniquely de ned if and only if (zk ; gk?1;k )G 6= 0. But since zk = uk , it follows from the determinantal expression of gk?1;k that (zk ; gk?1;k )G = (uk ; gk?1;k )G = det[(ui; uj )G ]; i; j = 1; : : :; k:

Theorem 5 The vectors generated by the ORIA satisfy the following relations a) (uj ; gi? ;i)G = 0, for j = 1; : : :; i ? 1 1

b) (uj ; pk )G = (uj ; y )G, for j = 1; : : :; k c) (gj ?1;j ; pk )G = (gj ?1;j ; y )G , for j = 1; : : :; k d) (gj ?1;j ; gi?1;i)G = 0 for i 6= j e) (gi?1;i; pk )G = 0, i > k f) spanfg0;1 ; g1;2; : : :; gk?1;k g = spanfu1 ; u2; : : :; uk g g) pk minimizes k y ? x kG over the linear subspace spanfg0;1; g1;2; : : :; gk?1;k g.

Proof: a) Follows directly by using the determinantal expression of gi?1;i. b) As uj = zj for j = 1; : : :; k, we have from the determinantal expression of Ek , (uj ; Ek )G = 0, then (uj ; pk )G = (uj ; y )G. c) For j = 1; : : :; k, gj?1;j can be developped as

gj?1;j = uj +

jX ?1 l=1

ajl ul :

Multiplying this expression by Ek , we obtain (gj ?1;j ; Ek )G = (uj ; Ek )G +

jX ?1 l=1

ajl (ul; Ek )G :

But since (ui; Ek )G = 0 for i = 1; : : :; k, it follows that (gj ?1;j ; Ek )G = 0; hence (gj ?1;j ; pk )G = (gj ?1;j ; y )G: d)For i > j , let us multiply (2.17) by gi?1;i . Hence, using a) it follows that (gi?1;i; gj ?1;j ) = 0:

e) The vector pk is computed as pk = pk?1 ? (gk?1;k(;gpk?1);Gg? (gk)?1;k ; y)G gk?1;k ; k > 0: k?1;k k?1;k G 7

(2:17)

For i > k, multiplying pk by gi?1;i and using d), we get (gi?1;i ; pk )G = (gi?1;i; pk?1 )G = : : : = (gi?1;i ; p0)G = 0:

f ) From (2.17) we see that, for i = 1; : : :; k, gi?1;i 2 spanfu1 ; : : :; uk g. On the other hand, using c), the vectors g0;1; : : :; gk?1;k are linearly independent. Therefore spanfg0;1; g1;2; : : :; gk?1;k g = spanfu1 ; u2; : : :; uk g: g) Follows directly from (2.12). In the following section, we will use the RIA and the ORIA to derive direct methods for solving linear systems and give the connections with some known methods.

3 Direct methods for solving linear systems Let us consider the linear system

Ax=b

(3:1)

where A is a nonsingular N  N matrix and b is a vector of RN . Let x denotes the exact solution of (3.1). In what follows, we will take G = I , hence the G-inner product becomes the usual Euclidean product in RN .

3.1 Method A

Let ai = (ai;1; : : :; ai;N ) denotes the i-th row of the matrix A and bi the i-th component of the vector b. We choose the vectors zi and ui of RN +1 as follows

zi = (ai;1; : : :; ai;N ; ?bi)T ; and ui = ei ; i = 1; : : :; N + 1; where fei g; i = 1; : : :; N + 1, is the canonical basis of RN +1. With these notations, the vector gk;i is in RN +1 and is given by

e : : : ek (z ; e ) : : : (z ; e ) ei k gk;i = (z ; ei) (z ; e ) : : : (z ; ek ) = .. . : . .. .. . ... . . . (zk ; ei ) (zk ; e ) : : : (zk ; ek ) (zk ; e ) : : : (zk ; ek ) 1

1

1

1

1

1

1

1

1

8

1

(3:2)

Hence, for k = 0; : : :; N and k < i  N + 1, gk;i can be developped as

gk;i = ei + 1;i e1 + : : : + k;i ek :

(3:3)

Now, as (zj ; gk;i) = 0 for j = 1; : : :; k and i = k + 1; : : :; N + 1, the j;i ' s are solution of the k  k linear system

8 > k;i + a ;i = 0 > < aa ;;  ;i;i ++ :: :: :: ++ aa ;k;k k;i + a ;i = 0 > > : :a: ::: :: :+:: :: :::: +: ::a: :  + a = 0: 11

1

1

1

21

1

2

2

k;k k;i

k;i

k;1 1;i

We notice that for i = N + 1, we set aj;N +1 = ?bj ; j = 1; : : :; N . Let us de ne 'k;i = (1;i ; : : :; k;i )T ; i = k + 1; : : :; N + 1. Then we have gk;i = ('k;i ; 0; : : :; 1; 0; : : :; 0)T , where the value 1 corresponds to the i-th component of the vector gk;i . For each i 2 f1; : : :; N + 1g, the last k  k linear system can be written as

Ak 'k;i = ?(ai;1 ; : : :; ak;i)T ;

(3:4)

where Ak is the submatrix formed by the rst k rows and the rst k columns of the matrix A. When k = N and i = N + 1, the linear system (3.4) is exactly the initial system (3.1). In this case, we have 'N;N +1 = x and gN;N +1 = (x ; 1)T . Hence, the solution of the system (3.1) can be computed by the following algorithm

8 g = e ; i = 1; : : :; N + 1: > ;i i > > > < gk;i = gk? ;i ? ((zzk;; ggk? ;i )) gk? ;k ; k = 1; : : :; N ; i = k + 1; : : :; N + 1 > k k? ;k > > > : gN;N = (x; 1)T 0

1

1

1

1

+1

This method is equivalent to the Purcell's method [13]. The algorithm requires

N 3=3 + O(N 2) multiplications to calculate x .

9

3.2 Connection with the L U decomposition

Let M be the N  N matrix whose columns are the vectors g0;1; : : :; gN ?1;N . Using the results of the last subsection, M is the unit upper triangular matrix given by 0 1 1 1;2 : : : 1;N BB 0 1 : : : 2;N CC M =B B@ ... ... . . . ... CCA : 0 0 ::: 1 Let us set L1 = A M , then the matrix L1 is expressed as 1 0 a1;1 a1;11;2 + a1;2 : : : a1;11;N + a1;2 2;N + : : : + a1;N B a1;2 a2;11;2 + a2;2 : : : a2;11;N + a2;2 2;N + : : : + a2;N C CC B B L1 = B .. . .. . CA : . . . @ . . . a1;N aN;11;2 + aN;2 : : : aN;11;N + aN;22;N + : : : + aN;N Now, for j = 1; : : :; k and i = k + 1; : : :; N + 1, we have from (3.4)

aj;11;i + aj;22;i + : : : + aj;k k;i + aj;i = 0: This shows that L1 is a lower triangular matrix which nonzero elements are given by 1 lp;q = ap;1 1;q + ap;22;q + : : : + ap;q ; q = 1; : : :; N and p  q: 1 Let D denote the diagonal matrix de ned by D = diag (l11;1; l21;2; : : :; lN;N ) and L 1 1 the unit lower triangular N  N matrix whose elements are lp;q = lp;q =lq;q . Then since L1 = A M , we obtain A = L D M ?1 : Setting U = D M ?1 , the matrix U is upper triangular and we obtain the decomposition A = L U:

3.3 Method B

Let fei g1iN denote the canonical basis of RN . The solution x of the linear system (3.1) can be expressed as where the xj 's verify

x = x1e1 + x2e2 + : : : + xN eN ;

ai;1 x1 + ai;2 x2 + : : : + ai;N xN = bi ; i = 1; : : :; N: 10

Let Ak be the k  k submatrix formed by the rst k row and the rst k columns, and consider the associated linear system of dimension k Ak x(k) = b(k); (3:7) where b(k) = (b1; : : :; bk )T and x(k) = (x(1k) ; : : :; x(kk))T are two vectors of Rk . If we set pk = (x(1k); : : :; x(kk); 0; : : :; 0)T and Ai;k = (ai;1; : : :; ai;k ; 0; : : :; 0)T , then the linear system (3.7) is written as (Ai;k ; pk ) = (Ai;k ; x) = bi; i = 1; : : :; k: Hence, at step k, solving (3.7) is equivalent to solve the generalized interpolation problem 8 > < pk = x(1k)e1 + x(2k)e2 + : : : + x(kk)ek (3:8) > (A ; p ) = b ; i = 1; : : :; k: : i;k k i  When k = N , we have pN = x the exact solution of the linear system (3.1). pk can be recursively computed by 8 p = 0; g = e ; i = 1; : : :; N > 0;i i > 0

> > > < pk = pk? + bk ? (Ak;k ; pk? ) gk? ;k ; k = 1; : : :; N (Ak;k ; gk? ;k ) > > > > > : gk;i = gk? ;i ? ((AAk;k;; ggk? ;i)) gk? ;k ; k = 1; : : :; N ? 1; i > k: 1

1

1

1

1

1

1

k;k k?1;k

The computation of pN by the preceding algorithm requires N 3=3 + O(N 2) multiplications. Note that this algorithm is equivalent to the bordering method [6] and to the method of Sloboda [17]. Remark that the methods A and B are oblique projection methods. In the following two subsections, we will consider orthogonal methods and show how to derive the Huang algorithm [9] and the Modi ed Gram-Schmidt (MGS) algorithm [8] from the ORIA.

3.4 The Huang algorithm

Let ai = (ai;1; : : :; ai;N )T ; i = 1; : : :; N , denotes the vector formed by the i-th row of the matrix A. We set y = x and ui = zi = ai ; i = 1; : : :; N: (3:10) 11

Using the RIA and setting xk = pk for k = 0; 1; : : :, we have  xk = xk?1 + (ak ; x(a) ;?g (ak ; x) k?1) gk?1;k ; k = 1; : : :; N: k k?1;k But since x is the exact solution of the system (3.1), it follows that (ak ; x) = bk ; k = 1; : : :; N: If we set qk?1 = gk?1;k and use the fact that (ak ; qk?1 ) = (qk?1 ; qk?1 ), the vector xk is computed by xk = xk?1 + bk(q? (a;kq; xk?)1) qk?1 ; k = 1; : : :; N: (3:11) k?1 k?1 Now, as x0 = p0 = 0, it follows from (3.11) that

xk =

k b ? (a ; x ) X i i i? i=1

1 (qi?1 ; qi?1 ) qi?1 :

(3:12)

Note that as we are dealing with an orthogonal method, xk can be computed by the ORIA as follows k b ? (q ; x ) X i i?1 i?1 q : xk = (3:13) i?1 i=1 (qi?1 ; qi?1 ) On the other hand, the vectors of direction qk = gk;k+1 are given by

8 > > < g ;j = aj ; j = 1; : : :; N (g ; g ) > > : gk;i = gk? ;i ? k? ;k k? ;i gk? ;k ; i = k + 1; : : :; N: 0

1

1

(3:14)

(gk?1 ; gk?1;k ) 1 Another way for computing the vectors qk is as follows. From the result c) of theorem 5, we have (gi?1;i; gj ?1;j ) = 0 for i 6= j . Using this orthogonality relation in (3.14), we get for k = 1; : : :; N ? 1 , the expression 1

gk;k+1 = g0;k+1 ?

k (g X ;k ; qi? ) q : (q ; q ) i? 0

i=1

+1

1

1

i?1 i?1

(3:15)

Then, since g0;k+1 = ak+1 and gk;k+1 = qk , we nally obtain

qk = ak+1 ?

k (a ; q ) X k i? (q ; q ) qi? : +1

i=1

12

1

i?1 i?1

1

(3:16)

The algorithm de ned by (3.13) and (3.16) is known as the classical Huang algorithm [9]. Notice that since xN = pN = x (theorem 2), the algorithm gives the exact solution of the system (3.1). As we will see in the next subsection, (3.14) is identical to the stabilized Gram-Schmidt procedure, which is numerically preferable to (3.16). Hence, the algorithm de ned by (3.11) and (3.14) is numerically better than the classical Huang algorithm.

3.5 The Classical and the Modi ed Gram-Schmidt algorithm We also consider the case G = I and set

ui = zi = a(i); i = 1; : : :; N; where a(i) denotes the i-th column of the matrix A. We will be interested in the gk;i 's given by the auxiliary rule the ORIA. These vectors are given by 8 (j ) > > g0;j = a ; j = 1; : : :; N

< (g ;g ) > > : gk;i = gk? ;i ? k? ;k k? ;i gk? ;k ; i > k = 1; : : :; N ? 1: 1

1

(gk?1;k ; gk?1;k ) 1 Let Q be the N  N matrix whose columns are de ned by 1

q(j) = k ggj?1;j k ; j = 1; : : :; N: j ?1;j

Now, since ui = zi ; i = 1; : : :; N , we have, from theorem 5, the orthogonality relation (q (i); q (j )) = 0; i 6= j: (3:17) This relation shows that Q is an orthonormal matrix. Let us set

R = QT A:

(3:18)

Then, using the fact that uj = a(j ) and the result a) of theorem 5, we obtain (q (i); a(j )) =k gi?1;i k?1 (gi?1;i; uj ) = 0; for j < i:

(3:19)

It follows from (3.18) and (3.19) that R is upper triangular and we get the decomposition A = QR 13

which is the Modi ed Gram-Shcmidt procedure for the QR decomposition of the matrix A. Remark that the vector gj ?1;j is the orthogonal projection of a(j ) onto the orthogonal of the subspace spanfa(1) ; : : :; a(j ?1)g. Let us notice that, since (gi?1;i ; gj ?1;j ) = 0 for i 6= j , the vector gk;k+1 can be given as k X gk;k+1 = g0;k+1 ? ((ggi?1;i ;;gg0;k+1)) gi?1;i: i=1

i?1;i i?1;i

Therefore, as g0;k+1 = a(k+1) , it follows that

gk;k+1 = a(k+1) ?

k X i=1

ri;k gi?1;i; where ri;k = ((ggi?1;i ;;ag

k

): i?1;i i?1;i) ( +1)

(3:20)

This leads to the Classical Gram-Schmidt algorithm (GS). Although the two algorithms are mathematically equivalent, MGS is numerically better than GS; see [14].

4 Iterative methods for solving linear systems

In what follows, we will take G = AT A and use the ORIA to derive an iterative method for solving the system (3.1). For any intitial guess x0 , we set pk = xk ? x0; k = 0; 1; : : :; where pk is the vector obtained by the ORIA with the choices

y = x ? x0 ; and ui = zi = ri?1 = b ? A xi?1 ; i = 1; : : :; k: Using the ORIA, the vector pk is given by

(4:1)

8 > > < p =0 (g ; y) ? (g ; p ) > > : pk = pk? + k? ;k(g G ; g k? ;k) k? G gk? ;k ; k = 1; : : :: 0

1

Let us set

1

1

1

k?1;k k?1;k G

qk?1 = gk?1;k ; k = 1; 2; : : ::

14

1

(4:2)

Now, using the fact that y = x ? x0, pk = xk ? x0 and G = AT A, the approximation xk is computed as Aqk?1 ; rk?1) : xk = xk?1 + k qk?1 ; where k = ((Aq (4:3) k?1 ; Aqk?1 ) The direction vectors qk = gk;k+1 are obtained from the auxiliary rule of the ORIA as follows 8 g = u = r ; j = 1; : : :; k + 1 > > 0;j j j?1

> > < (Agm? ;m ; Agm? ;i) g g = g ? m;i m ? ;i > (Agm? ;m ; Agm? ;m) m? ;m ; m = 1; : : :; k; i > k > > > : qk = gk;k : 1

1

1

1

1

1

(4:4)

+1

Note that the algorithm de ned by (4.3) and (4.4) converges in at most N iterations. In fact, when k = N , we have pN = y (theorem 2). But since pN = xN ? x0 and y = x ? x0, it follows that xN = x. Let us give now some properties of the preceding algorithm Theorem 6 the vectors xk , rk and qk generated by the algorithm (4.3)-(4.4) satisfy the following relations a) (Arj ; Aqi ) = 0; j < i. b) (Arj ; ri) = 0; j < i. c) (Aqj ; Ari) = 0; j < i. d) (Aqj ; Aqi ) = 0; j 6= i. e (Aqj ; ri) = (Aqj ; r0); j  i. f ) spanfq0 ; : : :; qk?1 g = spanfr0 ; : : :; rk?1g. g) xk minimizes k b ? Ax k over the ane subspace x0 + spanfq0 ; : : :; qk?1 g. Proof: It is just an application of the theorem 5. There is another way for computing the direction vectors qk . From(4.4), we have k?1 ; Agk?1;k+1) q ; k  1: (4:5) gk;k+1 = gk?1;k+1 ? (Aq (Aq ; Aq ) k?1

k?1 k?1 Using the orthogonality relation (Aqi ; Aqj ) = 0; for i 6= j , we can easily k X i?1 ; Ag0;k+1) q : gk;k+1 = g0;k+1 ? (Aq i?1 i=1 (Aqi?1 ; Aqi?1) 15

show that

(4:6)

Hence, as g0;k+1 = rk , it follows from (4.6) that

qk = rk ?

k X i=1

(Aqi?1 ; Ark ) :

k;i qi?1 ; where k;i = (Aq ; Aq ) i?1

(4:7)

i?1

The algorithm de ned by (4.3) and (4.7) is exactly the GCR algorithm. The two ways (4.4) and (4.7), for computing the qk 's, require the same number of arithmetic operations. However, as (4.5) is based on the stabilized Gram-Schmidt procedure, it is numerically more stable than (4.7); see [16] for some examples. Using the determinantal expressions of pk and gk;k+1 , the vectors xk and qk can be expressed as ratios of two determinants

x r ::: rk ? ?(Ar ; Ar ) (Ar ; Ar ) : : : (Ar ; Ark? ) .. .. .. . . ?(Ar ; Ar ) (Ar ; Ar ) : : : (Ar ;. Ar ) ? k? k? k? xk = (kAr ; Ar ) : : : (Ar ; Ark? ) .. .. . . 0

0

0

0

0

0

1

0

0

1

0

0

1

0

1

1

0

1

1

(4:8)

(Ar0; Ark?1) : : : (Ark?1 ; Ark?1)

The determinantal expression of qk is obtained from that of xk by replacing the rst column in the numerator of (4.8) by [rk ; (Ar0; Ark ); : : :; (Ark?1; Ark )]T . Other iterative projection algorithms such as the orthogonal-error method [7], the Orthodir [18] and the method of Arnoldi [15] could be obtained from the ORIA and the RIA.

References [1] C. Brezinski, Recursive interpolation extrapolation and projection, J. Comp. Appl. Math., 9(1983) 369-376. [2] C. Brezinski, Other manifestations of the Schur complement, Linear. Alg. Appl., 111(1988) 231-247. [3] C. Brezinski, Redivo Zalia, Extrapolation Methods, Theory and Practice, North-Holland, Amsterdam, 1991. [4] P. J. Davis, Interpolation and Approximation, Blaisdell Publ. Co., New-York, 1961. 16

[5] H.C. Elman , Iterative methods for large sparse nonsymmetric systems of linear equations, Ph.D.thesis, Computer Science Dept., Yale Univ., New Haven, CT, 1982. [6] D. K. Faddeev, V. N. Faddeeva, Computational Methods of Linear Algebra, W. H. Freeman, San Fransisco, 1963. [7] V. Faber, T. Manteufel, Orthogonal error methods, SIAM J. Numer. Anal., 24 (1987) 170-187. [8] G. H. Golub, C. F. Van Loan, Matrix Computations, Johns Hopkins Press, Baltimore, MD., 1989. [9] H. Y. Huang, A direct method for the general solution of a system of linear equations, J. Opt. Th. Appl., 16(1975) 429-445. [10] K. Jbilou, A general projection algorithm for solving systems of linear equations, Numerical Algorithms, 4(1993) 361-377. [11] K. Jbilou, H. Sadok, Some results about vector extrapolation methods and related xed point iterations, J. Comput. Appl. Math. 36 (1991) 385-398. [12] A. Lembarki, Methodes de projection et extensions: etude theorique et pratique, These de 3eme cycle, Universite de Lille 1, 1984. [13] E. W. Purcell , The vector method for solving simultaneous linear equations, J. Math. and Phys., 32(1954) 180-183. [14] J. R. Rice , Numerical aspects of Gram-Schmidt orthogonalization of vectors, Linear Alg. Appl., 52/53(1983) 591-601. [15] Y. Saad , Krylov subspace methods for solving large unsymmetric linear systems , Math. Comput., 37 (1981) 105-126. [16] H. Sadok, A uni ed approach to conjugate gradient algorithms for solving nonsymmetric linear systems, submitted. [17] F. Sloboda, A parallel projection method for linear algebraic systems, Aplikace Mathematiky, 23(1978) 185-198. [18] D.M. Young and K.C Jea, Generalized conjugate-gradient acceleration of nonsymmetrisable iterative methods , Lin. Alg. Appl., 34(1980) 159-194. 17