Synchronous and Asynchronous Two-stage Multisplitting Methods Daniel B. Szyld
y
Abstract
Di erent types of synchronous and asynchronous two-stage multisplitting methods for the solution of linear systems are analyzed. The di erent methods which appeared in the literature are reviewed, and new ones are presented. Convergence properties of these methods are studied.
1 Synchronous Methods
In this paper we analyze di erent types of synchronous and asynchronous two-stage multisplitting methods for the solution of linear systems of the form Ax = b, where A is an n n nonsingular matrix. We concentrate our study in two important cases: when A is monotone, i.e., when A?1 0 [1], and when A is an H -matrix [21], [26]. The multisplitting method was introduced in [20] and was further studied, e.g., in [7], [8], [19], [27], [28]. This method consists of having a collection of splittings
A = M ? N ; ` = 1; ; L;
(1)
`
`
with M nonsingular, and diagonal nonnegative weighting matrices E which add to the identity, and performing the following: Algorithm 1. (Multisplitting). Given the initial vector x0 . For i = 1; 2; , until convergence. For ` = 1 to L (2) M y = N x ?1 + b `
`
`
x=
`
`
i
X L
i
Ey `
`
`=1
Convergence of the multisplitting method was established for A?1 0 in [20] when the splittings (1) are weak regular [1], [25]. Comparison of convergence of di erent splittings (1) when they are M -splittings [17], [22], was studied in [19]. Often the splittings (1) are all the same, i.e., M = M , N = N , ` = 1; ; L, and the weighting matrices are such that the multisplitting algorithm is a rendition of classical block iterative methods, such as block-Jacobi; see, e.g., [25], [29] and [10, Def. 1.1 (a)]. In this case, we say that the weighting matrices form a partition of the identity. In fact, the common application of the multisplitting algorithm is when the splittings A = M ? N correspond to M being a diagonal block of A even when the corresponding weighting matrix `
`
`
`
`
This work was supported by the National Science Foundation grant DMS-9201728. Department of Mathematics, Temple University, Philadelphia, Pennsylvania 19122-2585, USA (
[email protected]). y
1
2
Daniel B. Szyld
has ones in fewer entries than the order of that diagonal block; see, e.g., [10, Def. 1.1 (b)]. In this case the method is called overlapping block-Jacobi multisplitting; see also [9], where it was established that the overlapping method with weights forming a partition of the identity is asymptotically faster than block-Jacobi. Of course, multisplitting methods were conceived for parallel computations, since each linear system (2) can be solved by a di erent processor. These methods can also be used to precondition conjugate gradient type methods; see, e.g., [2], [14], [28]. When the linear systems (2) are solved iteratively in each processor, using the splittings
M = B ? C ; ` = 1; ; L; and performing a xed number s of iterations, one obtains the following:
(3)
`
`
`
Algorithm 2. (Two-stage Multisplitting). Given the initial vector x0, and the xed number s of inner iterations. For i = 1; 2; , until convergence. For ` = 1 to L y 0 = x ?1 For j = 1 to s (4) B y = C y ?1 + N x ?1 + b `;
i
`
x=
(5)
`;j
`
`;j
`
i
X L
Ey `
i
`;s
`=1
Convergence of this method for any number of inner iterations s was established for 0 in [24] when the outer splittings (1) are regular splittings [1], [25], and the inner splittings (3) are weak regular splittings, cf. [11], [16]. Recently, in [15], the comparison of the rate of convergence between a two-stage overlapping multisplitting method and twostage block-Jacobi was established, extending the result in [9], [10] to the two-stage method. When the number of inner iterations varies for each splitting and for each outer iteration, i.e., when s = s(`; i) in Algorithm 2, we say that we have a Non-stationary Two-stage Multisplitting Algorithm. Model A in [3] is a special case of this algorithm, when the outer splittings (1) are all A = A ? O; see also [4], [13] where extrapolated methods are considered. Theorem 1.1. Let A?1 0. Let the splittings (1) be regular and the splittings (3) be weak regular. Then, the Non-stationary Two-stage Multisplitting Algorithm converges to x with Ax = b for any initial vector x0 and any sequence of numbers of inner iterations s(`; i) 1, ` = 1; 2; ; L, i = 1; Proof. Let e = x ? x be the error at the i-th outer iteration. Let R = B ?1 C . We rewrite (5) as
A?1
?
?
i
?
i
x=
`
X L
i
E [R ( )x ?1 + `
`=1
X?1
s(`;i)
s `;i
i
`
R B?1(N x ?1 + b)]; j `
j =0
`
`
i
cf. [12], [16]. Thus, e = H (i)e ?1 = H (i)H (i ? 1) H (1)e0, where i
(6)
H (i) =
i
X L
E [R
s(`;i)
`
`
X L
j `
`
`
E [R ( ) + (I ? R ( ))M ?1N ] = I ? Q(i)A; `
`=1
R B?1 N ]
j =0
`=1
=
+
X?1
s(`;i)
s `;i
s `;i
`
`
`
`
`
`
3
Asynchronous Two-stage Multisplitting Methods
where
Q(i) =
(7)
X L
E (I ? R ( ))M ?1: s `;i
`
`
`
`=1
The theorem follows now using the same argument as in [12, Theo. 2.2] with the obvious extensions to this case. H -matrices are not necessarily monotone; see [11] for an extensive bibliography and for de nitions of splittings of H -matrices, and [18], [23] for relaxed and extrapolated multisplitting methods for H -matrices. Theorem 1.2. Let A be an H -matrix. Let the splittings (1) be H -splittings and the splittings (3) be H -compatible splittings. Then, the Non-stationary Two-stage Multisplitting Algorithm converges to x with Ax = b for any initial vector x0 and any sequence of numbers of inner iterations s(`; i) 1, ` = 1; ; L, i = 1; 2; Proof. The proof follows from that of Theo. 1.1 in a way similar to the way that [12, Theo. 2.3] follows from [12, Theo. 2.2]. ?
?
2 Asynchronous Methods
All methods described in x1 are synchronous in the sense that step (5) is performed only after all approximations to the solutions of (2) are completed (` = 1; ; L). Alternatively, if the weighting matrices form a partition of the identity, each part of x , say x( ) = E x , can be updated as soon as the approximation to the solution of the corresponding system (2) is completed, without waiting for the other parts of x to be updated; here the matrices E correspond to the sets S in [10, Def. 1.1 (a)]. Thus, the previous iterate x ?1 is no longer available for the computation of (2) or (4). Instead, parts of the current iterate are updated using a vector composed of parts of di erent previous, not necessarily the latest, iterates; cf. [3, Model B], [12, x3], [13, x4]. As is customary in the description and analysis of asynchronous algorithms, the iteration subscript is increased every time any part of the iteration vector is computed; see, e.g., the references in [3], [5], [6], [12]. In a formal way, the sets J f1; 2; ; Lg, i = 1; 2; , are de ned by ` 2 J if the `-th part of the iteration vector is computed at the i-th step. The subscripts r(k; i) are used to denote the iteration number of the k-th part being used in the computation of any part in the i-th iteration, i.e., the iteration number of the k-th part available at the beginning of the computation of x( ) , if ` 2 J . Each n n matrix K can be decomposed into L n n matrices K ( ) , ` = 1; ; L, so X that Ku = K ( ) u, where the nonzeros in K ( ) u correspond to the elements in S in [10, =1 Def. 1.1 (a)]. With this notation, we write the following: Algorithm 3. (Outer Asynchronous Two-stage Multisplitting). Given the ( ) initial vector x0 = x(1) 0 + + x0 For i = 1; 2; `
i
`
i
i
`
`
i
i
i
`
i
i
`
L
`
`
`
`
L
(8)
x
(`) i
8 < =:
x(?)1 H ( )(i) x(1)(1 ) + + x( ( ) `
i
L
`
r
;i
r L;i)
if ` 62 J + Q (i)b if ` 2 J :
with H (i) and Q(i) as de ned in (6) and (7), respectively.
i
(`)
i
i
4
Daniel B. Szyld
For easy comparison with Algorithm 4, we rewrite (8) explicitly as 8 > > < ( ) x => > :
(9)
`
i
x(?)1
if ` 62 J
`
i
E [R ( )x( () ) + `
s `;i
`
`
X?1
s(`;i)
R B?1(N
X
j `
r `;i
L
`
`
j =0
x( () ) + b)] if ` 2 J : k
i
r k;i
k =1
i
We always assume that the asynchronous iterations satisfy the following conditions. They are very natural in asynchronous computations; see the explanation in [12, x3].
r(`; i) < i for all ` = 1; ; L; i = 1; 2; lim r(`; i) = 1 for all ` = 1; ; L: !1 The set fi j ` 2 J g is unbounded for all ` = 1; ; L: Theorem 2.1. Let A?1 0. Let the splittings (1) be regular and the splittings (3) be weak regular. Assume that the sequence r(`; i) and the sets J , ` = 1; ; L, i = 1; 2; ,
(10) (11) (12)
i
i
i
satisfy conditions (10){(12). Then, the Outer Asynchronous Two-stage Multisplitting Algorithm 3 converges to x with Ax = b for any initial vector x0 and for any sequence of numbers of inner iterations s(`; i) 1, ` = 1; ; L, i = 1; 2; Proof. The proof follows in the same way as the proof of [12, Theo. 3.3] which in turn is based on [5, Theo. 3.4]. Theorem 2.2. Let A be an H -matrix. Let the splittings (1) be H -splittings and the splittings (3) be H -compatible splittings. Assume that the sequence r(`; i) and the sets J , ` = 1; ; L, i = 1; 2; , satisfy conditions (10){(12). Then, the Outer Asynchronous Two-stage Multisplitting Algorithm 3 converges to x with Ax = b for any initial vector x0 and for any sequence of numbers of inner iterations s(`; i) 1, ` = 1; ; L, i = 1; 2; Proof. The proof follows in the same way as the proof of [12, Theo. 3.4]. Theorems 2.1 and 2.2 extend [3, Theo. 2.2] to more general splittings and to H -matrices. We consider now asynchronous two-stage multisplittings algorithm where, at each inner iteration, the most recent information from the other parts of the iterate is used. In other words, the parts x( ( ) ) in (9) may di er for di erent values of j , j = 0; ; s(`; i) ? 1 (` 2 J ). To re ect this, we therefore use indices of the form r(k; j; i). These algorithms are called totally asynchronous to distinguish them from the outer asynchronous ones; see [12, x4] for a discussion of possible advantages of these methods. Algorithm 4. (Totally Asynchronous Two-stage Multisplitting). Given ( ) the initial vector x0 = x(1) 0 + + x0 For i = 1; 2; ?
?
i
?
?
k
r k;i
i
L
x
(`) i
8 > > < => > :
x(?)1 E [R ( )x( () 0 ) + s `;i
`
`
`
X?1
s(`;i)
R B?1 (N
j =0
X L
j `
r `; ;i
Analogous to (10){(11) we now assume (13)
if ` 62 J
`
i
`
` k =1
x( () k
r k;j;i)
+ b)] if ` 2 J :
8 < r(k; j; i) < i; for all k = 1; ; L; j = 0; ; s(k; i) ? 1; min r(k; j; i) = 1; for all k = 1; ; L: : lim !1 =0 ( )?1 i
j
;
i
i
i = 1; 2; ;
;s k;i
Theorem 2.3. Let A?1 0. Let the splittings (1) be regular and the splittings (3) be
weak regular. Assume that the numbers r(k; j; i) and the sets J , k = 1; ; L, i = 1; 2; , i
Asynchronous Two-stage Multisplitting Methods
5
satisfy conditions (12){(13). Then, the Totally Asynchronous Two-stage Multisplitting Algorithm 4 converges to x with Ax = b for any initial vector x0 and for any sequence of numbers of inner iterations s(`; i) 1, ` = 1; ; L, i = 1; 2; Proof. The proof follows in the same way as the proof of [12, Theo. 4.3] which in turn is based on [6, Theo. 2.1]. Theorem 2.4. Let A be an H -matrix. Let the splittings (1) be H -splittings and the splittings (3) be H -compatible splittings. Assume that the numbers r(k; j; i) and the sets J , k = 1; ; L, i = 1; 2; , satisfy conditions (12){(13). Then, the Totally Asynchronous Two-stage Multisplitting Algorithm 4 converges to x with Ax = b for any initial vector x0 and for any sequence of numbers of inner iterations s(`; i) 1, ` = 1; ; L, i = 1; 2; Proof. The proof follows in the same way as the proof of [12, Theo. 4.4]. ?
?
i
?
?
References [1] A. Berman and R. J. Plemmons, Nonnegative Matrices in the Mathematical Sciences, Academic Press, New York, third ed., 1979. Reprinted by SIAM, Philadelphia, 1994. [2] R. Bru, C. Corral, and J. Mas, Multisplitting preconditioners based on incomplete Choleski factorizations. Preprint, Departamento de Matematica Aplicada, Universidad Politecnica de Valencia, 1993. [3] R. Bru, L. Elsner, and M. Neumann, Models of parallel chaotic iteration methods, Linear Algebra and its Applications, 103 (1988), pp. 175{192. [4] R. Bru and R. Fuster, Parallel chaotic extrapolated Jacobi method, Applied Mathematics Letters, 3 (1990), pp. 65{69. [5] M. N. El Tarazi, Some convergence results for asynchronous algorithms, Numerische Mathematik, 39 (1982), pp. 325{340. [6] A. Frommer, On asynchronous iterations in partially ordered spaces, Numerical Functional Analysis and Optimization, 12 (1991), pp. 315{325. [7] A. Frommer and G. Mayer, Convergence of relaxed parallel multisplitting methods, Linear Algebra and its Applications, 119 (1989), pp. 141{152. , On the theory and practice of multisplitting methods in parallel computation, Computing, [8] 49 (1992), pp. 63{74. [9] A. Frommer and B. Pohl, A comparison result for multisplittings based on overlapping blocks and its application to waveform relaxation methods, Research Report 93-05, Eidgenossiche Technische Hochschule, Seminar fur Angewandte Mathematik, Zurich, May 1993. To appear in Numerical Linear Algebra with Applications. [10] , Comparison results for splittings based on overlapping blocks, in Proceedings of the Fifth SIAM Conference on Applied Linear Algebra, SIAM, Philadelphia, 1994. [11] A. Frommer and D. B. Szyld, H -splittings and two-stage iterative methods, Numerische Mathematik, 63 (1992), pp. 345{356. [12] , Asynchronous two-stage iterative methods, Numerische Mathematik, 69 (1994), pp. 141{ 153. [13] R. Fuster, V. Migallon, and J. Penades, Asynchronous parallel extrapolated methods. Preprint, Departamento de Tecnologa Informatica y Computacion, Universidad de Alicante, 1993. [14] C.-M. Huang and D. P. O'Leary, A Krylov multisplitting algorithm for solving linear systems of equations, Linear Algebra and its Applications, 194 (1993), pp. 9{29. [15] M. T. Jones and D. B. Szyld, Two-stage multisplitting methods with overlapping blocks, Research Report 94-31, Department of Mathematics, Temple University, Philadelphia, March 1994. Also Available as Technical Report CS-94-224, Computer Science Department, University of Tenessee, Knoxville, March 1994. [16] P. J. Lanzkron, D. J. Rose, and D. B. Szyld, Convergence of nested classical iterative methods for linear systems, Numerische Mathematik, 58 (1991), pp. 685{702. [17] I. Marek and D. B. Szyld, Splittings of M -operators: Irreducibility and the index of the iteration operator, Numerical Functional Analysis and Optimization, 11 (1990), pp. 529{553.
6
Daniel B. Szyld
[18] V. Migallon and J. Penades, Metodos caoticos extrapolados en paralelo basados en multiparticiones. Preprint, Departamento de Tecnologa Informatica y Computacion, Universidad de Alicante, 1993. In Spanish. [19] M. Neumann and R. J. Plemmons, Convergence of parallel multisplitting methods for M matrices, Linear Algebra and its Applications, 88-89 (1987), pp. 559{574. [20] D. P. O'Leary and R. E. White, Multi-splittings of matrices and parallel solution of linear systems, SIAM Journal on Algebraic and Discrete Methods, 6 (1985), pp. 630{640. die Determinanten mit uberwiegender Hauptdiagonale, Comentarii [21] A. M. Ostrowski, Uber Mathematici Helvetici, 10 (1937), pp. 69{96. [22] H. Schneider, Theorems on M -splittings of a singular M -matrix which depend on graph structure, Linear Algebra and its Applications, 58 (1984), pp. 407{424. [23] X. Sun, Convergence of parallel multisplitting chaotic iterative methods, Numer. Math. J. Chinese Univ., 14 (1992), pp. 183{187. In Chinese. [24] D. B. Szyld and M. T. Jones, Two-stage and multisplitting methods for the parallel solution of linear systems, SIAM Journal on Matrix Analysis and Applications, 13 (1992), pp. 671{679. [25] R. S. Varga, Matrix Iterative Analysis, Prentice-Hall, Englewood Cli s, New Jersey, 1962. [26] , On recurring theorems on diagonal dominance, Linear Algebra and its Applications, 13 (1976), pp. 1{9. [27] R. E. White, Multisplittings and parallel iterative methods, Computer Methods in Applied Mechanics and Engineering, 64 (1987), pp. 567{577. , Multisplitting of a symmetric positive de nite matrix, SIAM Journal on Matrix Analysis [28] and Applications, 11 (1990), pp. 69{82. [29] D. M. Young, Iterative Solution of Large Linear Systems, Academic Press, New York, 1971.
