Decomposition of a symmetric matrix

Author's personal copy

Numer. Math. 27, 95--t09 (t976) 9 by Springer-Verlag 1976

Handbook Series Linear Algebra Decomposition of a Symmetric Matrix* James R. Bunch, Linda Kaufman and Beresford N. Parlett Received October 6, t975

Summary. An algorithm is presented to compute a triangular factorization and the inertia of a symmetric matrix. The algorithm is stable even when the matrix is not positive definite and is as fast as Cholesky. Programs for solving associated systems of linear equations are included. 1. Theoretical Background A real symmetric matrix A usually possesses a unique triangular factorization

A = M D M t,

(t)

where M is unit lower triangular, M t denotes the transpose of M, and D is diagonal. However this factorization does not always exist and, what is worse, it m a y exist and be hopelessly unstable in the sense that the intermediate quantities I m~tdi], i ~ j, m a y be arbitrarily greater than the elements of A. The use of interchanges to produce P~A P = M D M t (2) where P is a permutation matrix, removes some but not all of the troublesome cases. The simplest recalcitrant example is

These difficulties have nothing to do with the condition number of A for inversion, namely IIA I1" II1-111It is interesting that an adequate degree of stability can be maintained b y accepting the small increase in complexity that comes from allowing D above to be block diagonal with blocks of order I or 2. A procedure parsymdec for obtaining the faetorization of the form (2) with block diagonal D is presented in this contribution. The algorithm is easily modified to accept complex Hermitian or complex symmetric matrices. The method described here is the latest in a sequence which can be traced in the articles Parlett and Reid [9J, Bunch and Parlett [6J, Bunch [4J, Aasen [t ], * The authors gratefully acknowledge research support from N S F grant MCS 75-065 t 0, NSF grant MCS75-23333, and ONR contract NO0014-69-A-0200-t0t7, respectively

Author's personal copy

96

Handbook Series Linear Algebra:

Bunch and Kaufman [5]. All these methods are special variants of the familiar triangular decomposition (t) which are designed both to preserve symmetry and to prevent harmful element growth. Crucial points in the implementation of these algorithms are (a) the bounds on the Im~idit , i ~ ] , (b) the computing time required for the factorization, (c) the form of the permutation P, and (d) the storage requirements for P, M, and D. There is an additional constraint that the algorithms should be no slower than those in [3] which solve A x = b without maintaining symmetry. Block factorization techniques for symmetric A begin with A (=A In)), and produce a sequence of reduced matrices {A IO} as described below. Note that the order of A l0 is i. At the typical step a permutation of A IO is made and written in partitioned form as

P(O'A(O P(')= [X

~] ,

where the pivot X is s • s and nonsingular with s-----l or 2. The next reduced matrix is defined by A (i-,) = y _ C X - 1C*. (3) The integer s is chosen to be 2 whenever it is predicted that this will cause less growth in the elements of A ~i-s) than would the choice s = t . The goal is to choose p(O as simply as possible consistent with a modest bound on element growth at each step. Our strategy depends on a parameter of 0 < a < t, an appropriate value for which is given below. In order to simplify the description of the algorithm let Ira, denote the permutation matrix obtained by interchanging rows m and r of the identity matrix I. The reduction of A (~ to A (~-') is as follows: (i) Determine )t = ] a~[ = ,r~a~i [ a~i)l[. (it) If taint > : ~ then take p r O = i , s - - i , go to (vii). (iii) Determine a = max [a~)], k 4:r. l < k < i

(iv) (v) (vi) (vii)

"

"

If ]a[~[a>o~ ~ take P(')=I, s = t , and go to (vii). If[ a~9[ > a a the take P ( ' ) = I i , , s = l , and go to (vii). Take P ( 0 = I t , , s = 2 , and go to (vii). Compute A (~-') according to ()) and decrement i by s.

The following remarks explain the structure of the algorithm. Let #(0= max [aJ~)[ I", k = t . . . . . i. Note that 2 _ a = < / z (0. Case t : s = t. After permuting A {0, a(i-1) -7"--1, k - - 1 - -

~(O

a(~),,i0t~(0

t a l a - - --71 t~k 1 I~+II,

whence /~(~-i) ~ u(0 4- J;t/0t if (it) holds --~- --(a/o~ if (iv) or (v) holds

0, v > 0). The algorithm was designed for this case. Nevertheless the procedures unsymdet and unsymsol in [3 ] can solve A x = b stably in approximately n313 multiplications and n ~ storage by sacrificing symmetry. Rival techniques must not spend so much time choosing and executing interchanges that they exceed this operation count. Parsymdec requires n3/6 multiplications. Singular Case (~ > 0). For the inertia problem it is important that parsymdec can accept singular matrices. 3. Formal Parameter List

3.1. Input to procedure parsymdec a n

the elements of an n • n symmetric matrix A. Only the upper triangular portion need be stored. order of the matrix A.

Output of procedure parsymdec a

elements of the MDM t decomposition of A are stored in its upper triangular portion including the diagonal. Its strictly lower triangular portion is left untouched.

change a vector recording the interchanges performed on A during the computation of the decomposition and block structure of D. 3.2. Input to procedure parsymsol a

elements of the MDM t decomposition of a symmetric matrix A as produced b y parsymdec.

change a record produced by parsymdec of the interchanges performed on b

the A matrix and of the block structure of D. a vector of length n containing the right hand side of the equation

Ax=b. n

order of the matrix A.

Output from procedure parsymsol b

/ail

the solution to the problem A x = b. This is the exit used when A, possibly as a result of rounding errors, is singular.

Author's personal copy Decomposition of a Symmetric Matrix. J. R. Bunch et al.

99

3.3. Input to procedure inertia a elements of the M D M t decomposition of a symmetric matrix A as produced by parsymdec.

change a record produced by parsymdec of the block structure of D. n

order of the matrix A.

Output from procedure inertia. poselg the number of positive eigenvalues of A. negeig the number of negative eigenvalues of A. 4. ALGOL Programs

procedure parsymdec (a, change, n): value n; integer n ; array a, change; c o m m e n t Given a symmetric matrix A of order n, this procedure computes the decomposition I~tAP=MDM t where M is a unit lower triangular matrix. D is a block diagonal matrix with blocks of order t or 2 and M~+I,~=0 when Di+l, ~ is nonzero, and P is a permutation matrix. This procedure uses a partial pivoting scheme to form the decomposition. A is assumed to be stored only in its upper triangular part, M and D are written over A and the diagonal of A will be destroyed. On output the real array change of length n contains a record of the interchanges performed, i.e. the permutation matrix P; begin integer i, j, k, m, ipl, ip2; real det, s, t, alpha, lambda, sigma, aii, alp1, aipli ; procedure interchange(i) ; value i; integer i; c o m m e n t this procedure interchanges row and column j of a and row and column i where i < i, and a is the reduced matrix of order n - - i + 1 ; begin for k : = i + t step 1 until n do begin t : = a ~ ' , k ] ; aEj, k] :----all, k]; aEi, k] : = t end k; for k : = i + t step I until i--1 do begin t :-=aIi, k]; aIi, k] :=a[k, i]; a[k, /'] : = t end k; t

:--a[i,i]; a[i,i] :=a~',i]; a~,i]

end interchange; alpha : = (I + sqrtO 7))/8; change[n] : = n ; i : = t ; reduce:

if i < n then begin

ipl:=i+t; ip2:=i+2; aii :=abs(a[i, i]); change[i] : = i ;

:=t

100

Author's personal copy Handbook Series Linear Algebra:

comment Find the maximum element in the first column of the reduced matrix below the diagonal; lambda :=abs(a[i, ipl]); i : = i p l ; for m : = i p 2 step t until n do if abs(a[i, m]) > lambda then begin 1' : = m ; lambda :-~abs(a[i, m]) end; t : = alpha • lambda; if aii > = t t h e n g o t o onebyone; c o m m e n t Determine the maximum element in the /'th column of the reduced matrix off the diagonal; sigma : = lambda ; for m:----ipl step 1 until j - - I do if abs(a[m, /']) > sigma t h e n sigma :=abs(a[m, /']) ; f o r m : = / ' + t step t until n d o if abs(a[/', m]) > sigma then sigma :-----abs(a[/', m] ; if sigma X aii > = t X lambda t h e n g o t o onebyone; if abs(a[/', i]) > =alpha x sigma then begin c o m m e n t Interchange the ith and/'th row and column and do a t x I pivot; change[i] :=1"; interchange(i); goto onebyone end; c o m m e n t We do a 2 x 2 pivot; if/" > ipl t h e n

begin interchange(ipl) ; t:=a[i,i]; a[i,i]:=a[i, ipl]; a[i, ipl]: =t end; det : =a[i, i] • ipl]--a[i, ipl] T2; c o m m e n t det must be negative; aipli :=a[i, ipl]/det; aii : = a [ i , i]/det; alp1 : --- a lip1, ipl]/det; changeEi] :=/'; change[ipl] :=det; f o r / ' : = i p 2 step 1 until n do begin s :=aipli • xaE i,/']; t :=aiplixa[i,/']--aiixa[ipl,/']; f o r k :=/' step 1 until n do a[/', k] : =a[/., k] +a[i, k] x s + a [ i p l , k] • a[i, i] : = s ; a[ipl, i] :----t

end/.; i : = @ 2 ; goto reduce; o.ebyone: if a[i, i] not equal 0 then begin aii : = a [i, i] ; f o r i : = i p l step t until n d o

Author's personal copy Decomposition of a Symmetric Matrix. J. R. Bunch et al.

t01

begin s :------a[i, ~ ]/aii;

for k : = i step ! until n do a[/', k] : = a[?', k ] + s •

a[i, iJ

k];

:=s

end j end; i : = i p l ; goto reduce end i e n d pa~symdec ;

procedure parsymsol (a, change, b, n,/ail): value n; i n t e g e r n;

array a, b, change; label/ail; c o m m e n t This subroutine uses the decomposition c o m p u t e d in the subroutine

parsyrndec to solve A x - - b where A is a nonsingular s y m m e t r i c m a t r i x of order n and b is a vector of order n. The real vector change of length n is generated in parsymdec. The solution is returned in the vector b. If it is detected t h a t the m a t r i x A is singular, control will pass to the location given b y the l a b e l / a i l ;

begin i n t e g e r ii, i, j, k, ipl; real det, temp, save; c o m m e n t Solve M D y = b

and store y in b;

i:=t;

repeat: if i < n then

begin ipl : = i +1 ; save : =b[change[i]]; if change [ipl] > 0 t h e n begin

b[change [i]] :----b[i]; if a[i, i] = 0 then goto/ail; b[i] : = save~all, i ] ; f o r i : = i p l step t until n d o b[/'] :=b~1"]+a[i, {] • i : = ipl end else

begin temp : = b [ i ] ; b[change[i]] :----blip1]; det : -----change [ipl] ; b [i] : = (temp • a lip1, ipl] -- save X a [i, ipl])/det; blip1] : - - (save xa[i, i] --temp • ipl])/det;

Author's personal copy

t 02

Handbook Series Linear Algebra:

for i : = i + 2

step t until n do b ~'] : = b ~'] + a [i, i] • temp + a [ipl, i] • save;

i:=i+2 end; goto repeat end i; if i = . then

begin

ifa[i, i ] = 0 then goto ]ail; b[i]:=b[il/a[i,i];

i:=n--t

end else i : = n-- 2; comment Now solve M t x = y in b;

for x, where y is stored in the vector b and store x

calc:

if i > 0 then begin if change[i] < 0 t h e n ii : = i - - t f o r k : - ~ i i s t e p I until i d o

e l s e ii :----i;

begin save : = b [k] ; for i:=i+1 s t e p t until n d o save : ----save + a [k, i] • b [l'] ; b [k] : = save

end; b [i] : = b [change [ii]~ ; b [change [ii] ] : = save; i : = i i - - 1 ; g o t o calc end i end parsymsol ;

procedure inertia (a, change, n, poseig, negeig) ; value n; integer n, poseig, negeig ; a r r a y a, change; comment

This procedure uses the decomposition computed by the subroutine parsymdec to compute the inertia of a symmetric matrix A of order n. The real vector change of length n is generated in parsymdec. The procedure returns the number of positive eigenvalues in poseig and the number of negative eigenvalues in negeig.The decomposition in

the A matrix is not affected;

begin integer i; negeig : = poseig : = 0; next: if i < n then

i := t ;

begin if c h a n g e [ i + 1 ] > 0 then

Author's personal copy Decomposition of a Symmetric Matrix. J. R. Bunch et al.

103

begin comment a t • block has been found; if a[i, i] < 0 then negeig : = n e g e i g + t ; if a[i, i] > 0 t h e n poseig :=poseig+ 1 ; i:----i+1

end else begin c o m m e n t a 2 • 2 block has been found; poseig : = p o s e i g + t ; negeig :-~negeig+ 1 ; i : = i + 2

end ; goto next end i; if i---n then begin if a[i, i] > 0 then poseig : =poseig + 1 ; if a[i, i] < 0 then negeig : = n e g e i g + t end end inertia; 5. Organizational and Notational Details a) The procedure parsymdec takes A in conventional form as an n • n array. Without increasing execution time A can be stored as a one dimensional array of length n (n+1)/2. This saves storage but the square array has the advantage of being able to hold M, D, and the strictly lower triangular part of A. The diagonal of A must be saved separately. Snapshots of two possible configurations after two steps of the algorithm are given below.

dll d2l ,---1 -211d22 a31 a32 a41 a42 asz a52 a61 a62

m31 t/~41 m51 m61 ma2 "~42 m52 m62 dz3Im43 n~53 ~v~63 a43 a44 a45 a46 a53 a54 a55 a56 a63 a64 a65 a66

V ~

n~31 tn41 msl m61

a lt d221m32 m42 m52 m62 a31 a321 d33 d43 ~t~53m6z a41 a42 a43 j d44 m54 m64 a51 a52 a53 a54 a55 ~56 a61 a62 a63 6164 a65 a66

Note that m i + z , j d i + l , j = 0 always, so M and D never overlap. b) The derivation of the permutation P from the sequence of interchanges is a standard technique, see Wilkinson C10], p. 206, for more details. Let Pi denote the interchange at i. If a 2 • 2 pivot is used on A (0 then, b y convention, M i_ z = P i - z = I. Also let M -t denote (M-Z) t. The block diagonal matrix D is computed in stages as M ~ z . . . M ~ ~ P**m ~ ~ P~ AP~ M~-' P , M ~ ' . . . M~t_~ = O. This can be rewritten in the form M - 1 P t A P M -t ----D

Author's personal copy

t 04

Handbook Series Linear Algebra:

where

P = PI P2 " . P.-1, M . - l --- M. - ~1- ~ ( PJ _2 ~. - 1 M .-1

(P~-I '

P~-d

.

.

P~ M~-~ P2" P.-1)" .

.

.

The matrices in parentheses are all unit lower triangular.

c) I n the table below c o m p a r i n g three different m e t h o d s for c o m p u t i n g factorizations of s y m m e t r i c matrices, the algorithm comsymdec is the complete p i v o t i n g scheme of Bunch a n d P a r l e t t I6] which searches each A (i) to find its largest element. Method

Restrictions

Choleski comsymdec parsymdec

pos. def. sym. sym.

Multiplications {n a + ~n 2 + {n i,

where n is the order of the matrix. The code choldet 1" was obtained by deleting the d e t e r m i n a n t calculation from choldet 1 [7]. I n Table 2 the matrices are given by

aii=abs(i--j)

for

i4=i,

a,i=t.69.

These matrices are indefinite a n d the D matrices produced b y parsymdec contained a n u m b e r of 2 • blocks. The code unsymdet* was obtained b y deleting the d e t e r m i n a n t calculation from unsyrndet [2? a n d computing inner products in line in single precision. I n order to make fair comparisons unsymsol [2], wlfich uses a m a t r i x for the r i g h t h a n d side(s), was modified to unsymsol*, which stores b in a one dimensional array. Timing comparisons are, of course, dependent on the compiler and machine. Table 1. Execution times on a positive definite matrix Order of Matrix

10 20 40 80

Burroughs6 700 parsymdec and parsymsol

choldet1 and cholsol 1

choldet1" and choIsol 1

0.025 s 0.123 s 0.795 s 5.672 s

0.037 s 0.205 s t.265 s 8.817 s

0.037 s 0.183 s 1.132 s 7.883 s

CDC 6400

10 20 40 80

parsymdec and parsymsol

choldet1" and cholsol 1

0.027 s 0.127 s 0.742 s 4.957 s

0.026 s 0.t26 s 0.777 s

5.277 s

Author's personal copy t06

Handbook T a b l e 2. E x e c u t i o n Order of Matrix

t0 20 40 80

Series Linear

Algebra:

times on an indefinite matrix

Burroughs 6700 parsymdec a n d parsymsol 0.023 0.t27 0.817 6.003

s s s s

unsymdet* ( w i t h o u t d e t e r m i n a n t c a l c u l a t i o n ) a n d unsymsol* 0.047 0.275 t .902 14.093

s s s s

CDC 6400 parsymdec a n d parsymsol 10 20 40 80

0.025 0.117 0.629 4.349

T a b l e 3. R a t i o t o Order of Matrix

10 20 40 8o

s s s s

unsymdet* ( w i t h o u t d e t e r m i n a n t c a l c u l a t i o n ) a n d unsymsol* 0.058 0.297 2.001 13.395

s s s s

parsymdec a n d parsymsol o n t h e B u r r o u g h s 6 7 0 0

parsymdec

choldet 1 a n d cholsol i o n

choldet1" a n d cholsol1 o n a

unsymdet* a n d unsymsol* o n

parsymsol

a positive def. matrix

positive def. matrix

an indefinite matrix

t .00 1.00 t.00 1.00

1.48 1.67 t.59 t.55

t .48 1.49 1.42 1.39

2.04 2.17 2.33 2.35

and

T a b l e 4. R a t i o t o

parsymdec a n d parsymsol o n t h e C D C 6 4 0 0

Order of Matrix

parsymdec

choldet1" a n d cholsol 1 o n a

unsymdet* a n d unsymsol* o n

parsymsol

positive def. matrix

an indefinite matrix

10 20 40 80

t.O0 1.00 t.O0 ! .00

0.96 0.99 1.05 1.06

2.32 2.54 3.18 3.08

and

Author's personal copy Decomposition oI a Symmetric Matrix. J. R. Bunch et al.

tO7

In factoring positive definite matrices one might suppose that pivoting would give parsymdec-parsymsol a slight edge over choldet-cholsol I as regards accuracy. In many cases, such as Example t, this is so. However, in Example 2 the Cholesky method wins. There are two points to be made here. Firstly, all the answers are satisfactorily accurate. Secondly, pivoting is designed solely to keep step by step element growth to a modest level and that will not necessarily enhance accuracy. Let us recall the following general truth. The fact that Method A has a smaller error bound than Method B in no way implies that it produces smaller errors. In fact the diagonal pivoting method usually produces slightly better approximations than Gaussian elimination with partial pivoting. All examples were run on tile CDC 6400 with machine precision 2 -4~ ~ 71o--t 5.

Example 1: Ma~ix 4 4 24 40 --24

4 7 45 t3 --39

24 45 296 91 --289

40 t3 91 964 --420

--24 --39 --289 --420 572

True Solution

Right Hand Side

--7 --2 --1 --4 9

--436 --490 --35t9 --8033 7363

Computed Solutions:

parsymdec and parsymsol

choMetl* and cholsoll

--6.9999999999964 --2.0000000000075 --0.99999999999910 --4.0000000000001 9.0000000000000

--6.9999999969468 --2.0000000078596 --0.99999999900285 --4.000000000t073 9.0000000000173

Example 2: Matrix 1

--2

3

7

--9

--2 3 7 -9

8 -6 2 50

--6 18 -15 -18

2 -15 273 173

50 -t8 173 t667

True Solution

Right Hand Side

--6 --5 --8 5 --7

78 --320 --81 222 --10856

Computed Solutions:

parsymdec and parsymsol --5.9999999998895 --4.9999999999870 --8.0000000000170 4.9999999999959 --6.9999999999995

choldet 1" and cholsol g --6.0000000000000 -- 5.0000000000000

--8.0000000000000 5.0000000000000 --7.0000000000000

Author's personal copy

t08

Handbook Series Linear Algebra:

Example 8: Mat~x -3 -3 -3 -t8 -30 t8

-t -4 -48 8

-18

-30

-4 -6 -274 6

-48 -274 tt9 t9

t8 8 6 t9 2t6

Computed Solutions: parsymdec and parsymsol --6.9999999999914 --2.0000000000214 --0.99999999999727 --4.0000000000003 9.0000000000000

True Solution

Right Hand Side

--7 --2 --1 --4 9

327 29t 1290 275 t720

unsymdet* and unsymsol* --6.9999999922446 --2.0000000207192 --0.99999999732927 --4.0000000002640 9.0000000000702

As determined by inertia, this matrix has 3 positive and 2 negative eigenvalueq

Example 4: Matrix --4 0 --t6 --32 28

0 1 5 t0 --6

True Solution --16 5 --37 --66 64

--32 10 --66 --85 53

28 --6 64 53 --t5

--8

Right Hand Side 448

-3

-1ti

--2 --5 8

1029 1207 --719

Computed Solutions:

parsymdecand parsymsol

unsymdet* and unsymsol*

--8.0000000015746

--7.9999999993815 --2.9999999992122 --2.00O0000002219 --1.9999999994367 --5.0000000000716 --4.99999999997tt 8.0000000000152 7.9999999999945 As determined by inertia, this matrix has 2 positive and 3 negative eigenvalues.

--3.0000000020098

References

1. Aasen, J. O. : On the reduction of a symmetric matrix to tridiagonal form. BIT 11, 233-242 (1971) 2. Barwell, V., George, A. : A comparison of algorithms for solving symmetric indefinite systems of linear equations. TOMS 2, 242-251 (1976) 3. Bowdler, H. J., Martin, R. S., Peters, G., Wilkinson, J . H . : Solution of real and complex systems of linear equations. Numer. Math. 8, 2t 7-234 (1966) 4. Bunch, J. R. : Analysis of the diagonal pivoting method. SIAM Numerical AnMysis 8, 656-680 (t971) 5. Bunch, J. R., Kaufman, L. : Some stable methods for solving symmetric linear systems. Math of. Comp.. (to appear), t977. 6. Bunch, J.R., Paflett, B. N. : Direct methods for solving symmetric indefinite systems of linear equations. SIAM Numerical Analysis 8, 639-655 (1971)

Author's personal copy Decomposition of a Symmetric Matrix. J. R. Bunch et al.

t09

7. Cottle, R. W.: Manifestations of the Schur complement. Linear Algebra and its Applications 8, t 89-2t I (1974) 8. Martin, R. S., Peters, G., Wilkinson, J. H. : Symmetric decomposition of a positive definite matrix. Numer. Math. 7, 362-383 (1965) 9. Parlett, B. N., Reid, J. K. : On the solution of a system of linear equations whose matrix is symmetric but not definite. BIT 10, 386-397 (1970) 10. Wilkinson, J. H.: The Algebraic Eigenvalue Problem. Oxford: Clarendon Press 1965 James R. Bunch Department of Mathematics University of California San Diego USA Linda Kaufman Bell Laboratories Murray Hill New Jersey 07974 USA Beresford N. Parlett Department of Mathematics and Department of Electrical Engineering and Computer Science University of California Berkeley USA