Solving linear least squares problems by Gram-Schmidt ... - CiteSeerX

BIT 7 (1967), 1--21

SOLVING L I N E A R LEAST SQUARES PROBLEMS BY GRAM-SCHMIDT ORTHOGONALIZATION AKE BJORCK Abstract. A general analysis of the condit4on of the linear least squares problem is given. The influence of rounding errors is studied in detail for a modified version of t h e G r a m - S e h m i d t orthogonalization to obtain a faetorization A = Q R of a given ra x n m a t r i x A, where R is upper triangular a n d QTQ = L L e t ~c be t h e vector which minimizes lib-Azcll, and r = b - A x . I t is shown t h a t if inner-products are accumulated in double precision then the errors in the computed x and t are less t h a n the errors resulting from some simultaneous initial perturbation ~A, ~b such that [I~AIIs/IIAIIE ~ []6bll,/[[b[l, ~ 2"n 3/2 machine u n i t s . No reorthogonalization is needed and the result is independent of the pivoting strategy used. K e y words: Least square, linear equations, orthogonalization.

1. Introduction. L e t A b e a g i v e n m x n r e a l m a t r i x of r a n k n, m > n, a n d b a g i v e n m x 1 real vector. Then there exists a unique vector x which solves the l e a s t s q u a r e s p r o b l e m of m i n i m i z i n g

ltb - A x l t ~

•

I t is w e l l k n o w n t h a t t h e s o l u t i o n x s a t i s f i e s t h e c o n d i t i o n A~'(b-Ax)

= O,

(1.I)

i.e. t h e r e s i d u a l v e c t o r r = b - A x is o r t h o g o n a l t o t h e c o l u m n s of A . It follows that we can compute x from the normal equations ATAx

= Arb.

(1.2)

W e d e f i n e f o l l o w i n g B a u e r [1] t h e ( c o l u m n w i s e ) c o n d i t i o n of t h e r e c t angular matrix A to be c o n d ( A ) = m a x []Axll/min []Ax][. I~1=I II~i=I T h e c o n d i t i o n n u m b e r c o r r e s p o n d i n g t o t h e L 2 - n o r m is d e n o t e d b y n(A). In section 7 we will show that under some restrictions, z(A) can be con-

2

AKE BJORCK

sidered as an approximate condition number for the problem of determining x. 1 On the other hand u(ATA)

--

~2(A) .

This shows t h a t in general using t-digit binary arithmetic, we will not be able to obtain even an approximate solution to (1.2) unless z(A) < 2t/2 . Now let B = A S , where S is square and non-singular. Then from (1.1) follows B T ( b - A x ) = 0 and thus the equations BTAx

= B~rb

(1.3)

can be used instead of the normal equations. A natural question to ask is if we can chose B in such a way t h a t u(BrA) = ~(A).

(1.4)

This is indeed possible. Since the columns of A are linearly independent we have a faetorization of A A = QR

(1.5)

where QTQ = I and R is an upper triangle. Moreover, each row of R and each column of Q is uniquely determined up to a scalar factor of modulus one ([4] p. 7). Choosing B =AR-1 = Q the matrix B T A = ~T~/~_--R is triangular. The equations (1.3) are thus easily solved by back-substitution. Further the condition (1.4) is satisfied since ~(R) = ~ ( Q R ) = ~ ( A ) .

(1.6)

This factorization can be obtained mainly in two different ways, and both have been proposed for solving least squares problems. One way is to annihilate the subdiagonal elements of A by a sequence of elementary Hermitian matrices H (k) = ( I - 2WkWkT),

k = 1, 2 . . . . . n ,

II(n)...H(~)H(1)A = Q T A = R .

A detailed error estimate for this algorithm has been given by Wilkinson [7] pp. 152-161, and the application to least squares problems is treated by Golub in [2]. A second way to obtain the factorization is by applying the G r a m Schmidt orthogonalization procedure to the columns of A. This is however known to have poor numerical stability. 1 T h e r e a d e r m a y p r e f e r to r e a d t h i s s e c t i o n a t t h i s s t a g e .

SOLVING

LINEAR

LEAST

SQUARES

PROBLEMS

...

In this paper we will use a slightly modified version of the GramSehmidt procedure, which is equivalent to the elimination with weighted row combinations in [1]. This will be applied to the solution of linear least squares problems. The propagation of rounding errors will be studied in detail and in particular an estimate of the deviation from orthogonality of the computed Q and an error bound for the computed solution will be derived.

2. Description of the algorithm. In the actual computation it is preferable to use a different normalization of the factors in (1.5). We avoid computing square roots if we take instead A = Q'R' (2.1) where R' is a u n i t upper triangle and (Q,)TQ, diagonal. For completeness we give below the algorithms both for the classical and the modified Gram-Schmidt procedure. In both cases we compute a sequence of matrices A = A ( ~ ) , A ( 2 ) , . . . , A (n+~) = Q ' . Classical Gram-Schmidt :

Here the elements of R' are computed one column at a time. We define A(k) =

(ql',

• •, q ~" l ,

•

ak, "

"', an)

and assume (q , ) r q , = ~ , d ~ ,

I