SIGNAL
PROCESSING ELSEVIER
Signal
Processing 52 (1996) 65-73
A new algorithm for the factorization and inversion of recursively generated matrices Manuel D. Ortigueira ISTfINESC. R. Alves Redol, 9. 2”. 1000 Lisboa, Portugal Received
1 July 1994; revised 11 May 1995, 4 October
1995 and 13 March 1996
Abstract
In this paper we present a recursive algorithm for factorization and inversion of matrices generated by adding dyads (elementary or rank-one matrices) as it happens in recursive array signal processing. The algorithm is valid for any recursively generated rectangular matrix and it has two parts: one valid for rank-deficient and the other for full rank matrices. The second part is used to obtain a generalization of the Sherman-Morrison algorithm for the recursive inversion of the covariance matrix. From the proposed algorithm we derive two others: one to compute the inverse (pseudo-inverse) of any matrix and the other to invert simultaneously two matrices. Zusammenfassung
In diesem Beitrag prbentieren wir einen rekursiven Algorithmus fiir die Faktorisierung und Inversion von Matrizen, die durch die Addition von Dyaden (Elementar-Matrizen oder Matrizen mit dem Rang eins) generiert werden, wie dies bei der rekursiven Array-Signalverarbeitung geschieht. Der Algorithmus ist fiir jede rekursiv generierte Rechteck-Matrix giiltig und besteht aus zwei Teilen: der eine gilt fiir Matrizen mit nicht vollem Rang und der andere fiir Matrizen mit vollem Rang. Der zweite Teil wird benutzt, urn eine Verallgemeinerung des Sherman-Morrison Algorithmus fiir die rekursive inversion der Kovarianz-Matrix zu erhalten. Aus dem vorgeschlagenen Algorithmus leiten wir zwei weitere ab: einer berechnet die Inverse (Pseudo-Inverse) jeder beliebigen Matrix und der andere invertiert gleichzeitig zwei Matrizen.
On prBsente dans cet article un algorithme rCcursif pour la factorisation et l’inversion de matrices gCntrCes par l’addition de diades (matrices Clkmentaires ou de rang unitaire), comme dans le traitement de signal matriciel rCcursif. L’algorithme convient pour toute matrice rectangulaire g&ntrCe rt.cursivement et contient deux parties: une pour les matrices qui ne sont pas de plein rang et l’autre pour les matrices de plein rang. La seconde partie est consacr6e g l’obtention d’une g&nCralisation de l’algorithme de Sherman-Morrison pour l’inversion r6cursive de la matrice de covariance. Deux autres algorithmes sont d&iv&s B partir du premier: l’un pour calculer l’inverse (ou le pseudo-inverse) de n’importe quelle matrice, et l’autre pour inverser simultantment deux matrices. Keywords:
Dyad; Householder matrix; Gauss-Jordan
transformation;
SOl65-1684/96/$15.00 c(( 1996 Elsevier Science B.V. All rights reserved PII SOl65-1684(96)00062-X
Pseudo-inverse
66
M.D. Oriigueira J Signal Processing 52 (1996) 65-73
1. Introduction The increasing application of the multisensor array techniques and adaptive systems led to the development of some recursive algorithms for matrix analysis [l, 81. The most famous one is the Sherman-Morrison algorithm [l, 121 used for the recursive computation of the covariance matrix inverse. This goal was also the first motivation for the development of the proposed algorithm. In adaptive filtering we need frequently to invert the covariance matrix, for instance, in the computation of the Wiener filter [2]. However, in the computation of the covariance (or its inverse) there is a matrix multiplication. In fact, this is the cause of the bad numerical performance of the Sherman-Morrison algorithm. On the other hand, the Sherman-Morrison algorithm is valid for regular matrices only. In practice, the covariance matrices may become singular, needing either to sum ‘noise’ by adding a positive diagonal matrix or to execute a periodic reinitialization. In the latter case, the reinitialization is accomplished by means of a matrix which is proportional to the identity instead of the null matrix. Besides, a factored form of the autocovariance (or inverse) matrix is enough in most practical situations. In developing the algorithm we present here, we had in mind replacing the Sherman-Morrison algorithm by one with better numerical behavior. We intended to (a) get a more stable algorithm, valid for any matrix and (b) avoid the matrix multiplication. The algorithms for the inversion came as a logical sequence. We discovered later that the algorithms were suitable to derive a recursive algorithm for the implementation of the propagator [4]. In previous works [S-S] we presented some results concerning the recursive processing using rank-one matrices (dyads [lo]). There, we showed how to perform the eigendecomposition or the SVD of recursively generated matrices. However, those algorithms are very heavy from a computational point of view. On the other hand, there are algorithms that deal with other decompositions of the covariance matrix [ll]. The algorithm we present here is suitable for array signal processing where we use covariance matrices, but without needing to compute it explicitly. For example, in
computing the Capon estimate we need to compute a quadratic form such as aHR- ‘a. With the algorithms presented here, that form is converted into a product of the type bHb = (P- 1a)“. (P- ’ a), where P and P-l are computed directly from the snapshots, without forming R or R- ‘. In the rest of the paper we shall consider complex matrices that can be considered as sums of dyads: i1
ui’wY,
(1)
where vi and wi are complex vectors assumed to be non-null and with dimensions N and M (N > M), respectively. Without loss of generality, we can deal with matrices generated accordingly to R, = a.R,_l
+ b.v,,wt,
n = 1,2, . . . ,
a, b > 0,
(2)
which is more suitable for array processing or adaptive processing applications. The proposed algorithm is based on the properties of the dyads. We review some of them in Appendix A. As we said the algorithm has two branches: the one presented in Section 2 is suitable for the factorization/inversion of rank-deficient rectangular matrices; the other described in Section 3 is valid for full rank matrices. The first branch of the algorithm is useful to (pseudo) invert simultaneously two matrices. In particular, we present in Section 4 a new inversion algorithm for any matrix provided that it is constructed by joining columns. Some simulation results are presented in Section 5.
2. Factorization/inversion of rank-deficient recursive matrices In the following we shall be considering the simultaneous factorization and inversion of recursively generated matrices, say R = RL, obtained by adding L dyads: R,=a*R,_l+b-v,wr,
n=l,2
,...,
L,
(3)
where u,, and w, are vectors with dimensions N and M, respectively, and a, b > 0. In Array Signal Processing, R, is a covariance matrix. As the inverse of the covariance matrix appears frequently in the
68
M.D. Ortigueira J Signal Processing 52 (1996) 65- 73
that we obtained a factorizaton which can be written as
of a matrix, R,
R = P. [JDQ],
& = (17)
where P and [DQ] are regular matrices and [JDQ] is an N x M full rank matrix. The pseudo-inverse, R-, of R is given by R- =
[Q-‘D-‘JT]P-l.
(18)
If R is square, J becomes the identity matrix as the matrix JDQ becomes regular. In symmetrical case, D is positive and we can write R = (PD”‘)(PD”‘)“.
matrices
(21)
where P,, is a regular square matrix and Qna full rank N x M matrix.
Next, we will prove this theorem and simultaneously show how to compute the factors P, and Qn and their (pseudo-)inverses. Assuming that R,_, =Pn_I’Qn_I, we have R, = P;Q,
= a.P,,_l.Q,,_l
= P,-l[a.Z+
+ b.v;wr
b.xn.yf]Qn-l,
(22)
[a.Z + b.x;yz]
In the previous section we proposed a means for the factorization/inversion of rank-deficient recursively generated matrices. As soon as the matrices stop to be rank deficient, we must follow another procedure. This procedure we are going to present was discovered first in the Hermitian case when trying to find an alternative to the Sherman-Morrison algorithm and readily extended to the nonHermitian case [S]. Here, we present the general version valid for rectangular matrices, too. We would like to obtain a factorization in the form (19)
with P and Q being regular matrices. However, we could not find a recursive algorithm with that feature. In what follows, we will incorporate the matrix J into Q; so, our factorization will be R =P.Q.
f’n*Qn,
with X, and yn as in (8), but with the pseudo-inverse of Qn_ 1 instead of the inverse. Using the results of Theorem A.2, we obtain
3. Factorization/inversion of regular recursive
R = P.J.Q,
these conditions, R, is factorable in the form
(20)
The procedure can be stated in the following theorem.
= H,,*[a.Z +
[email protected],)e,e:]
*H,,
k < N,
(23)
with H, being an N x N Householder matrix given by (A.3). Then, from (22) and (23) we get 4, =P~-IH~~~,~KQ,-I,
(24)
with’ A,, = [a.Z + b.(jf.x,)e,el].
(25)
To obtain (21) we just have to make P,, = P,,_l.H,,.A,
and
Qn = H,,*Qn_l.
(26)
It is now a simple matter to obtain the (pseudo) inverse of R,: R, = Qn .P,-‘,
(27)
with Q; = Q;_ lH, Note that use A,‘12 procedure ment and
and
Pi ’ = Ai lH,,P,=‘l.
(28)
in the Hermitian case, it is preferable to instead of A,, and make Q,, = c. The we just described is very easy to impleit is not computationally heavy, because
Theorem 3.1. Let R, be an N x M matrix that verifyes the formation law of Eq. (3), where v, and w, are two vectors of dimensions N and M. Also, assume that RO = PO. Qo, with PO invertible square (N x N) matrix and Q, is a full rank N x M matrix. Under
’ If ($.x.) = - b/a the matrix A. becomes singular and we must choose k = N and switch to the algorithm of the previous section.
M.D. Ortigueira / Signal Processing 52 (1996) 65-73
69
we do not need to compute the Householder matrices explicitly. The Sherman-Morrison algorithm can readily be obtained from the just presented results, as described next.
which is a generalized version of the usual Sherman-Morrison algorithm. This is obtained for square matrices with a = b = 1. This algorithm is valid only for full rank matrices. It seems not to be possible to generalize it for rank-deficient matrices.
4. Consequences of the previous results
4.2. Simultaneous inversion of two matrices
4. I. The generalized Sherman-Morrison algorithm
The algorithm presented in Section 2 gives us the possibility of simultaneously (pseudo)-invert two matrices with suitable dimensions, N x M and M x K, with K d N < M. This interesting result can be obtained easily. First, remark that the product of two matrices V and W can be written as a sum of dyads:
The ordinary Sherman-Morrison algorithm is a valid procedure for the recursive inversion of square regular matrices. It can readily be obtained from the results of Section 3. However, these results are valid for full rank rectangular matrices. So, we can generalize the usual Sherman-Morrison algorithm by extending its validity to include the rectangular matrix case. We begin by noting that the inverse of the diagonal matrix A,, given by (25) is
II a.
V,NXM)~MXK) = [VI
v2
v3
I
(29)
So, the product H, A, ‘H;, transforms into HJ,‘H,
=
AZ_ “bf*xn)la fhek eTf-6, a a + b.(_y:.x,,)
(34) (30) As it is easily deduced from (lo), the matrix TK can be factorized:
and, using the results of Theorem A.2, H,A,‘H,
zz
AZa
bJa
(31)
a + b.@!f.x.)X”‘Y”
This relation, together with (27) allows us to write 0
R, = Q;_ 1 IIa
a + b.(yy.x,)x”‘Y”
”
1
Z’;Al
(32)
TK = Dv. J.D$, where Dv is a rank M diagonal matrix that depends only on V, and D$ is also a rank K diagonal matrix depending only on IV. We conclude that W+‘=QK1.(D$)-,J.D;Pil.
W- = Qi’.(D$)-
= IR,l a
-
bla R, a + b.(vf;I.xn)
V- = JMXN.D;PN1. 1v, . wf;‘R,
(36)
(37)
,
4.3. Inversion of any matrix
and, at last, =lR,, a
.JKxM
and
or, using (8)
R,,
(35)
As J is a K x N matrix, JKxN, it can be considered as the product JK X,,, = JK XMJM XN. Then
and obtain successively
R,,
Vhf1
-
bla a+b.(w!f.R;_lv,)
R,--lv,.wf;lR,l, (33)
The algorithm presented in the previous section may be used for the computation of (pseudo)-inverse of any matrix, VNXM. We only have to assume
70
M.D. Ortigueira / Signal Processing 52 (1996) 65-73
it as being constructed by joining columns: M
c
v=
u&T
i=l
In this case, we obtain the factorization V=P.D*J,
(39)
where J is an M x N matrix and the pseudo-inverse A-
J=D-.p-'.
=
(40) 13’
5. Simulation results
0
I
ml
1OJl
snapshot
In order to make a comparison of the proposed and Sherman-Morrison algorithms we simulated a 6 sensor array where two sinusoidal waves impinged on. White noise was added to each snapshot. We generated 2000 snapshots that we used to compute the autocovariance matrix. Both algorithms were used to compute R-‘. For every ten snapshots we computed the following error matrices for both algorithms: EN = P-‘R(P-‘)H
-Z
and
and the corresponding ment: &N =
log,,
-
-
cj2
=
-
Fig. 1. Recursive inversion of the covariance matrix. Signal to noise ration 0 dB.
ESM=RelR-z
1
6-
and
PFPGH)
%H
Xca
log-error per matrix ele-
PF@N)
[
1500
log,, [ 62 1 ’
where PF is the Frobenius norm. The matrix R was initialized with I. In Figs. 1 and 2 we show the results we obtained using a signal to noise ratio of 0 and 40 dB, respectively. As it can be seen, the new algorithm is clearly better. Naturally, both become worse with the decrease of the noise variance, since the ‘noise’ eigenvalues become smaller and the matrix is near to be singular. To illustrate the performance of the algorithm proposed in Section 4.3, we ran it together with another one based on the Moore-Penrose pseudoinverse: A- = (AHA)-‘A. The elements of matrix A were samples of a sum of two sinusoids in Gaussian noise. The signal to noise ratio was 20 dB. We
4-
snapshol Fig. 2. Recursive inversion of the covariance matrix. Signal to noise ratio 40 dB.
ran the algorithms for N x M matrices with N=2,3, . . . . 16 and M = 2, . . . , N. As before, after the computation of the pseudo-inverse, we computed p(N, M), the Frobenius norm of the matrix A-A -I. This norm was used to compute the standard log-error by matrix element E = - log p(N)/NM. In Fig. 3, we depict that error, and in Fig. 4 we repeat the experiment for a comparison with the built-in SVD-based algorithm used in MatLab. As expected, the latter one is better but it is computationally much heavier.
M.D. Ortigueira / Signal Processing 52 (I 996) 65- 73
71
algorithm for the recursive implementation of the propagator. The algorithm is simple and easily implementable. As a particular case, we derived a new algorithm for the (pseudo)-inversion of any matrix.
Appendix A. The dyads
Let u and w be two non-null vectors of dimensions N and M, respectively.
EL
I
120
0
psElD-ifwA~#*f/f Fig. 3. New versus Moore-Penrose
Definition 1. A matrix E of dimensions N x A4 is
a dyad if it is of the form
peudo-inverse.
E=v.wH.
(A.11
E is a matrix of rank 1. It is not hard to conclude that if E is a square matrix, then it has only a nonnull eigenvalue (wH. u); if E is a rectangular matrix (N # M), then it has a non-null singular value equal to pU.pw with p. = F and p,,, = fi
CO A.l. If a6 is a scalar difirent from - l/y” ‘x and x and y two vectors of dimension N, the inverse of [Z + ax .yH] is given bq
Theorem
+
[Z+olxyH]-’ 101 0
20 (+)A
40
60
m
lol
1M
new a l gor i thmf ) -Z pseudo- i nversebySVD
Fig. 4. New versus SVD pseudo-inverse.
= Z-(1
+;P.X)ryH
1
(A.2)
The proof is immediate [3]. A case of special interest is obtained with c1= - 2/9x. In this case, we introduce a matrix, H, called Householder matrix defined as
6. Conclusions
We presented an algorithm for the recursive analysis of matrices using the properties of the dyads. We showed how to make their factorization and inversion. The algorithm may be useful in several situations where we need adaptive algorithms: speech, arrays, spectral analysis, etc. We used the Hermitian algorithm to implement a recursive Capon algorithm. In short time we will expect to present results concerning the application of this
.
(A.3)
As it can be verified, using Eq. (A.3), H2 = Z and then H = H- ‘. The more important property of the Householder matrices will be stated in the following theorem.
6 If a is equal to deficient.
- l/y”x,
the matrix
Z - (x.y/ynx)
is rank
M.D. Ortigueira / Signal Processing 52 (1996) 6% 73
72
Theorem A.2. Let u and w be two non-orthogonal
Corollary A.2.1. Let E be a dyad (Eq. (AI)). Under
vectors and ek the kth column of the identity matrix. Introduce’
the conditions of Theorem A.2 the eigendecomposition of E is
rv = let.ul,
E=H+A.H,
r, = (eT.wl,
8 = arg(e: . u),
v, = arg(e: . w)”
(A.4)
(A.12)
where
and rj =
A = diag
?A.
W”U,
(A.51
r,
Define H a Householder matrix by Eq. (A.3) with the vectors x and y given by x = u - aejeek
and
Y=w-/I’ej’ek
(A4
i
O,O, . . . ,O w”v ,O, . . . ,O . -posk-
I
These results can be used in the eigendecomposition of the covariance matrix leading to recursive MUSIC and Min-Norm algorithms [S]. In [7], the dyads were used also in a recursive algorithm for SVD of the data matrix.
with’ Corollary A.2.2. Let I, be the Mth order identity cx= fi v
=
matrix and HNI an M x M Householder matrix dejined by
ej’,
arid + 4 - 0 2
.
(A.7)
Then, the matrix H verifies the following equalities: crejO.H*e, = u
(A.9
HN=IN-
(A.lO)
and wHH = /j* e-ja,$.
(A.13)
accordingly to Theorem A.2 with n = wHJTu. In a similar way, denote by HN the matrix defined in the following relation:
The verification is simple. The relations (A.8) and (A.9) allow us to show that the Householder matrices defined by Eq. (A.3) transform any two vectors into two others proportional to columns (or rows) of the identity matrix. In fact, as H2 = I, it is immediate to obtain H. u = uejeek
2 HT JTx.yH y J ax
(A.8)
and p* e-jq. e:H = w”.
H,=I,-
2 rx.yHJT. y J .x
(A.14)
With these matrices, a rectangular represented by 0 0 0 E=HN.J.
“’
‘HM, il
(A.1 1)
These properties are useful in finding the eigendecomposition and singular value decomposition of a dyad. For example:
‘We will not consider the cases w(k) = 0 or u(k) = 0. ‘If x and _v are orthogonal, define new x and y by x = a: u - ej@e, and y = p. w - eJ’p- e,
dyad can be
(A.15)
... 0
0 L 1 where the inner matrix, A, is a square (M x M). This relation expresses a decomposition similar to the SVD of a rectangular dyad. Another approach to transform a dyad into a diagonal matrix is supplied by the Gauss-Jordan condensation matrix. Let us introduce it, next. Let x(k) be the k component of a generic vectorx(N x 1)
M.D. Ortigueira / Signal Processing
and define a vector X by9 x
(A.16)
X=a-ek, and the Gauss-Jordan
transformation
matrix G by
G=I,+_%.el
(A.17)
G-’ =IN -3.e:
(A.18)
and verify (A.19)
Let G and C be two Gauss-Jordan matrices corresponding to vectors x (N x 1) and y (A4x l), N > M, and T a N x M rank-deficient matrix defined by Tk = J.Dk,
(A.20)
where Dk is an M x M diagonal matrix of rank k. They have the property 1 ,< k & M
if i k.
transformation
(B.1)
such that Q-52)
Asy=PxandPP=I, x = y(k)PG.e,.
X =X(k)&,.
73
difficulty, if there is an x(j) # 0,j > k. In this case, introduce a permutation matrix P that transforms x into y such that
y =y(k)G.ek.
with inverse
G-’ Ti(C-‘)H = Ti
52 (1996) 65-73
(B.3)
So, the algorithm in Section 2 remains almost equal. We only have to perform a permutation when computing the factor matrices or their inverses.
References and C.F. Van Loan, Matrix Computations, Johns Hopkins University Press, Baltimore, MD, 1983. VI S. Haykin, Adaptive Filter Theory, Prentice-Hall, Englewood Cliffs, NJ, 1988. Englewood c31T. Kailath, Linear Systems, Prentice-Hall, Cliffs, NJ, 1980. M J. Munier and G.Y. Delisle, “Spectral analysis using new properties of the cross-spectral matrix”, IEEE Trans. Signal Process., Vol. 39, No. 3, March 1991. 151M.D. Ortigueira, “Matrices generadas por adicibn de diadas (matrices de Rango 1): Propiedades y aplicaciones”, Mttodos Nunkricos para Calculo Diserio en Ingieneria, Spain, To appear. C61M.D. Ortigueira and M.A. Lagunas, “A recursive algorithm for the computation of eigenvalues and eigenvectors with application to array processing”, Proc. Latvian Signal Processing Internat. Co@, Riga, April 1990. c71M.D. Ortigueira and M.A. Lagunas, “A recursive SVD algorithm for array signal processing”, Proc. EUSIPCO90, Barcelona, Spain, September 1990. PI M.D. Ortigueira and M.-A. Lagunas, “Eigendecomposition versus singular value decomposition in adaptive array signal processing”, Signal Processing, Vol. 25, No. 1, October 1991, pp. 3549. c91N.L. Owsley, “Sonar array processing”, in: S. Haykin, ed., Array Signal Processing, Prentice-Hall, Englewood Cliffs, NJ, 1988. Cl01M.C. Pease, Methods of Matrix Algebra, Academic Press, New York, 1965. estimation Cl11S. Prasad and B. Chandra, Direction-of-arrival using rank revealing QR factorization, IEEE Trans. Signal Process., Vol. 39, No. 5, May 1991. Cl21J. Sherman and W.J. Morrison, “Adjustment of an inverse matrix corresponding to changes in the elements of a given column or a given row of the original matrix,” Ann. Math. Statist., Vol. 20, 1949, p. 261.
Cl1 G.H. Golub
