Bayes estimation of number of signals

Ann. Inst. Statist. Math. Vol. 43, No. 2, 227-243 (1991)

BAYES ESTIMATION OF NUMBER OF SIGNALS N. K. BANSAL 1 AND M. BHANDARY 2

1Department of Mathematics, Statistics and Computer Science, Marquette University, Milwaukee, WI 53233, U.S.A. 2 Chesapeake Biological Laboratory, University of Maryland, P.O. Box 38, Solomons, MD 20688, U.S.A.

(Received October 31, 1988; revised February 9, 1990)

A b s t r a c t . Bayes estimation of the number of signals, q, based on a binomial prior distribution is studied. It is found that the Bayes estimate depends on the eigenvalues of the sample covariance matrix S for white-noise case and the eigenvalues of the matrix S2(SI+A) -1 for the colored-noise case, where $1 is the sample covariance matrix of observations consisting only noise, $2 the sample covariance matrix of observations consisting both noise and signals and A is some positive definite matrix. Posterior distributions for both the cases are derived by expanding zonal polynomial in terms of monomial symmetric functions and using some of the important formulae of James (1964, Ann. Math. Statist., 35, 475-501).

Key words and phrases: measure, partitions.

i.

Zonal polynomial, white-noise, colored-noise, Haar

Introduction

In the area of signal processing, signals are observed at different time points from different sources to different sensors. However, due to atmospheric interference, the signals received by the sensors do not remain undistorted. A noise factor affects the signals in the receivers. In this area, a model often used is t h a t the observed signal vector is the sum of a r a n d o m noise vector and a linear transform of a r a n d o m signal vector. Noise vector and signal vectors are assumed to be indep e n d e n t l y distributed as multivariate normal with zero m e a n vector. One of the i m p o r t a n t problems in this case is estimating the n u m b e r of signals transmitted. In the classical case, this problem is equivalent to estimate the multiplicity of smallest eigenvalue of the covariance m a t r i x of the observation vector. Anderson (1963), Krishnaiah (1976), Rao (1983) considered the problem of testing the hypothesis of the multiplicity of the smallest eigenvalue of the covariance matrix. Wax and Kailath (1985), Zhao et al. (1986a, 1986b) considered the problem of estimation of n u m b e r of signals by information theoretic criteria proposed by Akaike (1972), Rissanen (1978) and Schwartz (1978). 227

228

N. K. BANSAL AND M. BHANDARY

In this paper, we will consider the problem of estimation of the number of signals from a Bayesian point of view. Bayes estimation of the binomial integer parameter n has been discussed by Hamedani and Walter (1988). In the similar fashion, we will estimate the integer parameter q, the number of signals transmitted from sources to the sensors. In Section 2, we discuss the model in signal processing and discuss different cases of the problem. In Section 3, we define some notations and derived some preliminary results useful for solving the problem. In Section 4, we discuss the Bayesian solution for estimating the number of signals for the white-noise case by choosing appropriate prior distributions for the parameters. A discussion for the colored-noise case is in Section 5.

2.

Model in signal processing The following model is used in signal processing

(2.1)

x(t) = As(t) + n(t),

where z ( t ) : p × 1 observation vector at time t, A: p x q matrix of unknown parameters associated with signals, s(t): q × 1 vector of unknown random signals,

n(t):p x 1 vector of random noise. The assumption for the model (2.1) is as follows

(2.2)

s(t) ~ Nq(0, ~ Np(0, Zl)

and s(t) and n(t) are independent. In (2.2), if E1 = a2I then model (2.1) is called the white-noise model, otherwise it is called a colored-noise model. We will discuss the solution of the problem described in Section 1 for both cases. Under assumption (2.2), we can say from model (2.1) that (2.3)

x(t) ~ Np(0, Aq2A'+ El).

The number of signals transmitted is q (< p) which is the rank of A ~ A ~. So, here estimation of number of signals is equivalent to the estimation of the rank of

A ~ A I. For the white-noise case, i.e. when E1 -- a2I, we will assume that n independent observations on x(t) are available as x ( t l ) , . . . , x(tn). Let the sample covariance matrix be n - i S = (l/n)~'~=lx(ti)~(t~)'. Then using (2.3) it is obvious that S ~ Wp(E, n), where E ----r + a2I, F (= A~A') is non-negative of rank q (< p). We will discuss the problem of estimation of q from a Bayesian point of view, when a 2 is known and when it is unknown. For colored-noise case, i.e. when E1 is arbitrary positive definite matrix, we will assume that n21S2 based on the original data set ~ ( t l ) , . . . , ~(t~2) estimates

BAYES ESTIMATION OF NUMBER OF SIGNALS

229

E2 (= F + El) and n11S1 based on a different data set X(tn~+l),..., x(t~) independent of the original data set, where n = nl + n2, estimates El. Then n2

$2 = E =(ti)x(ti)' ~

Wp(E2, n2),

i=1

(2.4)

n

S1 :

E

X(ti)X(ti)F ~" W p ( ~ I ' Ttl)

i=n2+1 and S1, S 2 are independent. We will discuss the problem of estimation of q in this case by Bayesian procedure.

3.

Notations and some preliminary results Denote

(3.1) (3.2) (3.3) (3.4) (3.s)

Ar+l = Diag()~l,..., At, , ~ r + l , . ' . , P×P A~I = Diag(A1,..., A~, 1 , . . . , 1), pxp

•~r = ()~1, A 2 , ' ' ' ,

/~r+l),

At) l,

:Dr = {A~: 0 < ~1 < ,~2 _< ' " < At-1 < ,~ < oc},

7),.1 = {A~:0_< 11 _< . . . _< A~ < 1}.

Let P(j, m) be the set of all partitions ~- = (tl, t 2 , . . . ) of j into no more than m parts such that tl + t2 + . . . . j, and for T E P(j, m), let C~(A) be the zonal polynomial formed from the eigenvalues of A. The zonal polynomial C~(A) can be expressed as a linear combination of monomial symmetric functions M~ (A) of eigenvalues of A, see James (1964) and Muirhead (1988). If A is a symmetric matrix of order m x m with eigenvalues )h, A2,..., Am, then

M,~(A) = E Ahkl ,~k2 ,~ . .',kipkp ,

(3.6)

where p is the number of non-zero parts in the partion ,~ = (kl, k2,...) and the summation is over the distinct permutations (il, i2,..., ip) of p different integers 1, 2 , . . . , m.

C~(A)=Ec~,~M~(A),

(3.7)

,~ O. Then (3.14)

Br~(a, b, c)

cJ+r~+bI'(j + ra + b) 2..~ % ~

-~-~p)

p,a --

r!

-

Z e~,~ (r - I~1)~b~(r, j), d + b r ( j + b),

for

r T~ O,

for

r = 0

where Ial represents the number of non-zero parts in ~ and (3.15)

b~(r, j ) =

/o1 /o111/=1 .~-,-~--:--~-j:.a-~ ,,-1i aul...aur •

"

'

r#0.

for

--i-2_,~=i i)

LEMMA 3.3.

(3.16)

Let ~- E P(j, p). Let

Dr(a, b) = r! ~

~ I A a - l ( 1 - Ai)b-lcr(Arl)/Cr(I) d ~

rl i=1 for some a, b > O. Then . C~(Ip_r) r[ Ep,a apa C.(Ip) tcEo

e-trStsIa-(k+l)/2°F°(ST' U)dS = 1Fo(a; T, U),

where

Fk(a) = fs e-t~Slsla-(k+l)/2dS

(3.21)

>0

----Trk(k-1)/4 H F

a--~(i--1)

,

i=1

1F0(a; T, U ) =

(3.22)

E E j=O r e P ( j , k )

LEMMA 3.6.

(a)rCr(T)Cr(U) j!C.(I)

Let ~- E P(j, p). Let P

Err(a, b) = [

J D ~,

1

)ta-1 (1 -- )ti) b-1H()~i - )~j) Cr(Arl)dA~ i>j

C,(I)

for some a, b > O. Then x--" ~ C,(I)Cp(I) C--~

(3.23) E:(a, b) = 2_.., ap,er p,¢r

F~(a + b + r - 1, p)

r(

where

F~(t,p)--Tr r ( r - ' ) / 4 H F

7(r2/2

1)

t+p~-~(i-1)

i=l

if (Pl, P2,...,

Pr) is the

partition corresponding

to p.

PROOF OF LEMMA 3.2. We first prove the lemma for r ~ 0. Since Cr(A), for any smmetric matrix A, is symmetric in eigenvalues of A, it can be seen that, the integrand in (3.13) is symmetric in A1, k 2 , . . . , Ar and therefore the left-hand side of (3.13) is

B~(a, b, c) = fo~ fo ~+1

" " "

"'i

"'r+l

{

i=l

C.(A~+~)dA1..,dA~+1. C~(I)

)

BAYES ESTIMATION OF NUMBER OF SIGNALS

Now using the transformation ui = )~i/;~r+l, i = 1 , . . . , r, ;k~+l for any constant h, we get

233 =

)~r-I-1,

C~(hA) -- hJC~(A)

and since

r b, c) = L L "''L lrr|ll.~i r [ a--l'A~+ j-l-ra-1-5--1 B,(a, 1 c,~

1

kl

Cr([P) dUl"" d~rdar+l

1 = d+"~+br(j + ra

L1 L1

....

(1 +

+ b)

1-[1~ ~--1U~ C,(U@lp_~)du 1 ~-,~ , j + ~ + b C~-(Ip) "..du,, 2-~1 uo

where U -- D i a g ( u l , . . . , u~). Now the result, for r ~ 0, follows from L e m m a 3.1. For r = O, since

C~-(AII)= AJc~-(I),

B°r(a, b, c ) =

A~-I

=

/?

exp{-A1/c}C~-(Ali)/Cr(I)dAl

I j+b-1 e x p { - ) ~ l / c } d , X l = d + b r ( j

+ b).

Hence, the lemma.

PROOF OF LEMMA 3.3. First, let r ¢ 0, then, by the same argument of s y m m e t r y as in the proof of L e m m a 3.2, r

Dr~(a,b) = r! i~ H )~-I(1 - Ai)b-Ic~(Arl)/C~(I)dA~ rl i=1 ?-

L 1 L1 I IA~-,(1

Ai)b-IC.(A.Ip_,.)IC.(I)dA.,

l

where A = D i a g ( A i , . . . , A~). Now the result, for r ¢ 0, follows from L e m m a 3.1. For r = 0, clearly, b) = 1. This proves the lemma.

D°(a,

PROOF OF LEMMA 3.6. (3.24)

From (3.9)

a;~Cp(A)C~(I~_~),

C~(Arl) = E p~cr

where h = Diag(A1,..., A~). Hence from (3.23) and (3.24), it is enough to show r

(3.25) ~, 1

i>j

p( )

rr(a+~(r-i),P)r~(b+ 2(r-i))F~(2r ) Fr(a + b + r - 1, p)

7rr~/2

234

N.K. BANSAL AND M. BHANDARY

Now, using equation (22) of Constantine (1963) and substituting R = I~, m = r, n = p, we have

i

(3.26)

i~

ISl'-(r+~)/~lZ -

L(t, p)r~(u)

s ~-(~+~)/2 c . ( s )

C . ( I ~ ) d S = L ( t + ~, p) "

Using the transformation S ~ P A P ~, where P is orthogonal of order r and the first column of P is non-negative and using the result (Anderson (1984))

o

(~)

J(S ~ P' A )dP =

we get, from (3.26), fz~ l~I Ati-(~+l)/2(1-A~)~-(~+a)/2 H ( A i - A Y ). cCp(A) (r/2) - - ~ d A r = l'~(t, r~(t +p)r~(u) u, p) L~.~/2 rl

1

i>j

and this proves (3.25). 4.

Bayes estimation of q under white-noise Let S ~ Wp(E, n)

(n > p),

where E = F + a2I, F is of rank q. We consider two cases, when a 2 unknown and when a 2 = 1.

Case 1. 0-2 unknown: Since F is of rank q, E can be reparametrized into an orthogonal matrix P and Aq+l defined as in Section 3, such that E -1 = P'Aq+IP, where Aq+l is defined in Section 3. Then the parameter space O = {(P, ~q+l,

q): P is o r t h o g o n a l ,

0 < AI _(. ~

-

aj)C.(A.,)/C.(Oe),.] 1 (m-j-n~ CT (-~('2+A)-Is2)

oo

= c..(s,, s.)p(.)r. Z ~. \ j=o reP(j,p)

(1~(.1 +n2 - p +

• E;

1

2 /T

1), ~(~2 - p + * )

),

where C.(S1, $2) is some constant. Finally, the estimate is

p-1

p-1

1 oo 1 (m 2______~n) ~ C. (-~(S1-~1 A)-'S2) (p--)¢Ej=0E~_~I.I V'~P-2 L:'t_,~(r+i) (1 1 ) •/_~=0 ~'2t,j~r ~(ul + n 2 - p + 1), ~(v2 - p + 1)

oc

.1

1 (m 2___~n )

.~=oFt(r)E(r)

( ~("1 1 +n2 - p +

A)-1S2)

1 - p + 1) 1), ~(v2

)

where, for i = 1, 2

F~(r) = ( P - i) Here also the series in ~ converge if Amax((1/2)(S1 + A)-Is2) < 1.

Acknowledgements The authors wish to thank the referee and the editorial committee for their careful reading of this paper, and for their valuable comments which helped us to improve its original version and broaden its content.

242

N . K . BANSAL AND M. BHANDARY

Appendix T h e table for coefficients Dl(n/2 + a, #) and B~(n/2 + a, n/2 + a,/3), where = r ( n / 2 + a + i ) r ( # ) / r ( n / 2 + a + # + i), i = o, 1, 2 , . . . and ~ni = r ( n / 2 -~c~+ i)~ n/2+a+i, i = 0, 1, 2 , . . . . ~

degree

T

D~(n/2 + a,/3)

B~ (n/2 + a, n/2 + a,/3)

o

(o)

~.o

2 e.o/2

1

(1)

(TriO+ ~ n 2 ) / 2

~n2~nl/2

2

(2) (15) (3) (21)

(3~n0 + 2~nl T 3~n2)/8 ~nl (5~n0 +3~nl + 3~2 + 5~3)/16 (7~1 +7~2)/2

(~n0~n2 + ~ 1 ) / 8 ~1/2 ( 5 ~ 0 ~ 3 + 3~1~n2)/16 ~n~2/2

3

(4)

(35~n0 + 20"Ynl -b 18~,n2 +20"Yn3 -t- 357n4)/128 (31) (3~nl + 2~n2 -t- 3~/n3)/8 (22 ) "~n2

(35~n0~n4 +20~nl~n3 + 9~2)/128

~2

n2

REFERENCES Akaike, H. (1972). Information theory and an extension of the maximum likelihood principle, Proc. Second International Symposium on Information Theory, Supp. to Problems of Control and Information Theory, 267-281. Anderson, T. W. (1963). Asymptotic theory for principal component analysis, Ann. Math. Statist., 34, 122-148. Anderson, T. W. (1984). An Introduction to Multivariate Statistical Analysis, Wiley, New York. Constantine, A. G. (1963). Some non-central distribution problems in multivariate analysis, Ann. Math. Statist., 34, 1270-1285. Davis, A. W. (1979). Invariant polynomials with two matrix arguments extending the zonal polynomials: Application to multivariate distribution theory, Ann. Inst. Statist Math., 31, Vol. No. 465-485. Gross, K. and Richards, D. (1987). Special functions of matrix argument I: Algebraic induction, zonal polynomials and hypergeometric functions, Trans. Amer. Math. Soc., 301,781-811. Hamedani, G. G. and Walter, G. G. (1988). Bayes estimation of the Binomial parameter n, Comm. Statist. A--Theory Methods, 17, 1829-1843. Hayakawa, T. (1967). On the distribution of the maximum latent root of a positive definite symmetric random matrix, Ann. Inst. Statist. Math., 19, 1-17. James, A. T. (1960). The distribution of the latent roots of the covariance matrix, Ann. Math. Statist., 31, 151-158. James, A. T. (1964). Distributions of matrix variates and latent roots derived from normal samples, Ann. Math. Statist., 35, 475 501. Khatri, C. G. and Pillai, K. C. S. (1968). On the noncentral distributions of two test criteria in multivariate analysis of variance, Ann. Math. Statist., 39, 215-226. Krishnaiah, P. R. (1976). Some recent developments on complex multivariate distributions, J. Multivariate Anal., 6, 1-30. Kushner, H. B. (1985). On the expansion of C~(V + I) as a sum of zonal polynomials, J. Multivariate Anal., 17, 84-98. Kushner, H. B. (1988). The linearization of the product of two zonal polynomials, SIAM J. Math. Anal., 19, 687-717.

BAYES ESTIMATION OF NUMBER OF SIGNALS

243

Kushner, H. B. and Meisner, M. (1984). Formulas for zonal polynomials, J. Multivariate Anal., 14, 336-347. Muirhead, R. J. (1988). Zonal polynomials, Encyclopedia of Statistical Sciences, Vol. 8 (eds. S. Kotz, N. L. Johnson and C. B. Read), 676-682, Wiley, New York. Parkhurst, A. M. and James, A. T. (1974). Zonal polynomials of order 1 through 12, Selected Tables in Mathematical Statistics (eds. H. L. Harter and D. B. Owen), 199-388, American Mathematical Society, Providence, Rhode Island. Rao, C. R. (1983). Likelihood ratio tests for relationships between two covariance matrices, Studies in Econometric, Time Series and Multivariate Statistics (eds. T. Amemiya, S. Karlin and L. Goodman), Academic Press, New York. Rissanen, J. (1978). Modeling by shortest data description, Automatica--J. IFAC, 14, 463-471. Saw, J. G. (1977). Zonal polynomials: An alternative approach, J. Multivariate Anal., 7, 461467. Schwartz, G. (1978). Estimating the dimension of a model, Ann. Statist., 6,461-464. Wax, M. and Kallath, T. (1985). Determination of the number of signals by information theoretic criteria, IEEE Trans. Acoust. Speech Signal Process., 33, 387-392. Zhao, L. C., Krishnalah, P. R. and Bai, Z. D. (1986a). On detection of number of signals in presence of white noise, or. Multivariate Anal., 20, 1-25. Zhao, L. C., Krishnaiah, P. R. and Bal, Z. D. (1986b). On detection of the number of signals when the noise covariance matrix is arbitrary, J. Multivariate Anal., 20, 26-49.