Subspace algorithms for system identi cation and stochastic realization Bart De Moor
y
Peter Van Overschee
Johan Suykens
ESAT Department of Electrical Engineering Katholieke Universiteit Leuven Kardinaal Mercierlaan 94 B-3001 Leuven (Heverlee) Belgium tel: 32/16/220931 fax: 32/16/221855 email:
[email protected] or
[email protected]
ESAT-SISTA Report 1990-28 Abstract The subspace approach for linear realization and identi cation problems is a promising alternative for the 'classical' identi cation methods. It has advantages with respect to structure determination and parametrization of linear models, is computationally simple and numerically robust. A summary is given of existing techniques based upon the singular value and qr decomposition. Algebraic, geometric, statistical and numerical points are emphasized. A new idea is outlined for the joint stochastic realization - deterministic identi cation problem. Several examples are given. Keywords: Singular value and qr decomposition, canonical correlations and angles, stochastic realization, instrumental variables.
1 Introduction While subspace techniques are by now quite common in the solution of the direction-of-arrival problem [13] [14], it is perhaps less well known that they are equally succesful in an identi cation and stochastic realization context. In this paper, we give a summary of some existing strategies and point out some new ideas. We shall discuss geometrical, algebraic, statistical and numerical issues, which are all related to the following central problem: Let a sequence of outputs yk 2 > li) block Hankel matrix Y with the output vectors and partition it as
Y=
Ypast Yfuture
!
Using the input-output matrix equation (3), we nd Ypast = ?h X^ past + Hh Epast and Yfuture = ?h X^ future + Hh Efuture . It is also straightforward to derive that X^ future = Aih X^ past +h Epast where h is a reverse extended controllability matrix: h = ( Ajh?1 Bh : : : Ah Bh Bh ). It can now be shown that:
Theorem 5 t ) = nh . Fact 1: The rank of the matrix E(Yfuture Ypast t Fact 2: The column space of E(Yfuture Ypast) coincides with the column space of ?h , i.e. it
has a shift structure generated by the matrices Ah and Ch (or similarity transforms of these matrices). t ) has a shift structure generated by the Fact 3: The row space of the matrix E(Yfuture Ypast matrices Ah and G (or similarity transforms of these matrices). Some relevant references are [1] [9]. Fact 1 permits to estimate the order of the stochastic system. Fact 2 and 3 allow to estimate the matrices Ah ; G and Ch by constructing a block Hankel matrix with the output covariances (see the algorithm below). t ) is rank de cient implies the existence of an (li ? n)The fact that the matrix E(Yfuture Ypast dimensional subspace of the li-dimensional row space of Ypast , which is orthogonal to an (li ? n)-dimensional subspace of the row space of Yfuture . In both spaces, there are ndimensional subspaces that are not orthogonal to one another. In other words, there are n canonical angles (which is a generalization for subspaces of the angle between two vectors, see e.g. [10] for a de nition) between the row spaces of Ypast and Yfuture that are not equal to =2. All others are equal to =2. It is interesting to note that the expected value of these canonical angles can be computed if the matrices Ah ; Ch and G are known:
Theorem 6
Let P and N be the solutions of the forward (8), resp. backward (9) Riccati equations. Then, the expected values of the cosines of the n canonical angles between the row spaces of the random matrices Yfuture and Ypast (of which the expected value is unequal to =2) are the square roots of the eigenvalues of the product NP .
7
This result was proved in [8]. Theorem 5 and 6 lead us to the following Algorithm for stochastic realization
- Let Yfuture and Ypast be the li j block Hankel matrices containing the past and future
outputs. Estimate the order nh of the stochastic system either from the singular values of t =j (which is an estimate of E(Yfuture Y t ) for large j ), or from the canonical Yfuture Ypast past angles between the row spaces of Yfuture and Ypast (for an algorithm, see [10]). - Obtain the SVD of S1 0 ! V1t ! t Yfuture Ypast = U1 U2 V2t 0 S2 where the dimension of S1 is an estimate of the system order. - Obtain the estimates
Ch Gt Ah
rst l rows of U1 last l rows of V1S1
U1+ U1
- Estimates of all possible noise covariance matrices Q; R; S can be obtained as described in Theorem 4.
4.3 Canonical angles: Finite sample properties
Assuming ergodicity, the order of the system can be estimated by considering the canonical angles between the row spaces of the block Hankel matrices Ypast and Yfuture for growing j . As j ! 1, li ? n angles will converge to =2 while n angles will reach an asymptotic level (which could be derived from Theorem 6 if Ah , Ch and G were known). An example is shown in gure 5 for a 3-th order system with innovations representation:
1 0 1:7 ?1:2 0:35 0 0C Ah = B A @ 1 0 1 0 ! 3 : 7 ? 2 : 2 ? 0 : 15 Ch = ?0:1 0:3 0
1 0 1 0 Bh = B @ 0 1 CA 0 0:5
We took E(ek etk ) = I3. The expected values of the canonical angles were computed using Theorem 6. They are 10:190; 36:780; 59:090. These are also the asymptotes in gure 5. We emphasize that the canonical angles are plotted, not just for one value of j , but for increasing j . This allows a clearer vizualization of the order estimation. There are li ? n canonical angles for which the expected value is =2. It is less well known that the nite sample probability distributions of these canonical angles are known as a function of j . By a random subspace we understand the row space generated by a random matrix, the elements of which are identically and independently normally distributed. 8
80 60 40
* * ** *
** * ** **
* * * **
* * ** *
* ** ***
* * ***
** * **
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
100
200
300
400
500
600
700
800
* * ** * * *
* *
20 0
Figure 5: Canonical angles between the row spaces of Ypast and Yfuture as a function of j for simulated data with i = 5. The system has two outputs and is third order. The asymptotes of the angles as a function of j were computed using Theorem 6.
Theorem 7
In an j -dimensional ambient space, the joint probability density p(cos21 ; : : :; cos2p ) of the cosines squared of the canonical angles between a p-dimensional random subspace U and a q-dimensional random subspace V is given by
K (p; q; j )pk=1(cosk )q?p?1 (sini )j?q?p?1 pk
