Estimation of Markovian Jump Systems with Unknown Transition ...

Estimation of Markovian Jump Systems with Unknown Transition Probabilities through Bayesian Sampling? Vesselin P. Jilkov1 , X. Rong Li1 , and D. S. Angelova2 1

2

Department of Electrical Engineering, University of New Orleans New Orleans, LA 70148, USA {vjilkov, xli}@uno.edu Central Laboratory for Parallel Processing, Bulgarian Academy Of Sciences 25A Acad. G. Bonchev St, 1113 Sofia, Bulgaria [email protected]

Abstract. Addressed is the problem of state estimation for dynamic Markovian jump systems (MJS) with unknown transitional probability matrix (TPM) of the embedded Markov chain governing the system jumps. Based on recent authors’ results, proposed is a new TPM-estimation algorithm that utilizes stochastic simulation methods (viz. Bayesian sampling) for finite mixtures’ estimation. Monte Carlo simulation results of TMP-adaptive interacting multiple model algorithms for a system with failures and maneuvering target tracking are presented.

1 Introduction Markovian jump systems (MJS) evolve in a jump-wise manner by switching among predefined dynamic models, according to a finite Markov chain. Theoretically, in the estimation for MJS (or more generally, hybrid system [1]) the TPM is almost always assumed known. In practice it is a “design parameter” whose choice is done a priori. For most applications, however, a priori information about the TPM may be inadequate or even lacking. Using an inaccurate TPM may lead to performance degradation of the state estimation. The uncertainty regarding the TPM is a major issue in the application of MM to real-life problems, e.g., maneuvering target tracking, systems’ fault detection and isolation, and many others. Thus, it is highly desirable to have algorithms which can identify the TPM or estimate it recursively in the course of processing measurement data. There exist only a few publications in the engineering literature devoted to solving this important and difficult problem [2–5] (See [6] for a review of previous contributions). Our work, initiated in [7, 6] and furthered in this paper, is conceptually based on the assumption that the unknown TPM is a random constant matrix with a given prior distribution defined over the continuous set of valid TPMs. Within this Bayesian framework we have obtained a recursion for the TPM’s posterior PDFs in terms of an MM estimator’s model probabilities and likelihoods and developed an optimal estimator for ?

Research Supported in part by ONR grant N00014-00-1-0677, NSF grant ECS-9734285, NASA/LEQSF grant (2001-4)-01, and by Center of Excellence BIS21 grant ICA1-200070016.

the case of a two-state Markov chain. Since the optimal solution has (linearly) increasing computation, we also proposed (for the general case of a multiple-state Markov chain) three computationally feasible, approximate algorithms which utilize alternative approximation techniques — viz. second moment approximation, quasi-Bayesian (QB) approximation, and numerical integration. In this paper we continue with development of a new approximate algorithm for TPM estimation. It has been stimulated by the recent advances in the stochastic sampling (Markov chain Monte Carlo (MCMC)) methods for highly nonlinear estimation/filtering problems, e.g., [8]. Specifically, we utilize the stochastic expectation maximization (SEM) (also referred to as data augmentation (DA)) for finite mixture estimation [9] to implement our approach [7, 6] for joint TPM and MM state estimation. To illustrate and compare the performance of the proposed sampling-based TPM-adaptive MM estimation algorithm, we also provide simulation results from two most typical applications of MM filtering.

2 Problem Formulation Consider the following model of a Markovian jump system in discrete time k = 1, 2, . . . x(k) = f [k, m(k), x (k − 1)] + g [k, m(k), w [k, m(k)]] z (k) = h [k, m(k), x (k)] + v [k, m(k)]

(1) (2)

where x is the base (continuous) state vector with transitional function f, w is the random input vector, z is the measurement vector with measurement function h, and v is the random measurement error vector. All vectors are assumed of appropriate dimensions. The modal (discrete) state m (k) ∈ M , {1, 2, . . . , m} is a Markov chain with initial and transition probabilities respectively P {mj (0)} = µj (0) (3) © ª k P {mj (k) |mi (k − 1)} = P mj (k) |mi (k − 1) , z = π ij , i, j = 1, . . . m (4) where mi (k) stands for the event {m(k) = i} and z k = {z(1), . . . , z(k)}. Provided all parameters £ ¤ of the model (1)–(4) are known the MMSE-optimal estimate x ˆ(k) = E x (k) |z k of the base state x can be obtained by the Bayesian fullhypothesis-tree (FHT) estimator [10]. The FHT is however infeasible because of its exponentially growing computation and thus suboptimal approximations with limited complexity are of interest in practice. There exist a number of such approximations, commonly referred to as multiple model (MM) algorithms. For the class of Bayesian MM estimation a good trade-off between performance and computational cost is provided by the popular Interacting MM (IMM) estimator [11, 12]. To be more specific we will apply our further development to the case of IMM algorithm as an example. Let us consider now the state estimation problem for the above hybrid system model (1)–(4) without the presumed knowledge of the transition probability matrix 0 0 Π = [π 1 , π 2 , . . . , π m ] with π i = [π i1 , π i2 , . . . , π im ] , i = 1, . . . , m, defined by (4). We assume that Π is a random time-invariant matrix with given prior distribution defined over the simplex of valid TPMs. In such a Bayesian framework we£ consider ¤ the problem of finding recursively the posterior MMSE estimate Π (k) = E Π|z k of Π for k = 1, 2, . . . , and formulate the following.

Bayesian TPM-Adaptive MM State Estimation. On receipt of a new measurement zk : – Run the MM algorithm with the previous state/covariance estimates, model probabilities µ(k − 1) and Π(k − 1) to update the current state/ covariance estimates, model probabilities ª © 0 (5) µ (k) = [µ1 (k) , . . . , µm (k)] with µi (k) , P mi (k) | Π, z k−1 and model likelihoods £ ¤ 0 Λ (k) = [Λ1 (k) , . . . , Λm (k)] with Λj (k) , p z (k) | mj (k) , Π, z k−1 (6) – Update the TPM estimate Π(k) based on Π(k £ − 1) and the¤ new measurement information contained in the TPM likelihood p z (k) |Π, z k−1 . Note that this scheme is not restricted to the IMM algorithm only. Provided a feasible recursive TPM estimator, as specified above, is available the adaptation scheme is applicable to various MM state estimation algorithms in a straightforward manner.

3 Posterior PDF of TPM Bayesian estimation of the TPM is based on the recursive relationships for updating the TPM’s posterior PDF in terms of the model probabilities and likelihoods, defined by (5)–(6). As obtained in [6] ¡ ¢ p Π|z k =

¡ ¢ µ0 (k − 1) ΠΛ (k) k−1 p Π|z µ0 (k − 1) Π (k − 1) Λ (k)

(7)

where µ(k − 1) and Λ(k) are computed by replacing the unknown Π with its best estimate Π(k z k−1 ) available at that time. Analogically, the marginal ¡ − k1)¢ (conditioned on 0 PDF p π i |z of each row π i , i = 1, . . . , m of Π is given by the following. Theorem. If Z ¡ ¢ ¡ ¢ π l p π 1 , . . . , π m |z k−1 dπ 1 . . . dπ i−1 dπ i+1 . . . dπ m = π l (k − 1) p π i |z k−1 (8) for l, i = 1, 2, . . . m; l 6= i then ¡ ¢ ¡ ¢ p π i |z k = {1 + η i (k) [π i − π i (k − 1)]0 Λ (k)} p π i |z k−1

(9)

where η i (k) =

µi (k − 1) µ0 (k − 1) Π (k − 1) Λ (k)

(10)

Proof. See [6]. Remark 1. The condition (8) is satisfied if π i , i = 1, . . . , m are independent. However, it is in general looser than the independence assumption. The recursion (9) has been used in [6] for derivation of four MMSE TPM estimators. In this work it is the basis of the sampling-based TPM estimator, developed next.

4 TPM Estimation by Stochastic Simulation An immediate use of the Theorem is that it allows (under the stated assumptions) to decompose the estimation of the TPM into estimation of its rows. Indeed, from (9) the likelihood of each π i , i = 1, . . . , m, is represented in the following mixture form [6] m £ ¤ X f [z (k) |π i ] = p z (k) |π i , z k−1 = π ij gij (k)

with

(11)

j=1

gij (k) = gij [z (k)] = 1 + η i (k) [Λj (k) − π 0i (k − 1) Λ (k)]

(12)

Thus for each π i , i = 1, . . . , m, our TMP-estimation problem can be boiled down to a well known classification problem referred to as prior probability estimation of finite mixtures [13, 14], or as unsupervised learning [15]. 4.1 Mixture Estimation by Data Augmentation The prior probability estimation problem can be stated as follows: For the finite mixture model (11)–(12) with known mixture component PDFs gj [z(k)], estimate the unknown probabilistic weights π j , j = 1, . . . , m given a sequence of independent observations z(1), z(2), . . ., z(K) There exist a number of publications and results in solving this problem, see e.g. [13, 14, 16, 15]. One of the most powerful and promising recent approaches in this direction is based on the idea of stochastic simulation — represent (approximately) the distributions (prior and posterior) by random samples rather than by some analytical, parametric (e.g. by moments), or numerical (by deterministic samples) descriptions. Bayesian sampling is loosely speaking a numerical mechanization (simulation) of the Bayesian update equation. In the context of the prior probability estimation problem its particular interpretation is known as data augmentation method [9]. It can be interpreted as a Bayesian extension of the well known expectation-maximization (EM) method [17] and is therefore also referred to as stochastic EM (SEM). Specifically, represent first the mixture model (11) (the index i is dropped) in terms of missing (incomplete) data. That is, define vectors δ(k) = (δ 1 (k), δ 2 (k), . . . , δ m (k))0 , k = 1, 2, . . . , K with components δ j (k) ∈ {0, 1}, j = 1, 2, . . . , m, which indicate that the measurement zk has density gj [z(k)]. Then the SEM algorithm, similarly to the regular EM, implements a two-step iterative scheme. In the first step, instead of estimating the “missing data” δ(k) by their expectation (as in the EM), the missing data are simulated by sampling from their conditional distribution, i.e., δ (s) ∼ p(δ|π (s) ). Then, in the second step of the iteration, the unknown mixture proportions π are sampled from their posterior, conditioned on the simulated missing parameters, i.e. π (s+1) ∼ p(π|z, δ (s) ). Thus, by iterating with s = 1, 2, . . . this procedure, as shown in [9], converges to simulate from p(π|z). In a more general setting this method is well known as Gibbs (or Bayesian) sampling and belongs to the class of the Markov chain Monte Carlo (MCMC) methods. Generally speaking, the Bayesian sampling produces an ergodic Markov chain π (s) with stationary distribution p(π|z). For a sufficiently large s0 a set of samples π (s0 +1) , . . . , π (s0 +S) is approximately distributed as p(π|z) and, due to ergodicity, averaging can be made with respect to time.

4.2 TPM-Adaptive MM Algorithm By applying the outlined above SEM algorithm for mixture densities [9] to the TPM estimation formulated as a set of mixture estimation problems for each π i , i = 1, . . . , m via (11)–(12), we obtain the following SEM-TPM estimator. The algorithm operates within a generic Bayesian TPM-adaptive MM estimation scheme (e.g. IMM) as specified in Section 2. Algorithm (SEM-TPM Estimator) For k = 1, 2, . . . For i = 1, 2, . . . m (possibly in parallel) • Conditional PDFs Computation gij (k) = 1 +

1 µ (k − 1) [Λj (k) − π 0i (k − 1) Λ (k)], ck i with ck = µ0 (k − 1) Π (k − 1) Λ (k)

j = 1, 2, . . . m

• Data Augmentation (Stochastic EM) - Initialization (0)

πi

= π i (k − 1)

- Iterations (s = 0, 1, . . . , s0 + S − 1) * Missing Data Conditional PMFs (s)

π ij gij (l) (s) , qij (l) = Pm (s) j=1 π ij gij (l)

l = 1, 2, . . . k,

j = 1, 2, . . . m

* Missing Data Generation (Multinomial Sampling) n om (s) (s) δ i (l) = (0, . . . , 0, 1, 0, . . . , 0) ∼ qij (l) , j=1

l = 1, 2, . . . k

* Parameter Evaluation (Dirichlet Distribution Sampling) Ã ! k k X X (s) (s) (s+1) δ i1 (l) , . . . , αim + δ im (l) πi ∼ D π; αi1 + l=1

l=1

• Output Estimates π i (k) =

S 1 X (s0 +σ) π S σ=1 i

Remark 2. The use of the Dirichlet distribution (DD) [18] D (π; α1 , . . . , αm ), αj ≥ 0, j = 1, . . . , m for a mixture’s prior probabilities π = (π 1 , . . . , π m ) in the DA algorithm is justified by the fact that the conjugate priors on π are DDs [9]. The SEMTPM estimator requires knowledge of the prior parameters of αij . This however is not a shortcoming of the algorithm, because if αi1 = αi2 = . . . = αim = 1 the DD reduces to an uniform distribution and the algorithm can be initialized with noniformative (uniform) prior.

5 Simulation In this section we illustrate the performances of the proposed TPM-adaptive IMM algorithm. Two typical applications of Markov switching models are simulated − a system subject to measurement failures and maneuvering target tracking. 5.1 Example 1: A System with Failures Under consideration is a scalar dynamic system described for k = 1, 2, . . . by x (k) = x (k − 1) + w (k) z(k) = m(k)x(k) + [100 − 90m(k)] v(k) with x(0) ∼ N (0, 202 ),© w(k) ∼ Nª(0, 22 ), v(k) ∼ N (0, 1). The Markov model of switching is: m(k) ∈ m(1) , m(2) = {0, 1} , µ1 (0) = µ2 (0) = 0.5, π ij = ª © P m(k + 1) = m(j) |m(k) = m(i) , i, j = 1, 2. Note that m(k) = 0 corresponds to a measurement failure at k. Three IMM algorithms were implemented: an IMM knowing the true TPM (TPMexact IMM), an IMM with a typical design of fixed TPM with diagonally-dominant elements (π 11 = π 22 = 0.9) (non-adaptive IMM), and an IMM using the SEM-TPM estimator (TPM-adaptive IMM). In the scenario simulated the true TPM was with π 11 = 0.6, π 22 = 0.15. Results from 100 Monte Carlo runs are shown in Figure 1: (a) Accuracy/convergence of the SEM-TPM estimator and (b) comparative accuracy of the three IMM algorithms. The estimation accuracies of TPM and state estimation are evaluated in terms of mean absolute error – the absolute value of the actual error, averaged over runs. Figure 1 (a) illustrates the convergence of the TPM estimator. Figure 1 (b) shows that in case of a mismatch regarding the TPM the adaptive algorithms perform better than the typical IMM design (with π 22 fairly largely mismatched in this scenario). 0.4

40 π 11 π

Exact−TMP Non−Adaptive S−Adaptive

22

0.35

35

0.3

30

0.25

25

0.2

20

0.15

15

0.1

10

0.05

5

a 0

0

10

20

30

40

50

b 60

70

80

90

100

0

0

10

20

30

40

50

60

70

80

Fig. 1. Deterministic TPM Scenario: Error of (a) TPM and (b) IMM estimation

90

100

5.2 Example 2: Maneuvering Target Tracking We considered a simplified illustrative example of maneuvering target with Markov jumping acceleration. The one-dimensional target dynamics is given by · ¸ · ¸· ¸ · 2 ¸ p 1T p T /2 (k + 1) = (k) + [a (k + 1) + w(k + 1)] v 01 v T where p, v and a stand for the target position, velocity and acceleration ¤ £ respectively and w models “small” perturbations in the acceleration. p (0) ∼ N 80000, 1002 , £ ¤ £ ¤ v(0) ∼ N 400, 1002 and w (k) ∼ N 0, 22 is white noise sequence independent of p (0) and v (0). The acceleration process a (k) is a Markov chain with three states a1 = 0, a2 = 20, a2 = −20 and initial probabilities µ1 (0) = 0.8, µ2 (0) = µ3 (0) = 0.1. The sampling period is T = 10. The measurement £ ¤ equation is z (k) = x (k) + ν(k) where the measurement error ν(k) ∼ N 0, 1002 is white noise. Again three IMM algorithms were implemented: TPM-exact, non-adaptive (with π ii = 0.9, π ij = 0.05 j 6= i, i, j = 1, 2, 3), and TPM-adaptive. To obtain an overall, scenario independent evaluation we ran the algorithms over an ensemble of TPMs, randomly (uniformly) chosen from the set of all valid TPMs. Results from 100 MC runs of such randomly generated TPM scenarios are presented in Figure 2. The adaptive IMMs show better overall performance than the non-adaptive one (Figure 2 (b)). Clearly, in quite a lot of the runs the fixed TPM of the latter is mismatched with the true one, while the adaptive algorithms provide better estimated TPM. This indicates that for situations where the target behavior can not be described by one TPM it is preferable to use the adaptive TPM.

0.4

40 π 11 π 22 π 33 π21 π31

0.35

Exact−TMP Non−Adaptive S−Adaptive 35

0.3 30 0.25

0.2

25

0.15 20 0.1 15 0.05

a 0

0

10

20

30

40

50

b 60

70

80

90

100

10

0

10

20

30

40

50

60

70

80

90

Fig. 2. Random-TPM Scenario: Accuracy of (a) TPM and (b) IMM velocity estimation

100

6 Conclusion A new, stochastic sampling-based algorithm for estimating the unknown transition probabilities of a Markovian jumping dynamic system within the multiple model state estimation paradigm has been proposed in this paper. As illustrated by simulation examples, the proposed algorithm can effectively serve for off-line identification of hybrid (Markovian jump) dynamic systems, or for on-line adaptation of the transition probability matrix in multiple model state estimation algorithms.

References 1. Li, X.R.: Hybrid Estimation Techniques. In Leondes, C.T., ed.: Control and Dynamic Systems: Advances in Theory and Applications. Volume 76. Academic Press, New York (1996) 213–287 2. Sawaragi, Y., Katayama, T., Fujichige, S.: Adaptive Estimation for a Linear System with Interrupted Observations. IEEE Trans. Automatic Control(1973) 152–154 3. Tugnait, J.K., Haddad, A.H.: Adaptive Estimation in Linear Systems with Unknown Markovian Noise Statistics. IEEE. Trans. Information Theory26 (1980) 66–78 4. Tugnait, J.K.: Adaptive Estimation and Identification for Discrete Systems with Markov Jump Parameters. IEEE Trans. Automatic ControlAC-27 (1982) 1054–1065 5. Goutsias, J., Mendel, J.M.: Optimal Simultaneous Detection and Estimation of Filtered Discrete semi-Markov Chains. IEEE. Trans. Information TheoryIT-34 (1988) 551–568 6. Jilkov, V.P., Li, X.R.: Bayesian Estimation of Transition Probabilities for Markovian Jump Systems. IEEE Trans. Signal Processing(2002) Submitted. 7. Jilkov, V.P., Li, X.R.: Adaptation of Transition Probability Matrix for Multiple Model Estimators. In: Proc. Fourth Intern. Conf. on Information Fusion, Montreal, QC, Canada (2001) 8. Doucet, A., de Frietas, N., Gordon, N., eds.: Sequential Monte Carlo Methods in Practice. Statistics for Engineering and Information Science. Springer-Verlag, New York (2001) 9. Diebolt, J., Robert, C.P.: Estimation of Finite Mixture Distributions through Bayesian Sampling. J. R. Statist. Soc. 56 (1994) 363–375 10. Bar-Shalom, Y., Li, X.R., Kirubarajan, T.: Estimation with Applications to Tracking and Navigation: Theory, Algorithms, and Software. Wiley, New York (2001) 11. Blom, H.A.P., Bar-Shalom, Y.: The Interacting Multiple Model Algorithm for Systems with Markovian Switching Coefficients. IEEE Trans. Automatic ControlAC-33 (1988) 780–783 12. Mazor, E., Averbuch, A., Bar-Shalom, Y., Dayan, J.: Interacting Multiple Model Methods in Target Tracking: A Survey. IEEE Trans. Aerospace and Electronic SystemsAES-34 (1996) 103–123 13. Kazakos, D.: Recursive Estimation of Prior Probabilities Using a Mixture. IEEE. Trans. Information TheoryIT-23 (1977) 203–211 14. Dattatreya, G.R., Kanal, L.N.: Estimation of Mixing Probabilities in Multiclass Finite Mixtures. IEEE Trans. Systems, Man, and Cybernetics SMC-20 (1990) 149–158 15. Titterington, D., Smith, A., Makov, U.: Statistical Analysis of Finite Mixture Distributions. John Wiley & Sons (1985) 16. Smith, A.F.M., Makov, U.E.: A Quasi-Bayesian Sequential Procedures for Mixtures. J. R. Statist. Soc. 40 (1978) 106–112 17. Dempster, A.P., Liard, N.M., Rubin, D.B.: Maximum Likelihood from Incomplete Data Via the EM Algorithm. J. R. Statist. Soc. B 39 (1977) 1–38 18. Port, S.C.: Theoretical Probability for Applications. John Wiley & Sons (1994)