IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 2, FEBRUARY 2008
1
2
Discrete-Time Expectation Maximization Algorithms for Markov-Modulated Poisson Processes Robert J. Elliott and W. P. Malcolm
3
Abstract—In this paper, we consider parameter estimation Markov-modulated Poisson processes via robust filtering and smoothing techniques. Using the expectation maximization algorithm framework, our filters and smoothers can be applied to estimate the parameters of our model in either an online configuration or an offline configuration. Further, our estimator dynamics do not involve stochastic integrals and our new formulas, in terms of time integrals, are easily discretized, and are written in numerically stable forms in W. P. Malcolm, R. J. Elliott, and J. van der Hoek, “On the numerical stability of time-discretized state estimation via clark transformations,” presented at the IEEE Conf. Decision Control, Mauii, HI, Dec. 2003.
16 17
Index Terms—Change of measure, counting processes, expectation maximization (EM) algorithm, martingales.
I. INTRODUCTION
18 19 Q1 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
HE WELL-KNOWN expectation maximization (EM) algorithm [8], [17] provides a scheme for solving a problem common in signal processing: estimating the parameters of a probability distribution for a known, partially observed dynamical system. This problem has received considerable attention for common signal models, such as the discrete-time Gaus-Markov model or the observation of a Markov process through a Brownian motion, [10], [24]. In this paper, we propose EM algorithms for the so-called Markov-modulated Poisson process (MMPP). A MMPP is conditionally a Poisson counting process, whose rate of arrivals depends upon the state of an indirectly observed Markov chain. These models have enjoyed many successful applications in queueing theory, and more recently, have been studied in the context of packet traffic estimation, and biomedical and optical-signal processing. Since our hidden-state process models are continuous-time Markov chains, the parameter estimation problem we consider, concerns computing estimates for the rate matrix of the Markov chain and the vector of Poisson intensities for the observation process. Traditionally, the EM algorithm is implemented by maximizing a log-likelihood function over a parameter space [11], [21], [22]. In some applications, this approach can lead to
T
technical difficulties. For example, the form of the log-likelihood function could be complicated or the operation of maximization of this function might be difficult. The implementations of the EM algorithms we present are the so-called filter-based and smoother-based EM algorithm [4], [10]. In the filter-based scheme, the parameter estimates are computed online by running a set of four recursive filters whose only storage requirements are previous estimates. Adapting the transformation techniques introduced by Clark [1], we compute the so-called robust versions of these filter, where the observation processes appear as parameters rather than as stochastic integrators. These formulations have been shown to have some numerical advantages [16]. Our smoother-based EM algorithm exploits a type of identity between the forward robust filter and its reverse-time counterpart. Smoothed estimates are obtained without recourse to stochastic integration. The paper is organized as follows. In Section II, the signal models for the state and observation processes are defined; our reference probability measure is also defined in this section. In Section III, we briefly recall the EM algorithm and compute a filter-based EM algorithm for MMPPs. In this section, we also compute robust filter dynamics that do not include stochastic integrals. In Section IV, we compute a robust smoother-based EM algorithm for an MMPP. Finally, in Section V, we compute a discrete-time data-recursive smoother-based EM algorithm for an MMPP.
66
II. DYNAMICS AND REFERENCE PROBABILITY
67
Initially, we suppose that all processes are defined on the measurable space (Ω, F) with probability measure P .
69
A. State Process Dynamics
70
IE E Pr E oo f
4 5 6 7 8 9 10 11 12 13 14 15
20
1
Manuscript received Jun 7, 2004; revised December 9, 2005 and December 1, 2006. Recommended by Associate Editor xxxx. The work of W. P. Malcolm was supported by the National Information and Communication Technology (ICT) Australia under the Grant given by the Australian Government’s Department of Communications, Information Technology and the Arts and the Australian Research Council through Backing Australia’s Ability and the ICT Centre of Excellence program. R. J. Elliott was with the Haskayne School of Business, University of Calgary, Calgary, AB T2N 1N4, Canada. He is now with the University of Alberta, Edmonton, AB T6G 2E1, Canada (e-mail:
[email protected]). W. P. Malcolm is with the Statistical Machine Learning Program of the National Information and Communication Technology (ICT) Australia (NICTA), Canberra, A.C.T. 2601, Australia (e-mail:
[email protected]). Digital Object Identifier 10.1109/TAC.2007.914305
Suppose that the state process X = {Xt , 0 ≤ t} is a continuous-time finite-state Markov chain with rate matrix A and an initial probability distribution p0 . We now use the well-known canonical representation for a Markov chain, that is, without loss of generality, the state space of X is L = {e1 , e2 , . . . , en }, where ei denotes a column vector in Rn with unity in the ith position and zero elsewhere. The dynamics for this process are Xt = X0 +
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
68
71 72 73 74 75 76 77 78
t
A Xu du + Mt .
(2.1)
0
Here, M is a (σ{Xt , 0 ≤ t}, P )-martingale and the matrix A ∈ Rn ×n is a rate matrix for the process X.
0018-9286/$25.00 © 2007 IEEE
79 80
2
81 82 83
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 2, FEBRUARY 2008
B. Observation Process Dynamics Suppose that the state process X is observed through a counting process whose Doob–Meyer decomposition is t Nt = Xu , λdu + Vt . (2.2) 0
86
Here, V is a (σ{Nu , 0 ≤ u ≤ t}, P )-martingale, ·, · denotes an inner product, and λ ∈ Rn+ is a vector of n nonnegative Poisson intensities. Our filtrations are given by ∆
Ft = {Ft },
where Ft = σ{Xu ; 0 ≤ u ≤ t} ∆
Y0,t = {Y0,t }, G0,t = G0,t , 87 88 89 90 91 92 93 94
95
where Y0,t = σ{Nu ; 0 ≤ u ≤ t}
C. Reference Probability We define a probability measure P † on the space (Ω, F) such that, under P † , the following two conditions hold. 1) The state process X is a Markov process with intensity matrix A and initial probability distribution p0 . 2) The observation process N is a standard Poisson process, that is, N has a fixed intensity of unity. The real-world probability measure P is defined by setting dP = Λ0,t (2.6) dP † G0 , t
Λ0,t =
Xu , λ∆ N u exp
=1+ 96
t
Λu − 0
Here,
(1 − Xu , λ)du
Xu , λ − 1 dNu − du .
(2.7)
99 100 101 102
103
(2.8)
Lemma 1: Under the measure P , the dynamics for the Markov process X are unchanged and given by (2.1). A proof of Lemma 1 is given in the Appendix. Further detail on the theory of Girsanov’s theorem and its application to estimation problems for stochastic dynamical systems can be found in the texts [2] and [3]. III. FILTER-BASED EM ALGORITHM
104
A. EM Algorithm
105
The EM algorithm is a two-step iterative process for computing maximum likelihood (ML) estimates. This process is usually terminated when some imposed measure of convergence for the sequence of maximum likelihood estimators (MLEs) is satisfied. Let θ index a given family of probability measures Pθ , where θ ∈ Θ. All such measures Pθ defined on a measurable
106 107 108 109 110
117
The so-called filter-based form of the EM algorithm for a continuous-time Markov chain observed in Brownian motion was presented in [4] and a robust version is given in [10]. In this paper, we develop a version of the techniques used in [10] for parameter estimation with MMPPs. This method is based essentially on four quantities, each concerning the indirectly observed Markov process X and each computed by using the information up to and including time t. We now list the four quantities of interest for the filter-based EM algorithm. 1) Xt , the state of the Markov chain. We are interested in E[Xt |Yt ]. By Bayes’ Theorem this is E † [Λt Xt |Y0,t ]/E † [Λt |Y0,t ]. Write
∆ (3.11) qt = E † Λt Xt |Y0,t .
118
Then,
+
119 120 121 122 123 124 125 126 127 128 129
130
t
Aqu du
0
t
diag λ, e −1 dNu −du ∈ Rn .
(3.12)
Here, diag λ, e − 1
↓0
97
116
0
∆
98
115
B. State Estimation Filters
∆Nτ = Nτ − lim Nτ − = Nτ − N τ − .
113 114
(3.10)
θ ∈Θ
qt = q0 +
0
0< u ≤t
t
111 112
(2.4)
∆
θ τ +1 ∈ argmax Q(θ, θ∗ ).
(2.3)
where G0,t = σ{Nu , Xu ; 0 ≤ u ≤ t}. (2.5)
where
2) Maximization step: Maximize Q(θ, θ∗ ) over the space Θ
IE E Pr E oo f
84 85
space (Ω, F) are assumed absolutely continuous with respect to a fixed probability measure P . Suppose Y ⊂ F. The two iterative steps in the EM algorithm are as follows. 1) Expectation step: Fix θ∗ = θ τ , then compute Q(·, θ∗ ), where dP θ Q(θ, θ∗ ) = Eθ ∗ log |Y . (3.9) dPθ ∗
λ, e − 1 1 λ, e2 − 1 = .. .
131
(3.13)
λ, en − 1
and A is the rate matrix for the process X. The unnormalized probability q is converted toits corresponding normalized probability by noting that ni=1 Xt , ei = 1, so E † [Λt |Yt ] = qt , 1. Here, 1 = (1, 1, . . . , 1) ∈ Rn . Therefore, qt , ei . P (Xt = ei |Y0,t ) = n =1 qt , e
133 134 135 136
(3.14)
A proof of (3.12) is given in the Appendix. 2) Jti , the cumulative sojourn time spent by the process X in state ei is t Jti = Xu , ei du. (3.15) 0
132
137 138 139
ELLIOTT AND MALCOLM: DISCRETE-TIME EM ALGORITHMS FOR MMPP
140 141
(j,i)
3) Nt , the number of transitions ei → ej of X, where i = j, up to time t is t (j,i) Nt = Xu − , ei dXu , ej . (3.16)
3
Theoremm 2: The vector-valued process σ(N (j,i) X) ∈ Rn satisfies the stochastic integral equation t (j,i) σ(Nt Xt ) = Aσ(Nu(j,i) Xu )du
4) Git , the level integrals for the state ei , is t i Gt = Xu , ei dNu .
144
145 146
E † [Λ0,t Ht |Y0,t ] . E † [Λ0,t |Y0,t ]
(3.17)
+
148 149
0
(j,i) × σ(Nu − Xu − ) dNu − du .
E[GiT |Y0,T ] σ(GiT )
λτ +1 , ei = = . E[JTi |Y0,T ] σ(JTi )
151 152 153 154 155 156 157 158 159
161 162 163 164 165 166
(3.22)
The fundamental idea behind the filter-based EM algorithm is to compute recursive filters for quantities such as (3.22), then marginalize the state variable X to evaluate the estimators given by (3.20) and (3.21). We now give recursive filters to estimate, respectively, the product quantities J i X, N (j,i) X, and Gi X. Theorem 1: The vector-valued process σ(J i X) ∈ Rn satisfies the stochastic integral equation t t σ(Jti Xt ) = Aσ(Jui Xu )du + qu , ei du ei 0
+ 0
0 t
diag λ, e − 1 σ(Jui − Xu − ) dNu − du . (3.23)
167
Here,
σ(J0i X0 )
= 0 and q is the solution of (3.12).
172
qu − , ei λ, ei dNu ei
0
t
+
diag λ, e − 1}σ(Giu − Xu − ) dNu − du .
(3.20)
The conditional expectations in equations (3.20) and (3.21) are computed using the previous (at pass τ ) parameter estimates for A and λ. ˆk , ei are computed by evalThe updates for [Aˆk ](i,j ) and λ uating the expectations in (3.20) and (3.21), respectively. However, it is, in general, not possible to compute recursive dynamics for the processes J i , N (j,i) , and Gi . It is, however, possible to compute dynamics for the associated product quantities (j,i) σ(Jti Xt ), σ(Nt Xt ), and σ(Git Xt ), where, for example, σ(Git Xt ) = E † [Λt Git Xt |Yt ] ∈ Rn .
160
t
+
0
(3.21)
170 171
0
(3.19)
(j,i)
and
(3.24)
Here, σ(N0 X0 ) = 0 and q is the solution of (3.12). Theorem 3: The vector-valued process σ(Gi X) ∈ Rn satisfies the stochastic integral equation t i σ(Gt Xt ) = Aσ(Giu Xu )du
Indexing the sequence of passes of the EM algorithm by τ = 1, 2, 3 . . ., the update formulas for the parameter estimates are as follows: i,j
τ +1 ](i,j ) = E[NT |Y0,T ] = σ(NT ) [A E[JTi |Y0,T ] σ(JTi )
150
diag λ, e − 1
IE E Pr E oo f
147
t
(j,i)
(3.18)
Here, E † [·] denotes an expectation taken under the measure P † . Write σ(Ht ) = E † [Λ0,t Ht |Y0,t ].
qu , ei Aei , ej du ej
0
Using Bayes’ Theorem, if H = {Ht , 0 ≤ t} is any G adapted process E[Ht |Y0,t ] =
t
+
0 143
169
0
0 142
168
(3.25)
Here, σ(Gi0 X0 ) = 0 and q is the solution of (3.12). A proof of Theorem 3 is given in the Appendix. Theorems 1 and 2 can be readily proven by similar means. By using the solutions of (3.23), (3.24), and (3.25), the updates for the parameter estimates are given by
τ +1 ](i,j ) = [A
173 174 175 176 177
(j,i)
σ(NT XT ), 1 σ(JTi XT ), 1
(3.26)
and
σ(Git XT ), 1
λτ +1 , ei = . σ(JTi XT ), 1
178
(3.27)
C. Robust State Estimation Filters
179
Each of the dynamics given by (3.23)–(3.25) contain stochastic Lebesgue–Stieltjes integral terms. These stochastic integrals, with respect to the observation process N , can be eliminated by using a version of a gauge transformation due to Clark [1]. Consider the diagonal matrix ∆ Γt = diag γti ∈ Rn ×n . (3.28)
180
∆ Here, γti = exp (1 − λ, ei )t λ, ei N t with γ0i = 0. Note that the matrix Γ−1 o rule, one can t is nonsingular. Using the Itˆ show that t −1 Γt = diag λ, e − 1 Γ−1 u du 0
+ 0
t
−1 Γ−1 − 1 dNu . u − diag λ, e
(3.29)
181 182 183 184
185 186 187
4
188
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 2, FEBRUARY 2008
With q t = Γ−1 t qt , we have ∆
qt = q0 +
t
Γ−1 u AΓu q u du.
(3.30)
Finally, our dynamics for σ(Git Xt ) read t i i σ(Gt Xt ) = Γ−1 u AΓu σ(Gu Xu )du 0
0 189 190
Equation (3.30) was established in [13]. For any F-adapted integrable process H, we write σ(H) = Γ−1 t σ(Ht ).
191 192 193 194 195
+ q t , ei Nt ei −
(3.31)
Now, our objective is to compute filters to estimate the product processes Gi X, N (j,i) X, and J i X. Let us first consider the process σ(Gi X). Dynamics for the gauge transformed process i σ(Git Xt ) = Γ−1 t σ(Gt Xt ) can be computed by applying the product rule
0
0
and (j,i)
σ(Nt
t
t + diag λ, e − 1 σ(Giu Xu )du 0
t + diag λ, e −1 − 1 σ(Giu − Xu − )dNu 0
t + diag λ, e −1 − 1 q u , ei λ, ei dNu ei 0
t + diag λ, e −1 − 1 0
× diag λ, e −1
0
198
Several stochastic integrals in (3.33) cancel, noting diag λ, e − 1 + diag λ, e −1 − 1 +diag λ, e −1 − 1 diag λ, e −1 =0 ∈ Rn ×n giving σ(Git Xt ) =
t
i Γ−1 u AΓu σ(Gu Xu )du +
0
199 200
0
t
(3.34)
q u , ei dNu ei .
(3.35) The stochastic integral in (3.35) can be simplified by stochastic integration by parts t t q u , ei dNu ei = q t , ei Nt − Nu dq u , ei . (3.36) 0
t
q u , ei Aei , ej du ej .
0
(3.39)
206
For all time discretizations, we will consider a partition on an interval [0, T ] and write (K ) ∆ Π[0,T ] = 0 = t0 , t1 , . . . , tK = T . (3.40) Here, the partition is strict, that is, t0 < t1 < · · · < tK = T . To denote the mesh of the partition, we write (K ) Π[0,T ]
= max {tk − tk −1 }. 1≤k ≤K
∆
q t k ≈ q t k −1 + Γ−1 t k −1 AΓt k −1 q t k −1 ∆(k −1,k )
qt k = Γt k q t k ≈
Γt k Γ−1 t k −1
I + ∆(k −1,k ) A qt k −1 .
This suggests the recursion ∆ q k = Γk Γ−1
k −1 . k −1 I + ∆(k −1,k ) A q
207 208
209 210
(3.41)
For brevity, we shall use the notation ξk = ξt k , where ξk denotes a process ξ at a time point tk . Further, we write ∆(k −1,k ) = tk − tk −1 . Approximating the integral in (3.30), we get
so
− 1 σ(Giu − Xu − ) dNu .
(3.33)
197
(3.38)
D. Discrete-Time Filters
t + diag λ, e −1 σ(Giu − Xu − )(dNu −du) 0
204
0
t + q u , ei λ, ei dNu ei 0
203
(j,i) Γ−1 Xu )du u AΓu σ(Nu
Xt ) =
IE E Pr E oo f
0
202
205
(3.32)
The result of this calculation is t −1 i i Γt σ(Gt Xt ) = Γ−1 u AΓu σ(Gu Xu )du
(3.37)
Similarly, one can apply the product rule to compute process dy(j,i) namics for the quantities σ(Jti Xt ) and σ(Nt Xt ). The results of these calculations are, respectively, t t i −1 i σ(Jt Xt ) = Γu AΓu σ(Ju Xu )du + q u , ei du ei
+ 196
Nu dq u , ei ei .
0
i −1 i −1 i d(Γ−1 t σ(Gt Xt )) = Γt− d(σ(Gt Xt )) + dΓt σ(Gt− Xt− ) i + ∆Γ−1 t ∆σ(Gt Xt ).
t
201
211 212 213
(3.42) 214
(3.43) 215
(3.44)
Here, q denotes an estimate of the unnormalized probability generated by the suboptimal discrete-time recursion at (3.43). Remark 1: An important feature of the filter formulation at (3.44) is that the sampling interval or ∆(k −1,k ) can be chosen to ensure a certain type of numerical stability. Here, numerical stability is taken to mean q, ei ≥ 0 for all i ∈ {1, 2, . . . , n}. The details of this property are given in [16]. Writing the dynamics given by (3.37) recursively at sampling instants tk and tk −1 , we get
216 217 218 219 220 221 222 223 224
σ(Git k
Xt k ) =
σ(Git k −1 Xt k −1 )
tk
+ t k −1
i Γ−1 u AΓu σ(Gu Xu )du
ELLIOTT AND MALCOLM: DISCRETE-TIME EM ALGORITHMS FOR MMPP
+ q t k , ei Nt k ei − q t k −1 , ei Nt k −1 ei tk − Nu dq u , ei ei . (3.45) t k −1 225
E. Discrete-Time Filter-Based EM Algorithm
233
Summarizing the results from the previous sections, our filterbased EM algorithm reads
235
Initialization
Making an Euler–Maruyama1 approximation, we have tk tk Nu dq u , ei ei = Nu Γ−1 u AΓu q u du, ei ei t k −1
5
Step 1
t k −1
≈ Nt k −1 Γ−1 t k −1 Aqt k −1 ei ∆(k −1,k ) ei
Step 2
234
∀(i, j) ∈ {(1, 1), (1, 2), . . . , (n, n)},
0 ](i,j ) , for each i ∈ {1, 2, . . . , n} Choose [A choose
λ, ei . Using (3.26) and (3.27), compute
τ +1 ]i,j and
λτ +1 . the MLEs, [A Decide to stop or continue from step 2.
(3.46) 226
and with some algebraic manipulation q t k , ei Nt k ei − q t k −1 , ei Nt k −1 ei = Γ−1 t k −1 qt k −1 , ei (Nt k − Nt k −1 )ei +
227
∆(k −1,k ) Γ−1 t k −1 Aqt k −1 , ei Nt k
ei
(3.47)
we see that σ(Gik Xk ) ≈ σ(Gik −1 Xk −1 ) + Γ−1 qk −1 , ei (Nk − Nk −1 )ei k −1
+ ∆(k −1,k ) Γ−1 qk −1 , ei Nk ei k −1 A
− Nk −1 Γ−1 qk −1 , ei ∆(k −1,k ) ei . k −1 A
228 229
(3.48)
Now, by multiplying both sides of (3.48) on the left-hand side by the matrix Γk , we get i σ(Gik Xk ) ≈ Γk Γ−1 k −1 σ(Gk −1 Xk −1 )
i + Γk Γ−1 k −1 Aσ(Gk −1 Xk −1 )∆(k −1,k )
+
Γk Γ−1 qk −1 , ei (Nk k −1
+
Γk Γ−1 qk −1 , ei (Nk k −1 A
237
249
A. Smoothed State Estimation for the Process X
250
We first briefly recall the state estimation MMPP smoother presented in [14]. For a smoothed estimate for the process X ∈ Rn , we wish to evaluate the expectation E[Xt |Y0,T ], where 0 ≤ t ≤ T . By Bayes’ rule [3], we have
251
E[Xt |Y0,T ] =
− Nk −1 )ei .
qk −1 , ei Aei , ej ∆(k −1,k ) ei +Γk Γ−1 k −1
232
σ
(Jki Xk )
=
Γk Γ−1 k −1
I + ∆(k −1,k ) A σ
(Jki −1 Xk −1 )
qk −1 , ei ei . + Γk Γ−1 k −1
1 We
E [Λ0,T Xt |Y0,T ] . E † [Λ0,T |Y0,T ]
(3.52)
will use this particular approximation throughout this paper, however, other approximations could also be used.
241 242 243 244 245 246 247 248
252 253 254
(4.53) 255
∆
= E † [Λ0,t Λt,T Xt |Y0,T ]
= E † [E † [Λ0,t Λt,T Xt |Y0,T ∨ Ft ]|Y0,T ]
= E † [Λ0,t Xt E † [Λt,T |Y0,T ∨ Ft |Y0,T ].
(4.54)
†
Under the measure P , X is a Markov process, so the inner expectation in the previous line of (4.54) is †
†
E [Λt,T |Y0,T ∨ Ft ] = E [Λt,T |Y0,T ∨ σ{Xt }]. ∆ i vt,T = E † [Λt,T
|Y0,T and Xt = ei ].
256 257
(4.55)
Write
(3.15)
239 240
rt = E † [Λ0,T Xt |Y0,T ]
(3.50)
After similar calculations, the remaining discretized filters read (j,i) (j,i) σ
(Nk Xk ) = Γk Γ−1
(Nk −1 Xk −1 ) k −1 I + ∆(k −1,k ) A σ
238
†
Consider the numerator of (4.53)
Our estimator of the quantity σ(Gik Xk ) has dynamics ∆ σ
(Gik Xk ) = Γk Γ−1
(Gik −1 Xk −1 ) k −1 I + ∆(k −1,k ) A σ qk −1 , ei + ∆(k −1,k ) A
qk −1 , ei + Γk Γ−1 k −1
× (Nk − Nk −1 )ei .
231
In many implementations of the EM algorithm, for example, [24] and [29], the expectation step is completed with smoothed rather than (online) filtered estimates. Typically, the smoothing scheme used is the so-called “fixed interval smoother.” Computing smoothing schemes for MMPPs can be particularly difficult [23], [26]. One source of this difficulty is the task of developing backwards dynamics. This task usually leads to constructing stochastic integrals evolving backward in time. However, the approach we use to develop smoothing algorithms completely avoids these difficulties. To compute our smoothers we exploit a special identity between forward and backward robust dynamics, and as a consequence, do not need to consider the backward stochastic integration at all.
− Nk −1 )ei
(3.49)
230
236
IE E Pr E oo f
i + Γ−1 k −1 AΓk −1 σ(Gk −1 Xk −1 )∆(k −1,k )
IV. SMOOTHER-BASED EM ALGORITHM FOR MMPPS
258
(4.56)
Omitting further calculations, it can be shown [14] that
259
rt = qt , e1 vt,T , e1 e1 + qt , e2 vt,T , e2 e2 + · · · + qt , em vt,T , en en ∈ Rn . The normalized smoothed-state estimate of X is then rt . E[Xt |Y0,t ] = rt , 1
(4.57) 260
(4.58)
6
261
IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 53, NO. 2, FEBRUARY 2008
Note that
Now, rt , 1 = qt , vt,T
280
σ(Git Xt ), v t,T = σ (Git Xt ), v t,T + Nt q t , ei v t,T , ei
= E † [Λ0,T Xt , 1|Y0,T ] = E † [Λ0,T |Y0,T ] 262
(4.69) (4.59)
and
281
Nt q t , ei v t,T , ei = Nt qt , ei vt,T , ei .
is independent of t. Therefore d d rt , 1 = qt , vt,T dt dt = dq t , v t,T + q t , dv t,T
From the dynamics of σ (Git Xt ), we have d σ (Git
265 266 267 268 269 270
for all t ∈ [0, T ].
∆
so that
v t,T = 1 +
i = Γ−1 t AΓt σ(Gt Xt ), v t,T dt
− Nt Γ−1 t AΓt q t , ei ei , v t,T dt − σ(Git Xt )−q t , ei Nt ei , Γt A Γ−1 t v t,T dt = −Nt Γ−1 t AΓt q t , ei ei , v t,T dt + Nt q t , ei Γ−1 t AΓt ei , v t,T dt
v T ,T = vT ,T = 1
(4.62)
276 277
T
Γu A Γ−1 u v t,T du.
(4.63)
Nu q u , ei Γ−1 u AΓu ei , v u ,T du.
(4.72)
Therefore, σ(GiT
XT ), vT ,T =
Further, using the time discretization of (3.40) tk v k −1,T = v k ,T + Γu A Γ−1 u v u ,T du
so, our suboptimal estimator v ≈ v has dynamics ∆
k ,T . v k −1,T = Γ−1 k −1 Γk I + ∆(k −1,k ) A v
(4.64)
T
Nu q u , ei Γ−1 u AΓu ei , v u ,T du
+ NT qT , ei vT ,T , ei T =− Nu Aqu , ei ei , vu ,T du
(4.65)
0
T
+
Nu qu , ei ei , A vu ,T du
0
+ NT qT , ei vT ,T , ei .
(4.66)
σ (Git Xt ) = σ(Git Xt ) − q t , ei Nt ei .
(4.67)
Now, define
By using similar calculations, one can also show that T i σ(JT XT ), v T ,T = qu , ei vu ,T , ei dt
(j,i)
i dσ (Git Xt ) = Γ−1 t AΓt σ(Gt Xt ) dt
σ(NT (4.68)
(4.73) 285
(4.74)
0
and
Then − Nt Γ−1 t AΓt q t , ei ei dt.
XT ), v T
0
Xt ), v t,T .
∆
284
0
+
Following the same strategy as before, we consider the identity σ(Git
σ(GiT
= σ (GiT XT ), v T + NT q T , ei v T , ei T =− Nu Γ−1 u AΓu q u , ei ei , v u ,T du
B. Smoothers for the Quantities Nti , Jti , and Git
=
279
T
0
i σ(Git Xt ), vt,T = Γ−1 t σ(Gt Xt ), Γt vt,T
278
0
+
≈ v k ,T + Γk A Γ−1 k v k ,T ∆(k −1,k ) ,
275
283
σ (GiT XT ), v T ,T T =− Nu Γ−1 u AΓu q u , ei ei , v u ,T du
(4.61)
t k −1
274
(4.71)
i.e.,
t
273
− σ (Git Xt ), Γt AΓ−1 t v t,T dt
That is, v t,T = Γt vt,T . Using (4.60), one can show that dv t,T = −Γt A Γ−1 t v t,T , dt
272
− Nt Γ−1 t AΓt q t , ei ei , v t,T dt
= 0. (4.60) The vector vt,T = vt,T , e1 , vt,T , e2 , . . . , vt,T , en incorporates the extra information obtained from the observations between t and T . Computing dynamics for v can be difficult [18], [19]. However, by exploiting a special identity between the forward dynamics and the corresponding backward, process v, one can directly compute robust dynamics for the process v. What we must do is consider the process v, such that the following identity holds q t , v t,T = Γ−1 t qt , Γt vt,T = qt , vt,T ,
271
Xt ), v t,T
IE E Pr E oo f
264
282
i = Γ−1 t AΓt σ(Gt Xt ), v t,T dt
= Γ−1 t AΓt q t , v t,T + q t , dv t,T
263
(4.70)
XT ), v T ,T = 0
286 T
A ei , ej qu , ei vu ,T , ej du. (4.75)
ELLIOTT AND MALCOLM: DISCRETE-TIME EM ALGORITHMS FOR MMPP
287
C. Smoother-Based EM Algorithm
288
Recalling (3.20) and (3.21), our smoother-based update equations are T
τ +1 ](i,j ) = [A
τ ](i,j ) 0 qu , ei vu ,T , ej du (4.76) [A T q , e v , e du u i u ,T i 0
289
290
and
λτ +1 , ei =
T
Nu qu , ei ei , A vu ,T du T 0 qu , ei vu ,T , ei du T Nu Aqu , ei ei , vu ,T du − 0 T 0 qu , ei vu ,T , ei du 0
NT qT , ei vT ,T , ei + T . q , e v , e du u i u ,T i 0 291
(4.77)
293
Suppose that one observes data on the set [0, T ] and parameter estimates are computed by using these data. Further, suppose one receives a subsequent observation data on the set [T, T ], where T > T . What we would like to do is incorporate the new data on [T, T ] so as to reestimate the model parameters, but without complete recalculation from the origin. To utilize the information on [T, T ], we consider a time discretization on the total interval [0, T ] ∪ [T, T ], that is,
296 297 298 299 300
0 = t 0 < t1 · · · < tK = T < 301 302
t0
