656
IEEE TRANSACTIONS ON CONTROL SYSTEMS TECHNOLOGY, VOL. 19, NO. 3, MAY 2011
Identification of Time-Varying Systems Using Multi-Wavelet Basis Functions Yang Li, Hua-liang Wei, and S. A. Billings
Abstract—This brief introduces a new parametric modelling and identification method for linear time-varying systems using a block least mean square (LMS) approach where the time-varying parameters are approximated using multi-wavelet basis functions. This approach can be applied to track rapidly or even sharply varying processes and is developed by combining wavelet approximation theory with a block LMS algorithm. Numerical examples are provided to show the effectiveness of the proposed method for dealing with severely nonstationary processes. Application of the proposed approach to a real mechanical system indicates better tracking capability of the multi-wavelet basis function algorithm compared with the normalized least squares or recursive least squares routines. Index Terms—B-splines basis functions, block least mean squares (LMS), normalized least mean squares (LMS), parameter estimation, recursive least squares (RLS), system identification, time variation.
I. INTRODUCTION ANY processes are inherently time-varying and can not effectively be characterized using time invariant models. Modelling and analysis of time-varying systems is often a challenging problem. One feature of time-varying systems is that such signals contain nonstationary transient events. One approach to characterize such non-stationary processes is to employ time-varying parametric models for example the time-varying autoregressive with an exogenous (TVARX) model [1], or simply the time-varying autoregressive (TVAR) model [2]. Two main classes of methods can be used to resolve the TVARX and TVAR model estimation problem. The first uses recursive estimation of the time-varying coefficients and the second constrains the evolution of the coefficients to be linear or nonlinear combinations of some basis functions with appropriate properties. These approaches have been called stochastic and deterministic regression approaches, respectively [3]. Some of the most popular recursive algorithms are the least mean square (LMS) algorithm, the recursive least square (RLS) algorithm [4], the Kalman filter and the random walk Kalman filter (RWKF) algorithms [5]–[7]. The basis function expansion and regression method is a deterministic parametric
M
modelling approach, where the associated time-varying coefficients are expanded as a finite sequence of predetermined basis functions [8]–[10]. Generally, these coefficients are expressed using a linear or nonlinear combination of a finite number of basis functions. The problem is then reduced to time invariant coefficient estimation, where the unknown adjustable model parameters are those involved in the basis expansion. Hence, the initial time-varying modelling problem is simplified to deterministic regression selection and parameter estimation. An attractive approach is to expand the time-varying coefficients using wavelets as the basis functions. Wavelets have been proved to be a valuable tool for signal processing and have been shown to possess excellent linear or nonlinear approximation properties which outperform many other approximation schemes and are well suited for approximating general non-stationary signals, even those with very sharp or abrupt discontinuities. Wavelets have also successfully been used in system identification and modelling [11]–[14]. In this brief, a new wavelet multi-resolution parametric modelling and identification technique for the identification of systems with time-varying parameters is proposed, where the associated time dependent parameters are approximated using a set of multi-wavelet basis functions, which transforms the timevarying identification problem into a time-invariant parametric expansion. The identification of the model parameters can then be achieved by adopting a block LMS algorithm. One advantage of the proposed approach, which combines wavelet approximation theory with a block LMS algorithm, is that the new wavelet based algorithm can be used to track very rapidly or even sharply varying processes. The novel approach proposed can thus track rapid time variation and is more suitable for the estimation of process parameters of inherently non-stationary processes. A multi-wavelet basis function approach is used because of the ability to capture the signals characteristics at different scales. Two examples, one for a synthetic data set and a second for a real mechanical system are given to illustrate the capability and efficacy of the proposed method. It is shown that the proposed method can produce much better tracking performance compared with traditional LMS and RLS approaches. II. METHODOLOGY
Manuscript received March 30, 2010; accepted May 31, 2010. Manuscript received in final form June 01, 2010. Date of publication July 01, 2010; date of current version April 15, 2011. Recommended by Associate Editor A. Alessandri. The work of Y. Li was supported by the University of Sheffield under the scholarship scheme. This work was supported by the Engineering and Physical Sciences Research Council (EPSRC), U.K. and the European Research Council (ERC). The authors are with the Department of Automatic Control and Systems Engineering, the University of Sheffield, Sheffield, S1 3JD, U.K. (e-mail:
[email protected];
[email protected];
[email protected]. uk). Color versions of one or more of the figures in this brief are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TCST.2010.2052257
Consider an input-output relationship of a TVARX process which is described by the following equation: (1) where and are the sampled measurable input, output, and and are the time-varying paprediction error signals, rameters to be determined, and are the maximum model orders, and represents discrete time. The proposed method exand onto multipands the time varying parameters
1063-6536/$26.00 © 2010 IEEE
LI et al.: IDENTIFICATION OF TIME-VARYING SYSTEMS USING MULTI-WAVELET BASIS FUNCTIONS
wavelet basis functions for the following expressions hold:
, such that
(2) where , are wavelet basis functions, and are the expansion parameters, is the maximum number of basis sequences. Substituting (2) into (1), yields (3)
(3) Once basis functions have been chosen (see Section III), new variables can be defined as follows:
(4) Substituting (4) into (3) yields
(5)
657
and are the scaled and translated version of and the mother wavelet , , the scaling function and are the wavelet decomposition coefficients at the scale level and , respectively. The first summation indicates a low at the scale level resolution or coarse approximation of in (9). For each , a finer or higher resolution function including more detail of is added in the second summation. Like and in any regression representation of functions, can be interpreted as regressions. Equation (9) can be reduced to
where
(10) is the approximation at level where and is the detail at level . When the , higher resolution components make little contribution to the function can be approximated by a truncated multi-resoluas given in (10). tion wavelet expansion from up to Note that the multi-resolution wavelet of a signal is not unique and depends on the type of mother wavelet and scaling function that are used. In [18], properties of various types of wavelets including orthogonality, support, regularity and time-frequency localization window were discussed for various wavelet types. It was concluded that B-spline mother wavelets could be justified for wavelet-based high resolution image sequences. Cardinal B-splines are an important class of basis functions that can form multi-resolution wavelet decompositions [16]. The first order cardinal B-spline is the well-known Haar function defined as
Model (5) can be rewritten as (6) where
(7) and (8) and are the regression vector and coefficient vector, respectively, and the upper script “ ” indicates the transpose of a vector or a matrix. Equations (5) and (6) show that the TVARX model can now be treated as a time invariant model, since and are now time-invariant. III. MULTI-WAVELET BASIS FUNCTIONS From wavelet theory [15], [16], a signal can be decomposed into components spanned by the scaling and shifting wavelet basis functions at different resolutions. As defined by [17], any finite energy function can be arbitrarily approximated by (9)
otherwise.
(11)
, The second-, third-, and fourth-order cardinal B-splines , and are given in [19]. For detailed discussions on cardinal B-splines, the associated wavelets and details of the excellent approximation properties, see Chui [16]. One attractive feature of cardinal B-splines is that these functions are completely supported, and this property enables the operation of the multi-resolution decomposition (9) to be much more convenient. For example, the th order B-spline is defined , with the scale and shift indices and , for the family on of the functions (12) that is to be approximated with Assume that the function , then for any given decomposition (9) is defined within and scale index (resolution level) , based on , the effective values for the shift index , are with the restricted to the collection . Note that while the first and second order B-splines and are non-smooth piecewise functions, which would perform well for signals with sharp transients and burst-like spikes, B-splines of higher order would work well on smoothly changing signals. Motivated by this consideration, this study
658
IEEE TRANSACTIONS ON CONTROL SYSTEMS TECHNOLOGY, VOL. 19, NO. 3, MAY 2011
proposes using multi-wavelet basis functions for TVARX model identification. Two examples of the new multi-wavelet based algorithm are given in the following. Take the B-splines of order from 1 to 4 as an example, and , consider the decomposition (10). Let 1, 2, 3, 4 and , . and in model (1) can The time varying coefficients then be approximated using a combination of functions from . Several criteria the families for the determination of were given in [8]. For the B-splines wavelet used in this study, the lower level and higher level in (10) can be chosen . However, the higher level can be chosen as larger than as possible provided that the computation effort is not a concern. Therefore, one such combination can be represented as
(13) where , , and is number of observations of the signal. Simulation results with a large number of experiments have shown that for most time, , and varying problems, the choice of work well to recover the time-varying coefficients. If, however, there is strong evidence that the time-dependent coefficients have sharp changes, then the inclusion of the first and second order B-splines would work well. The decomposition (13) can easily be converted into the form of (2), where the colis replaced by the union of lection the three families: , , and . Further derivation can then lead to the standard linear regression (5). Equations (13) and (5) indicate that the initial full regression (5) may involve a great number of free time-dependent time-varying parameters, and least squares type algorithms may fail to produce reliable results for such ill-posed problems. These problems, however, can easily be overcome by performing an effective online block LMS algorithm, the reand in (13) will sulting recursive coefficient estimates and then be used to recover the time-varying coefficients in the TVARX in model (1). IV. MODEL IDENTIFICATION AND PARAMETER ESTIMATION WITH A BLOCK LMS APPROACH The conventional LMS algorithm and normalized LMS [20], [21] have been proven to be very effective to deal with dynamic regression problems. However, the performance of these algorithms is sensitive to the selection of step sizes and additional noise. In this study we introduce a block LMS algorithm, Table II presents a summary of the block LMS algorithm, where
the step size is divided by the maximum eigenvalue of the correlation matrix for a potentially-faster algorithm convergence time. An important issue that needs to be considered in the design of a block adaptive filter is how to choose the block size . From Table II we observe that the operation of the block LMS algorithm holds true for any integer value of equal to or greater than unity. Nevertheless, the option of choosing the block size equal to the filter length [that is, the number of is referred time-varying parameter coefficients in model (1)] in most applications as block adaptive filtering. This choice has been justified by Clark et al. [22] based on the following ob, redundant operations are involved in servations. When the adaptive process, because then the estimation of the gradient vector (computed over points) uses more input information , some of the tap weights in than the filter itself. When the filter are wasted, because the sequence of tap inputs is not long enough to feed the whole filter. . For It thus appears that the most practical choice is , the block LMS algorithm simplifies to the normalized LMS algorithm, where is a scalar. For , Table I summarizes the BLMS algorithm, where is a square matrix. The block LMS approach leads to two significant advantages , potenover the conventional LMS algorithm: 1) for tially-faster convergence speeds for both correlated and whitened input data [22], and stable behavior for a known range of param, stability condition) independent eter values ( , of the input data correlation statistics [23], [24] and 2) for block processing of data samples, a block of samples of the filter input and desired output are collected and then processed together to obtain a block of output samples. A good measure of computational complexity in a block processing system is given by the number of operations required to process one block of data divided by the block length. An implementation of the block LMS algorithm is more computationally efficient. In this study, the block LMS algorithm described above is applied to refine and solve the regression (5). This includes a model identification and time-dependent time-varying parameter estimation. The resultant estimates will then be used and in the to recover the time-varying coefficients TVARX model (1). To determine the proper model size selection for the TVARX case given by (5), we exploited the fact that time-dependent or time-varying coefficients are expanded onto multi-wavelet basis functions to give a time invariant system. Hence, the modified generalized cross-validation (GCV) criteria [25] can be employed to account for the increased number of time invariant coefficients for the system (5). This modified criterion allowed us not only to select the proper model size, but also to choose an appropriate number of multi-wavelet basis functions onto which the time-varying coefficients are expanded. V. SIMULATION EXAMPLES To verify the performance of the multi-wavelet basis functions approach, two examples will be studied. The first is a simulated experiment with measurement signal-to-noise ratios (SNRs) are presented. Consider the following TVARX(1,1) model (14)
LI et al.: IDENTIFICATION OF TIME-VARYING SYSTEMS USING MULTI-WAVELET BASIS FUNCTIONS
659
TABLE I SUMMARY OF THE BLOCK LMS ALGORITHM
TABLE II A COMPARISON OF THE MODEL PERFORMANCE FOR EXAMPLE 1 (SNR = 19.4024 dB)
which was a pseudo-random binary sequence (PRBS) [26] where is a discrete white noise sequence. The system parameters are estimated using the normalized LMS approach, RLS approach andtheblockLMS approachesbasedonmulti-wavelet basisfunctions, respectively, so that these three methods can be compared. The time-varying parameter variations were designed to change in an abruptly varying manner as
, (15)
The process parameters ways and the output
and will be varied in different is observed for the system input
is a frequency rich signal. The input The PRBS input signal signal is of 1 second duration and the sampling frequency was 1000 Hz. The output is a nonstationary signal with a
660
IEEE TRANSACTIONS ON CONTROL SYSTEMS TECHNOLOGY, VOL. 19, NO. 3, MAY 2011
Fig. 1. Time-varying parameter estimation using a normalized LMS approach with L true parameter a t and (b) estimated and true parameter b t .
()
()
= 1, = 0:6 and SNR of 19.4024 dB for Example 1. (a) Estimated and
19.4024 dB. Fig. 1 shows the true and estimated values of paand respectively for the noise measurement rameters of 19.4024 dB using the normalized LMS algorithm , . The estimated parameters follow the true with parameter variations quite well. Fig. 2 shows the results of the and respectrue and estimated values for parameters tively with the same noise measurement of 19.4024 dB using the . The RLS apRLS algorithm with forgetting factor proach obtains smoother estimates but does not faithfully track the rapidly changing time-varying parameters. Fig. 3 shows the and results of true and estimated values for parameters respectively with the noise measurement of 19.4024 dB , using the new block LMS based algorithm with using multi-wavelet basis functions. The estimated parameters follow the true parameters variations extremely well picking up the abrupt changes very quickly. Estimates were calculated for the given time varying coefficients in (15). The standard deviations of the parameter estimates (with respect to the true parameters) and the statistics of the obtained results are presented in Table II. The mean absolute error (MAE) of the parameter estimates, with respect to the corresponding true values, are also estimated and shown in Table II. Compared with the normalized LMS estimates and RLS estimates, the variance for the multi-wavelet basis functions block LMS method estimates is much smaller. The mean absolute error is defined by
with the traditional normalized LMS and RLS approaches, the simulation results above really show that the new method based on multi-wavelet basis functions proposed in this brief is more adaptive and possesses much better tracking ability in that it still can track the time-varying trend of the parameters even with noise contamination.
(16)
(18)
represents the estimates of in model (1), and where is the length of the data. Table II statistically confirms the better performance of the block LMS multi-wavelet basis functions method. Compared
where , are the parameter measuring the strength of the parametric excitation, and the excitation is given , where , by 6.91 rad/s and 5.65 rad/s, the step time used is 0.06 , s; the initial conditions for the free vibrations are
VI. APPLICATION-TRACKING OF A REAL MECHANICAL SYSTEM The proposed modelling scheme has been applied to the analysis of a real mechanical system to illustrate the application and tracking ability of the proposed multi-wavelet basis function method based TVARX modelling approach. The second-order continuous linear system equation subject to a general excitaas formulated in [27] is given by tion (17) The parameters , , and represent the time-varying mass, damping, and stiffness of the system, respectively. The properties and parameters of system are described in [27]. An idealized model of damage characterized by an abrupt change in the stiffness is considered, which is to assess the suitability of the identification procedure to abrupt variations in the timedependent parameters. The damped Mathieu equation is given by
LI et al.: IDENTIFICATION OF TIME-VARYING SYSTEMS USING MULTI-WAVELET BASIS FUNCTIONS
Fig. 2. Time-varying parameter estimation using an RLS with forgetting factor parameter a t and (b) estimated and true parameter b t .
()
()
661
= 0:9 and SNR of 19.4024 dB for simulation example. (a) Estimated and true
Fig. 3. Time-varying parameter estimation using the block LMS approach based on multi-wavelet basis functions with L for simulation Example. (a) Estimated and true parameter a t and (b) estimated and true parameter b t .
()
()
. The abrupt stiffness changes are given as , for , , for , and for . the value of and is periodic. The difference parameters are equation for the system (18) using the approximations and can be approximated by
for for
(19) Equation (19) can be considered as a discrete model approximation of (18). The selection of the order of B-splines, with the
= 2, = 1 and SNR of 19.4024 dB
resolution level (scale index), is the same as in the previous example. Comparison with (1), the linear TVARX model (19) with , and can be estimated by the proposed approach, , then the stiffness estimation with value of the time-varying parameter in model (19) can be represented by (20) with The stiffness estimate of the time varying parameter the new approach is given in Fig. 4. compared with the result given in [27], the parameter estimate here follows the true parameter variation very well and traces the abrupt changes quickly.
662
IEEE TRANSACTIONS ON CONTROL SYSTEMS TECHNOLOGY, VOL. 19, NO. 3, MAY 2011
Fig. 4. Time-varying abrupt stiffness parameter estimation k (t) using the block LMS approach based on multi-wavelet basis functions with L = 2, = 0:78. Solid (blue) line, the dashed (red) line and the dashed (black) line indicate the true value, estimate value (no noise) and estimate value with SNR (5.9735 dB), respectively.
VII. CONCLUSION Time-varying parameters in ARX models have been estimated using a new multi-wavelet basis function approach with a block LMS algorithm introduced in this study, where the associated time-dependent coefficients are expanded using multi-wavelet basis functions. Parameter variations including abrupt or sharp changes have been considered. Performance measures of the estimated parameters have been calculated. The experimental results indicate that the new approach based on multi-wavelet basis functions with the block LMS algorithm gives much better results for fast and abrupt changing parameters than the method which uses the traditional normalized LMS algorithm or the RLS algorithm directly. Furthermore, from the results above, it can be concluded that time-varying systems can be modelled using a TVARX and the identification problem of modelling fast and abrupt changing time-varying parameters is possible with good accuracy. The identification procedure has been shown to be effective in tracking time evolutions of the unknown parameters. The wavelet method is especially powerful for non-stationary signal analysis. We advocate the use of a multi-wavelet basis functions to identify and model time-varying signals including mechanical systems because of the flexibility in capturing the signal characteristics at different scales. REFERENCES [1] H. Peng, K. Nakano, and H. Shioya, “Nonlinear predictive control using neural nets-based local linearization ARX model-stability and industrial application,” IEEE Trans. Control Syst. Technol., vol. 15, no. 1, pp. 130–143, Jan. 2007. [2] F. N. Chowdhury, B. Jiang, and C. M. Belcastro, “Reduction of false alarms in fault detection problems,” Int. J. Innovative Comput. Inf. Control, vol. 2, no. 3, pp. 481–490, Jun. 2006. [3] B. Jiang and F. N. Chowdhury, “Fault estimation and accommodation for linear MIMO discrete-time systems,” IEEE Trans. Control Syst. Technol., vol. 13, no. 3, pp. 493–499, May 2005.
[4] L. Ljung and S. Gunnarsson, “Adaptation and tracking in system identification—A survey,” Automatica, vol. 26, no. 1, pp. 7–21, Jan. 1990. [5] M. Niedzwiecki, “Identification of time-varying systems with abrupt parameters changes source,” Automatica, vol. 30, no. 3, pp. 447–459, Mar. 1994. [6] F. Morbidi, A. Garulli, D. Prattichizzo, C. Rizzo, and S. Rossi, “Application of Kalman filter to remove TMS-induced artifacts from EEG recordings,” IEEE Trans. Control Syst. Technol., vol. 16, no. 6, pp. 1360–1366, Nov. 2008. [7] F. N. Chowdhury, “Input-output modelling of nonlinear systems with time-varying linear models,” IEEE Trans. Autom. Control, vol. 45, no. 6, pp. 1355–1358, Jun. 2000. [8] H. L. Wei and S. A. Billings, “Identification of time-varying systems using multiresolution wavelet models,” Int. J. Syst. Sci., vol. 33, no. 15, pp. 1217–1228, Dec. 2002. [9] K. H. Chon, H. Zhao, R. Zou, and K. Ju, “Multiple time-varying dynamic analysis using multiple sets of basis functions,” IEEE Trans. Biomed. Eng., vol. 52, no. 5, pp. 956–960, May 2005. [10] H. L. Wei, S. A. Billings, and J. Liu, “Time-varying parametric modelling and time-dependent spectral characterization with application to EEG signal using multiwavelets,” Int. J. Modell., Identification Control, vol. 9, no. 3, pp. 215–224, 2010. [11] M. K. Tsatsanis and G. B. Giannakis, “Time-varying system identification and model validation using wavelets,” IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3512–3523, Dec. 1993. [12] F. N. Chowdhury and J. L. Aravena, “A modular methodology for fast fault detection and classification in power systems,” IEEE Trans. Control Syst. Technol., vol. 6, no. 5, pp. 623–634, Sep. 1998. [13] S. A. Billings and H. L. Wei, “A new class of wavelet networks for nonlinear system identification,” IEEE Trans. Neural Netw., vol. 16, no. 4, pp. 862–874, Jun. 2005. [14] C. M. Chang and T. S. Liu, “A wavelet network control method for disk drives,” IEEE Trans. Control Syst. Technol., vol. 14, no. 1, pp. 63–68, Jan. 2006. [15] S. G. Mallat, “A theory for multiresolution signal decomposition: The wavelet representation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 11, no. 7, pp. 674–693, Jul. 1989. [16] C. K. Chui, An Introduction on Wavelets. Boston, MA: Academic Press, 1992. [17] S. Mallat, A Wavelet Tour of Signal Processing, second ed. London, U.K.: Academic Press, 1999. [18] N. Ahuja, S. Lertrattanapanich, and N. K. Bose, “Properties determining choice of mother wavelet,” IEE Proc. Vision Image Signal Process, vol. 152, no. 5, pp. 659–663, 2005.
LI et al.: IDENTIFICATION OF TIME-VARYING SYSTEMS USING MULTI-WAVELET BASIS FUNCTIONS
[19] H. L. Wei and S. A. Billings, “An efficient nonlinear cardinal B-spline model for high tide forecasts at the Venice lagoon,” Nonlinear Processes in Geophysics, vol. 13, no. 5, pp. 577–584, 2006. [20] J. J. Shynk, “Frequency-domain and multirate adaptive filtering,” IEEE Signal Process. Mag., vol. 9, no. 1, pp. 14–37, Jan. 1992. [21] S. Haykin, Adaptive Filter Theory, 3rd ed. Upper Saddle River, NJ: Prentice-Hall, 1996. [22] G. A. Clark, S. K. Mitra, and S. R. Parker, “Block implementation of adaptive digital filters,” IEEE Trans. Circuits Syst., vol. 28, no. 6, pp. 584–592, Jun. 1981. [23] J. Nagumo and A. Noda, “A learning method for system identification,” IEEE Trans. Autom. Control, vol. 12, no. 3, pp. 282–287, Jun. 1967.
663
[24] G. C. Goodwin and K. S. Sin, Adaptive Filtering, Prediction, and Control. Englewood Cliffs, NJ: Prentice-Hall, 1984. [25] S. A. Billings, H. L. Wei, and M. A. Balikhin, “Generalized multiscale radial basis function networks,” Neural Netw., vol. 20, no. 10, pp. 1081–1094, Dec. 2007. [26] I. Leontaritis and S. A. Billings, “Experimental design and identifiability for nonlinear systems,” Int. J. Syst. Sci., vol. 18, no. 1, pp. 189–202, Jan. 1987. [27] R. Ghanem and F. Romeo, “A wavelet-based approach for the identification of linear time-varying dynamical systems,” J. Sound Vibr., vol. 234, no. 4, pp. 555–576, 2000.
