Mining Dynamic Recurrences in Nonlinear and ...

Mining Dynamic Recurrences in Nonlinear and Nonstationary Systems for Feature Extraction, Process Monitoring and Fault Diagnosis Dr. Hui Yang

杨徽 Associate Professor The Harold and Inge Marcus Department of Industrial and Manufacturing Engineering The Pennsylvania State University University Park, PA 16802

October 1, 2017

Outline

1

Introduction

2

Recurrence Quantification Analysis

3

Heterogeneous Recurrence Monitoring

4

Research Opportunity

Relevant Literature H. Yang and Y. Chen, “Heterogeneous recurrence monitoring and control of nonlinear stochastic processes,”Chaos, Vol. 24, No. 1, p013138, 2014, DOI: 10.1063/1.4869306. Y. Chen and H. Yang, “Heterogeneous recurrence representation and quantification of dynamic transitions in continuous nonlinear processes, ”European Physical Journal (Complex Systems), Vol. 89, No. 6, p1-11, 2016, DOI: 10.1140/epjb/e2016-60850-y C. Cheng, C. Kan, and H. Yang, “Heterogeneous recurrence modeling and analysis of heartbeat dynamics for the identification of sleep apnea events,”Computers in Biology and Medicine, Vol. 75, p10-18, 2016, DOI: 10.1016/j.compbiomed.2016.05.006 C. Kan, C. Cheng, and H. Yang, “Heterogeneous recurrence monitoring of dynamic transients in ultraprecision machining processes,”Journal of Manufacturing Systems, Vol. 41, p. 178-187, 2016. DOI: 10.1016/j.jmsy.2016.08.007 Y. Chen and H. Yang, “Heterogeneous Recurrence T2 Charts for Monitoring and Control of Nonlinear Dynamic Processes,”Proceedings of the 11th Annual IEEE Conference on Automation Science and Engineering (CASE), p. 1066-1071, August 24-28, 2015, Gothenburg, Sweden. DOI: 10.1109/CoASE.2015.7294240 H. Yang, “Multiscale Recurrence Quantification Analysis of Spatial Vectorcardiogram (VCG) Signals,”IEEE Transactions on Biomedical Engineering, Vol. 58, No. 2, p339-347, 2011, DOI: 10.1109/TBME.2010.2063704 Y. Chen and H. Yang, “Multiscale recurrence analysis of long-term nonlinear and nonstationary time series,”Chaos, Solitons and Fractals, Vol. 45, No. 7, p978-987, 2012, DOI: 10.1016/j.chaos.2012.03.013

Introduction

RQA



Research Roadmap

Yang, Hui (PSU)

Nonlinear Recurrence Dynamics

October 1, 2017

4 / 44

Introduction

RQA



Research Projects Manufacturing Processes, Precision Machining

Publications: Pattern Recognition, IEEE Transactions, Chaos, IIE Transactions Yang, Hui (PSU)


October 1, 2017

5 / 44

Introduction

RQA



Research Projects Electro-mechanical Processes, Biomanufacturing

Publications: IEEE Transactions, Physical Review, Physiological Measurements

Yang, Hui (PSU)


October 1, 2017

6 / 44

Introduction

RQA



Opportunities and Challenges

Yang, Hui (PSU)


October 1, 2017

7 / 44

Introduction

RQA



Linear Systems

Linear System of D.E (system’s evolution is linear) X˙ = AX ; X ∈ Rn X(0) = X0 General solution X(t) = eAt X0 The solution is explicitly known for any t. Stability of linear systems is determined by eigenvalues of matrix A. If Re(λ) < 0, Stable If Re(λ) > 0, Unstable

Yang, Hui (PSU)


October 1, 2017

8 / 44

Introduction

RQA



Nonlinear Systems and Strange Attractors Difficult or impossible to solve analytically dX ˙ X(t) = = F (X), F ∈ Rn → Rn dt Components are interdependent

Yang, Hui (PSU)


October 1, 2017

9 / 44

Introduction

RQA



Nonlinear Dynamics Dynamical system – a rule for time evolution on a state space. State – a d dimensional vector defining the state of the dynamical system at time t x(t) ˙ = (x1 (t), x2 (t), · · · , xd (t))T Dynamics or equation of motion ˙ X(t) = F (X), F ∈ Rn → Rn Causal relation between the present state and the next state Deterministic rule which tells us what happen in the next step Linear dynamics - the causal relation is linear.

Experimental settings (not all states known or observable) Time Series: si = s(i∆t) , i = 1, . . . , N and ∆t is the sampling interval Yang, Hui (PSU)


October 1, 2017

10 / 44

Introduction

RQA



State Space Reconstruction

Takens’ embedding theorem F diffeomorphism - invariants: dimensions, Lyapunov exponents, ... Yang, Hui (PSU)


October 1, 2017

11 / 44

Introduction

RQA



An Example

Yang, Hui (PSU)


October 1, 2017

12 / 44

Introduction

RQA



Recurrence Poincar´ e Recurrence Theorem Let T be a measure-preserving transformation of a probability space (X, P ) and let A ⊂ X be a measurable set. Then, for any natural number N ∈ N, the trajectory will eventually reappear at neighborhood A of former states: P r({x ∈ A|{T n (x)}n≥N ⊂ X\A}) = 0

(a) Stamping Machine, from Dr. Jianjun Shi

Yang, Hui (PSU)


(b) Biological System

October 1, 2017

13 / 44

Introduction

RQA



Recurrence Plot Recurrence dynamics of nonlinear and nonstationary systems R(i, j) = Θ(ε − kx(i) − x(j)k)

Yang, Hui (PSU)


October 1, 2017

14 / 44

Introduction

RQA



Structures in Recurrence Plots

Small-scale structures: single dots, diagonal and vertical lines Large-scale structures: homogenous, periodic and disrupted patterns Yang, Hui (PSU)


October 1, 2017

15 / 44

Introduction

RQA



Recurrence Quantification Analysis (RQA) Statistical features to quantify topological structures and patterns from the Recurrence Plot (Webber, Marwan, and Kurths et al.): Recurrence rate (%REC): The percentage of recurrence points in an RP RR =

Entropy (ENT): EN T = −

N 1 X R(i, j) N 2 i,j=1

PN

l=lmin

p(l)ln(p(l)

Shannon entropy - the probability distribution of diagonal line lengths p(l)

Determinism (%DET) Linemax (LMAX) Laminarity (%LAM) Trapping time (TT)

Diagonal structures (the first four) and vertical structures (the last two) in the recurrence plot Yang, Hui (PSU)


October 1, 2017

16 / 44

Introduction

RQA



Homogeneous Recurrence

Yang, Hui (PSU)


October 1, 2017

17 / 44

Introduction

RQA



Heterogeneous Recurrences Two pairs of recurrence states: ~s(15) = (x15 , x17 , x19 ) and ~s(1) = (x1 , x3 , x5 ) ~s(32) = (x32 , x34 , x36 ) and ~s(30) = (x30 , x32 , x34 )

Yang, Hui (PSU)


October 1, 2017

18 / 44

Introduction

RQA



State Space Indexing

Database Management: Multi-dimensional Data Indexing Speed up data retrieval and data query in VLDB

Yang, Hui (PSU)


October 1, 2017

19 / 44

Introduction

RQA



Fractal Representation Iterative function systems (IFS) ~s(n) −→ K} K= {1, 2, . .. k∈ cx (n − 1) cx (n) = ϕ k, cy (n − 1) cy (n)

=

α 0

0 α

cx (n − 1) cos(k × + cy (n − 1) sin(k ×

2π ) K 2π ) K

cx (0) 0 Initial address: = cy (0) 0

Yang, Hui (PSU)


October 1, 2017

20 / 44

Introduction

RQA



Fractal Representation Iterative function systems (IFS) ~s(n) −→ K} k∈ K= {1, 2, . .. cx (n) cx (n − 1) = ϕ k, cy (n) cy (n − 1)

Initial address:

Yang, Hui (PSU)

=

α 0

0 α

cx (n − 1) cos(k × + sin(k × cy (n − 1)

2π ) K 2π ) K

cx (0) 0 = cy (0) 0


October 1, 2017

20 / 44

Introduction

RQA




Initial address:

Yang, Hui (PSU)

=

α 0

0 α

cx (n − 1) cos(k × + cy (n − 1) sin(k ×

2π ) K 2π ) K

cx (0) 0 = cy (0) 0


October 1, 2017

20 / 44

Introduction

RQA



Fractal Representation Iterative function systems (IFS) ~s(n) −→ K} k∈ K= {1, 2, . .. cx (n) cx (n − 1) = ϕ k, cy (n) cy (n − 1)

Initial address:

Yang, Hui (PSU)

=

α 0

0 α

cx (n − 1) cos(k × + cy (n − 1) sin(k ×

2π ) K 2π ) K

cx (0) 0 = cy (0) 0


October 1, 2017

20 / 44

Introduction

RQA




=

α 0

0 α

cx (n − 1) cos(k × + cy (n − 1) sin(k ×

2π ) K 2π ) K

cx (0) 0 Initial address: = cy (0) 0

Yang, Hui (PSU)


October 1, 2017

20 / 44

Introduction

RQA



Markov Process

Yang, Hui (PSU)


October 1, 2017

21 / 44

Introduction

RQA



Heterogeneous Recurrence Quantification Heterogeneous recurrence sets Wk1 ,k2 ,...,kL = {ϕ(k1 |k2 , . . . , kL ) : ~sn → k1 , ~sn−1 → k2 , ~sn−L+1 → kL }

Heterogeneous recurrence rate (HRR) HRR =

W k1 ,k2 ,...,k L N

2 , W k1 ,k2 ,...,kL is the cardinality of set Wk1 ,k2 ,...,kL

Heterogeneous mean (HMean) Dk1 ,k2 ,...,kL (i, j) = kϕi − ϕj k ϕi , ϕj ∈ Wk1 ,k2 ,...,kL ; i, j = 1, 2, . . . , W ; i < j PW PW 2 HM ean = i=1 j=i+1 Dk1 ,k2 ,...,kL (i, j) W (W −1)

Heterogeneous entropy (HENT) 1 #{ b−1 max(D) B W (W −1) PB HEN T = − b=1 p(b)lnp(b)

p(b) =

Yang, Hui (PSU)

< Dk1 ,k2 ,...,kL (i, j) ≤


b max(D)} B

October 1, 2017

22 / 44

Introduction

RQA



Multivariate Process Monitoring

Known mean µ and covariance matrix Σ χ2 = (y − µ)Σ−1 (y − µ) follows the chi-square distribution

Unknown mean µ and covariance matrix Σ T 2 = (y − y ¯)S−1 (y − y ¯) Replace µ and Σ with the sample mean y ¯ and covariance S What if the sample covariance matrix S is singular? Yang, Hui (PSU)


October 1, 2017

23 / 44

Introduction

RQA



Recurrence T 2 Chart Data centering: Y∗ = [y1 − y¯, y2 − y¯, . . . , yM − y¯] Singular vector decomposition: Y∗ = UΨ V T Principal components: Z = Y∗ V = UΨ VT V = UΨ S=

1 M −1

PM

i=1 (yi

−y ¯)(yi − y ¯)T =

1 VZT ZVT M −1

= VSz VT

S : sample covariance matrix SZ : sample covariance matrix of Z

Recurrence T 2 statistic for the ith sample: T 2 (i) = (yi − y ¯)T S −1 (yi − y ¯) = Z(i, :)VT (VSz VT )−1 VZ(i, :)T =

Z(i, :)SZ−1 Z(i, :)T

=

p X Z(i, k)2 k=1

λ2k

where λ1 > λ2 > · · · > λp are singular values of Y∗ P Recurrence T 2 statistic in the reduced dimension q: T˜2 (i) = qk=1 Yang, Hui (PSU)


Z(i,k)2 λ2 k

October 1, 2017

24 / 44

Introduction

RQA



Case Study - Discrete and Finite State Space In-control vs. out-of-control Markov processes Out-of-control transition matrix

In-control vs. slightly-changed Markov processes Change the 7th row of in-control transition matrix from [0, 0, 0.5, 0, 0, 0.5, 0, 0] to [0, 0, 0.6, 0, 0, 0.4, 0, 0]

Distribution-based processes (uniform vs. normal) Yang, Hui (PSU)


October 1, 2017

25 / 44

Introduction

RQA



Case 1: In-control vs. out-of-control Markov processes In-control:

Out-of-Control:

Yang, Hui (PSU)


October 1, 2017

26 / 44

Introduction

RQA



Case 2: In-control vs. slightly-changed Markov processes In-control:

Slight change:

Yang, Hui (PSU)


October 1, 2017

27 / 44

Introduction

RQA



Case 3: Distribution-based processes (uniform vs. normal) Uniform

Normal

Yang, Hui (PSU)


October 1, 2017

28 / 44

Introduction

RQA



Performance Comparison - Multivariate Projection Heterogeneous Recurrence T 2 statistics in the reduced dimension q

Yang, Hui (PSU)


October 1, 2017

29 / 44

Introduction

RQA



Case Study - Continuous State Space Autoregressive Model xi = axi−1 + bxi−2 + cε i− sample index a, b, c− model parameters ε ∼ N (0, 1)

Yang, Hui (PSU)

Lorenz Model   x˙ = σ (y − x) y˙ = x(ρ − z) − y  z˙ = xy − βz σ, ρ, β− model parameters


October 1, 2017

30 / 44

Introduction

RQA



Autoregressive Model

Figure: (a) AR2 time series. The blue segment is with parameters a = 1,b = −1,c = 0.5, and the parameter of red one are a = 1.05, b = −1, c = 0.5. (b) x chart for DET. (c) x chart for LAM. (d) Heterogeneous recurrence T 2 chart. Yang, Hui (PSU)


October 1, 2017

31 / 44

Introduction

RQA



Autoregressive Model

Figure: Performance comparison of detection power of DET, LAM charts and heterogeneous recurrence T 2 chart for AR(2) models. (a) varying parameter a. (b) varying parameter b. (a) varying parameter c.

Yang, Hui (PSU)


October 1, 2017

32 / 44

Introduction

RQA



Lorenz Model

Figure: (a) The state space of Lorenz system with parameter changing from σ = 10, ρ = 28, β = 8/3 (blue dots) and to σ = 10, ρ = 27, β = 8/3 (red lines); (b) x chart for DET. (c) x chart for LAM. (d) Heterogeneous recurrence T 2 chart.

Yang, Hui (PSU)


October 1, 2017

33 / 44

Introduction

RQA



Lorenz Model

Figure: Performance comparison of detection power of DET, LAM charts and heterogeneous recurrence T 2 chart for Lorenz models. (a) varying parameter σ. (b) varying parameter ρ. (c) varying parameter β.

Yang, Hui (PSU)


October 1, 2017

34 / 44

Introduction

RQA



ECG Dynamics

Figure: Heterogeneous recurrence T 2 chart for monitoring real-world ECG signals.

Yang, Hui (PSU)


October 1, 2017

35 / 44

Introduction

RQA



Summary Challenges Complex Systems =⇒ Advanced Sensing =⇒ Big Data Nonlinearity and Nonstationarity Homogeneous vs. Heterogeneous Recurrences

Methodology - Heterogeneous Recurrence Monitoring and Control Multi-dimensional state space indexing Fractal representation – characterize heterogeneous recurrences Heterogeneous recurrence quantification Multivariate monitoring of nonlinear dynamics

Significance Sensor-based monitoring and control of nonlinear dynamical systems Advanced sensing and control ⇐⇒ nonlinear dynamics theory Broad applications

Yang, Hui (PSU)


October 1, 2017

36 / 44

Introduction

RQA



Broad Applications

Yang, Hui (PSU)


October 1, 2017

37 / 44

Introduction

RQA




Open research opportunities for interested students and visiting scholars Research Directions Computer Experiments and Simulation Optimization Sensor-based Manufacturing Informatics and Control Nonlinear Dynamics Modeling and Control Innovative Biomanufacturing Smart Health, Biomechanics and Human Factors

Yang, Hui (PSU)


October 1, 2017

38 / 44

Introduction

RQA



Industrial and Manufacturing Engineering @ Penn State USNEWS: Top 10 Programs in the United States

Yang, Hui (PSU)


October 1, 2017

39 / 44

Introduction

RQA



Industrial and Manufacturing Engineering @ Penn State Factory for Advanced Manufacturing Education (FAME Lab)

Yang, Hui (PSU)


October 1, 2017

40 / 44

Introduction

RQA



Industrial and Manufacturing Engineering @ Penn State

Human Factors Ergonomics; Human Centered Design; Human-Computer Interaction; Manufacturing Distributed Systems and Control; Manufacturing Design; Manufacturing Processes; Operations Research Applied Probability and Stochastic Systems; Game Theory; Optimization; Statistics and Quality Engineering; Simulation; Production, Supply Chain and Service Engineering Health Systems Engineering; Production and Distribution Systems; Service Engineering; Supply Chain Engineering and Logistics;

Yang, Hui (PSU)


October 1, 2017

41 / 44

Introduction

RQA



Acknowledgements

NSF CAREER Award NSF CMMI-1454012 NSF CMMI-1266331

Yang, Hui (PSU)


October 1, 2017

42 / 44

Introduction

RQA



Questions?

Yang, Hui (PSU)


October 1, 2017

43 / 44

Introduction

RQA



Contact Information

Hui Yang Associate Professor Complex Systems Monitoring Modeling and Control Laboratory Harold and Inge Marcus Department of Industrial and Manufacturing Engineering The Pennsylvania State University Tel: (814) 865-7397 Fax: (814) 863-4745 Email: [email protected] Web: http://www.personal.psu.edu/huy25/

Yang, Hui (PSU)


October 1, 2017

44 / 44