Adaptive Neural Fuzzy Inference Systems (ANFIS ...

Adaptive Neural Fuzzy Inference Systems (ANFIS): Analysis and Applications

©Copyright 2002 by Piero P. Bonissone

1

Outline • Objective • Fuzzy Control – Background, Technology & Typology

• ANFIS: – as a Type III Fuzzy Control – as a fuzzification of CART – Characteristics – Pros and Cons – Opportunities – Applications – References ©Copyright 2002 by Piero P. Bonissone

2

ANFIS Objective • To integrate the best features of Fuzzy Systems and Neural Networks: – From FS: Representation of prior knowledge into a set of constraints (network topology) to reduce the optimization search space – From NN: Adaptation of backpropagation to structured network to automate FC parametric tuning

• ANFIS application to synthesize: – controllers (automated FC tuning) – models (to explain past data and predict future behavior) ©Copyright 2002 by Piero P. Bonissone

3

FC Technology & Typology • Fuzzy Control – A high level representation language with local semantics and an interpreter/compiler to synthesize non-linear (control) surfaces – A Universal Functional Approximator

• FC Types – Type I: RHS is a monotonic function – Type II: RHS is a fuzzy set – Type III: RHS is a (linear) function of state

©Copyright 2002 by Piero P. Bonissone

4

FC Technology (Background) • Fuzzy KB representation – Scaling factors, Termsets, Rules

• Rule inference (generalized modus ponens) • Development & Deployment – Interpreters, Tuners, Compilers, Run-time – Synthesis of control surface

• FC Types I, II, III

©Copyright 2002 by Piero P. Bonissone

5

FC of Type II, III, and ANFIS • Type II Fuzzy Control must be tuned manually • Type III Fuzzy Control (Takagi-Sugeno type) have an automatic Right Hand Side (RHS) tuning • ANFIS will provide both: – RHS tuning, by implementing the TSK controller as a network – and LHS tuning, by using back-propagation ©Copyright 2002 by Piero P. Bonissone

6

ANFIS Network

Inputs

IF-part

Rules + Norm

Input 1

Output

ω1

ω1 &

THEN-part

N

x1 &

N

Σ Input 2

Layers:

&

N

&

N

Output

x2

0

1

2

3

4

©Copyright 2002 by Piero P. Bonissone

5

7

ANFIS Neurons: Clarification note • Note that neurons in ANFIS have different structures: – Values [ Membership function defined by parameterized soft trapezoids (Generalized Bell Functions) ]

– Rules [ Differentiable T-norm - usually product ] – Normalization [ Sum and arithmetic division ] – Functions [ Linear regressions and multiplication with ω , i.e., normalized weights ω, ] – Output [ Algebraic Sum ]

©Copyright 2002 by Piero P. Bonissone

8

ANFIS as a generalization of CART • Classification and Regression Tree (CART) – Algorithm defined by Breiman et al in 1984 – Creates a binary decision tree to classify the data into one of 2n linear regression models to minimize the Gini index for the current node c: Gini(c) = where:

1 − ∑ p 2j j

• pj is the probability of class j in node c • Gini(c) measure the amount of “impurity” (incorrect classification) in node c ©Copyright 2002 by Piero P. Bonissone

9

CART Problems • Discontinuity • Lack of locality (sign of coefficients)

©Copyright 2002 by Piero P. Bonissone

10

CART: Binary Partition Tree and Rule Table Representation

x1

x1

a1 ≤ x1

a1 > x1 x2

x2

a2 ≤ x2

a2 > x2 a2 ≤ x2

f1(x1,x2)

f2(x1,x2)

f3 (x1,x2)

a2 > x2 f4 (x1,x2)

Partition Tree

©Copyright 2002 by Piero P. Bonissone

x2

y

a1 ≤

a2 ≤

f1(x1,x2)

a1 ≤

> a2

f2(x1,x2)

a1 >

a2 ≤

f3(x1,x2)

a1 >

> a2

f4(x1,x2)

Rule Table

11

Discontinuities Due to Small Input Perturbations

x1

y3= f3(x11, . )

a1 > x1

x1 ≤ a1

x2

x2

x2≤ a2

y1= f1(x11, . )

a2 ≤ x2 f1(x1,x2 )

1

a2 > x2

f2 (x1,x2)

f3 (x1,x2)

a2 > x2 f4 (x1,x2)

X1

µ(a ≤ )(x1) 1

Let's assume two inputs: I1=(x11,x12), and I2=(x21,x22) such that: x11 = ( a1- ε ) x21 = ( a1+ ε ) x12 = x22 < a2 Then I1 is assigned f1(x11,x12) while I2 is assigned f3(x1,x2) 0

X1 x11

a1

x21

©Copyright 2002 by Piero P. Bonissone

12

Takagi-Sugeno (TS) Model • Combines fuzzy sets in antecedents with crisp function in output: • IF (x1 is A) AND (x2 is B) THEN y = f(x1,x2) IF X is small THEN Y1=4 IF X is medium THEN Y2=-0.5X+4 IF X is large THEN Y3=X-1 n

∑Y w j

Y = ©Copyright 2002 by Piero P. Bonissone

j

j =1 n

∑w j =1

j 13

ANFIS Characteristics • Adaptive Neural Fuzzy Inference System (ANFIS) – Algorithm defined by J.-S. Roger Jang in 1992 – Creates a fuzzy decision tree to classify the data into one of 2n (or pn) linear regression models to minimize the sum of squared errors (SSE): SSE = ∑ e2j j

where: • ej is the error between the desired and the actual output • p is the number of fuzzy partitions of each variable • n is the number of input variables ©Copyright 2002 by Piero P. Bonissone

14

ANFIS as a Type III FC • L0: State variables are nodes in ANFIS inputs layer • L1: Termsets of each state variable are nodes in ANFIS values layer, computing the membership value • L2: Each rule in FC is a node in ANFIS rules layer using soft-min or product to compute the rule matching factor ωi • L3: Each ωi is scaled into ωi in the normalization layer • L4: Each ωi weighs the result of its linear regression fi in the function layer, generating the rule output • L5: Each rule output is added in the output layer ©Copyright 2002 by Piero P. Bonissone

15

ANFIS Architecture Rule Set: IF ( x 1 is A 1 ) AND (x 2 is B 1 ) THEN

f 1 = p 1 x 1 + q 1 x 2 + r1

IF ( x 1 is A 2 ) AND (x 2 is B 2 ) THEN

f 2 = p 2 x 1 + q 2 x 2 + r2

... x1 x2 x1 x2

Layers: 0

A1 A2 B1 B2 1

Π ω1 Π ω 2 2

ω1 f1

N ω1 N

3

Σ

ω2

y

ω2 f2 x1 x2 4

©Copyright 2002 by Piero P. Bonissone

5

16

ANFIS-Visualized (Example for n =2) where A1: Medium; A2: Small-Medium B1: Medium; B2: Small-Medium

• Fuzzy reasoning B1

A1

A2

B2

w1

w2

y1 = p1*x1 +q1*x2+r1 y2 = p2*x1+q2* x2 +r2

z=

w1*y1+w2*y2 w1+w2

y

x

• ANFIS (Adaptive Neuro-Fuzzy Inference System) A1

x

A2 B1

y

B2

Π

w1

Π

w1*y1

Σ Σwi*yi

w2*y2 w2

Σ

©Copyright 2002 by Piero P. Bonissone

/

Y

Σwi 17

Layer 1: Calculate Membership Value for Premise Parameter • Output O1,i for node i=1,2

O1,i = µ Ai (x1 )

• Output O1,i for node i=3,4

O1,i = µ Bi − 2 ( x 2 )

• where A is a linguistic label (small, large, …)

µ A ( x1 ) =

1 x1 − c i 1+ ai

2b

Node output: membership value of input ©Copyright 2002 by Piero P. Bonissone

18

Layer 1 (cont.): Effect of changing Parameters {a,b,c} (a ) C h a n ging 'a '

1

µA (x) = 1+

x −ci ai

2b

(b ) C ha n gin g 'b '

1

1

0.8

0 .8

0.6

0 .6

0.4

0 .4

0.2

0 .2

0 -10

-5

0

5

10

0 -1 0

(c ) C h a n ging 'c ' 1

0.8

0 .8

0.6

0 .6

0.4

0 .4

0.2

0 .2 -5

0

5

0

5

10

(d ) C ha ngin g 'a ' a n d 'b '

1

0 -10

-5

10

©Copyright 2002 by Piero P. Bonissone

0 -1 0

-5

0

5

10

19

Layer 2: Firing Strength of Rule • Use T-norm (min, product, fuzzy AND, ...)

O2 ,i = wi = µ Ai (x1 )µ Bi (x2 ) (for i=1,2)

Node output: firing strength of rule ©Copyright 2002 by Piero P. Bonissone

20

Layer 3: Normalize Firing Strength • Ratio of ith rule’s firing strength vs. all rules’ firing strength

O 3 ,i

wi = wi = w1 + w 2 (for i=1,2)

Node output: Normalized firing strengths ©Copyright 2002 by Piero P. Bonissone

21

Layer 4: Consequent Parameters • Takagi-Sugeno type output

O4,i = wi fi = wi ( pi x1 + qi x2 + ri ) • Consequent parameters {pi, qi, ri} Node output: Evaluation of Right Hand Side Polynomials ©Copyright 2002 by Piero P. Bonissone

22

Layer 5: Overall Output O 5 ,1 =

∑

wi fi =

i

∑w f ∑w i

i

i

i

i

• Note: – Output is linear in consequent parameters p,q,r: =

w1 w2 f1 + f2 w1 + w 2 w1 + w 2

= w 1 ( p 1 x 1 + q 1 x 2 + r1 ) + w

(

)

(

)

2

(p 2 x1

( )

(

+ q 2 x 2 + r2

)

)

(

)

( )r

= w 1 x 1 p 1 + w 1 x 2 q 1 + w 1 r1 + w 2 x 1 p 2 + w 2 x 2 q 2 + w

2

2

Node output: Weighted Evaluation of RHS Polynomials ©Copyright 2002 by Piero P. Bonissone

23

ANFIS Network

Inputs

IF-part

Rules + Norm

Input 1

Output

ω1

ω1 &

THEN-part

N

x1 &

N

Σ Input 2

Layers:

&

N

&

N

Output

x2

0

1

2

3

4

©Copyright 2002 by Piero P. Bonissone

5

24

ANFIS Computational Complexity Layer #

L-Type # Nodes

# Param

L0 L1

Inputs Values

n (p•n)

0 3•(p•n)=|S1|

L2

Rules

pn

0

L3 L4

Normalize Lin. Funct.

pn pn

0 (n+1)•pn=|S2|

L5

Sum

1

0

©Copyright 2002 by Piero P. Bonissone

25

ANFIS Parametric Representation • ANFIS uses two sets of parameters: S1 and S2 – S1 represents the fuzzy partitions used in the rules LHS

{

}

S1= {a11,b11,c11},{a12,b12,c12},...,{a1p ,b1p ,c1p },...,{anp,bnp,cnp}

– S2 represents the coefficients of the linear functions in the rules RHS S2 =

{{c

10 ,c11 ,...,c1n

}

},..., {c p 0 ,c p 1 ,...,c p n }

©Copyright 2002 by Piero P. Bonissone

n

n

n

26

ANFIS Learning Algorithms • ANFIS uses a two-pass learning cycle – Forward pass: • S1 is fixed and S2 is computed using a Least Squared Error (LSE) algorithm (Off-line Learning)

– Backward pass: • S2 is fixed and S1 is computed using a gradient descent algorithm (usually Back-propagation)

©Copyright 2002 by Piero P. Bonissone

27

Structure ID & Parameter ID Hybrid training method nonlinear parameters

x1 x2

A1

Π

A2

Π

B1

Π

B2

Π

w1

linear parameters w1*y1

Σ Σwi*yi w4*y4

w4

Σ

/

Σwi

Y

• Input space partitioning A1 B1

x2

A2 B2

x1

B2 B1

x2

A1

A2 x1

Forward stroke Backward stroke MF param. (nonlinear) Coef. param. (linear)

fixed

steepest descent

least-squares

fixed

©Copyright 2002 by Piero P. Bonissone

28

ANFIS Least Squares (LSE) Batch Algorithm • LSE used in Forward Stroke: – Parameter Set: S = {S1U S2}, and {S1I S2 = ∅}

( )

Output = F I , S , where I is the input vector

( )

H (Output ) = H o F I , S , where H o F is linear in S2

– For given values of S1, using K training data, we can transform the above equation into B=AX, where X contains the elements of S2 – This is solved by: (ATA)-1AT B=X* where (ATA)-1AT is the pseudo-inverse of A (if ATA is nonsingular) – The LSE minimizes the error ||AX-B||2 by approximating X with X* ©Copyright 2002 by Piero P. Bonissone

29

ANFIS LSE Batch Algorithm (cont.) • Rather than solving directly (ATA)-1AT B=X* , we resolve it iteratively (from numerical methods):   Si+1 = Si − , T  1 + a(i+1) Si a(i+1)   for i = 0,1,..., K −1  T T Xi+1 = Xi + S(i+1) a(i+1) (b(i+1) − a(i+1) Xi )    T Si Si a(i+1) a(i+1)

where:

X 0 = 0, S 0 = γ I , (where γ is a large number ) a Ti = i th line of matrix A b Ti = i th element of vector B X * = Xk ©Copyright 2002 by Piero P. Bonissone

30

ANFIS Back-propagation • Error measure Ek (for the kth (1≤k≤K) entry of the training data) Ek = where : N(L) = number

N ( L)

2 ( d − x ) ∑ i L ,i i =1

nodes in layer L

d i = i th component x L , i = i th component

of desired

output

vector

of actual

output

vector

• Overall error measure E: E =

K

∑

k=1

Ek

©Copyright 2002 by Piero P. Bonissone

31

ANFIS Back-propagation (cont.) • For each parameter αi the update formula is: ∆α

i

∂ +E = −η ∂α i

where :

η=

κ

κ 2

is the learning rate

 ∂E  ∑i  ∂α   i is the step size

∂ +E is the ordered derivative ∂α i ©Copyright 2002 by Piero P. Bonissone

32

ANFIS Pros and Cons • ANFIS is one of the best tradeoff between neural and fuzzy systems, providing: – smoothness, due to the FC interpolation – adaptability, due to the NN Backpropagation

• ANFIS however has strong computational complexity restrictions

©Copyright 2002 by Piero P. Bonissone

33

ANFIS Pros Characteristics Translates prior knowledge into network topology & initial fuzzy partitions

Advantages Network's first three layers not fully connected (inputs-valuesrules)

Smaller fan-out for Backprop

Induced partialorder is usually preserved

Uses data to determine rules RHS (TSK model)

Network implementation of Takagi-Sugeno-Kang FLC

©Copyright 2002 by Piero P. Bonissone

Faster convergency than typical feedforward NN

++

Smaller size training set

+++

Model compactness (smaller # rules than using labels)

+

34

ANFIS Cons Characteristics Translates prior knowledge into network topology & initial fuzzy partitions

Disadvantages

Sensitivity to initial number of partitions "

P"

Sensitivity to number of input variables " n "

Uses data to determine rules RHS (TSK model)

Partial loss of rule "locality"

©Copyright 2002 by Piero P. Bonissone

Surface oscillations around points (caused by high partition number)

--

Spatial exponential complexity: ^n # Rules = P

--

Coefficient signs not always consistent with underlying monotonic relations

-

35

ANFIS Pros (cont.) Characteristics Uses LMS algorithm to compute polynomials coefficients

Uses Backprop tune fuzzy partitions

Advantages Uses fuzzy partitions to discount outlier effects

to

Automatic FLC parametric tuning

+++

Smoothness guaranteed by interpolation

++

Error pressure to modify only "values" layer

Uses FLC inference mechanism to interpolate among rules

©Copyright 2002 by Piero P. Bonissone

36

ANFIS Cons (cont.) Disadvantages

Characteristics Uses LMS algorithm to compute polynomials coefficients

Uses Backprop tune fuzzy partitions

Batch process disregards previous state (or IC)

to

Error gradient calculation requires derivability of fuzzy partitions and T-norms used by FLC

Based on quadratic error cost function

Uses FLC inference mechanism to interpolate among rules

Uses convex sum: Σ λ f (X)/ Σ λ i i i

©Copyright 2002 by Piero P. Bonissone

Not possible to represent known monotonic relations

-

Cannot use trapezoids nor "Min"

-

Symmetric error treatment & great outliers influence

--

"Awkward" interpolation between slopes of different sign

-

37

ANFIS Opportunities • Changes to decrease ANFIS complexity – Use “don’t care” values in rules (no connection between any node of value layer and rule layer) – Use reduced subset of state vector in partition tree while evaluating linear functions on complete state X = ( X r U X (n−r) )

– Use heterogeneous partition granularity (different partitions pi for each state variable, instead of “p”) n

# RULES = ∏ pi i=1

©Copyright 2002 by Piero P. Bonissone

38

ANFIS Opportunities (cont.) • Changes to extend ANFIS applicability – Use other cost function (rather than SSE) to represent the user’s utility values of the error (error asymmetry, saturation effects of outliers,etc.) – Use other type of aggregation function (rather than convex sum) to better handle slopes of different signs.

©Copyright 2002 by Piero P. Bonissone

39

ANFIS Applications at GE

• • • •

Margoil Oil Thickness Estimator Voltage Instability Predictor (Smart Relay) Collateral Evaluation for Mortgage Approval Prediction of Time-to-Break for Paper Web

©Copyright 2002 by Piero P. Bonissone

40

ANFIS References • “ANFIS: Adaptive-Network-Based Fuzzy Inference System”, J.S.R. Jang, IEEE Trans. Systems, Man, Cybernetics, 23(5/6):665-685, 1993. • “Neuro-Fuzzy Modeling and Control”, J.S.R. Jang and C.-T. Sun, Proceedings of the IEEE, 83(3):378-406 • “Industrial Applications of Fuzzy Logic at General Electric”, Bonissone, Badami, Chiang, Khedkar, Marcelle, Schutten, Proceedings of the IEEE, 83(3):450-465 • The Fuzzy Logic Toolbox for use with MATLAB, J.S.R. Jang and N. Gulley, Natick, MA: The MathWorks Inc., 1995 • Machine Learning, Neural and Statistical Classification Michie, Spiegelhart & Taylor (Eds.), NY: Ellis Horwood 1994 • Classification and Regression Trees, Breiman, Friedman, Olshen & Stone, Monterey, CA: Wadsworth and Brooks, 1985 ©Copyright 2002 by Piero P. Bonissone

41