chapter 5 adaptive neuro fuzzy modeling and

42

CHAPTER 5 ADAPTIVE NEURO FUZZY MODELING AND SOFTWARE IMPLEMENTATION

5.1 INTRODUCTION TO NEURO FUZZY SYSTEM A neuro fuzzy system is a combination of neural network and fuzzy systems in such a way that neural network or neural network algorithms are used to determine the parameters of the fuzzy system. This means that the main intention of neuro fuzzy approach is to create or improve a fuzzy system automatically by means of neural network methods. An even more important aspect is that the system should always be interpretable in terms of fuzzy if-then rules, because it is based on a fuzzy system reflecting vague knowledge. The idea of a neuro fuzzy system is to find the parameters of a fuzzy system by the means of learning methods obtained from neural network. A common way to apply a learning algorithm to a fuzzy system is to represent it in a special neural- network-like architecture. Then a learning algorithm such as back propagation is used to train the system. However, neural network learning algorithms are usually gradient descent methods. This cannot be applied directly to a fuzzy system, because the functions used to realize the inference process are usually not differentiable. In order to realize the system, we need to replace the functions used in the fuzzy system (like min and max) by differentiable

43

functions or do not use a gradient-based neural learning algorithm but a better-suited procedure. Modern neuro fuzzy systems are often represented as multilayer feed forward neural network. A neuro fuzzy system is a fuzzy system that is trained by a learning algorithm (usually) derived from neural network theory. The (heuristic) learning procedure operates on local information, and causes only local modifications in the underlying fuzzy system. The learning process is not knowledge-based, but datadriven. Besides, a neural-fuzzy system can always be interpreted as a system of fuzzy rules. It is possible both to create the system out of training data from scratch, and to initialize it from prior knowledge in the form of fuzzy-rules. The learning procedure of a neural-fuzzy system takes the semantical properties of the underlying fuzzy system into account. This results in constraints on the possible modification of the system’s parameters. It also approximates an n- dimensional (unknown) function that is partially given by the training data. The fuzzy rules encoded within the system represent vague samples, and can be viewed as vague prototypes of the training data. Generally, a neuro fuzzy system should not be seen as a kind of (fuzzy) expert system, and it has nothing to do with fuzzy logic in the narrow sense. It can be viewed as a special kind of feed forward neural network. The units in this network use t-norms or t-conorms instead of the activation functions normally used in neural networks. Fuzzy sets are encoded as (fuzzy) connection weights. The neuro fuzzy system is broadly classified into three systems

44

namely; cooperative neuro fuzzy system, concurrent neuro fuzzy system and hybrid neuro fuzzy system as shown in following Figure 5.1.

Figure 5.1 Types of Neuro Fuzzy System The hybrid neuro fuzzy system is further classified into ten networks namely Adaptive Neuro Fuzzy Inference System (ANFIS), Fuzzy Adaptive Learning Control Network (FALCON), Neuronal Fuzzy Controller (NEFCON), Fuzzy Net (FUN), Generalized Approximate Reasoning based Intelligence Control (GARIC), Fuzzy Inference and Neural Network in Fuzzy Inference Software (FINEST), Self Constructing Neural Fuzzy Inference Network (SONFIN), Fuzzy Neural Network (FNN), Evolving Fuzzy Neural Network (EFuNN) and Dynamic Evolving Fuzzy Neural Network (dmEFuNN). 5.1.1 ANFIS Architecture The Adaptive Network based Fuzzy Inference System (ANFIS) (Jang, 1993) [29] shown in figure 5.2 implements a Takagi-Sugeno fuzzy inference system and it has five layers. The first hidden layer is responsible for the mapping of the input variable relatively to each membership functions. The operator T-norm is applied in the second hidden layer to calculate the antecedents of the rules. The third hidden layer normalizes the rules strengths followed by the fourth hidden layer where the consequents of the rules are determined. The output layer

45

calculates the global output as the summation of all the signals that arrive to this layer.

Figure 5.2 ANFIS Architecture ANFIS uses back-propagation learning to determine the input membership functions parameters and the least mean square method to determine the consequents parameters. Each step of the iterative learning algorithm has two parts. In the first part, the input patterns are propagated and the parameters of the consequents are calculated using the iterative minimum squared method algorithm, while the parameters of the premises are considered fixed. In the second part, the input patterns are propagated again and in each iteration, the learning algorithm back-propagation is used to modify the parameters of the premises, while the consequents remain fixed. 5.1.2 FALCON Architecture The Fuzzy Adaptive Learning Control Network FALCON as shown in figure 5.3 is an architecture of five layers. There are two linguistics nodes for each output. One is for the patterns and the other is for the real output of the FALCON. The first hidden layer is responsible

46

for the mapping of the input variables relatively to each membership functions.

Figure 5.3 FALCON Architecture The second hidden layer defines the antecedents of the rules followed by the consequents in the third hidden layer. FALCON uses an hybrid learning algorithm composed by a unsupervised learning to define the initial membership functions and initial rule base and it uses a learning algorithm based on the gradient descent to optimise/adjust the final parameters of the membership functions to produce the desired output. 5.1.3 NEFCON Architecture The Neural Fuzzy Controller (NEFCON) shown in figure 5.4

47

implement a Mamdani type inference fuzzy system. The connections in this architecture are weighted with fuzzy sets and rules using the same antecedents (called shared weights), which are represented by the drawn ellipses. They assure the integrity of the base of rules. The input units assume the function of fuzzyfication interface, the logical interface is represented by the propagation function and the output unit is responsible for the defuzzyfication interface. The process of learning in architecture NEFCON is based in a mixture of reinforcement learning with backpropagation algorithm. This architecture can be used to learn the rule base from the beginning, if there is no a prior knowledge of the system, or to optimise an initial manually defined rule base. NEFCON has two variants NEFPROX

(for

function

approximation)

and

classification tasks).

Figure 5.4 NEFCON Architecture

NEFCLASS

(for

48

5.1.4 FUN Architecture In Fuzzy Net (FUN) architecture shown in figure 5.5 the neurons in the first hidden layer contain the membership functions and this performs a fuzzification of the input values. In the second hidden layer, the conjunctions are calculated. Membership functions of the output variables are stored in the third hidden layer. Their activation function is a fuzzy-OR. Finally the output neuron performs the defuzzification. The network is initialized with a fuzzy rule base and the corresponding membership functions and thereafter uses a stochastic learning technique that randomly changes parameters of membership functions and connections within the network structure. The learning process is driven by a cost function, which is evaluated after the random modification. If the modification resulted in an improved performance, then the modification is kept, otherwise it is undone.

49

Figure 5.5 FUN Architecture 5.1.5 GARIC Achitecture The Generalized Approximate Reasoning based Intelligence Control (GARIC) shown in figure 5.6 implements a neuro-fuzzy system using two neural networks modules, ASN (Action Selection Network) and AEN (Action State Evaluation Network). The AEN is an adaptive evaluator of ASN actions. The ASN of the GARIC is an advanced network of five layers. The connections between the layers are not weighted. The first hidden layer stores the linguistics values of all input

50

variables. Each input can only connect to the first layer, which represents its associated linguistics values. The second hidden layer represents the fuzzy rule nodes that determine the compatibility degree of each rule using a softmin operator. The third hidden layer represents the linguistics values of the output variables. The conclusions of each rule are calculated depending on the strength of the rules antecedents calculated in the rule nodes. GARIC uses the mean of local mean of maximum method to calculate the output of the rules. This method needs for a numerical value in the exit of each rule. Thus, the conclusions should be transformed from fuzzy values for numerical values before being accumulated in the final output value of the system. GARIC uses a mixture of gradient descending and reinforcement learning for a fine adjustment of its internal parameters.

Figure 5.6 GARIC Architecture 5.1.6 FINEST Architecture The Fuzzy Inference and Neural Network in Fuzzy Inference

51

Software (FINEST) architecture is capable of two kinds of tuning process, the tuning of fuzzy predicates, combination functions and the tuning of an implication function. The generalized modus ponens is improved in the following four ways (1) Aggregation operators that have synergy and cancellation nature (2) A parameterized implication function (3) A combination function that can reduce fuzziness (4) Backward chaining based on generalized modus ponens. FINEST make use of a backpropagation algorithm for the fine-tuning of the parameters. Figure 5.7 shows the layered architecture of FINEST and the calculation process of the fuzzy inference. FINEST provides a framework to tune any parameter, which appears in the nodes of the network representing the calculation process of the fuzzy data if the derivative function with respect to the parameters is given.

Figure 5.7 FINEST Architecture 5.1.7 SONFIN Architecture The Self Constructing Neural Fuzzy Inference Network (SONFIN) architecture implements a modified Takagi-Sugeno FIS and is illustrated in figure 5.8. In the structure identification of the precondition part, the input space is partitioned in a flexible way according to an

52

aligned clustering based algorithm. As to the structure identification of the consequent part, only a singleton value selected by a clustering method is assigned to each rule initially. Afterwards, some additional significant terms (input variables) selected via a projection-based correlation measure for each rule are added to the consequent part incrementally as learning proceeds. For parameter identification, the consequent parameters are tuned optimally by either least mean squares or recursive least squares algorithms and the precondition parameters are tuned by back propagation algorithm.

Figure 5.8 SONFIN Architecture

53

5.1.8 EFuNN Architecture In Evolving Neural Fuzzy Network EFuNN all nodes are created during the learning phase. The first layer passes data to the second layer that calculates the degrees of compatibility in relation to the predefined membership functions. The third layer contains fuzzy rule nodes representing prototypes of input- output data as an association of hyperspheres from the fuzzy input and fuzzy output spaces. Each rule node is defined by two vectors of connection weights, which are adjusted through a hybrid learning technique. The fourth layer calculates the degree to which output membership functions are matched the input data and the fifth layer carries out the defuzzyfication and calculates the numerical value for the output variable. Neural Fuzzy Network (dmEFuNN) shown in figure 5.9 is a modified version of the EFuNN with the idea of not only the winning rule node’s activation is propagated but a group of rule nodes that is dynamic selected for every new input vector and their activation values are used to calculate the dynamical parameters of the output function. While EFuNN implements Mamdani type fuzzy rules, dmEFuNN implements TakagiSugeno fuzzy rules.

54

Figure 5.9 EFuNN Architecture In this research work, Adaptive Neuro Fuzzy Inference System (ANFIS) is taken into consideration for designing a neuro fuzzy controller network to control the two-stage KY boost converter. 5.2 ADAPTIVE NEURO-FUZZY INFERENCE SYSTEM (ANFIS) An Adaptive Neuro-Fuzzy Inference System (ANFIS) is a combination of neural network and fuzzy systems in such a way that neural network is used to determine the parameters of fuzzy system. ANFIS largely removes the requirement for manual optimization of the fuzzy system parameters. A neural network is used to automatically tune

55

the system parameters, for example the membership functions bounds, leading to improved performance without operator invention. The neuro fuzzy system with the learning capability of neural network and with the advantages of the rule-base fuzzy system can improve the performance significantly and can provide a mechanism to incorporate past observations into the classification process. In neural network the training essentially builds the system. However, using a neuro fuzzy scheme, the system is built by fuzzy logic definitions and is then refined using neural network training algorithms. Some advantages of ANFIS are:  Refines fuzzy if-then rules to describe the behaviour of a complex system.  Does not require prior human expertise  Uses membership functions with desired dataset to approximate  Greater choice of membership functions to use.  Very fast convergence time. 5.2.1 ANFIS ARCHITECTURE The ANFIS is a fuzzy Sugeno model put in the framework of adaptive systems to facilitate learning and adaptation (Jang, 1993 [29], 1995 [30]). Such framework makes the ANFIS modeling more systematic and less reliant on expert knowledge. To present the ANFIS architecture, two fuzzy if-then rules based on a first order Sugeno model are

56

considered: Rule 1: If (x is A1 ) and (y is B1 ) then (f = p1 x + q1 y + r1 ) Rule 2: If (x is A2 ) and (y is B2 ) then (f2 = p 2x + q 2 y + r2 ) where x and y are the inputs, Aj and Bj are the fuzzy sets, are the outputs within the fuzzy region specified by the fuzzy rule, p, q and r are the design parameters that are determined during the training process. Figure 5.10 illustrates the reasoning mechanism for this Sugeno model where it is the basis of the ANFIS model.

Figure 5.10 A two-input first-order Sugeno fuzzy model with two rules The ANFIS architecture to implement these two rules is shown in Figure 5.11, in which a circle indicates a fixed node, whereas a square indicates an adaptive node. Adaptive neuro fuzzy inference system basically has 5 layer architectures and each of the function is explained in detail later.

57

Figure 5.11 ANFIS architecture In the first layer, all the nodes are adaptive nodes. The outputs of layer 1 are the fuzzy membership grade of the inputs, which are given by equations 5.1 and 5.2; (5.1)

(5.2) where µ Ai (x), µBi(y) can adopt any fuzzy membership function. For example, if the bell shaped membership function is employed, µA(x) is given by equation 5.3;

(5.3) where ai, b i and ci are the parameters of the membership function, governing the bell shaped functions accordingly.

58

In the second layer, the nodes are fixed nodes. They are labeled with n, indicating that they perform as a simple multiplier. The outputs of this layer can be represented as equation 5.4;

(5.4) which are the so-called firing strengths of the rules. In the third layer, the nodes are also fixed nodes labeled by N, to indicate that they play a normalization role to the firing strengths from the previous layer. The output of this layer can be represented as equation 5.5;

(5.5) which are the so-called normalized firing strengths. In the fourth layer, the nodes are adaptive. The output of each node in this layer is simply the product of the normalized firing strength and a first order polynomial (for a first order Sugeno model). Thus, the output of this layer is given by equation 5.6;

(5.6) In the fifth layer, there is only one single fixed node labeled with £. This node performs the summation of all incoming signals. Hence, the overall output of the model is given by equation 5.7;

(5.7)

59

It can be observed that there are two adaptive layers in this ANFIS architecture, namely the first and the fourth layers. In the first layer, there are three modifiable parameters {a, b, c }, which are related to the input membership functions. These parameters are the so-called premise parameters. In the fourth layer, there are also three modifiable parameters {p, q, r}, pertaining to the first order polynomial. These parameters are the so-called consequent parameters (Jang, 1993 [29], 1995 [30]). Figure 5.12 shows the variation in the Sugeno model that is equivalent to a two-input first-order Sugeno fuzzy model with nine rules, where each input is assumed to have three associated MFs. Figure 5.13 illustrates how the two dimensional input space is partitioned into nine overlapping fuzzy regions, each of which is governed by a fuzzy if-then rule. In other words, the premise part of a rule defines a fuzzy region, while the consequent part specifies the output within the region.

Figure 5.12 Two-input first-order Sugeno fuzzy model with nine rules

60

Figure 5.13 The input space that are partitioned into nine fuzzy regions 5.2.2 LEARNING ALGORITHM OF ANFIS The task of the learning algorithm for this architecture is to tune all the modifiable parameters, namely {aiy b, c} and {p, q, r}, to make the ANFIS output match the training data. When the premise parameters a, b and c of the membership function are fixed, the output of the ANFIS model can be written as equation 5.8;

(5.8) Substituting Equation 5.5 into Equation 5.8 yields equation 5.9:

(5.9) Substituting the fuzzy if-then rules into Equation 5.9, it become as shown in equation 5.10;

61

(5.10) After rearrangement, the output can be expressed as equation 5.11;

(5.11) which is a linear combination of the modifiable consequent parameters p1, q1, ri, p 2, q 2 and r2 . The least squares method can be used to identify the optimal values of these parameters easily. When the premise parameters are not fixed, the search space becomes larger and the convergence of the training becomes slower. A hybrid algorithm combining the least squares method and the gradient descent method is adopted to solve this problem. The hybrid algorithm is composed of a forward and a backward pass. The least squares method (forward pass) is used to optimize the consequent parameters with the premise parameters fixed. Once the optimal consequent parameters are found, the backward pass starts immediately. The gradient descent method (backward pass) is used to adjust optimally the premise parameters corresponding to the fuzzy sets in the input domain. The output of the ANFIS is calculated by employing the consequent parameters found in the forward pass. The following Table 5.1 summarizes the activities in each pass. The output error is used to adapt the premise parameters by means of a standard backpropagation algorithm. It has been proven that this hybrid algorithm is highly efficient in training the ANFIS (Jang, 1993 [29], 1995 [30]).

62

Table 5.1 Activities of forward and backward pass Forward pass Fixed Premise parameters Least-squares estimator Consequent Signals

Node outputs

Backward pass Gradient descent Fixed Error signals

5.2.3 ANFIS CLASSIFIER Both neural network and fuzzy logic are universal estimators. They can approximate any function to any prescribed accuracy, provided that sufficient hidden neurons and fuzzy rules are available. Gradient descent and Backpropagation algorithms are used to adjust the parameters of membership functions (fuzzy sets) and the weights of defuzzification (neural networks) for fuzzy neural networks. ANFIS applies two techniques in updating parameters. The ANFIS is a FIS implemented in the framework of an adaptive fuzzy neural network. It combines the explicit knowledge representation of a FIS with the learning power of ANNs. The objective of ANFIS is to integrate the best features of fuzzy systems and neural network. The advantage of fuzzy is that prior knowledge is represented into a set of constraints to reduce the optimization research space. The adaptation of back propagation to structured network so as to automate fuzzy control parametric tuning is utilized from NN. For premise parameters that define membership functions, ANFIS employs gradient descent algorithm to fine-tune them. For consequent parameters that define the coefficients of each equation, ANFIS uses the least- squares method to identify them. This approach is thus called hybrid learning method since it combines gradient descent

63

algorithm and least-squares method. To achieve good generalization of unseen data, the size of the training data set should be at least as big as the number of modifiable parameters in ANFIS. Functionally there are almost no constrains on the node functions of an adaptive network except for the requirement of piecewise differentiability. The neurons in ANFIS have different structures. •

The Membership function is defined by parameterized soft

trapezoids (Generalized Bell Functions). •

The rules are differentiable T-norm usually product.

•

The Normalization is by Sum and Arithmetic division.

•

Functions are linear regressions and multiplication normalized

weights and Output (Algebraic Sum). 5.3 ANFIS EDITOR GUI The ANFIS Editor GUI menu bar can be used to load a FIS training initialization, save the trained FIS, open a new Sugeno system or any of the other GUIs to interpret the trained FIS model. Any data set is loaded into the ANFIS Editor GUI, (or that is applied to the commandline function ANFIS) must be a matrix with the input data arranged as vectors in all but the last column. The output data must be in the last column. A sample of ANFIS Editor GUI with input is shown in Figure 5.14.

64

Figure 5.14 ANFIS Editor GUI 5.4 FIS EDITOR The FIS Editor displays general information about a fuzzy inference system. There is a simple diagram at the top that shows the names of each input variable on the left, and those of each output variable on the right. The sample membership functions shown in the boxes are just icons and do not depict the actual shapes of the membership functions. This is shown in Figure 5.15.

65

Figure 5.15 FIS Editor 5.5 MEMBERSHIP FUNCTIONS EDITOR The Membership Function Editor shares some features with the FIS Editor. In fact, all of the five basic GUI tools have similar menu options, status lines, and Help and Close buttons. The Membership Function Editor is the tool that lets you display and edits all of the membership functions associated with all of the input and output variables for the entire fuzzy inference system. Figure 5.16 shows the membership function editor.

66

Figure 5.16 Membership Function Editor 5.6 RULE EDITOR The Rule Editor allows user to construct the rule statements automatically, by clicking on and selecting one item in each input variable box, one item in each output box, and one connection item as in Figure 5.17. Choosing none as one of the variable qualities will exclude that variable from a given rule. Choosing not under any variable name will negate the associated quality. Rules may be changed, deleted, or added, by clicking the appropriate button.

67

Figure 5.17 Rule Editor 5.7 RULE VIEWER The Rule Viewer as shown in Figure 5.18 allows users to interpret the entire fuzzy inference process at once. The Rule Viewer also shows how the shape of certain membership functions influences the overall result. Since it plots every part of every rule, it can become unwieldy for particularly large systems, but, for a relatively small number of inputs and outputs, it performs well (depending on how much screen space we devote to it) with up to 30 rules and as many as 6 or 7 variables. The Rule Viewer shows one calculation at a time and in great detail. In this sense, it presents a sort of micro view of the fuzzy inference system.

68

Figure 5.18 Rule Viewer 5.8 ANFIS MODEL STRUCTURE After the FIS is generated, the model structure can be viewed by clicking the Structure button in the middle of the right side of the GUI. A GUI can be seen in Figure 5.19. The branches in this graph are color coded. Color coding of branches characterize the rules and indicate whether or not and, not, or or are used in the rules. The input is represented by the left-most node and the output by the right-most node. The node represents a normalization factor for the rules. Clicking on the nodes indicates information about the structure.

69

Figure 5.19 ANFIS Model Structure 5.9 MODELING DATA THROUGH ANFIS The modeling approach used by ANFIS is similar to many system identification techniques. First, a parameterized model structure (relating inputs to membership functions to rules to outputs to membership functions, and so on) is hypothesized. Next, input/output data is collected in a form that will be usable by ANFIS for training. ANFIS can then be used to train the FIS model to emulate the training data presented to it by modifying the membership function parameters according to a chosen error criterion. Figures 5.20 and 5.21 shows the example training error in ANFIS and output of the ANFIS.

70

In general, this type of modeling works well if the training data presented to ANFIS for training (estimating) membership function parameters is fully representative of the features of the data that the trained FIS intend to model. This is not always the case, however. In some cases, data is collected using noisy measurements, and the training data cannot be representative of all the features of the data that will be presented to the model. This is where model validation comes into play. [Jang, 1993 [29]].

Figure 5.20 Training Error

71

Figure 5.21 Output of ANFIS 5.10 MODEL VALIDATION (CHECKING AND TESTING DATA) Model validation is the process by which the input vectors from input/output data sets on which the FIS was not trained, are presented to the trained FIS model, to see how well the FIS model predicts the corresponding data set output values. This is accomplished with the ANFIS Editor GUI using the so-called testing data set, and its use is described in a subsection that follows. Another type of data set can also be used for model validation in ANFIS. This type of validation data set is referred to as the checking

72

data set and it set is used to control the potential for the model over fitting the data. When the checking and training data are presented to ANFIS, the FIS model having parameters associated with the minimum checking data model error is then selected. The problem with model validation for models constructed using adaptive techniques is selecting a data set that is both representative of the data the trained model intends to emulate, yet sufficiently distinct from the training data set so as not to render the validation process trivial. If a large amount of data has been collected, hopefully this data contains all the necessary representative features, so the process of selecting a data set for checking or testing purposes is made easier. 5.11 SUMMARY In this chapter various architectures of neuro fuzzy system is explained in detail and also the modeling and software implementation of ANFIS architecture was explained in detail.