BEYOND INTEGRATED NEURO-FUZZY SYSTEMS: REVIEWS, PROSPECTS, PERSPECTIVES AND DIRECTIONS Ajith Abraham School of Computing and Information Technology Monash University (Gippsland Campus), Churchill, Victoria, Australia – 3842 http://ajith.softcomputing.net, Email:
[email protected]
Abstract Neuro-fuzzy computing, which provides efficient information processing capability by devising methodologies and algorithms for modeling uncertainty and imprecise information, forms at this juncture, a key component of soft computing. An integrated neuro-fuzzy system is simply a fuzzy inference system trained by a neural networklearning algorithm. The learning mechanism fine-tunes the underlying fuzzy inference system. Integrated neurofuzzy model makes use of the synergetic and complementary features of neural networks and fuzzy inference system and in most cases better results can be obtained rather than in a stand-alone mode. This paper presents some short fundamental concepts and modeling aspects of neuro-fuzzy systems emphasizing on Takagi Sugeno and Mamdani fuzzy inference system. Some short reviews of neuro-fuzzy models that have evolved in the past few years are discussed further. The paper concludes with an attempt to throw light on future research directions and some of the problems to be addressed that go beyond the current neuro-fuzzy algorithms.
1
Introduction
There are different approaches in the field of artificial intelligence for realizing human intelligence in machines. In recent years, neuro-fuzzy systems have found a wide gamut of industrial and commercial applications that requires the analysis of uncertain and imprecise information. Fuzzy inference systems and neural networks are complementary technologies in the design of adaptive intelligent systems. Artificial Neural Network (ANN) learns from scratch by adjusting the interconnections between layers. A valuable property of neural network is that of generalization, whereby a trained network is able to provide a correct matching in the form of output data for a set of previously unseen input data. Fuzzy Inference System (FIS) is a popular computing framework based on the concept of fuzzy set theory, fuzzy if-then rules, and fuzzy reasoning. With crisp inputs and outputs, fuzzy inference system implements a nonlinear mapping from its input space to output space by a number of if-then rules. There are several approaches to integrate neural networks and fuzzy inference systems and very often it depends on the application [9] [12]. However, this paper will focus only on fully integrated neuro-fuzzy models based on Takagi Sugeno and Mamdani FIS. This paper starts with some basic theoretical aspects of neuro-fuzzy systems and further analyzes some of the popular neuro-fuzzy models that have evolved in the past few years. Conclusions, research issues and future directions are also provided towards the end.
2 Neuro-Fuzzy Systems A FIS can utilize human expertise by storing its essential components in rule base and database, and perform fuzzy reasoning to infer the overall output value. However there is no systematic way to transform experiences of knowledge of human experts to the knowledge base of a FIS. For building a FIS, we have to specify the fuzzy sets, fuzzy operators and the knowledge base. For constructing an ANN for an application the user needs to specify the architecture and learning algorithm. Learning mechanism of ANN does not rely on human expertise. Due to the homogenous structure of ANN, it is difficult to extract structured knowledge from either the weights or the configuration of the ANN. For many practical problems, a priori knowledge is usually obtained from human experts and it is most appropriate to express the knowledge as a set of fuzzy if-then rules. However, it is not easy to encode prior knowledge into an ANN. Table 1 summarizes the comparison of ANN and FIS. Integrated neuro-fuzzy system combines the advantages of ANN and FIS. While the learning capability is an advantage from the viewpoint of FIS, the formation of linguistic rule base will be advantage from the viewpoint of ANN. Integrated neuro-fuzzy systems share data structures and knowledge representations. A common way to apply a learning algorithm to a fuzzy system is to represent it in a special ANN like architecture. However the conventional ANN learning algorithms (gradient descent) cannot be applied directly to such a system as the functions used in the inference process are usually non differentiable. This problem can be tackled by using differentiable functions in the inference system or by not using the standard neural learning algorithm. Some of the major works in this area are GARIC [11], FALCON [9], ANFIS [4], NEFCON [9], FUN [7], SONFIN [6], FINEST [8], EFuNN [5], dmEFuNN [5], evolutionary design of neuro fuzzy systems [2] and many others [1] [3].
Table 1. Comparison between neural networks and fuzzy inference system Neural Networks
Fuzzy Inference System
Prior rule-based knowledge cannot be used
Prior rule-based can be incorporated
Learning from scratch
Cannot learn (use linguistic knowledge)
Black box
Interpretable (if-then rules)
Complicated learning algorithms
Simple interpretation and implementation
Difficult to extract knowledge
Knowledge must be available
Based on the type of fuzzy inference method used, we can classify integrated neuro-fuzzy systems into different types. In the remaining portion of this section we will discuss how to model neuro-fuzzy systems implementing Mamdani and Takagi Sugeno FIS [2]. y
y
Layer 6 rule inference layer
Layer 5 rule inference and defuzzification layer
Layer 4 rule consequent layer
Layer 5 rule consequent layer
x1
x2 Layer 4 rule strength normalization
R
1
R
2
Layer 3 rule antecedent layer
R
3
R
1
R
2
R
3
Layer 3 rule antecedent layer
Layer 2 (fuzzification layer)
Layer 2 (fuzzification layer)
Layer 1 (input layer)
x1
x2
Figure 1. Generic Mamdani Neuro-fuzzy system
Layer 1 (input layer)
x1
x2
Figure 2. Generic Takagi Sugeno Neuro-fuzzy system
Neuro-fuzzy system (Mamdani FIS) The architecture of a generic Mamdani fuzzy inference system is depicted in Figure 1.The detailed functioning of each layer is as follows: Layer -1(input layer): No computation is done in this layer. Each node in this layer, which corresponds to one input variable, only transmits input values to the next layer directly. The link weight in layer 1 is unity. Layer-2 (fuzzification layer): Each node in this layer corresponds to one linguistic label (excellent, good, etc.) to one of the input variables in layer 1. In other words, the output link represent the membership value, which specifies the degree to which an input value belongs to a fuzzy set, is calculated in layer 2. A clustering algorithm will decide the initial number and type of membership functions to be allocated to each of the input variable. The final shapes of the MFs will be fine tuned during network learning. Layer-3 (rule antecedent layer): A node in this layer represents the antecedent part of a rule. Normally a T-norm operator is used in this node. The output of a layer 3 node represents the firing strength of the corresponding fuzzy rule. Layer-4 (rule consequent layer): This node basically has two tasks. To combine the incoming rule antecedents and determine the degree to which they belong to the output linguistic label (high, medium, low, etc.). The number of nodes in this layer will be equal to the number of rules. Layer-5 (Combination and defuzzification layer): This node does the combination of all the rules consequents using a T-conorm operator and finally computes the crisp output after defuzzification.
Neuro-fuzzy system (Takagi-Sugeno FIS) Block diagram of a Takagi Sugeno Neuro-fuzzy model is given in Figure 2. Layers 1,2 and 3 functions the same way as Mamdani FIS. The complete functioning of the remaining layers is as follows. Layer 4 (rule strength normalization): Every node in this layer calculates the ratio of the i-th rule’s firing strength to the sum of all rules firing strength.
wi =
wi , i = 1,2.... . w1 + w 2
Layer-5 (rule consequent layer): Every node i in this layer is with a node function
wi f i = wi ( pi x 1 + qi x 2 + ri ) , where wi is the output of layer 4, and {pi , qi , ri } is the parameter set. A well established way is to determine the consequent parameters using the least means squares algorithm as used in ANFIS and SONFIN. Layer-6 (rule inference layer) The single node in this layer computes the overall output as the summation of all åw f incoming signals: Overall output = å wi f i = i i i . å iwi i Learning in Fused NF Systems
Takagi Sugeno neuro-fuzzy systems make use of a mixture of back propagation to learn the membership functions) and least mean square estimation to determine the coefficients of the linear combinations in the rule’s conclusions. A step in the learning procedure got two parts: In the first part the input patterns are propagated, and the optimal conclusion parameters are estimated by an iterative least mean square procedure, while the antecedent parameters (membership functions) are assumed to be fixed for the current cycle through the training set. In the second part the patterns are propagated again, and in this epoch, back propagation is used to modify the antecedent parameters, while the conclusion parameters remain fixed. This procedure is then iterated. A Mamdani neuro-fuzzy system uses a supervised learning technique (backpropagation learning) to learn the membership functions.
3
Neuro-Fuzzy Systems – Reviews and Prospects
ANFIS implements a Takagi Sugeno inference system. In ANFIS [4] the adaptation (learning) process is only concerned with parameter level adaptation within fixed structures. The user has to specify the quantity and type of membership functions assigned to each input variable. The rulebase must be known in advance, as ANFIS adjusts only the membership functions of the antecedent and consequent parameters. For large-scale problems, it will be too complicated to determine the optimal premise-consequent structures, rule numbers etc. The structure of ANFIS ensures that each linguistic term is represented by only one fuzzy set. However the learning procedure of ANFIS does not provide the means to apply constraints that restrict the kind of modifications applied to the membership functions. When using Gaussian membership functions, operationally ANFIS can be compared with a radial basis function network. ANFIS performance will depend on the type and quantity of membership functions for each input variable, learning rates etc. It is difficult for ANFIS to handle high-dimensional problems, as this leads to a large number of input partitions, rules, and hence consequent parameters. GARIC [11] implements a neuro-fuzzy controller by using two neural network modules, the ASN (Action Selection Network) and AEN (Action state Evaluation Network). The AEN is an adaptive critic that evaluates the actions of the ASN. GARIC uses a differentiable soft minimum function to implement a fuzzy controller by a complicated supervised learning algorithm. GARIC is not easily interpretable (as Takagi Sugeno or Mamdani FIS) compared to other neuro-fuzzy counterparts. Initialisation using prior knowledge is also not easy. GARIC was further improved to incorporate prior knowledge, which is refined using reinforcement learning. Like other neuro-fuzzy systems, in FALCON [9] the adaptation process is only concerned with parameter level adaptation within fixed structures. FALCON uses a hybrid-learning algorithm comprising of unsupervised learning to locate initial membership functions/rulebase and a gradient descent learning to optimally adjust the parameters of the membership functions to produce the desired outputs. NEFCON [9] implements a Mamdani FIS and makes use of a reinforcement type of learning algorithm for learning the rule base (structure learning) and a fuzzy backpropagation algorithm for learning the fuzzy sets (parameter learning). NEFCON system is capable of incorporating prior knowledge as well as learning from scratch
(incremental learning). NEFCON uses a fuzzy error based on fuzzy goodness measure for the reinforcement learning. However the performance of the system will very much depend on heuristic factors like learning rate, error measure etc. Even though NEFCON was designed to perform as an intelligent controller, the algorithms were later modified to perform function approximation (NEFPROX) and pattern classification (NEFCLASS). FINEST [8] provides a mechanism based on the improved generalized modus ponens for fine-tuning of fuzzy predicates and the inference mechanism. Parameterization of the inference procedure is very much essential for proper application of the tuning algorithm. Aggregation functions, implication functions, combination functions and funny predicates can be tuned using FINEST. The FINEST system can also be used as a tool for quantifying the fuzzy meaning of sentences expressed in the form of fuzzy rules. SONFIN [6] is very much similar to ANFIS in terms of architecture but is adaptable to the users specification of required accuracy. The consequent parts are modified according to the users specified accuracy. Hence SONFIN could perform faster if the accuracy is sacrificed. Precondition parameters are tuned by backpropagation algorithm and consequent parameters by least mean squares or recursive least squares algorithms. Like ANFIS, SONFIN suffers high-dimensionality problem and the performance will depend on the type and quantity of membership functions for each input variable, learning rates etc. FUN [7]system is initialised by specifying a fixed number of rules and a fixed number of initial fuzzy sets for each variable and the network learns through a stochastic procedure that randomly changes parameters of membership functions and connections within the network structure. Rule base is optimised by a stochastic search procedure in the rule space. Membership functions are modified by a stochastic procedure and fine-tuned by gradient descent algorithm. FUN can be used to optimise an existing rule base. EFuNN [5] implements a Mamdani type of fuzzy rule base, based on a dynamic structure (creating and deleting strategy), and single rule inference, established on the winner-takes all rule for the rule node activation, with a onepass training, instance based learning and reasoning. dmEFuNN [5] is an improved version of the EFuNN capable of implementing Takagi-Sugeno inference system, using several (m) of the highest activated rule nodes instead of one (like EFuNN). The rule node aggregation is achieved by a C-means clustering algorithm. As EFuNN and dmEFuNN use single pass training (one epoch), it is highly suitable for online training. However an important disadvantage of EFuNN is the determination of the optimal network parameters like sensitivity threshold, error threshold and the learning rates. Even though a trial and error approach is practical, when the problem becomes complicated (large number of input variables) determining the optimal parameters will be a tedious task. Evolutionary design of neuro-fuzzy systems [2] is an attempt to optimize the design of neuro-fuzzy systems. The architecture of the evolving neuro fuzzy model can adapt to Mamdani or Takagi Sugeno type fuzzy inference system. The evolving mechanism can change their node functions, architecture and learning parameters according to different environments without human intervention. The evolving mechanism works on a hierarchical 5-tier evolutionary search procedure. The real success in modeling such systems will directly depend on the genotype representation of the different layers. Hierarchical evolutionary search processes attract considerable computational effort. Fortunately evolutionary algorithms work with a population of independent solutions, which makes it easy to distribute the computational load among several processors using parallel algorithms.
4. Beyond Neuro-Fuzzy Systems: Perspectives, Challenges and Directions Neuro-fuzzy computing is a boon to solve many of our complicated problems. Apart from control engineering, neuro-fuzzy systems have found wide applications in several function approximation problems, pattern recognition, as a backbone of several intelligent systems, financial modelling, etc. Knowledge acquisition and data mining is an important neuro-fuzzy application area where the if-then rules could aid the knowledge discovery within the data. Several experimentation results have shown that neuro-fuzzy systems do perform better than the stand-alone techniques [14-19]. Compared to a pure neural network approach a neuro-fuzzy system can learn the fuzzy sets and fuzzy rules which on one hand can perform the desired results, and on the other hand can also be linguistically interpreted. Thus it is easy to check the plausibility and to incorporate prior knowledge. Like most biological systems, which can adapt to any environment, we need fully adaptable neuro-fuzzy systems to tackle the demanding real world problems of the future. Most of the existing neuro-fuzzy models rely on user specified network parameters. For the system to be fully adaptable, performance should not be heavily dependant on user specified parameters. The neuro-fuzzy system should be able to learn from data in a continuous, incremental way, able to grow as they operate, update their knowledge and refine the model through interaction with the environment. The intelligence of such systems could be further improved if the adaptation process could learn from success and mistakes and apply that knowledge to new problems.
Neuro-fuzzy systems are based on Mamdani or Takagi Sugeno fuzzy inference system. More research is to be diverted to develop other inference methods which are computational inexpensive and efficient. Gradient descent learning methods are used for adaptation (parameter learning) in neuro-fuzzy systems. Second order optimization algorithms (conjugate gradient, Levenberg Marquardt algorithm etc.) have been successful in faster neural network training and achieving better generalization performance. Similar techniques could be easily extended to neurofuzzy systems. Overtime we become trapped in our shared visions of appropriate ways to tackle problems. Sometimes, it is worth stepping back and taking an entirely new look and re-engineer the problem solving methods. All the way, we were trying to mimic the human brain: the way of reasoning, modelling uncertainty, learning etc. It is time that we should think about building a system that would perform better than humans!
5
Conclusions
It is predicted, in the 21st century fundamental resource of wealth will be knowledge and communication rather than natural resources and physical labour. With the exponential growth of information in this world we need intelligent systems that learn from data in a continuous, incremental way, able to grow as they operate, update their knowledge and refine the model through interaction with the environment. Optimizing fuzzy systems has always been an interesting research area ever since they found several industrial applications. Neuro-fuzzy systems are the best answer to automate or support the process of developing a fuzzy inference system. This paper summarizes the strengths and weaknesses of some of the popular integrated neuro-fuzzy models. Takagi Sugeno neuro-fuzzy system performs better than Mamdani type even though it is computational expensive. However, Mamdani neuro-fuzzy system is faster and ideal for online learning. As a guideline, for NF systems to be highly intelligent some of the major requirements are fast learning (memory based - efficient storage and retrieval capacities), on-line adaptability (accommodating new features like inputs, outputs, nodes, connections etc), achieve a global error rate and computationally inexpensive. The data acquisition and preprocessing training data is also quite important for the success of neuro-fuzzy system implementation.
References [1]
Abraham A, Neuro-Fuzzy Systems: State-of-the-Art Modeling Techniques, In Proceedings of the Sixth International Work Conference on Artificial and Natural Neural Networks, IWANN 2001, Granada, Springer Verlag Germany, June 2001, (Forthcoming).
[2]
Abraham A and Nath B, Evolutionary Design of Neuro-Fuzzy Systems – A Generic Framework, In Proceedings of the 4th Japan – Australia joint Workshop on Intelligent and Evolutionary Systems, Japan, November 2000.
[3]
Abraham A & Nath B, Designing Optimal Neuro-Fuzzy Systems for Intelligent Control, In proceedings of the Sixth International Conference on Control Automation Robotics Computer Vision (ICARCV 2000), Singapore, December 2000.
[4]
Jang R, Neuro-Fuzzy Modeling: Architectures, Analyses and Applications, PhD Thesis, University of California, Berkeley, July 1992.
[5]
Kasabov N and Qun Song, Dynamic Evolving Fuzzy Neural Networks with 'm-out-of-n' Activation Nodes for On-line Adaptive Systems, Technical Report TR99/04, Department of information science, University of Otago, 1999.
[6]
Juang Chia Feng, Lin Chin Teng, An Online Self Constructing Neural Fuzzy Inference Network and its Applications, IEEE Transactions on Fuzzy Systems, Vol 6, No.1, pp. 12-32, 1998.
[7]
Sulzberger SM, Tschicholg-Gurman NN, Vestli SJ, FUN: Optimization of Fuzzy Rule Based Systems Using Neural Networks, In Proceedings of IEEE Conference on Neural Networks, San Francisco, pp 312-316, March 1993.
[8]
Tano S, Oyama T, Arnould T, Deep combination of Fuzzy Inference and Neural Network in Fuzzy Inference, Fuzzy Sets and Systems, 82(2) pp. 151-160, 1996.
[9]
Nauck D, Klawonn F and Kruse R, Foundations of Neuro-Fuzzy Systems, John Wiley and Sons, 1997.
[10] Lin C T and Lee C S G, Neural Network based Fuzzy Logic Control and Decision System, IEEE Transactions on Comput. (40(12): pp. 1320-1336, 1991.
[11] Bherenji H R and Khedkar P, Learning and Tuning Fuzzy Logic Controllers through Reinforcements, IEEE Transactions on Neural Networks, Vol (3), pp. 724-740, 1992. [12] Jang R, Sun C T and Mizutani E, Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence, Prentice Hall NJ, 1997. [13] Lin C T and George Lee C S, Neural Fuzzy Systems: A Neuro-Fuzzy Synergism to Intelligent Systems, Prentice Hall NJ, 1995. [14] Abraham A, Philip N S and Joseph B, Will We Have a Wet Summer? Soft Computing Models for Long- term Rainfall Forecasting (ESM 2001), 15th European Simulation Multiconference, Prague, June 2001. (Forthcoming) [15] Abraham A, A Soft Computing Approach for Fault Prediction of Electronic Systems, The 2nd International Conference on Computers in Industry ICCI 2000, Bahrain, pp. 83-91, November 2000. [16] Abraham A and Nath B, A Neuro-Fuzzy Approach for Forecasting Electricity Demand In Victoria, Applied Soft Computing Journal, Elsevier Science, April 2001. (Forthcoming). [17] Abraham A, An Evolving Fuzzy Neural Network Model Based Reactive Power Control, In Proceedings of The Second International Conference on Computers In Industry (ICCI 2000), Bahrain, pp. 247-253, (13-15), November 2000. [18] Sonja Petrovic-Lazerevic, Ken Coghill & Ajith Abraham, Neuro-Fuzzy Support of Knowledge Management in Social Regulation, In Proceedings of Fifth International Conference on Computing Anticipatory Systems, CASYS 2001, Belgium, August 2001. (Forthcoming) [19] Sorwar G, Abraham A & Dooley L, Texture Classification Based on DCT and Soft Computing, The 10th IEEE International Conference on Fuzzy Systems, Melbourne, Australia, (02-05) December 2001. (Submitted)
