Hybrid Model for the Training of Interval Type-2 Fuzzy Logic System Saima Hassan1? , Abbas Khosravi2 , Jafreezal Jaafar1 , and Mojtaba Ahmadieh Khanesar,3 1 Department of CIS, Universiti Teknologi PETRONAS, Malaysia Centre for Intelligent Systems Research, Deakin University, Australia Faculty of Electrical and Computer Engineering, Semnan University, Iran
2 3
Abstract. In this paper, a hybrid training model for interval type-2 fuzzy logic system is proposed. The hybrid training model uses extreme learning machine to tune the consequent part parameters and genetic algorithm to optimize the antecedent part parameters. The proposed hybrid learning model of interval type-2 fuzzy logic system is tested on the prediction of Mackey-Glass time series data sets with different levels of noise. The results are compared with the existing models in literature; extreme learning machine and Kalman filter based learning of consequent part parameters with randomly generated antecedent part parameters. It is observed that the interval type-2 fuzzy logic system provides improved performance with the proposed hybrid learning model. Keywords: Hybrid learning model, Extreme learning machine, genetic algorithm, interval type-2 fuzzy logic system, prediction.
1
Introduction
Information deficiencies such as incomplete, fragmentary, not fully reliable, vague and contradictory information [1] results in uncertainties in data and a process. Type-1 fuzzy logic system (T1FLS) can only handle the uncertainties about the meaning of the words by using precise membership functions. The choice of T1FLS is not appropriate in the presence of other sources of uncertainties in the real world data as it may cause problem in determining the exact and precise parameters of both the antecedents and consequents [2]. However, T2FLS can handle all type of uncertainties with their fuzzy grades [3]. T2FLS is computationally demanding because of the extra dimension. Interval T2FLS (IT2FLS) is the simplest form of T2FLS as all points in the third dimension are at unity and can be ignored for modelling purposes [4]. Though improvements of IT2FLS to its earlier version have been evidenced, yet it still lacks a systematic and coherent design. Different learning algorithms proposed for parameters optimization of IT2FLS include back propagation based learning ?
Kohat University of Science and Technology, Kohat, Pakistan. +60-104631749,
[email protected].
2
Lecture Notes in Computer Science: Authors’ Instructions
method [5], genetic and other bio-inspired algorithms [6–9], ant colony optimization [10], and extended Kalman filter based learning algorithm [11]. A hybrid model for IT2FLS was also proposed by [12] using orthogonal least-squares and back-propagation methods. The distressing issues of learning algorithms i.e. stopping criteria, learning rate, learning epochs and local minima may not be handled by the conventional learning algorithms. Huang et al. [13] introduced extreme learning machine (ELM) that can solve the stated issues of conventional training methods. Jang et al. established a functional equivalent between fuzzy and single hidden layer feed-forward neural network (SLFN) [14], that made it possible to hybridize fuzzy and ELM. Different hybrid models of fuzzy and ELM reported in literature include an evolutionary fuzzy extreme learning machine analyzed for mammographic risk [15], a hybrid model of fuzzy and ELM for fault detection method in power generation plant [16], ELM based fuzzy inference system [17] and an online sequential fuzzy ELM for function approximation and classification problems [18]. The antecedent parameters in the above mentioned models were randomly assigned and the consequent parameters were determined analytically. However, there are chances that the randomly assigned parameters might not create suitable membership function in fuzzy model. As it is noted that the randomly generated parameters may not be effective for network output [19] and can cause high learning risks [20] due to overfitting. Soon after the realization of this issue, optimal parameters (hidden node) are reported for ELM [19, 21, 22, 20]. However the hybrid model of fuzzy and ELM have not yet been reported with optimal parameters and are generated randomly. ELM is an efficient learning algorithm for T2FLSs [23], however, the antecedent part parameters were generated randomly. Inspired by the competitive performance of the T2FLS with ELM and motivated from the issue to find the optimal parameters of fuzzy and ELM hybrid model, this paper proposes a hybrid training model for IT2FLS, where the antecedent parameters are optimized using genetic algorithm (GA) and consequent parameters are determined analytically through ELM. GA proposes multiple solutions which evolve to find best point, so it is less probable that it fells in a local minima than other optimization methods. Moreover, this optimization method is suitable for the nonlinear optimization problems. These are the reasons why GA is proposed to be used to optimize the parameters of antecedent part. The proposed hybrid learning model for IT2FLS is described in Section 3. The parameters of optimization, such as length of chromosome, fitness function and simulation results are discussed in Section 4. Section 5 concludes the paper with some remarks and guidelines for future work.
2
Structure of the Interval Type-2 FLS used in this paper
An IT2FS A˜ can be defined as follows: Z Z A˜ = 1/(x, u) x∈X
u∈Jx
Jx ⊆ [0, 1]
(1)
Lecture Notes in Computer Science: Authors’ Instructions
Crisp Inputs
Fuzzifier
Rules
3
Crisp Output
Defuzzifier Type-1 FSs
Type-2 FSs
Inference Engine
Type-2 FSs
Type-reducer
Fig. 1. Block diagram of type-2 FLS.
A type-2 FLS (see Fig 1) maps crisp input into type-2 fuzzy sets by assigning membership grade to each fuzzy set in the Fuzzifier block. There are various type2 fuzzy membership functions (MFs), however the Gaussian MF is utilized here because of less parameters. The Gaussian MF with fixed mean mni and uncertain deviation [σin1 , σin2 ] can be represented as: 1 xi − mni 2 µA˜n (xi ) = exp[− ( ) ], i 2 σin as:
σin ∈ [σin1 , σin2 ]
(2)
where µA˜n is a Gaussian MF that has upper and lower MFs [µA˜n (xi ) µA˜n (xi )] i
i
i
µA˜n (xi ) = N (mni , σin2 ; xi )
(3)
µA˜n (xi ) = N (mni , σin1 ; xi )
(4)
i
i
A fuzzy Rule Base is a set of linguistic rules in the form of a two parts IF-THEN conditional statements. The IF part (known as antecedents) need to be satisfied to inferred the THEN part (known as consequents). The interval type-2 FLS’s with a rule-base of Nth rules (Rn ) are taken as: Rn : if x1 is A˜n1 ∧ x2 is A˜n2 ∧ · · · ∧ xd is A˜nd Then wn (x) = pn0 + pn1 x1 + · · · + pnd xd , n = 1, · · · , N In the Inference Engine, each fuzzy rule is premised on the input vector x = [x1 , x2 , · · · , xn ]T as a varying singleton wn . A˜ni is the ith IT2 fuzzy subset generated from the input variable xi in the nth rule domain. N and ∧ represent the number of fuzzy rules and conjunction operator respectively. pn = [pn0 , pn1 , · · · , pnd ]T denotes the nth fuzzy rule parameters of the consequent. The Output Processing block in type-2 FLS comprises of an additional component called the Type reducer followed by a Defuzzifier block. Because of the distinct nature of type-2 fuzzy membership functions, the output from the inference engine is type-2 FS. Since the defuzzifier block can only input the type-1
4
Lecture Notes in Computer Science: Authors’ Instructions
FSs to produce crisp output therefor, a type reducer is needed after the inference engine to produce a type-reduced set using a centroid calculation. This type-reduced set can be then defuzzified to crisp output.
3
Structure of the Hybrid Learning Model for Interval Type-2 FLS (IT2FELM-GA)
The major task in the design process of a type-2 FLS involves the selection of optimal parameters. In this paper, a hybrid learning model for IT2FLS is proposed based on ELM and GA. The proposed hybrid learning model tune the consequent part parameters using ELM with randomly generated antecedent part parameters initially. The antecedent part parameters are then encoded as chromosome and optimize using GA in the direction of having better performance. Figure 2 shows the flowchart of the hybrid learning model of IT2FLS using ELM and GA.
Start Initialize the Type-2 Gaussian MF parameters (m,δ1 and δ2) Calculate the Type-2 FLS’s consequents part using ELM
Use the optimal antecedent parameters
Initialize the consequents
Tune the consequents
Repeat the computation for test set Calculate the objective function for each IT2FLS GA operations: selection, crossover and mutation
No
stopping criteria achieved Yes
End
Fig. 2. Flowchart of the hybrid learning model of IT2FLS model.
Lecture Notes in Computer Science: Authors’ Instructions
3.1
5
Optimal Parameters Using GA
GA, an optimization tool, is based on a formalization of natural selection and genetics. A population of chromosomes, objective function and stopping criteria are defined in GA. The population then undergoes genetic operation to evolve and the best population is selected based on the objective function. An IT2FS described by a Gaussian membership function with fixed mean and uncertain standard deviation is encoded into a population of chromosomes. Root means square error (RMSE) is defined as the fitness function for the determination of the best chromosomes. Maximum number of iterations and relatively small changes in the value of RMSE are the stopping criteria. The GA runs for each iteration and calculated the RMSE for the IT2FLS with the consequents parameters are learnt through ELM. Learning of consequents through ELM is described in the next section. The optimal parameters are achieved once GA stops with the minimum RMSE. These optimal parameters are then used in the ELM strategy to develop a hybrid learning model for IT2FLS. 3.2
Extreme Learning Machine strategy for Interval Type-2 FLS (IT2FELM)[24]
ELM is originally proposed for SLFN [13]. From the functional relationship between FLSs and neural networks [14, 25], it is observed that under some mild conditions FLSs can be interpreted as a special case of SLFN and can be trained using its learning algorithms. The ELM considers the fuzzy rules as hidden nodes of the SLFN [18]. Learning of IT2FLS using ELM is done in three steps. – Generate the antecedent parameters. – Initialize the Consequents parameters. – Tune the Consequents parameters. Generate the antecedent parameters In the beginning of IT2FELM-GA model the GA initialize a population of chromosomes randomly which are used as the initial set of antecedent parameters in the ELM strategy. Initialize the Consequents parameters After the antecedent parameters are set by the GA, the IT1 fuzzy set [yl , yr ] are initially used to initialize the consequent parameters using the fuzzy basic function as: yl =
N X
f´¯n wn ,
fn f´¯n = PN ¯´ n ´ =1 fn
(5)
f´ n wn ,
f f´ n = PN n
(6)
n=1
yr =
N X n=1
n ´ =1
fn´
The initialized consequent parameters in IT2FELM-GA are then expressed as a function of linear system using the ELM strategy. Under the constraints of
6
Lecture Notes in Computer Science: Authors’ Instructions
minimum least square, the linear system is optimized by ELM. Huang et al. [13] observed that such an optimal solution has the smallest least-squares norm and has a unique solution. Tune the Consequents parameters Having the optimized initial consequent parameters of the IT2FLS in section 3.2, the K-M algorithm [26] is utilized to obtain the final consequent parameters. The obtained final parameters are once again expressed as a function of linear system and is optimized using ELM. This step gives an optimized output of the IT2FLS. Repeat the Computation for Test Data set The above parts of subsections 3.2 described training of IT2FLS with ELM strategy. Since the antecedents are computed once, the test data set is utilized with the last two parts for the prediction purposes. 3.3
Objective Function Evaluation
Once the ELM strategy is finished, the chromosome in each iteration is evaluated using the objective function of RMSE. The chromosomes’ population having lowest RMSE represents the best population of the solution. The chromosome having best value of the objective function is saved in each iteration. 3.4
GA Operations
The current population of chromosome in IT2FELM-GA is updated to generate the new set of chromosomes for the next iteration using genetic operations of selection, crossover and mutation. Parents are selected using the tournament selection mechanism. Crossover operation creates new chromosomes inherit information (genes) from parents. The mutation operation introduce new genetic information hence promote diversity in population. These genetic operation are performed to evolve and optimize the encoded antecedent part parameters (chromosomes). These chromosomes are iteratively utilized in the ELM strategy of IT2FLS for several generations until the optimum solution is achieved.
4
Simulation Results
In this paper, an IT2FS described by a Gaussian membership function with 5 number of MFs (nMF) and 4 inputs is optimized using GA. This means that each parameter of the Gaussian MF requires 20 chromosomes. Thus, a total of 60 chromosomes of population 10 are generated randomly in the range [0,1]. One-point crossover with a probability of 0.8 is utilized. The proposed hybrid learning model of interval type-2 fuzzy logic system is tested on the prediction of Mackey-Glass time series data sets with different levels of noise. The results are compared with the existing model in literature; extreme
Lecture Notes in Computer Science: Authors’ Instructions
7
learning machine and Kalman filter based learning of consequent parameters with randomly generated antecedent parameters are used. The IT2FEKM-GA is tested on the prediction of Mackey-Glass time series data sets with different levels of noise. The noise-free Mackey-Glass time series data is generated using a nonlinear time-delay differential equation as expressed as follows: ax(t − τ ) dx(t) = − bx(t) (7) dt 1 + xn (t − τ ) where x(t) is the time series data at time t, a, b and n are constants and τ is the delay parameter used to produce chaotic behavior in the data. The discretiszed data is obtained for simulation using the Fourth-Order Runge-Kutta method with an initial condition x0 and a time step ts. Table 1 shows the numerical values to generate Mackey-Glass time series data. The dataset with 4 inputs and one output is extracted in the form of x(t − 18), x(t − 12), x(t − 6), x(t) and x(t + 6). By adding different levels of noise to the Mackey-Glass time series data, five noisy data sets are generated. The training and testing data sets are obtained with a ratio of (70/30). Table 1. Numerical Values of Mackey-Glass Time Series Data Parameter a b τ x0 ts nData
Value 0.2 0.1 17 1.2 0.1 12000
The best and average trend of convergence of the IT2FELM-GA can be seen in Figs 3 and 5. Continuous reduction of the best and average values of the fitness function is observed. The average value of the fitness function drops from 0.182 to 0.173 for the most noisy data (0db) and from 0.1 to 0.02 for 40db. In order to show the effectiveness of optimal parameters, the IT2FELM-GA is compared with IT2FELM [23, 24] and Kalman filter based IT2FLS (IT2FKF) [11], where the antecedent parts are generated randomly and consequent parts are learnt using ELM and KF respectively. The results of the last two models are taken after 10 runs. The minimum among the 10 RMSE is selected as the best performance of these models. Table 2 shows the results of IT2FELM-GA over IT2ELM and IT2FKF in terms of RMSE. The prediction results of IT2FELGA gradually decrease with decrease in the level of noise in data. It is also observed that the prediction results of IT2FELM-GA is very stable even with higher level of noise. The IT2FELM produces higher errors with higher level of noise. Whereas the IT2FKF produces good results in the presence of high level of noise as compared to IT2FELM. However, increase in RMSE value is observed
8
Lecture Notes in Computer Science: Authors’ Instructions Best: 0.0117628 Mean: 0.0128354
Best: 0.171435 Mean: 0.172404
0.1
0.185 Best fitness Mean fitness
Best fitness Mean fitness
0.09 0.08 0.18 Fitness value
Fitness value
0.07
0.175
0.06 0.05 0.04 0.03 0.02
0.17 0
1
2
3
4
5
6
Generation
7
8
9
10
0.01 0
1
2
3
4
5 Generation
6
7
8
9
10
Fig. 3. The convergence of IT2FELM-GA Fig. 4. The convergence of IT2FELM-GA for 0db Mackey Glass time series data. for 40db Mackey Glass time series data.
with lower level of noise. It may be due to the fact that the KF algorithm are designed to perform well with noisy data. Table 2. RMSE of IT2FELM-GA, IT2FELM and IT2FKF Obtained With noisy Mackey-Glass data sets.
0db 10db 20db 30db 40db
IT2FELM-GA IT2FELM IT2FKF 0.171 0.198 0.192 0.075 0.125 0.106 0.030 0.105 0.073 0.014 0.095 0.112 0.011 0.060 0.110
Figure 5 shows the actual data and the forecasts obtained with the hybrid learning algorithm of IT2FLS. As can be seen from the figure, the results of IT2FELM-GA is quite satisfactory.
5
Conclusion
In this paper, effectiveness of the optimal parameters in ELM based fuzzy model is demonstrated with a hybrid learning model for IT2FLS using GA and ELM. The consequent parameters are tuned using ELM whereas the antecedent parameters are encoded as a chromosome and are optimized using GA. The proposed hybrid learning model is compared with IT2FELM and IT2FKF where the antecedent parameters are generated randomly. Competitive performance of the
Lecture Notes in Computer Science: Authors’ Instructions
9
1.6 Actual IT2FELMGA 1.4
1.2
1
0.8
0.6
0.4
0.2 600
700
800
900 Data
1000
1100
1200
Fig. 5. Actual and forecasted times series data using IT2FELM-GA.
proposed hybrid learning model with optimal parameters is observed as compared to IT2FELM and IT2FKF in the presence of uncertainty. Uncertainty in models is introduced using noisy Mackey-Glass data sets. It is concluded that, the results achieved with randomly generated parameters in the ELM based fuzzy models can be optimized by using various optimization algorithms.
References 1. Klir, G.J., Wierman, M.J.: Uncertainty-Based Information: Elements of Generalized Information Theory. 2nd edn. Physica-Verlag (1999) 2. Hagras, H.: Type-2 flcs: A new generation of fuzzy controllers. Computational Intelligence Magazine, IEEE 2(1) (Feb 2007) 30–43 3. Mendel, J.M.: Sources of Uncertainty. In: Uncertain Rule-Based Fuzzy Logic Systems: Introduction and New Directions. Prentice-Hall PTR (2001) 66–78 4. Mendel, J., John, R., Liu, F.: Interval type-2 fuzzy logic systems made simple. Fuzzy Systems, IEEE Transactions on 14(6) (2006) 808–821 5. Mendel, J.M.: Computing derivatives in interval type-2 fuzzy logic systems. IEEE Transaction on Fuzzy Systems 12(1) (2004) 84–98 6. Wu, D., Tan, W.W.: Genetic learning and performance evaluation of interval type2 fuzzy logic controllers. Engineering Applications of Artificial Intelligence 19(8) (2006) 829 – 841 7. Castillo, O., Melin, P., Alanis, A., Montiel, O., Sepulveda, R.: Optimization of interval type-2 fuzzy logic controllers using evolutionary algorithms. Soft Computing 15(6) (2011) 1145–1160 8. Khosravi, A., Nahavandi, S., Creighton, D.: Short term load forecasting using interval type-2 fuzzy logic systems. In: Fuzzy Systems (FUZZ), 2011 IEEE International Conference on. (June 2011) 502–508 9. Maldonado, Y., Castillo, O., Melin, P.: Particle swarm optimization of interval type-2 fuzzy systems for {FPGA} applications. Applied Soft Computing 13(1) (2013) 496 – 508 10. Juang, C.F., Hsu, C.H., Chuang, C.F.: Reinforcement self-organizing interval type2 fuzzy system with ant colony optimization. In: Systems, Man and Cybernetics, 2009. SMC 2009. IEEE International Conference on. (Oct 2009) 771–776
10
Lecture Notes in Computer Science: Authors’ Instructions
11. Khanesar, M., Kayacan, E., Teshnehlab, M., Kaynak, O.: Extended kalman filter based learning algorithm for type-2 fuzzy logic systems and its experimental evaluation. Industrial Electronics, IEEE Transactions on 59(11) (Nov 2012) 4443–4455 12. Mendez, G.M., de los Angeles Hernandez, M.: Hybrid learning for interval type-2 fuzzy logic systems based on orthogonal least-squares and back-propagation methods. Information Sciences 179(13) (2009) 2146 – 2157 13. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: Theory and applications. Neurocomputing 70(1-3) (2006) 489 – 501 14. Jang, J.S., Sun, C.T.: Functional equivalence between radial basis function networks and fuzzy inference systems. Neural Networks, IEEE Transactions on 4(1) (1993) 156–159 15. Qu, Y., Shang, C., Wu, W., Shen, Q.: Evolutionary fuzzy extreme learning machine for mammographic risk analysis. International Journal of Fuzzy Systems 13(4) (2011) 282–291 16. Aziz, N.L.A., Yap, S., Bunyamin, M.A.: A hybrid fuzzy logic and extreme learning machine for improving efficiency of circulating water systems in power generation plant. In: IOP Conference Series: Earth and Environmental Science. Volume 16. (2013) 17. Sun, Z.L., Au, K.-F.and Choi, T.M.: Neuro-fuzzy inference system through integration of fuzzy logic and extreme learning machines. IEEE Trans Syst Man Cybern B Cybern. 37(5) (2007) 1321–1352 18. Rong, H.J., Huang, G.B., Sundararajan, N., Saratchandran, P.: Online sequential fuzzy extreme learning machine for function approximation and classification problems. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on 39(4) (Aug 2009) 1067–1072 19. Huang, G.B., Chen, L.: Enhanced random search based incremental extreme learning machine. Neurocomput. 71(16-18) (October 2008) 3460–3468 20. Zhang, Y., Cai, Z., Gong, W., Wang, X.: Self-adaptive differential evolution extreme learning machine and its application in water quality eva. Computational Inforation Systems 11(4) (2015) 1443 –1451 21. Feng, G., Huang, G., Lin, Q., Gay, R.: Error minimized extreme learning machine with growth of hidden nodes and incremental learning. IEEE Transaction on Neural Network 20(8) (2009) 1352–1359 22. Zhang, R., Lan, Y., Huang, G.B., Soh, Y.: Extreme learning machine with adaptive growth of hidden nodes and incremental updating of output weights. In Kamel, M., Karray, F., Gueaieb, W., Khamis, A., eds.: Autonomous and Intelligent Systems. Volume 6752 of Lecture Notes in Computer Science. Springer Berlin Heidelberg (2011) 253–262 23. Deng, Z., Choi, K.S., Cao, L., Wang, S.: T2fela: Type-2 fuzzy extreme learning algorithm for fast training of interval type-2 tsk fuzzy logic system. Neural Networks and Learning Systems, IEEE Transactions on 25(4) (April 2014) 664–676 24. Hassan, S., Khosravi, A., Jaafar, J.: Training of interval type-2 fuzzy logic system using extreme learning machine for load forecasting. In: Proceedings of the 9th International Conference on Ubiquitous Information Management and Communication. IMCOM ’15 (2015) 87–91 25. Castro, J., Mantas, C., Benitez, J.: Interpretation of artificial neural networks by means of fuzzy rules. IEEE Transactions on Neural Networks 13(1) (2002) 101–117 26. Wu, D., Mendel, J.: Enhanced karnik–mendel algorithms. Fuzzy Systems, IEEE Transactions on 17(4) (Aug 2009) 923–934
