Multiobjective Differential Evolutionary Neural Network for Multi Class

Multiobjective Differential Evolutionary Neural Network for Multi Class Pattern Classification Ashraf Osman Ibrahim1, 2, Siti Mariyam Shamsuddin1, and Sultan Noman Qasem3 1

Soft Computing Research Group (SCRG), Faculty of Computing, Universiti Teknologi Malaysia (UTM), 81310, Skudai, Johor, Malaysia. [email protected]

2

Faculty of computer and technology, Alzaiem Alazhari University, Khartoum, Sudan 3 College of Computer and Information Sciences, Al-Imam Muhammad Ibn Saud Islamic University, Riyadh, Saudi Arabia Abstract. In this paper, a Differential Evolution (DE) algorithm for solving multiobjective optimization problems to solve the problem of tuning Artificial Neural Network (ANN) parameters is presented. The multiobjective evolutionary used in this study is a Differential Evolution algorithm while ANN used is Three-Term Backpropagation network (TBP). The proposed algorithm, named (MODETBP) utilizes the advantages of multi objective differential evolution to design the network architecture in order to find an appropriate number of hidden nodes in the hidden layer along with the network error rate. For performance evaluation, indicators, such as accuracy, sensitivity, specificity and 10-fold cross validation are used to evaluate the outcome of the proposed method. The results show that our proposed method is viable in multi class pattern classification problems when compared with TBP Network Based on Elitist Multiobjective Genetic Algorithm (MOGATBP) and some other methods found in literature. In addition, the empirical analysis of the numerical results shows the efficiency of the proposed algorithm. Keywords: Multiobjective Differential Evolution, Backpropagation network, Pareto optimal, classification.

1

Three-Term

Introduction

Over the past few decades, there was a significant increase in using soft computing approaches. Artificial Neural Network (ANN) has become the substrate of soft computing methods, successfully used for solving different problems. Due to the importance of using ANNs in many applications, there are some different methods in the previous studies that focused on solving the problems of ANNs optimization, the training and structure of the network [1, 2]. Recently, there has been a remarkable increase in the use of Evolutionary algorithms (EAs) for solving optimization problems. The design of ANNs is considered one of the most important problems that need to be solved using this kind of algorithms. The earlier approaches tackled the single objective optimization problems in some of the previous works, PSO [3], GA[2] and DE[4] were considered for optimizing ANNs. These optimization techniques optimize only one factor, such as, hidden nodes or connection weights or optimizing training error rate. Though in ANNs optimization, there is more than one parameter that need to be optimized. Therefore, multiobjective optimization problems are preferred because of their ability to optimize more than one objective simultaneously. Evolutionary Algorithms (EAs) are good candidates for Multi objective optimization problems (MOOPs). This is because of their abilities to search for multiple Pareto optimal solutions and they perform better in global search space. Multiobjective evolutionary algorithms (MOEAs) research area has become one of the hottest areas in the field of evolutionary computation [5]. They are suitable to

produce and design the appropriate and accurate ANNs with the optimization of two conflicting objectives, namely: minimization of ANNs structure complexity and maximization of network capacity. Hence, the MOEAs have been successfully applied recently to optimize both the structure, connection weights and network training simultaneously [6-8]. These methods have advantages over the conventional backpropagation (BP) method because of their low computational requirement when searching in a large solution space due to the fact that EAs are population based algorithms which allow for simultaneous exploration of different parts in the Pareto optimal set. Thus, Pareto optimal solutions are used to evolve artificial neural networks (ANNs) which are optimal both with respect to classification accuracy and network architectural complexity [9, 10]. The MOEAs have been receiving increased attention among researchers to solve this kind of problem. Recently, the trend to optimize and design ANNs architecture by using MOEAs is gaining popularity among researchers. It still attract significant interest of the evolving ANNs as optimization problems. In this paper, we proposed to use a MODE algorithm to optimize and design appropriate and accurate ANNs architecture, that will be able to find an appropriate number of hidden nodes in the hidden layer along with error rates of the network. The proposed method benefited from the concept of the Pareto optimal solutions, using Multiobjective Differential Evolution (MODE) for designing a good structure of ANNs for the multi class classification task. The other aspects which the paper delves on includes: Related Works given in Section 2. In Section 3, Material and methods are presented. The proposed method is presented in Section 4. In Section 5, Experimental Study, result and discussion are provided. Finally Section 6 presents the conclusions.

2

Related Works

There are various algorithms that have been proposed for ANNs with MOEAs. However, due to a large diversity of applications, various data and different purposes of optimization algorithms, it is so difficult to find a specific algorithm that can serve the needs of all, at the same time. Zhou, A., et al. [5] presented a survey paper on the development of Multiobjective evolutionary algorithms. The survey covers all areas that apply different types of MOEA to real world problems, such as, data mining, communications, bioinformatics, control systems and robotics, manufacturing, engineering, pattern recognition, image processing, fuzzy systems and Artificial neural networks. Furthermore, they presented some issues for the future work. In addition, they highlighted that the MOEAs are very promising for multiobjective optimization problems because of their capability to approximate a set of Pareto optimal solutions in a single run. Another study has proposed a solution for the regularization problems of complexity in the networks, based on multi-objective optimization algorithm to optimize the structure and minimize the number of connections in ANN [11]. Several studies have been in focusing on the extension of Differential Evolution (DE) to solve multi-objective optimization problems in continuous domains. One of the first papers to explore the potential of DE for solving multi objective optimization problems (MOPs) were written by [12, 13]. In both algorithms DE is employed to create new solutions and only the non-dominated solutions are retained as the basis for the next generation. Both methods used Pareto Differential Evolution (PDE) concept as the main objective of the study. Similar study introduced by [14] used the Pareto optimal approaches to train a multilayer perceptron network, they achieved Pareto optimal evolutionary neural network as parallel evolution of a population and considered multiple error measures

as objectives. H.A. Abbass in [15] proposed multi-objective method that includes DE algorithm to train the artificial neural network and to optimize the number of hidden nodes and connection weights simultaneously. His study, benefited from the concept of multiobjective optimization and multiobjective evolutionary algorithms to evolve ANNs. Another study by H.A. Abbass [16] that introduced multiobjective Pareto DE combined with local search algorithm for ANN to enhance the performance of algorithm. It minimizes the training error and the number of hidden nodes. This algorithm showed fast training and better generalization than traditional ANN. Similarly, in another study, the same author introduces Pareto differential evolution algorithm (PDE) hybrid with local search for evolutionary ANN to diagnose breast cancer [17] and this study has obtained promising results as well. The work by G. P. Liu and V. Kadirkamanathan in [18] studied the benefits of multi-objective optimization for selection and identifying nonlinear systems while optimizing the size of neural networks. Instead of a single objective, three objectives were considered for performance indices or error functions. In an attempts to optimize ANNs by multi-objective evolutionary algorithms, [8] also proposed Pareto-based multi-objective Differential Evolution algorithm that adapted to design Radial Basis Function Networks (RBFNs) hybrid with a local search for solving binary and multi class classification problems, to achieve simultaneous generalization and classification accuracy of the network. More recently, M. Cruz-Ramírez et al [19] introduced automatically designed artificial neural network and learning, the structure and weights of the ANN, for multi classification tasks in predictive microbiology using Hybrid Pareto Differential Evolution Neural Network named (HPDENN). They achieved two objectives, high classification level and highest classification level for each class.

3

Material and Methods

3.1

Differential Evolution (DE)

R. Storn and K. Price [20] introduced differential evolution (DE) algorithm for single strategy and R. Storn in [21] extended the study later to parallel computation, constraint satisfaction, constrained optimization and design centering problems. DE algorithm is an efficient population based metaheuristic search algorithm for optimization method. Recently, DE has been applied successfully in various areas and for complex non-linear problems. Due to its advantages such as simplicity, compact structure, minimal control parameters, powerful search capability and high convergence characteristics [21-24]. It has attracted a lot of researchers for its use in global optimization. The power of DE algorithm has attracted much attention as MOEA, it has been demonstrated over the years and successfully use to solve many types of multi objective optimization problems with varying degrees of complexity and in various fields of application [25-27]. 3.2

Multiobjective Differential Evolution (MODE)

A multiobjective evolutionary algorithm (MOEA), also known as multiobjective optimization algorithm (MOOA), is the process of simultaneously optimizing two or more conflicting objectives subject to certain constraints; they are a population based search. Hence, in a single run, it can get many of Pareto optimal solutions and that are attractive to this kind of algorithms. Recently, the original differential evolution algorithm DE has to be modified in order to apply the DE algorithm for

multiobjective optimization problems MOPs. Presently, several previous studies have presented the prospective achievements of DE that can be an attractive alternative to extending DE for solving multi objective numerical optimization problems [12, 26, 28]. 3.3

Three-Term Backpropagation algorithm

The Three-Term Backpropagation (TBP) proposed by [29], utilizes three parameters. Beside the learning rate and a momentum factor there is third parameter which is called proportional factor (PF), this parameter was introduced in order to speed up the weight adjusting process or to increase the BP learning speed. Generally, the TBP network employs the standard architecture and procedure of the standard backpropagation algorithm that contains input layer, hidden layer and the output layer. In all layers there are number of neurons that are connected together.

4

The Proposed MODE

In recent years, the used of evolutionary computational techniques have proven themselves useful in the area of evolving artificial neural networks. For the problem of optimizing ANN, several objective functions can be considered. For instance, network accuracy, network architecture and connection weights. This section presents a MOEA evolving Artificial Neural Network (ANN). The proposed MOEA is based on the Multiobjective Differential Evolution (MODE) algorithm for Three-Term Backpropagation network (TBP). The Pareto Multi-objective Evolutionary TBP network algorithm used in this work optimizes error rates and architectures of the network simultaneously. This proposed method allows us to design TBP network, in terms of choosing the number of hidden nodes and generalizations of the network, to obtain simple and accurate TBP network. The proposed method is implemented for multi class pattern classification problems. In this paper, MODETBP network algorithm has been proposed to determine the best performance and the corresponding architecture of the TBP network. To assist TBP network design, differential evolution (DE) and multiobjective optimization (MOO) are combined to carry out fitness evaluation and to enhance the performance capability of approximating a set of Pareto optimal solutions in a single run. The optimization goal is to minimize the objective function. In this work, we have taken the performance of the network (Accuracy) based on the Mean Square Error (MSE) on the training set as a first objective function by minimizing the network N error rate, E  1 (t j  o j )2 , where N is the number of samples. The second  N j 1 objective function, we have taken is to minimize the complexity of the network by H

optimizing the number of the nodes in the hidden layer of TBP network, H   ,  h h 1

where,  h belongs to vector ρ is the dimension of the maximum number of hidden nodes H of the network. In this objective function, the maximum number of hidden nodes H of the network as vector ρ is the binary value used to refer to the hidden node if it exists in the network or not.

4.1

Parameter setting

The parameter setting is very important for the algorithms. This is as a result of the significant impact they have on the optimum performance. Therefore, it is required to choose the parameters carefully to find optimal values for the parameters. In this paper, the parameters of the proposed MODE used for training the TBP

network for all datasets are the same. Depending on the previous studies in literature that used, to applied DE algorithm, we chose the parameters as follow: the population size is 100, probability of crossover ( CR ) used is 0.9 and the probability of mutation ( F ) is 0.5, while the maximum number of iterations is 1000. The fitness values are the hidden nodes and network training error or performance of the network. The training set is used to train the TBP network in order to obtain the Pareto optimal solutions, while the testing set is used to test the generalization performance of the Pareto TBP network.

5

Experimental results

In this section we present the experimental results of the study on MODE for TBP network. The proposed method (MODETBP) is evaluated by using 10-fold cross validation technique. The results were obtained for all datasets is a Pareto optimal solution to improve the generalization on unseen data. In the experimental design, we considered three multi class datasets. Multi class pattern classification is a major classification problem with more than two different classes which accurately maps an input feature space to an output space as output, and the number of features as inputs. Table 1 shows the dataset that was used in this study along with the following details of the dataset: number of features, number of classes and the total number of patterns. With regard to preprocessing stage, all the dataset values are normalized in the range of [0, 1]. Table 1. Summary of data sets used in the experiments. Dataset

Number of features

Number of classes

Number of patterns

Iris

4

3

150

Wine

13

3

178

Yeast

8

10

1484

For the measurements of the proposed method we used statistical measures which are Sensitivity to identify the correct positive samples, Specificity to predict the correct negative samples, accuracy to produce the level of accurate results and AUC is the area under the receiver operating characteristic curve (ROC). Since, the AUC is a portion of the area of the unit square, its value is between 0.0 and 1.0. The experimental result presented in Table 2 shows the values of the mean and standard deviation (STD) in generalization for training and testing error rates for all runs of the experiments performed. In addition, we can easily verify that all datasets on the average as shown in Table 2, MODETBP gives promising results in both training and testing sets. Moreover, it shows that the MODETBP obtained smallest error compared with our previous method Three-Term Backpropagation network based on the Elitist Multiobjective Genetic Algorithm (MOGATBP) [30], using same dataset and objective functions. Table 2. The training and testing error rates MODETBP Training Error

Dataset Iris

Testing Error

MOGATBP Training Error

Testing Error

Mean 0.1070

0.1036

0.1645

0.1654

0.0172

0.0298

0.0239

0.0224

STD

MODETBP Training Error

Dataset Wine

Testing Error

MOGATBP Training Error

Testing Error

Mean 0.1211

0.1227

0.1686

0.1682

0.0296

0.0257

0.0394

0.0433

0.0757

0.077

0.0816

0.0816

0.0065

0.0063

0.0088

0.0088

STD Yeast

Mean STD

Tables 3 present the results of the proposed method (MODETBP) and MOGATBP based on two objectives on iris, wine and yeast dataset. The Mean and SD indicate the average value and standard deviation respectively. The result of these algorithms is Pareto optimal solutions to improve the generalization on unseen data. All the results demonstrate that the MODETBP has the capability to perform better to classify the accuracy for all datasets against the MOGATBP algorithm. Additionally, Table 3 shows the statistical results for sensitivity (Sens), specificity (Spec) and Accuracy. It also gives detailed information on the MODETBP compared with MOGATBP in training and testing data. Equations 1, 2 and 3 show the calculation of those statistical measures as follows: Sensitivity 

TP TP  FN

(1)

Specificity 

TN TN  FP

(2)

Accuracy 

TP  TN TP  TN  FP  FN

(3)

Where, TP is true positive, FP is false positive, TN is true negative and FN is false negative. The experimental result presented in Table 3 show, among other things all the datasets that were used in this study, it can be observed that MODETBP achieved better accuracy than MOGATBP. Furthermore, we can clearly notice that the result of the MODETBP obtained an accurate result of 97.02 %, 93.67 % and 90.33% for iris, wine and yeast dataset respectively. Table 3. Accuracy, Sensitivity and Specificity Methods

MODETBP

Data

Dataset

Measure

Training

MOGATBP Testing

Training

Testing

Sens%

Spec%

Accuracy%

Sens%

Spec%

Accuracy%

Sens%

Spec%

Accuracy%

Sens%

Spec%

Accuracy%

Mean

96.81

97.76

97.59

93.33

96.67

97.02

34.89

99.41

78.17

34.00

99.33

77.56

STD

0.93

1.14

2.35

6.29

3.14

2.96

27.07

1.08

8.19

24.84

2.11

7.73

Mean

94.78

97.50

95.46

93.13

95.25

93.67

23.32

98.66

73.35

99.12

74.29

74.29

STD

5.03

0.83

2.34

7.51

3.80

5.17

35.71

2.39

10.42

2.02

11.95

11.95

Mean

2.98

97.89

90.23

3.90

98.19

90.33

0.00

100

90.00

0.00

100

90.01

Iris

Wine

Yeast

Methods

MODETBP

Data

Dataset

Measure

Training

MOGATBP Testing

Training

Testing

Sens%

Spec%

Accuracy%

Sens%

Spec%

Accuracy%

Sens%

Spec%

Accuracy%

Sens%

Spec%

Accuracy%

3.08

1.12

0.37

3.40

0.98

0.35

0.00

0.00

0.00

0.00

0.00

0.00

STD

Similarly, for the sensitivity, MODETBP has achieved 93.33% for iris, 93.13% for wine and 3.90% for yeast dataset. The sensitivity of the yeast data set is very difficult, due to their unbalanced data. Furthermore, besides accuracy and sensitivity, Table 3 shows the specificity for all datasets. We can note that the specificity rate achieved is as follows: iris data has achieved 96.67%, wine data has achieved 95.25% and 98.19% obtained by yeast dataset. In terms of mean and standard deviation, Table 3 also shows the MODETBP has produced small standard deviation for the test accuracy. It is clearly seen that From Table 4 and Figure 1, an analysis of the accuracy and AUC compared to MEPDENf1f2 [8], MEPDENf1-f3 [8] and MOGATBP, we found that the MODETBP has the highest classification accuracy and AUC as well followed by MEPDENf1-f3 algorithm in all classification accuracy results. Both MEPDENf1f2 and MEPDENf1-f3 we compared with our method, they are a memetic multiobjective evolutionary algorithm. They use a local search method to improve the solution and achieve better accuracy of the final result. Thus it enables these methods to improve their algorithm to achieve good results. While the results of MODETBP used only multiobjective DE algorithm without the local search algorithm. In spite of this, MODETBP performs better accuracy than all algorithms. Table 4. Comparison of the accuracy and AUC of the proposed method and other methods MODETBP

MOGATBP

MEPDENf1f2

MEPDENf1-f3

Dataset Accuracy

AUC

Accuracy

AUC

Accuracy

AUC

Accuracy

AUC

Iris

97.02

0.95

77.56

0.667

86.00

0.823

96.89

0.946

Wine

93.67

0.942

74.29

0.867

77.11

0.694

90.04

0.856

Yeast

90.33

0.510

90.01

0.500

90.00

0.500

90.16

0.506

Fig. 1. Comparison of Accuracy of the MODETBP and other methods

It is clearly seen that from Table 5, the complexity of MODETBP achieved better results than all methods in all dataset when using multi objective DE algorithms. The MODETBP design smallest network architecture against all methods, except MOGATBP that used GA technique, obtained better architecture than MODETBP in both of iris and yeast, while the MODETBP performs better architecture when used with wine dataset. The number of the hidden nodes is considered one of the objectives to optimize in this study. Table 5. Comparison of the hidden nodes of the proposed method and other methods MODETBP

MOGATBP

MEPDENf1f2

MEPDENf1-f3

Dataset Iris

3.8

3.6

3.9

3.9

Wine

3.8

4.6

3.8

3.8

Yeast

4.0

3.5

6.5

5.7

From Tables 4 and 5, we can conclude that MODETBP has achieved the optimization of the hidden nodes along with error rates simultaneously. It also performs well in accuracy and optimizing its generalization ability better than all methods.

6

Conclusions

This paper has presented a multiobjective evolutionary algorithm for optimizing TBP network, to achieve optimization of two objectives, which are accuracy of the network along with the complexity of the TBP network simultaneously. The proposed MODETBP algorithm has been evaluated using three types of performance evaluation indicators to assess the effect of MODETBP. In addition, MODETBP is used to develop generalization and classification accuracy for the TBP network. In this paper, an attempt was made to improve the generalization of the training and unseen data along with solving multi class pattern classification problem. The experimental results illustrate that MODETBP was able to obtain a TBP network with better classification accuracy and simpler structure compared to others. Acknowledgements. The authors would like to thank Soft Computing Research Group (SCRG) (UTM) for the continuous support and motivation. This work is supported by a Fundamental Research Grant Scheme (FRGS - 4F347).

References 1. 2. 3. 4. 5. 6.

N.L. Mineu, T.B. Ludermir, L.M. Almeida, Topology optimization for artificial neural networks using differential evolution, Neural Networks (IJCNN), The 2010 International Joint Conference on, (IEEE2010), pp. 1-7. S. Ding, C. Su, J. Yu, An optimizing BP neural network algorithm based on genetic algorithm, Artificial Intelligence Review, 36 (2011) 153-162. C. Zhang, H. Shao, Y. Li, Particle swarm optimisation for evolving artificial neural network, Systems, Man, and Cybernetics, 2000 IEEE International Conference on, (IEEE2000), pp. 2487-2490. J. Ilonen, J.-K. Kamarainen, J. Lampinen, Differential evolution training algorithm for feed-forward neural networks, Neural Processing Letters, 17 (2003) 93-105. A. Zhou, B.-Y. Qu, H. Li, S.-Z. Zhao, P.N. Suganthan, Q. Zhang, Multiobjective evolutionary algorithms: A survey of the state of the art, Swarm and Evolutionary Computation, 1 (2011) 32-49. C.-K. Goh, E.-J. Teoh, K.C. Tan, Hybrid multiobjective evolutionary design for artificial neural networks, Neural Networks, IEEE Transactions on, 19 (2008) 15311548.

7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30.

M. Delgado, M.P. Cuéllar, M.C. Pegalajar, Multiobjective hybrid optimization and training of recurrent neural networks, Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 38 (2008) 381-403. S.N. Qasem, S.M. Shamsuddin, Memetic elitist pareto differential evolution algorithm based radial basis function networks for classification problems, Applied Soft Computing, 11 (2011) 5565-5581. S.N. Qasem, S.M. Shamsuddin, Radial basis function network based on time variant multi-objective particle swarm optimization for medical diseases diagnosis, Applied Soft Computing, 11 (2011) 1427-1438. J.C. Fernandez Caballero, F.J. Martínez, C. Hervás, P.A. Gutiérrez, Sensitivity versus accuracy in multiclass problems using memetic Pareto evolutionary neural networks, Neural Networks, IEEE Transactions on, 21 (2010) 750-770. Y. Jin, T. Okabe, B. Sendhoff, Neural network regularization and ensembling using multi-objective evolutionary algorithms, Evolutionary Computation, 2004. CEC2004. Congress on, (IEEE2004), pp. 1-8. H.A. Abbass, R. Sarker, C. Newton, PDE: a Pareto-frontier differential evolution approach for multi-objective optimization problems, Evolutionary Computation, 2001. Proceedings of the 2001 Congress on, (IEEE2001), pp. 971-978. H.A. Abbass, R. Sarker, The Pareto differential evolution algorithm, International Journal on Artificial Intelligence Tools, 11 (2002) 531-552. J.E. Fieldsend, S. Singh, Pareto evolutionary neural networks, Neural Networks, IEEE Transactions on, 16 (2005) 338-354. H.A. Abbass, R. Sarker, Simultaneous evolution of architectures and connection weights in ANNs, Proceedings of Artificial Neural Networks and Expert System Conference2001), pp. 16-21. H.A. Abbass, A memetic pareto evolutionary approach to artificial neural networks, AI 2001: Advances in Artificial Intelligence, (Springer, 2001), pp. 1-12. H.A. Abbass, An evolutionary artificial neural networks approach for breast cancer diagnosis, Artificial Intelligence in Medicine, 25 (2002) 265-281. G. Liu, V. Kadirkamanathan, Multiobjective criteria for neural network structure selection and identification of nonlinear systems using genetic algorithms, IEE Proceedings-Control Theory and Applications, 146 (1999) 373-382. M. Cruz-Ramírez, C. Hervás-Martínez, P.A. Gutiérrez, M. Pérez-Ortiz, J. Briceño, M. de la Mata, Memetic Pareto differential evolutionary neural network used to solve an unbalanced liver transplantation problem, Soft Computing, 17 (2013) 275-284. R. Storn, K. Price, Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces, Journal of global optimization, 11 (1997) 341359. R. Storn, System design by constraint adaptation and differential evolution, Evolutionary Computation, IEEE Transactions on, 3 (1999) 22-34. J. Brest, S. Greiner, B. Boskovic, M. Mernik, V. Zumer, Self-adapting control parameters in differential evolution: A comparative study on numerical benchmark problems, Evolutionary Computation, IEEE Transactions on, 10 (2006) 646-657. S. Rahnamayan, H.R. Tizhoosh, M.M. Salama, Opposition-based differential evolution, Evolutionary Computation, IEEE Transactions on, 12 (2008) 64-79. J.-T. Tsai, W.-H. Ho, J.-H. Chou, C.-Y. Guo, Optimal approximation of linear systems using Taguchi-sliding-based differential evolution algorithm, Applied Soft Computing, 11 (2011) 2007-2016. B. Babu, M.M.L. Jehan, Differential evolution for multi-objective optimization, Evolutionary Computation, 2003. CEC'03. The 2003 Congress on, (IEEE2003), pp. 2696-2703. M. Ali, P. Siarry, M. Pant, An efficient differential evolution based algorithm for solving multi-objective optimization problems, European Journal of Operational Research, 217 (2012) 404-416. B. Alatas, E. Akin, A. Karci, MODENAR: Multi-objective differential evolution algorithm for mining numeric association rules, Applied Soft Computing, 8 (2008) 646-656. W. Gong, Z. Cai, A multiobjective differential evolution algorithm for constrained optimization, Evolutionary Computation, 2008. CEC 2008.(IEEE World Congress on Computational Intelligence). IEEE Congress on, (IEEE2008), pp. 181-188. Y. Zweiri, J. Whidborne, L. Seneviratne, A three-term backpropagation algorithm, Neurocomputing, 50 (2003) 305-318. A.O. Ibrahim, S.M. Shamsuddin, N.B. Ahmad, S.N. Qasem, Three-Term Backpropagation Network Based On Elitist Multiobjective Genetic Algorithm for Medical Diseases Diagnosis Classification, Life Science Journal, 10 (2013).