Neural Network Training by Hybrid Accelerated Cuckoo Particle Swarm Optimization Algorithm Nazri Mohd Nawi1, Abdullah khan1, M.Z. Rehman1, Maslina Abdul Aziz2, Tutut Herawan3,4, and Jemal H. Abawajy2 1
Faculty of Computer Science and Information Technology Universiti Tun Hussein Onn Malaysia 86400 Parit Raja, Johor, Malaysia 2 School of Information Technology, Deakin University, Waurn Ponds, Geelong, VIC, Australia 3 University of Malaya, 50603 Kuala Lumpur, Malaysia 4 AMCS Research Center, Yogyakarta, Indonesia
[email protected],
[email protected],
[email protected],
[email protected], {mabdula,jemal.abawajy}@deakin.edu.au
Abstract. Metaheuristic algorithm is one of the most popular methods in solving many optimization problems. This paper presents a new hybrid approach comprising of two natures inspired metaheuristic algorithms i.e. Cuckoo Search (CS) and Accelerated Particle Swarm Optimization (APSO) for training Artificial Neural Networks (ANN). In order to increase the probability of the egg’s survival, the cuckoo bird migrates by traversing more search space. It can successfully search better solutions by performing levy flight with APSO. In the proposed Hybrid Accelerated Cuckoo Particle Swarm Optimization (HACPSO) algorithm, the communication ability for the cuckoo birds have been provided by APSO, thus making cuckoo bird capable of searching for the best nest with better solution. Experimental results are carried-out on benchmarked datasets, and the performance of the proposed hybrid algorithm is compared with Artificial Bee Colony (ABC) and similar hybrid variants. The results show that the proposed HACPSO algorithm performs better than other algorithms in terms of convergence and accuracy. Keywords: Metaheuristic algorithm, Neural network, Cuckoo search, Particle Swarm Optimization, Optimization.
1
Introduction
Artificial Neural Network (ANN) is one of the widely used techniques for dataset classification in data mining. It is also known as a powerful prediction tool and has been applied in a wide variety of areas such as engineering, finance, military, telecommunication and so on. In general, the ANN presents and emulates the actual biological nervous systems with layers of interconnected individual artificial neurons. The ANN are structured by their number of layers (architecture), types of topology L.C. Kiong et al. (Eds.): ICONIP 2014, Part II, LNCS 8835, pp. 237–244, 2014. © Springer International Publishing Switzerland 2014
238
N.M. Nawi et al.
and learning process. One of the most popular supervised network classifiers is the Multilayer Perceptron (MLP) or the feed-forward ANN [1]. The MLP consists of three layers architecture. The first layer receives input, the second layer is the hidden layer, and the third layer produces output. Each of these layers contains nodes. Every node in these layers is connected to every adjacent layer [2]. There are different types of datasets: the training (supervised learning) and testing data set. The trained data sets results provide the flexibility to the learning process and can be used to make projections by using the pre-set target value. The MLP utilizes the Error Back Propagation (EBP) learning algorithm that is mainly used for solving classification tasks by sorting patterns in the datasets. The Back Propagation Neural Network (BPNN) [3-5] learning algorithm uses the gradient-descent technique in adjusting weights and biases for training an ANN in many domains. Along various advantages of ANN there are some downsides. Unfortunately, the EBP learning algorithm is not efficient enough in handling such large learning problems. The classifiers tend to make wrong selection of the characteristics for a specific task. Therefore, the whole network will be affected. It is also not easy to find the appropriate ANN architectures. Also, as ANN generate multifaceted error-planes with multiple local minimum, the BPNN fell prey to local minima instead of converging to global minimum [6]. A number of research studies have attempted to overcome these problems by introducing different techniques to analyze the performance of the standard steepest descent algorithm. These methods include the gradient descent with adaptive learning rate, gradient descent with momentum, gradient descent with momentum and adaptive learning rate, the resilient algorithm and standard steepest descent [6-8]. However, one limitation of gradient-descent technique is that it requires a differentiable neuron transfer function. Also, as neural networks generate complex error surfaces with multiple local minimum, the BPNN fall into local minima instead of a global minimum [7, 9-10]. In this paper, we solved the above optimization problems by improving the accuracy and decrease number of training errors with a fast convergence rate by using hybrid metaheuristic algorithms. Among the various metaheuristic populations based search algorithms for training BPNN are the Artificial Bee Colony (ABC) [1], Genetic Algorithm (GA) [11], Particle Swarm Optimization-Back Propagation (PSO-BP) [12], Ant Colony Optimization (ACO) [13] Cuckoo Search (CS) Algorithm [14-21], Bat Algorithm [22] and so on. This paper proposes an improved Accelerated Particle Cuckoo Swarm Optimization (HACPSO) algorithm. In the proposed HACPSO algorithm the communication ability for the cuckoo bird has been added. The Accelerated Particle Swarm Optimization (APSO) algorithm searches a better place with the best nest and share the information with Cuckoo bird. The convergence behavior and performance of the proposed HACPSO on classification datasets is analyzed. The results are compared with artificial bee colony using BPNN algorithm, and similar hybrid variants. The remaining paper is organized as follows: Section 2 describes the proposed method. Section 3 describes result and discussion. Finally, the paper is concluded in the Section 4.
Neural Network Training by HACPSO Algorithm
2
239
The Proposed HACPSO Algorithm
In this section, we examine the detail of the proposed hybrid algorithm. As mentioned above, the nature of cuckoo birds is that instead building their own nests, they lay their eggs in the nest of the other host bird. If the host bird detects an unknown egg, it will either throw the egg away or simply abandon its nest and build a new nest elsewhere. Thus, the cuckoo birds are always looking for a better place in order to reduce the chance of their eggs to be discoursed. In this proposed hybrid algorithm, the CS initializes the population for the nest, and randomly selects the best nest via levy flight. In addition, the communication ability for the cuckoo birds has been added, where the APSO search better place that has the best nest, and share the information with cuckoo search. Then cuckoo search selected the best nest among all via levy flight using the Equation (1) as follow , where
(1)
is the velocity vector generated from Equation (2) as follow ,
(2)
where and be the position vector for the particle , is the learning parameter or accelerating constant, is random vector which is draw from N(0, 1). The pseudo code of the proposed algorithm is given as follow. Generate initial population of N host nest for i= 1,...,n While (fmin
