2014 2014 International International Conference Conference on on Information, Information, Communication Communication Technology Technology and and System System
A New Operator in Gravitational Search Algorithm Based on The Law of Momentum R. V. Hari Ginardi1), Abidatul Izzah2) Department of Informatics Institut Teknologi Sepuluh Nopember Surabaya, Indonesia 1)
[email protected] , 2)
[email protected] Abstract—Gravitational Search Algorithm (GSA) is a new heuristic algorithm which derives its inspiration from natural phenomenon. As a heuristic algorithm, GSA has a good performance in global searching. However, if a premature convergence happens, GSA will stop and lose its ability to search. Therefore, a new operator that is inspired by conservation of linear momentum is added to improve GSA’s performance. In this paper, we propose this new operator called momentum operator which aims to solve the weakness of GSA. It will be applied in a non-linear optimization and data classification. The system will be simulated in MATLAB R2009a. The result showed that this operator was successful to solve data classification and minimization problem faster and more precisely in a high dimension. Keywords—Gravitational Momentum
I.
Search
Algorithm,
Heuristic,
INTRODUCTION
Heuristic is a strategy, which is widely used in a searching method for finding a new solution approximately based on an existing solution. In recent years, heuristic has been developed rapidly. However, it is not guaranteed that the solution is the optimum one [1]. Heuristic can also be applied to increase accuracy in data classification. For example, Pham (2009) proposed a meta-heuristic called Homogeneity Based Algorithm (HBA) to optimize accuracy in medical datasets classification. By optimizing problem in terms of a falsepositive, a false-negative, and undetected rates along with their penalty costs, HBA can enhance accuracy when it is combined with a classic algorithm [2]. Pham (2011) also proposed new a meta-heuristic approach called Convexity Based Algorithm (CBA) in [3]. CBA was designed for approach balancing between the data fitting and generalization behaviors of models. It also used penalty cost to define the total misclassification. Unlike the HBA, CBA uses partition training data step and genetic approach. The result showed that heuristic strategy made the effectiveness better. For heuristic techniques, Rashedi (2009) discovered a new heuristic technique that inspired by natural phenomenon called Gravitational Search Algorithm (GSA) [4]. GSA is based on the gravity law and mass interaction. The solutions, called agents, are represented by masses which interact with each other based on the Newtonian gravity and
978-1-4799-6858-9/14/$31.00 © 2014 IEEE
the motion law [4]. In general, GSA is used to solve continuous optimization problems because it was originally designed for it. Because of its good ability, GSA can also be used in combinatorial problem such as traveling salesman problem (TSP) [5]. The result showed that GSA was successful to approach the best solution for TSP. GSA was also effective in multimodal optimization problem [6]. For this case, GSA was improved by partitioning swarm of masses into sub-swarms and preserving them by applying a k-nearest neighbors (K-NN) strategy, an elitism strategy, and a modified active gravitational mass formulation. It is then called Niche GSA (NGSA). NGSA can solve multiple optima constrained and unconstrained problem. In fact, not only in optimization, GSA can also be applied in data mining, such as clustering and classification. In [7], GSA is applied for clustering with no prior information about the number of clusters. Automatic Clustering using GSA (ACGSA) has been made to determine the number of clusters well. In [8], GSA was also used as a learning method in artificial neural network (ANN) to classify iris dataset, glass dataset, and XOR. It was also used to classify medical data based on urinalysis to screen urinary treat infection (UTI) test [9]. Many researches mentioned that GSA has a good ability in searching. But, if too many agents on the boundary and especially when there are existence of some local optima on the boundary, the algorithm will lose its population diversity and stay in the local optima [10]. However, if a premature convergence happens, there will not be any recovery. Finally, GSA will lose its ability to explore and then becomes inactive [11]. A premature convergence occurs when some high rated but a solution quickly attain to dominate the swarm. In this situation, the algorithm cannot find a better solution [12]. In [11], a new operator, called “disruption”, based on astrophysics concept was proposed to address this weakness. The result showed that it was successful to avoid weakness of GSA in a premature convergence. Based on the issue(s), in this paper, we propose a new operator. This operator aims to avoid premature convergence by more exploring the solution and protecting the diversity. A new operator was inspired from the law of conservation of momentum. It can save the diversity because it made two masses move away from each other. By applying this, the algorithm may reach an optimum solution.
ICTS 2014, Surabaya, Indonesia
105
II.
GRAVITATIONAL SEARCH ALGORITHM
Gravitation is the force of masses to accelerate toward each other. The law of Newtonian Gravity said: “Every particle in the universe attracts every other particle with a force that is directly proportional to the product of its masses and inversely proportional to the square of the distance between them”. An increase in the distance between two particles indicates decreasing of the gravity force between them. The objects attract each other by the gravity force. Finally, the force makes all of them move towards the ones with heavier masses. The objects transform information by the gravitational force and the object with a heavier mass has a heavier force. It is illustrated in Fig. 1. Force F14 is smaller than force F13 because mass M4 is smaller than mass M3 [4]. In GSA, a population of solution is represented by agents. In each iteration, agents will move towards to the best one. GSA begins with initialization of agent. Assumed we generate N agents randomly and each of them has n dimension. If represents position of agent i, we defined an agent by using Eq. 1. After evaluated those agents, we must update ) at time t, best solution ( ), gravitational constant ( ). In maximization case, the best and worst solution ( was the agent which has the maximum fitness. On the other side, in minimization case, the best was the agent which has is the value of the the minimum fitness. Consider that gravitational constant at the first cosmic quantum-interval of can be calculated by using Eq. 2. time ,
Next, we calculated mass to determine the heavier agent. was fitness of agent i, and were If the best and the worst agent at time t, then we can calculate ) based on Eq. 3 and Eq. 4. the mass of the agent i (
We used euclidean distance to represent the distance between agent i and agent j ( ). The total force acted on agent i from agent j in the d-th dimension was defined by Eq.5.
(5) If is an acceleration of agent i in the dimension d is a new velocity to move, so they can be and calculated as Eq. 6 and Eq. 7.
Finally, the agents will move toward the best one and update a new position according to Eq. 8. . III.
NEW OPERATOR BASED ON MOMENTUM LAW
A. Conservation Linear of Momentum Law Momentum depends on the mass and the velocity of object so that momentum is defined as the product of the mass and is the mass of object and is the velocity. Consider that velocity then the momentum can be calculated by using Eq. 9 [13]. Newton’s second law stated that the “motive force” exerted on a particle is equal to the change in its linear momentum. is motive force, is momentum at time Assumed that , and is momentum at time , then the conservation showed in Eq. 10. If two or more objects move, they may move farther, closer, or collide. When a collision occurred, the law of conservation momentum is applied. Fig. 2(a) showed two objects are moving toward each other with velocity and . Otherwise Fig. 2(b) showed when they move farther after a collision. Both of them will turn around with velocity and .
Fig. 1. Masses Interaction
106
The law of conservation momentum says: “Momentum of the two objects before and after collision are equal”. It can be written by Eq. 11. Furthermore, there are three types of collisions, which are perfectly elastic collision, partially elastic collision, and inelastic collision. When a perfectly elastic collision happens, the law of conservation of momentum and law of kinetic energy conservation are applied. It means if a
perfectly elastic collision occurred, both momentum and kinetic energy of the object before and after collision are equal. The kinetic energy is formulated on Eq. 12.
(a)
(ii) Collisions that occurred was perfectly elastic collision so the law of conservation of momentum and law of kinetic energy conservation were applied If perfectly elastic collision occurred, the law of conservation of momentum (Eq. 12) was applied. Besides that the law of kinetic energy conservation (Eq. 16) was also used. Furthermore, these two constrains were used to determine new velocity after collision.
(b) (iii) Mass of agent before and after collision were same Fig. 2. Two objects collide
In a partially elastic collision, Conservation of kinetic energy is not applied because there is a changing in masses of object and kinetic energy before and after the collision. It causes only the law of conservation of momentum is applied. In an inelastic collision, two objects collide and then both are fused and move together with the same velocity. B. Modelling a new operator The concept of the law of conservation of momentum and conservation of kinetic energy will be used in the computation to avoid the weaknesses of GSA. We proposed three assumptions which were used in modeling a new operator. They are: (i) Two agents would collide if they reach a certain distance Sarafrazi (2011) explained a condition how two agents can was the distance be said as convergence [11]. Considered between agent i and agent j and was the distance between agent i and best agent. It was said that two agents and was less than convergence if only comparison of a value. Thus, two agents will collide if they reach a certain distance or less than threshold. If is a certain value, it can be formulated in Eq.13.
is a value that determined a collision is A threshold occurred or not. An increase in the time should decrease the value of . So is a variable which become smaller when the masses were converged or the masses get closer to each other, was a and vice versa. Consider that was a constant and maximum iteration then at time can be formulated in Eq. 14 [11].
If the mass of an agent before and after collision were equal, then and . Finally, from the previous equation, we obtained a new velocity for agent i and agent j as the following Eq 17 and Eq 18.
Based on both the law of conservation of momentum and the conservation of kinetic energy, we aimed to maintain the diversity of agents. By using this operator, if these two agents got closer each other or approaching a certain distance, they will collide and spread. Thus, the possibility of premature convergence will be solved. The pseudocode of momentum operator can be seen in Algorithm 1. Algorithm1. Pseudocode of Momentum Operator Input: agents, , , , Output: agents Process: foreach agenti foreach agentj = Euclid(agenti , agentj) ) t = Euclid(agenti , = * (1-t/maksepoh) if Calculate using Equation (17) Calculate using Equation (18) end if end for end for foreach agenti Evaluate new agentj after momentum operator has been applied if MSE new agentj< MSE agenti Update agenti end for end for Update
We can put momentum operator after GSA standard ended. Fig 3 shows the flowchart of GSA with Momentum Operator (GSAM).
107
TABLE II. Function 1
Initialize agents Evaluation
2
Update G, best, worse 3
Calculate Massa and acceleration 4
Update velocity and position 5
Momentum Operator Terminate Solution Fig. 3. Flowchart of GSAM
IV.
EXPERIMENT RESULT
A. Non linear optimization problem We applied GSAM on several minimization functions and compared the results with Standard GSA. We set population size to 50, dimension to 30, and maximum iteration was 100 for solving the problem. Table 1 represents the benchmark functions which were used in our experiments [4]. TABLE I. No.
RESULT OF OPTIMIZATION PROBLEM
Average Std Dev Best Average Std Dev Best Average Std Dev Best Average Std Dev Best Average Std Dev Best
GSAM 1061.590 331.640 606.748 5.652 2.027 3.191 2435.532 923.115 1738.336 2722.317 451.790 2056.654 7.267 1.517 4.780
GSA 1462.115 342.287 831.909 7.040 2.010 4.227 3554.914 2303.310 1615.027 2878.303 587.515 2110.939 9.223 3.372 4.123
The new operator in GSA aims to avoid premature convergence. Therefore, we can explore the success of GSAM about how to obtain an optimal solution but not in premature iteration. We set GSA and GSAM in the same conditions, to compare their performance. Specially, when the complexity and the dimension of the problem at hand increase, GSA will be more sensitive to premature convergence. It can be said that whatever the complexity and the dimension, GSAM obtained a better solution. Based on the experiment result in Table II, it was concluded that momentum operator can fixed the GSA in finding a global optimum. Moreover, we need not only the global optimum but also avoiding the premature convergence. According to the figure Fig 4, we solved the unimodal case F1 and the result showed that GSAM tended both to finding the global optimum faster and having a higher convergence rate than GSA.
BENCHMARK FUNCTIONS Function
Interval
1
[-100,100]
2
[-10,10]
3
[-100,100]
4
[-500,500]
5
[-65.53,65.53]
Fig. 4. Comparison Result of F1 Problem as 100 and For updating the constant gravity, , we set α as 20. For determining a threshold, we set as 100. The test was repeated ten times for every function and we obtained the average of mean, standard deviation, and the best of fitness function. Table II illustrated that GSAM produced better results than GSA for every function. Results showed that GSAM gave a good convergence rate.
108
B. Data classification problem For a verification purpose, we applied GSAM for classifying wine dataset from UCI Machine Learning and XOR 3 bit parity problem. We set the population size to 20 and maximum iteration to 50. Both of GSA and GSAM were used
as a learning method in ANN [7][8]. It was used to find an optimum weight and bias of ANN to classify datasets. In a classification problem, GSA encoded agents to be a sequence of value which consisted of weight and bias. Agent dimension depended on the input, the hidden neuron, and the output. Agent was saved in a 3D-array [14]. It was considered that architecture of the neural network was 4– 2 – 1, so agent represented weight between first can be encoded as Fig. 5. neuron input and first neuron hidden on the first layer and represented bias between first neuron hidden and neuron output on the second layer.
approximately the same accuracy, that is in 75% or 80%. We also perform a statistical test to see the significant effect. By using confidence interval in 95%, we get p-value 0.093 in Wine classification and p-value was 0.030 in XOR classification. It is proved that GSAM statistically significant at high dimensions. Moreover, the additional operator in GSA also kept the diversity and maked convergence slowly. Because of it, GSAM needed more time to find the optimum agent. Table III ilustrated that GSAM needed approximately twice longer time than GSA. The new operator in GSA aims to avoid premature convergence. Therefore, we also compare the result is both a global optimum and obtained not in the premature iteration. Fig. 6 showed that GSAM can find a better and faster solution in wine classification problem.
Fig. 5. Agent Representation
For updating the constant gravity, , we still used
as 100
and α as 20 and for determining a threshold, we set as 100. The minimum square error (MSE) is used as an objective function and the sigmoid (as activation fuction) was used as the fitness function. In fact, several activation function can be used such as mean square error (MSE), root mean square error (RMSE), or sum square error (SSE) [14]. TABLE III. Data
RESULT OF CLASSIFICATION
Epoch 50
Wine
100 200 50
XOR 3 BIT
100 200
s 7 11 15 7 11 15 7 11 15 7 11 15 7 11 15 7 11 15
GSA Accuracy 0.661 0.695 0.685 0.731 0.625 0.735 0.656 0.707 0.596 0.750 0.750 0.750 0.750 0.750 0.825 0.750 0.775 0.800
Time 7.9975 10.456 11.808 13.126 12.993 16.845 8.3883 16.040 17.863 0.9101 1.2462 1.3833 0.8652 1.1257 1.5132 0.8897 1.1440 1.5702
GSAM Accuracy Time 0.644 13.723 0.819 15.504 0.714 13.979 0.747 11.320 0.652 18.853 0.764 31.025 0.763 14.758 0.702 21.764 0.702 17.863 0.800 2.0333 0.750 2.0208 0.775 2.6037 0.800 2.0521 0.875 2.6253 0.775 2.7653 0.800 1.7452 0.800 2.2323 0.800 1.5702
Table III showed the result of the classification experiment. As in the previous one, GSA had converged in a small epoch and escaped in a local optima solution. However, momentum operator effected in a high dimension dataset (e.g. wine dataset with 13 features) and it was not for small dimension dataset (e.g. XOR with 3 features). As seen on Table III, GSAM provided an accuracy as high as that of GSA. Based on the result, we can conclude that momentum operator showed a significant effect for a high dimension. For example in wine classification with 11 node hidden, the operator can increase the accuracy from 69.5% until 81.9%. But in contrast to previous, in XOR classification GSA and GSAM obtained
Fig. 6. Comparison Result of Wine Classification Problem
V.
CONCLUSIONS
GSA is a good algorithm in searching, but it is not effective enough if it escapes in a local optima and a premature convergence. The goal of this paper is to increase the exploration and exploitation abilities of GSA using the momentum operator. It is inspired by linearization of the law of conservation of momentum. The momentum operator explores the search space by recalculating the velocity and position of the agents when they collide. The result shows that this operator was successful to solve minimization problem and data classification faster and more precisely. However, a significant effect appears in problem with a high dimension. For the future work, the complexity in momentum operator that makes the algorithm needs more time must be addressed. REFERENCES [1] [2]
S.J. Russell and P. Norvig, “Artificial Intelligence a Modern Approach”, Prentice Hall, Upper Saddle River, New Jersy, 1995 H.N.Pham and E. Triantaphyllou,“An application of a new metaheuristic for optimizing the classificationaccuracy when analyzing some medical datasets”, Expert System with Applications, Vol. 36, pp. 92409249, 2009.
109
[3]
[4]
[5]
[6]
[7]
[8]
H.N. Pham and E. Triantaphyllou,“A meta-heuristic approach for improving the accuracy in someclassification algorithms”, Computer and Operation Research, Vol. 38, pp. 174-189, 2011. E. Rashedi, H. Nezamabadi-pour,and S. Saryazdi, “GSA: A Gravitational Search Algorithm”, Information Science, Vol. 179, pp. 2232–2248,2009 M.B.Dowlastshashi and H.N. Pour,“A discrete gravitational search algorithm for solving combinatorial optimization problems”, Information Sciences, Vol. 258, pp. 94-107, 2014. S. Yazdani, H.N. Pour, and S. Kamyab,“A gravitational search algorithm for multimodal optimization”, Swarm and Evolutionary Computation, Vol. 14, pp. 1-14, 2014. V. Kumar, J.,K. Chhabra,and D. Kumar,“Automatic cluster evolution using gravitational search algorithm and its application on image segmentation”, Engineering Applications of Artificial Intelligence,Vol. 29, pp. 93-103, 2014. A. Izzah, R.V.H. Ginardi, and R. Sarno, “Gravitational Search Algorithm dengan Operator Disruption sebagai Optimasi pada Artificial Neural Network untuk Klasifikasi Data”, Prosiding Konferensi Nasional Informatika ITB, Vol. 3, pp. 102-107, 2013.
110
[9]
[10]
[11]
[12]
[13] [14]
A. Izzah, R.V.H Ginardi, andA. Saikhu, “Pendekatan Algoritma Heuristik dan Neural Network untuk Screening Test pada Urinalysis”, Jurnal Cybermatika, Vol. 1, No 2, pp. 29-35, 2013 M. Yin, Y. Hu, F. Yang, X. Li, and W. Gu,“A novel hybrid K-harmonic means and gravitational search algorithm approach for clustering”, Expert Systems with Applications, Vol. 38, pp. 9319–9324, 2011. S. Sarafrazi, H. Nezamabadi-pour, S. Saryazdi, “Disruption: A new operator in gravitational search algorithm”, Scientia Iranica D, Vol. 18, pp. 539–548, 2011. E.S. Nicoara, “Mechanisms to Avoid the Premature Convergence of Genetic Algorithms”, Matematică - Informatică – Fizică, Vol. 59, pp. 87-96, 2009. Tipler, Fisika Untuk Sains Dan Teknik Jilid 1, Erlangga, Jakarta 1998 S. Mirjalili, S. Hashim, H. Sardroudi, “Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm”, Applied Mathematics and Computation, Vol. 218, pp. 11125–11137, 2012.
