Systematic Testing of Genetic Algorithms: A Metamorphic Testing

Download PDF
1 downloads 0 Views 235KB Size Report
Comment
2 Aug 2018 - In that instance, if the true answer was known, the land ... To test genetic algorithms we can use a testing technique called meta- ... that evaluates each potential solution and returns a value based on that ..... Statements.
Systematic Testing of Genetic Algorithms: A Metamorphic Testing based Approach

arXiv:1808.01033v1 [cs.SE] 2 Aug 2018

Janette Rounds and Upulee Kanewala Gianforte School of Computing Montana State University Bozeman, Montana 59717 Email: [email protected] Email: [email protected]

Abstract Genetic Algorithms are a popular set of optimization algorithms often used to aid software testing. However, no work has been done to apply systematic software testing techniques to genetic algorithms because of the stochasticity and the lack of known outputs for genetic algorithms. Statistical metamorphic testing is a useful technique for testing programs when the output is unknown or when the program has random elements. In this paper, we identify 17 metamorphic relations for testing a genetic algorithm and show, through mutation testing, that these relations are more effective at finding defects than traditional unit tests based on known outputs. We examined the failure rates of the system-level relations when initialized with various fitness functions. We found three relations failed excessively and we then modified these relations so that they failed less often. We also identified some metamorphic relations for genetic algorithms that are generalizable across different types of evolutionary algorithms and showed that our relations had similar mutation scores between two implementations. This is the first time statistical metamorphic testing has been applied for testing genetic algorithms. Keywords: Metamorphic Testing, Genetic Algorithm, Differential Evolution

Preprint submitted to Journal of Systems and Software

August 6, 2018

1. Introduction Genetic Algorithms are a popular optimization technique that have been used in domains from consumer goods quality assurance [1] and modeling traffic flows [2] to medical imaging [3] and materials science [4]. Additionally, Genetic Algorithms are often used to optimize the parameters or structure of other machine learning algorithms [5]. There has been a significant amount of work done on using genetic algorithms to test software [6, 7, 8] especially on using genetic algorithms in mutation testing. However, there is nothing in the literature about testing the genetic algorithms themselves even though they are often used for critical applications such as those mentioned above. This is especially problematic given that genetic algorithm’s are being used as a tool to help medical diagnoses [9, 10] and in other high-stakes situations. There has been little work on testing machine learning algorithms in general [11]. The reasons why so little work has been done in this area are myriad. The first reason is that most machine learning algorithms incorporate at least one random element. Random elements are challenging to test because the same input can produce different results [12]. Furthermore, the correct answer is often unknown. For example, a person can design a genetic algorithm to optimize land zoning subject to multiple constraints [13]. In that instance, if the true answer was known, the land would already have been zoned. If the true answer is in fact known, like many of the optimization problems we discuss here, the true answer and the answer a genetic algorithm returns may not match, even if the genetic algorithm is correct. For example, in the Ackley’s function which we discuss later in this paper, a genetic algorithm may not find the true global minimum for the function at (0,...,0) for a number of dimensions. Instead the genetic algorithm may find one of the many local minima for the Ackleys function and return this value. Although this local minimum may not be the true answer, the genetic algorithm still may be correct. Genetic algorithms are useful tools for approximation. However, they tend not to return exact answers, but an approximation of a good answer. If a genetic algorithm returned an exact and correct answer for a problem in which the answer was known, especially if it returned the same answer several times, we would probably suspect the genetic algorithm of having a coding error. To test genetic algorithms we can use a testing technique called metamorphic testing, first proposed by Chen et. al. [14]. Metamorphic testing 2

involves defining properties by which we can relate two or more outputs from an algorithm based on the input. If the outputs of two related inputs do not follow the property, there is an error in the program under test. Statistical metamorphic testing [12] is a useful technique when the program under test has random elements. Instead of an initial and follow-up test case, statistical metamorphic testing uses an initial and follow-up sample, which is then tested using statistical hypothesis testing. We identified 17 metamorphic relations from the literature on genetic algorithms. We then used mutation testing to demonstrate the effectiveness of the various relations on two different implementations. We also demonstrated the effectiveness of three of these relations on two different differential evolution implementations. We found that traditional deterministic unit tests were not as effective at finding mutations as metamorphic relations, and that system-level relations and unit-level relations perform well on different parts of the genetic algorithm. We also found that our three relations for differential evolution performed as well or better than the twelve relations for genetic algorithms. The rest of this paper is organized as follows: Section 2 contains related work and background on metamorphic testing, genetic algorithms, and mutation testing. In Section 3, we discuss our implementation of a genetic algorithm. In Section 4, we identify 17 metamorphic relations, five for fitness functions, three for genetic algorithm operators, and nine system-level level relations. In Section 5, we lay out several experiments we conducted. Finally, Section 6 contains our conclusions and future work. 2. Background and Related Work Genetic algorithms are often used in software testing. The most common uses of genetic algorithms are in test case generation [15], multi-objective test case generation [16], test case prioritization [17], and mutation testing [8]. There has been no work published in the literature on how to test genetic algorithms. However, Arcuri and Briand [18] state that many randomized algorithms (like a genetic algorithm) are used in software applications. Additionally, Arcuri and Briand argue for the use of statistical testing in software applications that use random algorithms. Metamorphic testing is a testing technique used when the algorithm is non-deterministic and/or when there is no way to determine the correct output [14]. To conduct metamorphic testing, first one defines a metamorphic 3

relation (or a set of metamorphic relations) for the program under test. A metamorphic relation is a property by which we can relate two outputs based on the input. Then, tests are defined for each metamorphic relation. Each test consists of an initial test case and a follow-up test case based on the metamorphic relation. Tests are then executed on the program under test. The output of the initial and follow-up test cases are evaluated to determine if the text cases follow the metamorphic relation. When the algorithm is stochastic, one cannot simply check if the result of the follow-up test is equal to the expected value. The result may be close, but because of the stochasticity, the result will not be exactly equal. Therefore, statistical tests are applied to determine whether the difference between the expected output and the actual output is statistically significant. This is called statistical a relationship between two samples, an initial and a follow-up test sample, is specified, usually taking the form of null and alternative hypotheses. The output of the program generates the samples and is compared using statistical hypothesis testing. Genetic Algorithms all consist of an encoding of potential solutions as chromosomes, e.g. a bit string, and a fitness function [19], which is a function that evaluates each potential solution and returns a value based on that evaluation. A generic genetic algorithm has a population of properly encoded potential solutions which is usually randomly generated, and a randomized way or ways to change potential solutions to see if the fitness of that solution improves. Unfit solutions are removed from the population. Testing a genetic algorithm presents several problems. Genetic algorithms are randomized algorithms. Each time the genetic algorithm is run, the output of the algorithm will be different. There is also no way to determine the ”correct” output of a genetic algorithm. In many cases, if the target output of a genetic algorithm is known, there is often no reason to use a genetic algorithm at all. Furthermore, if the data provided to the genetic algorithm is misleading in some way, the output of the genetic algorithm will not match the desired output. Thus, testing approaches that require one to compare an output against a true answer will not work for testing a genetic algorithm. In this paper, we use several problems where the correct answer is already known in order to demonstrate the process of testing a genetic algorithm. However, we also provide ways to expand the testing process in the case where the target answer is unknown. There has been no published research on how to test genetic algorithms. 4

However, Xie et al. [20] defined several possible metamorphic relations to use when testing a machine learning classifier. In machine learning, classification is the problem of delineating decision boundaries so that all examples inside of a boundary are of one class. Genetic algorithms have been shown to be effective classifiers in a multitude of cases [21, 22]. Xie et al. [20] demonstrated how to test two other machine learning methods, the k-Nearest Neighbor and the Naive Bayes Classifiers, using metamorphic testing. Xie et al. also defined several possible metamorphic relations to use when testing a machine learning classifier. These authors showed that metamorphic testing is an effective way to test machine learning classifiers. However, they only tested this approach on WEKA [23], an open source tool for performing classification, regression and other data mining tasks. Shin Yoo [24] showed how one could use metamorphic testing to validate a machine learning approach, called simulated annealing, to an optimization problem. Optimization problems consist of an objective function, f (x) that we must either minimize or maximize subject to constraints. Shin Yoo showed that metamorphic relations can be an effective way to test machine learning approaches to optimization problems, especially for certain kinds of faults. Genetic algorithms are another way to approach an optimization problem. Murphy et.al. [25] showed how an automated metamorphic testing framework can be used to test support vector machines, decision trees, and ranking algorithms. The most complex algorithm tested by Murphy et. al. is the MartiRank algorithm. This algorithm is a type of ensemble method that divides the data into a series of sub-lists which it then orders according to the ”quality” of the features, similar to a fitness function, except that it is iteration dependent. At each iteration, the model describes how to divide the data into lists and updates the quality measure. When all the rankings are completed, the algorithm reconstructs a final ranking based on the divisions and quality measures of previous iterations. Murphy et. al. designed an automated metamorphic testing system to improve the speed at which metamorphic tests can be developed. None of these papers test very complicated algorithms. The most complex of these is the MartiRank algorithm, which is based on a series of simple ranking algorithms. Genetic algorithms involve many more random components than even the most complex algorithm tested to date. In addition, Murphy et. al. and Xie et. al. used the Weka [23] implementation of these algorithms. There is no Weka implementation for a genetic algorithm. All 5

of this means that there is a great need for testing genetic algorithms thoroughly and until now there has been no established way to accomplish this task. In order to demonstrate the effectiveness of the testing approach outlined in this paper, we will use mutation testing. Mutation testing is technique that has been shown to be effective for comparing testing techniques [26]. The first step in mutation testing is to generate a number of mutants, given the source code to the program under test. These mutants are identical to the program under test except that one line has been changed. Next, the test set is run on each mutant. If the test set detects the changed line (in other words, if at least one test fails with the changed line where the test passed without the line being changed), the mutant is said to have been ’killed’. If the test set does not detect the changed line, the mutant is said to have ’survived’. Compilation and run-time errors and tests that time out are considered ’killed’ as well. Then, the number of killed mutants is counted and is divided by the total number of mutants. This gives a mutation score. A perfect mutation score would be 1, or all of the mutants detected by the test set. However, a perfect mutation score is usually impossible. Consider the example of a for loop in a Java program. f o r ( i n t i = 0 ; i < 1 0 ; i ++){} If the code above is the original program and the following is the mutant: for ( int i

= 0 ; i != 1 0 ; i ++){}

there would be no way to detect the mutant as it would be functionally equivalent to the original program. However, most mutants would not be functionally equivalent and could be detected by the test set, if an appropriate test exists. Therefore, the goal of mutation testing is to get the mutation score as close to 1 as is possible by changing tests so that they detect more mutants. In this paper, we compare the mutation score of different types of tests in order to determine which tests detect the most number of faults. 3. Genetic Algorithm Implementation Most often genetic algorithms used in classification are included as part of a larger solution. For example, a common use of genetic algorithms is to train the weights of a neural network. Genetic algorithms in an optimization context usually do not use any other algorithms. Additionally, it is trivial to 6

turn a minimization problem into a maximization problem. This is important for the future generalization of the testing technique we develop here. Therefore we implemented a genetic algorithm that optimizes a function. Algorithm 1: Genetic Algorithm Input : popSize, killRate, mutRate, δ, maxGen Output: the best solution from Pt Randomly initialize population (P0 ); Evaluate fitness for P0 ; while t < maxGen and best fitness > δ do Childt = selection Pt−1 ; Child′t = crossover Childt ; Child′′t = mutate Child′t ; Evaluate fitness for Child′′t ; Pt = replace Pt−1 , Child′′t ; t+ = 1; end A typical example of a genetic algorithm can be seen in Algorithm 1. The generic genetic algorithm takes as input a fitness function, a population size and a fraction of the population to be replaced at each time step, a mutation rate and a fitness threshold [19]. The output will be a set of real numbers that produce the “best” output of the fitness function. The first step is to randomly initialize potential solutions, and then evaluate the fitness of each potential solutions. Then while the termination criteria are unmet, a number of potential solutions are selected from the population, recombined to form children and then mutated. Selection, crossover (also called recombination), and mutation are considered the operators for the genetic algorithm. Then each child’s fitness is evaluated and some number in the original population will be replaced by the child. Our genetic algorithm uses uniform crossover with 2 parents and fitness proportionate selection. Fitness proportionate selection associates a probability of selection with a particular individual in the population [27]. If f itt is the fitness of individual t, the probability of selection is f itt pt = PN j=1 f itj 7

where N is the number of individuals in the population. Fitness proportionate selection tends to be relatively slow [28] and has a risk of slow convergence [19]. Uniform crossover allows parents to contribute individual genes to a child individual rather than sequences of genes. This eliminates positional bias [19] which was necessary as positional bias could interfere with certain metamorphic relations. The probability that a parent will contribute a particular gene to the child is pt = a1 where a represents the number of parents. When designing a genetic algorithm for optimization, if there is a single objective function and no constraints, creating a fitness function is very simple. We simply use the objective function itself as a measure of fitness. It is also trivial to regularize the output. In this project we will test both stochastic and deterministic objective functions. In addition to implementing our own Genetic Algorithm, we contacted Strasser et.al. [29] who allowed us to use their Factored Evolutionary Algorithms framework (FEA framework) as an additional implementation to test. 3.1. Differential Evolution Differential Evolution (DE) is another evolutionary algorithm that was designed for continuous spaces. DE centers around creating offspring by conducting crossover on parents and what is known as a “trial vector” [30]. A trial vector is similar to the child concept in genetic algorithms. A trial vector is generated by choosing 3 individuals – call them xi (t), x2 (t), and x3 (t) – from the population, without replacement, where x2 , and x3 are randomly selected. The trial vector, ui , is then ui (t) = xi (t) + β(x2 (t) − x3 (t))

(1)

The subtraction term generates what is known as a “difference vector”, where the beta multiplier is some user-defined positive number. Crossover then combines xi (t) and ui (t) into an offspring, which has the effect of “pushing” xi in a particular direction in the search space. There are multiple forms this crossover can take; we chose to implement binomial crossover, which is similar to uniform crossover in a genetic algorithm. Because of the representation of our chromosomes, we used binomial crossover. We used a single difference vector, and to select the target vector we used random selection. As such our differential evolution algorithm could 8

be described as DE/random/1/binomial. We implemented differential evolution ourselves and used the differential evolution algorithm from the FEA framework. 3.2. Test Problems There are a variety of established test problems used in order to assess the ability of the genetic algorithm to optimize. For the purposes of this project, we wanted problems that were continuous and scalable to more than three dimensions. Differential evolution operates almost exclusively in continuous spaces. Although the current genetic algorithm will be able to optimize in discrete and categorical spaces, we wanted problems that would also work for the differential evolution algorithm. We need problems that are scalable to more than three dimensions because most real-world problems in machine learning have feature spaces (or numbers of dimensions) much greater than three. We ran the genetic algorithm on a variety of problems in order to show how the metamorphic relations would change, or remain constant depending on the problem. To that end, we wanted at least one random function. We also wanted at least one function with multiple local minima, because local minima have a tendency to ”trap” genetic algorithms. For this project, we selected three test problems from the literature [31], Ackleys function, the Quartic function, and the Rosenbrock function. Ackleys function is a continuous, deterministic, scalable function that has multiple local minima. This function fulfills our requirement for a function with multiple local minima. √ 1 PD 2 1 PD f (x1 , ..., xD ) = −20 ∗ e−0.2 D i=1 xi − e D i=1 cos(2π∗xi ) + 20 + e (2) The Quartic function is a continuous, scalable, stochastic function that, because of the random element, may or may not have local minima. This function fulfills our requirement for a random function. f (x1 , ..., xD ) =

D X i=1

i ∗ x4i + random[0, 1)

(3)

The Rosenbrock function is a continuous, deterministic, scalable function with a single minimum. This function has neither a random element, nor multiple local minima. This means it is a useful function for comparisons

9

Class Chromosome Fitness Function Ackleys Quartic Rosenbrock Genetic Algorithm Differential Evolution Total Total without DE

LOC # of tests 22 3 14 1 17 7 11 6 18 6 112 14 82 5 271 42 189 37

# of trad. tests 2 0 5 3 5 5 2 22 20

# of MRs 1 1 2 3 1 9 3 20 17

Table 1: Lines of code, number of tests written, number of deterministic tests and number of tests written using metamorphic relations.

against both the Quartic and Ackleys functions. f (x1 , ..., xD ) =

D−1 X i=1

100(xi+1 − x2i )2 + (xi − 1)2



(4)

The genetic algorithm we implemented is organized as follows: We used three fitness functions in this work, Ackleys, Quartic and Rosenbrock functions. The Chromosome class encapsulates a potential solution to a particular problem. The Genetic Algorithm class is the biggest class by far. It contains a list of Chromosomes which represent the population. It also contains all the operators for the Genetic Algorithm, a multitude of getters and setters, and the genetic algorithm itself, modeled on the genetic algorithm we outlined in Algorithm 1. The Differential Evolution class is similar to the Genetic Algorithm class. For most analyses, the Differential Evolution class was not included. The FEA framework is a much bigger program, containing over 5000 lines of code. For comparison, our implementation used only 271 lines of code. However, much of the FEA framework code implements other algorithms such as Particle Swarm Optimization or other test problems such as the Rastrigrin Function. The lines of code directly relating the the genetic algorithm, the differential evolution algorithm or the three fitness functions only amounts to 245 lines of code. The FEA framework can be loosely organized into the genetic algorithm, the fitness functions and the differential evolution algorithm. 10

4. Metamorphic Relations In this section we lay out 17 metamorphic relations. Three of these relations also applied to differential evolution. 4.1. Metamorphic Relations for Fitness Functions The fitness functions are very different from the rest of the genetic algorithm because we do have the true answers for two of the functions. For Ackleys function, the minimum output is 0, and this occurs when the input is (0,0,...,0) for all dimensions. The maximum value of Ackleys function is approximately 22.3. Since there are many peaks in Ackleys function, there are many different sets of inputs that could reach this value. In two dimensions, one of those input sets is at (-21.6, 31.5) when the number of dimensions is 2. For the Rosenbrock function, the minimum output is 0. This occurs when the input is (1,1,...,1) for all dimensions. The maximum output is approximately 8.6490961 ∗ 107 . These values were tested using deterministic unit tests. Relation 1.1. Unit tests for the Quartic function were tested using metamorphic relations. Since the Quartic function adds a random number ∈ [0, 1) for each dimension, we do not know what the exact value of the Quartic function will be. However, we do know that the minimum value without the random numbers would be 0. We also know that the random numbers will all be strictly less than 1. Therefore, we know that the minimum value MV for the Quartic function will be 0 ≤ MV < d where d is the number of dimensions. In certain circumstances, other inputs will cause the output to be in this range. The follow up test will be (d − 1) 0’s and the last input value is 1. This input will produce a result always larger than (d) 0’s. As an example, in 4 dimensions, the initial test case would be (0,0,0,0) and the follow-up test case would be (0,0,0,1). With these inputs, the output from the initial test case would always be less than the output from the follow-up test case. Relation 1.2. The maximum value the Quartic function would take if the random elements were not added is 2.68435456∗ d(d+1) . This happens at (1.28, 2 1.28, ..., 1.28), or (-1.28, -1.28, ..., -1.28) for all dimensions. The random number generator includes 0, so the maximum value is always greater than or equal to this number. Since there are d random numbers added, and the random numbers are pulled from a uniform distribution, the mean maximum d value will be 2.68435456∗ d(d+1) + d2 and the variance is 12 . For this test, since 2 11

we have fairly complete knowledge of the distribution, we could have used simple statistical tests. However, we used statistical metamorphic testing [12] because it is not always possible to know the distribution of the fitness function. To perform the statistical metamorphic tests, we generated two samples, each with 20 observations, by running the Quartic function on the input (1.28, 1.28, ..., 1.28) 20 times for each sample (e.g (1.28, 1.28) for 2 dimensions) and recording the output of the function. Our null hypothesis was that the mean of the two samples would not be equal to each other. Our alternative hypothesis was that the means would be equal to each other. Relation 1.3. One of the metamorphic relations often mentioned is changing the order of the attributes [20]. For Ackleys function, this is a valid metamorphic relation that we use in the testing process. As an example, an initial input of (6.4, 2.5, 1.25) and a follow-up input of (1.25, 2.5, 1.28) will both produce an output of 13.24197384. However, for the Quartic function and the Rosenbrock function, this is invalid. Consider the following example for the Quartic function in 3 dimensions. The first input is (0.25, 0.5, 1.28). This produces a result of 9.94196993. The second input is (1.28, 0.5, 0.25). As you can see, only the order changes between the two inputs. However, the output for the second input is 4.64107331, less than half the output of the first example. Relation 1.4. Another metamorphic relation for the Quartic and Rosenbrock functions is to compare the maximum output of the fitness functions given some potential solution inside of the range of expected values, given some potential solution outside of the range of expected values, and finally the first solution again. The fitness function class will adjust the maximum output if some solution produces a higher output than is currently the case. This happens so that we do not get a fitness greater than the maximum fitness. The fitness should change between the first and the last test, and the fitness of the first test should be higher. This does not apply to the Ackleys function because the values outside the range of expected values are not necessarily higher or lower than those inside the range. Again, we used statistical metamorphic testing in this relation. We generated 2 samples, each with 20 observations. The initial sample was generated by feeding the maximum input (e.g. (30, 30, 30) for 3 dimensions in the Rosenbrock function) into the function 20 times. Next we ran the function with a single input that was larger than the maximum input (e.g. (80, 80, 80) in 3 dimensions for the 12

Rosenbrock function). Finally, we generated the follow-up sample by running the function on the same input as the initial sample. Our null hypothesis was that the mean of the two samples would be equal to each other, while our alternative hypothesis was that the mean of the initial sample would be greater than the mean of the follow-up sample. Relation 1.5. Since all of our fitness functions can generalize to more than three dimensions, another metamorphic relation for the fitness functions is to create a fitness function object for two dimensions and input a good solution in two dimensions, say (1,1) for the Rosenbrock function. The follow-up test would be to create a fitness function object for greater than two dimensions and input the same good solution, simply with more dimensions, e.g. (1,1,1,1) for the Rosenbrock function. The scaled fitness for the test and the follow-up test will be the same. 4.2. Metamorphic relations for genetic algorithm operators 4.2.1. Metamorphic relations for mutation Relation 2.1. Let us assume that the mutation rate is 1. Let us also assume that we have a solution with m dimensions. We then run our mutation operator on our solution. Each observation of a value in the mutated solution will be different from the corresponding un-mutated observation by some amount. This amount will be greater than or equal to 0 and less than 0.1. The average amount will be 0.05. An example input initial test case would be the randomly generated solution (3, 27, 6, 14, 30, 16, 1, 16, 10, 29) for 10 dimensions. We then run the mutation operator, with the mutation rate set to 0.1 on this solution and record the difference between the input and the output. The follow-up test case would use another randomly generated solution with the mutation operator set to 0.9 and the difference between the input and output of the mutation operator would be recorded. Our null hypothesis was that the mean difference between the input and output for the two test cases would be equal. Our alternative hypothesis was that the mean difference for the follow-up test case when the mutation rate was set to 0.9 would be greater than the mean difference for the initial test case. 4.2.2. Metamorphic relations for crossover Relation 2.2. The crossover operator takes as input a set of ’parent’ solutions parents and outputs a ’child’ solution that is a combination of elements of the parent solutions. For each dimension in the solution, the probability that 13

1 the parent will contribute their value for that dimension is |parents| . Given a set of parents that are unique (i.e. no common elements between the parents), we can determine which element of the child solution came from which parent. This relation also used statistical metamorphic testing. With parents (1,2,3,4) and (5,6,7,8), our initial test case would set the crossover rate to 0.5 and run the crossover operator 20 times, generating 20 children. The follow-up test case would set the crossover rate to 1 and again generate 20 children. We would then calculate the proportion of elements in each child that came from the first parent. Our null hypothesis was that the average proportion for the initial sample would be equal to the average proportion for the follow-up sample. Our alternative hypothesis was that the average proportion for the initial sample would be greater than the average proportion for the follow-up sample. This metamorphic relation also applies to Differential Evolution.

4.2.3. Metamorphic relations for selection Our implementation of the genetic algorithm uses fitness proportionate selection. This means that each solution’s fitness is scaled by the total fitness of the population. These scaled fitnesses all add up to 1. More specifically, this algorithm uses roulette wheel selection. This means that we create a scaled fitness vector, and we select a potential solution based on these scaled fitness vectors. Smaller fitness values are more likely to be selected, but in order to maintain diversity in our population of solutions, there must be some chance that solutions with higher fitness can be selected. Relation 2.3. This metamorphic relation involves running the selection operator several times on two populations, one that contains several copies of the ideal solution, and one that does not but is identical in every other way. For an example with the Rosenbrock fitness function and 2 dimensions, our initial test case might have the individuals (2,3), (5,10), (27, 8), (17, 11), and (29, 2). The follow-up test case might have the individuals (3,4), (5,10), (17,11), (1,1), and (1,1). The average fitness for the initial test case would be worse than the average fitness for the follow-up test case. If the ideal solution is unknown, a good solution, as determined by the fitness function, could be used instead of the ideal solution. 4.3. System-level metamorphic relations Relation 3.1. The number of generations we allow the genetic algorithm to run is a crucial parameter. If we increase the number of generations, the 14

average fitness will improve, unless the fitness threshold is reached, and the algorithm exits early. One way to prevent the early exit is to select a test problem that has many local optima so as to prevent early convergence, such as the Ackley’s function. An example initial test case is setting the number of generations to 50. An example follow-up test case would be setting the number of generations to 5000. This relation applies to Differential Evolution. Relation 3.2. Another crucial parameter is the population size. As the population size increases, the average fitness will also improve. This is because with increased individuals in the population, we increase the chance that one of those individuals will encounter the true solution. For an initial test case, we used a population size of 5. The population size for the follow-up test case was 500. The average fitness for the follow-up test case will be better (lower) than the average fitness for the initial test case. Interestingly, for Differential Evolution smaller population sizes improve average fitness more than larger population sizes (assuming some low number of iterations and/or a problem with many local minima). This could be because the decreased diversity of the population leads to a faster decrease in fitness. Alternatively, there could be an interaction between the population size and another important parameter, β that influences the ideal size of the population. We used the same initial and follow-up test cases for differential evolution. The average fitness for the follow-up test case will be worse (larger) than the average fitness for the initial test case. Relation 3.3. Finally, if we increase the threshold parameter, the average fitness will be worse, but the average number of iterations run by the algorithm will decrease. This is because when the threshold is set at a higher value the algorithm reaches the threshold relatively quickly and exits. When the threshold is set at a lower value, the algorithm continues searching until it reaches the lower value, so the fitness will be lower (better). Our initial test case was 0.5 and our follow-up test case was 0.05. The average fitness for the initial test case was greater than the average fitness for the follow-up test case. The average number of iterations run for the initial test case was less than the average number of iterations run for the follow-up test case. The other two parameters, mutation rate and replacement rate, are much more difficult to define relations for. In Differential Evolution, β and the crossover rate can be seen as approximate substitutions for mutation and 15

fitness per iteration

fitness

10−3 10−5 10−7 0

50

100 150 200 250 300 350 400 450 500 550 600 iterations mut=0 kill=0 mut=0.8 kill=1

mut=0.1 kill=0.4 mut=0.8 kill=2

Figure 1: Fitness for the best solution at each iteration when the parameters are varied. When fitness decreases, the solution is improving.

16

replacement rates. As you can see from Figure 1, when both of these parameters are at 0, we will see no improvement in total fitness in the population over the generations. As we increase the parameters, we will see improvements in the total fitness and the best fitness will be reached in fewer iterations. At a certain point, total fitness will oscillate. If the oscillations are small, we should still see convergence towards our ideal solution. However, these oscillations, no matter how small, will lead to a greater number of generations needed to find the ideal solution. This point where total fitness begins to oscillate is not easily identifiable and is thought to depend on the problem. We can see that in this instance, unstable oscillations occur when the mutation rate is at 0.8 and the replacement rate is at 2. When the unstable oscillations occur, overall fitness increases rather than decreasing. It is unclear how the mutation rate and replacement rate interact to speed or slow the rates of convergence towards the best solution in general. However, we can still define some metamorphic relations for mutation rate and replacement rate. Relation 3.4. When both parameters are at 0, the average fitness will be worse than if both parameters are at 0.5. This is because the final solution depends only on the randomly initialized potential solutions, rather than the changes made to those solutions. This applies to crossover and β in Differential Evolution. Relation 3.5. On the other hand, if both parameters at 1 the average fitness does not follow the same behavior. We think this is because each time mutation happens, the solution will only change by a small amount. Recombination never adds new solutions. Thus each solution is only being changed by a small amount. It is possible that the mutation operator can be changed so that setting both parameters equal to 1 will be roughly equivalent to a random search. This relation was tested in section 5.4, but was not tested in the remainder of the experiments. Relation 3.6. If we hold mutation rate constant at 0, and increase the replacement rate to 0.5, the average fitness of the best solution in the population will be better than when both mutation rate and replacement rate are at 0. Performing Relation 3.6 with mutation and replacement rate switched, however, does not produce the same results. This is because mutation happens only when a potential solution is selected, so the mutation rate depends on the selection rate. 17

Relation 3.7. If however, we hold the recombination rate constant at some low number, say 0.1, and set the mutation rate at 0 and 0.5, for example, the average fitness for the best solution will be better when the mutation rate is at 0.5 than when the mutation rate is set at 0. Relation 3.8. For this reason, the common wisdom in the genetic algorithm literature is that mutation rate should be set lower than the replacement rate [32]. If this common wisdom holds, one metamorphic relation for mutation is to run the whole algorithm when the parameters follow this common wisdom, and then run the algorithm again when the values for the parameters are swapped. The average fitness for the best solution when the parameters follows this common wisdom should be better than when the parameters do not follow this common wisdom. However, when the values are 0.1 and 0.8, this is not the case. Much of the time, the swapped parameters (i.e. the mutation rate set higher than the replacement rate), performed better than the parameters that followed the common wisdom. This high rate of failure is one reason this test was not used in the analyses, other than in section 5.4. Relation 3.9. On the other hand, we can test this type of interaction between the replacement and mutation rates by using the values 0 and any other number strictly greater than 0 and less than or equal to 1. In that case, the algorithm should behave like Relation 3.6 with the parameters switched. In other words, the average best fitness for this algorithm will be no better than when both parameters are set at 0. 4.4. Deterministic unit tests We implemented several unit tests for both the genetic algorithm and differential evolution that did not use metamorphic relations. These relations included testing initialization, selecting the best chromosome from the population, that fitness is changed when the update fitness function was called, and checking constant and known values. These were used for comparisons against the metamorphic relations. The scope of these deterministic tests is limited, but testing many parts of a genetic algorithm is difficult, if not impossible, without metamorphic testing. 5. Experiments In order to conduct mutation testing, we used the PIT mutation testing tool [33]. Although the PIT tool does not include source code of the mutants, 18

Statements Covered Our Implementation FEA Framework

Total Deterministic Tests Tests 243 87 125 119

Total Function System MRs Level MRs Level MRs 169 136 165 125 124 125

Table 2: Statement Coverage for each set of tests, reported as a ratio between lines of code covered over total lines of code, leaving aside the differential

it does include the line number where the mutation happened and the type of mutation (e.g. negated conditional, replaced addition with subtraction). PIT reports each mutant that survived, was killed, timed out or was not covered for each class. For each test, 182 mutants were generated. PIT uses Line coverage to assess mutants not covered by test cases. The PIT mutation tool generates the following types of mutants: • Replaced operator (+, −, ∗, /) with another operator. • Changed conditional boundary. • Negated conditional. • Changed increment. • Mutated return value. • Removed call to other function. Unless otherwise specified by the metamorphic relation, the parameters for the genetic algorithm were as follows: • popSize = 50. • mutRate = 0.1. • killRate = 0.4. • δ = 0. • maxGen = 1000.

19

Total Lines 194 245

Genetic Algorithm Chromosome Fitness Function Ackleys Function Quartic Function Rosenbrock Function Combined Gen. Algo. Combined Fit. Func. Total

All Tests 90/117 77% 14/15 93% 9/11 81% 13/14 93% 8/8 100% 18/18 100% 104/132 79% 48/51 94% 152/182 84%

Deterministic Tests 20/117 17% 8/15 53% 5/11 46% 11/14 79% 0/8 0% 16/18 89% 28/132 21% 32/51 63% 60/182 33%

All Relations 90/117 77% 11/15 73% 7/11 64% 12/14 86% 8/8 100% 18/18 100% 101/132 77% 45/51 88% 147/182 81%

Table 3: Mutation scores for the genetic algorithm implementation comparing deterministic tests and all tests involving metamorphic relations. We report totals by class and then totals for the all classes with Genetic Algorithm components (Genetic Algorithm and Chromosome) and all classes for fitness function components (Fitness Function, Ackleys Function, Quartic Function, and Rosenbrock Function)

5.1. Overall results Relation 2.3 detected a fault in our original implementation of the genetic algorithm. The fault in the selection function occurred because we were prioritizing higher fitness, rather than lower fitness. Once we had fixed this error, we generated mutation scores for all types of tests. As shown in Table 3, the overall mutation score is 85% for all the tests written. Several classes received a mutation score of 100%. The vast majority of mutants that were not killed were due to there being no coverage, or tests, for the code that was changed (44 of the 59 mutants that were not killed). This is a problem that can be easily remedied with more tests.

20

Genetic Algorithm Chromosome Fitness Function Ackleys Function Quartic Function Rosenbrock Function Combined Gen. Algo. Combined Fit. Func. Total

All Relations Function-Level 90/117 48/117 77% 41% 11/15 11/15 73% 73% 7/11 7/11 64% 64% 12/14 12/14 86% 86% 8/8 8/8 100% 100% 18/18 18/18 100% 100% 101/132 59/132 77% 45% 45/51 45/51 88% 88% 147/182 104/182 81% 57%

System-Level 74/117 64% 11/15 73% 6/11 55% 8/14 57% 8/8 100% 7/18 39% 85/132 64% 29/51 57% 114/182 63%

Table 4: Mutation scores for the genetic algorithm implementation comparing functionlevel and system-level tests. We report totals by class and then totals for the all classes with Genetic Algorithm components (Genetic Algorithm and Chromosome) and all classes for fitness function components (Fitness Function, Ackleys Function, Quartic Function, and Rosenbrock Function)

5.2. Deterministic and Metamorphic Comparison We next divided the tests into deterministic and metamorphic tests. All tests were run again and the mutation score was calculated for each type of test. For the deterministic tests, the mutation score was 33%, as seen in Table 3. This is quite low, due at least in part to not being able to test the random elements of the genetic algorithm. The mutation score for metamorphic testing was 81%. Part of the reason for the big disparity was potentially due to the fact that we have identified two different types of metamorphic tests, function-level and system-level metamorphic relations.

21

5.3. System-level and Function-level comparison Since the deterministic tests were implemented only at the function level and there were no system-level deterministic tests, we divided the metamorphic relations into system-level and function level tests. The mutation score for the function level metamorphic relations was 0.577, from Table 4. This was higher than the deterministic tests. We were originally concerned that this was due to differences in coverage of the different types of tests. However if we approximate a normalization by dividing the number of mutants killed by the lines of code covered, we see that the deterministic tests only scored 0.588, while the function level metamorphic relations obtained a score of 0.772 (the best possible score would be 1.055). This means that the deterministic tests had a lower mutation score, even when measured relative to statement coverage. The mutation score for the system-level relations was even higher than for the function-level relations, at 0.648. Most of what is driving that number is the higher coverage for the genetic algorithm class. As you can see in Table 4, when separated out by class, the whole algorithm relations had a mutation score of 0.638 for the Genetic Algorithm class, while the function-level relations only had a mutation score of 0.414. On every other class, the function-level relations had higher mutation scores. 5.4. System-level Tests with Different Fitness Functions Finally, we examined the failure rates of the whole algorithm tests. We expected that some small number of tests would fail if the tests were run a sufficient number of times, given the nature of statistical tests. However, we suspected we were seeing too many failures for statistical likelihood. If this was due to an actual error in the genetic algorithm, we expected that the test failure would be consistent. The failures we were seeing occurred inconsistently. One potential cause of this was the fact that we were using the Ackleys function to set up the tests for several metamorphic relations. Ackleys function has many local optima. We hypothesize that if there were more failures than expected, these failures were due to the Ackleys Function getting stuck in local optima and not converging towards a global optima. We restricted the set of tests run to the system-level metamorphic relations, specifically to Relations 3.1, 3.2, 3.3, 3.4, 3.5, and 3.8. We ran each set of relations ten times. We then changed the fitness functions for each relation and ran the relations another ten times. For example, we would run Relation 3.1 ten times with Ackleys function, then ten times with the Quartic function, and finally ten times with the Rosenbrock function. If statistical 22

Ackleys Quartic Rosenbrock

3.1 7 6 0

3.2 3.3 3.4 3.5 0 0 0 2 0 0 0 3 0 0 0 4

3.8 8 5 5

Table 5: Number of failures in ten runs for each relation when the fitness function is varied

tests were causing the failures, we expected that failure would occur less than one time for each fitness function with each relation. As you can see from Table 5, only 3 relations failed at all, and all the failing relations failed much more than we would expect given a 95% confidence interval. Relations 3.2, 3.3 and 3.4 succeeded consistently. We restricted our further tests to Relations 3.1, 3.5 and 3.8. Relation 3.1 failed only when run with Ackleys and Quartic functions. This fit our hypothesis that failing tests were getting stuck in local optima. We ran this relation 20 more times with just the Rosenbrock function and found that it did not fail in any of those runs. However, running Relations 3.5 and 3.8 with the Rosenbrock function did not alter the amount of failures seen. One option for Relation 3.5 was to increase the average alteration made to each value performed by the mutation operator, in hopes that it would improve the failure rate. However, this would require altering Relation 2.1 to match. Instead we developed Relations 3.6 and 3.7 to substitute for this relation. Relation 3.8 was the most problematic relation, also probably requiring a change to the mutation operator. We identified Relation 3.9 to replace Relation 3.8. After we replaced the faulty relations, we ran Relations 3.1, 3.2, 3.3, 3.4, 3.6, 3.7, and 3.9 another twenty times each, and saw no more failures. 5.5. FEA framework We used the same tests and relations for the FEA framework as we did for our implementation. The authors of [29] had previously constructed 70 unit tests for the FEA framework. All but 2 of the unit tests were irrelevant to the genetic algorithm, fitness functions and differential evolution. As you can see in Table 6 the whole algorithm relations achieved the highest mutation score in the genetic algorithm classes, while the unit level relations achieved higher mutation scores for the fitness function classes. Both types of relation had higher mutation scores than the deterministic unit tests. Since more deterministic tests were implemented, but the mutation scores for the 23

Genetic Algorithm Fitness Functions Total

Initial All Tests 0/89 62/89 0% 70% 24/71 47/71 33% 66% 24/160 109/160 15% 68%

Deterministic Tests 35/89 39% 35/71 49% 70/160 44%

All Relations 62/89 70% 43/71 60% 105/160 66%

Table 6: Mutation scores for the FEA framework comparing initial results, deterministic tests and all tests involving metamorphic relations

Genetic Algorithm Fitness Functions Total

All Relations Function-Level 62/89 37/89 70% 42% 43/71 39/71 60% 54% 105/160 76/160 66% 48%

System-Level 62/89 70% 36/71 50% 98/160 61%

Table 7: Mutation scores for the FEA framework comparing function-level and systemlevel tests.

relations was higher, we can surmise that the metamorphic relations were far more effective than the deterministic unit tests. When we compared the results of our implementation to the FEA framework we found lower mutation scores for the FEA framework, as seen in Table 8. This suggests that we were possibly targeting our relations to the implementation. However, the mutation scores for the metamorphic relations were still higher than the mutation scores for the deterministic tests, suggesting that these relations are still more effective at detecting errors than deterministic tests. 5.6. Differential Evolution We ran five tests from the genetic algorithm on the differential evolution algorithm on each implementation. Two of these tests were deterministic while the remaining three were metamoprhic relations. Tests performed much better than expected on the differential evolution algorithm, given that all but one of the tests was virtually identical to the genetic algorithm tests. 24

FEA framework Our impl.

Initial 15%

All 68%

Deterministic 44%

0%

83%

33%

MRs Function MRs System MRs 66% 48% 61% 81%

58%

64%

Table 8: Comparison between mutation scores for our implementation and the FEA framework

FEA framework Our impl.

All Deterministic 37/39 19/39 95% 49% 50/58 7/58 86% 12%

MRs 31/39 79% 42/58 72%

Table 9: Comparison between mutation scores for our implementation and the FEA framework on the differential evolution algorithm

The mutation scores for all tests for the FEA framework implementation of differential evolutions was 95% while for our implementation the mutation score for all tests was 86%. This was the opposite of the genetic algorithm tests in that the FEA framework tests performed better than the tests on our implementation. Additionally, in the FEA framework, differential evolution tests outperformed our implementation of the genetic algorithm in terms of mutation score, while our implementation of the genetic algorithm slightly outperformed our implementation of differential evolution. This increase in performance is intriguing, although it probably can be attributed to differential evolution being a simpler program with fewer random elements. 6. Conclusions 6.1. Contributions In this work we identified metamorphic relations of genetic algorithms, and genetic algorithm operators. Additionally, we have identified metamorphic relations of genetic algorithms that translate to differential evolution, and may be more illustrative in differential evolution than in genetic algorithms. We defined 17 metamorphic relations, five for the fitness functions, 25

three for the genetic algorithm operators and nine for the whole algorithm. We compared the metamorphic relations to the deterministic unit tests for the genetic algorithm. We found that metamorphic relations for our implementation had a mutation score of 81% while traditional deterministic unit tests had a mutation score of 33%. We also compared the functionlevel metamorphic relations to the system-level metamorphic relations on the genetic algorithm for our implementation. Function-level metamorphic relations had a mutation score of 57% while system-level metamorphic relations had a mutation score of 63%. For the FEA framework, the mutation score for the metamorphic relations was 66% while the mutation score for the deterministic tests was 44%. When comparing system-level and functionlevel metamorphic relations for the FEA framework, the mutation score for system-level metamorphic relations was 61%. The mutation score for the function-level metamorphic relations was 48%. Additionally, we modified two relations that failed more often than was statistically likely on both implementations of the genetic algorithm when no fault was present. Finally, we used 5 tests, two deterministic unit tests and three metamorphic relations, on the differential evolution algorithm. The mutation score for the metamorphic relations was 79% for the FEA framework and 72% for our implementation. The mutation score for the deterministic tests was 49% for the FEA framework and 12% for our implementation. These comparisons demonstrated the effectiveness of the metamorphic testing approach when testing genetic algorithms. We assessed the mutation score relative to the statement coverage, and found that function-level metamorphic relations performed better than function-level deterministic tests, despite there being more deterministic tests. This result was consistent across both genetic algorithm implementations we tested. We examined the failure rates of the system-level relations when initialized with various fitness functions. We found one relation that only performed well when paired with a particular fitness function. Additionally, we found two relations that do not perform well no matter the fitness function. We developed two new relations to replace these problematic ones. 6.2. Future Work Future work for metamorphic testing on genetic algorithms would include identifying individual metamorphic relations that are either more generalizable, or kill more mutants than other metamorphic relations. We also plan to test these relations on different types of operators and different purposes for 26

a genetic algorithm. We plan to develop relations for other operators, purposes, or algorithms. We can then also test those relations on this genetic algorithm implementation. We would like to identify types of mutants that survive more often, in order to identify metamorphic relations that are able to target these mutants, or show that these mutants are more likely to be equivalent. Finally, any future work would have to include the identification of more metamorphic relations, and the implementation of more tests. References [1] C. Lee, K. Choy, G. Ho, C. Lam, A slippery genetic algorithm-based process mining system for achieving better quality assurance in the garment industry, Expert Systems with Applications 46 (2016) 236–248. [2] S. Chiappone, O. Giuffr`e, A. Gran`a, R. Mauro, A. Sferlazza, Traffic simulation models calibration using speed–density relationship: An automated procedure based on genetic algorithm, Expert Systems with Applications 44 (2016) 147–155. [3] N. K. Pareek, V. Patidar, Medical image protection using genetic algorithm operations, Soft Computing 20 (2016) 763–772. [4] J. B. A. Davis, S. L. Horswell, R. L. Johnston, The application of a parallel genetic algorithm to the global optimisation of gas-phase and supported gold-iridium nanoalloys, The Journal of Physical Chemistry C (2016). [5] L. J. Perreault, M. Thornton, R. Goodman, J. W. Sheppard, A swarmbased approach to learning phase-type distributions for continuous time bayesian networks, in: Computational Intelligence, 2015 IEEE Symposium Series on, IEEE, 2015, pp. 1860–1867. [6] B. Jones, H.-H. Sthamer, D. Eyres, Automatic structural testing using genetic algorithms, Software Engineering Journal 11 (1996) 299–306. [7] J. Wegener, H. Sthamer, B. F. Jones, D. E. Eyres, Testing real-time systems using genetic algorithms, Software Quality Journal 6 (1997) 127–135.

27

[8] C. P. Rao, P. Govindarajulu, Genetic algorithm for automatic generation of representative test suite for mutation testing, International Journal of Computer Science and Network Security (IJCSNS) 15 (2015) 11. [9] K. C. Tan, Q. Yu, C. Heng, T. H. Lee, Evolutionary computing for knowledge discovery in medical diagnosis, Artificial Intelligence in Medicine 27 (2003) 129–154. [10] C. A. Pena-Reyes, M. Sipper, A fuzzy-genetic approach to breast cancer diagnosis, Artificial intelligence in medicine 17 (1999) 131–155. [11] X. Xie, J. Ho, C. Murphy, G. Kaiser, B. Xu, T. Y. Chen, Application of metamorphic testing to supervised classifiers, in: Quality Software, 2009. QSIC’09. 9th International Conference on, IEEE, 2009, pp. 135– 144. [12] R. Guderlei, J. Mayer, Statistical metamorphic testing testing programs with random output by means of statistical hypothesis tests and metamorphic testing, in: Quality Software, 2007. QSIC’07. Seventh International Conference on, IEEE, 2007, pp. 404–409. [13] T. J. Stewart, R. Janssen, M. van Herwijnen, A genetic algorithm approach to multiobjective land use planning, Computers & Operations Research 31 (2004) 2293–2313. [14] S. C. C. Chen, Tsong Y., S. M. Yiu, Metamorphic testing: a new approach for generating next test cases (1998). [15] L. D. Geronimo, F. Ferrucci, A. Murolo, F. Sarro, A parallel genetic algorithm based on hadoop mapreduce for the automatic generation of junit test suites, in: Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conference on, IEEE, 2012, pp. 785–793. [16] C. Henard, M. Papadakis, G. Perrouin, J. Klein, Y. L. Traon, Multiobjective test generation for software product lines, in: Proceedings of the 17th International Software Product Line Conference, ACM, 2013, pp. 62–71.

28

[17] C. Sharma, S. Sabharwal, R. Sibal, Applying genetic algorithm for prioritization of test case scenarios derived from uml diagrams, arXiv preprint arXiv:1410.4838 (????). [18] A. Arcuri, L. Briand, A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering, Software Testing, Verification and Reliability 24 (2014) 219–250. [19] T. M. Mitchell, et al., Machine learning. wcb, 1997. [20] X. Xie, J. W. Ho, C. Murphy, G. Kaiser, B. Xu, T. Y. Chen, Testing and validating machine learning classifiers by metamorphic testing, Journal of Systems and Software 84 (2011) 544–558. [21] H. Ishibuchi, T. Murata, I. T¨ urksen, Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems, Fuzzy Sets and Systems 89 (1997) 135–150. [22] A. L. Corcoran, S. Sen, Using real-valued genetic algorithms to evolve rule sets for classification, in: Evolutionary Computation, 1994. IEEE World Congress on Computational Intelligence., Proceedings of the First IEEE Conference on, IEEE, 1994, pp. 120–124. [23] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I. H. Witten, The weka data mining software: an update, ACM SIGKDD explorations newsletter 11 (2009) 10–18. [24] S. Yoo, Metamorphic testing of stochastic optimisation, in: Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, IEEE, 2010, pp. 192–201. [25] C. Murphy, K. Shen, G. Kaiser, Automatic system testing of programs without test oracles, in: Proceedings of the eighteenth international symposium on Software testing and analysis, ACM, 2009, pp. 189–200. [26] J. H. Andrews, L. C. Briand, Y. Labiche, Is mutation an appropriate tool for testing experiments?[software testing], in: Software Engineering, 2005. ICSE 2005. Proceedings. 27th International Conference on, IEEE, 2005, pp. 402–411.

29

[27] P. J. Hancock, An empirical comparison of selection methods in evolutionary algorithms, in: AISB Workshop on Evolutionary Computing, Springer, 1994, pp. 80–94. [28] D. E. Goldberg, K. Deb, A comparative analysis of selection schemes used in genetic algorithms, Foundations of genetic algorithms 1 (1991) 69–93. [29] S. Strasser, J. Sheppard, N. Fortier, R. Goodman, Factored evolutionary algorithms, in: IEEE Transactions on Evolutionary Computation, 2016, p. Under review. [30] R. Storn, K. Price, Differential evolution-a simple and efficient adaptive scheme for global optimization over continuous spaces, volume 3, ICSI Berkeley, 1995. [31] M. Jamil, X.-S. Yang, A literature survey of benchmark functions for global optimisation problems, International Journal of Mathematical Modelling and Numerical Optimisation 4 (2013) 150–194. [32] K. Deb, S. Agrawal, Understanding interactions among genetic algorithm parameters., in: FOGA, 1998, pp. 265–286. [33] H. Coles, Pit mutation testing, 2012.

30