Using Chaos in Genetic Algorithms John Determan and James A. Foster, U. Idaho, Dept of Computer Science, Moscow ID 83844-1010
[email protected],
[email protected]
1 Abstract We have performed several experiments to study the possible use of chaos in simulated evolution. Chaos is often associated with dynamic situations in which there is feedback, hence there is speculation in the literature that chaos is a factor in natural evolution. We chose the iterated prisoner’s dilemma problem as a test case, since it is a dynamic environment with feedback. To perform an initial assessment of the use of chaos we used the logistic function, a simple equation involving chaos, as the basis of a special mutation operator, which we call λ mutation. The behavior of the logistic function is well known and comprises three characteristic ranges of operation: convergent, bifurcating, and chaotic. We hypothesize that the chaotic regime will aid exploration, and the convergent range will aid exploitation. The bifurcating range is likely neutral, and hence an insignificant factor. Our results confirm these expectations.
2 Introduction Chaos underlies many natural phenomena, from turbulent fluid flow to global weather patterns (Gleick), from healthy heart rhythms (Goldberger, Rigney, and West) to DNA coding sequences (Ohno). Complex dynamic systems with intricate feedback paths may often be understood in terms of chaos theory. Therefore it is natural to speculate that evolutionary processes may also be understood in terms of chaos. The book Beyond Natural Selection (Wesson) thoroughly motivates and discusses this speculation. To explore of this idea, we incorporated a simple chaotic equation, the logistic function, into a simple genetic algorithm. 2.1
The Iterated Prisoner’s Dilemma Problem A problem involving coevolution is a natural candidate for this initial test. Coevolution introduces a dynamic environment, and a natural feedback path. The iterated prisoners dilemma problem with each individual scored against the entire population is such a problem, and will be our target application. The Iterated Prisoner’s Dilemma (IPD) was first implemented in genetic algorithms by (Axelrod). To understand the IPD, suppose that two prisoners are being held as suspects in a crime. The prisoners attempt independently to get the most advantageous deal from the police by either supporting or turning on their partner, that is by cooperating with their partner or defecting. However, the consequences of their actions are intertwined, such that although the decision is binary, it is not simple. A payoff matrix, Table 1, represents the severity of a criminals sentence as a function of both his and his partner’s actions. A high score represents a light sentence and therefore a successful pair of decisions for the criminal receiving the score. In the iterated version, the score is summed over multiple plays, which makes the study of general strategies interesting, since the only rational move in a single game is defection. Move Player 1 Cooperate Cooperate Defect Defect
Player 2 Cooperate Defect Cooperate Defect
Score Player 1 Player 2 3 3 0 5 5 0 1 1
Table 1. Payoff matrix for the Iterated Prisoner’s Dilemma game. As in Axelrod’s initial experiments, we represent an IPD strategy as a response to a history of three previous pairs of moves, together with six initial moves. Since there are two possible moves for each player,
this means our strategies will be represented as binary strings with 22*3 (including itself) for 10 iterations and computing the average cumulative score per opponent. 2.2 Chaos A common and simple chaotic function, the logistic equation (Moon) is: (1) xn+1 = λ xn (1-xn) The properties of the logistic function are well known, but we briefly discuss them here. For values of λ between 0 and 3, (1) will converge to some value x. For λ between 3 and about 3.56 the solution to (1) bifurcates into 2, then 4, then 8 (and so on) periodic solutions. For λ between 3.56 and 4 the solutions to (1) become fully chaotic: neither convergent nor periodic, but variable with no discernable pattern. As λ approaches 4, the variation in solutions to (1) appear increasingly random. Figure 1 illustrates this behavior. We refer to these three regimes as convergent, bifurcating, and chaotic, respectively. λ=2
λ = 3.3
(Convergent)
λ = 3.9
(Bifurcating)
(Chaotic)
1 0.8 0.6
x 0.4 0.2 0 0
10
20
n
30
0
10
20
n
30
0
10
20
30
n
Figure 1. Solutions to the logistic equation for various λ (x0=0.25). 2.3
The Logistic Function as a Genetic Operator Our individual chromosomes maintain an IPD strategy (70 bits), a value for λ (10 bits), and a uniform crossover mask (70 bits). We use uniform crossover with a mask: one child from crossover of parents A and B has strategy bits selected from B when the mask bits from A are ones and from A when they are zeros, and it inherits its other bits (including the mask) from A. The case for the other child is similar, with the roles of A and B reversed. Since λ must be a real between zero and four, we scale the ten bit binary value in the chromosome appropriately. Though arbitrary, a ten-bit precision seemed adequate. We modify the mask itself with the logistic equation, using the λ value stored in the chromosome and treating the mask itself as the input. That is, we interpret the 70-bit mask as a real value, scaled into the range (0,1) to get xn, and set the new mask, xn+1, to the 70-bit representation of λxn (1-xn). This operation, which we call λ mutation, incorporates a type of self-directed chaos into the evolutionary search for IPD strategies via crossover. We also subjected both λ and the strategy bits to ordinary, bit-flipping mutation. See Figure 2 as a summary of our representation and use of mutation operators. An incidental requirement is the ability to perform floating-point math with a minimum of 70-bit precision (plus a few for rounding). The Windows specific type long double is limited to 64-bit precision, so we implemented a high precision floating point class that supports user specified precision. We tested this high
precision class by iterating the logistic equation on 75-bit precision numbers and verifying that the code displayed the correct behavior and values. We then used a precision of 75 bits for λ mutation. Contents: IPD Strategy Mask = xn λ Size in bits: 70 10 70 Range of phenotype: n/a (0,4) (0,1) Crossover Type: Uniform with Mask none none Mutation Type: bit flip bit flip λ mutation Figure 2. Representation. For the chromosome in the boxed region, we describe the contents and size of the gene, the range of real valued alleles and the type of genetic operations performed on that gene. Expected Effect of λ mutation Consider, for a moment, the behavior of the logistic function. For values of λ below 3, iteratively applying the logistic function to some value x0 results in convergence to some value x. Speed of convergence decreases, and the value of x increases, as λ approaches 3, and for λ > 2 the convergence is oscillatory. This will produce convergent mutation and will tend to produce crossover masks that preserve the higher order bits of the mask, but vary the lower order bits. Near convergence, the mask will become fixed. Thus an individual with convergent λ will tend to produce offspring with progressively more rigid crossover masks. Individuals with non-convergent λ will tend to have a high degree of variability in the crossover masks of their descendants, with the variability increasing as λ approaches the maximum value of 4. Note that the probability of λ mutation has a large effect on this qualitative behavior. For generations where λ mutation does not occur, the crossover mask will be inherited unchanged with high probability from the mother or father, thus good masks will tend to propagate unchanged most of the time, but with occasional modification from λ mutation. So, low values of λ should favor exploitation of good masks, and large values of λ should favor exploration of the space of possible masks (and therefore strategies). Thus, one should observe greater exploration with λ mutation than without. Similarly, one should observe increased exploration when λ is in the chaotic domain rather than the convergent one. Also, exploration should increase with the probability of λ mutation. We can quantify exploration as the total number of strategies explored to test these hypotheses. 2.4
3 Results Our control case was the basic IPD with masked uniform crossover and bit flipping mutation. Each chromosome contained a crossover mask, which was subject to only bit flipping mutation, and not λ mutation. The probability of mutation was 0.001. We also performed a sensitivity experiment with the probability of mutation at 0.01. The probability of crossover in all tests was 0.7. Our GA was the SimpleGA of GALib (Wall) with elitism. To determine which scaling and selection algorithms gave the best performance on the control problem, we compared several combinations of scaling and selection algorithms, namely linear and ST scaling, and tournament and SUS selection algorithms. We used a population of 20 individuals, and all calculations were performed for 50 generations for these tests. We repeated each run ten times. Table 3 shows the average IPD scores and their standard deviations from these series of calculations. The data show that ST outperforms linear scaling, and that SUS selection outperforms tournament selection on the IPD, but just barely. Because of these tests, we used ST scaling and SUS selection in all subsequent experiments. Scaling Linear Linear Sigma-truncation Sigma-truncation
Selection tournament stochastic uniform tournament stochastic uniform
Avg IPD score Std deviation 28.4 2.65 32.0 4.47 30.2 3.57 33.8 4.21
Table 3. Average and standard deviation of ten IPD calculations for various combinations of scaling and selection operators.
There is a basic but important difference between bit flipping mutation and λ mutation. The occurrence of bit flipping mutation is decided for each bit of each chromosome, whereas the occurrence of λ mutation is decided once per chromosome. Each chromosome is 70 strategy bits, 70 mask bits, and 10 bits for λ, for a total of 150 bits per chromosome. With 1000 individuals, for example, this produces 150,000 opportunities for mutation. With the typical mutation probability of 0.001, then, one expects approximately 150 bits to be flipped each generation. Although a single λ mutation has more effect on a chromosome than a single bit flipping mutation, a probability of λ mutation of 0.001 is far too low to be a significant test of λ mutation. Therefore, we performed experiments with the probability of λ mutation set to 0.01, 0.05 and 0.1 (but the probability of bit flipping mutation was 0.001). To ensure that the comparison to the control remained fair, we also ran the control with an increased mutation probability (0.01). As shown in Table 4, and as expected, the average IPD score (over ten trials) decreased when the mutation probability increased from 0.001 to 0.01. Thus, it is fair to compare calculations with higher λ mutation probabilities to the control case. Calculation Control case, mutation prob. 0.001 Control case, mutation prob. 0.01
Avg. IPD score Std. deviation 33.8 4.21 31.4 4.90
Table 4. Average and standard deviation of ten IPD calculations for the control case and increased mutation probability. We repeated all tests ten times. Summary results of these tests are in Table 5. The first two cases are the control case and the mutation rate sensitivity calculations. As mentioned before, the average IPD score decreased with increasing mutation rate. The column labeled “Average Exploration Rate” is the number of strategies generated divided by the number of individuals in the population, averaged over the ten calculations in a series. For the control case, increasing the mutation probability increased the exploration rate by more than a factor of three, yet the average IPD score decreased. This sensitivity clearly indicates that merely increasing exploration may not improve system performance, or that there is such a thing as too much exploration.
ID 1 2 3 4 5 6 7 8 9 10
Type of λ Mutation None None Full Full Convergent Convergent Even Even Chaotic Chaotic
Probability of Avg bit Exploration Max IPD λ mutation Avg IPD Std mutation Score deviation Rate Score 0.001 0 33.8 4.21 0.231 44 0.010 0 31.4 4.90 0.845 43 0.001 0.01 32.3 5.46 0.210 48 0.001 0.10 33.5 5.22 0.314 48 0.001 0.01 34.7 5.40 0.223 44 0.001 0.05 29.8 4.49 0.282 42 0.001 0.01 31.2 3.66 0.212 40 0.001 0.05 32.1 5.59 0.270 47 0.001 0.01 31.5 2.38 0.223 35 0.001 0.05 30.9 4.46 0.251 40
Table 5. Comparison of average, best-of-ten IPD scores, and exploration rate for different types and rates of mutation. Full λ mutation randomly selects λ from all domains, convergent and chaotic λ mutation selects from convergent and chaotic domains, and even λ mutation selects with equal probability from all three domains. The next two cases in Table 5 (IDs 3 and 4) employ λ mutation with random λ values in the initial populations. However, because the convergent λ values cover three quarters of the range of possible λ values, convergent λ values are likely to dominate in a random initial population. We include these results because the
greatest IPD score obtained in any of the tests occurred during these tests. However, because of the random, but biased, initialization, the only conclusion that we can draw from these tests is that some mixture of λ values in the population appears to be beneficial. In addition, increasing the rate of λ mutation slightly improved the average IPD score. Without a better handle on the initial λ distribution it is not possible to speculate on what meaning, if any, this increase has. Note, however that the average IPD scores for these test series fall between the average IPD scores of the control case and its sensitivity (IDs 1 and 2), even though the maximum score is higher for the λ mutation cases than for the control cases. Also, note that the increasing the λ mutation produced a moderate increase in the exploration rate, which may be related to the increased average IPD score. To get a better handle on the effect of λ mutation, we forced several known λ value distributions on the initial population: including all convergent values (IDs 5 and 6), an even distribution of λ values from all three regimes (IDs 7 and 8), and all chaotic values (IDs 9 and 10). All three series of tests were performed with the λ mutation probability at both 0.01 and 0.05. For all of these cases, an increase in the λ mutation probability is accompanied by an increase in the exploration rate, though a modest increase by comparison to the control cases. However, the average IPD scores for the convergent and chaotic test series decreased with increasing λ mutation rate, while the average IPD score for the even distribution case increased with the λ mutation rate. We observed this same trend with the random initialization cases. Also note that the highest maximum IPD score for these six tests occurred for one of the even distribution cases. Apparently, a mixture of initial λ values in the population improves the effectiveness of λ mutation, so that increasing the λ mutation rate increases the average performance on the IPD. There is a weak suggestion that λ mutation with a mixed initial population is beneficial for obtaining high scores on the IPD. As a more extensive comparison to the control case, we perform an additional series of fifty tests with greatly reduced data collection for the control case and the even λ distribution case. We used the mutation rates for which each case performed the best. The results are in Table 6. Both series obtained the same maximum IPD score of 48, equal to the highest in any of our runs. The even λ distribution case had the higher average IPD score. This likely indicates that average performance on the IPD is related to the ability of the algorithm, whereas the maximum performance is significantly influenced by chance. ID
Description
1 2
No λ mutation, mutation rate .001 Even λ mutation, mutation rate 0.05
Avg IPD Std Max IPD Score deviation Score 31.6 4.90 48 32.9 5.33 48
Table 6. Comparison of control case and even λ mutation average and maximum values over 50 tests. The win rates of different categories of λ values in the even distribution case affords yet another perspective on the effect of chaos in GAs. Even if a mixture of λ values in the initial population is beneficial, one category or another of λ values might tend to produce the highest scoring individual more frequently than the others do. Table 7 presents the fraction of maximum scoring individuals per λ category in the final generation for the series of fifty even λ distribution tests. It might have been expected that whatever value chaotic individuals have during a generation, they would tend to die off by the final generation. Yet Table 7 indicates that in 30% of fifty tests, the maximum scoring individual possessed a chaotic value of λ. If the IPD score of an individual were independent of the λ category it belonged to, then an even one third split among the categories would be expected. Table 7 indicates that it is definitely harmful for an individual to possess a bifurcating λ value, neutral to possess a chaotic value, and beneficial to possess a convergent value. Thus, while chaotic values may not always be the winners, they will remain in the population, and continue to provide diversity and promote exploration. In summary, the results indicate that the introduction of chaos into GAs can be beneficial, but that too much chaos, like too much mutation, is detrimental. Chaotic operations, like ordinary bit flipping mutation, appear to be good in small quantities. Further, λ mutation with a mixture of λ categories improved both the exploration rate and average IPD scores. Chaos may improve exploration, but must be used judiciously.
Description Even λ mutation, probability 0.05
Chaotic Bifurcating Convergent 30% 8% 62%
Table 7. Percentage of maximum scoring individuals from each λ category over fifty tests.
4 Conclusions We examined using chaos in simulated evolution to direct crossover for a dynamic, coevoltionary problem. A number of interesting trends are suggested by the data. Although increasing ordinary bit flipping mutation beyond a small degree greatly increases exploration, it also degrades system performance. Increasing λ mutation improves system performance by increasing exploration in a more focused, exploitive manner than bit flipping. In a sense, chaos directed mutation better than randomness. In a large number of tests, where the λ values were initially uniformly distributed, individuals with chaotic or convergent λ values achieved the maximum IPD score most of the time. This indicates that there is pressure to keep both of these components in the population. The chaotic values presumably aid exploration, and the convergent values aid exploitation. There must be benefit in mixing of chaotic and non-chaotic individuals, because experiments where the populations were initially all chaotic or all convergent performed poorly. It is perhaps the mixing of genetic material from exploring (chaotic) individuals and exploitive (convergent) individuals that gives λ mutation is greatest advantage. Individuals with bifurcating λ values rarely achieved the maximum IPD scores. Bifurcating mutation is neither exploration, nor exploitation, but merely cyclic repetition of previously used genetic material. Unfortunately, few of our conclusions are clearly statistically significant. We suspect that this is due in part to our representation of strategies, since the convergent regimes tended to concentrate changes in the low order bits of the strategy, though these hold no special significance for the IPD. It would also be interesting to isolate the effects of bit flipping mutation, λ mutation, and other aspects of the evolutionary process more thoroughly. Nonetheless, we have seen that a judicious use of chaos in simulated evolution can be beneficial. This also again raises the question, with which we began, of whether chaos might not play a role in natural evolution.
5 References Axelrod, R. 1987. “The Evolution of Strategies in the Iterated Prisoner’s Dilemma,” in L.D. Davis, ed., Genetic Algorithms and Simulated Annealing. Morgan Kaufmann. Gleick, James. 1988. Chaos: Making a New Science. New York: Viking. Goldberger, Ary L., Rigney, David R., West, Bruce J. 1990. “Chaos and Fractals in Human Physiology.” Scientific American 262 (February): 42 – 49. Moon, Francis C. 1992. Chaotic and Fractal Dynamics, an Introduction for Applied Scientists and Engineers. New York: John Wiley & Sons, Inc. Ohno, S. 1988. “Codon Sequence is But an Illusion Created by the Construction Principle of Codon Sequence.” Proceedings of the National Academy of Science USA 85: 4378-4386. Wall, M. 1996. GALib, A C++ Library of Genetic Algorithm Components, Version 2.4, Documentation Revision B, Massachusetts Institute of Technology (http://lancet.mit.edu/ga/). Wesson, Robert G. 1981. Beyond Natural Selection. Cambridge, Massachusetts: The MIT Press.
