Lecture Notes in Computer Science, vol. 866, Davidor, Y., Schwefel, H-P., Manner, R. (eds.), Springer Verlag.
Co-evolutionary Constraint Satisfaction Jan Paredis Research Institute for Knowledge Systems Postbus 463, NL-6200 AL Maastricht, The Netherlands e-mail:
[email protected]
Abstract. This paper introduces CCS, a Co-evolutionary approach to Constraint Satisfaction. Two types of objects - constraints and solutions - interact in a way modelled after predator and prey relations in nature. It is shown that co-evolution considerably focuses the genetic search. In addition, the new technique of life-time fitness evaluation (LTFE) is introduced. Its partial but continuous nature allows for efficient fitness evaluation. Moreover, co-evolution and LTFE nicely complement each other. Hence, their combined use further improves the performance of the evolutionary search. CCS also provides a new perspective on the problems associated with high degrees of epistasis and the use of penalty functions.
Keywords: co-evolution, constraint satisfaction, epistatic problems, genetic algorithms, life-time fitness evaluation, predator-prey systems.
1. Introduction Arms races constitute an important evolutionary driving force leading towards highly complex adaptations. These races occur, for example, in predator-prey relations. There is a strong evolutionary pressure for prey to defend themselves better (e.g. by running quicker, growing bigger shields, better camouflage ...) in response to which future generations of predators develop better attacking strategies (e.g. stronger claws, better eye-sight ...). In arms races, success on the one side is felt by the other side as a failure to which must be "responded" in order to maintain one's chances of survival. This, in turn, calls for a reaction of the other side. This inverse fitness interaction leads towards a stepwise increase in complexity of both: predator and prey. Hence, both populations are said to co-evolve. Recent research by Hillis (92) has shown the use of the principles behind arms races to boost the performance of evolutionary algorithms (EAs). Hillis used the design of sorting hardware as a test domain for his technique. Paredis (94a) has adapted this approach to evolve classification neural networks. It combines another biologically inspired technique, called life-time
fitness evaluation (LTFE), with co-evolution. Here we show that this combination can be used successfully to solve constraint satisfaction problems as well. Constraint Satisfaction Problems (CSPs) typically consist of a set of n variables xi (i ≤ n) which have an associated domain Di of possible values. There is also a set of constraints, called C, which describe relations between the values of the xi. (e.g. the value of x1 should be different from the value of x3). A solution consists of an assignment of values to the xi such that all constraints in C are satisfied, i.e. the solution is valid. Typically, highly constrained CSPs have a high degree of epistasis: the choices made during the search are closely coupled. In general, highly epistatic problems are characterised by the fact that no decomposition in independent subproblems is possible. In that case, it is difficult to combine subparts of two valid solutions into another valid solution. This is bad news for GAs which typically search by combining features of different "solutions". The last couple of years a number of methods for constraint handling have been proposed within the GA community. The first one - genetic repair (Mühlenbein 92) - removes constraint violations in invalid solutions generated by the genetic operators (such as mutation and crossover). The second one uses decoders such that all possible representations give rise to a valid solution (Davis 88). A third one uses penalty functions, see for example (Richardson et al 89). This approach defines the fitness function as the objective function one tries to optimize minus the penalty function which represents the "degree of invalidity" of a solution. Hence, the search process is allowed to wander around in portions of the search space containing only invalid solutions. The rationale being that it might be easier to find a solution if one is allowed to reach it from both sides of the valid/invalid border. All three approaches mentioned above, have a major disadvantage: they are all problem specific. For every CSP one has to determine a good decoder, a good genetic repair method or a penalty function which balances between convergence towards suboptimal valid solutions (when the penalty function is too harsh) or towards invalid solutions (too tolerant a penalty function is used). Recently, Paredis (93) proposed GSSS, a general method for introducing domain knowledge - through the use of constraint programming - to guide the evolutionary search when solving constraint problems. Now, an alternative approach - called Co-evolutionary Constraint Satisfaction (CCS) - is presented. Both CCS and GSSS are specifically designed to solve constraint problems. The explicit focus on this class of problems brings several advantages with it. A first one is that it allows for a natural problem specification in terms of constraints. Moreover, these constraints explicitly represent the epistatic interactions. This information can be used to guide the evolutionary search. Paredis (93) also shows the use of the constraints as domain elements in a general, abstract, genetic repair method.
In contrast with GSSS, CCS only checks whether a proposed solution satisfies the constraints. No active knowledge on how to enforce the constraints is required. Hence CCS relies on less domain knowledge than GSSS. Or in other words, GSSS is more knowledge-rich than CCS. Hence, the choice between both approaches depends on issues such as the availability of knowledge and its quality (i.e. do the constraints focus the search enough to overcome the computational overhead created by the use of the additional knowledge?). The structure of this paper is as follows. The next section describes the bench-mark problem on which CCS will be tested. Section three describes the algorithm. Empirical results and comparison with other algorithms are given in section four. The paper closes with conclusions and a discussion.
2. The Test Problem In this section, we describe the bench-mark problem on which our approach will be tested empirically. The n-queens problem consists of placing n queens on a n×n chess board so that no two queens attack each other (i.e. they are not in the same row, column, or diagonal). A frequently used representation of this problem - which we use as well - consists of n variables xi. Each such variable represents one column on the chess board. The assignment x2=3, for example, indicates that a queen is positioned in the third row of the second column. Hence, the xi take a value from the set {1,2, ...,n}. The constraints for this problem are simple: • xi ≠ xj • |xi - xj| ≠ |i-j|
i≠j i≠j
; row-constraint ; diagonal-constraint
The first line above prohibits two queens to be placed in the same row. The second line ensures that no two queens are on a same diagonal. Note that the column constraint (only one queen is allowed per column) is implicit in the representation. Here, we test our approach on a 50-queens CSP. Hence, a solution consists of 50 variables. Each of which has to be assigned a value from the set {1,2, ..., 50}.
3. The Algorithm The algorithm involves two populations. The first one contains the constraints. The second population contains potential - possibly invalid - solutions to the CSP to be solved. Just as in predator-prey models the selection pressure on members of one population depends on the fitness of the mem-
bers of the other population. Or, in other words, there is an inverse fitness interaction between both populations. Figure 1 depicts the data structures used in CCS. The "genetic representation" of both types of individuals is quite simple. The solutions are represented as 50-dimensional vectors containing integers from the set {1,2,3 ..., 50}. A constraint, on the other hand, is an à priori defined piece of LISP-code which checks whether it is violated by a given solution. The constraints can be taken directly from the problem specification. For each xi two constraints are used. A first one checks whether any of the other columns have a queen positioned at the same row as xi. The second one checks the diagonal constraint. Or, in a more formal way: for each i ∈ {1,2,3 ..., 50} the following two constraints are present in the constraint population: ∀j ∈ {1,2,3 ..., 50}: i=j or xi≠xj ∀j ∈ {1,2,3 ..., 50}: i=j or abs(xi-xj)≠abs(i-j). As we discuss in more detail below, the constraint population contains all the time the same hundred constraints. This in contrast to the population of solutions in which new solutions - created through reproduction - gradually replace older ones. Hence, the constraints do not form a real population in a strict sense.
SOL-pop
f i t n e s s e s
CONSTRpop
f i t n e s s e s
Figure 1: Co-evolving solutions and constraints
CCS uses GENITOR (Whitley 1989a; 1989b) as its core evolutionary algorithm. An important novelty, however, is the notion of an encounter between a solution and a constraint. During such an encounter the solution is checked with respect to the constraint it encounters. The solution receives a reward when it satisfies the constraint, it receives a penalty in case the constraint is violated. For the constraint involved in an encounter the result of an encounter is opposite: it gets a penalty when the solution encountered satisfies
it and a reward in case of violation. In our implementation rewards have the value of 1 and penalties -1. Both can be interpreted as the pay-off resulting from an encounter. Each individual - whether constraint or solution - has an associated history containing the rewards and penalties it received during its most recent encounters. Here we use a history of length 25. Hence, the fitness of an individual is defined as the sum of the pay-off it received during the 25 most recent encounters it was involved in. This is a clear example of the use of life-time fitness evaluation (LTFE). A continuous series of tests during the life-time is used to approximate the fitness of an individual. This in contrast with the traditional "once and for all" fitness calculation. The limited length of the histories ensures that only the most recent encounters are taken into account. That is why LTFE is said to provide a partial fitness measure. The pseudo-code below describes the basic cycle of the algorithm. First 20 - this number is rather arbitrarily chosen - constraint-solution encounters are executed. The function which SELECTs the individuals to be involved in an encounter is biased towards highly ranking individuals: the most fit individual is 1.5 times more likely to be selected than the individual with median fitness. This way, fit solutions and constraints are often involved in an encounter. Or in other words, good solutions have to prove themselves more often. At the same time, the algorithm concentrates on satisfying difficult, i.e. not yet solved, constraints. This is because constraints have a high fitness when they are violated by many members of the solution population. The function ENCOUNTER returns 1 if the solution satisfies the constraint. In the other case, -1 is returned. Next, the histories of the solution and constraint involved in the encounter are UPDATEd. This is done by pushing the pay-off associated with the current encounter on the history of the individuals. At the same time the pay-off associated with the least recent encounter is removed from these histories. Finally, the fitness of an individual - which is defined as the sum of rewards in its history - is updated. As the populations are continually kept sorted on fitness, this change of fitness might move the individual up or down in its population.
DO 20 TIMES sol:= SELECT(sol-pop) constr:= SELECT(constr-pop) res:= ENCOUNTER(sol,constr) UPDATE-HISTORY-AND-FITNESS(sol,res) UPDATE-HISTORY-AND-FITNESS(constr,-res) sol1:= SELECT(sol-pop) ; parent1 sol2:= SELECT(sol-pop) ; parent2 child:= MUTATE-CROSSOVER(sol1,sol2) f:= FITNESS(child) INSERT(child,f,sol-pop)
After the execution of the 20 encounters one new solution is created using one-at-a-time reproduction (cf. GENITOR): 1) two parent solutions are SELECTed. This selection is - again - biased towards fitter individuals. 2) a new solution is generated from these parents through the application of MUTATION (which replaces a value by a randomly drawn member from the set {1,2, ..., 50}) and CROSSOVER (two-point crossover is used here), 3) the initial FITNESS of this child is calculated by having it to encounter 25 SELECTed constraints. The rewards associated with these encounters are pushed on the history. Again, the fitness of the newly created solution is the sum of these rewards, 4) if this fitness is higher than the minimum fitness in the population then the child is INSERTed into the appropriate rank location in the NN population. At the same time, the individual with the minimum fitness is removed. As mentioned above the members of the constraint population do not change. Hence, no selection and reproduction operates on the constraint population. Just as in (Hillis 92), the fitness interaction between solutions and constraints keeps both populations in constant flux. As soon as the solutions are becoming successful on a certain set of constraints then these constraints get a lower fitness. As a consequence, other, not yet satisfied constraints, move up in the population and will be selected more often. This forces the solutions to concentrate on these "more difficult" constraints (because of the fitness proportional selection of the individuals involved in an encounter). In our test problem, for example, the diagonal-constraints rapidly move up in the population. This is because they check both diagonals. Hence, for every occupied board location, they constrain 2 × 49 other locations. The row-constraints, on the other hand, only constrain the 49 board locations in one row. Let us now look at the benefits gained from the use of LTFE. Its continuous partial nature allows for an early detection of good and bad solutions: the solutions need not evaluate all 100 constraints at once. A few, well chosen, i.e. highly informative, constraint-checks provide sufficient information. This is true during different stages of the genetic search. Initially, all solutions are rather bad. In this case, one does not need extensive testing. A rough indication of their quality is sufficient. Even if - due to an unlucky choice of constraints in its initial encounters - the fitness is rated a bit too low then future encounters will certainly raise this fitness. At later stages - during which more mutation occurs1 - LTFE is advantageous as well: it immediately weeds out clearly inferior offspring at minimal computational expense. When almost all solutions have a high fitness, e.g. they satisfy more than 80% of the constraints, then clearly inferior solutions can be spotted quite easily. Here too, one does not need to check all constraints. In addition, the biased selection of constraints allows to focus on the most relevant ones. In
1 The probability of mutation is linearly proportional to the "genetic similarity" of the parents. This is known as adaptive mutation [Whitley 1989b].
this way, LTFE and the negative fitness interaction of the co-evolution nicely complement each other. Obviously the fitness of an individual - based on 25 encounters - is only approximate. The continuous life-time fitness feedback together with the limited history length limits the effect of unrepresentative pay-off. Moreover, the same mechanisms allow the individuals to keep up with the changing individuals or rankings in the other population. This is the role of the 20 encounters executed in the DO-loop in the pseudo-code given above. As the empirical results of the next section show, the combined use of coevolution and LTFE considerably focuses the genetic search.
4. Empirical Results In this section we compare CCS with two other genetic approaches. The first one, called TRAD, is a traditional GENITOR-like algorithm. This algorithm uses only a population of solutions. The fitness of a solution is defined as the number of constraints (out of 100) it satisfies. Hence, TRAD uses the traditional once-and-for-all fitness evaluation (instead of LTFE) and no co-evolution. The code of the basic cycle of TRAD is given below (compare this with the code in section 3) sol1:= SELECT(sol-pop) ; parent1 sol2:= SELECT(sol-pop) ; parent2 child:= MUTATE-CROSSOVER(sol1,sol2) f:= FITNESS(child) INSERT(child,f,sol-pop)
The second one, called CCS-, is identical to CCS except for the population structure of the constraints. In CCS- the constraints do not have any fitness. Hence, the selection of constraints involved in an encounter is not biased towards "difficult" constraints. Or, in other words, the constraints form a set instead of a sorted "population". The main difference between CCS- and TRAD is that the former uses LTFE. CCS-, however, does not use co-evolution. In this way an empirical comparison of these three systems can point out the relative merits of LTFE, co-evolution, and their combination. All three approaches were run 10 times. Each run used a population size of 250. A run is stopped after the creation of 200000 offspring. In terms of solution quality both, CCS- and TRAD, are poor performers. On the 10 runs, TRAD never found a global solution satisfying all constraints. CCSfound such a solution only once. CCS, on the other hand, was seven out of ten times successful. And this after having created, on average, 125000 offspring which is significantly less than the maximally allowed 200000. In the three other runs CCS found solutions which satisfy 98 out of the 100 constraints. This is also consistently higher than the quality obtained by TRAD and CCS-. Even after the generation of 200000 offspring TRAD and CCS-, on
average, only found solutions satisfying 95.7 and 97.2 constraints, respectively. For CCS this average is 99.4. This comparison between TRAD, CCS-, and CCS gives a good indication of the contribution of the successive introduction of LTFE and co-evolution.
5. Conclusions and Discussion To our knowledge, CCS is the first system using co-evolution for solving constraint satisfaction problems. Earlier work (Paredis 94a) showed that the integrated use of LTFE and co-evolution clearly boosts the performance of genetic neural net search. Here, we have illustrated the power of the combination of these same techniques in the completely different field of constraint satisfaction. The empirical results presented in both papers confirm the potential of the approach. The observed performance increase can be attributed to two factors. Primo, the partial nature of life-time fitness evaluation allows for early detection of particularly good or bad solutions. Secundo, co-evolution concentrates its effort on the constraints which are most relevant at a given moment of the search. Furthermore, the combination of LTFE and coevolution has a clearly beneficial effect. The partial and continuous nature of LTFE - which only takes into account the most recent encounters - is well suited when dealing with coupled fitness landscapes. Here the fitnesses of solutions and constraints are coupled because changes in one population (through a different ranking or through the introduction of new individuals) clearly affect the fitness of the members of the other population. For this reason only the pay-off resulting from the most recent encounters provides an up-to-date approximation of the fitness of an individual. Moreover the use of LTFE makes the steady-state evolutionary algorithm more robust and adaptive. This is a consequence of the relation between LTFE, noise and steadystate reproduction as discussed in more detail in (Paredis 94a). CCS also provides an interesting approach towards some wellknown research issues of EAs. The first one relates to the notion of building blocks. Holland showed that for problems with a low degree of epistasis, operators that splice together parts of two different individuals might yield good solutions. For such problems, sets of functionally dependent genes are relatively small. In that case, it becomes possible to use a string representation in which the "correlated" genes are placed near to each other. Once the EA finds good values for these genes, they are unlikely to be split apart during sexual reproduction. Analogously, independent genes should be located far apart from each other on the string. Otherwise there will be too little exploration: suboptimal values for these genes are unlikely to get disrupted. This last case typically results in premature convergence. In general, a linear string might not be sufficient to place interacting bits near to each other, and to place non-interacting bits far apart. This is particularly the case when ad-
dressing problems with a high degree of epistasis, such as tightly constrained CSPs. In CCS each constraint checks the consistency between a number of "genes" (here constraint variables). Good values for such a set of correlated genes should then be combined in order to obtain a global solution. As is the case in other approaches, crossover tends to disrupt genes which are located far apart from each other. But on the other hand, the constraints checking such "difficult genes" will rapidly become fit. In this way, more search effort will be devoted to these genes. This is particularly relevant for problems consisting of too many dependencies to be packed on a linear string. The nqueens problem used here is an example of this: each variable is related to each other variable. The uniformity of the constraints does not allow to say which variables are most tightly coupled. As a matter of fact, the degree of interaction between two variables is not constant over the search space. It strongly depends on the assignment of the other variables. LTFE together with co-evolution enables CCS to concentrate on the relevant difficult (sets of) genes. A second important issue is the relation between CCS and approaches using penalty functions. As discussed in the introduction of this paper, the success of these approaches critically depends on the choice of a good penalty function. In CCS, on the other hand, the co-evolution gives rise to an "auto-adjusting" penalty function. This is because the fitness evaluation is a kind of a penalty function, i.e. it counts the number of constraint violations. The changing ranking of the constraints and the biased selection of constraints on which the solutions are tested, allows CCS to focus on the more difficult - not yet satisfied - constraints. Or, in other words, the penalty function becomes more and more harsh as better solutions are found. Informally, one could say that the penalty function co-evolves in step with the quality of the solution population. Various researchers already observed that gradually adding constraints during the evolutionary search improves the results obtained. CCS automatically performs such a gradual tightening of the problem. The research reported here can be extended in multiple directions. A first extension is to tackle constrained optimization problems. In these problems one has to find a valid solution to a CSP which at the same time maximizes a given objective function. Constraint relaxation is another interesting line of future research. The goal is then to delete constraints from an overconstrained problem such that the problem is no longer over-constrained, i.e. a valid solution exists. Two observations of CCS's mode of operation are relevant here. Primo, the ranking of the constraints during the search reflects the difficulty of a constraint. This information may be used to determine which constraints to delete or relax. Secundo, each individual in the solution population is a valid solution with respect to some less constrained version of the problem to be solved.
Further research will concentrate on the application of the method proposed here to other tasks. Here we addressed constraint satisfaction problems. Paredis (94a) applies LTFE and co-evolution on a classification task using neural nets. Path finding is another application area currently under investigation. Undoubtedly, many other tasks are still waiting for exploration. In addition to the application to other tasks the current framework can be extended in yet another way. All examples discussed above have used predator-prey relations to improve the power of artificial search. Obviously, many other mechanisms - not necessarily based on inverse fitness interaction - exist in nature. Symbiosis is such an important and widely occurring counter example. It consists of a positive fitness feedback in which a success on one side improves the chances of survival of the other. As far as we know, the use of symbiosis to enhance the power of evolutionary search is as yet unexplored. Paredis (94b) describes a first investigation into the symbiotic evolution of solutions and their genetic representation (i.e. the ordering of the genes). A representation adapted to the solutions currently in the population speeds up the search for even better solutions which in their turn might progress optimally when yet another representation is used. Finally, we hope that this paper may prove instrumental in further stimulating research in the computational use of co-evolution and its combination with LTFE.
Acknowledgements The author is indebted to Desiree Baaten for proofreading this paper.
References Davis, L., (1988), Applying Adaptive Algorithms to Epistatic Domains, Proc. IJCAI-88. Hillis, W.D., (1992), Co-evolving Parasites Improve Simulated Evolution as an Optimization Procedure, in Artificial Life II, Langton, C.G.; Taylor, C.; Farmer, J.D., and Rasmussen, S., (eds), Addison-Wesley, California. Mühlenbein, H., (1992), Parallel Genetic Algorithms in Combinatorial Optimization, Proc. Computer Science and Operations Research: New Developments in Their Interfaces, ORSA, Pergamon Press. Paredis, J., (1993), Genetic State-Space Search for Constrained Optimization Problems, Proc. Thirteenth International Joint Conference on Artificial Intelligence (IJCAI 93), Morgan Kaufmann Publishers. Paredis, J., (1994a), Steps towards Co-evolutionary Classification Neural Networks, Proc. Artificial Life IV, R. Brooks, P. Maes (eds), MIT Press / Bradford Books. Paredis, J., (1994b), The Symbiotic Evolution of Solutions and their Representation, (in preparation).
Richardson, J.T. ; Palmer, M.R. ; Liepins, G.; Hilliard M., (1989), Some Guidelines for Genetic Algorithms with Penalty Functions. Proc. Third Int. Conf. on Genetic Algorithms, Morgan Kaufmann. Whitley, D., (1989a), The Genitor Algorithm and Selection Pressure: Why RankBased Allocation of Reproductive Trails is Best. Proc. Third Int. Conf. on Genetic Algorithms, Morgan Kaufmann. Whitley, D., (1989b), Optimizing Neural Networks using Faster, more Accurate Genetic Search. Proc. Third Int. Conf. on Genetic Algorithms, Morgan Kaufmann.
