Solving dynamic constrained optimisation problems ...

3 downloads 2947 Views 2MB Size Report
of CH and DO strategies such as stochastic-ranking, repair, ..... the search and drive more individuals away from the good regions ...... G24 6a (2DR, hard). 17.
1

Solving dynamic constrained optimisation problems using stochastic ranking and repair methods Trung Thanh Nguyen, Member, IEEE and Xin Yao, Fellow, IEEE

Abstract—It has been shown that (i) dynamic constrained optimisation problems (DCOPs)have some special characteristics that make them very different from other types of problems and (ii) existing dynamic optimisation (DO) and constraint handling (CH) algorithms might not work effectively in solving DCOPs. This creates an important gap in current research about DO. In this paper, we propose a set of new mechanisms and algorithms to effectively solve DCOPs. The goal is to combine the advantages of CH and DO strategies such as stochastic-ranking, repair, random-immigrant and hyper-mutation, while overcoming the drawbacks of these methods in solving DCOPs. Test results confirm the advantages of the newly proposed mechanisms and algorithms. Not only do they overcome all existing drawbacks and hence perform significantly better than the tested existing state-of-the-art algorithms in solving DCOPs, they also perform equally to or better than these existing DO and CH algorithms in other groups of tested problems except in static problems. We also (i) propose new measures and analyse how/why the newly mechanisms/algorithms work better in DCOPs, (ii) investigate their individual contribution and (iii) study the influence of different parameter values on algorithm performance in solving DCOPs. These analyses reveal some interesting and counter-intuitive findings about the characteristics of DCOPs and the way we can solve DCOPs. Index Terms—Dynamic optimisation, dynamic environments, dynamic constraints, constraint handling, benchmark problems, evolutionary algorithms, performance measures.

I. I NTRODUCTION There have been very few studies on continuous dynamic constrained optimisation (DCO) despite their high popularity in real-world aplications (see Subsection II-A). Dynamic constrained optimisation problems (DCOPs) also have some special characteristics that make them very different from unconstrained dynamic problems and stationary constrained problems (see Subsection II-A). Due to these different characteristics, some existing dynamic optimisation (DO) and constraint handling (CH) algorithms might not work effectively in solving DCOPs. The lack of knowledge about DCOPs, the ineffectiveness of existing algorithms in solving continuous DCOPs, and the lack of algorithms specifically designed for solving continuous DCOPs creates an important gap in current dynamic optimisation research. Manuscript received 17-June-2010. This work was supported by an UK ORS Award and a School of Computer Science studentship and was partially supported by an EPSRC grant (EP/E058884/1) on ”Evolutionary Algorithms for Dynamic Optimisation Problems: Design, Analysis & Applications”. T.T. Nguyen is with The School of Engineering, Liverpool John Moores University, L3 3AF, United Kingdom; email: [email protected]. X. Yao is with The Centre of Excellence for Research in Computational Intelligence and Applications,(CERCIA), School of Computer Science,University of Birmingham, B15 2TT, United Kingdom; email: [email protected].

This paper contributes to the task of closing this research gap by developing new methods to solve DCOPs more effectively. Based on detailed studies of the characteristics of realworld DCOPs and the weaknesses of some existing algorithms in solving DCOPs (Section II), we modify two existing CH techniques, the stochastic ranking method [1], [2] and the repair method [3] [4], to create two new frameworks with special mechanisms to support solving DCOPs, and then integrated two DO techniques, random-immigrant [5] and hyper-mutation [6], into the frameworks to develop new algorithms able to solve DCOPs better than the original DO and CH methods (Section III). We will also undertake two detailed analyses to (a) study the behaviours and performance of some existing DO and CH algorithms as well as the newly proposed algorithms in solving DCOPs (Section IV); and (b) study which factors have made the proposed algorithms work well (Section V). Another analysis (Section VI) will be carried out to investigate more about the characteristics of DCOPs as well as the influence of each algorithmic component on algorithm performance in DCOPs. Based on this observation we will suggest some recommendations on choosing the suitable parameter values. Some new measures will be developed to assist these two aforementioned analyses. Finally, in Section VII, we will discuss the advantages and disadvantages of the proposed methods and outline future directions. II. BACKGROUND A. Characteristics of real-world DCOPs Many real-world problems are DCOPs. A comprehensive review of 56 real-world dynamic applications has shown that a majority of the applications are DCOPs (73% in the combinatorial and 74% in the continuous domain - see Chapter 3, Section 5.1 in Chapter 5 and the Appendix (Tables 1-5) in [7]). Some examples of real-world DCOPs can be found in [8], [9], [10], [11], [12], [13]. Despite this popularity of DCOPs, there is little research specifically on answering the questions of what are the special characteristics of DCOPs and how different these characteristics are from unconstrained dynamic problems. Answering these questions, however, is fundamental because only by learning the characteristics of DCOPs, can we develop algorithms to solve the problems effectively. DCOPs may have the following characteristics, which make them very different from unconstrained DOPs: • •

The objective function, constraints or both can change The dynamic of constraints can lead to changes in the shape/percentage/structure of the feasible/infeasible areas

2







The dynamic in objective function might cause the global optima to switch from one disconnected feasible region to another in problems with disconnected feasible regions. In problems with fixed objective functions and dynamic constraints, moving infeasible areas might expose new, better global optima without changing existing optima. DCOPs also have some common characteristics of constrained problems such as global optima in boundaries of feasible regions, global optima in boundaries of the search region, and multiple disconnected feasible regions.

B. Difficulties of existing DO algorithms in solving DCOPs A previous study [14] revealed that existing DO strategies might suffer from some of the following four types of difficulties in solving DCOPs. First, the presence of infeasible areas might make diversity-introducing/ diversity-maintaining strategies1 such as Random-Immigrant (RIGA, [5]) and HyperMutation (HyperM, [6]) less effective, especially when they are used with penalty functions because diversified infeasible individuals are likely be rejected by penalty methods and consequently no longer serve the purpose of maintaining diversity. Second, the presence of switching optima (between disconnected regions) might have a negative impact on the performance of DO strategies if they are combined with penalty functions because penalty methods make it difficul to create an infeasible path between disconnected regions. Third, the presence of moving infeasible areas might make tracking-previous-optima strategies such as HyperM less effective. Fourth, the use of elitism has a positive impact on the performance of existing diversity-maintaining strategies like RIGA. C. Difficulties of existing CH algorithms in solving DCOPs It was suggested that, when solving DCOPs, besides difficulties of handling dynamics, some of existing representative CH strategies may also suffer from additional difficulties in handling constraints [14]. These difficulties are caused by that the knowledge CH strategies have about the problem and even the CH strategies themselves might become outdated due to environmental changes. Experimental results in the aforementioned study show that these difficulties significantly affect the performance of CH algorithms like the tested repair method and penalty method in solving DCOPs. In this paper, we will show that these difficulties also significantly affect the performance of current state-of-the-art CH algorithms such as stochastic ranking ES (SRES, [1], [2]) and simple multi-membered ES (SMES, [16]). Details will be shown in Subsection IV-E. D. Possible requirements for DO and CH algorithms to solve DCOPs effectively Based on the experimental results, Nguyen and Yao [14] suggested some possible requirements for DO and CH strategies to solve DCOPs effectively as follows: 1 As

categorised by Jin and Branke in [15]

1) Diversity maintaining should be used with elitism 2) Detect changes/search in both feasible/infeasible regions 3) Track the moving feasible regions instead of tracking the moving existing optima. 4) The CH strategy should : a) Allow diversified individuals to be distributed in the whole search space. b) Not reject diversified individuals even if they do not contribute to the CH process. c) Distinguish between performance drops caused by the constraint handling process and those caused by environmental changes . 5) The CH strategy needs to get updated per change E. Related research in the continuous domain Only until vere recently a few algorithms specially designed for DCOPs were proposed. The first attempt was made in [17], to develop a GA-based algorithm named RepairGA to solve DCOPs. The algorithm is based on the repair method [3]. It is a combination of the repair-based GA+Repair method [7] and a simple mechanism to update reference individuals at every generation. Details of this algorithm (i.e. the combination of GA+Repair and the update mechanism) will be described in Subsection III-B. Experimental results in [17] show that RepairGA performs better than existing DO algorithms such as GA/RIGA/HyperM in the four tested problems. However, the method has not been able to satisfy all of the requirements mentioned in the above section because it lacks the following mechanisms to handle DCOPs: (1) an explicit mechanism to adaptively detect changes; (2) a mechanism to update outdated search individuals; and (3) a mechanism to maintain/introduce diversity in dynamic environments. Details of these drawbacks will be discussed in Section III-B4. Experimental results in [14] (and also in Section V) confirm the drawbacks and show that the method become less effective in solving DCOPs, compared to the static constrained cases. Using two of the benchmark problems proposed in [17], in [18] a static constraint optimisation algorithm (IDEA) was evaluated on DCOPs. IDEA is an EA that uses a special ranking mechanism to select individuals: the algorithm explicitly maintains some infeasible individuals during the search and ranks “good’ infeasible solutions higher than feasible solutions. This algorithm, however, was not actually designed for solving dynamic environments but static environments. In [18], to make it work on DCOPs IDEA was modified by adding a simple change-detection method in which at each generation a random individual is chosen to detect changes. Whenever a change is detected, the whole population is reevaluated to make sure that the information that the algorithm has is updated. This simple change-detection method does not guarantee that changes are always detectable because it assumes that changes can be detectable by re-evaluating any random individual in the search space, which is not always the case. Experimental results show that IDEA perform betters than the chosen peer EA in the two tested problems. Similar to the case of RepairGA above, the IDEA algorithm also does not satisfy all the necessary requirements to solve DCOPs because:

3

(1) it does not use any DO techniques to handle environmental dynamics; (2) its change-detection is simple and possibly insufficient in certain cases. Of these two main drawbacks, the lack of a DO technique is a significant disadvantage. As will be shown later in Section IV, without any DO technique, even current state-of-the-art methods in CH such as SMES [16] and SRES[2] become siginifcant less effective in solving DCOPs compared to the static cases. The latest algorithm proposed specifically to solve DCOPs is [19] which adapted abstract memory for DCOPs. Abstract memory is a special scheme which relies on a probabilistic model of the occurrence of good solutions in the search space to memorise a ”spatio–temporal cartographing” [20] of promising regions. In [19], this scheme was adapted for DCOPs by separately memorising the good candidates for solving the unconstrained objective function as well as the likely feasible regions. Elements from the two memories then are processed in two ways: blending and censoring, and were applied to EAs with penalty function and repair method. The methods were then tested in a special DCOP where the constraints and objective function might change at different periods [21]. The methods, however, were not designed to satisfy all the requirements for solving DCOPs. Particularly, to detect changes, they only use the HyperM’s fitness-drop monitoring mechanism, which might not be effective in the case of newly-appearing-optima DCOPs. Experimental results in [17] and [7] have already demonstrated that using only HyperM to detect changes does not guarantee that constraint changes are detected. In addition, because the proposed method only increases diversity when changes are detected, if the algorithm is not able to detect a change, it might lose diversity and consequently becomes less effective in DCOPs. In summary, it has been shown that none of existing methods is able to satisfy all the requirements set out in II-D. III. A

NEW CLASS OF ALGORITHMS TO SOLVE

DCOP S

As DCOPs have the properties of both DO problems and constrained problems, it is natural that in order to solve DCOPs, an algorithm needs to use both DO and CH strategies. These strategies need to be modified to satisfy the special requirements outlined in Subsection II-D. In this section, the possibility of modifying and combining DO and CH strategies will be discussed, and new algorithms following the idea will be introduced. A. Choosing DO and CH strategies Some representative DO and CH methods were chosen. The DO techniques are the classical triggered hyper-mutation GA (HyperM [6]) and random-immigrant GA (RIGA [5]) and the CH techniques are the canonical repair method [3] and the current state-of-the-art stochastic ranking [1], [2]. These were chosen because they are effective, easy to implement/evaluate/compare/analyse and more importantly because they represent commonly used strategies in many current stateof-the-art algorithms in DO and CH. HyperM is a simple GA with an adaptive mechanism to switch from a low mutation rate (standard-mutation-rate) to a

high mutation rate (hyper-mutation-rate) to increase diversity and vice versa depending on whether there is a degradation of the best solution in the population or not. It represents the ”introducing diversity” and ”tracking previous optima” strategies in DO. RIGA is another derivative of basic GA. After the normal mutation step, a fraction of the population (determined by the random-immigrant rate) is replaced by randomly generated individuals. By continuously introducing random solutions, the algorithm is able to maintain diversity throughout the search to cope with dynamics. RIGA represents the ”maintaining diversity” strategy in DO. The repair method in [3] represents the repair strategies in CH. The idea is that, if it is possible to map (repair) an infeasible solution to a feasible solution, then instead of searching the best feasible solution directly, it might be possible to look for an individual that can potentially produce the best repaired solution. Stochastic ranking [1], [2] represents the rank-based selection strategies in CH. The idea is to accept not only feasible solutions but also infeasible solutions with a certain probability Pf . Specifically, when comparing two individuals, if at least one of them is infeasible, the probability of comparing them according to their objective function regardless of their feasibility is Pf . In this paper, the repair method and stochastic ranking method will be modified to create a framework which supports DO strategies and then existing DO strategies will be integrated to the framework. As the repair method and stochastic ranking method are the foundations for our new algorithm, in the next subsection the two methods, plus their strengths and weaknesses in solving DCOPs, will be discussed in details. B. The repair method and GA+Repair 1) General idea: Repair approaches follow three steps: 1) For each new individual s, use a heuristic repair () to repair s, mapping s to a new, feasible individual z. 2) The objective value f (z) is then used to calculate the fitness value eval (s) of s, i.e. eval (s) = h (f (z)) where h is the mapping: objective values -¿ fitness values. 3) If the repair approach is Lamarckian, replace one or more search individuals with z The repair method used in this paper was proposed in [3] and was integrated as a part of the Genocop III algorithm [4] for the continuous domain. This method also follows the general three steps given above, in which the repair () heuristic can be described as follows: 1) The population is divided into two sub-populations: a search population S containing normally-evolving individuals (can be fully feasible or not), and a reference population R containing only fully feasible individuals. 2) During the search, while each individual r in R is evaluated using their objective function, each individual s in S is considered to be repaired based on an individual from R. Detail of the routine is in Algorithm 1. It is important to note that there are two possible variants of deciding whether a search individual s needs to be repaired in Genocop III (step 2 above). In the first variant [3], a search individual s is repaired only if s is infeasible. In the second

4

and latest variant [4], the implementation shows that search individuals are repaired regardless of their feasibility. In all experiments in this paper, the second variant, i.e., to repair search individuals regardless of their feasibility, was implemented. From now on, in this paper unless stated otherwise the term repair method will be used to refer to the continuousbased repair approach proposed in [4]. Algorithm 1 routine Repair(Indiv s) 1) Randomly pick an individual r ∈ R 2) Generate individual z in the segment between s and r a) a = U (0, 1) ;z= a.s+ (1 − a) .r b) While z is infeasible, return to step 2a c) If feasible z is not found after 100 trials, z = r; eval (z) = eval (r) 3) a) Evaluate z b) If (f (z) better than f (r)): r = z; eval (r) = f (z) c) Update the fitness value of s: eval (s) = f (z) 4) Return the individual s

Algorithm 2 GA+Repair 1) Initialise: a) Randomly initialise mS individuals in S. b) Randomly initialise mR feasible indivs in R. 2) Search: For i = 1 : mS a) Use nonlinear-ranking selection to choose an indiv s b) If probabilities allow, crossover and mutate s c) Evaluate s and repair s using the routine Repair (s) d) Use nonlinear ranking selection to replace one of the worst individuals in S by s 3) Evolve the reference population after each 100 evaluations: For i = 1 : mR a) Use nonlinear-ranking selection to choose an indiv r b) If probabilities allow, crossover and mutate r c) If new individual better than parent, replace the parent 4) Return to step 2 2) The GA+Repair algorithm: One good characteristic of the repair method is that, the method can be integrated with many different continuous EAs. To make it easier to analyse the effect of the repair method, besides testing Genocop III, in [14] the repair method was hybridised with basic GA to create an algorithm named GA+Repair, in which the only difference between GA+Repair and basic GA is the repair method and its repair procedures. The brief prosecode of GA+Repair can be found in Algorithm 2 and its details can be found in [14]. This algorithm will be used as a foundation to develop one of our new algorithms to solve DCOPs. 3) Problem knowledge and the balancing strategy in the repair method: The repair method relies on the information (which is stored in S and R) that it achieves during the search to give a balance between exploiting feasible regions and exploring infeasible regions. R provides information about particular feasible regions that are promising, and S provides

information indicating which points in the landscape would lead to these promising regions (via the repair process). Given the information above, the balance of feasibility/infeasibility is achieved by two procedures. First, both infeasible and feasible individuals are accepted provided that they can produce good repaired solutions. This is accomplished by updating the fitness value eval (s) of each search individual s with the objective value f (z) of the repaired solution z i.e. eval (s) = h (f (z)). Second, by updating the fitness values of s like this, the repair method ensures that while infeasible solutions are accepted, they cannot have better fitness values than the best feasible solution available. 4) The impact of outdated information and outdated strategy on the repair method : When solving DCOPs the repair method significantly suffers from being outdated. First, the reference individuals, which are used to repair all other search individuals, might become outdated when a change occurs: they might no longer remain the best in the population or might even become infeasible and hence might wrongly bias the search and drive more individuals away from the good regions, making the search process less effective. Evidence for this drawback can be seen in [14, Fig 5] where it shows that after a change most of the reference individuals become infeasible and hence mislead the search population. Second, the fitness values of search individuals might also become outdated. Because these values are always based on the objective value of the corresponding repaired feasible solution, they can become outdated right after a change if the objective value of the corresponding repaired solution changes. Due to this, search individuals with high-but-outdated fitness values might wrongly bias the selection process and hence the search. Evidence for this drawback can be seen in Figure 5 in Section V where the outdated fitness values cause the search individuals to remain in one feasible region even when the global optimum has moved away from this region. 5) Potential directions to improve the repair method for solving DCOPs: Despite the drawback of being outdated when a change happens, the repair method satisfies a number of requirements for solving DCOPs as set out in Subsection II-D: First, its operation does not interfere with such DO strategies as the diversity-maintaining/introducing mechanisms because it accepts both feasible and infeasible individuals in the same way. Second, the repair method is naturally suitable for tracking the moving feasible region: whenever a search individual is repaired, the newly created repaired individuals will always be closer to existing reference individuals than the search individual is (see Algorithm 1). As a result, if the algorithm is able to have at least one reference individual in the moving feasible region, the repair method will have a chance to track this moving region. Third, the repair method also naturally supports elitism because the best found feasible solutions will always be stored in the reference population. The aforementioned advantages of the repair method are empirically confirmed by the experimental results in Section IV where although the presence of environmental dynamics does significantly decrease the performance of GA+Repair, the algorithm still has better performance than all other GA-based existing DO algorithms in solving DCOPs.

5

To apply the repair method to solving DCOPs more effectively, we need to improve it to meet the three major requirements not satisfied by the original method: the ability to detect changes; the ability to increase/maintain diversity; and the ability to update the strategy per change. 6) Disadvantages of the repair method: A major disadvantage of repair methods like GA+Repair and Genocop III is that they require a considerable number of feasibility checkings to find feasible individuals for the reference population and for the repair operator. This disadvantage makes repair methods unsuitable for solving DCOPs with very small feasible areas or with very expensive constraint functions. For such problems, we may need to find a different CH strategy that does not require finding feasible individuals. In the next subsection, we will discuss the stochastic ranking method, one of the current state-of-the-art CH methods, and its suitability as an alternative to the repair method in solving DCOPs. C. The stochastic ranking method and SRES 1) The stochastic ranking mechanism and the improved SRES algorithm: As mentioned earlier, stochastic ranking (SR) is a rank-based selection mechanism to adaptively balance feasibility and infeasibility by accepting not only feasible solutions but also infeasible solutions with a certain probability Pf . The ranking method is based on a buble-sort-like procedure, where depending on Pf individuals might be sorted based on their feasibility or on their objective functions. A pseudo code of the SR procedure is given in Algorithm 3. Algorithm 3 routine StochasticRanking() Note: Pf = 0.45 P 2 Penalty function φ (x) = max {0, gk (x)} λ - Population size 1) For i=1 to λ : For j=1 to λ − 1 a) If (φ (xj ) = φ (xj+1 ))or (U (0, 1) < Pf ) then • If (f (xj ) > f (xj+1 ))then swap(xj , xj+1 ) b) Else If (φ (xj ) > φ (xj+1 ))then swap(xj , xj+1 ) The SR mechanism can be integrated into the selection scheme of any evolutionary algorithm. In [1] SR was integrated into a (µ, λ)-ES algorithm to create the stochasticranking evolution strategy (SRES) algorithm. The algorithm was then improved in [2] where an exponential smoothing technique to reduce random fluctuations of mutation step sizes and a differential variation technique to reduce search bias toward the coordination system were introduced (improved SRES). The improved SRES achieved the best results in solving the tested problems and still remains the current stateof-the-art in CH. A brief pseudo code of this algorithm is given in Algorithm 4. From now on we will refer this improved version of stochastic ranking evolution strategy as SRES. 2) Problem knowledge and balancing strategy in SRES: As mentioned above, SRES balances feasibility/infeasibility based on a simple yet effective mechanism: to select not only feasible solutions but also infeasible solutions based on a probability Pf . Pf ’s value, which can be greater than,

Algorithm 4 improved SRES Note:

λ - Population p √ size √ τ = ϕ/ 2 n; τ ′ = ϕ/ 2n ϕ, γ, α are pre-defined constants

′ 1) Initialisation: = √ For each individual k : σk (uk − lk ) / n; x′k = lk + (uk − lk ) .Uk (0, 1) 2) Evaluation: For each individual k : f (x′ k ), g(x′k ) 3) Selection: a) Call routine StochasticRanking() (Algorithm 3); ′ b) Select the best µ: (xi , σi ) ← (x′i;λ , σi;λ ), i = 1, .., µ 4) Mutation: For each individual k a) Cycle through best µ: i = mod (k − 1, µ) + 1 b) Differential variation: If (k < µ)then ′ • σk = σi ′ • xk = xi + γ (x1 − xi+1 ) c) Standard mutation: Else ′ ′ ′ • σk,j = σi,j exp (τ N (0, 1) + τ Nj (0, 1)) , j = 1, .., n ′ ′ • xk = xi + σk N (0, 1) ′ ′ • Exponential smoothing: σk = σi + α (σk − σi ) 5) If termination criteria not satisfied, back to step 2.

or less than, or equal to 0.5, determines if there is a bias against feasible solutions, or agains infeasible solutions, or no bias at all, respectively. The authors suggested a value of 0.4 < Pf < 0.5 (best reported value is 0.45[1]), which assures that although feasible solutions are slightly more favoured, infeasible individuals with good objective values still have a good chance to be preserved until the population converges. The balancing mechanism above is self-adaptive and relies totally on the search population, which stores the knowledge that the algorithm has learnt about the problem. The population plays the role as a special ”memory” to indicate promising feasible and infeasible regions in the search space. In addition, SRES also retains the knowledge about the best mutation strategies (σi ), which are learnt during the search process thanks to the self-adaptive nature of ES. 3) The impact of outdated information and outdated strategy on the stochastic ranking method: Despite having the selfadaptive features, SRES also suffers significantly from the issue of being outdated. First, if the algorithm has already converged to a region before a change occurs, the population no longer covers the whole landscape and consequently it might no longer be able to react to changes in regions not covered by the population. Second, because the stochastic ranking relies entirely on the population, if the population is outdated, the ranking mechanism would not be able to balance feasibility/infeasibility effectively. Third, the mutation strategies previously learnt by the ES algorithm might also no longer be suitable for the newly changed environments. Experimental results and analyses in Section IV and V confirm these difficulties. Table IX shows that SRES lack the necessary diversity for DCOPs, Figure 4 shows that SRES failed to track the moving optimal feasible region effectively, and Figure 5 shows that SRES did not effectively balance

6

feasibility/infeasibility in DCOPs. 4) Potential directions to improve stochastic ranking for solving DCOPs: Similar to the repair method, stochastic ranking also satisfies some requirements for solving DCOPs set out in Subsection II-D: First, the probability of ranking individuals based on either their objective values or their constraint values is nearly the same. Thanks to this, the operations of such DO strategies as the diversity-maintaining/introducing mechanisms should not be significantly affected because both feasible and infeasible individuals are accepted. Second, the method search in both feasible and infeasible regions, and hence facilitate detecting changes in both regions. Third, the ranking method has a big advantage compared to the repair method: it does not require additional feasibility checkings or that feasible solutions have to be found. Similar to the case of the repair method, to solve DCOPs we need to improve SR to resolve its drawbacks by introducing the followings to satisfy the requirements set out in Subsection II-D: a diversity mechanism, a change detection mechanism, a better method to track the moving feasible regions, and a mechanism to update the ranking strategy.

D. Combining the advantages of DO and CH techniques In this subsection, we will discuss our proposed method to modify the repair method and SR into a framework that supports all the necessary requirements to solve DCOPs. We will then discuss the possible ways to integrate existing DO strategies into the framework to solve DCOPs more effectively. 1) Maintaining/introducing diversity: The experimental results in Sections III and V show that diversity is not maintained effectively in GA+Repair and especially in SRES. The algorithm suffers significantly from the lack of diversity because the population has already converged to a feasible region and remained in this region (see Figure 5). Consequently it is not able to react to changes in other regions and/or follow the moving optima to other regions (see Figure 4). To improve the performance of GA+Repair and SRES, we need to equip them with some mechanisms to maintain/introduce more diversity as follows. For GA+Repair, we just replace the normal GA mutation with the mutation strategies of RIGA and HyperM. For SRES, after the mutation step of SRES, we carry out the following: 1) Replace a fraction of the population with randomly generated individuals based on the random-immigration rate or the hyper-mutation rate, 2) For each random-immigrant/hyper-mutated individual √ k, re-initialise its mutation strategy: σk,i = (ui − li ) / n, i = 1, ..., n where ui , li are the upper, lower bound of the i-th decision variable, and n is the dimension. For the random-immigration version, the two steps above will be carried out at every genration. For the hyper-mutation versions, the two steps above will only be carried out whenever the algorithm detects a change or is informed of a change and will be maintained until the change no longer has impact on the population (as specified in [6]).

2) Tracking the moving feasible regions: The next step is to improve the ability of GA+Repair and SRES in tracking the moving feasible regions to satisfy the requirements in Subsection II-D. Figure 4 and the analysis in Section V show that GA+Repair did not find much difficulty in tracking moving feasible regions. This is because, whenever a search individual is repaired, the newly repaired individuals will always be closer to the chosen reference individuals. As a result, if at least one reference individual is in the moving feasible region, the algorithm will be able to send more individuals toward the region and hence the region is tracked. SRES, on the other hand, suffers significantly from the difficulty of tracking the moving feasible regions, as shown in Figure 4 and the analysis in Section V. To improve SRES’s performance in tracking moving feasible regions, we can equip the algorithm with a similar tracking mechanism as that of GA+Repair. Of course, it is not a good idea to apply exactly the repair operator from GA+Repair to SRES because the repair operator needs a lot of feasibility checkings, which consequently eliminate the advantage of SRES in not requiring additional constraint evaluations. We can, however, maintain a separate reference population containing the current promising feasible individuals (and their corresponding mutation strategies σ), and frequently reintroduce these individuals to the main population. This way, if any of the found feasible individual is in the moving feasible region, the algorithm might have a better chance to follow the region2. This mechanism can be enhanced when being combined with the diversity maintaining/introducing scheme introduced earlier so that the reference population is frequently updated with diversified individuals from other regions in the landscape to track any newly appearing region. The reference population itself does not evolve but is regularly updated with individuals from the main population3. Another advantage of adding a reference population to SRES is that, because reference individuals are always the best feasible solutions found, maintaining a reference population means that elitism is naturally supported. This satisfies another requirement set out in Subsection II-D for solving DCOPs. Details of the tracking routine is described in Algorithm 5. 3) Updating the feasibility/infeasibility balance strategies: As discussed earlier, SRES suffers from the outdated issues because (i) the population might have already converged to one region and (ii) the cumulative mutation strategies learnt so far might become outdated. These issues can be alleviated using the proposed diversity-maintenance and feasible-tracking mechanisms because they can automatically help updating the population and the mutation strategies. However, the introduction of the new feasible-tracking mechanism leads to a new issue: the reference population itself might become outdated 2 Although the reference population must contain feasible solutions, the proposed mechanism does not require spending any additional evaluation to find a feasible solution. Promising feasible solutions will be added when they are found during the search process. If no feasible solution is found yet, the algorithm work as normal with no reference individual. 3 Although maintaining the reference population to some extent is similar to maintaining elitism, the reference individuals are different from elite members in that reference individuals are only re-introduced after certain period while elite members are always retained at every generation.

7

Algorithm 5 routine TrackFeasible() Variables: S - Search population; R - Reference population mR - maximum size of R ωU - Period to update S with individuals from R 1) For each gen, after evaluation: For each feasible x ∈ S, a) If (kRk < m) then x → R; σ (x) → R b) Else if (∃r ∈ R| f (x) < f (r)) then r = x; σ (r) = σ (x) 2) For each ωU gen, after evaluation: For each r ∈ R: a) Randomly choose x ∈ S , b) x = r and σ (x) = σ (r)

because some individuals might no longer remain feasible and hence the reference population also needs updating. In this paper, we propose a method to update the outdated reference individuals. First, for each infeasible reference individual, we will try to replace it by a feasible individual from the search population S using non-linear ranking selection. If there is no feasible individual in S, we will replace the infeasible reference individual with a feasible individual from the reference population R. If there is no such feasible individual, we will then replace the infeasible reference individual with a randomly generated feasible solution (maximum ten attempts). After all infeasible reference individuals have been replaced by the feasible ones, all reference individuals will be re-evaluated if they have not been evaluated since the last change. The update procedure is described in Algorithm 6. Algorithm 6 routine UpdateReferencePop() Variables: S - Search population;R - Reference population nS - Number of feasible individuals in S nR - Number of feasible individuals in R mR - Number of unfeasible individuals in R 1) For i = 1 to min (nS , mR ): Replace each unfeasible xiu in R with a feasible one from S a) Non-linear ranking select a feasible xf from S b) Replace xiu by xf : xiu = xf 2) If (mR > nS )then For j = nS + 1 to nS + min (mR − nS , nR ): Replace each unfeasible xju in R with a feasible one from R a) Non-linear ranking select a feasible xf from R b) Replace xju with xf : xju = xf 3) If (mR > nS + nR )then For k = nS + nR + 1 to mR − nS − nR : Replace each unfeasible xku in R with a feasible, randomly generated individual (ten attempts)

4) Re-evaluate all reference indivs not evaluated since last change GA+Repair (and other repair methods) also suffers from the issue of being outdated, which prevents the algorithms from focusing resources on the optimal feasible region (see Figure 5). To improve GA+Repair, the knowledge that the repair operator uses, which are the reference and search populations, should be updated. For the reference population, we use the same routine in Algorithm 6. For the search population in GA+Repair, we need to update those search individuals that

have become outdated. The update mechanism is as follows: whenever we detect a change that affects the repair method, we firstly update the reference population and then use the newly updated reference individuals to repair all search individuals that have not been repaired since the last change. This is to make sure that the fitness values of all search individuals correctly reflect the feasible regions of the newly changed landscape. The update procedure is in Algorithm 7. Algorithm 7 routine UpdateSearchPop() Variables: Note:

S - Search population

This routine needs to be called after the routine UpdateReferencePop (Algorithm 6) 1) Create a set S1 ∈ S which includes all search individuals not been selected for the repair process since last change 2) For each individual s in S1 , update the fitness value of s by calling the routine Repair (s)

4) Detecting changes: In case the algorithm is not informed of the environmental changes, it is necessary for SRES and GA+Repair to detect the changes themselves so that they can prevent their problem information and constraint-handling strategies from being outdated. For the stochastic ranking and repair methods, it is not necessary to explicitly detect all types of change. Changes in feasible regions not covered by the population does not need to be detected. With the integration of the proposed diversitymaintenance mechanisms, it is likely that both the modified SRES and modified GA+Repair will be able to adaptively react, because some feasible solutions from these regions will probably be found by the diversity mechanisms and will consequently be added to the reference population. For changes that occur in the regions covered by the population, we only need to detect those that make the methods’ current knowledge outdated, i.e. the changes that alter the feasibility status or objective values of the reference individuals. This is because reference individuals are those that carry out the information about current promising moving feasible regions and are those that will influence the whole population to track these regions. To detect changes in objective values of reference individuals, we can adopt and modify the change-detection method originally used in HyperM [6]: monitor the drop of the average best fitness values over a certain period. Unlike the original change-detection method in HyperM, here we will only monitor the drop of the average fitness values of the best individuals in the reference population. If such a drop is detected and the drop persists over a certain period, it can be assumed that a change in the objective function has occurred. To detect changes in the constraints of reference individuals, we can just simply re-evaluate the constraint functions of after a certain period ωC . In this paper ωC is set to two generations4. Details of the procedure to detect changes in objective function can be found in the prosecode of routine DetectChange() 4 For methods not having reference individuals, it might be necessary to have an additional change detection mechanism to detect infeasible changes that alter the shape of infeasible areas (i.e. the shrink/extension or (dis)appearance of infeasible areas).

8

(Algorithm 8). The change detection routine should be maintained separately from the solving algorithms because it will only be needed in case changes are not known. Algorithm 8 routine DetectChange() Notes Inputs

Outputs

The routine needs to be called at each generation

k - index of the current generation bk - the fitness value of best feasible solution at gen. k R - the reference population true/false - whether a change is detected or not

 Pk−1  Pk 1) If k−4 bi /5 > k−5 bi /5 then return true; 2) For each r ∈ R: if gi (r) > 0then return true; 3) Return false; 5) Searching out of range: We observed that the original repair method becomes less effective in solving problems with global optima in search boundaries. This is because the probability of the method finding the optimum is smaller when the optimum is in search boundaries than when it is not. One way to make repair methods work better in such cases is to allow them to search out of range and consider all outof-range solutions infeasible. In an extended search space like this, the actual global optimum now is inside the search area, making it easier for the method to find the optimum. In this paper, when the out-of-range mechanism is tested we allow the algorithms to search 25% beyond the given search range. E. The new algorithms: dRepairRIGA, dGenocop, RIR-SRES and other variants In this subsection we will apply the previously proposed mechanisms to GA+Repair, Genocop III, and improved SRES to evaluate how effective they are in solving DCOPs. For improving GA+Repair, the proposed mechanisms were used to create three different versions of new algorithms. First, the change-detection and update mechanisms were integrated with GA+Repair to create a new algorithm called dRepairGA as follows: in addition to all existing operations of GA+Repair, at every two generations we call the routine DetectChange() to detect any environmental changes that may possibly affect the repair method. If a change is detected, we will invoke the routine UpdateReferencePop(), followed by the routine UpdateSearchPop(), to update the search and reference populations. The pseudocode of dRepairGA is described in Algorithm 9. dRepairGA was then extended to a second version: the mutation mechanism of dRepairGA was replaced with randomimmigrant and hyper-mutation strategies. The new algorithms are called dRepairRIGA and dRepairHyperM5. In the third version, dRepairGA/RIGA/HyperM were modified to search 25% beyond the given search range using the out-of-range (OOR) mechanism proposed in Subsection III-D5. The new algorithms are called dRepairGA OOR, dRepairRIGA OOR and dRepairHyperM OOR. 5 dRepairHyperM can trigger its hyper-mutation not only when it detects a drop in performance of the best individual like HyperM, but also when the DetectChange() routine returns true.

For improving Genocop III, the original Genocop III algorithms were hybridised with the newly proposed mechanisms (routines DetectChange(),UpdateReferencePop()and UpdateSearchPop()) in the same way as in dRepairGA in Algorithm 9. The new algorithm is called dGenocop. For improving SRES, two different versions of algorithms were created. First, the modified random-immigrant and hyper-mutation mechanisms, as described in Subsection III-D1, were integrated with the original SRES to create new algorithms named RI-SRES and HMSRES. Second, the newly proposed TrackFeasible(), DetectChange(),and UpdateReferencePop()were added to RI-SRES to improved its performance in tracking the moving feasible regions. DetectChange() and UpdateReferencePop()were applied in the same way as in dRepairGA while TrackFeasible()were called at every generation. The new algorithm is called RIR-SRES and its pseudo code is in Algorithm 10. We also developed a GAbased version of stochastic ranking named SRGA, and applied RI and HM to this algorithm (SR-RIGA and SR-HyperM) to compare with other GA-based algorithms. Algorithm 9 dRepairGA Routines: DetectChange() - described in Algorithm 8 UpdateReferencePop() - described in Algorithm 6 UpdateSearchPop() - described in Algorithm 7 1) Initialise: same as step 1 of GA+Repair (Algorithm 2,page 4) 2) Search at each generation: same as step 2 of GA+Repair 3) Evolve the reference population: same as step 3 of GA+Repair 4) Detect change and update search strategy: a) For each generation: If DetectChange()=true i) Update reference pop.: UpdateReferencePop() ii) Update the search pop.: UpdateSearchPop() b) Else do nothing 5) Return to step 2

IV. C OMPARING

AND ANALYSING

RIR-SRES,

D R EPAIR RIGA, D G ENOCOP AND THEIR VARIANTS AGAINST EXISTING ALGORITHMS

A. Chosen algorithms In this section the performance of different versions of RIRSRES, dRepairGA and dGenocop (as described in the previous section) will be compared with those of existing algorithms: GA, RIGA, HyperM, GA+Repair, Genocop III, SRGA and SRES to see if our newly proposed mechanisms can help improving the drawbacks of existing methods. We also compare the performance of the new algorithms with another state-of-the-art method in CH: the Simple-Multi-membered Evolution Strategies (SMES) algorithm [16]. The purpose is to investigate how another state-of-the-art CH method, besides SRES and Genocop III, perform on DCOPs. Note that to apply GA, RIGA and HyperM directly to solving DCOPs, we need to integrate them with a CH mechanism, which should not interfere with or change the original DO

9

Algorithm 10 RIR-SRES Routines: UpdateReferencePop() - described in Algorithm 6 TrackFeasible() - described in Algorithm 7 DetectChange() - described in Algorithm 8 Inputs: PR - random-immigrant rate S, R - the search and reference populations mR - size of R u, l - upper and lower bounds of the decision variables n - number of decision variables 1) Initialisation: a) Initialise S : same as step 1 of SRES (Algorithm 4) b) Initialise R: copy up to m best feasible from S to R. 2) Evaluation: same as step 2 of SRES 3) Tracking feasible regions: TrackFeasible();with ωU = 5 4) Selection: same as step 3 of SRES 5) Mutation: same as step 4 of SRES 6) Random-immig.: For each indiv k , If Uk (0, 1) < PR : a) xk = lk + (uk − √ lk ) .Uk (0, 1) b) σk = (uk − lk ) / n; 7) Detect change and update search strategy: a) For each generation: If DetectChange()=true then UpdateReferencePop(); b) Else do nothing 8) Return to step 2. TABLE I T HE OBJECTIVE FUNCTION FORM OF EACH BENCHMARK PROBLEM Benchmark problem G24 8a & G24 8b All other problems

objective function f (x) = f (2) f (x) = f (1)

minimise f (x) subject to gi (x) ≤ 0, gi (x) ∈ G, i = 1, .., n where the objective function f (x) can be one of the function forms in equation (1), each constraint gi (x) can be one of the forms in equation (2), and G is the set of n constraint functions for that particular problem. The detailed description of f (x) and gi (x) for each problem in the G24 benchmark set are described in Table I and Table II. Equation (1) describes the general function forms for the objective functions in the G24 set. Of these function forms, f (2) is used to design the objective function for G24 8a and G24 8b, and f (1) is used to design the objective functions for all other problems. f (1) is modified from a static function proposed in [23] and f (2) is a newly designed function. f (1) = f (2) =

where Xi (xi , t) = pi (t) (xi + qi (t));0 ≤ x1 ≤ 3; 0 ≤ x2 ≤ 4 with pi (t) and qi (t) (i = 1, 2) as the dynamic parameters, which determine how the dynamic objective function of each benchmark problem changes over time. Equation (2) describes the general function forms for the constraint functions in the G24 set. Of these function forms, g (1) and g (2) were modified from two static functions proposed in [23] and g (3) , g (4) , g (5) , g (6) are newly designed functions. g (1) = g (2) =

strategies in any way. To satisfy this requirement, the penalty function approach in [22] was chosen because it is the simplest and easiest way to apply existing unconstrained DO algorithms directly to solving DCOPs without changing anything in the algorithms. Additional reasons for chosing this penalty method is that it is reportedly effective in solving difficult numerical problems and more importantly it does not require users to choose any penalty factor or other parameter.

B. Test problems A set of 18 benchmark problems (named G24 set) able to capture the DCOP characteristics (Section II-A) was proposed in [14]. The general form for each problem is as follows: TABLE II T HE SET OF CONSTRAINT FUNCTION FORMS FOR EACH PROBLEM Benchmark problem G24 u; G24 uf; G24 2u; G24 8a G24 6a G24 6b G24 6c G24 6d All other problems

Set G of constraints G = {∅}  G = g (3) , g (6) (3) G = g G = g (3) , g (4) (5) (6) G = g , g G = g (1) , g (2)

(1) − (X1 (x1 , t) + X2 (x2 , t)) ! rq 2 2 −3 exp − (X1 (x1 , t)) + (X2 (x2 , t))

g (3) = g (4) = g (5) = g (6) =

4

3

2

−2Y1 (x1 , t) + 8Y1 (x1 , t) − 8Y1 (x1 , t) +Y2 (x2 , t) − 2

(2)

−4Y1 (x1 , t)4 + 32Y1 (x1 , t)3 − 88Y1 (x1 , t)2 +96Y1 (x1 , t) + Y2 (x2 , t) − 36

2Y1 (x1 , t) + 3Y2 (x2 , t) − 9  −1 if (0 ≤ Y1 (x1 , t) ≤ 1)or(2 ≤ Y1 (x1 , t) ≤ 3) 1 otherwise  −1 if (0 ≤ Y1 (x1 , t) ≤ 0.5)or(2 ≤ Y1 (x1 , t) ≤ 2.5) 1 otherwise   −1 if [(0 ≤ Y1 (x1 , t) ≤ 1) and (2 ≤ Y2 (x2 , t) ≤ 3)] or (2 ≤ Y1 (x1 , t) ≤ 3)  1 otherwise

where Yi (x, t) = ri (t) (x + si (t));0 ≤ x1 ≤ 3; 0 ≤ x2 ≤ 4 with ri (t) and si (t) (i = 1, 2) as the dynamic parameters, which determine how the constraint functions of each benchmark problem change over time. Each benchmark problem may have a different mathematical expression for pi (t), qi (t), ri (t) and si (t). The individual expressions of pi (t), qi (t), ri (t), and si (t) for each benchmark function are described in Table III. The problems were designed specifically so that they can be arranged in 21 pairs, of which the two problems in each pair are almost identical except that one has a particular characteristic (e.g. fixed constraints) and the other does not. By comparing the performance of an algorithm on one problem with its performance on the other problem in the pair, it is possible to analyse whether the considered characteristic has

10

TABLE III DYNAMIC PARAMETERS FOR ALL TEST PROBLEMS IN THE BENCHMARK SET G24. E ACH DYNAMIC PARAMETER IS A TIME - DEPENDANT RULE / FUNCTION WHICH GOVERNS THE WAY THE PROBLEMS CHANGE Prob G24 u G24 1 G24 f G24 uf G24 2

Parameter settings  p1 (t) = sin kπt + π2 ; p2 (t) = 1; qi (t) = 0 p2 (t) = ri (t) = 1; qi(t) = si (t) = 0 p1 (t) = sin kπt + π2 pi (t) = ri (t) = 1; qi (t) = si (t) = 0 pi (t) = 1; qi (t) = 1  p1 (t)=sin( kπt +π 2 2) n if (t mod 2 = 0) p2 (t−1) if t>0 if (t mod 2 6= 0)

G24 2u

qi (t) = si (t) = 0; ri (t) = 1  p1 (t)=sin( kπt +π 2 2) n if (t mod 2 = 0) p2 (t−1) if t>0 if (t mod 2 6= 0)

G24 3 G24 3b

G24 3f G24 4 G24 5

G24 7 G24 8a G24 8b k S c 1 , c 2 , ra (G24 8a/b only) i

p2 (t)= p (0)=0 if t=0 2 +π p1 (t)=sin( kπt 2 2)   kπ(t−1) p2 (t)=sin +π 2 2



qi (t) = 0 pi (t) = ri (t) = 1; qi (t) = s1 (t) = 0 2 min s2 (t) = 2 + t. x2 max S−x  π p1 (t) = sin kπt + 2 ; p2 (t) = 1 qi (t) = s1 (t) = 0; ri (t) = 1; s2 (t) = 2 + t. x2 max S−x2 min pi (t) = ri (t) = 1; qi (t) = s1 (t) = 0; s2 (t) = 2 p2 (t) = ri (t) = 1; qi(t) = s1 (t) = 0 p1 (t) = sin kπt + π2 ; s2 (t) = t. x2 max S−x2 min  p1 (t)=sin( kπt +π 2 2) n if (t mod 2 = 0) p2 (t−1) if t>0 p2 (t)=

if (t mod 2 6= 0)

G24 6a/b/c/d

p2 (t)= p (0)=0 if t=0 2 +π p1 (t)=sin( kπt 2 2)   kπ(t−1) +π p2 (t)=sin 2 2



p2 (0) if t=0

+π p1 (t)=sin( kπt 2 2)   kπ(t−1) +π p2 (t)=sin 2 2



qi (t) = s1 (t) = 0; ri (t) = 1; s2 (t) = t. x2 max S−x2 min p1 (t) = sin πt + π2 ; p2 (t) = 1; qi (t) = si (t) = 0; ri (t) = 1 pi (t) = ri (t) = 1; qi (t) = s1 (t) = 0; s2 (t) = t. x2 max S−x2 min pi (t) = −1; q1 (t) = − (c1 + ra . cos (kπt)) q2 (t) = − (c2 + ra . sin (kπt)) ; pi (t) = −1; q1 (t) = − (c1 + ra . cos (kπt)) q2 (t) = − (c2 + ra . sin (kπt)) ; ri (t) = 1; si (t) = 0 k determines the severity of function changes. k = 1 ∼large; k = 0.5 ∼ medium; k = 0.25 ∼ small S determines the severity of constraint changes S = 10 ∼large; S = 20 ∼ medium; S = 50 ∼ small c1 = 1 : 470561702; c2 = 3 : 442094786232; ra = 0 : 858958496 . i is the variable index, i = 1, 2

any effect on the tested algorithm and to what extent is the effect significant. The characteristics of each problem can be found in Table IV, and the description of each test case (pair of problems) can be found in Table V. The G24 benchmark set will be used in all experiments in this paper to evaluate and compare the performance of our newly developed algorithms with existing algorithms. C. Parameter settings Table VI shows the detailed parameter settings for all algorithms tested in this paper. All algorithms use real-valued representations. To create a fair testing environment and to draw reliable conclusions, the algorithms are tested in a wide range of dynamic settings (different values of population size,

TABLE IV P ROPERTIES OF EACH TEST PROBLEM IN THE G24 BENCHMARK SET Problem ObjF Constr DFR Type SwO NAO OICB OISB Path % feasible* G24 uf fF noC 1 NL No No No Yes N/A 100 G24 f fF fC 2 NL No No Yes No N/A 44.6 noC 1 No No No Yes N/A 100 G24 u dF G24 1 dF fC 2 NL Yes No Yes No N/A 44.6 G24 2u dF noC 1 No No No Yes N/A 100 G24 2** dF fC 2 Yes No Y&N Y&N N/A 44.6 dC 2-3 NL No Yes Yes No N/A 7.3-44.6 G24 3 fF G24 3b dF dC 2-3 NL Yes No Yes No N/A 7.3-44.6 G24 3f fF fF 1 NL No No Yes No N/A 7.3 dC 2-3 NL Yes No Yes No N/A 7.3-44.6 G24 4 dF G24 5** dF dC 2-3 NL Yes No Y&N Y&N N/A 7.3-44.6 G24 6a dF fC 2 L Yes No No Yes Hard 17 G24 6b dF fC 1 L No No No Yes N/A 50.5 G24 6c dF fC 2 L Yes No No Yes Easy 33.6 G24 6d dF fC 2 L Yes No No Yes Hard 17 G24 7 fF dC 2 NL No No Yes No N/A 7.3-44.6 noC 1 No No No No N/A 100 G24 8a dF G24 8b dF fC 2 NL Yes No Yes No N/A 44.6 DFR number of Disconnected Feasible Regions 1R/2DR One feasible region / Two disconnected feasible regions L/NL Linear / Nonlinear constraints SwO Switched global Optimum between disconnected regions (N)NAO (No) Newly Appear Optimum without changing existing ones O(N)ICB global Optimum is (Not) In the Constraint Boundary O(N)ISB global Optimum is (Not) In the Search Boundary Path Indicate if it is easy or difficult to use mutation to travel between feasible regions fF/dF Fixed objective function / Dynamic objective function fC/dC Fixed constraints / Dynamic constraints noC There is no constraint G24 xx G24 (x)f: fixed problem; G24 (x)u: unconstrained problem; G24 other: dynamic problem. * Average percentage of feasible regions within 10 medium changes. ** In some change periods, the landscape either is a plateau or contains infinite number of optima and all optima (including the existing optimum) lie in a line parallel to one of the axes TABLE V T HE 21 TEST CASES ( PAIRS OF PROBLEMS ) TO BE USED IN THIS PAPER . E XPLANATIONS OF ABBREVIATIONS ARE IN TABLE IV Static problems: Unconstrained vs Fixed constraints 1 G24 uf (fF, noC) vs G24 f (fF, fC) Fixed objectives vs Dynamic objectives 2 G24 uf (fF, noC) vs G24 u (dF, noC) 3 G24 f (fF, fC, OICB) vs G24 1 (dF, fC, OICB) vs G24 2 (dF, fC, ONICB) 4 G24 f (fF, fC, OICB) Dynamic objectives: Unconstrained vs Fixed constraints 5 G24 u (dF, noC) vs G24 1 (dF, fC, OICB) 6 G24 2u (dF, noC) vs G24 2 (dF, fC, ONICB) Fixed constraints vs Dynamic constraints 7 G24 1 (dF, fC, OICB) vs G24 4 (dF, dC, OICB) 8 G24 2 (dF, fC, ONICB) vs G24 5 (dF, dC, ONICB) 9 G24 f (fF, fC) vs G24 7 (fF, dC, NNAO) vs G24 3 (fF, dC, NAO) 10 G24 3f (fF, fC) No constraint vs Dynamic constraints 11 G24 u (dF, noC) vs G24 4 (dF, dC, OICB) 12 G24 2u (dF, noC) vs G24 5 (dF, dC, ONICB) 13 G24 uf (fF, noC) vs G24 7 (fF, dC) Moving constraints expose better optima vs not expose optima 14 G24 3f (fF, fC) vs G24 3 (fF, dC, NAO) 15 G24 3 (fF, dC, NAO) vs G24 3b (dF, dC, NAO) Connected feasible regions vs Disconnected feasible regions 16 G24 6b (1R) vs G24 6a (2DR, hard) 17 G24 6b (1R) vs G24 6d (2DR, hard) 18 G24 6c (2DR, easy) vs G24 6d (2DR, hard) Optima in constraint boundary vs Optima NOT in constr boundary 19 G24 1 (dF, fC, OICB) vs G24 2 (dF, fC, ONICB) 20 G24 4 (dF, dC, OICB) vs G24 5 (dF, dC, ONICB) 21 G24 8b (dF, fC, OICB) vs G24 8a (dF, noC, ONISB)

11

TABLE VI T EST SETTINGS FOR ALL ALGORITHMS USED IN THE PAPER . All algos. GA-based (exceptions below) HyperM RIGA GA+Repair, dRepairGA, & variants GenocopIII, dGenocop & variants SMES SRES & variants RIR-SRES

Benchmark problem settings

Pop size (pop size) Elitism Selection method Mutation method Crossover method Triggered mutate Rand-immig. rate Search pop size Reference pop size Replacement rate Search pop size Reference pop size Other parameters All parameters except pop size All parameters except pop size Rand-immig. rate Update period ωU Ref pop size mR Number of runs Number of changes Change frequency ObjFunc severity k Constr. severity S

5, 15, 25 (medium), 50, 100 Elitism & non-elitism if applicable Non-linear ranking as in [4] Uniform, P = 0.15. Arithmetic, P = 0.1. Uniform, P = 0.5 as in [6]. P = 0.3 as in [5]. pop size× (4/5) pop size× (1/5) 0 (default is 0.25 as in [4]). pop size× (4/5) pop size× (1/5) as in [4]. as in [16] as in [2] P = 0.3 as in [5]. 3 5 50 5/k (see below) 250, 500, 1K (med), 2K, 4K func evals 0.25 (small), 0.5 (med), 1.0 (large) 10 (small), 20 (medium), 50 (large)

severity of change and frequency of change) with five levels: small, medium small, medium, medium large, large. For each test setting all algorithms are tested under the same number of function evaluations regardless of the actual number of generations that they spent. The parameters of all algorithms were set to similar values or the best known values if possible. The base mutation rate is 0.15, which is the average of the best rates commonly used for GA-based in existing studies on continuous DO, which are 0.1 ([24] [20]) and 0.2 ([25] [26]). For HyperM and RIGA, the best hyper-mutation-rate and random-immigrantrate parameter values observed in the original papers [6] [5] were used. The same implementations as described in [6] and [5] were used to reproduce these two algorithms. A crossover rate of 0.1 was chosen for all algorithms because this is one of the few settings where all tested algorithms perform well (see subsection VI-C for more details). For Genocop-based and ESbased algorithms, the same parameter values as suggested by the original authors were used, except for the population sizes (which were set to the same five levels from small to large as in other algorithms) and the replacement ratio of Genocop (which was set to zero). For the newly proposed RIR-SRES and its variants, we set ωU - the period to update S with indivdiuals from R - to three, and set mR - the size of the reference population - to five to make it the same as in dRepairGAbased and Genocop-based algorithms6 . A further study of the effect of different values of the base mutation rates, hyper-mutation rates, random-immigrant rates, crossover rates, replacement ratios, ωU and mR on algorithm performance was also carried out in Section VI.

6 As will be analysed in Section VI, any positive values of ω U and mR would bring almost the same improvement in performance for RIR-SRES.

D. Performance measures and analysis criteria To measure the performance of the algorithms in this particular experiment, we proposed two new measures. First, for the detailed performance of each algorithm on each problem, we used the modified offline error for DCOPs, or offline error for short [14], which is our modified version of the original modified offline error [27]. The measure is calculated as the average over, at every evaluation, the error of the best feasible solution found since the last change of the environment. The fitness and errors of infeasible solutions will not be counted, regardless of their values. If in any generation there is no feasible solution, the measure will take the worst possible value that a feasible solution can have for that particular generation. This measure is always greater than or equal to zero and would be equal to zero for a perfect performance. 1 Xn eMO (j) (3) EMO = j=1 n where n is the number of generations so far, and eMO (j) is the best feasible error since the last change gained by the algorithm at the generation j. Although the offline error is useful for analysing algorithm performance on a single problem, in this experiment it is not feasible to evaluate algorithm performance by averaging all the offline errors we got from all test instances (very large number in this case) because the result might be biased toward larger errors. For example, if an algorithm has a large error of 1000 in problem A and a very small error of 1e-10 in problem B, the average value would be a large error of 500+5e-11. This large error obviously does not reflect the good performance that the algorithm has achieved in problem B. This limitation leads us to developing a new measure named normalised score to evaluate the overall performance of each algorithm on all possible test instances or on each group of problems in a normalised way. The idea is that, given a group of n tested algorithms and m problems, for each problem j the performance of each algorithm is normalised to the range (0, 1) so that the best algorithm in this problem j will have the score of 1 and the worst algorithm will get the score of 0. The final overall score of each algorithm will be calculated as the average of the normalised scores from each individual problem. If an algorithm is able to perform best in all tested problems, it gets an overall score of 1 and if it performs worst in all problems, it gets a score of 0. Formally, 1 Xm |emax (j) − e (i, j)| , ∀i = 1 : n. Snorm (i) = j=1 |emax (j) − emin (j)| m (4) where e (i, j) is the modified offline error of algorithm i in problem j; and emax (j) and emin (j) are the largest and smallest errors among all algorithms in solving problem j. The experimental results were analysed from different perspectives. First, using the normalised scores we summarised the average performance of the tested algorithms in each major group of problems under different parameter settings and dynamic ranges (see Figure 1 and Table VII) to have an overall picture of the behaviour of each algorithm in different types of problems. Second, we investigated the effect of each problem characteristic on each algorithm by analysing their offline error

12

TABLE VII N ORMALISED SCORE OF THE TESTED ALGORITHMS . S CORES GREATER THAN OR EQUAL TO 0.95 ARE IN BOLD .

Algorithm .GA-noElit .RIGA-noElit .HyperM-noElit .GA-elit .RIGA-elit .HyperM-elit .GA+Repair .dRepairGA .dRepairRIGA .dRepairHyperM .dRepairGA-OOR .dRepairRIGA-OOR .dRepairHyperM-OOR .GENOCOP .dGENOCOP .SMES .SRES .SRGA-noElit .SRGA-elit .SRRIGA-noElit .SRRIGA-elit .SRHyperM-noElit .SRHyperM-elit .RI-SRES .HM-SRES .RIR-SRES

Total 0.49 0.61 0.73 0.72 0.79 0.75 0.85 0.89 0.93 0.92 0.92 0.95 0.94 0.92 0.96 0.25 0.76 0.72 0.75 0.79 0.87 0.83 0.80 0.93 0.87 0.95

Normalised score DCOPs dF+noC fF+fC fF+noC 0.53 0.67 0.00 0.33 0.64 0.79 0.24 0.45 0.70 0.78 0.77 0.83 0.66 0.87 0.76 0.89 0.76 0.87 0.82 0.90 0.70 0.88 0.81 0.89 0.90 0.61 0.98 0.74 0.94 0.72 0.98 0.74 0.96 0.79 0.99 0.77 0.97 0.76 0.97 0.74 0.98 0.70 0.94 0.89 0.96 0.90 0.99 0.82 0.97 0.88 0.97 0.70 1.00 0.97 0.91 0.89 0.97 0.92 1.00 0.97 0.13 0.01 0.95 1.00 0.69 0.83 0.98 1.00 0.68 0.93 0.55 0.90 0.95 0.69 0.97 0.68 0.80 0.90 0.52 0.79 0.96 0.85 0.93 0.89 0.80 0.93 0.83 0.94 0.96 0.71 0.96 0.75 0.92 0.97 0.92 0.99 0.87 0.95 0.67 0.97 0.95 0.98 0.91 0.97

performance in 21 test cases as shown in Table V (pair of almost identical problems, one with a particular characteristic and one without). The test results for this analysis are in Figures 2 and 3. Third, besides the offline error and normalised score we also evaluated the number of feasibility checkings that an algorithm might need to take to solve a problem. These data are given in Table VIII. Note that because the tested algorithms still have the same behaviour under different parameter settings and dynamic ranges, only the summarised results in Figure 1 are presented with different settings (small / medium / large). For other detailed figures and tables, we only present the results in the default settings (all parameters/dynamic ranges are medium). E. Experimental results and analyses The results in Figure 1 (and the detailed pairwise comparisons in Figures 2 and 3) confirm our previous study of the negative impact of DCOPs on the performance of existing DO and CH methods. The figures show that CH methods like SRES, SMES and GA+Repair become much less effective (much lower scores and higher errors) in the DCOP cases (dF+fC, fF+dC, dF+dC) compared to the static constrained cases (fF+fC). Similarly, DO methods like GA/RIGA/HyperM have much lower scores (and higher errors) in the DCOP cases compared to the dynamic unconstrained case (dF+noC). The experimental results also confirm that all the new algorithms (RIR-SRES, dRepairRIGA, dGenocop and variants) achieved significantly better performance than the existing DO and CH algorithms on DCOPs problems (groups fF+dC, dF+fC, and fF+fC), both in term of the overall normalised

Fig. 1. Algorithm performance (normalised scores) in groups of problems. Explanations for abbreviations are in Table V. The best and worst offline errors recorded in each problem group (corresponding to the normalise scores of 1 and 0 in each group) are shown under the horizontal axis of the top plot.

13

scores (Table VII and Figure 1) and in term of the offline error at each problem (Figures 2 and 3). RIR-SRES, dGenocop, and dRepairRIGA OOR are the best overall performers, of which the first two are the two most robust for different types of problems because they have top performance with scores larger than 0.9 in all groups of problems (Table VII). Taking into account DCOPs only, then the repair-based algorithms (dGenocop, dRepairRIGA, dRepairHyperM and variants - scored 0.96-0.97) seem to perform better than RIRSRES (scored 0.95). However, this better performance come at a significant cost of additional feasibility checkings: Table VIII shows that the average number of feasbility checkings in repair-based methods are extremely high: from ˜450,000 to ˜1 million checkings, compared to only ˜12,928 of RIR-SRES and ˜11,000 of all other algorithms. We found that these high number of feasibility checkings mostly come from the DCOPs with moving feasible regions where the feasible regions become very small (less than 5% of the search space). This confirms the trade-off of using repair methods: although they can provide high quality solutions for constrained problems, they are not suitable for solving problems with small feasible areas or with expensive constraint functions. Figure 1 shows the good results of RIR-SRES also come with a trade-off: the algorithm become slightly worse than original SRES in solving static unconstrained (fF+noC) and static constrained problems (fF+fC) due to the introduction of the random immigrants and reference population. However, the decrease in performance is not significant (the scores of RIR-SRES in fF+noC and fF+fC still belong to the top: 0.97 and 0.91) while the improvement that RI-SRES achieved on DCOPs and on dynamic unconstrained problems are significant (increased scores from 0.69 and 0.83 to 0.94 and 0.98). Another interesting, counter-intuitive observation is that, although the presence of constraints makes the problems more difficult for existing DO algorithms, for repair methods the presence of constraints actually makes the problems (both static and dynamic) easier to solve. Evidence for this can be seen in all pairs of almost identical problems where one has constraints and the other does not: pairs 1, 5, 6, 11, 12 and 13 in Figure 2 and pair 21 in Figure 3. The reason is that the presence of constraints helps increase selection pressures in the repair routine. This routine selects newly repaired solutions only if they are feasible. Obviously this type of selection is only meaningful in case there are constraints. In the unconstrained case, any repaired solutions will be accepted and hence there is no pressure for selection in the repair process, leading to slower convergence speed. As to the impact of different dynamic ranges (change severity, change frequency, population size) on algorithm performance, the results show that (i) generally the behaviour of algorithms and their relative strengths/weaknesses in comparison with other algorithms still remain roughly the same when the dynamic settings change; and (ii) as expected in most cases algorithms’ performance decrease when the conditions (magnitude of change, change frequency, population size) become tougher. A counter-intuitive exception is the case of SMES: the algorithm has better performance under higher frequency of change. This behaviour is shown due to the way the offline-

TABLE VIII AVERAGE NO . OF

FEASIBILITY CHECKINGS OF THE ALGORITHMS

Avg. feasibility Standard checkings deviation .GA_noElit 11,000 0 .RIGA_noElit 11,000 0 .hyperM_noElit 11,000 0 .GA_Elit 11,000 0 .RIGA_Elit 11,000 0 .hyperM_Elit 11,000 0 .GENOCOP 584,425 690,960 .dGENOCOP 465,757 757,307 .GA+Repair 759,506 1,836,586 .dRepairGA 861,078 1,395,875 .dRepairGA_OOR 855,920 1,411,417 .dRepairRIGA 861,084 1,367,003 .dRepairHyperMGA 1,004,236 1,399,275 .SMES 11,008 0 .SRES 11,027 0 .RIR_SRES 12,928 72 Algorithm

errors are calculated: they are added accumulatively as shown in Equation 3. Accordingly, if an algorithm converges to a local optimum, gets trapped there and fails to follow the moving optima, then the longer the number of generations it spends, the higher the offline error. This is exactly the case of SMES when the change frequency is low because the algorithm has enough time to converged on one optimum, lost its diversity and hence no longer be able to follow the moving global optimum. When the change frequency is high, SMES does not have enough time to converge to an optimum and hence it still has enough diversity to follow the moving global optimum. This is the reason why SMES perform betters in case the change frequency is high. This is another interesting example of situations where higher level of dynamics actually help the algorithm to perform better. Among the variations in dynamic settings, it seems that the variations in frequency of change affect performance the most, followed by variations in magnitude of changes. Variations in population size have the least impact on GA-based and repair-based algorithms, but has a stronger impact on ES-based algorithms, especially the existing CH methods like SMES and SRES. This observation conforms to the general rule-of-thumb to use large population sizes for ES-based algorithms. In summary, it seems that RIR-SRES is the best algorithms in the group in almost all settings: it provides high quality solutions (second highest overall score); it is robust and perform well in all types of problems (one of the only two able to achieve >0.90 scores in all types of problem); and it requires minimum additional feasibility checkings (only ˜12,928 compared to ˜11,000 of all other algorithms). The results also reveal some counter-intuitive situations where for certain algorithms (repair-based methods and SMES), higher levels of dynamics might make the algorithms perform better. V. W HAT

MAKES THE NEW ALGORITHMS WORK WELL ON DCOP S - A FURTHER ANALYSIS

A. Ability to retain diversified solutions that are infeasible The experimental results show that one of the reasons for the new algorithms to work better than existing DO

14

TABLE IX T HE AVERAGE percentage of selected infeasible individuals OVER ALL PROBLEMS FOR EACH TESTED ALGORITHM . T HE LAST ROW SHOWS THE AVERAGE percentage of infeasible areas OVER ALL PROBLEMS .

Fig. 2. The effect of twelve problem characteristics on existing DO algorithms (GA, RIGA, HyperM), existing CH algorithm (SMES, SRES, SRGA, GA+Repair), and the newly proposed RIR-SRES and dRepairRIGA (other variants are not included because their behaviours are roughly the same as those presented here). Each subplot represents algorithm performance in a pair of almost identical problems (one has a special characteristic and the other does not). The heights of the bars indicate how well the tested algorithms perform in solving the pair of problems (the higher the better). The larger the difference between the bar heights, the greater the impact of the corresponding DCOP characteristic. Algorithms’ performance (vertical axis) is evaluated based on the ratio between the base line error (the worst observed error) and the error of each algorithm to see how many times their performance is better (smaller) than the base line error. The title of each subplot represents the test case number (in brackets) followed by an abbreviation of the test case. Explanations for the abbreviations can be found in Table IV.

Fig. 3. The effect of the other eight different problem properties on the performance of GA, RIGA, HyperM, GA+Repair, dRepairGA and dRepairRIGA. Instruction to read this figure can be found in the caption of Figure 2.

and CH algorithms is that diversified but infeasible solutions are accepted with a higher percentage. This helps the new algorithms maintain a higher level of diversity and hence be able to react to changes in/near infeasible regions effectively. Evidence can be seen in Table IX where an analysis of the measure percentage of selected infeasible individuals [14] was carried out. Among the individuals selected for the next generation, this measure counts the percentage of those that are infeasible. The average score of this measure (over all tested generations) is then compared with the percentage of infeasible areas over the total search area of the landscape. If

Algorithms

Percent of infeasible solutions

.GA-elit .RIGA-elit .HyperM-elit .GA-noElit .RIGA-noElit .HyperM-noElit .dRepairGA .dRepairRIGA .dRepairHyperM .dRepairGA-OOR .dRepairRIGA-OOR .dRepairHyperM-OOR .GA+Repair .GENOCOP .dGENOCOP .SMES .SRES .RI_SRES .HM_SRES .RIR_SRES Percentage of infeasible areas

12.5% 26.3% 14.8% 41.8% 46.8% 42.8% 43.8% 44.9% 48.0% 46.0% 48.3% 50.8% 47.0% 40.8% 41.5% 25.6% 29.8% 47.0% 56.3% 60.9% 60.8%

the considered algorithm is able to treat infeasible diversified individuals and feasible diversified individuals on an equal basis (and hence to maintain diversity effectively), the two percentage values should be equal. As can be seen in Table IX, the new algorithms (RIR-SRES, dRepairRIGA and variants) have higher percentage of infeasible individuals (43.9%-60.9%) than their SMES/SRES/RIGAelit/HyperM-elit/GA-elit counterparts (12.5%-29.8%). This means that the new algorithms are able to maintain more diversified but infeasible individuals.7 It is interesting that although not equipped with an enhanced diversity maintaining mechanism such as RI or HM, dRepairGA and GA+Repair still achieve high percentages of infeasible individuals (same level as that of dRepairRIGA and dRepairHyperM). This confirms that the advantage is due to repair methods accepting both infeasible solutions and feasible solutions, provided that they can contribute to the search. The result also shows that SMES and SRES have low percentage of infeasible individuals although they were reported as able to accept infeasible solutions in static constrained problems. When combined with RI and HM (RI-SRES and HMSRES) and especially when also combined with the tracking mechanism (RIR-SRES), the score increases significantly. This result confirms that in order for SMES and SRES to work well on DCOPs, they need to maintain a higher level of diversity and also need to track the moving feasible regions. The results also confirm the effectiveness of the proposed schemes in improving SRES in this aspect. B. Ability to track the moving feasible regions Another reason for the new algorithms to perform better is that they are able to track the moving feasible regions. As 7 Although the non-elitism versions of GA/RIGA/HyperM also have a nearly as high percentage as dRepairGA-based algorithms, as already discussed in [14], in non-elitism versions, this high score comes with a trade-off of slower convergence which eventually decreases performance. The detailed results in [28] also show that non-elitism GA-based algorithms have much worse performance than their elitism counterparts.

15

Fig. 4. The optimal region tracking (ORT) scores that show how well the tested algorithms does in tracking the moving optimal feasible regions. Explanations for the abbreviations in the name of problem groups can be found in Table V (page 10). Note that the following groups: fF+fC and fF/dF+noC are not included because they are static or there is no constraint.

been briefly mentioned in Subsection II-D, to solve DCOPs effectively it is necessary to track the moving feasible regions because the global optimum always either move along with the feasible regions or appear in a new feasible region. To evaluate algorithm performance regarding this aspect, we analyse their ability in tracking the optimal region, i.e. the feasible region where the global optimum is currently in. Algorithms are evaluated using a new measure: the optimal region tracking measure (ORT), which is calculated as the ratio between (a) the number of generations at which the algorithm has at least one individual in the optimal region and (b) the total number of generations: ORT =

Pm Pp(i)

n (i, j) Pmj=1 p (i) i=1

i=1

(5)

where n (i, j) is equal to 1 if the algorithm has at least one individual in the optimal region at the jth generation of the ith change and is equal to 0 otherwise; m is the number of changes and p (i) is the number of generations at change period i. The ORT would be 1 in the best case where the tested algorithm is able to track the region containing the global optimum at every generation, and would be zero in case the algorithm is not able to track that region at all. The summarised ORT scores are given in Figure 48 . As can be seen in Figure 4, the ORT scores of existing DO and CH algorithms are lower than that of the new algorithms counterparts, meaning that the latter are better in tracking the moving feasible regions. The reason for existing CH to not work well is that they lack the necessary diversity and tracking mechanism to track the moving regions. Figure 4 shows that when SRES is equipped with a random-immigrant scheme (RI-SRES), its tracking performance improve signifantly. The tracking performance is increased further when the TrackFeasible() routine is introduced (RIR-SRES). 8 To avoid the graph to be too cluttered, we do not include the scores of the non-elitism versions of GA/HyperM/RIGA because they are much worse than the elitism versions. We also do not include the score of dRepairHyperM, the out-of-range (OOR) versions of dRepairGA-based and Genocop/dGenocop because these scores are similar to the score of dRepairRIGA.

The reason for existing DO algorithms to not work well in that in many cases the global optimum appears in a new area where the algorithms are not aware of, or due to that the global optimum does not gradually move but jumps from one region to another and hence it is difficult to track its movement. It is also interesting that even when being used without such diversity-maintaining mechanisms as RI or HM, GA+Repair can still track the moving feasible regions very well (its score is only slighly lower than that of dRepairRIGA). It means that the repair operator plays the major role in helping the algorithms to track the moving feasible regions. This also confirms the hypothesis in Subsection III-B5 that the original repair operator can be used to track moving feasible regions. C. Ability to adaptively balance feasibility and infeasibility Another reason that makes RIRSRES/dRepairRIGA/dGenocop better is that they are able to adaptively balance feasibility/infeasibility when changes occur. As mentioned previously in Subsection II-C, existing CH strategies like the SRES/SMES and GA+Repair/Genocop might suffer from the outdated problem and hence do not balance feasibility/infeasibility well on DCOPs. To analyse if the newly proposed mechanism can really help improve this aspect, the new algorithms were compared with their original counterparts using a measure proposed in [14]: the plot of number of feasible individuals in each disconnected feasible region to monitor the number of feasible individuals in each disconnected feasible region and the ratio of feasibility/infeasibility. If the balancing mechanism works well in the DCOP case, it should be able to manage a good distribution of individuals so that the better feasible regions should have more feasible individuals. The most suitable environments to test this behaviour are DCOPs with two disconnected feasible regions where the global optimum keeps switching from one region to another after each or some consecutive changes. They are the G24 1, G24 2, G24 3b, G24 4, G24 5, G24 6a, G24 6c, G24 6d, and G24 8b. All these problems belong to the group SwO in Figure 1, where the performance of existing CH algorithms significantly decreases compared to the stationary constrained case (fF, fC). In such problems, if the balancing mechanisms work well, at each change period the algorithm should be able to focus most feasible individuals on the region where the global optimum is currently in while still maintaining the same ratio of feasibility/infeasibility for diversity purpose. The plots of number of feasible individuals in each disconnected feasible region of GA+Repair vs dRepairGA and SRES vs RIR-SRES in three typical example functions: G24 1, G24 3b, and G24 4 are given in Figure 5.9 The figure shows that except GA+Repair in G24 3b, the CH methods are not able to focus most feasible individuals on the region where the global optimum is currently in. Instead, the majority of feasible individuals still remained in one region (region 2) where the global optimum firstly was before the 9 The measure scores of the tested algorithms in other problems are not shown due to the lack of space, but in all problems the algorithms behave the same as in the three examples given here.

16

dRepairRIGA, the OOR version achieved better performance in group of problems with optima in search boundaries (OISB, fF+noC, dF+noC) while still having almost equal performance in other group of problems. The same behaviour was also observed in the OOR versions of other repair-based algorithms (not shown due to limited space). E. The contribution of each component on the performance of the new algorithms

Fig. 5. How the balance strategies of GA+Repair/SRES (left) and dRepairGA/RIR-SRES (right) distributes their feasible individuals in the disconnected feasible regions of problems G24 1, G24 3b and G24 4. The problems tested in this figure are those with global optima switching between two disconnected feasible regions after each change or after a few changes. e.g. in G24 1 and G24 3b the location of the global optimum is as follows: change 1: region 2, change 2-4: region 1, change 5: region 2 and so on.

changes happen. These results show that, due to their outdated information and strategy, the SRES and GA+Repair algorithms are not able to follow the switching optimum well. Figure 5 also shows that the drawback above has been resolved in the newly proposed RIR-SRES and dRepairGA. The presence of the routines proposed in Subsection III-D helps the new algorithms focus most of their feasible individuals to the region where the global optimum has moved into whenever a change happens while still maintaining the same ratio of feasibility/infeasibility for diversity purpose in all problems.

Because the newly proposed algorithms combine different components such as the routines DetectChange, UpdateReferencePop, UpdateSearchPop, TrackFeasible; the RIGA/HyperM mutation strategies; and the out-of-range mechanism OOR, it is of interest to analyse the contribution of each component on the performance of the complete algorithms. In order to carry out the analysis, we firstly integrated each of the above routines with the original GA+Repair, Genocop III and SRES to create a corresponding modified variation, then compared this variation with the original algorithms. The following combinations were tested: • SRES/GA+Repair/Genocop + wNU (No Update): Original versions of the CH algorithms. • SRES/GA+Repair/Genocop + wUSP (Update Search Population per change): Change detection is enabled. UpdateSearchPop routine is carried out adaptively whenever a change is detected. • SRES/GA+Repair/Genocop + wUSPwURP (Update Search Population and Reference Population per change): Change detection is enabled. Both the UpdateSearchPop and UpdateReferecePop routines are carried out adaptively whenever a change is detected. • SRES/GA+Repair/Genocop + RI/HM: The algorithms are hybridised with a random-immigrant or hyper-mutation using the mechanisms described previously. • SRES + wUPP (Update the Populations Periodically): This combination is applicable to SRES only: The populations are updated not only whenever a change is detected but also periodically. Detailed comparisons on groups of problems for GA+Repair-based, Genocop-based, and SRES-based algorithms are given in Figure 6. The results show that each of the proposed components does have its own contribution to improve the performance of the algorithms. In addition, random-immigrant seems to be more robust agains different types of DCOPs than hyper-mutation. The reason is that (a) hyper-mutation might not be able to maintain a high-enough level of diversity throughout the search to deal with the the more sudden changes found in DCOPs and (b) hyper-mutation might not be able to react to changes that lead to newly appearing optima without changing existing optima (NAO), as in the NAO case in Figure 6.

D. Ability to find optima in search boundary faster The ability to search out of range (OOR) of dRepairRIGA OOR, dRepairHydRepairGA OOR, perM OOR also brings them advantages in solving problems where the global optima is in search boundaries. Figure 6 shows an example where compared to the original

VI. A

DETAILED ANALYSIS ON PARAMETER VALUES

It is unknown if the default or best reported parameter values used in Subsection IV-C are most suitable for solving DCOPs. To answer this question, in this section a detailed analysis is carried out to investigate how different parameter values

17

0.95

0.9

Normalised score

Normalised score

0.95

0.85 0.8

RIGA dRepairRIGA

0.75

RI_SRES SRRIGA

0.7 0.2

0.3

0.4

0.6

0.9 0.85 0.8

HyperM dRepairHyperM HM_SRES SRHyperM

0.75 0.7

0.8

Random immigrant rate (a)

1

0.15

0.3

0.5

0.6

0.8

1

Hyper-mutation rate (b)

Fig. 7. The impact of different hyper-mutation rates and random-immigrant rates on algorithm performance.

RIGA and HyperM are still worse or not much better than restart GA and random search. This suggests that, for existing DO algorithms like HyperM and RIGA, the levels of change severity in the search spaces are so large that DO techniques such as RI or HM seem not to bring much benefit. This suggestion, however, seems to be contradicted with the medium level of change used in this experiment. This leads to the second interesting observation: although the objective and constraint functions only change gradually, when they are combined to create a DCOP, they can create much more sudden changes in the combined constrained search space.

Fig. 6. How each of the newly proposed components improves the performance of GA+Repair, SRES, and Genocop III in solving DCOPs.

would affect the performance of all tested algorithms. The analysis will provide suggestions on how to choose the best parameter values to solve not only DCOPs but also other types of problems including the unconstrained and static cases10 . A. Hyper-mutation rate and random-immigrant rate Here we analyse the impact of different hyper-mutation rate (PH ) and random-immigrant rate (PR ) on algorithm performance. Algorithms that use PR (RIGA/dRepairRIGA/RISRES/SRRIGA) and PH (HyperM/dRepairHyperM/HMSRES/SRHyperM) are tested in all different ranges of PH and PR . In addition, because HyperM (PH = 1.0)=restart GA+elitism and RIGA(PR = 1.0)=random search+elitism, it is possible to compare the performance of the tested algorithms with random search and restart GA. All other parameters are set using the default/medium values set out in Table VI (page 11).The results are summarised in Figure 7. Figure 7 shows some interesting observations. First, GAbased algorithms need very high mutation rates (PR and PH ∼0.8-1.0 ˜very high levels of diversity) but suprprisingly 10 For algorithms with both elitism and non-elitism versions as GA/RIGA/HyperM, in this section we will only present the results of the elitism versions because in our previous experiments the elitism versions have better performance than the non-elitism versions.

This fact, however, does not mean that there is no way to do better: the newly proposed algorithms all perform much better than both RIGA/HyperM and restart GA/random-search. Especially, SRRIGA/SRHyperM only need a mutation rate of 0.4-0.5 and RI SRES/HM SRES only need a mutation rate of 0.3 to perform best. The reason for the new algorithms to use lower mutation rates more effectively is that, as already analysed in Subsection V-A, by accepting both feasible and infeasible solutions, they are able to retain more diversified individuals to deal with environmental dynamics. As to the impact of selection scheme, comparing RIGA/HyperM (non-linear ranking selection) with SRRIGA/SRHyperM (stochastic ranking selection), the use of SR helps the algorithms achieved best results with much lower PR or PH rates. The reason is that SR accepts both feasible and infeasible solutions in a nearly equal basis and hence is able to utilise dieversified individuals better. As to the impact of underlying evolutionary schemes, comparing SRRIGA/SRHyperM (GA scheme) with RI-SRES/HMSRES (ES scheme), ES-based versions require lower PR or PH rates to achieve the best performance. This difference might be due to that while in GA only a proportion of the population are mutated, in ES all selected individuals are mutated and hence introduce more diversity. In sum, the results suggest that to solve the DCOPs effectively, higher mutation/random-immigrant rates are preferable because the levels of change severity in DCOPs are high. In addition, the results show that the suitable mutation rates highly depend on the selection and evolutionary schemes to be used. For selection schemes like stochastic ranking and evolutionary schemes like ES, the algorithms generally need considerably lower PR and PH rates.

18

Normalised score

1

Different replacement ratios

0.95 0.9

0.85 0.8

Genocop dGenocop dRepairGA

0.75 0 0.05 0.25 0.5 0.8 1 X axis: rratio (replacement ratio)

Fig. 8. The impact of different base-mutation and crossover rates on algorithm performance.

B. Base mutation rate The algorithms selected for this analysis are GA, dRepairGA, HyperM, dRepairHyperM, RIGA and dRepairRIGA11 . At the base mutation rate Pb = 1.0, GA-based algorithms (GA/RIGA/HyperM) become random search+elitism. Here we also compared the algorithms with restart GA. Figure 8 shows that all algorithms have the same general behaviours: performance is significantly worse at the smallest mutation rate Pb , then it increases when Pb increases until the performance reaches its peak at mid-range Pb values, and finally the performance decreases again (but only less slightly) when Pb reaches the extreme high values (Pb > 0.7). More detailed analysis on the results reveals some interesting observations as follows. First, the top normalised scores are gained at high mutation rates (larger than 0.2). Second, GA/RIGA/HM do not perform significantly better (and even worse in certain cases) than random search and restart GA for the same reason as in the case of PR and PH . Third, new repair-based algorithms always perform significantly better than existing algorithms given the same base mutation rate. The better performance of dRepair-based algorithms might be due to their ”divide-and-conquer” approach: tracking the moving unconstrained optima and tracking the moving feasible regions separately. Because individual changes in each of the constraint and objective function are less severe than the combination change in the search space, this approach makes it easier to handle environmental dynamics. Some other observations from the results are (1) The impact of changing the base mutation become less significant with any rate ≥ 0.1; (2) Algorithms with diversity-maintaining strategies (RIGA/HyperM/dRepairRIGA/dRepairHyperM) are more robust to deal with the variation in base mutation rate. C. Crossover rate Figure 8 shows how changing crossover rates affect GAbased algorithms when other parameters are kept constant12 . Some observations can be seen. First, the proposed GAbased algorithms have better overall scores than existing DO and CH algorithms in all cases regardless of the crossover rate. 11 Algorithms

like Genocop/dGenocop were not tested because, due to their special design, changing the mutation rate Pb would affect the rate of nine other specialised operators. In this case, any impact in performance might be caused by not only the change in mutation rate but also by the consequent changes in the probability rates of other specialised operators. 12 Algorithms like Genocop/dGenocop were not tested due to the same reason stated in note 11.

Fig. 9. The impact of different replacement ratio on the performance of repair-based algorithm

Second, the higher the crossover rate, the worse the performance of repair-based algorithms (they perform best where there is no crossover!). It seems that the use of the normal arithmetic crossover is not suitable for these algorithms. The repair operator (step 2a of Algorithm 1, page 4) is very similar to the normal arithmetic crossover except that (i) the parent must be from both S and R, and (ii) the offsprings are only accepted if they are feasible. Due to these two differences, in the repair operator the search outcome is biased so that individuals are constantly pushed toward feasible regions to track the moving feasible regions. Such a bias cannot be provided by the original arithmetic crossover. As a result, when the repair operator is combined with the crossover operator, we suspect that the effect of tracking moving feasible regions might be reduced or even be eliminated. Third, a crossover rate greater than or equal to 0.1 might be a good choice for existing GA-based DO algorithms to solve DCOPs. In addition, compared to HyperM and GA, RIGA is much less affected by the variation of the crossover rate. D. Replacement ratio and the effect of Lamarckian evolution (for repair-based algorithms) The replacement ratio determines the percentage of repaired individuals (the individual z in the routine Repair, Algorithm 1) to replace individuals in the population. Any non-zero value means that the algorithm follows a Lamarckian-evolution approach because the outcome of learning (repaired individuals in this case) can be applied directly to the evolution process by changing the chromosomes of individuals. In case the value is zero, the algorithm follows a Baldwinian-evolution approach because learning can only affects evolution indirectly. Figure 9 shows that Lamarckian evolution has little effect on the performance of the repair-based algorithms except that the performance of these algorithms drop when the replacement ratio becomes 100%. For this reason, it might be possible to fix this parameter to zero when solving DCOPs to simplify the parameter tuning process. E. Size of reference population and period to update search population (for RIR-SRES) As described in Algorithm 5, the routine TrackFeasible of RIR-SRES requires two parameters: ωU - the period to update S with indivdiuals from R , and mR - the size of R. Figure 10 shows that when we change these two values while keeping

19

Using different RefPop sizes

Using different update periods

0.950

0.950

0.945

0.945

0.940

0.940 Normalised score

0.935

Normalised score

0.935

40

30

15

20

10 15 20 25

5

7

10

5

1

3

3

1

N

0

ev er

0.930

0.930 Size of reference population

Fig. 10.

Period (no. of gens) to update the RefPop

The impact of changing the values of ωU (left) and mR (right)

all other parameters at medium level as set in Table VI, any positive value of these two parameters will bring (averagely) nearly the same improvement to the algorithm performance. The standard deviations (not shown) of the scores at different values are also similar (±0.061 − 0.074). Detailed results (not presented due to lack of space) show that for some problems (e.g. G24 4, G24 7, G24 8b), increasing ωU increases the performance while for some others (e.g. G24 2, G24 3, G24 6a/b/c/d) the reverse thing happens. Similarly, for some problems like G24 8b and G24 1, increasing mR increases the performance while for some others e.g. G24 3, G24 7 the reverse thing happens. The result suggests that these two parameters can fixed at the previously chosen values ωU = 3 and mR = 5 to simplify the parameter tuning process without affecting performance. VII. C ONCLUSION

AND FUTURE RESEARCH

A. Summary of contributions and findings This paper has made the following contributions. First, a new approach has been proposed to solving DCOPs (with superior results to the tested state-of-the-art CH and the tested DO algorithms): combining existing DO and CH techniques to handle objective-function changes and constraint-function changes separately and differently. Specifically, a new method was developed to track the moving feasible regions (either by modifying the existing repair method or by adding a new reference population for tracking) and it was combined with existing DO techniques (random-immigrant and hypermutation) to handle objective-function changes. We also used different techniques to detect objective-function changes and constraint-function changes separately and differently. This newly proposed approach was applied to several different EAs: SRES, GA+stochastic ranking, GA+Repair, and Genocop III to create new algorithms named RIR-SRES, dRepairRIGA, dGenocop and variants. The test results in 18 benchmark problems confirms the advantages of the new approach: the new algorithms are able to significantly outperform existing DO and state-of-the-art CH algorithms in solving DCOPs while still maintaining equal or better overall performance in solving other groups of problems except in the static cases. Because the new mechanisms are designed to improve ubiquitous DO and CH strategies such as the stochastic ranking, random-immigrant, hyper-mutation and repair, it is expected that the usefulness of the mechanisms can be generalised to any algorithm that use these strategies.

Second, this paper offers a deeper understanding of the behaviours and characteristics of DCOPs. On the one hand, the analyses show that changes in DCOPs are usually more sudden than the unconstrained case even if the elementary changes in objective function and constraint functions are small. Such sudden changes might make existing DO algorithms like RIGA and HyperM perform worse than restart and random search. On the other hand, the analyses also show that in DCOPs the presence of dynamics and constraints might not always make the problems more difficult as we intuitively expect. For example, the higher the change frequency, the better the performance of algorithms that quickly lose their diversity like SMES. In addition, for the newly proposed repair-based algorithms, the presence of constraints actually makes some of the tested problems easier to solve. Third, this paper gives an insight into the influence of different algorithmic components and parameter values on algorithm performance on DCOPs. The paper also provides a guideline for choosing the ideal parameter settings for each algorithm to solve DCOPs. Fourth, two new performance measures were proposed to provide in-depth analysis on algorithm performance on DCOPs. B. Advantages of the proposed methods The first advantage of the new algorithms is that they overcome the drawbacks of DO and CH strategies in solving DCOPs. This helps them perform significantly better than the tested state-of-the-art algorithms in the tested DCOPs. The second advantage is that they are robust. Among the tested problems the new algorithms, especially RIR-SRES and dGenocop, are able to work well not only on DCOPs but on all other types of the tested problems. The third advantage is their simplicity and generality. The proposed methods are very straightforward, easy to implement, and can be hybridised with any population-based EA. In this paper we have integrated them with ES, GA and Genocop III, all with good results. The fourth advantage is the ability to work without the need of choosing many parameters. The only mandatory parameters are the population size and the mutation/hyper-mutation rate or random-immigrant rate. Optional parameters are the replacement ratio in the case of dRepairRIGA/dGenocop, and the update period ωU and the reference population size mR in the case of RIR-SRES but the algorithms are not sensitive to changes in these parameters. There is also a crossover rate parameter in the case of dRepairRIGA and variants but our detailed analysis suggests that for the newly proposed algorithms no crossover would give the best results. C. Shortcomings of the proposed methods For RIR-SRES, one minor disadvantage is that the algorithm become less effective in solving static problems, comparing to the original state-of-the-art SRES. However, the difference in performance is not significant and experimental results showed that RIR-SRES is still one of the top solvers in solving this group of problem. In addition, the algorithm needs some

20

additional feasibility checkings. However, as shown in Figure VIII, this number is insignificant. For repair-based methods (dGenocop, dRepairRIGA and variants), the biggest disadvantage is that they require a considerable number of feasibility checkings. Due to this reason, repair-based methods in general might not be suitable for solving problems with very expensive constraint functions and problems with very small feasible areas (because the repairbased methods would need to take a lot of feasibility checkings to find a feasible solution). In such cases, RIR-SRES is a better alternative because it can give similar performance with much fewer aditional feasibility checkings. D. Future research One of the future directions is to test the proposed algorithms on large-scale DCOPs. We are also interested in doing theoretical studies to confirm the experimental analyses of why the proposed algorithms are robust against different types of problems and why they are suitable for solving DCOPs. ACKNOWLEDGMENT The authors thank the anonymous reviewers for their insightful comments, Prof. J. Branke, Prof. Y. Jin, Dr. S. Yang, C. Li, Dr. P. Rohlfshagen, Dr. T. Ray, Dr. J. Rowe, Dr. A. Kaban and H. Nguyen for their fruitful discussions, and H. Maxwell and R. Gidwaney from EISU for their help in academic writing styles. The programs used in this paper are developed from the open source code provided by Ken Williams [29] and Zbigniew Michalewicz [4]. R EFERENCES [1] T. P. Runarsson and X. Yao, “Stochastic ranking for constrained evolutionary optimization,” IEEE Trans. Evolutionary Computation, vol. 4, no. 3, pp. 284–294, 2000. [2] ——, “Search biases in constrained evolutionary optimization,” IEEE Transactions on Systems, Man, and Cybernetics, Part C, vol. 35, no. 2, pp. 233–243, 2005. [3] Z. Michalewicz and G. Nazhiyath, “Genocop III: A co-evolutionary algorithm for numerical optimization with nonlinear constraints,” in Proceedings of the Second IEEE International Conference on Evolutionary Computation, D. B. Fogel, Ed. Piscataway, New Jersey: IEEE Press, 1995, pp. 647–651. [4] Z. Michalewicz, “The second version of Genocop III: a system which handles also nonlinear constraints,” School of Computer Science, University of Adelaide, [Accessed February 2009]. [Online]. Available: {http://www.cs.adelaide.edu.au/∼ zbyszek/EvolSyst/gcopIII10.tar.Z} [5] J. J. Grefenstette, “Genetic algorithms for changing environments,” in Parallel Problem Solving from Nature 2, R. Maenner and B. Manderick, Eds. North Holland, 1992, pp. 137–144. [6] H. G. Cobb, “An investigation into the use of hypermutation as an adaptive operator in genetic algorithms having continuouis, time-dependent nonstationary environments,” Naval Research Laboratory, Washington, USA, Technical Report AIC-90-001, 1990. [7] T. T. Nguyen, “Continuous Dynamic Optimisation Using Evolutionary Algorithms,” Ph.D. dissertation, School of Computer Science, University of Birmingham, January 2011, http://etheses.bham.ac.uk/1296 and http: //www.staff.ljmu.ac.uk/enrtngu1/theses/phd thesis nguyen.pdf. [8] M. Andrews and A. L. Tuson, “Dynamic optimisation: A practitioner requirements study,” in Proceedings of the The 24th Annual Workshop of the UK Planning and Scheduling Special Interest Group (PlanSIG 2005), London, UK, 2005. [9] Y. Wang and M. Wineberg, “Estimation of evolvability genetic algorithm and dynamic environments,” Genetic Programming and Evolvable Machines, vol. 7, no. 4, pp. 355–382, 2006.

[10] D. M. Prata, E. L. Lima, and J. C. Pinto, “Simultaneous data reconciliation and parameter estimation in bulk polypropylene polymerizations in real time,” Macromolecular Symposia, vol. 243, no. 1, pp. 91–103, 2006. [11] L. Araujo and J. J. Merelo, “A genetic algorithm for dynamic modelling and prediction of activity in document streams,” in GECCO ’07: Proceedings of the 9th annual conference on Genetic and evolutionary computation. New York, NY, USA: ACM, 2007, pp. 1896–1903. [12] P. Tawdross, S. K. Lakshmanan, and A. Konig, “Intrinsic evolution of predictable behavior evolvable hardware in dynamic environment,” in HIS ’06: Proceedings of the Sixth International Conference on Hybrid Intelligent Systems. Washington, DC, USA: IEEE Computer Society, 2006, p. 60. [13] M. Rocha, J. Neves, and A. Veloso, “Evolutionary algorithms for static and dynamic optimization of fed-batch fermentation processes,” in Adaptive and Natural Computing Algorithms, B. Ribeiro et al., Eds. Springer, 2005, pp. 288–291. [14] T. T. Nguyen and X. Yao, “Continuous dynamic constrained optimisation - the challenges,” IEEE Transactions on Evolutionary Computation (submitted), 2010, [online]. [Online]. Available: {\url{http://www.cs. bham.ac.uk/∼ txn/Papers/Nguyen Yao DCOP.pdf}} [15] Y. Jin and J. Branke, “Evolutionary optimization in uncertain environments—a survey,” IEEE Transactions on Evolutionary Computation, vol. 9, no. 3, pp. 303–317, 2005. [16] E. Mezura-Montes and C. A. C. Coello, “A simple multimembered evolution strategy to solve constrained optimization problems,” IEEE Trans. Evolutionary Computation, vol. 9, no. 1, pp. 1–17, 2005. [17] T. T. Nguyen and X. Yao, “Benchmarking and solving dynamic constrained problems,” in Proceedings of the IEEE Congress on Evolutionary Computation CEC2009. IEEE Press, May 2009, pp. 690–697. [18] H. K. Singh, A. Isaacs, T. T. Nguyen, T. Ray, and X. Yao, “Performance of infeasibility driven evolutionary algorithm (IDEA) on constrained dynamic single objective optimization problems,” in Proceedings of the IEEE Congress on Evolutionary Computation CEC2009. Trondheim, Norway,: IEEE Press, May 2009, pp. 3127–3134. [19] H. Richter, “Memory design for constrained dynamic optimization problems,” in Proceedings of the European Conference on the Applications of Evolutionary Computation 2010, EvoApplications 2010, ser. Lecture Notes in Computer Science, vol. 6024. Springer, 2010, pp. 552–561. [20] H. Richter and S. Yang, “Learning behavior in abstract memory schemes for dynamic optimization problems,” Soft Computing, vol. 13, no. 12, pp. 1163–1173, 2009. [21] H. Richter and F. Dietel, “Solving dynamic constrained optimization problems with asynchronous change pattern,” in Applications of Evolutionary Computation, ser. Lecture Notes in Computer Science, C. Di Chio, S. Cagnoni, C. Cotta, M. Ebner, A. Ekrt, A. Esparcia-Alczar, J. Merelo, F. Neri, M. Preuss, H. Richter, J. Togelius, and G. Yannakakis, Eds., vol. 6624. Springer Berlin / Heidelberg, 2011, pp. 334–343. [22] K. A. Morales and C. Quezada, “A universal eclectic genetic algorithm for constrained optimization,” in Proceedings 6th European Congress on Intelligent & Soft Computing, EUFIT’98, 1998, pp. 518–522. [23] C. Floudas, P. Pardalos, C. Adjiman, W. Esposito, Z. Gumus, S. Harding, J. Klepeis, C. Meyer, and C. Schweiger, Handbook of Test Problems in Local and Global Optimization, ser. Noncovex Optimization and Its Applications. Kluwer Academic Publishers, 1999, vol. 33. [24] H. Richter, “Detecting change in dynamic fitness landscapes,” in Proceedings of the IEEE 2009 Congress on Evolutionary Computation, CEC2009, 2009, pp. 1613–1620. [25] J. Branke, Evolutionary Optimization in Dynamic Environments. Kluwer, 2001. [26] D. Ayvaz, H. Topcuoglu, and F. Gurgen, “A comparative study of evolutionary optimization techniques in dynamic environments,” in GECCO ’06: Proceedings of the 8th annual conference on Genetic and evolutionary computation. New York, NY, USA: ACM, 2006, pp. 1397– 1398. [27] J. Branke and H. Schmeck, “Designing evolutionary algorithms for dynamic optimization problems,” in Theory and Application of Evolutionary Computation: Recent Trends, S. Tsutsui and A. Ghosh, Eds. Springer, 2003, pp. 239–262. [28] T. T. Nguyen and X. Yao, “Detailed experimental results of GA, RIGA, HyperM and GA+Repair on the G24 set of benchmark problems,” School of Computer Science, University of Birmingham, Tech. Rep., 2010, online. [Online]. Available: \url{http://www.cs.bham.ac.uk/∼ txn/ Papers/DCOP fulldata.pdf} [29] K. P. Williams, “Simple genetic algorithm (SGA) source code (in C),” [Accessed December 2008]. [Online]. Available: http: //www.kenwilliams.org.uk/code/ga2.c