CMA-ES with Restarts for Solving CEC 2013 Benchmark ... - CiteSeerX

CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems Ilya Loshchilov Laboratory of Intelligent Systems ´ Ecole Polytechnique Fédérale de Lausanne, Switzerland Email: [email protected] Abstract—This paper investigates the performance of 6 versions of Covariance Matrix Adaptation Evolution Strategy (CMAES) with restarts on a set of 28 noiseless optimization problems (including 23 multi-modal ones) designed for the special session on real-parameter optimization of CEC 2013. The experimental validation of the restart strategies shows that: i). the versions of CMA-ES with weighted active covariance matrix update outperform the original versions of CMA-ES, especially on illconditioned problems; ii). the original restart strategies with increasing population size (IPOP) are usually outperformed by the bi-population restart strategies where the initial mutation stepsize is also varied; iii). the recently proposed alternative restart strategies for CMA-ES demonstrate a competitive performance and are ranked first w.r.t. the proportion of function-target pairs solved after the full run on all 10-, 30- and 50-dimensional problems.

I.

II presents the main principles of the CMA-ES algorithm. Section III describes the restart strategies of CMA-ES. Section IV explains the experimental procedure and comments the experimental results. Section V concludes the paper with a discussion and some perspectives for further research. II.

T HE (µ/µw , λ)-CMA-ES

The CMA-ES algorithm [8], [7] optimizes an objective function f : x ∈ Rn → f (x) ∈ R by sampling λ candidate solutions from a multivariate normal distribution. It exploits the best µ solutions out of the λ ones to adaptively estimate the local covariance matrix of the objective function, in order to increase the probability of successful samples in the next iteration. More formally, at iteration t, (µ/µw , λ)-CMA-ES samples λ individuals (k = 1 . . . λ) according to

I NTRODUCTION

The Covariance Matrix Adaptation Evolution Strategy (CMA-ES) proposed by [8], [7] has become a standard for continuous black-box evolutionary optimization. The main advantage of CMA-ES over classical Evolution Strategies comes from the use of correlated mutations instead of axis-parallel ones. The adaptation of the covariance matrix C allows to steadily learn appropriate mutation distribution and increase the probability of repeating the successful search steps. However, there are several properties of black-box optimization problems which may lead to a premature convergence of CMA-ES, among the most common are multi-modality and uncertainty. To increase the probability of finding the global optima, IPOP-CMA-ES [2] and BIPOP-CMA-ES [4] restart strategies for CMA-ES have been proposed. The IPOPCMA-ES was ranked first on the continuous optimization benchmark at CEC 2005 [3]; and BIPOP-CMA-ES showed the best results together with IPOP-CMA-ES on the blackbox optimization benchmark (BBOB) in 2009 and 2010 [1]. Later, alternative restart strategies for CMA-ES proposed in [12] demonstrated an even more competitive performance on some of multi-modal functions during the BBOB 2012. The recently proposed weighted active covariance matrix update of CMA-ES [11], [9] is also a competitive alternative to the original update procedure, it allows to substantially improve the performance both on unimodal and multi-modal functions [9]. This paper focuses on analyzing the performance of the restart strategies of CMA-ES with the original and weighted active covariance matrix updates on the CEC 2013 benchmark test [10]. The remainder of this paper is organized as follows. Section

(t+1)

xk

2 = N m(t) , σ (t) C(t) = m(t) +σ (t) ·N 0, C(t) , (1)

where m(t) denotes the mean of a normally distributed random vector, C(t) is the covariance matrix and σ (t) is the mutation step-size. These λ individuals are evaluated and ranked. The mean of the distribution is updated and set to the weighted sum of the Pµ (t) best µ individuals m(t+1) = i=1 wi xi:λ , with wi > 0 for Pas µ i = 1 . . . µ and i=1 wi = 1, where index i : λ denotes the i-th best individual after the objective function. In the original CMA-ES the information about the remaining (worst λ − µ) solutions is used only implicitly during the selection process. However, it has been shown in [11] that the information from the worst solutions also can be used to reduce the variance of the mutation distribution in unpromising directions. The corresponding active (µ/µI , λ)-CMA-ES algorithm demonstrates a performance gain up to a factor of 2 without loss of performance on any of tested functions in [11]. Later, the active update of (µ/µI , λ)-CMA-ES was extended to the weighted case of (µ/µW , λ)-CMA-ES, where wi > wi+1 for i = 1 . . . λ−1. This weighted active (µ/µW , λ)-CMA-ES (also referred to as aCMA-ES) was implemented in the IPOP regime of restarts as IPOP-aCMA-ES and demonstrated improvements up to a factor of 2 on a set of noiseless and noisy functions from the BBOB [9]. More formally, the active CMA-ES only differs from the original CMA-ES in the adaptation of the covariance matrix C(t) . Like for CMA-ES, the covariance matrixt is Pµ xi:λ −m computed from the best µ solutions, C+ w × i µ = i=1 σt

(xi:λ −mt )T σt

. The main novelty is to exploit the worst solutions Pµ−1 wi+1 yλ−i:λ yTλ−i:λ , where yλ−i:λ = to compute C− µ = i=0

t −1/2

(xλ−µ+1+i:λ −mt )

C

xλ−i:λ −mt . σt

× The covariance matrix k C t −1/2 (xλ−i:λ −mt )k estimation of these worst solutions is used to decrease the variance of the mutation distribution along these directions: t Ct+1 = (1 − c1 − cµ + c− α− old )C + + − − − t+1 t+1 T + (cµ + c (1 − α− +c1 pc pc old ))Cµ − c Cµ ,

(2)

where pt+1 is adapted along the evolution path and coefc ficients c1 , cµ , c− and α− old are defined such that c1 + cµ − c− α− ≤ 1. The interested reader is referred to [7], [9] for a old more detailed description of these algorithms. A potential issue of the active update is that the positive definiteness of the covariance matrix cannot be guaranteed anymore, that may result in algorithmic instability. According to [12], this issue is not observed on the BBOB benchmark suite [5]. In our experiments with the CEC 2013 benchmark suite this issue is also never observed. III.

R ESTART S TRATEGIES

FOR

CMA-ES

A. Preliminary Analysis The CMA-ES algorithm is a local search optimizer and its default population size λdef ault has been tuned for unimodal functions. On multi-modal functions, however, it can get stuck in local optima and the convergence to global optima is not guaranteed. Various approaches to increase the probability of finding global optima have been proposed, many of them belong to i). niching approaches and ii). restart strategies.

B. The IPOP-CMA-ES and IPOP-aCMA-ES As mentioned, [6] demonstrated that increasing the population size improves the performance of CMA-ES on multimodal functions. The authors of [6] suggested a restart strategy for CMA-ES with successively increasing population size. Such an algorithm was later introduced in [2] as IPOP-CMAES. IPOP-CMA-ES only aims at increasing the population size λ. Each time at least one of the stopping criteria is met by the CMA-ES, it launches a new CMA-ES with population restart λdef ault , where irestart is the index of the size λ = ρiinc restart and λdef ault is the default population size. Factor ρinc must be not too large to avoid ”overjumping” some possibly optimal population size λ∗ ; in [2] it is set to ρinc = 2 that in certain cases allows to keep a potential loss in terms of function evaluations (compared to the “oracle“ restart strategy which would directly set the population size to the optimal value λ∗ ) by about a factor of 2. The active version of IPOP-CMA-ES (IPOP-aCMA-ES) has been proposed in [9]. C. The BIPOP-CMA-ES and BIPOP-aCMA-ES In BIPOP-CMA-ES [4] after the first single run with default population size, the algorithm is restarted in one of two possible regimes and account the budget of function evaluations spent in the corresponding regime. Each time the algorithm is restarted, the regime with smallest budget used so far is used. Under the first regime the population size is doubled as λlarge = 2irestart λdef ault in each restart irestart and use 0 0 = σdef some fixed initial step-size σlarge ault . This regime corresponds to the IPOP-CMA-ES.

A representative approach of the first category is the CMAES with the fitness sharing [15], where the niche radius is adapted during the search that allows to keep several running individual CMA-ES instances on a certain distance from each other, and, thus, maintain some diversity. Another example is the NBC-CMA-ES algorithm [14] with the niching via Nearest-Better Clustering (NBC) which is employing a radiusfree basin identification method. In this approach, the niches are dynamically identified and the corresponding points are used to form populations for individual CMA-ES instances. According to [14], for very highly multi-modal functions, the effort invested into the coordination of local searches often does not pay off as it becomes almost impossible to identify enough basins of attraction to obtain an advantage over uncoordinated restarts.

Under the second regime the CMA-ES is restarted with 0 some small population size λsmall and step-size σsmall , where λsmall is set to

The second category of restart strategies is not that different from the first one since restarts also can be viewed as a parallelized search, but rather in the time than in space [14]. A milestone paper [6] investigated the probability of reaching the global optimum (and the overall number of function evaluations needed to do so) w.r.t. the population size of CMA-ES. The analysis of empirical results demonstrated that, indeed, this probability is very sensitive to the population size and that the default population size of CMA-ES is rather too small. The restart strategies described in the following sections are inspired by an idea of exploring CMA-ES hyper-parameters such as the population size and the initial step-size.

The active version of BIPOP-CMA-ES (BIPOP-aCMA-ES) has been proposed in [12].

U[0,1]2 λlarge λsmall = λdef ault 12 λdef , ault

(3)

Here U [0, 1] denotes independent uniformly distributed numbers in [0, 1] and λsmall ∈ [λdef ault , λ/2]. The initial step0 0 −2U[0,1] size is set to σsmall = σdef . ault × 10 In each restart, BIPOP-CMA-ES selects the restart regime with less function evaluations used so far. Since the second regime uses a smaller population size, it is therefore launched more often.

D. The NIPOP-aCMA-ES The NIPOP-aCMA-ES [12] is an alternative restart strategy to the IPOP-aCMA-ES, where in addition to increasing of population size in each restart, the initial step-size is also decreased by some factor kσdec . In [12], this factor is set to kσdec = 1.6 such that σ value after 9 restarts (the default maximum number of restarts in BIPOP-aCMA-ES) roughly corresponds to the minimum possible initial σ = 10−2 σdef ault

F2 in 10-D, 30-D and 50-D

0.7 0.6 0.5 0.4 0.3 IPOP-CMA BIPOP-CMA IPOP-aCMA NIPOP-aCMA BIPOP-aCMA NBIPOP-aCMA

0.2 0.1 0 1 10

2

10

3

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 2 10

4

10

0.8

IPOP-CMA BIPOP-CMA IPOP-aCMA NIPOP-aCMA BIPOP-aCMA NBIPOP-aCMA

10

# evaluations / dimension

3

4

10

10


F4 in 10-D, 30-D and 50-D

1 0.9 0.8 0.7

Proportion of function-target pairs

0.8

F3 in 10-D, 30-D and 50-D

1 0.9




F1 in 10-D, 30-D and 50-D 1 0.9


0.6 0.5 0.4 0.3 0.2 0.1 0 2 10

3

4

10

10

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 IPOP-CMA BIPOP-CMA IPOP-aCMA NIPOP-aCMA BIPOP-aCMA NBIPOP-aCMA

0.2 0.1 0 2 10


3

10

4

10


Fig. 2. Empirical cumulative distribution of the number of objective function evaluations divided by dimension (FEvals/D) for 300 function-target pairs in 10[−1..4] (100 pairs for each of dimensions 10, 30 and 50) for F1, F2, F3.

The NBIPOP-aCMA-ES typically outperforms IPOPaCMA-ES, BIPOP-aCMA-ES and NIPOP-aCMA-ES on the BBOB noiseless problems [13], especially in larger dimensions.

0

σ/σ

default

10

All the above described algorithms can be viewed as some search algorithms in the space of hyper-parameters λ and σ. The typical patterns of these search algorithms are shown in Fig. 1.

−1

10

IV.

−2

10

0

10

1

2

10

10

3

10

λ/λ

default

Fig. 1. An illustration of λ and σ hyper-parameters distribution for 9 restarts of IPOP-aCMA-ES (◦), BIPOP-aCMA-ES (◦ and · for 10 runs), NIPOPaCMA-ES () and NBIPOP-aCMA-ES ( and many △ for λ/λdef ault = 1, σ/σdef ault ∈ [10−2 , 100 ]). The first run of all algorithms corresponds to the point with λ/λdef ault = 1, σ/σdef ault = 1.

used for BIPOP-aCMA-ES. This strategy represents an alternative to the BIPOP-aCMA-ES in the case if the restart strategy is restricted to increasing of population size. It also outperforms IPOP-aCMA-ES and is competitive with BIPOP-aCMA-ES on the BBOB noiseless problems [13]. E. The NBIPOP-aCMA-ES In NBIPOP-aCMA-ES [12] as well as in BIPOP-aCMAES there are two restart regimes: i). Double the population size and decrease the initial step-size by kσdec = 1.6 (NIPOP-aCMA-ES). ii). Launch CMA-ES with default population size λdef ault and 0 −2U[0,1] σ 0 = σdef . ault × 10 In contrast with BIPOP-aCMA-ES, where both regimes have the same budget, the budget is adapted here according to the performance of the regime: the best solutions x∗A and x∗B found by regimes A and B are used as an estimate of the quality of the regimes. Thus, kbudget = 2 times larger computation budget is allocated for regime A if it performs better than B (i.e., if x∗A is better than x∗B ), and vice versa.

E XPERIMENTAL VALIDATION

The experimental validation investigates the performance of 6 CMA-ES restart strategies: IPOP-CMA-ES, BIPOPCMA-ES, IPOP-aCMA-ES, BIPOP-aCMA-ES, NBIPOPaCMA-ES, NBIPOP-aCMA-ES. We use the source code 1 provided by the authors of [12], which is based on the original MATLAB code 2 of CMA-ES provided by N. Hansen. Both for IPOP and BIPOP versions the default parameter settings are used as given in [9], [4], [12]. The initial step-size σ is chosen according to the given search range [−100; 100] as 0.6 · 200 = 120. For all functions and dimensions the maximum number of function evaluations was set to 10000n. A. Results The results individually for each function and problem dimension are given according to [10] in Tables II-XIX after the maximum number of function evaluations. To assess the performance of the algorithms we use a procedure similar to one used in BBOB framework: for each objective function we define a set of function-target pairs ∆ft in the range [10−1 , 104 ]. The lower bound of 10−1 is chosen because for most of multi-modal functions the objective values below 10−1 are usually difficult to achieve. Fig. 2 and 3 depict the empirical cumulative distribution of running time of the annotated algorithm individually on all objective functions. Importantly, the results for all 3 problem dimensions and 51 runs are aggregated such that if the proportion of functiontarget pairs equals to 1 after a given number of function evaluations, then all 3 · 100 = 300 function-target pairs have been solved 51 times (i.e., 15300 problems solved) by the corresponding algorithm. For some functions, e.g., F20, the yaxis is scaled to better illustrate the difference in performances. 1 https://sites.google.com/site/ppsnbipop/ 2 https://www.lri.fr/∼ hansen/cmaes

inmatlab.html

F6 in 10-D, 30-D and 50-D


0 2 10

4

10

# evaluations / dimension Proportion of function-target pairs


F9 in 10-D, 30-D and 50-D

0.7


0.65 0.6 0.55 0.5 0.45 0.4 2 10

3

10



0.6 0.5 0.4

0.7 0.6 0.5 0.4 0.3

0.1



0.45

0.4

0.35

3

0.35 0.3

0.2 0.15 0.1 0.05

0.46 0.44 0.42 0.4 0.38 0.36 0.34 0.32


0.32 0.3 0.28 0.26 0.24 0.22 0.2 2 10

3

10

0.25

0.15

0.1

0.05


0.32 0.3 0.28 0.26 0.24 0.22 0.2 2 10

3

10


Fig. 3.

Continuation of Fig. 2.

0.5 0.45

4

10

0.36 0.34

0.35 0.3

0.2

0.1 0.05

0.3 0.28 0.26 0.24 0.22 3


0.7

0.6

0.5

0.95 0.9 0.85 0.8 0.75 0.7 0.65

0.6

0.55

0.605 0.6 0.595 0.59 0.585 0.58 0.575

0.1

0.05

4

10

0.26 0.24

0.4 0.38 0.36 0.34

0.3 0.28 0.26 0.24 0.22

0.18 0.16 0.14 0.12


4

10

F28 in 10-D, 30-D and 50-D

0.2

3

3

10



10


0.32

0.2 2 10

4

10

0.22

0.1 2 10

4

10

F24 in 10-D, 30-D and 50-D

0.15

0.28

3

10


0.2

3


0.57 2 10

4

10


10

4

10

F20 in 10-D, 30-D and 50-D

0.65

0.25

3

10


0.7

3

4

10


0.6 2 10

4

10


10

3

10

1

0.8

0.75


0.8

F27 in 10-D, 30-D and 50-D


10

0.9

F16 in 10-D, 30-D and 50-D

0.15

3

4

10


0.25

10

3

10

1


0.32

0.2 2 10

0.1

0.4 2 10

4

10


0 2 10

4

10

0.4 0.38

3

10

0.4

F26 in 10-D, 30-D and 50-D Proportion of function-target pairs


F25 in 10-D, 30-D and 50-D

0.34

0.55


0.4

0.36

3

10

0.2

# evaluations / dimension F23 in 10-D, 30-D and 50-D

0.2


0.38

0.6

0.5 2 10

4

10


0 2 10

4

10

0.3

F12 in 10-D, 30-D and 50-D

0.65

# evaluations / dimension F22 in 10-D, 30-D and 50-D Proportion of function-target pairs


0.34

3

0.4



10

0.5


0.7

0 2 10

4

10

0.5 0.48

0.3 2 10

4

10

0.4

0.36

0.8 0.75


0.6

F15 in 10-D, 30-D and 50-D

0.25

3

0.7



10

0.8

0 2 10

4

10


0.4 2 10

4

10

0.4

# evaluations / dimension F21 in 10-D, 30-D and 50-D 0.38

3

10

3

10

0.9 0.85


0.5

10

2

10

0 2 10

4

10


0.3 2 10


0.2

# evaluations / dimension F17 in 10-D, 30-D and 50-D 0.55

0.1

F14 in 10-D, 30-D and 50-D

0.7

3

0.2



10

0.3

F11 in 10-D, 30-D and 50-D

0.8

F13 in 10-D, 30-D and 50-D

0.3 2 10

0.4

1 0.9


1

0 1 10

4

10

1

0.8

0.5

0 2 10

4

10

0.9


0.9

0.6


0.8 0.75

3

10


3

10

0.1


2

10

0.2


0 1 10



0.1

0.3


0.2

0.4


0.3

0.5

0.7


0.4

0.6


0.5

0.7

0.8



0.6

0.8

F8 in 10-D, 30-D and 50-D

1 0.9


0.7

0.9


0.8

F7 in 10-D, 30-D and 50-D

1




F5 in 10-D, 30-D and 50-D 1 0.9

4

10

0.29 0.28 0.27 0.26 0.25 0.24 0.23 IPOP-CMA BIPOP-CMA IPOP-aCMA NIPOP-aCMA BIPOP-aCMA NBIPOP-aCMA

0.22 0.21 0.2 2 10

3

10


4

10

T0 0.277 0.277 0.277

T1 1.778 2.929 4.159

(T2 - T1) / T0 for IPOP-CMA 22.64 38.20 57.29

IPOP-aCMA 25.45 45.66 69.41

Active covariance matrix update. The active versions of CMA-ES clearly outperform the original ones on unimodal illconditioned functions F2, F3, F4. A substantial improvement is also observed on F5, F6, F7. The only function, where the original versions seem to perform better is F21 composition function of functions F1, F3, F4, F5 and F6, i.e., on which the active versions actually perform better. This is an unexpected result and requires further analysis. BIPOP vs IPOP. BIPOP-based algorithms outperform IPOP-based algorithms on F9, F14, F16, F20, F21, F24, F25, F26, F27, F28, and are outperformed by the latter on F11, F12, F13, F14 and F15. While in some cases the difference is minor, in overall, BIPOP-based algorithms perform better on composition functions. NBIPOP and NIPOP vs BIPOP and IPOP. The alternative restart strategies outperform the original ones on F9, F12, F16, F20, F24, F25, F26, F27, F28, and demonstrate a comparable performance on other functions. Computational Complexity. The results of experimental runs on F14 Schwefel’s function are given in Table I according to [10]. The restart strategies where smaller population sizes are used (e.g., NBIPOP-aCMA-ES) spend more time on internal computations per function evaluation, and are typically up to 2 times slower in terms of time than IPOP-CMA-ES. V.

C ONCLUSION

AND

P ERSPECTIVES

In this paper, we have compared the original and recently proposed restart strategies for CMA-ES on the CEC 2013 test suite. The aggregated results depicted in Fig. 4 demonstrate a slightly better performance of the NBIPOP-aCMA-ES and NIPOP-aCMA-ES. A possible reason is that a smaller initial step-size is especially useful on composition functions where the basins of attractions are relatively small. The results also confirm some superiority of the active covariance matrix update. The main limitation of all tested approaches is that the search in the hyper-parameter space of the population size and initial step-size seems to be inefficient and some potentially useful information from the restarts (e.g., the location of the best found solution) is not used. Another important limitation inherited from the CMA-ES is a lack of functionality which would allow to detect and exploit the separability of the objective function. Thus, the algorithms which specifically focus on separable and partially-separable functions will very likely outperform the CMA-ES and its restarts strategies. The above-described issues need to be addressed in future work. R EFERENCES [1]

A. Auger, S. Finck, N. Hansen, and R. Ros. BBOB 2010: Comparison Tables of All Algorithms on All Noiseless Functions. Technical Report RR-7215, INRIA, 2010.

BIPOP-CMA 45.97 56.20 84.06

BIPOP-aCMA 45.08 64.96 85.43

NIPOP-aCMA 59.26 63.64 102.38

All 28 functions in 10-D, 30-D and 50-D 1 0.9

IPOP-CMA BIPOP-CMA

0.8

IPOP-aCMA

0.7

NIPOP-aCMA

0.6

BIPOP-aCMA

0.5

NBIPOP-aCMA

0.4 0.3 0.2 0.1 0 1 10

2

10

3

10


4

10


D=10 D=30 D=50

C OMPUTATIONAL COMPLEXITY OF ALL 6 ALGORITHMS GIVEN FOR 10-, 30- AND 50- DIMENSIONAL S CHWEFEL’ S FUNCTION (F14).


TABLE I.

NBIPOP-aCMA 62.39 68.11 103.79

All 28 functions in 10-D, 30-D and 50-D 0.64 0.62 0.6 0.58 0.56 0.54 0.52 0.5 3 10

IPOP-CMA BIPOP-CMA IPOP-aCMA NIPOP-aCMA BIPOP-aCMA NBIPOP-aCMA 4

10


Fig. 4. Empirical cumulative distribution of all function-target pairs solved on all functions, dimensions and runs (in overall, 428400 pairs).

[2] A. Auger and N. Hansen. A Restart CMA Evolution Strategy With Increasing Population Size. In IEEE Congress on Evolutionary Computation, pages 1769–1776. IEEE Press, 2005. [3] N. Hansen. Compilation of results on the 2005 CEC benchmark function set. Online, May, 2006. [4] N. Hansen. Benchmarking a BI-population CMA-ES on the BBOB2009 function testbed. In GECCO Companion, pages 2389–2396, 2009. [5] N. Hansen, A. Auger, S. Finck, and R. Ros. Real-Parameter BlackBox Optimization Benchmarking 2010: Experimental Setup. Technical Report RR-7215, INRIA, 2010. [6] N. Hansen and S. Kern. Evaluating the CMA Evolution Strategy on Multimodal Test Functions. In PPSN’04, pages 282–291, 2004. [7] N. Hansen, S. Müller, and P. Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11(1):1–18, 2003. [8] N. Hansen and A. Ostermeier. Completely Derandomized SelfAdaptation in Evolution Strategies. Evol. Comput., 9(2):159–195, June 2001. [9] N. Hansen and R. Ros. Benchmarking a weighted negative covariance matrix update on the BBOB-2010 noiseless testbed. In GECCO ’10: Proceedings of the 12th annual conference comp on Genetic and evolutionary computation, pages 1673–1680, New York, NY, USA, 2010. ACM. [10] P. N. S. J. J. Liang, B-Y. Qu and A. G. Hernandez-Diaz. Problem Definitions and Evaluation Criteria for the CEC 2013 Special Session and Competition on Real-Parameter Optimization. Technical report, Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, 2013. [11] G. A. Jastrebski and D. V. Arnold. Improving Evolution Strategies through Active Covariance Matrix Adaptation. In IEEE Congress on Evolutionary Computation, pages 2814–2821, 2006. [12] I. Loshchilov, M. Schoenauer, and M. Sebag. Alternative Restart Strategies for CMA-ES. In V. C. et al., editor, Parallel Problem Solving from Nature (PPSN XII), LNCS, pages 296–305. Springer, 2012. [13] I. Loshchilov, M. Schoenauer, and M. Sebag. Black-box Optimization Benchmarking of NIPOP-aCMA-ES and NBIPOP-aCMA-ES on the BBOB-2012 Noiseless Testbed. In T. Soule and J. H. Moore, editors, Genetic and Evolutionary Computation Conference (GECCO Companion), pages 269–276. ACM Press, July 2012. [14] M. Preuss. Niching the cma-es via nearest-better clustering. In Proceedings of the 12th annual conference companion on Genetic and evolutionary computation, pages 1711–1718. ACM, 2010. [15] O. M. Shir, M. Emmerich, and T. Bäck. Adaptive niche radii and niche shapes approaches for niching with the cma-es. Evolutionary Computation, 18(1):97–126, 2010.

TABLE II. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.187 0.000 0.000 0.000 0.000 0.000 3.602 0.312 0.905 10.382 10.258 0.440 1.547 100.000 9.902 14.224 200.000 200.000 106.960 319.700 300.000

TABLE III. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.185 0.000 0.000 0.000 0.000 0.000 3.665 3.665 0.000 10.550 6.299 0.378 0.828 100.000 38.725 34.232 100.000 109.600 100.000 186.920 100.000

TABLE IV. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.189 0.000 0.000 0.000 0.000 0.000 3.602 3.727 0.000 10.310 10.346 0.059 1.563 100.000 21.790 18.237 100.000 100.000 49.144 300.000 109.340

IPOP-CMA-ES IN 10-D Worst 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.471 3.121 0.000 1.990 0.995 1.990 167.830 58.398 1.542 12.430 11.953 0.919 4.019 400.190 313.190 318.930 225.200 224.120 218.160 560.530 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.352 0.199 0.000 0.000 0.000 0.000 18.535 18.535 1.124 10.984 10.951 0.646 3.019 400.190 56.512 59.212 208.450 203.610 205.810 446.630 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.342 0.582 0.000 0.332 0.098 0.313 26.681 21.955 1.152 11.068 10.974 0.646 2.763 374.677 73.252 86.163 209.465 205.517 204.201 454.562 300.000

Std 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.070 0.721 0.000 0.551 0.299 0.508 26.673 15.243 0.136 0.430 0.414 0.115 0.592 71.735 49.446 66.062 7.022 6.709 15.094 79.234 0.000

BIPOP-CMA-ES IN 10-D Worst 0.000 0.000 0.000 0.000 0.000 0.000 0.002 20.517 2.638 0.000 2.985 1.990 3.651 359.080 258.190 1.593 13.109 14.018 0.951 3.559 400.190 339.800 302.550 207.740 207.740 200.020 447.100 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.359 0.104 0.000 0.995 0.995 0.995 33.529 39.989 0.090 11.500 11.706 0.591 2.621 300.000 78.204 110.520 110.700 202.260 107.960 354.120 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.339 0.554 0.000 0.936 0.585 0.944 54.762 52.837 0.305 11.551 11.565 0.604 2.582 284.407 99.325 117.411 130.304 192.516 118.570 346.434 280.392

Std 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.082 0.684 0.000 0.806 0.603 0.838 69.842 57.800 0.457 0.559 1.159 0.120 0.552 122.356 65.612 65.091 38.730 28.442 24.895 44.001 60.065

NIPOP- ACMA-ES IN 10-D Worst 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.470 1.782 0.000 0.995 0.995 1.026 336.200 332.870 1.529 11.968 11.439 0.953 3.606 400.190 375.240 506.830 206.400 207.430 200.020 547.980 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.357 0.000 0.000 0.000 0.000 0.000 142.300 109.320 1.121 10.980 10.824 0.679 2.479 400.190 98.857 180.540 108.070 200.000 100.990 325.980 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.353 0.254 0.000 0.267 0.078 0.254 140.243 129.231 1.055 11.006 10.849 0.658 2.417 350.538 146.533 196.642 149.495 196.967 123.776 350.654 292.859

Std 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.064 0.457 0.000 0.441 0.270 0.439 97.669 96.480 0.314 0.382 0.257 0.159 0.455 87.857 110.690 117.469 49.703 19.858 43.711 67.183 35.738

TABLE V. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.156 0.000 0.000 0.000 0.000 0.000 0.187 0.250 0.526 10.262 10.227 0.458 1.512 200.000 16.683 16.404 200.000 200.000 108.950 304.010 300.000

TABLE VI. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.000 0.000 0.000 0.000 0.000 0.000 0.312 0.125 0.000 4.490 7.703 0.339 1.001 100.000 29.353 25.760 102.720 103.460 73.031 300.000 100.000

TABLE VII. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.000 0.000 0.000 0.000 0.000 0.000 6.892 18.597 0.011 10.333 7.956 0.010 1.198 100.000 36.355 24.453 100.000 100.000 100.000 172.980 100.000

IPOP- ACMA-ES IN 10-D Worst 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.474 3.000 0.000 1.990 0.995 1.990 65.170 125.390 1.598 12.014 13.002 0.873 4.035 400.190 259.830 243.670 225.180 222.810 220.010 559.070 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.359 0.000 0.000 0.000 0.000 0.000 21.825 18.472 1.222 10.784 11.021 0.658 2.604 400.190 58.989 59.730 205.850 203.370 202.730 462.610 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.353 0.504 0.000 0.351 0.078 0.254 23.576 24.250 1.169 10.846 11.076 0.655 2.719 380.564 70.667 80.393 209.341 203.944 199.885 450.354 300.000

Std 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.075 0.720 0.000 0.520 0.270 0.482 14.898 22.438 0.228 0.317 0.548 0.097 0.609 60.122 42.842 49.941 8.700 4.543 20.951 89.518 0.000

BIPOP- ACMA-ES IN 10-D Worst 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.468 2.635 0.000 2.985 2.985 2.396 271.630 317.210 1.386 13.014 13.686 0.900 3.547 400.190 233.900 367.530 212.520 206.820 200.020 400.000 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.355 0.263 0.000 0.000 0.013 0.995 27.012 28.596 0.051 11.466 11.340 0.575 2.551 400.190 66.714 85.528 108.570 201.300 107.420 337.710 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.345 0.522 0.000 0.587 0.644 0.733 40.118 47.743 0.174 11.409 11.336 0.589 2.538 315.805 82.324 96.398 127.376 184.849 121.129 340.883 260.784

Std 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.082 0.644 0.000 0.721 0.766 0.688 49.940 65.849 0.331 1.101 0.974 0.116 0.583 113.884 45.521 55.902 37.218 36.093 32.253 29.434 80.196

NBIPOP- ACMA-ES IN 10-D Worst 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.520 1.503 0.000 1.990 2.985 2.836 356.120 659.330 1.369 12.390 16.995 0.876 3.795 200.000 451.250 512.870 202.240 205.770 246.650 360.600 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.353 0.000 0.000 0.000 0.000 0.001 76.816 151.310 0.054 11.369 11.071 0.518 2.761 200.000 141.820 129.180 107.870 200.060 105.970 311.620 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.339 0.232 0.000 0.364 0.238 0.484 114.997 158.161 0.120 11.334 11.288 0.525 2.726 152.941 175.131 174.230 119.885 176.972 111.035 316.684 249.020

Std 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.090 0.440 0.000 0.506 0.542 0.676 92.377 117.317 0.263 0.545 1.276 0.139 0.650 50.410 114.655 122.831 32.220 39.918 24.986 29.556 88.029

TABLE VIII. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.765 1.213 0.000 0.000 0.071 0.000 60.072 29.083 1.914 32.431 32.044 1.177 13.737 200.000 120.550 91.710 219.630 205.270 200.000 483.550 300.000

TABLE IX. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.799 0.231 0.000 0.000 0.000 0.000 51.251 56.319 0.002 33.612 32.257 1.265 12.392 100.000 113.670 189.780 117.230 214.860 111.940 378.550 100.000

TABLE X. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.755 0.000 0.000 0.000 0.000 0.000 201.210 90.572 1.508 31.754 31.905 1.130 10.012 200.000 116.060 82.274 220.740 207.770 125.110 320.160 300.000

IPOP-CMA-ES IN 30-D

Worst 0.000 0.000 64.878 0.000 0.000 0.000 55.451 21.006 41.165 0.000 6.965 5.970 12.135 1277.400 1055.100 3.191 39.212 181.730 3.203 15.000 300.000 1483.100 1869.600 306.160 302.720 403.450 1326.300 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 2.843 20.956 37.269 0.000 2.096 1.990 1.990 185.840 344.610 2.539 33.577 40.312 2.527 14.585 300.000 420.510 517.520 300.270 298.280 323.380 1281.600 300.000

Mean 0.000 0.000 1.732 0.000 0.000 0.000 16.835 20.931 24.463 0.000 2.290 1.853 2.414 287.008 337.708 2.528 34.073 81.650 2.484 14.603 254.902 502.379 576.071 285.725 286.874 314.510 1141.729 300.000

Std 0.000 0.000 9.296 0.000 0.000 0.000 19.624 0.059 16.090 0.000 1.452 1.164 2.266 272.130 241.796 0.273 1.355 61.282 0.402 0.349 50.254 309.407 350.245 30.214 28.505 81.420 290.392 0.000

BIPOP-CMA-ES IN 30-D Worst 0.000 0.000 3.638 0.000 0.000 0.000 46.223 21.017 12.008 0.000 6.965 5.970 6.853 4146.200 2228.500 2.826 40.999 172.370 3.309 15.000 300.000 2906.700 2776.000 300.540 302.750 205.460 660.750 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 1.156 20.936 6.419 0.000 2.985 1.990 1.990 514.060 492.590 0.042 36.002 41.362 2.497 14.344 200.000 705.840 664.470 161.900 224.960 148.750 513.360 300.000

Mean 0.000 0.000 0.082 0.000 0.000 0.000 9.426 20.935 6.489 0.000 3.082 2.410 2.391 669.207 609.778 0.775 36.343 54.364 2.395 14.237 200.000 838.581 716.942 180.398 231.191 163.826 503.581 292.157

Std 0.000 0.000 0.509 0.000 0.000 0.000 13.302 0.051 2.380 0.000 1.519 1.430 1.474 697.842 450.755 1.143 1.770 33.956 0.418 0.636 28.284 577.102 452.235 50.160 21.156 32.299 71.345 39.208


Median 0.000 0.000 0.000 0.000 0.000 0.000 0.044 20.928 2.927 0.000 0.995 0.031 0.995 712.620 668.950 2.549 34.100 35.104 2.482 13.529 200.000 530.650 632.300 298.520 298.610 249.560 639.970 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 4.055 20.921 2.823 0.000 1.032 0.656 0.931 716.645 670.256 2.484 34.248 53.961 2.408 13.365 241.176 572.759 667.436 290.623 278.962 251.038 870.294 300.000

Std 0.000 0.000 0.000 0.000 0.000 0.000 9.172 0.062 1.228 0.000 1.040 0.825 0.928 244.372 280.430 0.314 1.716 44.520 0.465 1.260 49.705 341.016 326.261 22.816 35.199 57.982 422.247 0.000

TABLE XI. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.834 1.500 0.000 0.000 0.000 0.000 54.523 60.736 1.968 31.726 31.678 1.207 13.716 200.000 99.474 119.140 218.430 211.300 200.000 380.180 300.000

TABLE XII. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.797 1.305 0.000 0.000 0.000 0.995 113.210 46.739 0.000 32.490 32.241 1.350 11.551 200.000 118.070 190.510 120.200 210.660 118.910 400.000 100.000

TABLE XIII. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.000 20.797 0.401 0.000 0.000 0.000 0.000 278.650 282.650 0.014 32.451 32.191 1.103 11.117 100.000 129.850 188.220 122.800 154.010 128.850 350.550 100.000

IPOP- ACMA-ES IN 30-D Worst 0.000 0.000 0.000 0.000 0.000 0.000 120.400 21.023 41.265 0.000 4.396 2.985 4.120 1195.800 1203.100 2.987 39.384 176.400 3.398 15.000 300.000 1249.100 1195.500 304.830 303.330 406.060 1334.000 300.000

Median 0.000 0.000 0.000 0.000 0.000 0.000 0.118 20.950 38.879 0.000 0.997 0.327 0.995 225.710 298.050 2.550 33.396 36.888 2.532 14.537 300.000 401.690 444.300 297.660 299.040 350.090 1272.900 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 8.854 20.944 27.216 0.000 1.174 0.704 1.117 271.876 336.269 2.529 33.764 70.653 2.466 14.596 254.902 477.206 492.216 276.327 289.376 329.264 1079.047 300.000

Std 0.000 0.000 0.000 0.000 0.000 0.000 22.129 0.045 16.452 0.000 1.092 0.833 1.225 220.390 257.532 0.252 1.442 56.460 0.449 0.329 50.254 293.492 292.454 34.673 26.732 77.967 344.867 0.000


Median 0.000 0.000 0.000 0.000 0.000 0.000 0.027 20.950 5.168 0.000 2.985 2.985 1.990 429.630 465.140 0.060 35.802 37.687 2.291 14.155 200.000 636.750 649.110 213.840 225.560 154.730 513.390 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 2.727 20.939 5.214 0.000 3.142 2.810 2.646 495.496 544.527 0.940 35.702 58.053 2.285 14.015 213.725 662.775 702.919 186.292 226.475 168.368 516.864 284.314

Std 0.000 0.000 0.000 0.000 0.000 0.000 10.937 0.055 1.949 0.000 1.426 1.575 1.437 277.647 416.219 1.203 1.712 43.411 0.323 0.770 34.754 301.895 312.690 41.916 13.732 30.175 93.743 54.305


Median 0.000 0.000 0.000 0.000 0.000 0.000 0.057 20.946 2.768 0.000 2.985 2.985 2.985 739.970 744.850 0.041 33.593 39.560 2.233 13.131 200.000 734.260 666.730 155.520 221.920 146.760 471.890 300.000

Mean 0.000 0.000 0.000 0.000 0.000 0.000 2.313 20.942 3.300 0.000 3.043 2.907 2.778 810.125 765.493 0.440 34.419 62.289 2.228 12.940 192.157 838.392 667.086 161.757 219.984 158.223 468.925 268.627

Std 0.000 0.000 0.000 0.000 0.000 0.000 6.049 0.048 1.383 0.000 1.413 1.376 1.453 360.294 294.867 0.926 1.869 45.591 0.341 0.598 27.152 459.988 289.554 30.045 11.094 29.999 73.770 73.458

TABLE XIV. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.048 20.841 3.167 0.000 0.001 0.000 0.000 150.500 102.580 2.717 53.291 54.106 2.022 25.000 200.000 152.770 164.350 244.560 239.930 200.000 699.690 400.000

Worst 0.000 0.000 173190.000 0.000 0.000 0.000 195.500 21.178 75.313 0.000 21.889 20.894 93.992 13329.000 12866.000 3.776 79.900 360.530 5.935 25.000 1122.200 13113.000 13349.000 391.680 388.480 491.740 2200.700 3400.600

TABLE XV. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.095 20.996 4.838 0.000 1.054 0.995 0.000 217.950 159.450 0.003 54.361 54.870 3.099 22.118 200.000 269.680 202.850 169.060 229.770 128.880 400.060 400.000

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.035 20.963 2.038 0.000 0.000 0.000 0.000 450.280 365.230 2.580 53.824 55.423 2.571 19.790 200.000 218.400 253.900 237.470 215.640 200.000 520.440 400.000

Median 0.000 0.000 1.988 0.000 0.000 0.000 11.322 21.134 71.931 0.000 6.965 5.970 5.573 780.840 801.890 3.336 57.218 328.620 4.518 25.000 200.000 1042.700 1133.400 385.340 383.120 481.770 2130.500 400.000

Mean 0.000 0.000 6506.587 0.000 0.000 0.000 22.928 21.123 59.601 0.000 8.506 6.117 10.804 1625.565 1357.597 3.315 58.214 228.534 4.413 25.000 516.822 1825.791 2986.475 375.023 373.787 382.372 1936.220 1034.771

Std 0.000 0.000 27617.764 0.000 0.000 0.000 39.616 0.052 25.270 0.000 5.594 4.359 16.777 2921.834 2387.453 0.277 4.370 135.806 0.789 0.000 408.086 2860.190 4190.174 33.360 33.452 129.421 454.537 1222.556

BIPOP-CMA-ES IN 50-D

Worst 0.000 0.000 226180.000 0.000 0.000 0.000 358.290 21.190 25.721 0.000 21.889 17.909 33.058 5743.500 4883.900 3.801 94.885 359.620 5.892 25.000 836.440 12181.000 6981.000 385.270 386.390 205.240 929.000 400.000

TABLE XVI. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

IPOP-CMA-ES IN 50-D

Median 0.000 0.000 0.044 0.000 0.000 0.000 11.802 21.139 13.283 0.000 7.960 6.965 5.970 1354.700 930.790 0.069 61.199 74.722 4.264 23.573 200.000 1262.300 1329.100 248.760 253.630 178.600 743.580 400.000

Mean 0.000 0.000 8858.325 0.000 0.000 0.000 45.355 21.128 13.711 0.000 9.275 8.116 7.636 1697.035 1305.709 1.562 61.619 138.883 4.347 23.527 224.958 1765.215 1922.464 246.799 259.932 177.402 736.518 400.000

Std 0.000 0.000 37582.750 0.000 0.000 0.000 79.408 0.039 4.507 0.000 5.319 3.704 6.106 1445.075 1101.514 1.662 5.778 117.742 0.560 0.642 124.767 1731.078 1655.305 32.156 32.909 23.863 101.757 0.000


Median 0.000 0.000 0.000 0.000 0.000 0.000 1.910 21.117 7.054 0.000 1.990 0.995 0.995 1204.200 1331.800 3.386 57.287 106.450 4.544 22.969 200.000 895.340 938.890 382.040 382.570 311.420 2119.100 400.000

Mean 0.000 0.000 18.773 0.000 0.000 0.000 11.714 21.111 6.882 0.000 1.991 1.366 1.481 1257.493 1352.059 3.370 57.737 193.660 4.467 22.985 365.287 1017.509 1186.484 370.404 365.032 288.263 1898.799 571.590

Std 0.000 0.000 116.751 0.000 0.000 0.000 37.728 0.044 1.819 0.000 1.305 1.289 1.009 442.403 504.095 0.296 2.373 134.807 0.521 1.370 325.247 466.984 690.262 37.743 48.659 98.170 520.000 693.198

TABLE XVII. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.074 21.012 4.115 0.000 0.000 0.000 0.000 155.470 115.590 2.593 55.082 54.672 2.439 25.000 200.000 238.960 225.600 240.740 218.540 200.000 664.700 400.000

Worst 0.000 0.000 151.320 0.000 0.000 0.000 334.040 21.184 75.639 0.000 21.889 11.940 66.642 11724.000 11956.000 3.883 76.011 360.110 6.125 25.000 1122.200 12547.000 12190.000 389.950 387.800 492.920 2181.700 3345.400

TABLE XVIII. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.084 20.996 5.075 0.000 0.000 2.985 1.026 220.000 195.040 0.005 54.955 54.805 2.587 22.080 200.000 283.460 102.160 142.300 223.190 117.910 400.000 400.000

TABLE XIX. Func. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Best 0.000 0.000 0.000 0.000 0.000 0.000 0.039 20.969 2.022 0.000 0.998 0.007 0.995 381.240 333.750 0.018 54.741 55.241 3.040 18.746 100.000 188.710 477.380 194.580 233.430 113.930 390.610 400.000

IPOP- ACMA-ES IN 50-D Median 0.000 0.000 0.002 0.000 0.000 0.000 3.701 21.124 72.368 0.000 7.960 1.991 3.980 632.460 639.630 3.392 57.460 73.931 4.507 25.000 836.440 832.130 895.690 385.920 382.090 360.670 2128.900 400.000

Mean 0.000 0.000 5.446 0.000 0.000 0.000 14.791 21.122 51.913 0.000 8.545 3.535 6.066 1138.034 1067.475 3.356 58.697 164.545 4.472 25.000 645.944 1406.937 1201.597 380.249 366.253 370.441 2048.754 856.214

Std 0.000 0.000 22.085 0.000 0.000 0.000 47.056 0.031 29.071 0.000 4.595 3.597 10.345 2040.146 1732.580 0.271 4.181 129.500 0.553 0.000 407.072 1989.910 1669.776 27.161 45.626 125.046 315.653 1067.395


Median 0.000 0.000 0.012 0.000 0.000 0.000 8.221 21.127 12.349 0.000 6.965 6.965 6.064 1129.500 1218.600 0.418 58.751 81.181 4.294 23.663 200.000 1402.500 1330.600 248.860 251.220 200.000 735.360 400.000

Mean 0.000 0.000 28.663 0.000 0.000 0.000 20.688 21.127 12.505 0.000 7.323 6.773 7.131 1253.627 1399.566 1.673 60.136 158.085 4.240 23.752 267.999 1628.679 1752.374 245.757 256.735 184.264 716.665 400.000

Std 0.000 0.000 150.674 0.000 0.000 0.000 36.151 0.034 3.778 0.000 3.542 2.896 4.187 857.811 923.669 1.696 3.608 123.762 0.572 0.626 211.417 1025.878 1054.910 32.071 33.322 24.067 141.348 0.000


Median 0.000 0.000 0.000 0.000 0.000 0.000 3.646 21.131 7.058 0.000 5.970 4.975 6.965 1335.000 1495.500 0.044 56.737 104.030 4.422 22.738 200.000 1336.300 1492.400 244.990 248.680 200.000 777.220 400.000

Mean 0.000 0.000 18.166 0.000 0.000 0.000 4.971 21.119 7.220 0.000 5.505 5.371 7.595 1375.403 1553.688 0.878 57.369 133.647 4.458 22.547 198.039 1448.353 1712.552 239.643 247.570 196.091 727.829 400.000

Std 0.000 0.000 121.348 0.000 0.000 0.000 5.724 0.045 2.286 0.000 2.959 2.540 5.468 566.544 548.191 1.441 2.726 100.310 0.593 1.175 14.003 601.295 809.352 20.380 5.059 14.340 144.098 0.000

CMA-ES with Restarts for Solving CEC 2013 Benchmark ... - CiteSeerX

CMA-ES with Restarts for Solving CEC 2013 Benchmark ... - CiteSeerX

Suggest Documents

Benchmark Functions for the CEC'2013 Special ... - RMIT University

Solving Two Supervisory Control Benchmark Problems ... - CiteSeerX

Solving Two Supervisory Control Benchmark Problems ... - CiteSeerX

Benchmark Functions for CEC'2017 Competition on ... - Cercia

Benchmark Generator for CEC'2009 Competition on Dynamic ...

CEC Partners with Indiana CASE for CEC 2019 - Council for ...

CEC-2013 - Nanyang Technological University

Restarts - NSRDEC

CEC

SGDR: STOCHASTIC GRADIENT DESCENT WITH WARM RESTARTS

Efficient and stable Arnoldi restarts for matrix functions ... - CiteSeerX

Efficient and stable Arnoldi restarts for matrix functions ... - CiteSeerX

Problem Definitions and Evaluation Criteria for the CEC 2013 Special ...

IAB Europe Adex Benchmark 2013

IAB Europe Adex Benchmark 2013

IAB Europe Adex Benchmark 2013

IAB Europe Adex Benchmark 2013

IAB Europe Adex Benchmark 2013

Problem Definitions and Evaluation Criteria for the CEC 2013 Special ...

Experiments with Subsetting Benchmark Suites - CiteSeerX

Experiments with Subsetting Benchmark Suites - CiteSeerX

Local Restarts in SAT

Download CEC 2019 Flyers - IEEE CEC 2019

Object Tracking Benchmark - CiteSeerX