Improving Coevolution by Random Sampling

Improving Coevolution by Random Sampling Wojciech Jaśkowski

Paweł Liskowski Krzysztof Krawiec

Institute of Computing Science

July 8, 2013

Marcin Szubert

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Inspired by Chong, Tiˇno, Ku & Yao

statistical framework for evaluating strategies in coevolution coevolution < evolution with random sampling Improving Coevolution by Random Sampling

2 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Position Evaluation Function for Othello Othello Board

Strategy encoding .92 -.27 .55 -.10 .08 .47 -.38 .99 -.13 -.52 -.18 -.07 -.18 -.29 -.68 -.44 .55 -.24 .02 -.01 -.01 .10 -.13 .77 -.10 -.10 .01 -.01 .00 -.01 -.09 -.05 .05 -.17 .02 -.04 -.03 .03 -.09 -.05 .56 -.25 .05 .02 -.02 .17 -.35 .42 -.25 -.71 -.24 -.23 -.08 -.29 -.63 -.24 .93 -.44 .55 .22 -.15 .74 -.57 .97

1 look ahead

Improving Coevolution by Random Sampling

3 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Setups: Coevolution and Evolution with Random Sampling population

one-population coevolution (1-Coev)

population

random individuals

evolution with random sampling (Evol-RS)

ES (50+50), same settings; difference only in computing fitness Improving Coevolution by Random Sampling

4 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Results for Expected Utility

Performance (expected utility)

0

50

Generation 100

150

200

85%

Expected Utility = average result over 50 000 games

80%

Algorithm Exp. Util. [%]

75%

1-Coev

70%

65%

0

200

400 600 800 Games played (×1000)

Improving Coevolution by Random Sampling

80.34 ± 0.54

1000

5 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Results for Expected Utility

Performance (expected utility)

0

50

Generation 100

150

200

85%

Expected Utility = average result over 50 000 games

80%

Algorithm Exp. Util. [%] Evol-RS

86.46 ± 0.25

1-Coev

80.34 ± 0.54

75%

70%

65%

0

200

400 600 800 Games played (×1000)

Improving Coevolution by Random Sampling

1000

5 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Research Questions

Initial result: Evolution with RS > coevolution for expected utility Research questions: 1

Any coevolution > evolution with RS for expected utility?

2

Any performance measure for which coevolution > evolution with RS?

Improving Coevolution by Random Sampling

6 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Two population coevolution population

one-population coevolution (1-Coev)

population 1

population 2

two-population coevolution (2-Coev)

Improving Coevolution by Random Sampling

7 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

1- and 2-Coevolution vs. Evolution with Random Sampling

Performance (expected utility)

0

50

Generation 100

150

200

85%

Expected Utility = average result over 50 000 games

80%

Algorithm Exp. Util. [%] Evol-RS

86.46 ± 0.25

1-Coev

80.34 ± 0.54

75%

70%

2-Coev 65%

0

200

400 600 800 Games played (×1000)

Improving Coevolution by Random Sampling

79.97 ± 0.58

1000

8 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Coevolution with Random Sampling

2-Coev-RS

1-Coev-RS population

random individuals

population 1

population 2

random individuals

Improving Coevolution by Random Sampling

9 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

1-Coevolution with Random Sampling

Performance (expected utility)

0

50

Generation 100

150

200

85%

Expected Utility = average result over 50 000 games

80%

Algorithm Exp. Util. [%] Evol-RS

75%

86.46 ± 0.25

1-Coev-RS 83.63 ± 0.37 1-Coev

70%

2-Coev 65%

0

200

400 600 800 Games played (×1000)

Improving Coevolution by Random Sampling

80.34 ± 0.54 79.97 ± 0.58

1000

10 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

2-Coevolution with Random Sampling

Performance (expected utility)

0

50

Generation 100

150

200

85%

Expected Utility = average result over 50 000 games

80%

Algorithm Exp. Util. [%] Evol-RS

75%

86.46 ± 0.25

2-Coev-RS 86.44 ± 0.26 1-Coev-RS 83.63 ± 0.37 1-Coev

70%

2-Coev 65%

0

200

400 600 800 Games played (×1000)

Improving Coevolution by Random Sampling

80.34 ± 0.54 79.97 ± 0.58

1000

10 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Algorithms Analyzed with Performance Profiles 100%

Performance Profile — a vector measure — how a strategy copes against opponents of different strength.

95%

Method performance

90% 85% 80% 75%

Algorithm Exp. Util. [%]

70%

Evol-RS

65%

86.46 ± 0.25

60% 55% 10% 20% 30% 40% 50% 60% 70% 80% Opponent performance

Performance Profiles

Improving Coevolution by Random Sampling

11 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Performance Profile — how prepared? random individuals

Opponent performance 10% 20% 30% 40% 50% 60% 70% 80%

54-55% 55-56% 56-57%

100 buckets. bucket — at most 1000 opponents of 1% performance range. opponents generated by randomly sampling the strategy space Improving Coevolution by Random Sampling

12 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Algorithms Analyzed with Performance Profiles 100%

Performance Profile — a vector measure — how a strategy copes against opponents of different strength.

95%

Method performance

90% 85% 80% 75%

Algorithm Exp. Util. [%]

70%

Evol-RS

65%

2-Coev-RS 86.44 ± 0.26

86.46 ± 0.25

60% 55% 10% 20% 30% 40% 50% 60% 70% 80% Opponent performance

Performance Profiles

Improving Coevolution by Random Sampling

13 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Algorithms Analyzed with Performance Profiles 100%

Performance Profile — a vector measure — how a strategy copes against opponents of different strength.

95%

Method performance

90% 85% 80% 75%

Algorithm Exp. Util. [%]

70%

Evol-RS

65%

2-Coev-RS 86.44 ± 0.26

86.46 ± 0.25

1-Coev-RS 83.63 ± 0.37

60% 55% 10% 20% 30% 40% 50% 60% 70% 80% Opponent performance

Performance Profiles

Improving Coevolution by Random Sampling

13 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Algorithms Analyzed with Performance Profiles 100%

Performance Profile — a vector measure — how a strategy copes against opponents of different strength.

95%

Method performance

90% 85% 80% 75%

Algorithm Exp. Util. [%]

70%

Evol-RS

65%

2-Coev-RS 86.44 ± 0.26 1-Coev-RS 83.63 ± 0.37

60% 55% 10% 20% 30% 40% 50% 60% 70% 80% Opponent performance

Performance Profiles

Improving Coevolution by Random Sampling

86.46 ± 0.25

13 / 19

1-Coev 2-Coev

80.34 ± 0.54 79.97 ± 0.58 W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Results of Round-Robin Tournament

Algorithm

1-Coev-RS 2-Coev-RS 1-Coev Evol-RS 2-Coev Overall

1-Coev-RS

-

53.4%

53.3%

56.3%

2-Coev-RS 1-Coev

56.7%

54.9%

46.6%

-

51.2%

46.7%

48.8%

-

53.2%

53.9%

51.3%

51.8%

52.7%

50.0%

Evol-RS

43.7%

46.8%

48.2%

-

52.0%

47.7%

2-Coev

43.3%

46.1%

47.3%

48.0%

-

46.2%

each algorithm = 120 strategies method vs. method = 120 × 120 = 14 400 games

Improving Coevolution by Random Sampling

14 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Expected Utilility vs. Round-Robin Tournament Discrepancy between the order of methods in Expected Utility and Round-Robin Tournament: Algorithm

Exp. Utility [%]

Algorithm

Round Robin [%]

Evol-RS 2-Coev-RS 1-Coev-RS 1-Coev 2-Coev

86.46 86.44 83.63 80.34 79.97

1-Coev-RS 2-Coev-RS 1-Coev Evol-RS 2-Coev

54.9 51.3 50.0 47.7 46.2

Can be explain the discrepancy using Performance Profiles?

Improving Coevolution by Random Sampling

15 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Explaining Exp. Util. with Performance Profiles 100%

Method performance

90%

Algorithm Exp.Util. [%] 80%

Evol-RS 86.46 ± 0.25 2-Coev-RS 86.44 ± 0.26 1-Coev-RS 83.63 ± 0.37

70%

1-Coev 2-Coev

80.34 ± 0.54 79.97 ± 0.58

60%

50% 10% 20% 30% 40% 50% 60% 70% 80% 90% Opponent performance

Improving Coevolution by Random Sampling

16 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Explaining Round-Robin with Performance Profiles 100%

Method performance

90%

Algorithm Round Robin [%] 80%

1-Coev-RS 2-Coev-RS 1-Coev

54.9 51.3 50.0

70%

Evol-RS 2-Coev

47.7 46.2

60%

50% 10% 20% 30% 40% 50% 60% 70% 80% 90% Opponent performance

Improving Coevolution by Random Sampling

17 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Explaining Round-Robin with Performance Profiles 80%

Method performance

75%

Algorithm Round Robin [%] 70% 65% 60%

1-Coev-RS 2-Coev-RS 1-Coev

54.9 51.3 50.0

Evol-RS 2-Coev

47.7 46.2

55% 50% 50% 55% 60% 65% 70% 75% 80% 85% 90% Opponent performance

Improving Coevolution by Random Sampling

17 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Two-objectives analysis

90% Expected utility

Evol-RS 2-Coev-RS 1-Coev-RS 1-Coev 2-Coev

70% 30%

60% Round-robin result

Improving Coevolution by Random Sampling

18 / 19

W. Jaśkowski, et al.

Motivation

Coevolutionary Algorithms

Performance Profiles

Round-Robin

Explaining with PP

Summary

Summary

1 2

Coevolution with RS dominates coevolution. For strong opponents Coevolution with RS > Evol with RS bias towards playing well against strong opponents larger bias for 1-coevolution

3

Performance profiles — deeper analysis than scalar performance measures. code in Java: http://www.cs.put.poznan.pl/wjaskowski/projects/performance-profiles

Questions?

Improving Coevolution by Random Sampling

19 / 19

W. Jaśkowski, et al.

Appendix

Strategies are evaluated by their expected utility

Strategy performance is expected utility of the strategy Expected Utility is expected game result over all possible strategies 1 X performance(s) = othello(s, t), N t∈TN

where: TN

is set of N random strategies (here: N = 50 000)

othello(s, t) is result of the game (here: ∈ {0, 12 , 1})

Improving Coevolution by Random Sampling

20 / 19

W. Jaśkowski, et al.