Evolutionary Network Minimization: Adaptive Implicit Pruning of Successful Agents Zohar Ganon1 , Alon Keinan1 , and Eytan Ruppin1,2 1
School of Computer Sciences, Tel-Aviv University, Tel-Aviv 69978, Israel {zganon,keinan}@cns.tau.ac.il,
[email protected] 2 School of Medicine, Tel-Aviv University, Tel-Aviv 69978, Israel
Abstract. Neurocontroller minimization is beneficial for constructing small parsimonious networks that permit a better understanding of their workings. This paper presents a novel, Evolutionary Network Minimization (ENM) algorithm which is applied to fully recurrent neurocontrollers. ENM is a simple, standard genetic algorithm with an additional step in which small weights are irreversibly eliminated. ENM has a unique combination of features which distinguish it from previous evolutionary minimization algorithms: 1. An explicit penalty term is not added to the fitness function. 2. Minimization begins after functional neurocontrollers have been successfully evolved. 3. Successful minimization relies solely on the workings of a drift that removes unimportant weights and, importantly, on continuing adaptive modifications of the magnitudes of the remaining weights. Our results testify that ENM is successful in extensively minimizing recurrent evolved neurocontrollers while keeping their fitness intact and maintaining their principal functional characteristics.
1
Introduction
Pruning a neural network is a standard approach in the connectionist literature by which unimportant weights are removed, usually in order to enhance the network’s generalization performance [1]. In this paper we focus on pruning and its applications in the context of Evolved Autonomous Agents (EAA). To demonstrate this we propose a new network minimization algorithm, Evolutionary Network Minimization (ENM). ENM prunes successful networks while maintaining their fitness and principal functional characteristics intact. It is essentially a standard genetic algorithm with an additional step during reproduction in which weights are irreversibly eliminated. Minimization of successful agents is motivated by several goals. First, it enables better understanding of the way an agent works: Finding a small underlying skeleton of the agent enables a direct inspection of its functioning. Second, the minimized agent is more efficient computationally, hence, if the agent is implemented in hardware, fewer components will be needed. Third, the size and complexity of the minimized agent may stand as an estimate for the complexity of the task it solves (while the size of the original agent is arbitrary).
Most pruning algorithms in the neural networks literature, such as pruning by the weights magnitude, Optimal Brain Damage (OBD) [2], Optimal Brain Surgeon (OBS) [3] and contribution based pruning [4, 5], are non evolutionary methods. OBD and OBS analytically predict the effect of a weight removal on the error function and all algorithms allow for retraining after the elimination of weights3 . However, in the case of EAA no explicit error function is available for such an analysis and for retraining. Furthermore, the neurocontrollers may be fully recurrent, as in the agents presented throughout this paper. Pruning algorithms incorporating an explicit network complexity term in the fitness function and performing an evolutionary search have also been suggested [6, 7]. ENM does not rely on such an arbitrarily defined explicit complexity term in the fitness function. It is more akin to variable length encodings algorithms, an example of which is a genetic algorithm with an evolved mask representing the eliminated weights [8]. Within ENM, the pruning is based on accumulation of eliminated weights. The rest of this paper is organized as follows : Sect. 2 describes the EAA environment. Sect. 3 presents the ENM algorithm, and in Sect. 4 we describe and analyze the results of minimizing neurocontrollers of autonomous agents. These results testify to the effectiveness and consistency of ENM, and illustrate that the minimized agents preserve the essential functional characteristics of the original ones. They further demonstrate the usage of a minimized network for understanding the agent’s functional characteristics. We summarize in Sect. 5.
2
The Evolved Autonomous Agents Environment
The EAA model used in this paper is described in detail in [9]. A population of agents performing a task of navigation and foraging is evolved in a discrete grid arena surrounded by walls. Poison items are scattered in the whole arena, while food items are scattered only in a “food zone” in one corner. The agent’s goal is to find and eat as many food items as possible during its life, while avoiding poison. Its fitness is the number of food items minus the number of poison items it has consumed, normalized by the total number of food items that were available giving a maximal value of 1. Due to the short lifetime, a fitness score of 1 is practically unattainable. The sensory system of the agent is made of 5 sensors: front, left, right and here provide an indication whether there is a resource (food or poison, without a distinction between the two), a wall or an empty cell in front, left-front, right-front and underneath the agent, respectively. The smell sensor gives an indication of food or poison if they are directly underneath the agent and a random reading otherwise. In several simulations the agent was also equipped with a position sensor indicating its x and y grid coordinates, otherwise it had only local information from its 5 sensors. Four motor neurons dictate movement forward, a turn left or right, and control the state of the mouth (open or closed). Eating takes 3
OBS suggests a formula for recalculating the magnitude of the weights, rather than simple retraining.
place if the agent opens its mouth and is neither moving nor turning. The agent is controlled by a fully recurrent neurocontroller of binary McCulloch-Pitts neurons, with the exception of the non-binary sensor neurons which have no input from other neurons. Previous analysis [9] revealed that successful agents possess one or more command neurons that determine the agent’s behavioural strategy. Artificially clamping these command neurons to either constant firing activity or to complete quiescence causes the agent to constantly maintain one of the two behavioural modes it exhibits, regardless of its sensory input. These two behavioural modes are exploration and grazing. Exploration, which takes place when the agent is outside of the food zone, consists of moving in straight lines, ignoring resources in the sensory field that are not directly under the agent, and turning at walls. Grazing, which takes place when the agent is in the food zone, consists of turning towards resources to examine them, turning at walls and maintaining the agent’s location on the grid in a relatively small region. In this paper we apply the ENM to three successful evolved agents: Z10, S22 and SP10 with 10, 22 and 10 neurons (including the motors), respectively. SP10 is also equipped with a position sensor.
3
The ENM Algorithm
The ENM algorithm receives as input an evolved neurocontroller and outputs a pruned neurocontroller evolved to achieve a high fitness on the original task. ENM is a genetic algorithm which evolves a population of genomes, directly encoding the networks’ weights. The algorithm starts by cloning the input network’s genome to form the first generation. Then, a standard genetic algorithm is used, with an additional important elimination step after the selection, crossover and mutation steps. In this step, weights are eliminated (zeroed) according to an elimination criterion. In order to create pressure for the accumulation of eliminated weights, they cannot be revived by mutation. The fitness function used for evaluating the new generation genomes is simply the fitness function of the original task. The evolution continues until both the average fitness and the average number of non-pruned weights in the population converge to stable values. The result of the evolutionary process is then a population of networks from which ENM selects and outputs the network with the best fitness. The elimination criterion used throughout this paper is as follows: Weights in the range [−0.2, 0.2] are eliminated, while keeping all weights in the range [−10, 10] by truncating oversized weights in order to keep them within a reasonable distance from the elimination range. Before running ENM, the input network is scaled, without changing the network’s operation, such that the magnitude of all weights is smaller than the maximal value (10). The genetic operators are as in the original evolution creating the successful agents: Roulette wheel selection is used with uniform point crossover (probability of 0.35). Mutation is applied to all non zero weights, adding a uniformly distributed random value between −0.2 and 0.2.
4
Results
4.1
Effectiveness and Consistency
We applied ENM for minimizing agent S22 (described in Sect. 2). Figure 1 shows the average fitness and average fraction of non zero weights throughout the minimization process. Evidently, ENM markedly reduces the number of network connections while preserving high fitness values. The minimized neurocontroller consists of only 26 weights out of the original 594. The average fitness declines in the beginning of the evolutionary process, then is gradually restored. This initial decline is due to the elimination of small but important weights that were mutated to magnitudes below the elimination threshold in some of the agents during the first generations. As the evolution continues, the important weights are “pushed” farther away from the elimination threshold, allowing for the regain of the average fitness.
1
1
0.9
0.9
0.8
0.8
0.7 0.6 0.5
0.7 0.6 0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1 0 0
0.1 5000
10000
15000
20000
25000
30000
0 0
200
400
600
800
1000
1200
1400
1600
1800
2000
Fig. 1. Average fitness and average fraction of non zero weights across generations during ENM of S22. The fitness is normalized such that the fitness of the original agent equals 1. The averaging is over a population of 100 agents. (a) ENM during 30000 generations. (b) Focusing on the first 2000 generations, where the high fitness is restored after a decrease in the first generations A comparison of the ENM with random pruning and magnitude based pruning is presented in Fig. 2 . Magnitude based pruning removes in each iteration the weight with the smallest absolute size4 . ENM prunes 95% of the weights while maintaining the original fitness almost intact, whereas the other methods result in a large decrease in the fitness already at much smaller levels of pruning. In order to test the consistency of the ENM, we repeatedly applied it 10 times for minimizing each of the evolved agents S22, Z10 and SP10. Fig. 3a 4
This is done without retraining since the agents are evolved, rather than trained.
! "$# &% "'( )*$
0.4
0.3
0.2
0.1
0
−0.1
−0.2 0
10
20
30
40
50
60
70
80
90
100
Fig. 2. Comparison of different pruning methods for minimizing S22. The agent’s fitness is plotted against the percentage of pruned weights, using three different pruning methods: ENM, magnitude based pruning and pruning the weights in random order (mean and standard deviation across 25 runs). ENM converged to a final network architecture after 95% of the weights were pruned
presents the fitness values of the resulting minimized agents. In all three cases the fitness is very close to the fitness of the original intact agent, with small standard deviations, testifying to the consistency of ENM. Figure 3b presents the number of weights of the minimized agents. Evidently, the number of weights is very small compared with the original intact agents and is consistent across different ENM runs.
4.2
Preservation of Functional Characteristics
Several findings lead to the conclusion that the ENM, although pruning most of the weights, preserves the essential functional characteristics of the network. Observing the agents reveals similar patterns of behavior in the original and minimized agents. For example, Z10 has a unique characteristic, not found in other successful agents evolved in the same environment: When walking along a wall, the agent keeps a one cell distance from it. The ENM minimized agent of Z10 exhibits this exact same unique characteristic. Repeating the ENM many times and considering the surviving weights in each, we note that about half the weights of a typical minimized network (9 out of 21) are weights that survive in all the runs (Fig. 4). This observation indicates that the ENM strives to preserve important weights: If the surviving weights were to be selected randomly then the number of surviving weights should have been
0.5
! "$#%"$&('*)
30 28
@A B >? =9
0.4
