Keywords: antnet, ant colony optimisation, beehive, routing, reinforcement learning, swarm intelligence. 1. Introduction. In today's fast growing Internet traffic ...
Antnet Routing Algorithm with link evaporation and multiple ant colonies to overcome Stagnation Problem Abstract –Antnet is a software agent based routing algorithm that is influenced by the unsophisticated and individual ant’s emergent behaviour. The aim of this paper is twofold, firstly to introduce improvements to antnet routing algorithm and then to critically review the work that is done around antnet and reinforcement learning applications on routing. In this paper a modified antnet algorithm for packet based network has been proposed, which offers improvement in the throughput and average delay by detecting and dropping packets routed through the non-optimal routes and the effect of traffic fluctuations has been limited by applying boundaries to the reinforcement parameter. The round trip feedback information supplied by the software agents is reinforced by updated probability entries in the distance vector table. In addition link usage information is also used to prevent stagnation problems. Also discussed is antnet with multiple ant colonies applied to packet switched networks. Simulation results show that the average delay experienced by data packets is reduced for evaporation for all cases when non-uniform traffic model traffic is used. However, there was no performance gain on the uniform traffic models. In addition, multiple ant colonies was applied to the packet switched networks and results were compared with the other approaches. Results showed that the throughput could be increased when compared with other schemes, but with no gain in average packet delay. Keywords: antnet, ant colony optimisation, beehive, routing, reinforcement learning, swarm intelligence.
1. Introduction In today’s fast growing Internet traffic conditions changes and failures occurs at some parts of the network from occasionally, in an unpredictable manner. Therefore, there is a need for an algorithm to manage traffic flows and deliver packets from the source to the destination in a realistic time. The routing algorithm is the key element in networks performance and reliability, and can be seen as the “brain” of the network which manages the traffic flow through the network. An ideal routing algorithm should be node and link independent, and be able to deliver packets to their destination with the minimum amount of delay, regardless of the network size and the traffic load. The only way to achieve this would be by employing an intelligent and distributed routing algorithm. The routing algorithms currently in use lack intelligence, and need human assistance and interpretation in order to adapt themselves to failures and changes within the network. Routing is considered to be an NP-Hard Optimization problem, therefore widely used optimisation problems have been applied in the literature: to name a few, Genetic Algorithms, Neural Networks, Simulated Annealing, Software Agents and Reinforcement Learning. In recent years, agent based systems and reinforcement learning have attracted researchers’ interest. This is because these methods do not need any supervision and are distributed in nature. Swarm intelligence particularly ant based systems [1], Qlearning [2] methods and hybrid agent based Distance Vector algorithms [3] have shown promising and encouraging results. The ant-based approach applied to routing problem was first reported in [4], which itself was influenced by the work done in [5] on the software agents used for control in telecommunication networks. An improved version of the algorithm in [4] was applied to the connection-oriented systems [6]. In [7] for the first time ant based routing was applied to the packet based connection-less systems. In [8] the idea was applied to application of mobile agents in adaptive routing, namely antnet, which is also used as the basis in this work. In real life, ants deposit some kind of chemical substance called pheromone to mark the path that they used. Then on their way back they choose the path with the most pheromones which becomes the shortest path. Ants (nothing but software agents) in antnet are used to collect traffic information and to update the probabilistic distance vector routing table entries. Recently research focused on improving antnet’s performance. [9] carried out performance analysis of antnet and compared with Open Shortest Path First (OSPF) routing algorithm and found out that, in low traffic loads antnet’s end-to-end packet delay is larger than the OSPF whereas it is opposite in heavy traffic loads. [10] Implemented on simple network and compared with OSPF and suggested to tune number of variables automatically for optimum performance. Although ant based routing algorithms have shown some interesting results, they are still far away from being ideal. Tests carried out have shown that by detecting and dropping a few of the packets, routed through the non-optimal routes, the average delay per packet is decreased and network throughput is increased. In addition, one of the major problems with antnet approach is the stagnation, in which the network freezes and consequently routing algorithm gets trapped in the local optima and therefore is unable to find new improved paths [11]. A number of methods have been used to overcome this problem, such as: evaporation, ageing, pheromone control and hybrid algorithms [12]. However, most of these methods, except for evaporation,
are rather complex or not efficient and requiring other heuristics such as genetic algorithms [13]. Evaporation links is a simple method and adding negligible overhead to the node itself. In this paper the focus is on the antnet routing algorithm. In the original antnet algorithm the only feedback provided to the system by the software agents is the trip time observed on their way from source to destination. This feedback signal is reinforced by updating the related probability entry in the distance vector table. In this work link usage information is used as the second feedback to the system in order to prevent stagnation problems. For every link the usage information is held at every node based on a predefined time window. Routing table entries then are reinforced with negative signal based on the calculated evaporation information based on the link usage statistics. In addition multiple colonies used in antnet on packet switched networks which also improved the performance of the algorithm compared to the original approach. However, this improved node and algorithm complexity as separate data variables are kept for each colony. The paper is organised as follows: In section two the behaviour ants in real life will be given. This will be followed by the description of the antnet routing algorithm. Section four describes the test environment and outlines the assumptions made. The proposed improvements together with evaporation and multiple ant colonies is given in section 5. Results and discussion is given in section 6 and 7, critical discussion into antnet improvements and other nature inspired algorithms is given in section 8 which is followed by the concluding remarks.
2. Ants in real life Individual ants are unsophisticated and simple insects, but when they act in a collaborative way that they are capable of performing a variety of complex tasks [14]. Real ant behaviour has been observed by many researches and as a result a set of collective behaviour of several species has been agreed [1, 4, 14]. Ants in real life: • • • • • • • • • • • •
explore particular areas for food build and protect nests sort brood and food items cooperate in carrying large items immigrate as a colony leave pheromones on their way back preferentially exploit the richest available food source store information on the nature (uses world as a memory) make decision on a stochastic way find the shortest paths to their nests or food source are blind, cannot foresee the future have very limited memory.
Fig. 1. Ants laying pheromones. It is believed that these behaviours emerge from interactions when in a large colony and environment. A notion of stigmergy , which is a form of indirect communication through the environment, is used by ants [1,4]. They lay pheromone, some kind of chemical, to mark the path used to find the food source, which evaporates over a certain time. For example, in Fig. 1(i) initially ants have no idea which direction to go. Path is chosen purely randomly. It is expected that on average, half of the ants will choose the shorter path and the other half will choose the longer path, since they have no
information or knowledge of the paths ahead. It can be observed from Fig. 1(ii) that the ants chosen the shorter path (see Fig. 1(ii) will reach the food source and return to their nest well before the others. Following the foot steps of these ants, more will choose the shortest path, identified by the laid pheromones, see Fig. 1(iii)
3. Antnet routing algorithm Antnet [15, 16] is an agent based routing algorithm, influenced by the real ants behaviour, where ants (nothing but software agents) explores the network to find the optimal paths from the randomly selected source destination pairs. Moreover, while exploring the network ants update the probabilistic routing tables and build a statistical model of the nodes local traffic and uses these information to communicate with each other. To collect network statistics and to update the routing table, the algorithm uses two types of agents namely; forward and backward agents. In each node there are two types of queues, low priority queues are used by the data packets and the forward ants and high priority queues are used by backward ants. In [15] forward ants also uses the high priority queues and use some heuristics in order to predict T rather than carrying this information[15].
Fig. 2. Probabilistic routing table and traffic statistic. Agents communicate via two data structures that are stored in every node (see Fig. 2) as outlined below [15,16]: i.
A distance vector routing table Tk with distance metric is defined with probabilistic entries where, for each possible destination d and neighbour node n, there is a probability value Pnd, which reflects the goodness of the link (path), given as:
∑P
nd
nε N
= 1, d ε [1, N ] ,
k
(1)
Ν k = {Neighbours (k )} An array Mk (µd, σd2, Wd) defines a simple statistical traffic model experienced by node k over network. Where Wd is the observation window used to compute the estimated mean µd and the variance σd2 parameters 1 given as, respectively:
ii.
µ d = µ d + η(o k →d − µ d )
(2)
σd2 = σd2 + η((o k →d − µ d )2 − σd2 ) ,
(3)
where ok →d is the new observed trip time experienced by the agent while travelling from node k to destination d. The antnet behaviour is summarised as [16]: 1.
At regular intervals (defined by the user) from every node an agent A is sent to a randomly selected destination node d.
2.
Each agent first defines the possible neighbour nodes (unvisited neighbour nodes) at the current node by using its routing table. Then, an agent chooses the next node n within the identified possible (unvisited) nodes by using the probabilistic values in the table, taking into account the state of the associated queue as follows2:
1 2
η weights is the number of samples that affect the average and set to 0.05 [14]. α weights ln with respect to routing table entries and set to 0.3[14].
′ = Pnd
Pnd + α ln , 1 + α( N k − 1)
(4)
where ln is the heuristic correction value with respect to the probability values stored in the routing table that gives a quantitative measure of the queue waiting time, defined as:
ln = 1 −
qn
,
Nk
(5)
∑q n′=1
n′
where qn is the bit length (or number of packets if packet size is fixed) waiting to be sent to the queue on the output port of node n. 3.
If the selected port is full, then an agent is stored in the first-in-first-out output buffer for a time being until availability of a port. It is assumed that the buffer size is infinitely large.
4.
Information on every node visited and time elapsed since the agent is despatched from the source are stored in the stack memory S(k) carried by the agent.
5.
If an agent is forced to visit previously visited node, i.e. if cycle exists, then it deletes all the entries for the nodes associated with the cycle.
6.
On reaching the destination, the agent changes its type to a backward agent and uses the same path to return back to its source node. On the return journey, the agent pops every visited node from its stack and updates the associated routing table entries (probabilities) for all the nodes along the path by using the following rules: (i) Mk (µd, σd2, Wd) is updated with values stored in S(k). The time elapsed to arrive (for the forward ant) at the destination ( ok →d ) is used to update the estimated mean µ΄d, variance σ΄d2 and the best trip value over the observation window W΄d. (ii) Tk is updated by incrementing the probability of selecting neighbour f when the destination is d΄ given as:
P fd ′ = P fd ′ + r (1 − P fd ′ ) .
(6)
Probabilities Pnd΄ for rest of the neighbour nodes are updated as:
Pnd ′ = Pnd ′ − rPnd ′ , nεN k ,
n≠ f .
(7)
The goodness of the path taken by the ant, and is used as the only feedback (reinforcement) information used in the original antnet routing algorithm to update the routing tables (eq. 8).
I sup − I inf W r = c1 best + c2 (I sup − I inf ) + (T − I inf ) T
(8)
where T is the observed trip time, c1 and c23 are the constants used to control the effect of the last T, Wbest is the ant’s best trip time for a given destination and last observation period. Iinf = Wbest and I sup = µ * z * (σ
| w | ) 4 are the
lower and upper limits for the confidence interval µ, respectively. More details and discussion about the variables and parameters used can be found in [16].
4. Test Environment and assumptions made 3 4
c1 and c2 are chosen experimentally to be 0.3 and 0.7 respectively.
z =1
(1 − γ ) , γ lies is the confidence, γ є [0.75, 0.8].
In this work, 36-Node irregular grid and 29-Node randomly created network will be used for carrying out the tests. Evaporation criteria are applied to the modified antnet routing algorithm [17] and are implemented in the following environment.
Fig. 3. 29-Node random network • • • • • • • • • • •
Fig.4. 36-Node irregular grid
Algorithms are implemented in the C language and mapped onto a parallel environment using a Parallel Virtual Machine (PVM) model. Inter-process behaviour is simulated by assigning processes to different nodes both on the same or different machines. Poisson traffic distribution is used on non-uniform data traffic where 33% of the traffic is forwarded to nodes 1 and 21 with 29-node random network (see Fig. 3). Non-uniform data traffic is used with the 36-node irregular grid where nodes 0-5 generate traffic for the nodes 30-35 and vice versa (see dark coloured nodes in Fig. 4). There are only two paths available from left of the grid to the right of the grid, and vice versa. No packet creation is destined to the immediate neighbours. Paths discovery and initialisation of the probabilistic routing table entries are completed in 10 seconds. Ant creation rate is set to 10 seconds per node For each simulation, packet generation is stopped after creation of 2500 packets per node and simulation is stopped after all packets arrive at their destinations or are detected and deleted from the network. Every simulation is run 10 times and the average of the results is used for accuracy. It is assumed that packet size is fixed and there are no packet losses. All experiments are implemented for varying link evaporation rates, since it has a significant effect on the performance of the algorithm.
Average packet delay is used for evaluating the performance of the algorithm, which is the average delay experienced by the packet while it is being routed from source to the destination.
5. Antnet Modifications Based on the original antnet routing algorithm, initially two modifications has been proposed and tested. In addition, [18] showed that number of ants in the network has to be limited for optimum performance, their results suggested to control the ant rate between 0.40-0.70 per packet after a series of test, we have also limited the number of ants in the systems. 5.1 Deleting aged packets During the simulations run on the original antnet routing algorithm it was discovered that some packets travel within the network for a number of hops until they reach their destination. This problem occurs because of probabilistic nature of the routing tables, where few packets are directed to non-optimal routes and cycles. As explained earlier there is a mechanism that
deletes the memory associated with the cycles previously visited by the forward ants. A simple rule is defined to detect and drop these packets from the network as follows: if PACKET AGE > 2 x NO_OF_NODES DROP PACKET
(9)
This rule is based on the information gained from experimental results. It was observed that only 1% and 1.2% of the packets experience this problem for 29-node and 36-node configurations, respectively. Further details regarding packet loss information can be found in [17].
5.2 Limiting the effect of r due to traffic fluctuations The reinforcement r applied to the routing table entries is limited by the lower and upper bounds defined as follows: if (NO_OF_NODES
