SLA-Based Inter-Domain QoS Routing Rui Prior1 and Susana Sargento2 1 LIACC, University of Porto, Portugal 2 Instituto de Telecomunicações – University of Aveiro, Portugal
[email protected],
[email protected] Abstract This technical report describes a solution for inter-domain QoS to be part of the framework for end-to-end QoS control. The solution is based on the use of virtual-trunk type aggregates, defined by means of Service Level Agreements, for data transport between peering domains. The problem is formally stated and formulated in Integer Linear Programming. As a practical solution, we define an extension to the BGP routing protocol to transport three different QoS metrics (light load delay, assigned bandwidth and a congestion alarm), and a path selection algorithm using a combination of these metrics.
1
Introduction
Today’s commercial Internet has far surpassed the initial concept of a data transport network mostly for academic and military purposes. Its multi-service nature becomes evident with the ever increasing number of online audio and video broadcasting services, podcasting, instant messaging with audio and video extensions and internet telephony services like skype, among others. Its success in replacing the traditional networks to provide multimedia services with real-time requirements and in providing new services is, however, conditioned by the ability to provide a high and guaranteed quality level. The introduction of Quality of Service (QoS) support in Internet routing is, therefore, of major importance towards this goal. Minimization of the number of AS hops provided by the current, non-QoS-aware inter-domain routing protocol does not guarantee availability of resources required to transport the data or better QoS: a path with a smaller number of AS hops may traverse congested links or may comprise a larger span of fiber and/or a higher number of traversed routers than another one with a larger number of AS hops. Other metrics like delay, loss probability or available capacity, need to be taken into account in choosing the paths. Internet routing is performed in two layers: intra-domain routing, within networks controlled by a single operator or administrative entity, and inter-domain routing, for the interconnection and transport service between different administrative domains. Though much attention has been paid to QoS in packet switching networks in general and IP networks in particular, most of the effort has been centered on intra-domain QoS. We have now readily available solutions based on the traffic engineering capabilities of MultiProtocol Label Switching (MPLS) [2] and on QoS extensions of intra-domain routing protocols such as OSPF [3]. Much less has been done in the scope of inter-domain QoS, though. Several different factors contribute to the difficulty of solving this problem. The Internet is a complex entity, comprised of Autonomous Systems (AS) managed by very diverse operators. If it is to be widely deployed, an inter-domain QoS routing mechanism must be capable of handling the heterogeneity of the Internet and impose minimum requirements on intradomain routing in order to be appealing to the different operators. The introduction of QoS routing should not disrupt the currently existing inter-domain routing: both should inter-operate, allowing for incremental deployment among the different networks, and the stability of intra-domain routing should not be significantly affected by the QoS mechanisms. Conciliating the required stability with the dynamic nature of QoS information is a major challenge in inter-domain QoS routing. A related issue is that of scalability: a solution that does not scale to the dimension of the Internet cannot be deployed widely enough to be useful. A further requirement is the optimization of the use of network resources. As bad as the current path selection mechanism is, based mostly on the minimization of the number of traversed ASs, it has the advantage of avoiding unnecessarily long paths, in general. A minimization of the number of AS hops is not, however, guaranteed to optimize the amount of resources required to transport the data from origin to destination, as a path with a smaller number of AS hops may comprise a larger span of fiber and/or a higher number of traversed routers than another one with a larger number of AS hops. In this respect, a better metric for optimization would be the expected delay in light load conditions that, nowadays, when the bandwidth in backbones and transit networks is quite high, corresponds approximately to the propagation delay, directly proportional to the traversed span of fiber. Minimization of this metric by the path selection mechanism 1
would lead to a more rational use of the network resources, as well as better QoS to data packets, which would suffer smaller delay penalties, as long as the path is not congested1. An appropriate choice of the path selection metrics is key to the behavior of QoS routing. Two fundamentally distinct types of path metrics may be used for the selection: static or dynamic. Static metrics, such as the bottleneck link capacity, do not vary over time, meaning that no information exchange is required after the initial propagation of routes to the entire network and the chosen routes are stable. However, they are unable to express the dynamic character of the networks, and therefore QoS routing based solely on static metrics is unable to prevent and/or react to network congestion. By using dynamic metrics, such as the available path bandwidth or the time window averaged instantaneous packet delay, it is possible to react to congestion. However, a high overhead is incurred in message exchange for updated metrics and in path recomputation, and the route stability property is lost. QoS routing based on purely dynamic metrics cannot, therefore, be applied to the Internet scale. Two main approches may be used for reducing the overhead associated with QoS routing with dynamic metrics, in order to make it scale [4]. The first one is based on the distinction between significant changes in the metrics that trigger QoS information updates and path recomputations from minor changes that do not. The second one is to convey longer term information on the path characteristics through statistical characterization of the route metrics. Both approaches provide coarser-grained information: the first one in terms of scale, the second one in terms of time. In a commercial Internet, data transport is a service subject to contracts between the interested parties. Network operators establish Service Level Agreements (SLA) between them that specify the conditions for data transport, and these agreements are translated to path selection policies in the routing protocol, frequently configured by hand. The ability of inter-domain routing to automatically adjust to these agreements should not be overlooked. The combined use of SLAs for transport service with an inter-domain QoS routing protocol is preconized in the DAIDALOS project [5], in the scope of which this work was developed. In this report we formally state the problem of SLA-aware inter-domain QoS routing and formulate it as an Integer Linear Programming (ILP) optimization problem, proving that routes thus obtained are cycle-free. We propose a practical solution for inter-domain QoS routing based on both static and coarse-grained dynamic metrics: it uses the light load delay and assigned bandwidth (both static) in order to improve the packet QoS and make better use of network resources, and a coarse-grained dynamic metric for path congestion to avoid overloaded paths. We define an extension to the Border Gateway Protocol (BGP) [6] routing protocol to transport these QoS metrics, and a path selection algorithm using a combination of these metrics. This report is organized as follows. The next section describes related work. Section 3 contains the formal description of the problem and its formulation in ILP. Section 4 presents the proposed protocol and the associated path selection algorithm. Finally, section 5 draws some conclusions and points out directions for further development of the work herein presented.
2
Related work
A framework for QoS-based Internet routing, adopting the traditional separation between intra- and inter-domain routing, was defined by Crawley et al. [4]. They discussed the goals of an inter-domain QoS routing and the associated issues that must be addressed, and provided general guidelines that should be followed by any viable solution to QoS routing in the Internet. However, they do not specify the set of QoS metrics to be transported or the algorithms for using such metrics in the choice of inter-domain routes. A series of statistical metrics for QoS information advertisement and routing, tailored for inter-domain QoS routing, though also applicable to intra-domain routing, were defined by Xiao et al. [12], along with algorithms to compute them along the path. These metrics, the Available Bandwidth Index (ABI), the Delay Index (DI), the Available Bandwidth Histogram (ABH) and the Delay Histogram (DH), convey information expressed in terms of one or more probabilistic intervals. Simulation results show that by using these metrics, selected routes are closer to optimality than when using static metrics; moreover, the overhead is lower and the stability higher than when using the corresponding instantaneous (purely dynamic) metrics. However, these approaches consider only a single QoS parameter, making it difficult to simultaneously satisfy different requirements. When optimizing by bandwidth, paths with large delay may be chosen while 1 If the links do not have these characteristics and the light load delay diverges significantly from the sum of propagation delays, the optimality regarding resource utilization will decrease, but the QoS optimization will remain intact.
2
others with less, yet sufficient, available bandwidth and much lower delay may be available. Conversely, when optimizing by delay, a route with low available bandwidth may be selected; switching to this route may cause congestion, increasing the delay. When the delay information is updated, the previous route might be selected again, and so on, causing route flapping (though on longer time scales than with dynamic metrics). Cristallo and Jacquenet proposed an extension to the BGP with a new optional and transitive attribute, QoS_NLRI, for the transport of several types of QoS information [13]. This work is focused on the specification of the attribute, including the formats for transporting the different parameters, such as reserved data rate or minimum one-way delay, and does not specify how the information is to be used in path selection. Simulation results demonstrating its use with (static) information on one-way packet delay are provided, though.
3
Inter-Domain QoS Routing with Virtual Trunks
In this section we formally describe the problem of inter-domain routing with virtual trunks and formulate it as an Integer Linear Programming (ILP) problem. This formulation can be used in a Mixed Interger Programming (MIP) solver to obtain optimal route sets for a given topology, configuration and traffic matrix. 3.1
Virtual trunk model of the Autonomous Systems
Though the use of some inner information of the ASs is important for inter-domain QoS routing, the exact topology and configuration of the ASs should not be used for two reasons: (1) the level of detail would be excessive, complicating the route computation task and, most important, (2) network operators usually want to disclose the minimum possible amount of internal information about their networks. In this work, we use a “black box” model where only externally observable AS information is disclosed. The intra-domain connections between edge routers are replaced by virtual trunks with specific characteristics interconnecting the peering ASs. Each virtual trunk has a specific amount of reserved bandwidth and an expected delay. These values depend on the internal topology of the AS, on the intradomain routing and on resource management performed by the operators, and usually reflect Service Level Agreements established between the operator of the AS and the operators of the peering ASs. The virtual trunk model of ASs is illustrated in figure 1. Each virtual trunk corresponds to a particular (ingress point, egress point) pair. Figure 1 illustrates the concept: an SLS between domain S1 and domain T1 specifies that X traffic may flow between S1 and domain T3; an SLS between domain T1 and domain T3 specifies that Y traffic may flow between T1 and domain D1. Aggregates are managed internally within each (transit) domain, ensuring that enough resources are assigned, and no imposition is made regarding mechanism used to this end.
Figure 1 - Virtual trunk type SLSs
The configuration of the virtual trunks must be consistent with the inter-domain links. In particular, the summed bandwidth of all virtual trunks traversing AS j and going to AS k must be less than the bandwidth of the inter-domain link connecting ASs j and k; similarly, the summed bandwidth of all virtual trunks coming from AS i and traversing AS j must be less than the bandwidth of the inter-domain link connecting ASs i and j.
3
3.2
Problem statement Let G = (V , E ) be an undirected graph with edge capacities ci , j and edge delays wi , j . Each node
represents an AS, and the edges correspond to the inter-domain links. Additionally, we define a set F of aggregate flows between pairs of nodes and a corresponding matrix of traffic demands a s ,d for all
( s, d ) ∈ V 2 , where s and d denote the source and destination nodes, respectively. Given any three nodes i , j and k , where i is directly connected to j and j is directly connected to k , there may be a traffic contract (SLS) stating that j provides a virtual trunk between i and k with reserved capacity ri , j ,k . The amount of data transported from i to k via j is, therefore, bounded by ri , j ,k . If no such contract exists, we say that ri , j ,k = 0 . Since each virtual trunk is mapped to an actual path inside the AS, it has an associated delay yi , j ,k , corresponding to the delay of that path. Call L the set of all virtual trunks (i, j , k ) . The virtual trunks must satisfy the conditions o j ,k +
∑r
i , j ,k
≤ c j ,k , where o j ,k is the capacity reserved
i
for traffic originated at node j and destined to or traversing node k , and t i , j +
∑r
i , j ,k
≤ ci , j where ti , j is
k
the capacity reserved for traffic destined to node j and originated at or traversing node i . The expected total delay suffered by packets of a given flow is the sum of the wi , j and yi , j ,k parameters along the path followed by the flow. Our goal is to find the set of hop-by-hop routes that minimize the delay while guaranteeing that interdomain link and virtual trunk capacities are not exceeded. 3.3
Problem statement transform
In order to formulate the stated problem as an ILP problem, we first transform the original graph into a transformed graph where the virtual trunks are explicitly accounted for. 3.3.1
Transform graph
It is possible to transform the graph into a directed multigraph where each edge corresponds to a virtual trunk; however it is difficult to account for the delays of all links in the original graph (inter-domain links) without counting some of them twice. Moreover, graphs are better studied and easier to deal with than multigraphs. Therefore, we add virtual nodes to the directed multigraph in order to obtain a resulting directed graph. Virtual trunks are established between an entry link and an exit link. Therefore, we add two virtual vertices per link of the original graph, one for each direction, and virtual trunks are represented by edges connecting these virtual nodes. Moreover, in order to forbid a node of the original graph from being traversed directly (instead of via a virtual trunk) we split each original node into two: one source virtual node with outgoing edges only and one destination virtual node with incoming edges only. Flows on the transform graph exist between source virtual nodes and destination virtual nodes. A very simple example of an original graph and its transform with all possible virtual trunks is shown in figure 2. Link (A,B) on the original graph is represented by virtual nodes AB and BA, virtual trunk (A,B,C) is represented by an edge connecting AB to BC, node A is represented by the virtual source and destination nodes AS and AD, and flow (A,D) is represented by flow (AS,DD), for example.
a) Original graph
b) Transform graph Figure 2: Simple network with 4 nodes
4
The solid edge connecting the virtual nodes ij and jk correspond to the virtual trunk for sending traffic from node i to node k via node j , and has capacity ri , j ,k (that of the virtual trunk), and delay
yi , j ,k + w j ,k , where yi , j ,k is the internal delay of the virtual trunk and w j ,k the delay of the inter-domain exit link. Each dashed edge ( j S , jk ) corresponds to the inter-domain exit link from node j to node k , and has delay w j ,k and capacity o j ,k . Each dotted edge (ij , j D ) corresponds to the inter-domain entry link in node
j from node i , and has zero delay and capacity ti , j . Notice that even if there is no explicit o j ,k and ti , j values, they may be computed using o j ,k = c j ,k −
∑r
i , j ,k
and t i , j = ci , j −
i
∑r
i , j ,k
.
k
In the example of figure 2 there is only one possible path from A to C, corresponding to the virtual trunk through node B. Traffic sent from A to C is subject to a delay equal to the sum of wa ,b (from the dashed edge ASAB) and y a ,b ,c + wb ,c (from the solid edge ABBC); the dotted edge BCCD has zero delay. Regarding bandwidth, it is constrained by oa ,b , ra ,b ,c , and t b ,c , all of them shared with other traffic. Figure 3 provide a somewhat more complex example with a cyclic graph and its transform containing all possible virtual trunks. Though the transform graph looks overly complex when compared to the original one, the number of variables and constraints in the ILP formulation is not increased, since a formulation based on the original graph would require variable unfolding in order to be linear. Also keep in mind that an undirected graph has half the number of edges of the equivalent directed graph.
a) Original graph
b) Transform graph Figure 3: Cyclic network with 5 nodes
3.3.2
Generation of the transform graph
In this section we present an algorithm for the generation of the tranform graph G ' = (V ' , E ' ) from the original graph and the set of virtual trunks, informally described above. The algorithm is as follows: 1. For each node i ∈ V 1.1. Add node iS to the set S of sources and to the set V ' of nodes 1.2. Add node iD to the set D of destinations and to V ' 2. For each (undirected) edge {i, j} ∈ E 2.1. Add node ij to V ' 2.2. Add node ji to V ' 2.3. Add edge (ij , j D ) to the set E ' of edges 2.3.1.
Set capacity c'ij , jD = ti , j
2.3.2.
Set delay w'ij , jD = 0 5
2.4. Add edge ( ji, iD ) to E ' 2.4.1.
Set capacity c' ji ,iD = t j ,i
2.4.2.
Set delay w' ji ,iD = 0
2.5. Add edge (iS , ij ) to E ' 2.5.1.
Set capacity c'iS ,ij = oi , j
2.5.2.
Set delay w'iS ,ij = wi , j
2.6. Add edge ( j S , ji ) to E ' 2.6.1.
Set capacity c' jS , ji = o j ,i
2.6.2.
Set delay w' jS , ji = w j ,i
3. For each (directed) virtual trunk (i, j , k ) ∈ L 3.1. Add edge (ij , jk ) to E ' and to the set L' of virtual trunk edges 3.1.1.
Set capacity c'ij , jk = ri , j ,k
3.1.2.
Set delay w'ij , jk = yi , j ,k + w j ,k
4. For each flow (i, j ) ∈ F 4.1. Add flow (iS , j D ) to the set F ' of flows 4.2. Set traffic demand a 'iS , jD = ai , j When the algorithm finishes, we have the transform graph G ' = (V ' , E ' ) , the associated edge capacity and edge delay matrices C ' and W ' , a set L' ⊂ E ' of virtual trunk edges, a set S ⊂ V ' of source nodes and a set D ⊂ V ' of destination nodes, a set F ' of flows, and the respective traffic demand matrix A' . 3.3.3
Complexity of the transform graph
The number of nodes and edges of the transform graph G ' is related to the original (undirected) graph G and the set of virtual trunks in the following way. The number of nodes is two per node of the original graph (one source and one destination, e.g., AS and AD) plus two per edge of the original graph (one for each direction, e.g., AB and BA). The number of edges is four per edge of the original graph (combinations of source/destination and transmission/reception, e.g., (AS,AB), (AB, BD), (BS,BA) and (BA,AD)) plus one per virtual trunk (e.g., (AB,BC)). In the example of figure 3, the original graph has 5 nodes, 5 edges and 12 possible virtual trunks. The transform, therefore, has 20 nodes (2*5+2*5) and 32 edges (4*5+12). 3.3.4
Conversion of routes from the transform to the original graph
A route p ' on the transform graph may be converted back to a route p on the original graph by analysing the traversed edges. Each traversed edge on the transform graph corresponds to a traversed node on the original graph according to the following table: Edge on the transform graph
Node on the original graph
(iS , ij ) (ij , jk ) ( jk , k D )
i j k
For example, the route (BS,BC,CE,ED) on the transform graph of figure 3.b) corresponds to the route (B,C,E) on the original graph. 3.4
Formulation as ILP problem
We now formulate our bandwidth-constrained global route and delay optimization hop-by-hop routing problem as an ILP problem with boolean variables using the transform graph. In the transform graph, 6
formulation is somewhat simpler than in a generic graph, since some restrictions are already enforced by the topology (since there are no incoming edges in source nodes, it is not necessary to use a restriction disallowing incoming traffic for flows originated at those nodes, and similarly for destination nodes). Our objective is to minimize the global delay while respecting the bandwidth limits, assuming that the network has enough capacity to satisfy all demands. In addition to the transform data obtained by the above described algorithm, let us define a set of positive flow weights bs ,d for all ( s, d ) ∈ F ' . Two different kinds of optimization may be obtained by using different weight values. The first alternative uses bs ,d = 1, ∀( s, d ) ∈ F ' , stating that all flows have equal importance; in this case, optimization is performed on a per-route basis. The second alternative uses bs ,d ∝ a ' s ,d , ∀( s, d ) ∈ F ' , stating that a flow’s importance is proportional to its traffic demand; in this case, optimization is performed on a traffic volume basis. Let us define the boolean decision variables xijsd which take the value 1 if the flow ( s, d ) ∈ F ' is routed through the edge (i, j ) ∈ E ' and the value 0 otherwise. The problem can, thus, be formulated as follows:
∑ ∑b
Minimize
s ,d
w'i , j xis,,jd subject to
( i , j )∈E ' ( s ,d )∈F '
xis,,jd ∈ {0,1}, ∀( s, d ) ∈ F ' , (i, j ) ∈ E '
∑ a'
s ,d
x
s ,d i, j
(1)
≤ c'i , j , ∀(i, j ) ∈ E '
(2)
( s , d )∈F '
s ,d j ,k
∑x
s ,d s, j
= 1, ∀( s, d ) ∈ F '
(4)
s ,d i ,d
= 1, ∀( s, d ) ∈ F '
(5)
( j , k )∈E '
−
∑x
= 0, ∀( s, d ) ∈ F ' , j ∈ (V '−{s, d })
∑x
s ,d i, j
( i , j )∈E '
( s , j )∈E '
∑x
( i ,d )∈E '
∑ ∑x
t ,d i, j
≤ S ⋅ x ss,,id , ∀( s, d ) ∈ F ' , i ∈ S : ( s, i ) ∈ E '
j∈V ':( i , j )∈E ' t∈S t ≠d
∑
∑x
( i , j )∈E ':( j , d )∈E ' s∈S s≠d
s ,d i, j
=
∑ ∑x
s ,d j ,d
, ∀d ∈ D
j∈V ':( j ,d )∈E ' s∈S s≠d
(3)
(6) (7)
Constraint set (1) imposes boolean decision variables, meaning that flows cannot be split over multiple paths. Constraint set (2) means that the sum of all flows traversing an edge will not exceed its capacity. Constraint sets (3), (4) and (5) are the mass balance equations: (3) means that each flow entering a node that is neither source nor destination for that flow must leave it and vice-versa; (4) means that each flow leaves the source node once and, conversely, (5) means that each flow enters the destination node once. Constraint set (6) imposes hop-by-hop routing on the original graph: it means that if a flow from a given node to a certain destination leaves that node by a given link, no flow to the same destination traversing that node may leave that node by a different link. On the transform graph, it means that if a flow from a source to a destination traverses a given virtual node directly connected to that source, no other flows to the same destination may traverse a different virtual node connected to the same source. Finally, constraint set (7) prevents routing loops at the destination nodes of flows in the original graph by forcing flows arriving at a node directly connected to their destination virtual node to use that direct path. Failing this, a flow would be counted twice (or more) on the left hand side and only once on the right hand side, invalidating the equality. Theorem 1: Paths obtained through this optimization procedure are guaranteed to be cycle-free on the transform graph.
7
Proof: Satisfaction of conditions (4) and (5) implies that each flow leaves the source virtual node exactly once (4) and arrives at the destination virtual node exactly once (5), therefore the source and destination virtual nodes belong to the path. There are no incident edges to source virtual nodes, therefore these nodes cannot be in a cycle. Conversely, there are no incident edges from destination virtual nodes, therefore these nodes cannot be part of a cycle either. Now let p be a path with a cycle from a given source s ∈ S to a given destination d ∈ D and p * the same path with the cycle removed. The cycle may only include intermediate nodes, since source and destination nodes cannot be part of cycles. If the above constraints (notably the capacity constraints) are satisfied with path p from s to d , then they are also satisfied with path p * from s to d . Since
bs ,d > 0, ∀( s, d ) ∈ F ' and w'i , j > 0, ∀(i, j ) ∈ E ': i ∉ S ∧ j ∉ D , the cost of using p * would be lower than the cost of using p , therefore p could not be in the optimal route set, as a route set including it would not minimize the cost function. ■ Theorem 2: Paths obtained through this optimization procedure are also guaranteed to be cycle-free on the original graph. Proof: The proof is based on three lemmas regarding cycles without the destination node, cycles with the destination node and the preceding edge, and cycles with the destination node without the preceding edge. Lemma 1: A path satisfying the above conditions cannot contain cycles that do not include the destination node on the original graph. Proof: Let p1 be a path on the original graph from source s to destination d containing a cycle that does not include node d , and p '1 the equivalent path in the transform graph. Since it does not contain the destination d , the cycle on p1 contains a node i left by the flow twice by different edges, (i, j ) and (i, k ) , towards the cycle and towards the destination node. Therefore, in the transform graph, p '1 contains virtual nodes ij and ik . Constraint (6) implies that xii,,dj = 1 and xii,,kd = 1 . However, this contradicts constraint (4), therefore p1 can only contain cycles that include the destination node d . ■ Lemma 2: A path satisfying the above conditions cannot contain cycles that include the destination node and the preceding edge on the original graph. Proof: Let p2 be a path on the original graph from source s to destination d containing a cycle that contains node d and the edge incident to d , (i, d ) , and p '2 the equivalent path in the transform graph. In this case, the flow enters d twice by the edge (i, d ) , from the source and from the cycle. Therefore, in the transform graph, p '2 contains a cycle containing the virtual nodes id . This contradicts theorem 1, which states that p '2 is guaranteed to be cycle-free on the tranform graph, meaning that a cycle containing node d and the edge incident to d cannot exist in the returned route set. ■ Lemma 3: A path satisfying the above conditions cannot contain cycles that include the destination node but not the preceding edge on the original graph. Proof: Let p3 be a path on the original graph from source s to destination d containing a cycle that includes node d but not the edge incident to d , and p '3 the equivalent path in the transform graph. In this case, the flow enters node d twice by different edges, (i, d ) and ( j , d ) , from the source and from the cycle. Therefore, in the transform graph, p '3 contains virtual nodes id and jd , contributing two units to the left hand side of constraint (7). According to constraint (5), the destination virtual node can only be entered once, therefore this flow’s contribution to the right hand side of constraint (7) can only be one unit. Since the same applies to all flows, no flow on the original graph can have a cycle containing the destination node d but not the edge incident to d . ■
8
The preceding lemmas cover all possible cycles on the original graph, therefore all paths satisfying the above conditions are guaranteed to be cycle-free on the original graph. ■ 3.4.1
Reducing the number of variables
The transform graphs have particular characteristics that allow us to significantly reduce the number of decision variables. Notice that the source virtual nodes have only out-edges (dashed edges in figure 3.b)) and the destination virtual nodes have only in-edges (dotted edges in figure 3.b)). As such, the following conditions hold:
xis,,jd = 0, ∀( s, d ) ∈ F ' , (i, j ) ∈ E ': i ∈ (S − {s})
(8)
xis,,jd = 0, ∀( s, d ) ∈ F ' , (i, j ) ∈ E ': j ∈ (D − {d })
(9)
Therefore, we may restrict the domain of (i, j ) in variables xis,,jd to L'∪{(i, j ) ∈ E ': i = s ∨ j = d }. 3.5
Variant formulation
The problem formulation above assumed that a certain amount of capacity is reserved at the interdomain links for traffic generated at the transmitting AS (initial node) and destined to or traversing the receiving AS (terminal node), therefore not corresponding to any virtual trunk, and, similarly, that at the ingress policing a given amount of capacity was assigned to traffic destined to the AS itself. In terms of constraints, they are represented, respectively, by the capacities of the edges incident from the virtual source nodes (dashed) and by the capacities of the edges incident to virtual destination nodes (dotted). A perhaps more reasonable assumption is that traffic outside the virtual trunks may use all the capacity left unused by traffic inside the virtual trunks (including reserved but unused virtual trunk capacity). Adapting the problem formulation to this new assumption involves the following changes: 1. Restricting the domain of (i, j ) in constraint (2) to L' instead of E ' , that is, to the set of virtual trunk edges, removing the fixed capacity constraints outside the virtual trunks. 2. Introducing an additional constraint at each virtual node i corresponding to an inter-domain link
∑
∑ a'
s ,d
xis,,jd ≤ ci , ∀i ∈ (V '− S − D)
(10)
i∈V ':( i , j )∈E ' ( s , d )∈F '
where ci is the capacity of the corresponding inter-domain link in the original graph. This constraint set states that the sum of all flows traversing (leaving) the inter-domain link must be less than the link’s capacity. Usually, ci = c 'i , j .
∑
i∈V ':( i , j )∈E '
4
Proposed protocol and associated algorithms
While the ILP formulation of the inter-domain QoS routing problem presented in the previous section is useful as a baseline for comparison with real protocols in controlled environments where all the input data is known, it cannot be used in the implementation of a real protocol itself for several reasons: first, the problem of 0-1 integer programming is known to be NP-complete [14]; second, because it requires knowledge of the traffic matrix, which is not easy to obtain real utilization scenarios; and third because it requires knowledge of the virtual trunk SLAs, which are usually disclosed only to the involved peers. In this section we propose a virtual-trunk-aware inter-domain QoS routing protocol, based on an extension of BGP, for use practical implementation in real internetworks. 4.1
QoS routing
Currently, version 4 of BGP has become the de facto standard for inter-domain routing in the Internet. BGP is a path vector protocol for the exchange of reachability information between connected ASs (eBGP External BGP). It is also used to share information on learned routes among the different edge routers of a given domain (iBGP - Internal BGP). The routes selected by BGP are propagated to the intra-domain routing protocol used in the AS (usually referred to as the Interior Gateway Protocol – IGP) by the edge routers. The 9
reachability information is conveyed in UPDATE messages, each containing an advertisement of a new or changed route to a given network destination and/or a set of withdrawals of routes to destinations that may no longer be reached via the AS originating the UPDATE. Besides the destination prefix, information on the traversed ASs (AS_PATH) and on the next hop (NEXT_HOP) is provided for advertised routes. The reception of an UPDATE message triggers a three step decision process: (1) a degree of preference is assigned to the new route (if any) based on a set of policies; (2) one of the available routes to the destination is selected and propagated to the IGP; (3) if the route is different from the previously installed one, it is propagated to the peering ASs. The most common policy for path selection is the minimum number of hops in the AS_PATH. Though the AS_PATH length metric bears only a very loose relation to QoS parameters, BGP can easily be extended to convey virtually any kind of relevant QoS information. The decision processes may also be changed to use the QoS information (if present) for path selection without breaking backward compatibility. We extended BGP to transport and use three QoS metrics: assigned bandwidth (static), path delay under light load (static) and a dynamic metric for path congestion described below. 4.1.1
Metrics
Virtual trunk information is explicitly included using BGP to carry information on the amount of bandwidth contracted between two domains regarding data transport to a third one. The assigned bandwidth, reflecting traffic contracts, is essentially static. It is updated along the path to be the minimum, that is, the bottleneck bandwidth (concave metric). Notice that our model does not require explicit and quantified agreements, only that transport operators assign a certain capacity for data transport between their connected peers; explicit SLAs are just a means to guarantee that reasonable assignments are performed. Information on the expected delay in light load conditions is also carried. This information not only allows for the minimization of packet delays, but also for a more rational use of the network resources, since in high capacity links with significant length, such as those found in today’s inter-domain connections, it consists mostly on the sum of propagation delays [15], directly proportional to the traversed span of fiber, as long as there is no congestion. It also provides a lower bound for the expected packet delay. We argue that the minimization of this metric by the path selection mechanism will lead to a more rational use of the network resources, as well as better QoS to data packets, which would suffer smaller delay penalties. The light load delay metric is static, and is summed along the path (additive metric). The third QoS metric conveyed by our proposed extension is path congestion. The concept of congestion is deliberately vague and may, therefore, be translated into a coarse objective metric, minimizing the inherent overhead in message exchange and path re-computation typical of dynamic metrics. In our case, the congestion alarm is expressed by an integer with three possible values, whose meaning is the following: 0 – not congested; 1 – very lightly congested; 2 – congested. This metric is updated along the path to the maximum value (convex metric). In the most basic version, congestion may be inferred from the utilization of the aggregates; in a more advanced version, other parameters, such as packet loss, average length of traversed router queues or measured delay, may be used as inputs to the function for computing the alarm level of virtual trunk aggregates. The main requirement for the congestion alarms, the sole dynamic metric in our proposal, is that changes should be infrequent, for scalability and stability reasons; hysteresis and related techniques may be applied in the assignment of alarm levels to this end. An effective value of the congestion alarm is used for path selection instead of the received value, aiming at reducing the fluctuations in the usage of the aggregates. This effective value is the same as the received value, unless the received value is 1 and the route is already in use, in which case the effective alarm is 0. This means that when level 1 (light congestion) is reached, no domain should use that route to replace a previously established one, but domains that were already using the route should not switch to a different one unless a higher congestion level is reached. This behavior is meant to avoid the synchronized route flapping problem. The next section describes the propagation and updating of these metrics and their use in path selection. 4.1.2
Path selection algorithm
The three above mentioned QoS metrics are conveyed in the UPDATE messages by a newly defined Path Attribute, QoS_INFO, which is optional and transitive (meaning that ASs which do not yet support the extension simply forward the received value), and are updated by the BGP-speaking routers at each transit domain, taking into account the virtual trunks between the domain to which the route is advertised and the “next hop domain” for the route. Notice that these virtual trunks are shared among different source to
10
destination routes: in figure 1, for example, all traffic transported from T1 to D1 via T3 shares the T1:D1 virtual trunk, independently of being originated at S1 or S2.
Figure 4 - Propagation of metrics in the QoS_INFO attribute
Figure 4 illustrates the propagation of the delay, bandwidth and congestion alarm metrics in the QoS_INFO attribute of UPDATE messages. When the destination AS (AD2) first announces the route to an internal network, it may omit the QoS_INFO attribute if this network is directly connected to the announced NEXT_HOP. On receiving the UPDATE, the edge router at transit domain TD2 creates (or updates, if already present) the QoS_INFO attribute with metrics of the outgoing link, for route selection purposes (this step is omitted in the figure). If the route is selected, it is propagated to all peering domains; the QoS_INFO attribute sent to the different upstream domains is different, since the metrics are updated with respect to the virtual trunk aggregates. The same process is repeated at transit domain TD1. Notice that in the UPDATE sent from TD1 to AD1, the delay metric (17ms) is the sum of the delays of the concatenated virtual trunks (2ms and 15ms), the reserved bandwidth is the minimum along the path (300Mbps) and the congestion alarm the maximum (1). The virtual trunk values that contribute to the final values received by AD1 for this route are underlined in the figure. set Traffic to dest = Local traffic to dest + Transit traffic to dest for both routes if Alarmrcv = 1 and route in use then set Alarmeff = 0 else set Alarmeff = Alarmrcv if both routes have Assigned BW < traffic to dest, choose the one with larger Assigned BW else if one route has Assigned BW < traffic to dest, choose the other one else if Alarmeff is different, choose the route with lower Alarmeff else if Delay is different, choose the route with least Delay else if Assigned BW is different, choose the route with larger Assigned BW else use normal BGP rules (AS_PATH length, etc.)
Figure 5 - Route comparison/selection function
Figure 5 shows the algorithm for route comparison used in the decision processes in pseudo-code. Delay information is used to select the fastest/shortest route. The benefits of doing this, as previously mentioned, are twofold: packets will suffer lower delays and network resource usage will be more rational. The information on the reserved bandwidth is used to eliminate, from the set of possible choices, routes with insufficient bandwidth to support the current outgoing traffic aggregate from the local AS to the destination (including flows generated at the local AS and flows traversing it); it is also used as tie breaker when two routes for the same destination have the same announced delay. The alarm levels are used to eliminate congested routes from the set of possible choices. The elimination of routes with insufficient capacity from the set of possible choices prevents congestion of those routes to a certain degree, contributing to lower message and processing overhead and to route stability. 4.1.3
Route aggregation
A very important aspect in inter-domain routing is the possibility of aggregating routes. Without the deployment of route aggregation and Classless Inter-Domain Routing (CIDR) [16] in the 1990s, the routers would not have been able to support the increasing number of advertised routes. Paradoxically, little attention is given to aggregation in inter-domain QoS routing proposals, in general. The use of a metric so coarse-grained as the congestion alarm in this proposal is aggregation-friendly. While the introduction of new metrics reduces the possibilities of aggregation compared to the standard, nonQoS-aware BGP, congestion alarm values will almost always be either 0 or 1, meaning that much aggregation is still possible. This is particularly true if congestion is introduced in transit domains, since it is 11
common to all routes sharing the congested virtual trunk. The light load expected delay metric may easily be made compatible with aggregation if assumed to be an indicator rather than an exact value: in this case, two routes may be aggregated if the smaller delay is more than a certain fraction (say 75%) of the larger one, announcing the larger value for the aggregated route. The hierarchical structure of the Internet allows for a large degree of aggregation with this approach. The bandwidth metric, however, is more difficult to deal with, even with a hierarchical structure. We are currently working on how to conciliate the bandwidth metric with high levels of aggregation. We are also evaluating the use of the congestion metric alone and of a combination of congestion and light load delay metrics without the bandwidth metric. An evaluation of the aggregation possibilities is out of the scope of this report, though.
5
Conclusions
This report addressed the problem of inter-domain QoS routing. Our proposal is based on the use of virtual trunk type aggregates for the indirect transport of traffic between two different administrative domains across a third one. These virtual trunks are usually (though not necessarily) defined by means of SLSs between the operators of peering domains, and each inter-domain path consists on a concatenation of virtual trunks. Each virtual trunk, defined by a triplet of ASs, is shared by all active routes containing that sequence of ASs in the AS path. We formally stated the problem of SLA-aware inter-domain QoS routing and formulated it as an Integer Linear Programming (ILP) optimization problem, providing a proof that routes thus obtained do not contain cycles. As a practical solution, we proposed the QoS_INFO extension to the BGP protocol using of a combination of three different metrics (assigned bandwidth, expected light load delay and congestion alarm) in order to simultaneously achieve different and conflicting goals: finding non-congested paths that satisfy the QoS requirements of the data flows, minimize the network resources used to transport the flows, interdomain route stability and minimization of the message exchange and path computation overheads. Though one of the metrics, congestion alarm, is dynamic, its coarse granularity and the rules for its use in the path selection algorithm are such that the impact overhead in message exchange and in path re-computation is minimized, and route stability increased. There are several topics for further improvement of this work. Devising a more advanced algorithm for the assignment of alarm levels to aggregates is one of the topics. Besides aggregate usage, this algorithm would use parameters such as measured delay, packet losses, queue lengths and historical records as inputs, improving the assessment of congestion while reducing even further the number of required updates. Another topic is the introduction of route aggregation and its conciliation with accurate QoS information, in order to improve scalability. Finally, we want to simulate the proposed QoS routing mechanism in order to evaluate it and compare it to standard BGP without QoS support, to BGP with the QoS_NLRI extension and to the optimal results provided by the ILP formulation. Simulations will be performed first with a single, representative topology and then with a large number of randomly generated topologies, in order to better ascertain its behavior in the real word.
6
Acknowledgement
This work is based on results of the IST FP6 Integrated Project DAIDALOS [5], funded by the European Community's Sixth Framework Programme. It reflects the authors' views, and the EC is not liable for any use that may be made of this information.
7 [1] [2] [3] [4] [5] [6] [7] [8]
References S. Blake (ed.) et al., “An Architecture for Differentiated Services.” IETF RFC 2475, December 1998. E. Rosen, A. Wiswanathan and R. Callon, “Multiprotocol Label Switching Architecture.” IETF RFC 3031, January 2001. R. Guerin, A. Orda and D. Williams, “QoS Routing Mechanisms and OSPF Extensions.” In Proceedings of IEEE Globecom’97, November 1997. E. Crawley, R. Nair, B. Rajagopalan and H. Sandick, “A Framework for QoS-based Routing in the Internet.” IETF RFC 2386, August 1998. DAIDALOS project (IST-2002-506997) homepage, www.ist-daidalos.org Y. Rekhter and T. Li (eds.), “A Border Gateway Protocol 4 (BGP-4).” IETF RFC 1771, March 1995. M. Winter (ed.) et al., “Final System Specification.” AQUILA Consortium Deliverable D1203 (v. b1), February 2003. P. Pan, E. Hahne, H. Schulzrinne: "BGRP: Sink-Tree-Based Aggregation for Inter-Domain Reservations." In Journal of Communications and Networks, Vol. 2, No. 2, June 2000, pp. 157-167.
12
[9] [10] [11] [12] [13] [14] [15] [16]
T. Damilatis (ed.) et al., “D3.4: Final System Evaluation. Part B: D1.4: Final Architecture, Protocol and Algorithm Specification.” TEQUILA Consortium Deliverable, October 2002. P. Flegkas (ed.) et al., “Specification of Business Models and a Functional Architecture for Inter-domain QoS Delivery.” MESCAL Consortium Deliverable D1.1, June 2003. P. Chimento et al., “QBone Signalling Design Team – Final Report.” Internet2 QoS Working Group, July 2002. http://qos.internet2.edu/wg/documents-informational/20020709-chimento-etal-qbone-signaling/, L. Xiao, J. Wang, K. Lui and K. Nahrstedt, “Advertising Inter-Domain QoS Routing Information.” In IEEE Journal on Selected Areas in Communications, Vol. 22, No. 10, December 2004. G. Cristallo and C. Jacquenet, “The BGP QOS_NLRI Attribute.” IETF Internet Draft, February 2004. R. M. Karp, “Reducibility among combinatorial problems.” In Complexity of Computer Computations, pages 85-103. Plenum Press, 1972. K. Papagiannaki et al., “Measurement and analysis of single-hop delay on an ip backbone network.” IEEE JSAC Special Issue on Internet and WWW Measurement, Mapping, and Modeling, Vol. 21, No. 6, August 2003. V. Fuller et al., “Classless Inter-Domain Routing (CIDR): an Address Assignment and Aggregation Strategy.” IETF RFC 1519, September 1993.
13
