Load Distribution with Queuing Delay Bound over

AIAA 2008-5456

26th International Communications Satellite Systems Conference (ICSSC) 10 - 12 June 2008, San Diego, CA

Load Distribution with Queuing Delay Bound over Multipath Networks: Rate Control using Stochastic Delay Prediction Sumet Prabhavat*, Hiroki Nishiyama*, Yoshiaki Nemoto†, Nei Kato†, Graduate School of Information Sciences, Tohoku University, Sendai, JAPAN Nirwan Ansari‡ Advanced Networking Laboratory, Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ, USA Network connections established via multiple paths are becoming more eminent; efficient utilization of network resources provided by multiple interfaces, e.g., network gateways equipped with International Mobile Telecommunications–Advanced (IMT-Advanced) technologies or enabled with multipath routing, is necessary for next generation networks. In reality, both wired and wireless links are sometimes over-provisioned in which some are utilized while others are reserved for backup. One of the key issues in our study is to investigate how these interfaces can be used simultaneously for transmitting real time traffic with a short delay. Optimally distributing network traffic over multiple paths connected between source and destination gateways is the main purpose of this research. Our proposed model takes into account of both propagation delay and bandwidth for determining the optimal path. If there are paths with different propagation delays between a source and a destination, the path with a larger propagation delay will likely be assigned with a lower precedence. For input traffic, the splitting ratios are optimally adjusted in order to minimize the maximum path delay. We then show that the delay controlled load distribution model can optimally utilize the path with different path parameters in minimizing the maximum path delay.

I. Introduction

A

majority of the future network applications will require multimedia broadband services. Meanwhile, network infrastructure technologies, e.g., International Mobile Telecommunications–Advanced (IMTAdvanced) Technology or multipath routing protocols, are being introduced to provision multipath communications. Devices with multiple interfaces are widely deployed. The number and type of interfaces in facilitating path selection are important factors in distributing load efficiently. However, distributing load among multiple paths in a heterogeneous network to minimize path delay is a challenging research problem which has not been studied much. In reality, propagation delay in a wireline network is significantly

Low/High Bandwidth Large Delay Channel

LAN or End-User Network

LAN or End-User Network

High Bandwidth Small/Large Delay Channel High speed link

Gateway enabled traffic controller

Low Bandwidth Small/Large Delay Channel

Gateway enabled traffic controller

Low-priced link

Figure 1. Multipath configuration.

*

Ph.D. candidate, Graduate School of Information Sciences, Tohoku University, JAPAN. Professor, Graduate School of Information Sciences, Tohoku University, JAPAN. ‡ Professor, Department of Electrical and Computer Engineering, New Jersey Institute of Technology, USA. 1 American Institute of Aeronautics and Astronautics †

092407 Copyright © 2008 by the American Institute of Aeronautics and Astronautics, Inc. All rights reserved.

different from that of a wireless network. Therefore, we focus on how to distribute traffic and control the path delay. Figure 1 depicts the multiple paths which can be provisioned by the multiple-interface gateways. The multipath established between the gateways can be wired links or wireless as well as satellite links which are channels with large delay. These links can also be high or low bandwidth channels. We propose a model to efficiently utilize these different channels by optimally distributing traffic over these paths. In previous works, load distribution models have been developed to capitalize on redundant channels. The problem of load distribution is originated from the problem of distributing load among servers such that a certain objective function is optimized, e.g., jobs are completed in the least amount of time like the generic job shop scheduling1-3 and the multiprocessor scheduling4. Input load is distributed by a central job scheduler according to its control scheme and objective, i.e., throughput guarantee or delay control1, 2. However, those models are based on a local network system, and so the latency experienced by a job traveling from the source to a server has not been considered. Therefore, some modifications are needed in order to appropriately apply load distribution over a multipath where at least one larger delay path exists. After an overview of multipath and related works in IP networks in Section II, our proposed model, i.e., Delay-Controlled Load Distribution (DCLD), will be described in Section III, its performance is substantiated by the numerical results in Section IV, and concluding remarks will be presented in Section V. Note that since traffic splitting ratios are defined as normalized path weights, which will be detailed in Section III, the term “splitting ratios” will be used instead of the “path weights”.

II. Traffic Engineering in IP Networks: Load Distribution over Multipath The objective of traffic engineering is to distribute traffic such that performance based on some criteria is optimized5. Optimal traffic distribution depends on both routing protocol and forwarding mechanisms adopted in the network. In IP networks, Routing Information Protocol (RIP)6 and Open Shortest Path First (OSPF)7 have been the most widely deployed standard routing protocols, which use the shortest path routing where the path cost is derived from either hop count or bandwidth of the path. Only paths with the smallest cost are used for data forwarding while the other paths are for backups. When the routing protocol generates multiple equal cost paths or next-hops for a given destination routing prefix, the forwarding mechanism equally splits the corresponding traffic across them to balance the load. The selection is based either on packet header information or on the state of simple round-robin scheme cycling through the possible next-hops. In Ref. 8 and Ref. 9, the performance of the equal cost load balancing approach and the implementation issues on traffic splitting in the context of next-hop forwarding are studied. The effectiveness of these approaches relies on how many equal cost shortest paths exist between each source and destination pair. However, the use of both shortest path routing and equal splitting across equal cost paths make it difficult or even impossible to achieve optimal traffic engineering. To enhance the performance of OSPF, some related works have been proposed. For example, Ref. 10 and Ref. 11 study how path costs can be optimally determined, while Ref. 12 studies how traffic can be optimally distributed over the equal cost multipath. On the other hand, Enhanced Interior Gateway Routing Protocol (EIGRP)13, which is Cisco’s proprietary routing protocol, introduces additional parameters into the routing process, and unequal load balancing into the forwarding process. Ref. 10 has proposed a way to select proper path weights. While the link weight optimization problem under equal load sharing is an NP-hard problem, Ref. 10 introduces a local search heuristic algorithm. This scheme achieves the performance quite close to the optimal routing in some specific situations. On the other hand, Ref. 11 tries to solve a similar problem by using linear programming. The result presented in Ref. 11 assumes forwarding decisions that are specific to each ingress-egress pair and the ability to split traffic in an arbitrary ratio over different shortest paths. These assumptions conflict with current IP forwarding mechanisms. Compatibility with destination based forwarding can be achieved through a simple extension of the result, by taking advantage of a property of shortest paths14 or readjusting the program formulation itself15. OSPF Optimized Multipath (OSPF-OMP)12 takes into account not only bandwidth but also link utilization as well as packet drops in its forwarding mechanism. By using path determination based on this information, OSPFOMP unequally splits traffic and can achieve better load distribution. However, the performance improvement by the previously described approaches trying to enhance OSPF’s distribution mechanism is limited because they distribute the traffic only over equal cost paths as the original OSPF does. EIGRP is different from OSPF in both path cost calculation and traffic splitting control. EIGRP allows more than one metric to be used for determining the path cost while OSPF uses only one metric which is inversely proportional to bandwidth. In general, EIGRP uses both bandwidth and delay as metrics while it can support five 2 American Institute of Aeronautics and Astronautics 092407

different metric types. These metrics prevent the routing protocol from choosing paths with a large delay and/or small bottleneck bandwidth. Furthermore, EIGRP forwarding mechanism supports load balancing over equal and unequal cost multipath. However, existing routing and load distribution methods do not consider delay minimization, and optimal traffic distribution for minimizing delay has not been emphasized. Although EIGRP uses the delay metric for computing path costs, the impact of growing queuing delay due to network congestion has not been discussed. In this paper, we focus on how to select optimal path among potential end-to-end paths in minimizing the maximum delay between source and destination. Note that possible end-to-end routes are supposed to be determined by a shortest path algorithm or Multiprotocol Label Switching (MPLS)16. Setting up of such paths is beyond the scope of this work. In our model, a network is expressed as a simple queuing model, and end-to-end delay, including queuing delay, can be estimated by applying the Poisson process. The proposed method, DCLD, employing an optimization technique gradually controls load splitting ratios, and so traffic is distributed among available paths so as to minimize the maximum delay.

III. Delay-Controlled Load Distribution (DCLD) A. Network Topology and Path Setup Suppose that a multipath has been established between two gateways, i.e., nodes S and T, as depicted in Fig. 2. The gateways are the source and destination nodes at edges of the core network. The other nodes in the core network are forwarding nodes. The path setup depends on the implemented routing protocol. There is no shared link between paths. In MPLS networks, each path is a Label Switched Path (LSP) having been set up according to Ref. 16 where source and destination nodes are ingress and egress nodes, respectively. In IP networks, each path is a series of consecutive single hops. Each hop is a selected route from itself to another adjacent router according to the routing protocols, e.g., RIP, OSPF, and EIGRP. The source and destination nodes are a pair of gateway routers. Thus, both a pair of sourcedestination nodes and a pair of ingress-egress nodes are herein equivalent to a pair of source and destination gateways. B. Load Distribution Problem Network traffic is a flow of packets that originated from nodes outside the core network and is enrouted via forwarding nodes to a destination gateway. Input traffic is inbound network traffic aggregated at a source gateway. Network load is the total demanded bandwidth of the input traffic. The network load is split and then distributed proportionally according to the splitting ratios. Our model determines the optimal splitting ratios so as to minimize the maximum path delay. In Figure 2, P denotes a set of multiple paths connected between the source gateway S, and the destination gateway

p1 p1

p1

p1 p2

p2

p2 T

S

pN

pN

pN

pN

Figure 2. Example of multipath network model (N paths) established in a core network.

p1, B1, D1 p2, B2, D2 Gateway

p3, B3, D3

Gateway

pN, BN, DN

Figure 3. Logical link model. (Path pn is defined with bandwidth Bn and propagation delay Dn.)

3 American Institute of Aeronautics and Astronautics 092407

T. If

| Ρ | = N, then Ρ = { p1 , p2 ,..., p N } . From a user’s viewpoint, the path p ∈ Ρ connected between the source

and destination gateways can be considered a logical link between the gateways as illustrated in Fig. 3. The network load will be distributed and assigned to each path according to the optimal solution of the load assignment problem. Let bp,l represent bandwidth and dp,l represent propagation delay of link l ∈ p . Bandwidth and delay along the path can be herein approximated by the following: B p = min (bp ,l ) , l∈ p

D p = ∑ d p ,l l∈ p

where Bp is the bandwidth and D p is the propagation delay of path p. Path delay of path p ∈ Ρ can be expressed as a summation of propagation delay, D p , packet forwarding time,

D pf , and queuing delay, D pq , as shown in Eq. (1). f

q

(Path Delay) = D p + D p + D p The second and third terms in Eq. (1) are variables depending on the bandwidth and queuing policy in path p while D p is a constant depending on the physical properties and the length of the path, p. Assuming that packet arrival follows the Poisson process with exponential service time and each queue is equipped an infinite buffer and adopts a First Come First Serve (FCFS) discipline, the sum of

(1)

λu1 λ

Traffic Allocator

λu2

λuN

D pf and D pq can be derived from the M/M/1

B1 B2

BN

Figure 4. Queuing model in load distribution model.

queuing analysis and is equal to

1 , B p − λu p where

λ

is the mean input traffic rate and

u p is the splitting ratio for path p. By this assumption, the logical link

model in Fig. 3 is transformed to a queuing model as described in Fig. 4, and the corresponding cost function is formulated as in Eq. (2).

C p (u p ) = D p +

1 B p − λu p

(2)

The optimal splitting ratios can be derived by solving the optimization problem described in Eq. (3). Minimize

max C p (u p ) ,

subject to constraints:

∑u

p∈Ρ

p∈Ρ

and

p

=1

0 ≤ up ≤

Bp

λ

4 American Institute of Aeronautics and Astronautics 092407

(3)

C. Adaptive Load Distribution Algorithm The splitting ratios, uk = {upk} for all p ∈ Ρ , are the control variables of the problem described in Eq. (3) and the proportion of traffic allocated to path p at time tk. The initial splitting ratios, u0, are calculated from Eq. (4).

Bp

∀p ∈ Ρ : u 0p =

∑B p∈Ρ

. p

(4) When the kth packet arrives at time tk, the input traffic rate at that instant in time can be expressed by

λ (t k ) = λ + δ k , where

δk

is the traffic fluctuation around the mean input traffic rate

unknown constant ignoring traffic fluctuation such that

λ . Assuming that the traffic demand is an

λ (t k ) ≈ λ , the objective of the dynamic load distribution

problem is to choose, after each measurement duration (tk-1 , tk), the ratios uk in such a way that they converge to the unknown optimal values u*. The value of λ can be easily estimated from inter-arrival times of packets. Actually, the inverse of the average value of all inter-arrival times observed so far is used as λ in numerical analysis in Section IV. After λ is obtained, the gateway router will perform the following steps: 1)

Calculate C p (u p ) by using Eq. (2) for each path p ∈ Ρ .

2)

Select p worst ∈ Ρ which has a maximum cost among all paths.

3)

Select pbest ∈ Ρ which has a minimum cost among all paths.

4)

Calculate Δu such that C pworst u pworst − Δu = C pbest u pbest + Δu by using Eq. (5).

(

)

(

)

( S pbest − S pworst ) ⎧ ⎪ 2λ ⎪⎪ 2 Δu = ⎨ ⎛ 2 ⎞ 2 2 ⎪ ( S pbest − S pworst ) + ΔD − υ ( S pbest + S pworst ) + ⎜⎜ ΔD ⎟⎟ p p ⎠ ⎝ ⎪ ⎪⎩ 2λ ΔD p where S p = B p − λu p , ΔD p = D pbest − D pworst , and υ = ΔD p

; ΔD p = 0

; ΔD p ≠ 0

(5) 5)

Determine the appropriate Δu by Δu ← min(u pworst , Δu ) .

6)

Update u pworst = u pworst − Δu and u pbest = u pbest + Δu .

7)

For all paths p ∈ Ρ except pbest and p worst , u p = u p .

k

k −1

k

k −1

k

k −1

For an arrival, upk is gradually adjusted based on the measured input traffic and predicted delay. Note that these splitting ratios determine the proportions of the distributed traffic, not the absolute amounts. In addition to splitting the traffic at the packet level, traffic splitting may be done at the flow level. If traffic splitting has been done at the packet level, packet reordering may be required; whereas splitting traffic at the flow level avoids this problem. However, poor and unpredictable granularity may occur. 5 American Institute of Aeronautics and Astronautics 092407

IV. Numerical Results We evaluate DCLD in a high-load condition where network utilization is 90% ( ρ = λ / ∑ B p = 0.9 ). The p∈Ρ

network model is illustrated in Fig. 3 with N=5, and so we have Ρ = { p1 , p2 , p3 , p4 , p5 } . The performance of DCLD is studied through numerical analysis. The splitting ratios and the maximum path delay are represented by graphs. We demonstrate the DCLD performance in the case that all paths have unequal bandwidth and/or unequal propagation delay. A. Unequal Bandwidth We consider how network load will be divided and how much the maximum path delay will be, if there are paths with different bandwidths. We assume that, in the multiple paths Ρ , bandwidths of path p1, p2, p3, p4, and p5 are given to be 10, 15, 20, 25, and 30, respectively; and propagation delay is negligible. In this case, the splitting ratios are adjusted based on path bandwidths in order to minimize the maximum path delay. Fig. 5(a) presents the splitting ratios. Traffic load is allocated to paths with large bandwidths than those with small ones. At the initial state (before arrival of the first packet), the splitting ratios are similar to the ratios of the existing unequal cost multipath load balancing. When packets arrive, the proposed algorithm dynamically updates the ratios in order to minimize the maximum path delay as illustrated in Fig. 5(b). Therefore, the higher the bandwidth, the higher the path precedence.

0.4 Splitting ratio up

u1

u3

u4

u5

0.3 0.2 0.1 0 0

Maximum path delay max Cp(up);p∈P

u2

1

2 # Packet arrival (a)

3

4

1

2 # Packet arrival (b)

3

4

1 0.8 0.6 0.4 0.2 0 0

(Bandwidth B p1

Figure 5. Unequal bandwidth multipath. = 10, B p2 = 15, B p3 = 20, B p4 = 25, B p5 = 30 ; Delay D p =

6 American Institute of Aeronautics and Astronautics 092407

0 for ∀p ∈ Ρ )

B. Unequal Propagation Delay In this evaluation, suppose that there are paths with different propagation delays but bandwidths are equal. Propagation delays of path p1, p2, p3, p4, and p5 are given to be 1, 1.5, 2, 2.5, and 3, respectively; and bandwidths of all paths are equal to 20. Figure 6 shows the results when propagation delays are different. At the initial state, all ratios are equal to 0.2. When a packet arrives, the splitting ratios are changed, even if bandwidths of all paths are not different. The results prove that propagation delay is also a parameter that plays an important role in the precedence of path utilization. If path propagation delays are different, the splitting ratios are adjusted so as to balance the delay time of the paths. Therefore, paths with larger delay will be utilized less than smaller delay paths.

Splitting ratio up

0.22 0.2 u1

0.18

u2

u3

u4

u5

0.16

Maximum path delay max Cp(up);p∈P

0

5

10

15

20 # Packet arrival (a)

25

30

35

40

5

10

15

20 # Packet arrival (b)

25

30

35

40

3.6 3.5 3.4 3.3 3.2 3.1 3 0

Figure 6. Unequal propagation delay multipath. (Bandwidth B p = 0 for ∀p ∈ Ρ ; Delay D p1 = 1, D p2 = 1.5, D p3 = 2, D p4 =

2.5, D p5 = 3 )

C. Unequal Bandwidth and Unequal Propagation Delay Now, suppose that bandwidths of path p1, p2, p3, p4, and p5 are given to be 10, 15, 20, 25, and 30, respectively. In addition to having different bandwidths, the propagation delays of these paths are also unequal, i.e., 1, 2, 3, 4, and 5, respectively. Figure 7 shows the results when both bandwidths and propagation delays are different among all paths and proves that both bandwidth and propagation delay parameters affect the adjustment of splitting ratios. Our results show that the model achieves load distribution and minimization of the maximum path delay. Moreover, regarding the convergence time of the optimal approach, the maximum path delay is minimized within a few arrivals. 7 American Institute of Aeronautics and Astronautics 092407

Splitting ratio up

u1

0.3

u2

u3

u4

u5

0.25 0.2 0.15 0.1

Maximum path delay max Cp(up);p∈P

0.05 0

5

10

15

20 # Packet arrival (a)

25

30

35

40

5

10

15

20 # Packet arrival (b)

25

30

35

40

5.5 5.4 5.3 5.2 5.1 5 0

Figure 7. Unequal bandwidth and propagation delay multipath. (Bandwidth B p1 = 10, B p2 = 15, B p3 = 20, B p4 = 25, B p5 = 30 ; Delay D p1 = 1, D p2 =

2, D p3 = 3, D p4 = 4, D p5 = 5 )

V. Conclusion We have proposed a multipath load distribution model named DCLD. The objective of DCLD is to optimally utilize the multipath with different path parameters, i.e., bandwidth and propagation delay, in minimizing the maximum path delay. DCLD has been derived from the load assignment problem formulated as the minimization of a non-linear objective function. The objective function is defined as a function of the path delay including propagation and queuing delays as well as queue service time. Since DCLD updates the splitting ratios according to the proposed algorithm iteratively for each arrival packet, the packet will likely be forwarded via a path with better conditions, i.e., higher bandwidth and/or smaller delay. Moreover, by updating the splitting ratios of the maximum and minimum delay paths only, network disturbance is suppressed while the optimal point will be converged rapidly. In reality, the multipath established between the gateways can be wired links or wireless as well as satellite links which represent channels with large delay. These links can also be high or low bandwidth channels. Based on the numerical results which show that the maximum path delays have been minimized, it has been proven that DCLD can efficiently utilize different channels by optimally distributing traffic over multiple paths.

8 American Institute of Aeronautics and Astronautics 092407

References 1

F. Bonomi, B. Doshi, J. S. Kaufman, T.P. Lee, and A. Kumar, “A case study of an adaptive load balancing algorithm,” Queueing Syst. Theory Appl. 7, 1 (Nov. 1990), pp. 23-49. 2 F. Bonomi and A. Kumar, “Adaptive optimal load balancing in a heterogeneous multiserver system with a central job scheduler,” IEEE Transactions on Computers, Vol. 39, No.10, Oct. 1990, pp. 1232-1250. 3 K. P. Bubendorfer, “Resource Based Policies for Load Distribution,” Master's Thesis of Victoria University of Wellington, Aug. 1996. 4 E.S.H. Hou, N. Ansari, and H. Ren, " A Genetic Algorithm for Multiprocessor Scheduling," IEEE Trans. on Parallel and Distributed Systems, vol. 5, no. 2, pp. 113-120, Feb. 1994. 5 D. Awduche, A. Chiu, A. Elwalid, I. Widjaja, and X. Xiao. “Overview and Principles of Internet Traffic Engineering,” RFC 3272, May 2002. 6 G. Malkin, “RIP Version 2,” RFC 2453, Nov. 1998. 7 J. Moy, “OSPF Version 2,” RFC 2328, Apr. 1998. 8 D. Thaler and C. Hopps, “Multipath Issues in Unicast and Multicast Next-Hop Selection,” RFC 2991, Nov. 2000. 9 C. Hopps, “Analysis of an Equal-Cost Multi-Path Algorithm,” RFC 2992, Nov. 2000. 10 B. Fortz and M. Thorup, “Internet Traffic Engineering by Optimizing OSPF weights,” Proceedings of IEEE INFOCOM 2000, Vol. 2, Mar. 2000, pp. 519–528. 11 Y. Wang, Z. Wang, and L. Zhang, “Internet Traffic engineering without full mesh overlaying,” Proceedings of IEEE INFOCOM 2001, Vol.1, Apr. 2001, pp. 565 – 571. 12 C. Villamizar, “OSPF optimized multipath (OSPF-OMP),” Internet draft , Feb. 1999. 13 Cisco Systems Inc., “Enhanced Interior Gateway Routing Protocol (EIGRP),” Cisco White Paper EIGRP, http://www.cisco.com/warp/public/103/eigrp-toc.html. 14 A. Sridharan, R. Guerin and C. Diot, “Achieving Near-Optimal Traffic Engineering Solutions for Current OSPF/IS-IS Networks,” IEEE/ACM Transactions on Networking, Vol. 13, No. 2, Apr. 2005, pp. 234-247. 15 H. Abrahamsson, B. Ahlgren, J. Alonso, A. Andersson, and P. Kreuger, "A multi-path routing algorithm for IP networks based on flow optimization," In International Workshop on Quality of Future Internet Services (QofIS'02), Zurich, Switzerland, Oct. 2002. 16 E. Rosen and A. Viswanathan, “Multiprotocol Label Switching Architecture,” RFC 3031, Jan. 2001.

9 American Institute of Aeronautics and Astronautics 092407