Traffic Characteristics-based Flow Assignment ...

Traffic Characteristics-based Flow Assignment Method for Reducing Queuing Delay Yoshinor Kitatsuji∗ , Satoshi Katsuno† , Masato Tsuru‡ , Tetsuya Takine§ and Yuji Oie‡ ∗ National

Institute of Information and Communications Technology, Fukuoka Japan Email: kitaji[at]kyushu.jgn2.jp † KDDI R&D Laboratories, Inc., Saitama Japan Email: katsuno[at]kddilabs.jp ‡ Kyushu Institute of Technology, Fukuoka Japan Email: {tsuru[at]ndrc, oie[at]cse}.kyutech.ac.jp § Osaka University, Osaka Japan Email: takine[at]comm.osaka-u.ac.jp

Abstract— In traffic engineering, for effectively distributing traffic flows over multiple network paths, it is vital to take traffic characteristics of individual flows into account in appropriately assigning the flows to the network paths to achieve better delay performance as a whole. We have developed a traffic characteristic-based flow assignment method that reduces queuing delay in a buffer at the beginning of each network path. By taking the traffic characteristics of individual flows into account the method can effectively assigns flows to the paths so as to minimize overall queuing delay. We consider the simple case in which two types of flows with distinct traffic characteristics (burstiness) are assigned to two bandwidth-guaranteed paths. The flow assignment of our method reduces the queuing delay (at the edge router) of the worse path, by attempting to optimize the min-max fairness between the paths in terms of queuing delay. Numerical simulation showed that the method assigned the flows such that the queuing delay was reduced up to 40% compared with that when conventional path utilization based flow assignment was used. This means that taking the traffic characteristics of flows into account significantly improves the queuing delay performance when the flows have distinct characteristics.

I. I NTRODUCTION Internet service providers are facing the challenge of dynamically and adaptively designing their networks to satisfy customer demands for fast, reliable, and differentiated services with the minimal cost. Internet traffic engineering (TE) [1] is a key tool for meeting this challenge because it can effectively map traffic demands onto the network topology and adaptively reconfigure the mapping as network conditions change. More specifically, TE can adaptively distribute traffic flows over networks so that network resources are efficiently allocated. The fundamental philosophy for making the Internet scalable is that an algorithmically complex processing should be pushed to the edge of the network whenever possible. When designing a backbone network that supports TE the backbone network having two levels of components – a high-speed core and edge routers surrounding the core – reflects this philosophy. Consider the case in which individual pairs of edge routers establish multiple paths to convey traffic from one edge (ingress) router to another edge (egress) router. Effective

Backbone network

Edge Router

Edge Router

Fig. 1. Simple case considered in this paper: a number of flows, each having one of two distinct sets of traffic characteristics, are assigned to one of two paths with different bandwidths.

traffic distribution can be achieved by balancing traffic flows among these paths. [2] [3]. In distributing the traffic flows, however, inadequate traffic balancing into multiple paths significantly degrades queuing delay performance, especially when the path is highly utilized [4] [5]. When different (bursty and non-bursty) types of flows are given to be balanced over multiple paths, the non-bursty flows assigned to the same path as the bursty flows encounter the same queuing delay performance as the bursty ones. In addition, when the path bandwidths are different, conventional path utilization based flow assignment, that is, assigning flows on the basis of the ratio of path bandwidths for each flow type, results in dropping the queuing delay performance of the narrower path more severely. Therefore, given a set of flows with distinct traffic characteristics and different path bandwidths, it is not trivial to determine how to distribute them so as to minimize overall queuing delay, that is, to achieve a good balance of the flows in terms of their queuing delays. In this paper, we consider a simple case: a number of flows, each having one of two distinct traffic characteristics, are assigned to one of two paths with different bandwidths (Figure 1). A queuing delay statistic, such as, the mean queuing delay, the standard deviation of the queuing delay, or the 99.5-th

percentile queuing delay for the worse path, is used as a metric representing the overall delay performance. The flows are assigned using the min-max fairness optimization of the queuing delay. We describe a flow assignment method that improves this delay metric, and compare the queuing delays obtained experimentally with those using the conventional path utilization based flow assignment. The flow are defined as a sequence of packets distinguished by five tuples (a pair of IP addresses, a protocol number, and port numbers), but we do not limit ourselves to this definition. The propose method is applicable to flows in which 1) each packet can be promptly categorized on the basis of information it contains, and 2) each flow can be categorized by traffic types on the basis of its measured or predefined traffic characteristics. The method is showed to be practical for roughly estimating the queuing delay statistics for an arbitrary assignment of two given types of flows to two given paths where traffic characteristic of each flow type and bandwidth of each path (each output link) are known. It learns the relationship between the flow assignment and the queuing delay statistics from the queuing delay statistics for a small number of previous flow assignments. Using this relationship, it estimates the queuing delay statistics for a period short enough to be used in an online manner, which is not possible with either fluid-flow model-based analysis nor packet-based simulation. Using the estimated statistics, it searches, in an online manner, for the better flow distribution over the two paths in terms of overall queuing delay performance to determine which path should be assigned to the next incoming flow. The effectiveness of flow assignment on the basis of the traffic characteristics of each flow showed through preliminary experimental evaluation demonstrates the great potential of this method. The rest of this paper is organized as follows. Section II describes our approach to the flow assignment and our objectives. Section III distinguishes our approach from previously proposed ones. Section IV defines the delay metric representing flow assignment effectiveness. In Section V, we describe our method. In Section VI, we evaluate the effectiveness of our method by comparing the delay metric from the flow assignment estimated by our method with that by the path utilization based assignment. Finally, Section VII concludes with a brief summary. II. OVERVIEW OF F LOW A SSIGNMENT A. Approach We assume that the core of a high-speed network accepts bandwidth-guaranteed paths established between pairs of edge routers. Each pair establishes two or more paths. The traffic is distributed in a flow-based manner instead of a packet-based one which degrades the performance of TCP data transmission and the stability of UDP streams due to packet reordering. The procedure of balancing flows over multiple paths at the edge router is 1) first grouping on-going flows into multiple types based on their traffic characteristics (peak rate, average rate,

traffic-varying interval [cycle of on-off-states]) in an online manner, and 2) secondly distributing the flows over multiple paths on the basis of the multiplexing character of the multiple flow types. We assume that the traffic characteristics are stationary or vary slowly as long as the flows are balanced. Our goal is to develop a flow assignment method that assigns flows grouped into m types to n paths on the basis of the traffic characteristics of each group and the number of flows in each group. The criteria for characterizing the flows in order to group them are: • TCP vs. UDP flows, • video vs. VoIP streams, where only UDP traffic is focused on by separately treating TCP and UDP traffics (e.g. by assigning different queue classes for them) so that, in some case, the packet loss is minimized for TCP flows and the delay variation is reduced for UDP flows, and • the bandwidths of the links through which traffic enters the edge router, which connects multiple types of access links, including those aggregating DSL and/or FTTH customers (1-Gbit/s class) and enterprise customers (10Gbit/s class). Using these criteria, we can group flows into two types (bursty and non-bursty) that have stationary traffic characteristics. Therefore, we initially develop a method for assigning two types of flows. In addition, to avoid unnecessary complexity, we should limit the number of paths between each pair of edge routers to two or three. For simplicity, we consider two-path case for flow assignment. This method is a first step in developing a method that can easily and efficiently optimize the assignment of actual traffic flows. While a fluid-flow model [6], or packet-based simulation would enable accurate estimation of queuing delay for each path, their computation takes too long for online use. For real-time application, we need a method that can quickly estimate the queuing delay for each path so that the flows can be assigned so as to minimize the overall queuing delay. The estimated flow assignment is used to compute a flow assignment probability which is used for assigning a flow to a path as follows. 1) When a packet arrives, the router checks the routing table and identifies a flow from the incoming packet. 2) The path mapped to the identified flow is retrieved, if the flow had already been active and the mapped path was registered. Otherwise, a path is selected using the flow assignment probability p(l,k) (where l and k denote a path [l = 1, 2] and a flow type [k = 1, 2], respectively, and p(1,k) + p(2,k) = 1), and mapped to the flow. The mapping information is registered for the subsequent packets in the identified flow. The flow assignment probability is periodically updated in order to adaptively reduce the gap between the actual and estimated number of flows in the paths because the number of actual flows varies over time. The gap is dominated by the

dynamics of the flow arrival process, and leads to the actual queuing delay differing from that expected from the estimated flow assignment. There is a trade-off between updating the flow assignment probability more often in order to reduce the gap and conserving the computational resources needed to compute the flow assignment probability. Future improvements to this method include calculating the flow assignment probability on the basis of the flow arrival changes and on the the estimated flow assignment. B. This work In our previous work, we analyzed the relationship between the combination of traffic characteristics (burstiness) and the bandwidths of two paths in terms of the queuing delay [6]. We analyzed the effect of using the mean queuing delay , the standard deviation of the queuing delay, and the 99.5th percentile queuing delay when flows are multiplexed into a single path using the fluid-flow model, which assumes an infinite buffer and admits on-off-state fluid-flows [7] [8] [9]. Using these analytical results we examined ways of assigning a set of flows to achieve min-max optimization, that is, ways to minimize the larger queuing delays, as described in Section IV-A. There were three significant findings. • Assigning the same number of flows to each path can minimizes the overall queuing delay if the two paths have the same bandwidth. • If the two path have different bandwidths, there are several assignments with a lower overall queuing delay than that of the path utilization based assignment. • The low-delay flow assignments obtained using one of the queuing delay statistics, (the mean, the standard deviation, or the 99.5-th percentile) are very similar to those obtained using the other queuing delay statistics. In the work reported here, we focused on the case of different path bandwidths and developed a method for finding a flow assignment that reduces the queuing delay more than does the path utilization-based flow assignment. We term path utilization based flow assignment ’naive flow assignment’ hereafter. The fluid-flow model we used to evaluate the proposed method uses only the first moment in the traffic arrival process, such as, the peak rate of the on-off-state flows and the average duration of on-off-state. Using the second moment as well as the first one better emulates the traffic arrival process and better demonstrates greater effectiveness of the proposed method when applied to the actual traffic. However, the nature of the delay contour lines used to estimate the queuing delay for a path when assigning flows, described in Section V-A, should be the same as that when using only the first moment. Therefore, we should be able to demonstrate the effectiveness of the proposed method is confirmed even if the model uses only the first moment in the traffic arrival process.

an MPLS available network in an online manner. E.g., the proposed methods in the literature [10] [11] search for a route that minimizes the forthcoming tunnel to be established under the assumption that the bandwidth required for the tunnel to aggregate the arriving flows can be predicted in an a priori manner. These method are applicable to traffic flows varying over a longer time scale (tens of minutes to several hours). For a shorter period (a few seconds to a few minutes), they often over or under allocate bandwidth. Our approach is applicable to tunnel links because it adaptively assigns flows which vary over a short period. Goplan, Chiueh, and Lin described an improved admission control algorithm for tunnel bandwidth allocation that uses the probability of violating the delay tolerance with deterministic delay bounds [12]. Acceptance of a (small) violation probability of delay tolerance enables the algorithm to improve the utilization of the base (physical) links admitting the tunnel links. However, the algorithm has a scalability problem when used for admitting flows with various traffic characteristics due to its assumption that the traffic assigned to each tunnel has homogeneous characteristics. In contrast, the principal challenge we face is assigning various types of flows to the same path. Previously proposed methods for balancing traffic over multiple paths between a pair of edge routers [2] [3] are based on the assumption that reducing path utilization reduces the queuing delay. However, as we showed in our previous work [6], flow can be assigned in such a way that low queuing delay can be retained even if the utilization increases. The principal feature distinguishing our work from previous work is the use of the multiplexing characteristic of distinct types of traffic to minimize the overall queuing delay among multiple paths. IV. D ELAY M ETRIC A. Metric Representing Delay Performance for Two Paths

III. R ELATED W ORK

In this work, several queuing delay statistics, including the mean queuing delay, the standard deviation of queuing delay, and the 99.5-th percentile queuing delay, are used to define a delay metric for the overall delay performance among paths. When two types of flows are assigned to two paths with different bandwidths, even if the ratio of the total rates of the assigned flows are equalized to that of path bandwidths, the queuing delay statistics for the two paths differ depending on the traffic characteristics, the number of flows, and the path bandwidths [6]. There are various definitions of the delay metric used to represent the effectiveness of the flow assignment. The definition differs with the operational situation and the service level specification (SLS). Our goal is to find the flow assignment approximately maximizing (or minimizing in some cases) the representative delay metric. One operational requirement is to reducing the largest queuing delay among multiple paths. A flow assignment maximizing the following delay metric meets this requirement,

Previously proposed admission control algorithms for routing tunnel paths are routed between two edge routers in

Mmin = min (−dl (f(1,l) , f(2,l) )), l=1,2

(1)

0

0

-5e-05

-0.02

-0.0001 - d2

- d2

-0.04 -0.00015

-0.06 -0.0002 -0.08

-0.00025

best assignment for min metric naive flow assignment y=x

-0.0003 -0.0003 -0.00025 -0.0002 -0.00015 -0.0001 - d1

-5e-05

0

where dl denotes one of the queuing delay statistics for path l (l = 1, 2) to which f(k,l) flows of type k (k = 1, 2) is assigned. Maximizing this metric saves users from the worst delay deterioration. Though simple, this strategy is an important strategy in network operations. The suitability of a metric definition may depend on the SLS. For example, for those applications where packets arriving beyond the delay threshold (D) are useless (lost packets), maximizing the following metric saves flows from severe queuing delay deterioration for a unit of a path, 2


  wl ln D − dl (f(1,l) , f(2,l) ) .

-0.08

-0.06

-0.04

-0.02

0

- d1

Fig. 2. Mean of queuing delay for possible flow assignments to two paths with bandwidths of 4 and 8 Mbit/s and the total utilization of 0.6.

Mlog =

-0.1 -0.1

best assignment for min metric naive flow assignment y=x

(2)

l=1

The logarithm makes this metric sensitive to large queuing delay (near the threshold). Although the proposed method was found to be effective for both metrics, we show the results only for the first one (termed the ’min metric’) due to space limitations. B. Relationship with Best Flow Assignment In the fluid simulation paradigm, traffic is modeled in terms of a (time) discrete or continuous rate-based model rather than discrete packet instances. A fluid simulator technique tracks the fluid rate changes at traffic sources and network nodes. An on-off-state fluid-flow is one in which the source is either in a “bursty state,” transmitting at its peak rate, or in an “idle state,” transmitting at a zero bit rate. Such a source model has the advantage of being both concise and flexible because it can be used to represent connections with bit streams ranging from bursty to continuous [8]. Figure 2 shows the mean queuing delay for possible flow assignments to two paths with bandwidths of 4 and 8 Mbit/s. The traffic characteristics of the flows represent high-bit-rate (video) and low-bit-rate a(VoIP) streams in the case of the optimization of the queuing delay for UDP flows, as described in Section II-A. There are seven high bit-rate (bursty) flows

Fig. 3. The 99.5-th percentile queuing delay for possible flow assignments to two paths with bandwidths of 4 and 8 Mbit/s and the total utilization of 0.9.

with a peak rate (rate in the on-state) of 1 Mbit/s and average on and off durations of 10 and 30 ms, respectively. There are 174 low-bit-rate (non-bursty) flows with a peak rate of 64 Kbit/s and average on and off durations of 10 ms each. The on and off durations of both flows follow an exponential distribution. The average rate of the total flows corresponds to a total path capacity of 60%. The d1 and d2 are the mean queuing delays for the 4- and 8-Mbit/s paths, respectively. The min metric for a flow assignment is represented by the corresponding value on the x-axis if its plot point is above the line y = x, and by the corresponding value on the y-axis otherwise. The highest min metric is achieved using the flow assignment indicated by the square (2): 93 non-bursty flows are assigned to the 4-Mbit/s path, and the remainder are assigned to the 8-Mbit/s path. The queuing delays are 0.0000937 and 0.0000928 s, respectively. This best flow assignment almost equalizes the queuing delays between the two paths. The naive flow assignment is indicated by the triangle (
): 2 bursty and 58 non-bursty flows are assigned to the 4-Mbit/s path, and the remainder are assigned to the 8Mbit/s path. The points falling on the line x = 0 represent the queuing delay for the 4-Mbit/s path when the peak rate for all the flows assigned to the 4-Mbit/s path is less than 4 Mbit/s, meaning there is no delay. In this case, the flow assignment is highly unbalanced in terms of the queuing delay. Figure 3 shows another case – the 99.5-th percentile queuing delay. The flows are composed of 30 bursty and 105 non-bursty flows, and The total rate of all the flows is 90% of the total path capacity. The best flow assignment (2) is 3 bursty flows and 96 non-bursty flows assigned to the 4-Mbit/s path and the remainder to the 8-Mbit/s path. The queuing delays are 0.0276 and 0.0253 s, respectively. Comparing the 60 and 90% cases, we see that the properties of the best flow assignments differ completely: in the 60% case, only non-bursty flows are assigned to the 4-Mbit/s path and both types of flows are assigned to the 8-Mbit/s path; in

1

0.8 Ratio of class 1 flow (s)

the 90% case, both types of flows are assigned to both paths. This illustrates how the best flow assignment depends on the given set of flows and that it is not trivial to find the best flow assignment. Similar results were obtained when we used the standard deviation of queuing delay, different combinations of traffic characteristics, and different path bandwidths. The method described in the following section can find a flow assignment that results in better overall delay performance than the naive flow assignment when two types of flows having distinct traffic characteristics are distributed between two paths.

0.6

0.4

queuing delay: 0.00025 s queuing delay: 0.00050 s queuing delay: 0.00075 s queuing delay: 0.00100 s queuing delay: 0.00125 s

0.2

V. F LOW A SSIGNMENT M ETHOD As mentioned, our proposed flow assignment method reduces the queuing delay by taking into account the traffic characteristics of each flow types. It is targeted at the case where the path bandwidths are different as described in Section II-B. When a set of flows are given, the method takes two steps for finding the best flow assignment 1) approximately placing delay contour lines, as described in Section V-A, and 2) searching the best (highest min metric) flow assignment by iteratively estimating the queuing delay for a certain number for flows from a given set of flows are assigned to one of two paths with the delay contour lines. Our contribution is an algorithm for the first step which places the delay contour lines on the basis of the path bandwidth and traffic characteristics of two flow types. The obtained delay contour lines allow the second step to approximately estimate the queuing delay promptly. For the second step, we applied the hill-climbing algorithm to the search for the best flow assignment. In the search process, the algorithm iterates the following steps until the highest min metric is found 1) dividing flows into two paths, 2) estimating the queuing delay for each path with the delay contour lines, and 3) computing the min metric from the two queuing delays. In every instance of the iteration, the flows are divided into the different numbers. In the following subsections, we describe the queuing delay characteristics for a single path in order to indicate the nature of the delay contour lines in Section V-A, the algorithm we developed for approximately determining the placement of the delay contour lines in Section V-B, and how we applied the hill-climbing algorithm for searching for the best flow assignment in Section V-C. A. Queuing Delay Characteristics We first define class 1 and class 2 flows. The steady state probability of the on-state for a class 1 flow, denoted by p1 , is less than or equal to that for a class 2 flow, denoted by p2 (p1 ≤ p2 ) [6]. Given a set of flows to be assigned to a path, we represent the queuing delay statistics as the delay function ˜ r) parameterized by the average path utilization u (0 ≤ d(u,

0 0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Average path utilization (u)

Fig. 4. Delay contour line for 99.5-th percentile queuing delay against ratio of class-1 flows and average path utilization for a single path.

u < 1) and the average ratio of class 1 flows r (0 ≤ r ≤ 1) to total flows. Figure 4 shows the relationship between the average path utilization (x-axis) and the ratio of class 1 flows (y-axis) for a single path. The flows have the same traffic characteristics as described in Section IV-B, and the path bandwidth is 4 Mbit/s. All the points on each line have the same 99.5-th percentile queuing delay. Points on the x-axis (or r = 1), where the ratio of class 1 flows is 0 (1), represent the path occupied by only the class 2 (1) flows. We term this delay curve the ’delay contour line’ hereafter. From observations of various combinations of flow characteristics and path bandwidths, we found that: • the delay contour lines do not intersect; • the larger the utilization, the larger the queuing delay; and • there exist multiple points (flow assignments) having the same queuing delay. The figure shows that reducing either the ratio of class 1 flows or the average path utilization reduces the queuing delay when flows are assigned to a single path. The nature of the delay contour lines was the same as the the other queuing delay statistics (the mean queuing delay and the standard deviation of queuing delay) although the gradient of the delay contour lines and the space between the lines differed depending on the path bandwidth and/or the combination of traffic characteristics. For a given set of flows, the proposed search algorithm (for the second step) can easily estimate the queuing delay from the nearest delay contour lines once the lines for each path have been suitably placed using the algorithm described in the following section. B. Algorithm for Approximately Placing Delay Contour Lines In our flow assignment approach (described in Section II-A), we assume that the number of flows varies between assignment

periods. The used queuing delay statistic, such as, the mean queuing delay, is acquired every period. This means that the three-tuple data (average path utilization, ratio of class 1 flows, and queuing delay) are collected every period. our algorithm for placing the delay contour lines uses these data and the amount of data increases as the assignment period proceeds. Note that initially, the flows are assigned so as to maintain equal path utilization (the naive flow assignment). Once sufficient three-tuple data has been collected, the flows are assigned using the our method. The algorithm first identifies an appropriate area for placing the delay contour lines. Given a set of flows to be assigned to a path, the sum of the peak rate (denoted by Rk ) of all the flows must exceed the path bandwidth (denoted by cl ) so that ˜ r) > 0, there is a queuing delay d(u, R1 f(1,l) + R2 f(2,l) > cl .

(3)

If the average rate of each flow type are given by p1 R1 f(1,l) = rucl and p2 R2 f(2,l) = (r − 1)ucl , u and r hold 1 r > p2p−p ( pu2 − 1) 1 u > p1

if p1 < p2 otherwise (p1 = p2 ),

(4)

which specify where the delay contour lines should be placed. We hypothesize that the delay contour lines have the form, r = a/u + b (a > 0),

(5)

from Inequality (4). If this hypothesis holds, they can approximately placed by deriving parameters a and b if we are given two three-tuple data having identical queuing delays. The ease of placing the delay contour lines is a key feature of our proposed algorithm. Preliminary experiments using various parameters for the traffic characteristics and the various combinations of path bandwidths, showed that The delay contour lines derived by Equation (5) with two three-tuple data nicely fitted in the result of the numerical analysis with fluidflow model. We therefore use Equation (5) for approximately placing the delay contour lines. To approximately place the lines using a single three-tuple data, we simultaneously estimate the placement of several of them in an iterative manner under two restrictions. r-1 Equation (5) satisfies u < 1 and Inequalities (4) for 0 ≤ r ≤ 1. r-2 The delay contour lines do not intersect, as mentioned in Section V-A. The delay contour lines (j = 1, 2, · · · ) are numbered in the order of queuing delay, denoted by qj , where qj < qj+1 . A delay contour line j and the amount of a(j,i) being moved for the line j are defined as r = a(j,i) /u + b(j,i) , and αj , respectively. Note that bj is derived from aj and the given three-tuple data in each iteration. A pair of parameters a(j,i) and b(j,i) moves the delay contour lines independently. The algorithm works as follow. 1) The initial values are set to parameters a(j,0) and αj . p2 if p1 < p2 . An example of a(j,0) is pp21−p 1

2) In iteration i, a(j,i) is updated by increasing or decreasing a(j,i−1) by αj on the basis of the following rules. a) If delay contour line j violates r-1 a(j,i) is increased. b) if delay contour lines j and j + 1 intersect (violate r-2), a(j,i) is increased (decreased) and a(j+1,i) is decreased (increased) if intersection point is lower (higher) than the three-tuple data for each line. The lines are moved starting with the one with the largest queuing delays because the a higher delay contour line is more likely to violate r-1. αj is halved if aj continues to move back and forth. 3) The iteration is terminated when all the lines satisfy r-1 and r-2 or when αj is less than , which is a sufficiently small value. If there are delay contour lines representing multiple threetuple data, the algorithm retains their positions and moves the others. A preliminary experiment showed that the computational cost of this algorithm was low enough for it to be used in an online manner for a moderate number of delay contour lines (10 – 50). C. Search for Best Flow Assignment During the search for the best flow assignment using the hill-climbing algorithm, most of the possible flow assignments do not fall exactly onto a delay contour line. Given a flow assignment denoted by v(ux , rx ), the algorithm first identifies the two delay contour lines closest to v. Next, the queuing delay is linearly derived from the queuing delays represented by those two lines. In this process, the distance is measured with respect to the horizontal line from v (r = rx ). Although using four or more delay contour lines would enable more accurate estimation of the queuing delay, we use only two because, even though the delay contour line placement includes a more inaccurate factor, the estimated flow assignment still results in a good queuing delay, as described in Section VI. The way of dividing a given set of flows into two paths in the search process is: 1) initially the number of naive flow assignment is used; 2) the algorithm shifts a small amount flows of one flow type (flow type A) from one path to the other from the flow assignment in the previous iteration until the min metric reaches a local maximum; and 3) Next, the algorithm shifts a small amount of flows of the other flow type (flow type B) and returned to the movement of the flow type A. until the local maximum for step 2 no longer increases. The amount of shifted flows should reflect the amount of the increase of min metric (the larger increasing, the larger amount of flows are shifted). VI. E VALUATION We compared the queuing delay performance for two paths obtained using our proposed method with that using the naive

Ratio of delay metrics for proposed algorithm and naive assignment

1

10 data 20 data 30 data 40 data

0.9

0.8

0.7

0.6

0.5 0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Average utilization over two paths

Fig. 5. Ratio of min metric with proposed method to that with naive flow assignment against average utilization for two paths. The 99.5-th percentile queuing delay is used for computing of the min metric.

flow assignment. Figures 5 shows the ratio of the delay metric (the greater 99.5-th percentile queuing delay for the two paths) achieved with the proposed method to that with naive flow assignment (y-axis). The x-axis represents the average utilization for two paths. Traffic characteristics and bandwidth of paths were the same as in Section IV-B. 10–40 randomly selected three-tuple data were used to place the delay contour lines. The ratio of two min metrics (y-axis) took the average metric value over 100 runs using various combinations of the number of flows and random seeds. A ratio lower than 1.0 means that the proposed method achieved a higher (better) delay metric than did naive flow assignment. The proposed method outperformed the naive flow assignment in all cases covering a wide range of utilizations. Even with only ten three-tuple data the delay metric decreased up to 30%. As the number of three-tuple data increased (20, 30, 40), the delay metric decreased up to an additional 10%. The improvement was more significant the higher the utilization. These results indicate that even early in the process of collecting the three-tuple data (e.g., with only ten data), this method assigns flows so that the queuing delay is less than with the naive flow assignment. In addition, when the number of balanced flows increases resulting in the acceptance of more flows, the overall delay performance is remarkably improved by using the proposed method. These findings were consistent for all cases using other delay statistics (the mean and the standard deviation), different combinations of path bandwidths, and distinct traffic characteristics. VII. C ONCLUSION When balancing flows with distinct traffic characteristics over multiple paths solely on the basis of the path utilization, the queuing delay for the narrower-band path increases than does the more broadband path. So taking the traffic characteristics and the number of flows, and the path bandwidths into

account is vital for a good balancing of the flows in terms of the overall delay performance. We have presented a method for finding a flow assignment that gives good overall delay performance when distributing two types of flows having distinct traffic characteristics over two paths. It estimates the queuing delay statistics for an arbitrary assignment of flows with two given traffic characteristic types. The estimation requires little measured data for learning and does not rely on an analytical formula for the relationship between the traffic characteristics of the flows and the queuing delay. Experiments showed that it can assign flows in an online manner, and reduce the queuing delay in the path with the larger delay by up to 40% compared with the conventional path utilization based flow assignment. These results show that taking the traffic characteristics of the flows into account improves distribution over several paths. Future work includes evaluating the proposed method using actual traffic balanced over three or more paths to emulate a more practical situations and enhancing it so that it can handle flows with three or more distinct traffic characteristics. ACKNOWLEDGMENTS This work was supported in part by the Japan Society for the Promotion of Science, Grant-in-Aid for Scientific Research (S) (No. 18100001). R EFERENCES [1] D. Awduche, A. Chiu, A. Elwalid, I. Widjaja, and X. Xiao, “Overview and principles of Internet traffic engineering,” Internet Engineering Task Force, RFC 3272, May 2002. [Online]. Available: http://www.rfc-editor.org/rfc/rfc3272.txt [2] A. Elwalid, C. Jin, S. Low, and I. Widjaja, “MATE: MPLS adaptive traffic engineering,” in Proceedings of IEEE INFOCOM, Apr. 2001, pp. 1300–1309. [3] T. Guven, C. Kommareddy, R. J. La, M. A. Shayman, and B. Bhattacharjee, “Measurement based optimal multi-path routing,” in Proceedings of IEEE INFOCOM, Mar. 2004, pp. 187–196. [4] V. Trecordi and G. Verticale, “Per-flow delay performance in a FIFO scheduler fed by policed UDP sources,” Computer Communications, vol. 23, no. 4, pp. 309–316, 2000. [5] Y. Xu and R. Guerin, “Individual QoS versus aggregate QoS: A loss performance study,” in Proceedings of IEEE INFOCOM, June 2002, pp. 1170–1179. [6] Y. Kitatsuji, S. Katsuno, M. Tsuru, T. Takine, and Y. Oie, “On flow distribution over multiple path based on traffic characteristics,” in LNCS 3961, Proceedings of the ICOIN, Sept. 2006, pp. 483–492. [7] D. Anick, D. Mitra, and M. M. Sondhi, “Stochastic theory of a datahandling system with multiple sources,” The Bell System Technical Journal, vol. 61, no. 8, pp. 1871–1894, 1982. [8] M. Decina, T. Toniatti, P. Vaccari, and L. Verri, “Bandwidth assignment and virtual call blocking in atm networks,” in Proceedings of IEEE INFOCOM, 1990, pp. 881–888. [9] V. Ramaswami, “Matrix analytic methods for stochastic fluid flows,” in Proceedings of the 16th ITC, June 1999, pp. 1019–1030. [10] K. Kar, M. Kodialam, and T. Lakshman, “Minimum interference routing of bandwidth guaranteed tunnels with MPLS traffic engineering applications,” IEEE Journal of Selected Areas in Communications, vol. 18, no. 12, pp. 2566–2579, 2000. [11] K. Gopalan, T. Chiueh, and Y.-J. Lin, “Load balancing routing with bandwidth-delay guarantees,” IEEE Communications, vol. 42, no. 6, pp. 108–113, 2004. [12] ——, “Probabilistic delay guarantees using delay distribution measurement,” in Proceedings of the 12th annual ACM international conference on Multimedea, Oct. 2004, pp. 900–907.