Performance comparison of TCP, HSTCP and XCP in high-speed ...

Performance comparison of TCP, HSTCP and XCP in high-speed, highly variable-bandwidth environments D. M. Lopez-Pacheco and C. Pham RESO/LIP, University Lyon 1 email:{dmlopezp,congduc.pham}@ens-lyon.fr Abstract

bandwidth to the transmission quality (which is very subject to climatic and interference phenomenon). Hand-overs may also introduce bandwidth variations if the user moves to a lower capacity cell. In wireless networks, most optimizations have focused in hiding the (unavoidable) losses caused by physical phenomenon. The fixed and constant available bandwidth assumption was true in wired networks, and still hold in a large part of the Internet, but we believe things may change in a near future. The reasons for this change is because many telco operators and Internet providers (ISP) are beginning to deploy Quality of Service (QoS) features with reservation-like or priority-like mechanisms in their networks (both access and backbone networks). In core networks, where DWDM (Dense Wavelength Division Multiplexing) techniques are widely deployed, (G)MPLS or RSVP-TE paths are foreseen to offer an elegant solution for traffic engineering concerns. In access networks, DiffServ with its various Per-Hop-Behavior (PHB) is used to introduce priority between packets and to emulate in some cases leased-lines with guaranteed QoS level. Note that some network technologies may also introduce dynamic bandwidth features [3, 10]. These new technologies, along with the strong desire to provide bandwidth-on-demand features, may turn wired networks into highly variable bandwidth environments for the best-effort traffic. This paper studies the behavior and the performance issues of TCP and its new variants in such environments. For constant bandwidth environments, Low et al. have shown in [15] that standard TCP becomes oscillatory as delay and bandwidth increase, and this regardless of the queue mechanism in use. The first work on variable bandwidth environments, to the best of our knowledge, has been done by Dutta and Zhang in [5]. They showed that TCP (NewReno) behaves quite badly when the bandwidth is varied over time. Sine-based variations were investigated and a PEPbased (Performance Enhancement Proxies, RFC 3135) solution was proposed. In this paper, the focus is not standard TCP’s performances (TCP NewReno) but rather the new propositions (High Speed TCP [8] and XCP [14]) for high-speed and high-delay networks, oftenly referred to as

The assumption of constant bandwidth capacity may be not true anymore because many telco operators and Internet providers (ISP) are beginning to deploy Quality of Service (QoS) features with reservation-like or priority-like mechanisms in their networks. Therefore, the available bandwidth for best-effort traffic can vary over time. This paper studies the behavior and the performance issues of TCP and its new variants in such environments. Both sinebased and step-based bandwidth variations models, which representing more closely dynamic bandwidth provisioning scenario, are used. The results highlight the problem of deterministic increase of the congestion windows which is not suitable for variable bandwidth environments. In our simulations, XCP that does not have pre-determined congestion window increment is always more performant.

1 Introduction TCP (Transmission Control Protocol) originally defined in RFC 793 has been the transport protocol of the Internet for more than 2 decades (almost 3 decades if we consider the first work initiated in 1974 by Khan and Cerf [4]). Since the congestion collapse observed by V. Jacobson in 1986 and the well-known slow-start and congestion avoidance algorithms proposed in 1988 [12], the networking community has proposed many enhancements and optimizations to the original proposition in order to make TCP more efficient in a large variety of network conditions and technologies [2, 11]: wireless links, asymmetric links and very highspeed and long delay links [8, 13, 14]. As a result, TCP’s congestion control performs quite good in many scenario and succeeds in sharing the available bandwidth and in offering acceptable fairness among the millions of Internet users. However most of the TCP’s optimizations have assumed than the bottleneck bandwidth remains constant over time. This assumption is not true in wireless networks because of the natural dependency of 1

offered by the ISP to the end-users willing to pay more for more guarantees. In the core network, very high bandwidth links for very high traffic aggregation may entirely rely on optical networking (without any conversion into the electrical domain) with GMPLS and RSVP-TE/LDP technology for traffic engineering. These links are likely to see a high concentration of flows. At the edge of the core network, before the pure optical domain, DiffServ and MPLS could again be used by the tier 1 operator to offer QoS and wavelength assignment to ISPs. In this scenario, the end-user will not be assigned a wavelength, but rather a small part of the available bandwidth bought by its ISP from the tier 1 operator. If we place ourselves a bit in the future, it is not unrealistic to foreseen very dynamic subscriptions to guaranteed bandwidth for very resource-consuming applications (grid computing, VoD, database replication, distributed simulation, virtual reality experiences, etc...). In this case, the bandwidth available for best-effort traffic will certainly suffer from large variations over time, especially in highaggregation links. It is interesting to mention that a highly variable-bandwidth environment could also be perceived by TCP-friendly protocols when competing with much more aggressive TCP versions (with very large initial slow-start threshold, large buffers or faster response function)

high bandwidth.delay product networks. HSTCP modifies the standard TCP response function to both acquire faster the available bandwidth (more efficiency) and to recover faster from packet losses in the network. XCP uses the assistance of routers to more accurately signal congestion conditions in the network. These 3 protocols represent the two trends currently in study in the research community for exploiting future generation networks: end-to-end (TCP, HSTCP) and router-assisted (XCP). The main goal of this paper is to study how these modifications behave on variable-bandwidth environments, and not to propose at this stage of the study any modification of these propositions. Also, with a focus on variable-bandwidth environments, we use only single flow configurations in order to better capture the behavior of the protocols, and let fairness issues for future works. In this paper, in addition to sinebased variations, we also use a different variation model representing more closely dynamic bandwidth provisioning scenario. Foreseen problems that might be amplified by variable-bandwidth environments are inefficiency due to convergence time (because it is difficult to quickly acquire the available bandwidth when it suddenly increases) and huge amount of packet losses due to dramatic reductions of the available bandwidth. The paper is organized as follows. Section 2 describes variable bandwidth environments and presents the variation models. Section 3 summarizes the properties of three transport protocols under study: TCP (NewReno), High Speed TCP and XCP. Section 4 presents the simulation results and our discussion. Section 5 concludes the paper and gives future work directions.

2.1 Bandwidth variation models In order to study the performance of TCP in these highly variable bandwidth environments, we present the two variation models used in this paper to represent such variations. In the rest of the paper, a guaranteed TCP flow or traffic will be referred to as a TCP flow or traffic with negotiated resources enabling strong QoS guarantees. We do not assume how these resources are negotiated and assigned to the flow, but only assume that the resulting effect on the best effort traffic is a decrease in the available bandwidth.

2 Variable bandwidth environments As stated in the introduction, variable bandwidth environments could also be found in wired networks due to the introduction of QoS mechanisms by network operators or ISPs. In the network example illustrated by figure 1, end users (campus, offices, residentials, etc...) are connected to intermediate ISPs which acts as a client to the tier 1 operator.

2.1.1 Continuous variations So-called continuous variation models are those where the increase or decrease of the available bandwidth for the best effort traffic does not present significant jumps as time is varied. In the real world, pure continuous variations (in the mathematical meaning) do not exist as the granularity of bandwidth is at the packet level, however, if the change in bandwidth is not significant when ∆t is small then we will refer such variations to as continuous. Figure 2 shows a 200Mbps link with both a pure sinebased bandwidth variations from 200Mbps to 25Mbps with a period of about 30s (dash line) and a more complex variation model (plain line). The y-axis represent the bandwidth available for best-effort traffic over time (therefore

Figure 1. Where bandwidth will vary. DiffServ Premium PHB or VPN/MPLS circuit could be 2

3 A quick review of transport protocols

the bandwidth reserved for guaranteed traffic is 200-y). This bandwidth variation model could represent a large number of guaranteed TCP flows coming in and out, resulting in variations for the best effort traffic. available bandwidth in Mbps

200

Transport protocols have usually the difficult task of providing reliability and fair sharing of the bandwidth to endusers. In the Internet, TCP Reno/NewReno is the most deployed protocol. With high bandwidth-delay product networks, transport protocols with more efficient mechanisms for acquiring bandwidth faster or for reacting better to congestions have been proposed in the research community. In this section, we briefly review the basic TCP (Tahoe/Reno/NewReno) mechanisms and the new High Speed TCP and XCP propositions for high bandwidth-delay product networks as some concepts (especially the congestion control mechanism and the response function) are important for the rest of the study.

more complex pure sinusoide

150

100

50

0

0

10

20

30

40

50

60

time is second

Figure 2. Sine-based continuous bandwidth variation model.

3.1 TCP Tahoe/Reno/NewReno

Dutta and Zhang [5] used a pure sine-based variation model to study TCP’s performances. They did not claim that such a simple function could represent closely the real variations that could be found in the Internet, but rather that such a model could be used to evaluate TCP in a variable bandwidth environment. We totally agree with this statement and use sine-based variations to model continuous variations and believe that such patterns (especially the more complex one) could be used to evaluate the behavior of TCP in core network’s links where the aggregation level is very high.

TCP Tahoe is the TCP protocol proposed by Jacobson after the congestion collapse observed in 1986. In TCP Tahoe, the sender maintains a congestion window (usually noted cwnd) that limits the amount of packets that it is allowed to send into the network without waiting for acknowledgments. The congestion control mechanism is a 2-phase control: a slow-start phase and a congestion avoidance phase. Following is a very simple description of the TCP Tahoe/Reno/NewReno protocols, readers are invited to read the original papers ([12], RFC 2581, RFC 2582) for the detailed and full description of the algorithms. TCP Reno/NewReno are evolutions of TCP Tahoe. TCP Tahoe: the TCP sender dynamically increases/decreases the congestion window size according to the congestion level of the network, which is conjectured by packet losses. Switching from the first phase to the second phase depends on the slow-start threshold (usually noted ssthres): starting with cwnd = 1pkt, if cwnd < ssthres then cwnd is doubled at every RTT (1, 2, 4, 8,. . . ), otherwise, it is increased by 1 pkt. When a timeout occurs or more than 3 duplicated ACKs are received (indicating either severe losses or a single segment loss), ssthres is divided by 2 and cwnd is set to 1 in order to start a new slow-start phase. ssthres could initially be set as high as possible. Note that fast retransmit can be used to retransmit the lost packet (and only this one) when duplicated ACKs are received without waiting for the timeout. This mechanism falls into the Additive Increase Multiplicative Decrease (AIMD) principle to achieve fairness with additive increase of 1 packet in the congestion avoidance phase and a multiplicative decrease of 1/2 when a congestion occurs. TCP Reno: when more than 3 duplicated ACKs are received, fast retransmit is used as in TCP Tahoe but a new fast recovery mechanism is implemented by dividing

2.1.2 Discrete variations So-called discrete variation models are those where the increase or decrease of the available bandwidth for the best effort traffic does present significant jumps (with discontinuity) as time is varied. Figure 3 illustrates the step model we use in this paper to represent a discrete variation model.

Figure 3. Step-based discrete bandwidth variation model. When compared to the previous model, the discrete model tries to represent large and sudden variations of the available bandwidth. Such variation patterns could model dynamic bandwidth provisioning of guaranteed traffic in both access and backbone networks. 3

ssthres by 2 and setting cwnd = ssthres, followed with a congestion avoidance phase (and not a slow-start phase as in TCP Tahoe, slow-start is only triggered by a timeout). TCP NewReno ( RFC 2582): in case of multiple losses within the same window of data, TCP NewReno uses the first acknowledgment received after the fast retransmit procedure to decide if more retransmissions should be performed. TCP NewReno is the most deployed version of TCP on the Internet. TCP achieves good fairness among other flows but known problems of TCP are low efficiency and high oscillations on high bandwidth.delay product networks.

Figure 4a shows the average congestion window size of HSTCP and TCP as a function of the packet loss rate p (the axis are in logarithmic scale). This curve uses the data provided in [8]. For high packet loss rate, up to 10−3 , HSTCP and TCP are similar. The HSTCP response function could be expressed by cwnd = 0.12/pO.835 [8]. Therefore, in congestion avoidance phase, cwnd is not increased by 1 packet every RTT, but by a dynamic value that depends on the current value of cwnd. In case of packet losses, the multiplicative decrease factor is also dynamic (but is lower than 1/2). Figure 4b plots the increase–a(w)–and the decrease– b(w)–parameters of HSTCP with logarithmic scale on the x-axis for cwnd ≥ 38, using the data provided in [8]. Note that for cwnd = 38, a(w)=1 and b(w)=1/2 which are the standard TCP factors. HSTCP has very good convergence time to full utilization but known problems of HSTCP are low fairness with TCP flows (even in low bandwidth environments) and a higher convergence time to fairness among HSTCP flows.

3.2 High Speed TCP High Speed TCP (HSTCP) is a modification proposed by S. Floyd to the TCP response function1 in order to acquire faster the available bandwidth (and faster reach full utilization of the link) in high bandwidth-delay product networks. The targeted network environments for HSTCP are low packet loss rate networks, therefore HSTCP proposes a faster congestion window increase compared to TCP. cwnd size (# of packets sent/RTT)

1e+08

3.3 XCP XCP [14] (eXplicit Control Protocol) is not an end-toend approach to congestion control as opposed to TCP or HSTCP. XCP uses router-assistance to accurately inform the sender of the congestion conditions found in the network. It therefore generalizes the ECN (Explicit Congestion Notification) mechanism proposed in Frame Relay networks for instance, and later on reused by the TCP ECN proposition [16] for IP networks. In XCP, data packets carry a congestion header, filled by the source, containing the sender’s current cwnd value, the estimated RTT and a feedback field (that could be positive or negative) representing the sender’s desired window increase. This last field is the only one which could be modified at every hop (XCP router), based on the value of the two previous fields. On reception of data packets, the receiver copies the congestion header into ACK packets sent back to the source. On reception of ACK packets, the sender would update its congestion window size as follows: cwnd = max(cwnd+f eedback, packetsize), with cwnd expressed in bytes. The core mechanism resides in XCP routers that use an efficiency controller and a fairness controller to update the value of the feedback field over the average RTT, which is the control interval. The efficiency controller (EC) has the responsibility of maximizing link utilization while minimizing packet drop rate. The EC basically assigns a feedback value proportional to the spare bandwidth S deducted from monitoring the difference between input traffic rate and output link capacity (S could be positive or negative) and to the persistent queue size Q (to avoid a feedback value of zero when input traffic is similar to output capacity). The authors in [14] proposes the following EC

TCP HSTCP

1e+07 1e+06 100000 10000 1000 100

10 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01

0.1

packet loss rate p

(a) 80

a(w) b(w)

70

0.45 0.35

50

0.3

40

0.25

30

0.2

20

0.15

10

0.1

0

10

100

1000

10000

b(w) value

0.4

60

a(w) value

0.5

0.05 100000

congestion window w

(b) Figure 4. HSTCP behavior. (a) response function compared to TCP, (b) a(w) and b(w) parameters. 1 The

commonly TCP response function expresses the congestion window size, or equivalently the throughput, as a function of the packet loss rate p. In the steady state, this function could be approximated by √ cwnd = 1.2/ p [9].

4

RED strategy, which drops packets according to the input buffer load, we use the ”gentle” mode and set the minimum threshold to one fourth the buffer size and the maximum threshold to 3 times the minimum threshold, as recommended in [17]. Only RED results are available for XCP simulations because the current XCP distribution code does not have the drop tail version for the XCP router.

equation: f eedback = α.rtt.S − β.Q, with α = 0.4 and β = 0.226. Then the fairness controller must translate this feedback value, which could be assimilated to an aggregated increase/decrease value, into feedback for individual flows (to be put in the data packet’s congestion header) following fairness rules similar to the TCP AIMD principles, but decoupled from drops because the difference between input traffic rate and output link capacity (S) is used in the EC. Note that no per-flow states are used by XCP routers to perform all these operations. XCP has very good convergence time to full utilization and fairness to other flows (TCP, HSTCP, XCP). There are no known problems to the best of our knowledge, except that router assistance is needed which limits its deployment.

4.2 Bandwidth variation models We use both pure sine-based variations with period of about 30s and step-based variations with changes every 30s. On 200Mbps links, the sine-based variation range is 30Mbps-200Mbps (that is the bandwidth available for besteffort traffic). For the step-based variation, we draw a random evolution and used it in all simulations. The maximum is 200Mbps, i.e. the capacity of the link, and the minimum is 30Mbps.

4 Simulations 4.1 Network model

4.3 Simulation results

The network model on which we simulate TCP, HSTCP and XCP is shown in figure 5. The bottleneck link is link b which propagation delay is either 50ms, 150ms or 300ms (i.e. RTT of 100ms, 300ms and 600ms respectively). 3 configurations are used which are detailed in the figure as case 1, case 2 and case 3.

This section presents the simulation results of TCP NewReno, HSTCP and XCP with the ns simulator [1]. The ns code we use for TCP NewReno and HSTCP is included in the ns package. The ns code we use for XCP can be downloaded on Dina Katabi’s web page. An XCP module is added to the ns simulator which realizes the XCP router functionalities. For TCP NewReno, we all network configurations. For HSTCP and XCP, only configuration 3 is used and we present simulation results for throughput, congestion window size cwnd and packet losses, both for sine and step variations. All results could be further found on http://perso.ens-lyon.fr/dino-martin.lopezpacheco/simulations/pages-web/simulations.html. All graphics show the bottleneck link delay, therefore the RTT is twice this value. The throughput is measured by monitoring the ACK packets sent back by the receiver. Therefore, packet losses causing duplicated ACK make the throughput to decrease.

Figure 5. Simple network model for ns simulations. R1 and R2 refer to the number of buffer slots (expressed in packets) that are available for each incoming link in the router. The optimal buffer size usually follows the rule of thumb of bandwidth.rtt product, e.g. for a 1Mbps link with RTT of 300ms (i.e. one way delay of 150ms), the buffer in number of 1Kbytes packets should be around 36. For 200Mbps, it is about 7324. For case 1&3, the buffer size is not optimal but reflects the reality. As the optimal buffer size is seldomly available due to design and cost constraints, we believe that an optimal buffer size at high-speed (case 3) would not be found in real systems. In case 2, we use a near optimal buffer size (according to the link capacity) when RTT is 300ms because 30 buffer slots is implementable in routers. s Two queue management strategies within routers are used: Random Early Drop (RED) and Drop Tail. In the

4.3.1 TCP NewReno Figure 6 shows the TCP throughput with network configuration 2&3 (1Mbps and 200 Mbps) when there are no bandwidth variations. The no variation case is included in this paper in order to serve as a reference. On the 1Mbps link, the window size advertised by the receiver is increased from 20 (default value in ns) to 256 in order to not being limited by the receiver window. On the 200 Mbps link, this value is increased to 4096 only in order to avoid high congestion situations at the beginning of the transmission (which cause in many timeouts and thus slow additive increase of the congestion window from a small value). 5

Without bandwidth variations (figure 6a), standard TCP on a rather low-speed 1Mbps link is very efficient and can grab all available bandwidth provided that sufficient buffer slots are available in routers (10 buffer slots are enough to considerably limit packet drops). On a 200Mbps link, when the RTT is small (about 100ms) standard TCP has no problem to acquire all the available bandwidth (results not shown) but as the RTT increases efficiency dramatically decreases as illustrated by figure 6b . This is the well-known problem of high bandwidth-delay product networks.

1.4

Bandwidth TCP New Reno - 50ms TCP New Reno - 150ms TCP New Reno - 300ms

Throughput (Mbps)

1.2 1 0.8 0.6 0.4 0.2 0

0

20

40

Throughput (Mbps)

Throughput (Mbps)

Bandwidth TCP New Reno - 50ms TCP New Reno - 150ms TCP New Reno - 300ms

1 0.8

1 0.8 0.6 0.4

0.6

0.2

0.4

0

0.2 0

100

Bandwidth TCP New Reno - 50ms TCP New Reno - 150ms TCP New Reno - 300ms

1.2

1.2

80

(a) 1.4

1.4

60 Time (s)

0

10

20

30

40

50

60

70

80

90

100

Time (s) 0

10

20

30

40

50

60

70

80

90

(b)

100

Time (s)

(a)

Throughput (Mbps)

200

Figure 7. TCP sine variations, 1Mbps link, DT. Buffer size (a) 5, (b) 30.

Bandwidth TCP New Reno - 300ms

150

We can also see on the figure that the TCP throughput is more stable when the router’s buffer size is small (case 1, figure 7a). The reason is that small buffers introduce more frequent packet losses that limit the growth of the sender cwnd. If we plot the congestion window evolution, we would see that the TCP sender in figure 7a always uses fast retransmit/fast recovery mechanism. On the contrary, when the buffer size is larger (30 packets), the buffer compensates for the bandwidth variations and lets the sender’s congestion window grow. Eventually, with a large cwnd, the router’s buffer will become full and severe losses, and then timeout, would occur which would trigger slow-start phases (this behavior has been verified when we plotted the congestion window evolution for the 30-packet case). We can use this simple experiment to try to better understand the TCP behavior in such a variable bandwidth environment. If we look at figure 7, we can see that the 600ms RTT case shows a throughput increase slope of about 0.02 in the congestion avoidance phase (meaning that throughput is increased at the rate of 0.02Mbps per second). Actually, we found easier to look at the throughput evolution than looking at the cwnd evolution. Figure 8a illustrates on a schematic graph the same bandwidth variation conditions that those of figure 7, and shows with a dashed line the slope of the throughput increase for the 600ms-RTT case. We have represented with the dashed-line arrows one possi-

100 50 0

0

50

100

150

200

250

300

350

400

450

500

Time (s)

(b) Figure 6. TCP no variations, DT. Throughput (a) 1Mbps link, (b) 200Mbps link.

When bandwidth changes as illustrated in figure 7 for a 1Mbps link (network configuration 1&2), we can see that TCP has many problems to recover from packet losses (which are more frequent in case 1) when the RTT is high. In this case, TCP exhibits much lower efficiency when compared to the maximum throughput shown in figure 6a. This is due to the convergence time. For small to medium RTT values (100ms and 300ms), the throughput curve follows quite closely the bandwidth variation curve because the router’s buffers can compensate in some extent the bandwidth variations. 6

parameters, we indeed got the same curves than those presented in [5]. However, we believe that our results presented in this paper are more realistic.

ble evolution of the sender throughput (this is a simple view since slow-start has not been taken into account): when the line crosses the bandwidth variation curve while it is decreasing, there are drops and cwnd will eventually restart from 1. There may be several re-initialization phases until the bandwidth curve increases again. When this happens, if the throughput increase slope is much smaller than the bandwidth increase slope then the efficiency is very low. The amount of wasted bandwidth is the surface captured by the bandwidth variation curve and the throughput increase line, between point a and point b.

Throughput (Mbps)

200

Bandwidth TCP New Reno - 300ms

150 100 50 0

0

50

100

150

200

250

300

350

400

450

500

Time (s)

(a) cwnd size (# of packets sent/RTT)

2500

1500 1000 500 0

(a)

TCP New Reno - 300ms

2000

0

50

100 150 200 250 300 350 400 450 500 Time (s)

(b) Figure 9. TCP sine variations, 200Mbps link, DT. (a) throughput, (b) cwnd. Figure 9 shows the throughput and the congestion window size for the sine variation case and 200Mbps link (network configuration 3). At high-speed, the inefficiency of TCP is amplified when bandwidth is varied. In figure 9a at time 10s, many losses occurred because of the dramatic decrease in bandwidth, resulting in 2 timeouts at the sender. We can see that in this case where cwnd is still small when losses occurred, the high convergence time of TCP makes inefficiency much higher than in figure 6b. Although not shown, RED routers give similar results.

(b) Figure 8. Schematic view. (a) experimental conditions (RTT=600ms), (b) longer period for bandwidth variation if RTT=600ms. The efficiency could be defined as the ratio between the surface under the throughput increase line and the total surface under the bandwidth variation curve, taken from a to b. In the case of the pure sine-based variation, if we want to match the bandwidth increase slope to the throughput increase slope, or vice-versa, the variation period must be much longer, or the RTT must be smaller. This is depicted in figure 8b. We do not say that this strategy is the optimal one for maximizing efficiency (the optimal throughput increase slope is quite difficult to determine!). The results presented in figure 7 show a better behavior of TCP on a low-speed link than those previously presented by Dutta et al in [5]. We believe the authors in [5] did not tune the TCP agent in ns (receiver window size for instance), and especially did not increase the router buffer size set by default to 1 packet. With the default TCP agent

4.3.2 HSTCP Figure 10, 11 and 12 show the throughput, the congestion window size and the packet losses with respectively no variations, sine variations and step variations. All tests are performed with network configuration 3 (200Mbps link). The no variation case is included in this paper in order to serve as a reference, extensive simulations of HSTCP could be found in [6, 7] and on the author’s web page. HSTCP has been proposed to better handle high bandwidth − delay product networks. We can see in figure 10 that it succeeds in doing so when bandwidth remains constant: cwnd is increased much faster than in TCP, resulting in more efficiency even at high RTT. HSTCP on RED routers in our 7

tests is not as performant as on Drop Tail routers regarding the maximum throughput, but the number of lost packets per congestion window is much lower (results not shown). Previous simulations [6] also show that fairness increases with RED routers when the number of flows is high. 350

Bandwidth HSTCP - 150ms HSTCP - 300ms

300

200

250 200

Throughput (Mbps)

Throughput (Mbps)

available rate, which results in an exact match between the throughput curve and the bandwidth curve. When buffers are empty, the throughput increase is regulated by the a(w) factor, with a slope much smaller than the bandwidth increase slope. However, if compared to the TCP NewReno additive increase of 1 packet (see figure 9a), HSTCP performs better and shows higher efficiency.

150 100 50 0

0

50

100

150

200

250

300

350

400

450

Bandwidth HSTCP - 150ms HSTCP - 300ms

150

100

50

500

0

Time (s)

0

50

100

150

(a)

250

300

350

400

450

500

450

500

Time (s)

(a)

300

Bandwidth HSTCP - 150ms HSTCP - 300ms

250

200

200

Throughput (Mbps)

Throughput (Mbps)

200

150 100

Bandwidth HSTCP - 150ms HSTCP - 300ms

150

100

50

50 0

0

0

50

100

150

200

250

300

350

400

450

500

0

50

100

150

cwnd size (# of packets sent/RTT)

cwnd size (# of packets sent/RTT)

7000

HSTCP - 150ms HSTCP - 300ms

16000 14000 12000 10000 8000 6000 4000

300

350

400

HSTCP - 150ms HSTCP - 300ms

6000 5000 4000 3000 2000 1000 0

2000 0

250

(b)

(b) 18000

200

Time (s)

Time (s)

0

50

100 150 200 250 300 350 400 450 500 Time (s)

0

50

(c)

100 150 200 250 300 350 400 450 500 Time (s)

7

(c)

HSTCP - 150ms HSTCP - 300ms

# of drops packets

6

Figure 10. HSTCP no variations. Throughput: (a) DT, (b) RED. cwnd: (c) DT.

5 4 3 2 1

For drop tail routers, figure 11c shows that during the initial slow-start phase, cwnd increases exponentially and can reach 6875 packets in 7s. When the bandwidth is decreased, many packets are dropped causing cwnd to decrease. Note that there are no timeouts when RTT=300ms therefore HSTCP is most of the time in the congestion avoidance phase. In this case, cwnd varies according to the a(w) and b(w) parameters shown in figure 4b. If we look at the throughput (figure 11a), when bandwidth increases again, packets in router’s buffers are sent at the

0

0

50

100

150

200

250

300

350

400

450

500

Time (s)

(d) Figure 11. HSTCP sine variations. Throughput: (a) DT, (b) RED. cwnd: (c) DT. Packet losses (d) RED. For RED routers, cwnd is generally smaller than for the 8

For step-based variations, it is difficult to capture the general behavior as the variation sequence may impact on the results from one experiment to an other (this result also shows the limit of the current simple step model, we will go back to this point later on). The main difference with sinebased variations is that the available bandwidth can drop significantly in a very short amount of time (possibly in several times). If this bandwidth drop can not be compensated by the buffer then there are unavoidable packet drops when cwnd is high. For instance, when both RTT and cwnd are high, if the bandwidth diminishes dramatically, a timeout can occur (figure 12c at time 272s which causes a slow-start and a significant degradation of the performance.

drop tail case because of the random and more frequent losses introduced by the RED policy. The direct impacts are a much smaller number of packet losses and a lower throughput. Regarding the ability of HSTCP to follow the bandwidth variations, the same remark than the previous case still holds: the throughput increase rate in the congestion avoidance phase is too small to provide high efficiency, especially for high RTTs. 250

Bandwidth HSTCP - 150ms HSTCP - 300ms

Throughput (Mbps)

200 150 100 50 0

0

50

100

150

200

250

300

350

400

450

500

Time (s)

(a) 250

Bandwidth HSTCP - 150ms HSTCP - 300ms

Throughput (Mbps)

200 150

Figure 13. The 3 cases in step-based variations.

100 50 0

0

50

100

150

200

250

300

350

400

450

We illustrate in figure 13 the 3 cases that can be found in step-based variations models (note that this discussion is also valid for TCP). Case (a) and (b) show an increase of the throughput to the maximum value before bandwidth is decrease to a smaller amount. In case (b), although there were several bandwidth changes, the behavior remains the same as the throughput increase is not limited by the amount of available bandwidth. In case (c), the bandwidth is decreased before the throughput could reach the maximum value. We identified 3 phases in this case. Phase a happens when bandwidth is decreased. Packets will certainly be dropped at the routers but since the buffer is not empty, the receiver still receives in-sequence packet and thus keeps sending outstanding ACK packets. This is why the throughput curve matches the bandwidth curve. At phase b, the sender is notified of the (previous) packet losses (with duplicated ACK, which causes the monitored throughput to decrease) and enters in a fast retransmit/fast recovery procedure (timeout can happen here if the number of packet drops is too high). The throughput could go up quickly if the retransmitted packets fill the gap at the receiver. This explains why the throughput curve matches the bandwidth curve again. In phase c, the receiver is paced by the sender’s ”normal” congestion window evolution, until the next bandwidth variation. Case c can be identified in figure 12a between time 170s and 250s for RTT=600ms. The fast recovery/fast retransmit phase can be seen in figure 12c at

500

Time (s)

(b) cwnd size (# of packets sent/RTT)

18000

HSTCP - 150ms HSTCP - 300ms

16000 14000 12000 10000 8000 6000 4000 2000 0

0

50

100 150 200 250 300 350 400 450 500 Time (s)

(c) 3500

HSTCP - 150ms HSTCP - 300ms

# of drops packets

3000 2500 2000 1500 1000 500 0

0

50

100 150 200 250 300 350 400 450 500 Time (s)

(d) Figure 12. HSTCP step variations. Throughput: (a) DT, (b) RED. cwnd: (c) drop tail. Packet losses (d) DT.

9

time 200s. For RED routers, the behavior is not exactly the same as packet drops are triggered by a given load threshold. The main change is in phase a where packets in the buffer are not necessarily in sequence, therefore sequence number gaps can cause fast retransmit/fast recovery procedure at the sender. This behavior can be seen in figure 12b at time 245s.

XCP router would always compute a higher feedback value than available. In XCP-e, we made XCP routers aware of the bandwidth variations so that they use the correct value for the available bandwidth. Real implementations of this scheme may be more difficult since XCP routers should then know, somehow, how much bandwidth is available for best-effort traffic (typically would need some states in routers, which is not required by XCP when there are no bandwidth variations).

4.3.3 XCP Figure 14 shows an XCP sender throughput and its congestion window evolution for RTT values of 300ms and 600ms, when there are no bandwidth variations. We can see that (i) XCP reaches full utilization very fast, and (ii) that cwnd is very stable even for high RTT. Having feedbacks from routers is very beneficial to the stability of the congestion control mechanism.

Throughput (Mbps)

100 50

Bandwidth XCP - 150ms XCP - 300ms

0

50

100

150

200

250

300

cwnd size (# of packets sent/RTT)

100 50

50

100

150

200

250

300

350

400

450

500

Time (s)

(a)

10000 8000 6000 4000 2000 0

50

100 150 200 250 300 350 400 450 500 Time (s)

(b)

12000 10000

700

8000

XCP - 150ms

600

# of drops packets

6000 4000 2000 0

500

12000

XCP - 150ms XCP - 300ms

14000

450

XCP - 150ms XCP - 300ms

14000

0

16000

400

(a) 16000

0

350

Time (s)

150

0

cwnd size (# of packets sent/RTT)

Bandwidth XCP - 150ms XCP - 300ms

150

0

200

Throughput (Mbps)

200

0

50

100 150 200 250 300 350 400 450 500 Time (s)

(b)

500 400 300 200 100 0

Figure 14. XCP no variations, RED. (a) throughput, (b) cwnd.

0

50

100

150

200

250

300

350

400

450

500

Time (s)

(c) Figure 15. XCP sine variations, RED. (a) throughput, (b) cwnd. Packet losses (c) 150ms.

When bandwidth is varied, we use 2 versions of the XCP simulation code: the original version distributed by Katabi, noted XCP, and a slightly modified (by ourself) version, noted XCP-e. In the original code, XCP routers compute the difference between the input traffic rate and the output link capacity to deduce a feedback value. As there were no bandwidth variations in Katabi’s study, the output link capacity is always considered constant. When bandwidth for best-effort is varied and is smaller than the link capacity, the

Figure 15 and figure 16 show the XCP sender throughput and cwnd evolution when sine-based variations and step-based variations are used respectively. We can see that the XCP sender can acquire very quickly (almost instantaneously) the available bandwidth, even when RTT is 10

routers are not able to store all the packets sent. This problem becomes more serious when the RTT increases, as the reaction time increases. In figure 17b, we can see this phenomenon happening many times when RTT=600ms.

high. However, in all cases, when the bandwidth decreases by large amounts, the computed feedback is erroneous and many packets are dropped by the routers. We can see in figures 15b and 16b that the congestion window is reset many times, but with small impacts on performances as cwnd can jump to a given value at the next feedback notification (and does not depend on previous value as in TCP or HSTCP).

Throughput (Mbps)

Throughput (Mbps)

200

200

Bandwidth XCP - 150ms XCP - 300ms

150 100

Bandwidth XCP - 150ms XCP - 300ms

150 100 50 0

0

50

100

150

200

250

300

350

400

450

500

Time (s)

50

(a) 0

50

100

150

200

250

300

350

400

450

16000

500

cwnd size (# of packets sent/RTT)

0

Time (s)

(a) cwnd size (# of packets sent/RTT)

16000

XCP - 150ms XCP - 300ms

14000 12000 10000 8000

12000 10000 8000 6000 4000 2000 0

6000

0

50

4000

100 150 200 250 300 350 400 450 500 Time (s)

(b)

2000 0

0

50

2000

100 150 200 250 300 350 400 450 500

1600

# of drops packets

(b) 2500

XCP - 150ms

1800

Time (s)

XCP - 150ms

2000

# of drops packets

XCP - 150ms XCP - 300ms

14000

1400 1200 1000 800 600 400

1500

200 0

1000

0

50

0

100 150 200 250 300 350 400 450 500 Time (s)

500

(c) 0

50

5000

100 150 200 250 300 350 400 450 500

XCP - 300ms

4500

Time (s)

4000

# of drops packets

(c) Figure 16. XCP step variations, RED. (a) throughput, (b) cwnd. Packet losses (c) 150ms.

3500 3000 2500 2000 1500 1000 500 0

When XCP routers are aware of the bandwidth variations, the protocol has a much better behavior (almost perfect) because cwnd is modified to take into account the new amount of available bandwidth. However, when cwnd has a considerable value and feedback is almost equal to 0, a dramatic decrease of the available bandwidth can cause high amount of packet losses (and thus producing timeouts) since

0

50

100 150 200 250 300 350 400 450 500 Time (s)

(d) Figure 17. XCP-e step variations, RED. (a) throughput, (b) cwnd. Packet losses (c) 150ms (d) 300ms.

11

5 Conclusion

[7] E. Souza, D. Agarwal. A HighSpeed TCP Study: Characteristics and Deployment Issues. LBNL Technical Report Number LBNL-53215.

In this paper, we focus on variable bandwidth environments and study the behavior of 3 transport protocols: TCP NewReno, High Speed TCP and XCP. The main result we got from this study which consider sine-based and stepbased bandwidth variations is that deterministic increase of the congestion windows is not suitable for variable bandwidth environments. In our simulations, XCP that does not have pre-determined congestion window increment is always more performant. However, as stated previously, XCP has deployment problems that TCP and HSTCP do not have. Therefore, XCP may not be available and in this case an end-to-end approach must be used. We do not say that these classes of protocols are bad (they were maybe not designed for these particular environments where bandwidth can vary with high amplitudes over time), we just want to point out that current propositions do have some problems if operated in such environments. We believe next generation networks may present such behavior and therefore we are seeking for more experiments and feedbacks. For instance, the simple step-based variation model we used in this paper is not accurate enough to allow extensive simulations of the protocols. More parameters of the variation models and how they relate to the protocol performances must be investigated.

[8] S. Floyd. HighSpeed TCP for Large Congestion Windows. RFC 3649, Experimental, December 2003. [9] S. Floyd, K. Fall. Promoting the use of end-to-end congestion control in the Internet. IEEE/ACM Transactions on Networking, August 1999. [10] L. Gauffin, L. Hakansson, B. Pehrson. Multi-gigabit networking based on DTM. Computer Networks and ISDN Systems, 24(2):119?130, April 1992. [11] G. Hasegawa, M. Murata. Survey on fairness issues in TCP congestion control mechanisms. IEICE Transactions on Communications, January 2001. [12] V. Jacobson. Congestion avoidance and control. In ACM SIGCOMM ’88, 1988. [13] C. Jin, D. X. Wei, S. H. Low. FAST TCP: Motivation, Architecture, Algorithms, Performance. IEEE Infocom 2004. [14] D. Katabi, M. Handley, C. Rohrs. Congestion Control for High Bandwidth-Delay Product Networks. ACM SIGCOMM 2002.

References [1] The network simulator http://www.isi.edu/nsnam/ns.

[15] S. H. Low et al. Dynamics of TCP/AQM and a scalable control. IEEE Infocom 2002.

ns-2.

[16] K. K. Ramakrishnan, S. Floyd. Proposal to add explicit congestion notification (ecn) to IP IETF RFC 2481.

[2] C. Barakat, E. Altman, W. Dabbous. On TCP performance in a heterogeneous network: a survey. IEEE Communication Magazine, January 2000.

[17] http://www.icir.org/floyd/REDparameters.html

[3] C. Bohm et al. Fast circuit switching for the next generation of high performance networks. IEEE Journal on Selected Areas in Communications, 14(2):298? 305, February 1996. [4] Cerf, V., and R. Kahn. A Protocol for Packet Network Intercommunication IEEE Transactions on Communications, Vol. COM-22, No. 5, pp 637-648, May 1974. [5] D. Dutta, Y. Zhang. An active proxy based architecture for TCP in heterogeneous variable bandwitdh networks. GlobeCom 2001. [6] E. Souza. A Simulation-based study of HighSpeed TCP and Its Deployment. LBNL Technical Report Number LBNL-52549. 12