A Transport Layer Load Sharing Mechanism for Mobile Wireless Hosts Ahmed Abd El Al, Tarek Saadawi, Myung Lee Dept. of Electrical Engineering, City University of New York, City College New York, NY 10031, USA
[email protected], {saadawi, mjlee}@ee1s0.engr.ccny.cuny.edu Abstract This paper introduces a transport layer solution for bandwidth aggregation, based on extending Stream Control Transmission Protocol (SCTP). The extended SCTP, which we refer to as Load-Sharing SCTP (LS-SCTP), is able to aggregate the bandwidth of the transmission paths between the communicating endpoints. In addition, LS-SCTP can dynamically add new paths to the bundle, as they become available. To insure that the transport connection does not stall due to the high loss rate or due to continuous failure of individual paths, LS-SCTP includes path monitoring and packet assignment mechanisms. These mechanisms monitor the paths and stripe the packets according to their condition. Simulation results demonstrated the effectiveness of our proposed mechanism in providing throughput and reliability to mobile applications.
1. Introduction As mobile devices become more prevalent, the demand for anytime/anywhere connectivity for those devices increases and the requirements on that connectivity become more stringent. Connectivity for mobile devices is governed by the communication technology on the device and the coverage area of the technology. To add versatility to the mobile devices, they are normally built with multiple communication technologies and the capacity for expansion. With the growing demand for mobility support, coverage areas for the wireless connectivity are growing and overlapping. If the mobile node has access to multiple communication channels, the application can be guided as to which channel currently the most appropriate. Such support is limited to the use of one communication channel per application data stream. The key contribution of our research is the implementation and performance evaluation of a mechanism for aggregation of resources across multiple heterogeneous channels in a mobile environment. The proposed mechanism provides the Prepared through collaborative participation in the Communications and Networks Consortium sponsored by the U.S. Army Research Laboratory under the Collaborative Technology Alliance Program, Cooperative Agreement DAAD19-01-2-0011.
mobile applications with a logical channel that has a bandwidth equal to the bandwidth of all the available channels. We implemented our bandwidth aggregation mechanism as an extension to Stream Control Transmission Protocol (SCTP) [1]. SCTP is a reliable, message-oriented data transport protocol that supports multiple streams within a single transport layer connection, an “association” in SCTP terminology, and hosts with multiple network addresses “multi-homed hosts”. SCTP provides different features that will be discussed in detail in section 3. The inherent support of multi-homing in SCTP makes it attractive than other transport protocols for bandwidth aggregation. Currently, SCTP uses multi-homing for redundancy, and not for load sharing. Each endpoint chooses a single destination address as the primary destination address for all the data units, “data chunks” in SCTP terminology. Retransmitted data chunks use the alternate address(es), under the assumption that this can improve the data chunks probability to reach the peer endpoint. Continues failure to send to the primary address ultimately results in the decision to transmit all the data chunks to an alternate destination until the primary destination become reachable again. Our proposed SCTP extension for bandwidth aggregation, which we refer to as Load Sharing SCTP (LS-SCTP), utilizes the available paths for simultaneous transmission of data chunks, while maintaining the SCTP congestion control on each path, so as to ensure fair integration with other traffic in the network. In order to aggregate the bandwidth, taking in account the differences in the paths congestion control characteristics such as loss rate, delay and delay jitter, LSSCTP separates between the congestion control and flow control. Congestion control is performed on a path basis, while the flow control is kept on association basis. In addition, LS-SCTP monitors the status of the available paths and as their status change, i.e. new paths become active or existing paths break, it updates the active paths list, which includes the paths that can be used for load sharing. In addition, LS-SCTP uses a retransmission technique that accelerates the delivery of the data chunks. Delaying the delivery of retransmitted data chunks can stall the association, while the receiver is waiting for these chunks. As will be shown in our performance study, in section 5, these features make LS-SCTP robust to the variation in the characteristics of the transmission paths.
Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops (PERCOMW’04) 0-7695-2106-1/04 $ 20.00 © 2004 IEEE
This paper is organized as follows. Section 2, provides a review for similar work in bandwidth aggregation. Section 3, presents a quick overview on SCTP features, highlighting the differences between SCTP and TCP. Section 4, describes in detail LS-SCTP design. Section 5 provides a performance evaluation for LS-SCTP. Finally, section 6 concludes the paper.
2. Related Work The idea of bandwidth aggregation to achieve higher throughput had been proposed at different levels of the protocol stack. On the link level, “Inverse Multiplexing”, is a standard application-transparent method to provide higher end-to-end bandwidth by splitting traffic across multiple physical channels, creating a single logical channel [2]. Inverse multiplexing implementations assume physical transport mechanisms with constant bit rates and stable channel characteristics, in terms of bandwidth, latency and loss rates. For this reason, they use a Round Robin approach to assign data fragments to the available channels. Although round robin data striping is simple, it not suitable when the channels have different characteristics. On the transport level, reference [3] introduces an end-to-end transport layer protocol called parallel TCP (pTCP). pTCP allows stripping the application data on multiple TCP connections. As appose to the multiple TCP connections in pTCP, our proposed bandwidth aggregation mechanism is based on a single LS-SCTP connection, due to the fact that SCTP inherently support multi-homing. This has several advantages: 1) It reduces the overhead of starting and ending multiple sockets; 2) there is no need for a module to manage the connections, similar to the Stripping Manager in pTCP; 3) LS-SCTP has direct access to the congestion status of all the paths. In addition, pTCP does not support dynamic addition/deletion of paths during the lifetime of the transport connection. On the application level, authors in [4] present an analytical model for the throughput of parallel sockets approach that uses multiple TCP connections (sockets) in order to achieve high throughput. The model is used to find the optimal number of sockets that achieve higher throughput for the applications without congesting the network. Implementing the bandwidth aggregation on the application level increases the complexity of the applications, as they are responsible for striping and re-sequencing the data. In addition, applications stripe the data on the sockets without considering the differences in the paths characteristics. As will be shown in our performance study, this can lead to a situation where a slow path can drag down the throughput of the whole bundle.
3. SCTP Overview SCTP is the fundamental member of a family of protocols designed by the SIGTRAN group to allow SS7 messages to be transported over an unreliable IP infrastructure. Data transfer
between two hosts in SCTP takes place in the context of a transport layer connection, an “association” in SCTP terminology. All transfer of data between hosts is encapsulated in SCTP packets. SCTP packet contains a common header and sequence of structures called “chunks”, which can be control chunks, such as a Selective Acknowledgement (SACK) chunks, or data chunks. The data chunk is the container for the user data transferred in SCTP. Data chunks incorporate flags to control the segmentation and reassembly, in addition to the following parameters: Transmission Sequence Number (TSN), Stream ID, Stream Sequence Number (SSN), and Payload Protocol Identifier. Stream ID and SSN are used for multistreaming support, while the Payload Protocol Identifier is included for future flexibility. Data chunks arriving at the SCTP receiver are acknowledged by transmitting SCTP packet with a SACK control chunk. SCTP has multiple features that make it different from TCP. The most attractive features are multi-streaming and multihoming support. Multi-streaming support allows data to be partitioned into multiple streams that have the property of being delivered independently, so that message loss in any of the streams will only affect the delivery within that stream, without affecting other streams. Multi-homing support allows a single SCTP endpoint to support multiple IP addresses. In addition, SCTP allows dynamic IP addresses reconfiguration [5]. This feature provides a mechanism that allows an SCTP endpoint to dynamically change (i.e. add/delete) IP addresses during the lifetime of an association. SCTP includes mechanisms for Path and Peer monitoring. An SCTP instance monitors all transmission paths to the peer instance of an association. To this end, Heartbeat control chunks are sent over all the idle paths, which are paths that are not currently used for the transmission of data chunks, regardless whether they are active or inactive. Each Heartbeat chunk has to be acknowledged by a Heartbeat-Ack chunk. A path is considered active if it has been used in the recent past to transmit an (arbitrary) SCTP packet that has been acknowledged by the peer. If transmissions on a certain path fail repeatedly more than a certain configurable limit, Path.Max.Retrans, the path is regarded as inactive. The number of events where Heartbeats were not acknowledged within a certain time, or retransmission occurred are counted on a per association basis, and when a certain configurable limit, Association.Max.Retrans, is exceeded the peer endpoint is considered unreachable, and the association terminated. SCTP congestion control is an amalgamation of current best practice for TCP implementations with extensions to deal with multi-homing aspect of SCTP and modifications due to the message rather than stream-based nature of the protocol. SCTP congestion control mechanism includes two major differences with the equivalent mechanism in TCP. First, the direct dependence of SCTP on the number of bytes acknowledged, rather than the number of acknowledgements received, to increase the congestion window. Secondly, the implicit dependence of SCTP on Selective Acknowledgement (SACK) messages for acknowledging the received data chunks.
Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops (PERCOMW’04) 0-7695-2106-1/04 $ 20.00 © 2004 IEEE
4. LS-SCTP Design In this section, we present our extensions for SCTP (LSSCTP) to support aggregating the bandwidth of the available transmission paths between the transport endpoints. An architectural view of LS-SCTP is shown in figure 1. Followed, we describe the overall architecture of the LSSCTP, as well as each of its modules. Application
A p p l i c a t i o n
LS-SCTP
Network
LS-SCTP
Path Monitor
Path Monitor
Path 1 Logical Buffer Association Buffer
Path Assignment Module
B u f f e r
Path 1 Logical Buffer
. . .
. . .
Path N Logical Buffer
Path N Logical Buffer
Sender
Association Buffer
Application
A p p l i c a t i o n B u f f e r
Receiver
Figure 1. Architectural view of LS-SCTP
4.1. LS-SCTP Architecture LS-SCTP is based on the idea of separating the association flow control from congestion control. In LS-SCTP the flow control is on association basis, thus both the sender and receiver endpoints use their association buffer to hold the data chunks regardless of the path these data chunks were sent or received. On the other hand, congestion control is performed on per path basis, thus the sender has a separate congestion control for each path, and these congestion control mechanisms can be used simultaneously for the data chunks transmission. The congestion control mechanism on each path is following RFC2960 [1], so as to insure fair integration with other traffic in the network. In order to separate flow control from congestion control, LS-STCP uses two different sequence numbers. The first sequence number is the Association Sequence Number (ASN), which is a per association sequence number, and is used to reorder the received data chunks at the receiver association buffer, regardless the path from which they have been received. The second sequence number is the Path Sequence Number (PSN), which is a per path sequence number, used for reliability and congestion control on each path. In addition, LS-SCTP continues to use the SSN for ordering the data chunks within the association streams. To support load sharing, we defined a new data chunk, Load Sharing Data chunk (LS-Data), by adding two new parameters to the standard SCTP data chunk [1]. The first parameter is a 4 bits Path Identifier (PID), which identifies the path used for the data chunk transmission. The second parameter is 12 bits Path Sequence Number (PSN), which is a monotonically increasing sequence number for the data chunks transmitted over the same path.
For acknowledging the received data chunks, LS-SCTP defines Load Sharing SACK (LS-SACK). LS-SACK includes two additional parameters than the standard SCTP SACK, Cumulative ASN ACK and Time Stamp. Cumulative ASN ACK is a per path cumulative acknowledgement, while the Time Stamp is used to order the LS-SACKs received from the different paths. The congestion control variables for each path, including congestion window size (cwnd), slow-start threshold (ssthresh), Round Trip Time (RTT), etc. are maintained in a data structure called the “Logical Buffer”. In addition, the logical buffer keeps track of the PSNs sent or received on each path. The receiver window size (rwnd), which represents the available space in the peer’s association buffer, as well as the total outstanding data count, are kept on an association basis.
4.2. Path Assignment Module The Path Assignment module in LS-SCTP is responsible for assigning transmission paths to the data chunks. The assignment is based on the bandwidth availability of the paths. The Path Assignment module uses the Congestion Window (cwnd) of each path, as an estimate of its bandwidth-delay product. The Path Assignment module assigns data chunks according to the cwnd/RTT of each path. After assigning a data chunk to a path, the data chunk is assigned the PID that identifies the path, as well as an in-sequence PSN. At the receiver, the data chunks do not compete on association buffer, as the sender controls the amount of data injected on all the paths, based on its view of the free space in the receiver’s association buffer (rwnd), as well as the total outstanding data on all the active paths. In order to compensate the differences in the paths RTT, the minimum receiver association buffer should be based on the summation of the bandwidth of all the paths and the maximum RTT, as shown in the following equation: Minimum Receiver Association Buffer = (1)
N
(
¦
B i ) * RTT
Max
i
where, Bi is the bandwidth of path i, N is the number of active path within the association and RTTMax is the maximum RTT. To improve the probability that a retransmitted data chunk(s) reaches the peer endpoint, the Path Assignment module retransmits the data chunk(s) on a different path than the one used for the original transmission. Delaying the arrival of a data chunk can stall the association, because data chunks that were sent on other paths will be queued at the receiver waiting the missing chunk. At a certain point the receiver’s association buffer space will limit the sender from sending more data chunks, until it receives the missing chunk. The PSN that was assigned to the data chunk(s) on the original path is returned to the PSN pool of this path, and can be used for new data chunk(s). The retransmitted data chunk(s) will be assigned PID and PSN on the retransmission path. In addition, the outstanding data counts on the old and new paths are
Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops (PERCOMW’04) 0-7695-2106-1/04 $ 20.00 © 2004 IEEE
adjusted according to the size of the retransmitted data chunk(s). Similarly, when time-out occurs on one of the active paths within the association, the Path Assignment module assigns the timed-out data chunks to different paths than the one they were originally timed-out on, that have enough capacity to transmit them. On the contrary of standard SCTP, that uses a new data chunk for probing the timed-out path, the Path Assignment module assigns a copy of one of the retransmitted data chunks for probing the path. The reason behind this, that there is a probability that the timed-out path could fail again, which delays the re-transmission of the new data chunk until the path is timed out again. This can lead the whole association to stall. The situation could be worse if the path fails for a long period. As the RTO of a path increases exponentially with consecutive time-outs, the association could repeatedly stall for exponentially increasing periods. This will continue until either the path recovers or it is marked as inactive by the Path Monitor, as will be described in the next section.
4.3. Path Monitor The Path Monitor module is responsible for updating the active paths list, which includes all the paths that can be used for load sharing, as well as for monitoring the status of the active paths. The Path Monitor updates the active paths list as it gets a feedback from the network regarding the failure of an exiting path or the availability of a new path. For example, when mobile IP protocol [6] reports a new Care Of Address (COA), with a PATH-ADD message, the Path Monitor performs the following actions: 1) It adds the new path to the existing association, using the Address Configuration Change (ASCONF) chunk [5]; 2) It creates a logical buffer for the new path; 3) Finally, it adds the new path to the active paths list. On the other hand, when the Path Monitor receives a PATHLOSS message from the mobile IP, it performs the following: 1) It removes the path from the active paths list; 2) It deletes the logical buffer that corresponds to the path; 3) Finally, it removes the path from the association using the ASCONF chunk. In addition, the Path Monitor removes a path from the active paths list when the number of consecutive retransmission time-outs on the path exceeds Path.Max.Retrans, which is set to 5, or when the path quality deteriorate to a limit that could affect the performance of the whole association. Currently, we are using a reasonable default threshold for the average loss rate on each path, and basing the path membership in the active paths list on that specified value. Inactive paths remain as a part of the association, and the sender keeps monitoring them through Heartbeat control chunks. As soon as a path recovers, the Path Monitor adds it again to the active paths list.
5. Performance Evaluation In order to examine the performance of LS-SCTP, we extended our SCTP implementation, in Opnet network
simulation software [7]. In our simulation, we used the network topology shown in figure 2. We assumed that an LSSCTP association is already initiated between the two hosts, and the association is unidirectional, which means that data chunks will only be sent from the host A (sender) to host B (receiver). In addition, we used only one stream within the association. Path1 Node 1
A1
Host A
A2
Path 2
. .
Node 2
AN
Path N
. .
B1
B2
. .
Host B
BN
Node N
Figure 2. Network model used for the simulation study Nodes 1-N are used to configure the characteristics of the transmission paths in terms of bandwidth, loss rate, delay and delay jitter. In addition, the nodes can be configured to represent different patterns for path failures. All the application packet sizes are set to 1 Kbytes. We set the application packets inter-arrival time so as to insure that the application will always have packets for transmission. The association buffer sizes (both sender and receiver) were set according to equation (1). Unless specified otherwise, the bandwidth of all the paths between host A and host B were set to 1.5 Mbps, with RTT = 100ms. In our performance study we used the association throughput as a performance metrics, which is defined as the amount of data delivered to the receiver’s application layer per second.
5.1. Effect of Bandwidth Disparity In this experiment we examine the performance of LSSCTP under the condition of paths with diverse capacities. We assumed that we have two paths, namely path 1 and path 2, such that the ratio of the bandwidths of the two paths is 1:5. We compared the performance of LS-SCTP to the ideal case, where the application’s throughput equals to the available bandwidth. Also we compared the LS-SCTP throughput to the throughput achieved by an application level bandwidth aggregation mechanism, i.e. the parallel sockets approach described in section 2. The x-axis in the figure 3 represents the total bandwidth of the two paths, while the y-axis represents the application throughput. As can be seen from the figure, that despite the difference in the bandwidths of the paths, the throughput achieved by LS-SCTP is close to the ideal throughput. The difference between the ideal case and LSSCTP is mostly due to the headers overhead from various protocol levels. The high throughput achieved by LS-SCTP is due to its striping mechanism that is based on the ratio of the bandwidth of the paths. On the other hand, the throughput of the parallel sockets is shown to be throttled by the low bandwidth path, as the application layer stripping is not based on knowledge about the characteristics of the paths.
Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops (PERCOMW’04) 0-7695-2106-1/04 $ 20.00 © 2004 IEEE
6 4 2 0 0
1
2
3
4
5
6
7
Available Bandw idth (M bps ) Ideal
Low BW Link Only
High BW Link Only
LS- SCTP
Par allel Socket s
Figure 3. Total throughput vs. available bandwidth
Association Throughput (Mbps)
Throughput(Mbps)
again. The reduction of the association throughput after path 1 failure, occurred for a duration approximately equal path one RTO, before the sender stopped considering path 1 for transmitting new data chunks. This throttling interval can be decreased by increasing the receiver association buffer than the limit set in equation (1), as this will allow the receiver buffer to absorb more sender transmission before filling up.
8
4 3 2 1 0 0
5.2. Effect of Bandwidth Fluctuation
4 3 2 1 120
110
100
90
80
70
60
50
40
30
20
10
0 0
20
30
40
50
60
70
80
90
Tim e (Sec.)
In this experiment, we study the performance of LS-SCTP with bandwidth fluctuation. We used two paths between host A and host B, path 1 and path 2. We represented the bandwidth fluctuation on path 1, by using a square-wave background traffic with amplitude 1 Mbps, and period t = 40 seconds. While we assumed that there is no background traffic on path 2. We run the simulation for 120 seconds. Figure 4 shows the association throughput. As can be seen that LSSCTP is able to adapt with the bandwidth fluctuation on the transmission paths, and utilize the available bandwidth, as the sender’s Path Assignment module is splitting the load on the paths based on an estimate of their bandwidths. Association Throughput (Mbps)
10
Figure 5. Association throughput with path failure
6. Conclusions The paper proposes an extension for SCTP, LS-SCTP, which aggregates the available bandwidth on the paths between the sender and receiver. LS-SCTP is based on separating the association flow control and congestion control. LS-SCTP distributes the data on the available paths based on an estimate of the bandwidth of each path. In addition, LSSCTP reacts to path failure and path quality deterioration without stalling the whole association. Performance study shows that LS-SCTP is able to utilize the available bandwidth efficiently, also it shows that LS-SCTP is able to react with path failures. We believe that our proposed bandwidth aggregation technique is extremely beneficial for networks with limited bandwidth and failure prone links.
Disclaimer
Time (Sec.)
Figure 4. Effect of bandwidth fluctuation on the throughput
The views and conclusions in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research laboratory or the U.S. Government.
5.3. Effect of Path Failure In this experiment, we examine the effect of path failure on the LS-SCTP. We assumed that there are two active paths within the association, path 1 and path 2, and path 1 fails for 40 seconds, after 5 seconds from starting the simulation. The failure duration was selected to insure that the sender’s Path Monitor would set path 1 as inactive early enough before its recovery. As shown from figure 5, after the failure of path 1 the sender continued to use path 2 for transmission of new data chunks. The data chunk copy on path 1 continued to time-out, until path 1 is set as inactive at t = 33 sec. The sender stopped probing path 1 cwnd, but at the same time it continued to monitor the status of path 1 using the Heartbeat chunks. After path 1 recovery at t = 50 sec., the first Heartbeat sent by the sender, at t = 52 sec., was acknowledged by a Heartbeat-Ack. Since then, the sender’s Path Monitor set path 1 as active
7. References [1] R. Stewart, Q. Xie et al., “Stream Control Transmission Protocol”, IETF, Request for Comments: 2960, 2000. [2] J. Duncanson, “Inverse Multiplexing”, IEEE Communications Magazine, 32 (4) (1994) 34- 41. [3] H.Hsieh, R.Sivakumar, pTCP: An End-to-End transport Layer Protocol for Striped Connections, Proceeding of IEEE International Conference on Network Protocols, Nov. 2002. [4] T. Hacker, B. Athey, “The End-to-End Performance Effects of Parallel TCP Sockets on a Lossy Wide-Area Network”, Proceedings of the IEEE IPDPS, 2002. [5] R. Stewart, M. Ramalho et al., “Stream Control Transmission Protocol (SCTP) Dynamic Address Reconfiguration”, Internet Draft (work in progress), draft-ietf-tsvwg-addip-sctp-07, August 2003. [6] D. Johnson, C. Perkins, “IP Mobility Support for IPv4”, RFC3344, Aug. 2002. [7] Opnet Modular, www.opnet.com.
Proceedings of the Second IEEE Annual Conference on Pervasive Computing and Communications Workshops (PERCOMW’04) 0-7695-2106-1/04 $ 20.00 © 2004 IEEE
