Performance of Asynchronous, Variable Bit Rate ... - CiteSeerX

Performance of Asynchronous, Variable Bit Rate Communications over ATM: A Comparison with FDDI Stephen Dempsey, Eric Fryland Lockheed Martin Government Electronic Systems Moorestown, NJ 08057 [email protected], [email protected] Brian Hazzard, David Cyganski Worcester Polytechnic Institute Electrical and Computer Engineering Department Worcester, MA 01609 [email protected], [email protected]

ABSTRACT Asynchronous Transfer Mode (ATM) technology performance is evaluated for Class C (variable bit rate, asynchronous) data transfer, as would be typically found when migrating legacy applications to an ATM environment. Of particular interest are the end-to-end latencies obtainable using ATM networks to transfer time critical data. Latencies obtained using Classical IP over ATM and raw ATM using the Fore ATM API are compared to performance of traditional TCP/IP on equivalent FDDI networks. It is shown that at OC-3, in it’s simplest configuration of back to back nodes, Classical IP over ATM latencies are greater than those obtained using FDDI, particularly at small message sizes. However, the results show that IP over ATM via ATM switches can exhibit better performance than FDDI switches or routers, due to the optimization of the ATM switch for fixed size cells. Thus, these benchmarks confirm that ATM provides a good foundation for building high performance IP internetworks. Using raw AAL 3/4 ATM, significantly lower latencies than the equivalent IP over ATM results or traditional FDDI results are presented, both in back-to-back configurations and through network devices such as switches and routers. A discussion of tradeoffs associated with using FDDI versus ATM for mission critical, real time data transfer applications is also included.1 I.

INTRODUCTION

In the past, networks suffered from a lack of available bandwidth. While network technologies operating at speeds of 100 Mbps or higher were 1

This research is funded by the Department of Defense through the US Navy Naval Sea Systems Command, contract N00024-95-C-5159.

available, their associated costs and complexity left them relegated primarily to the campus area backbone or other niche applications. 10 Mbps Ethernet dominated the markets, and continues to do so. Nevertheless, emerging high-speed networking technologies offer the promise of large bandwidth at low costs. Switched architectures retain the flexibility of the LAN while offering the promise of better bandwidth management and decreased response times. ATM technology offers the benefits of switched architectures, while adding broadband capabilities. As such, it is possible to develop architectures which allow for the establishment of multiple communications channels simultaneously, but with a tailored Quality of Service (QOS) for each one. ATM itself is designed to be efficient for time sensitive data such as digitized voice or video. In order to do so, a small, fixed length cell of 53 bytes is used for all data transfer (5 byte cell header, 48 byte payload) [1]. Classical IP over ATM, as defined in Request For Comment (RFC) 1577, provides users the ability to utilize ATM as the underlying protocol over which legacy applications may run. Thus, it is natural to assume that migrating legacy applications to an ATM environment is simple and will provide measurable performance increases. The AEGIS Weapon System (AWS) is the heart of the U. S. Navy’s DDG 51 class Destroyers. Because the computers in the AWS control all facets of the AEGIS Anti-Air Warfare mission (target detection, track, and, as required, engagement with weapons in real time), timing relationships between computers must be carefully maintained. Thus communications latency, more so than aggregate bandwidth, has been the focus of this research. It is felt that as high bandwidth becomes increasingly available at low cost, similar

concerns over end to end communication latencies will play an important role to developers of multimedia and real time network applications in the future. The primary objective of this paper is to describe the performance of ATM as a migration path for real time application communications. Measured ATM performance will be compared with similar results obtained from a comparable Fiber Distributed Data Interface (FDDI) network. Thus transfer of variable bit rate, asynchronous traffic using both Classical TCP/IP over ATM and raw ATM using the Fore ATM API are compared to the performance of conventional TCP/IP protocols over an isolated FDDI LAN. The remainder of the paper is organized as follows: In Section II, performance of the IP protocol suite over 100 Mbps FDDI is established for a simple two node back-to-back FDDI LAN and between distinct subnets interconnected in one instance with a router and in another via a FDDI switch. Section III discusses similar measurements made in a small ATM testbed using both Classical IP over ATM using AAL 5 and raw ATM AAL 3/4. In this case, a simple back-to-back arrangement is analyzed, as well as a simple switched arrangement. Section IV discusses tradeoffs associated with developing an ATM solution for LANs vice using FDDI or other traditional IP networks. Finally, Section V presents a summary of results and potential future research.

Figure 1: UDP/IP Latency Over FDDI for Several different processor types.

Round Trip Latency Characteristics of FDDI The performance of FDDI has been tested extensively to ensure it’s suitability for AEGIS. Results show that FDDI is, on average, deterministic, though overall end-to-end latencies tend to rely heavily on the specific TCP/IP protocol stack implementation. An example of this can be seen in Figure 1, where the Mean Round Trip Latency versus Message Size is shown for several different processor pairs. It has been shown that simple data of this type can be used to successfully predict network behavior under increasingly complex traffic conditions [3]. In order to characterize FDDI performance, three FDDI LAN scenarios were created. First, performance across a small isolated FDDI LAN without intervening devices was tested. Then the FDDI LAN was broken into two subnets and data was passed between them via a router. Finally, the router was replaced with a high performance FDDI switch. In all arrangements, both round trip latency and achievable throughput were measured across a range of transmitted packet sizes. In order to measure latency for a given message size, messages containing an embedded time stamp and necessary padding were sent from a client process on one node to a receiving process at the other node, where they were reflected back in entirety to the originating node. Upon receipt of echoed packets, the time stamp was extracted and compared with current time, yielding round trip end-to-end latency. Benchmarks can be obtained using either TCP or UDP at the transport layer.

II. PERFORMANCE OF LEGACY TCP/IP FDDI NETWORKS Before a meaningful analysis of the performance of ATM networks may be undertaken, it is first necessary to understand the capabilities of available high speed network communications technologies. Of primary interest in this research is 100 Mbps FDDI. FDDI couples deterministic upper bounds on media availability via token mechanisms with innate fault tolerance which makes it extremely attractive for mission critical computing in a real time environment such as the AEGIS Weapon System. As a result, FDDI has found extensive use as the LAN communications medium of choice within AEGIS [7].

1

Figure 2: UDP/IP and TCP/IP Latency Over FDDI between HP-J210 workstations on an isolated ring.

Figure 3: TCP/IP and UDP/IP Latency Over FDDI Subnets through a router.

FDDI Scenario 1: Back to Back Workstations For the first FDDI scenario, two Hewlett Packard J210 workstations were placed on an isolated ring with no other devices. Results of latency measurements versus data size using both UDP/IP and TCP/IP in this configuration are shown in Figure 2. Note that for both UDP and TCP, the performance of the J210 workstations yields a surprising latency characteristic at message payloads of roughly 2000 octets, where larger message sizes encounter lower latency. This is highly device specific, and has not been observed in any other processor pair, illustrating concretely the large degree of dependence on system hardware and software. It is apparent that the FDDI latencies which are observed are far more dependent on the protocol stack implementation and system hardware than on the FDDI data transfer mechanisms themselves. This can be seen by referring back to Figure 1, which shows the average UDP/IP round trip latency characteristic between various processor pairs on an isolated FDDI ring. In all cases TCP latencies were similar, though in some instances (the J210s, for example) they tend to be higher than those associated with UDP at large message sizes (> FDDI MTU).

Figure 4: TCP/IP and UDP/IP Latency Over FDDI Segments through a FDDI switch. expected, overall round trip latency was measurably increased as a result of both read-in and processing time within the router. Figure 3 shows the average measured latencies for TCP and UDP. It should be noted that the router configuration used is optimized to provide the fastest possible routing of packets, as opposed to the most cost effective usage of available ports. This, coupled with the relatively small routing tables present in the lab environment will tend to produce results somewhat more favorable than might be expected in practice. FDDI Scenario 3: Segments Connected Via a Switch The J210 workstations were disconnected from the Cisco router, and were placed on different ports of a high performance FDDI switch. As in Scenario 2, UDP and TCP latency measurements were repeated. As was the case with the router, the

FDDI Scenario 2: Subnets Connected Via a Router The two J210 workstations were removed from their shared LAN, and connected to different ports on a Cisco 7513 router. The TCP/IP and UDP/IP latency measurements were then duplicated. As

2

mode, like the router. As a result, their performances, while slightly better than those obtained using routers, still exhibit the latencies consistent with store and forward devices.

III. ATM AS A CARRIER FOR REAL TIME DATA When porting legacy applications to ATM, IP over ATM provides a seamless method of retaining existing TCP/IP code without modification. Additionally, establishing TCP/IP communications via the Berkeley Sockets API is well understood, and using standard socket calls for ATM communications will be attractive to developers of new applications for some time. There are also concerns over the lack of security mechanisms in raw ATM, in which the type of filtering rules used within conventional firewalls will not work [2], [5]. Because of this, it is expected that many

Figure 5: Summary of UDP/IP round trip Latencies between nodes in various topologies. switch added measurable latency to the round trip delay. This process was repeated using FDDI switches from several vendors, yielding similar results in virtually all cases. Figure 4 shows the average measured latencies for TCP and UDP through a typical FDDI switch. FDDI Latency Summary For reference, the TCP/IP latency results obtained from each of the three scenarios described for FDDI are summarized in Figure 5. The relative affects of going through switches and routers can be gauged from this figure. As expected, there is a measurable performance degradation when going through these devices. The latency incurred is not excessive, but on a large internet or intranet, the overall latency will be the summation of the individual latencies appended by each device between applications. The device latency for switches and routers is a consequence of two effects: read-in time and processing time. Because routers make routing decisions based upon the layer three IP header, they must read in the entire packet and look at it’s contents. Routers, which are inherently store and forward devices, will have a relatively high read-intime, as well as a processing delay. For each packet passing through a router, the FDDI MAC address must be physically adjusted to that of the next hop. Switches operate at the Data Link layer, allowing this MAC adjustment to be bypassed. As a result, switch performance would be expected to be slightly better than the equivalent performance through a router. Figure 5 shows that this is the case for the configuration tested. The performance difference is very slight, however, as the FDDI switches tested operate in a store and forward

Figure 6: IP over ATM latency between back to back nodes, as compared to similarly arranged FDDI. applications will tend to use IP over ATM rather than attempting to embrace the use of raw ATM via new, non-standardized APIs, at least in the near term. For this reason, and those specified in Section I, the latency characteristics of IP over ATM are of great interest. A. IP Over ATM Back to Back Workstations Figure 6 shows the round trip latencies for TCP and UDP obtained when running the same experiments as described in Section II between two J210 workstations connected back-to-back with no intermediate ATM switches. The equivalent results for TCP/IP over FDDI on a similar arrangement of HP-J210s (FDDI Scenario 1) are included for

3

reference. The latency of IP data over ATM at very low data sizes is noticeably higher than that of IP over FDDI, but it rapidly converges on the FDDI performance. Though the two curves approach each other for messages beyond roughly 2000 bytes in size, they remain slightly separated, with FDDI providing tens of microseconds lower round trip latencies. These results generalize for UDP as well. Similar experiments were conducted by Du, et. al. [4], though the results differed in that within their environment, TCP/IP over ATM provided far more substantial performance improvements, on the order of several milliseconds. It is believed that this was due to the specific hardware and software components used for their testing at the time. At that time, their components were only able to achieve 16 Mbps throughput on FDDI, while with the J210s sustained throughputs exceeding 80 Mbps are readily obtainable. Given this, it is felt that the results shown in Figure 6 are more representative of the type of performance that can be expected with the faster processors available today.

Figure 7: IP over ATM through an ATM switch, compared with switched FDDI results. improvement at large message sizes. It is interesting to note, though, that for messages smaller than 1000 Bytes, the switched FDDI results are better than those obtained for switched ATM. This is likely a consequence of the small switch read-in time associated with these small messages. B. Raw AAL 3/4 ATM When switches and routers are not used, performance of IP over ATM offers no advantages in end-to-end latencies obtainable via traditional FDDI LANs, and for very small messages IP over ATM latencies are significantly higher. To better realize the potential of ATM, a direct interface to the AAL is required without the intervening network and transport layers of the Internet Protocol suite. This is achievable via the use of APIs such as that developed by Fore Systems, Inc. The Fore API is reminiscent of the Berkeley Socket API, making the transition reasonably painless for new development or modification of existing application source. The ATM Adaptation Layers (AALs) are designed to offer distinct classes of service. AAL 1, for instance, is designed for constant bit rate synchronous data as would be required for smooth video. LAN traffic is generally asynchronous, variable bit rate, or Class C. AAL 3 was designed to handle such traffic on a connection oriented basis, while AAL 4 provided the same service via a connectionless paradigm. Because of their similarities, AAL 3 and 4 were eventually combined into a single AAL, known as AAL 3/4. Because the focus of this paper is on expected performance when migrating legacy LAN applications, only AAL 3/4 is addressed.

IP over ATM via ATM Switch As described in Section II, switching in IP networks allows one to avoid the delays associated with processing in intermediate devices at the Network layer, as required in routers. This benefit holds true for IP over ATM, and is accentuated by the fact that ATM supports only 53 byte cells. IP traffic can come in arbitrarily sized packets, up to the MTU of the LAN being used. Thus routers and IP switches must be capable of dealing with this size ambiguity. ATM switches, on the other hand, can be optimized for the fixed 53 byte cells ATM uses. Thus ATM switches should be expected to outperform IP routers and even IP switches. To test whether the improved performance of ATM switches can compensate for the penalty paid at the workstation to segment IP packets into ATM cells, a simple configuration was created with two HP-J210 workstations each connected to different ports on a Fore Systems ATM switch. Round trip latencies of UDP/IP and TCP/IP over ATM with an intermediate switch could then be measured, as shown in Figure 7. For reference, TCP/IP data from FDDI Scenario 3 is included to demonstrate the differences of IP through a FDDI switch versus IP over ATM through an ATM switch. As can be seen, latencies through the Fore Systems ATM switch are indeed lower than those through a typical FDDI switch for messages over roughly 1000 bytes, providing substantial performance

Raw AAL 3/4 Between Back to Back Nodes

4

contrast with the much larger latencies introduced by FDDI switches and routers, which must accommodate packets of indeterminate size. Using raw AAL 3/4 ATM through ATM switches, which are optimized for the 53 byte ATM cells, clearly offers tremendous latency advantages over equivalent IP internetwork devices. As had been noted previously, however, in order to take advantage of this, applications must be modified to use an appropriate ATM API such as the Fore ATM API. Migrating legacy code may not be a trivial matter, depending on the structure of the application source and the mechanisms employed in the specific API used. Nevertheless, for certain applications which are extremely time sensitive, this cycle may be justified in light of the performance results shown in Figures 8 and 9.

Figure 8: Raw ATM at AAL 3/4 between back-toback HP-J210 workstations.

IV. TRADEOFFS AND DESIGN CONSDERATIONS

The HP-J210s were again arranged in a back-toback ATM configuration and round trip latencies using ATM AAL 3/4 were measured between them. This data is presented in Figure 8, along with the equivalent classical UDP/IP over ATM results, for reference. Here the advantages of ATM begin to make themselves known, as there is a significant decrease in latency obtained by using raw ATM via AAL 3/4.

When designing a new LAN, or considering upgrading existing LAN assets to ATM, several tradeoffs should be considered. FDDI still retains substantial benefits for high speed, fault tolerant networks. Several issues to consider before migrating to ATM as opposed to FDDI as the optical network of choice are summarized below. As mentioned previously, a single predominant API has yet to emerge for directly accessing ATM AALs. While there are available APIs such as the Fore Systems ATM API, none dominate, and compatibility and portability of code designed specifically for raw ATM should be a concern for some time. The developer may bypass this by using better established APIs such as BSD sockets or even TLI [9], but in this situation it is not possible to directly access the ATM AALs. Instead, IP over ATM is being used. It was shown in Section III that IP over ATM may provide some performance benefits over FDDI, though these benefits tend to be far less than those which are often expected from ATM. Network security is another area where raw ATM fails to provide any benefits over IP networks, and in fact security in ATM networks remains a difficult problem that has yet to be adequately addressed. ATM cannot make use of the types of filtering rules which comprise the core of IP firewall security. Thus to retain the functionality of firewalls, one must run IP over ATM, and often must first convert to a more traditional IP media such as Ethernet or FDDI in order to accommodate the firewall architecture. A possible workaround for this is to encrypt the ATM

Raw AAL 3/4 Through an ATM Switch The J210s were once again connected to different ports on the Fore Systems ATM switch, and the raw ATM latency measurements were again conducted. The results are shown in Figure 9, along with the results obtained for back-to-back Raw AAL 3/4 ATM. Note here that the ATM switch introduces almost no latency when inserted into the communications path. This is in direct

Figure 9: Raw ATM at AAL 3/4 between HP-J210 workstations via an ATM switch.

5

cell payload, though such a solution comes with it’s own performance implications [2], [5]. At present, FDDI presents a far more fault tolerant architecture than ATM. In addition to the typical dual attached ring, which is itself able to withstand at least one fault with no degradation in communications, there are configurations possible which can tolerate up to three faults before a LAN wrap occurs. In these configurations, four faults are required before LAN communications are disrupted. By any standard, FDDI must be considered extremely fault tolerant [6], [8]. ATM, on the other hand, fails to effectively handle faults to end stations. A cable fault will immediately sever communications to/from the effected node. It may be possible to dual home nodes to a pair of switches, resulting in enhanced survivability, but at current no vendor provides a mechanism to allow this transparently to the end users. Inter-switch fault tolerance is quite good in ATM, however. FDDI specifications have been formalized by ANSI. They are not subject to capricious change, and FDDI is a mature, well known technology. ATM specifications are still evolving. As a result, vendors often incorporate advanced features which have not been incorporated as ATM standards by the ATM Forum. As these features are not standardized, vendor implementations tend to be incompatible. Thus it can be very difficult to develop an ATM environment using disparate ATM devices. It is with respect to the issue of LAN growth that ATM truly shines, however. FDDI standards are well defined, fixing available bandwidth at 100 Mbps. While this type of standardization makes interoperability simple, it does limit the capacity of the technology to grow. ATM was designed to scale to Gigabit speeds, assuming an appropriate cable plant and switch capacity. Furthermore, facilities for tailoring the quality of service for each ATM connection promise to allow a guaranteed bandwidth for specific applications, an important consideration for real-time systems. FDDI, on the other hand, has no such mechanism. Instead, access to the media is arbitrated via FDDI physical layer restrictions (token holding times, etc.) and the operating system in the computer. No allowance for urgent data is made.

ATM or a direct interface to the ATM adaptation layers. For the simplest connections of back-toback computers, FDDI greatly outperforms IP over ATM for very small messages (< 2000 bytes) and marginally outperforms IP over ATM for messages larger than 2000 bytes. However, it has been shown that IP over ATM, in an internetworked environment using switches, demonstrates significantly lower round trip latencies than similarly arranged FDDI internets for all but the smallest messages. This is believed to be a consequence of the ability to optimize ATM switches for a fixed cell size as opposed to the relatively arbitrary IP packet lengths encountered with FDDI. Using raw AAL 3/4 ATM does allow one to gain sizable performance increases for asynchronous, variable bit rate traffic. This is primarily the result of bypassing the costly TCP/IP protocol stack at the workstations, and of using switches optimized for passing the fixed sized ATM cells. In a larger internetwork, the extremely low switch latencies encountered should be expected to greatly reduce end to end latencies across the cloud, as opposed to using traditional routers or even IP switches. REFERENCES [1] U. Black, ATM: Foundation For Broadband Networks, Prentice Hall, Englewood Cliffs NJ, 1995. [2] J. Cappell, and D. Deeth, ATM Encryption Testing, [3] S. Dempsey, W. Reilly, M. Teter, L. Weinberg, Predicting FDDI Performance Using a Calibrated Software Model, IEEE 1997 International Performance, Computing, and Communications Conference, Tempe, AZ. February 5-7, 1997. [4] D. H. C. Du, M. J. Lin, et. al., Distributed Computing in ATM Networks, IEEE Journal on Selected Areas in Communications, Vol. 13, No. 4, pp. 733-748, May 1995. [5] G. M. Haskins, C. Paar and S. Dempsey, Securing ATM, presentation at 1997 RSA Data Security Conference, San Francisco, CA. January 31, 1997.

IV. CONCLUSIONS When porting legacy LAN applications to ATM or when developing new applications for an ATM environment, two approaches may be used: IP over

[6] S. Hiles and D. Marlow, Approaches for Survivability in FDDI Networks, IEEE

6

Computer Society 17th Conference on Local Computer Networks, September 13-16, 1992 Minneapolis, MN. [7] T. Mafarachisi, J. French, et. al, The AEGIS LAN Interconnect System: Providing a Reliable Infrastructure for Mission Critical Computing in Surface Combatants, Proceedings of The American Society of Naval Engineers Intelligent Ship Symposium II, November 25-26, Philadelphia, PA. [8] S. F. Ralph, R. H. Schellack, O. J. Ukrainsky, and L. Weinberg, Alternate Path FDDI Topology, IEEE Computer Society 17th Conference on Local Computer Networks, September 13-16, 1992 Minneapolis, MN. [9] W. R. Stevens, UNIX Network Programming, Prentice Hall P. T. R., Englewood Cliffs, NJ, 1990.

.

7