ATM Performance Measurement1 Yves A. Fouquet,2 Richard D. Schneeman, David E. Cypher, and Alan Mink Scalable Parallel Systems Group (Bldg 223, Rm B364) National Institute of Standards and Technology Gaithersburg, Maryland 20899 USA T´el. : +1 (301) 975-5681 Fax : +1 (301) 869-7429 E-mail :
[email protected]
R E´ SUM E´ . ATM est souvent consid´er´e comme la solution aux r´eseaux a` bande passante limit´ee. Il est donc n´ecessaire de d´eterminer la part de bande passante effectivement disponible au niveau de l’application, plutˆot que la bande passante brute du r´eseau. Afin de d´eterminer la bande passante d’ATM entre applications, les auteurs ont e´ tudi´e les performances d’un r´eseau local ATM bas´e sur du mat´eriel Fore Systems. Ceci a e´ t´e r´ealis´e en simulant a` la fois les transferts m´emoire et disque a` travers TCP/IP/ATM et l’API et protocole ATM de chez Fore Systems. Des variations sur des param`etres tels que le type de l’AAL, la taille de l’unit´e maximale de transmission et la configuration du r´eseau, nous ont permis de d´eterminer leurs effets respectifs sur la bande passante disponible a` l’application, ainsi que sur le taux de perte de cellules et les goults d’´etranglement (bottlenecks). A BSTRACT. ATM is seen by many users today as a solution to networks with bandwidth limitation. Therefore, it is necessary to determine the amount of communication bandwidth actually available to applications rather than raw network bandwidth. In order to determine ATM’s bandwidth between applications, we studied the performance of an ATM LAN based on Fore Systems’ hardware. This was achieved by emulating both memory-based and disk-based transfer over TCP/IP/ATM and the Fore Systems’ API/ATM protocols. Varying parameters such as the type of AAL, the size of the maximum transmission unit and the network configuration, allowed us to determine their effect on application bandwidth as well as cell loss and bottlenecks.
M OTS CL E´ S : ATM, TCP/IP, bande passante, protocoles de communication, mesure de performances, d´ebit. K EYWORDS : asynchronous transfer mode, ATM, TCP/IP, bandwidth, communication protocols, performance measurement, throughput
1 Introduction Asynchronous Transfer Mode (ATM) is rapidly advancing in the marketplace as one of the ways of achieving gigabit networking performance, touting high throughput and performance characteristics. In considering the possibilities of using emerging high-speed networks, the implications of integrating users’ current systems, software, and legacy networks with ATM need to be carefully examined. Currently, information concerning ATM bandwidth is typically reported as being a function of hardware bandwidth. Indeed, raw ATM cell-switching hardware is now extremely efficient; however the bandwidth offered to applications still varies with current internetworking mechanisms, system software, legacy protocols and networks, and existing end-system limitations. In order to determine ATM’s bandwidth between applications, National Institute of Standards and Technology (NIST) researchers studied the performances of an ATM local area network (LAN) based on the Fore Systems’ hardware and software environment. This was achieved by using a variety of data transfer techniques and benchmarks over the Fore Systems’ API/ATM and TCP/IP/ATM, in order to emulate both memory-based and disk-based transfer applications. Varying parameters such as the type of AAL, the size of the maximum transmission unit (MTU) and the network configuration, allowed us to determine their effect on application bandwidth as well as cell loss and bottlenecks.
2 Asynchronous Transfer Mode Network Overview An ATM network is an implementation of fast packet switching using fixed-length packets, called cells (53 bytes) [1] [4] [13] [16]. ATM network protocols use a layered structure that roughly corresponds to the OSI model’s physical layer and a portion of the data link layer. There are three distinct layers in the ATM protocol structure: a physical layer, the ATM layer, and an ATM adaptation layer (AAL). We briefly discuss the significance of each layer. 1 This National Institute of Standards and Technology contribution is not subject to copyright in the United States. Certain commercial equipment, instruments, or materials may be identified in this paper to adequately specify experimental procedures. Such identification does not imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that materials or equipment identified are necessarily the best available for the purpose. 2 Yves A. Fouquet is currently a Network engineer with EDF-DER, 1 Avenuedu General de Gaulle, 92141 Clamart, France,
[email protected]
2.1 Physical Layer The physical layer manages the transmission of ATM cells between communicating ATM entities. Synchronous Optical NETwork (SONET), DS3, and FDDI are examples of physical layer protocols. In our testbed, we used a 100 Mb/s optical fiber line module that conforms to the FDDI physical layer interface standard [15]. Physical layer synchronization on a 100 Mb/s FDDI-based optical fiber incurs approximately 2 Mb/s overhead, leaving usable bandwidth of 98 Mb/s [9].
2.2 ATM Layer The ATM layer defines the cell’s size and structure. A cell is composed of a 5byte header and a 48-byte information payload. Two of the fields in the cell header are a virtual circuit identifier (VCI) and a virtual path identifier (VPI). These two fields concatenated form a complete virtual connection identifier. These two fields are key in providing the routing, multiplexing, and demultiplexing of cells belonging to different network connections.
2.3 ATM Adaptation Layer The ATM Adaptation Layer (AAL) segments the upper layer data, such as datagrams, voice samples, and video frames, into ATM cells which can be sent via the ATM layer and then reassembled at the receiving end. In order to support these various data services, the following four service classes have been defined [10].
Class A: supports circuit emulation; constant bit-rate applications. Class B: supports variable bit-rate video and audio. Class C: supports connection-oriented data transfer (e.g., X.25 protocol). Class D: supports connectionless data transfer (e.g., IP protocol). There are currently three AALs defined to support a subset of these classes. AAL 1 supports class A. AAL 3/4 and AAL 5 both support classes C and D. AAL5 was developed to minimize the overhead of AAL 3/4. AAL5 calculates a 32 bit CRC on the whole packet as it is done by every IEEE.802.x technique and expected from IP, while AAL 3/4 provides a 10 bit CRC on each ATM cell. Another fundamental difference between the two AALs is the amount of data per cell; 48 bytes for AAL5 and 44 bytes for AAL3/4. Thus AAL5 requires fewer ATM cells than AAL3/4 to transmit the same amount of data. AAL types 1, 3/4, 5, and null were defined by ITU-T recommendations and Committee T1 standards, but some vendors have chosen not to implement all of them. In our case, only AAL 5 and AAL 3/4 were available.
3 NIST’s ATM Testbed NIST maintains an in-house ATM Testbed that is used to perform measurement studies and conduct experiments over a variety of networking substrates and technologies. In the following subsections, we discuss the configuration of our testbed.
3.1 ATM Testbed Network Configurations For our measurement environment illustrated in Figure 1, we use two Sun Sparc10 model 30 workstations, a Fore Systems’ ForeRunner ASX-100 ATM switch, and two SBA-200 Sbus adapter boards, also from Fore Systems. The two Sun workstations are connected to the switch using an optical fiber which provides the 100 Mb/s TAXI interface (FDDI fiber and signal encoding scheme) [15]. We performed experiments both with the ATM switch and without it in a back to back arrangement. Because the ATM switch imparts no measurable delay in our configuration, the majority of our experiments were done using the ATM switch. The workstations are rated at 86.1 Million Instructions Per Second (MIPS) and are running the SunOS 4.1.3 (Solaris 1.0) operating system. The only modifications to these standard workstations were to disable other login sessions during the testing period and to incorporate the ATM hardware and software. Standard UNIX daemons, processes, and network interfaces were allowed to execute normally.
3.2 Fore Systems’ Hardware and Software 3.2.1
ASX-100 ATM Switch
The Fore Systems’ ForeRunner ASX-100 [5] is a 2.5 gigabits per second (Gb/s) busbased ATM switch with an internal sparc-based microprocessor switch controller. It has four network line interface modules (LIMs), each supporting four full-duplex 155 Megabits per second (Mb/s) ATM interfaces per module. Two DS3 interfaces (44.736 Mb/s) provide connectivity to external networks. 3.2.2
SBA-200 Sbus Adapter
The ForeRunner SBA-200 ATM SBus adapter [7] is the second generation ATM interface card from Fore Systems. Unlike the first generation SBA-100 [6] which used software for the AAL layer processing, the SBA-200 is equipped with AAL 3/4 and AAL 5 hardware. The SBA-200 includes an embedded Intel i960 RISC processor and uses DMA (Direct Memory Access) for data transfer to/from the computer memory for both send and receive data. It also has two onboard FIFOs, a 16 Kbyte receiving FIFO and a one Kbyte sending FIFO. For our configuration, no flow control mechanisms exist either at the switch or the workstation adapter, nor is there any error detection or guaranteed sequencing. Sequencing for our configuration was guaranteed only because of our single dedicated path between source and destination. Implementation of any of these functions is the responsibility of the end-system’s application or higher layer protocol. 3.2.3
Application Programming Interface (API)
The Fore Systems’ ATM library routines provide two alternative paths to the ATM hardware as illustrated in Figure 2. Either traditional TCP/IP based sockets can be used over ATM or the Fore Systems’ API can be used to interface to ATM directly using a data link layer interface. Using the API, a client application would first call
atm-open() in order to acquire a file descriptor, and then use atm-bind() to bind a local Application Service Access Point (ASAP) to this file descriptor. An ASAP consists of an ATM switch identifier and a port number of the switch. Once an ASAP is bound, a connection is established using atm-connect() on the client side and bracketed by an atm-listen() and an atm-accept() on the server side. It is through an argument passed to atm-connect(), that an AAL is chosen. After a connection is established an atm-send() from the client and a subsequent atm-recv() from the server provide the transfer of one Protocol Data Unit (PDU) at a time. The Fore Systems’ API presents a socket layer style abstraction for writing applications; however, the application interaction actually occurs at the data link layer. This makes the programming paradigm confusing because transport layer semantics are implied with data link layer primitives resulting. This phenomenon also makes it very difficult to port any useful socket-layer applications (such as those found in the TCP/IP socket-based domain) to the Fore systems’ API. The ATM standard for AAL5 defines the Maximum Transmission Unit (MTU) size to be in the range of 1 to 65,535 bytes. The standard also provides an MTU negotiation procedure in case an ATM connection is made between systems that have different MTU sizes. We encountered a limit of 4 Kbytes on the MTU, imposed by the API in transfering data segments from application space down into the device driver. This restriction has been placed on the API because of a SunOS limit on the maximum amount of data that can be pushed down into the streams-based device driver interface to the adapter. A potential problem exists when the value of the MTU exceeds this limit on another communicating machine on which negotiating features are not implemented. ATM standards allow multiplexingof connections only over AAL3/4, not over AAL5. This could not be tested because multiplexing was not available.
4 Objectives and Related Work Our experiments focused on determining the communication bandwidth available at the application layer. We realize that most workstations cannot generate a data stream fast enough to utilize the available hardware bandwidth of high speed networks. Therefore we establish the performance of the workstation architecture to generate a data stream independent of communication networks and protocols. We then determined the communication bandwidth achievable with different protocol stacks and benchmark programs, relative to this architectural upperlimit bandwidth. Two optimized programs, further refered to as ”disk copy” and ”mem copy”, were written to define, respectively, the architectural upper limit of disk to memory copy speed, and memory to memory copy speed on a single machine. Two client-server based programs were developed, one to simulate memory-based data transfer, and the other to simulate diskbased data transfer between two machines over a network. We used these programs along with a public domain benchmark, NetPerf [8], to determine the available application communication bandwidth with our testbed over the Fore Systems’API/ATM and TCP/IP/ATM. Also we wanted to determine the potential for cell loss when using the API interface, which has no flow control. During the course of our work we discovered other related work. Moldeklev [14]
and Luckenbach [9] are both using a testbed similar to NIST’s. Moldeklev used a Fore Systems’ ATM ASX-100 switch, Sun Sparc10 clones and Fore Systems’ Sbus adaptor boards. Using ttcp3, a benchmark very similar to NetPerf which we used, they showed that with a reliable protocol such as TCP/IP over ATM, a maximum of 62 Mb/s was the highest sustainable rate for their configuration. They also showed the effect of data packet sizes and window sizes [3] on such transfer rates. Luckenbach also used a Fore Systems’ ASX-100 ATM switch, Sun Sparc10 workstations, and both the Fore Systems’ SBA-100 and SBA-200 Sbus adapter boards. They highlighted the saw tooth phenomenon, measured delays caused by cell segmentation and reassembly, and used ttcp to reach an average throughput of 60 Mb/s using TCP/IP over ATM, similar to that of Moldeklev. Mengjou [11] compared the communication performance of four different APIs: the Fore Systems’ ATM API, the Berkeley Software Distribution (BSD) Socket Programming Interface (with TCP/IP), the Sun Remote Procedure Call (RPC), and the Parallel Virtual Machine (PVM) message passing library. They evaluated two distributed applications over the various APIs; a parallel matrix multiplication and a parallel differential equation solver. Their results showed that the best performance was obtained using the Fore Systems’ ATM API, even though the MTU was limited to 4096 bytes as in our experiments.
5 Measurements Programs For our experiments we have developed two programs to determine the architectural upper limit of bandwidth for our SUNs, used an existing public domain benchmarking package, and developed two network benchmark programs that provide a more realistic sustainable measure of network throughput than the public domain package.
5.1 Architectural Limits Architectural limits of workstations, such as processor speed, bus bandwidth, and memory bandwidth, prevent most workstations from effectively using the bandwidth of high speed networks. We have developed the following two programs to determine the effective data movement rate of our workstations. These programs only moved data within a single machine, no networks were involved. These programs were written in assembly language and moved 32 bits of data (vs. 8 bits) per transfer. ”mem copy” This program consists of a loop where a fixed size array is copied to another array, both arrays are in primary memory on a single machine. The throughput is calculated by dividing the total amount of data transfered by the elapsed execution time of the loop. ”disk copy” This program, very similar to the previous one, moves a fixed size array stored on disk to an array in primary memory on the same machine. 3 This benchmark program was developed at the US Army Ballistics Research LAB, and is in the public domain. It is available from the following ftp site: wolf.brl.mil:pub/ttcp.
5.2 Public Domain Network Performance Software: NetPerf NetPerf [8] is a public domain benchmarking package developed by Hewlett-Packard and structured around a client-server model 4 . NetPerf can use either the Fore Systems’ API/ATM or the TCP/IP/ATM stacks. NetPerf provides the capability to vary the MTU size of the data packet being sent to the AAL layer from the Fore Systems’ API/ATM. One metric available from NetPerf is both the sending and receiving throughput over an ATM communication medium. An input to NetPerf allows the user to select the amount of time the data transfer will last over the network. The resulting throughput is calculated by dividing the amount of data transfered by the elapsed transmission time. To execute NetPerf, the user specifies the size of the desired memory array for the data to transmit (default: socket buffer size or MTU) and the length of time to transmit that data (default: 10 sec). NetPerf then continually retransmits the array until the transmission time elapses. This technique avoids dealing with disk access or large memory requirements and its performance may benefit from the caching of the memory array, thus yielding a higher throughput than a full memory based transfer. Because of these simplifications, we used NetPerf to obtain an optimistic value of communications throughput.
5.3 NIST’s Network Measurement Programs Our measurement programs were developed to provide more realistic sustainable throughput measurements than NetPerf from memory and disk over the Fore Systems’ API/ATM, using either AAL3/4 or AAL5. Memory-based Data Transfers The client copies the entire contents of a disk file into memory and then transfers this memory image in fixed size increments to the ATM Adaptation Layer using the Fore Systems’ API/ATM. Figure 3 illustrates the structure of our memory-based client and server code. File-based Data Transfer Similar to the memory-based data transfer, but in this case the client reads the data from the disk, a packet at a time, and then sends that packet to the ATM Adaptation Layer via the API. For both memory-based and file-based data transfer, the client initiates a connection with the server, which is waiting to accept it (iterative server model). Upon establishing connection with the server, the client sends its data and then proceeds to tear down the connection, closing the call. The server (identical for both memory and file based transfer) receives and copies the data into memory. The data is typically discarded; however, it can optionally be saved to a file. The total time to transfer the data between the client and server programs is calculated by using the standard UNIX gettimeofday() routine. The precision of the time returned by this routine is 10 milliseconds on our workstations. Just prior to sending 4 The software is available from the following ftp site: dist/networking/benchmarks.
ftp.cup.hp.com in the directory
data over the opened ATM connection, a time-stamp is acquired. After completion of the transfer, another time-stamp is taken. The difference between the two timestamps divided by the amount of data transferred provides an average value of the sending throughput at the application level. Since the duration of each of our data transfer experiments is several seconds, the 10 millisecond precision and the overhead of the two system calls to gettimeofday() do not significantly affect the results of our measurements. To determine cell loss, we used the Fore Systems’ atmstat library program, which allows statistics to be gathered and displayed on each participating machine. Statistics include the amount of cells transferred, received, or lost. The statistics can be a total for all protocol layers or focused on the physical layer, ATM Layer, AAL layer, or the device driver. The atmstat software accesses and uses the UNIX ioctl() system call to the ATM Sbus adapter device driver. We also used ioctl() in a similar fashion to gather statistics at the beginning and ending points of the data transfer for the NIST network measurement programs. Gathering such lower-level cell statistics helps to understand more clearly what is occurring within the ATM network as these transfers take place.
6 Varying Parameters and Performance Measurements The measurement data captured from these experiments were gathered over 20 consecutive executions and all the curves resulted from an average of these measurements. The large number of runs provides a smoothing effect on the throughput data points plotted. However this effect is very low since raw data showed a very small variance (less than 0.5 Mb/s). Throughput measurements were gather from the sender, while cell loss was measured on the receiver. Our intentionthroughout these experiments is to measure effective throughput between communicating application programs using both API/ATM and TCP/IP/ATM protocol stacks. We start by establishing an architectural limit on throughput by using our ”mem copy” and ”disk copy” programs. After establishing these upper limits we measured the performance of NetPerf and our network measurement programs, to determine how close they could come to these upper limits. While conducting these experiment, we varied a number of parameters and measured thier effect on throughput. Our results are summarized in Figures 4 and 6.
6.1 Architecture Limits Using ”mem copy” and ”disk copy” we measured the architectural limit for memory and disk based data transfer on our configuration for 5 Mbytes of data. Other than for very small amounts of data, the transfer rate was determined to be insensitive to the amount of data. These measurements represent the fastest rate that data can be moved within primary memory, and between disk and primary memory on a single machine. Moving data between different machines requires the additional overhead of a network and its associated protocols. The maximum transfer rate obtained from ”mem copy” is 75 Mb/s (see upper curve in Figure 4), and is 30 Mb/s for ”disk copy” (see upper curve in Figure 6).
6.2 Transmission Size The amount of data to transmit was used as a parameter in determining the sustainable throughput. Transmission size ranged from 1 Mbyte to 15 Mbytes. The various tests, of all the network measurement programs, showed that varying the transmission size did not have any measurable influence on the resulting throughput as can be seen in Figures 4 and 6, where the curves for 1, 5, and 10 Mbytes essentially form a single curve. Although we measured two exceptions, further investigation indicated that thier causes were due to other factors. When these factors were corrected invariance of this parameter was verified. In the case of the memory based transfers, the 15 Mbyte size introduced additional delays due to virtual memory management overhead (disk). We did not have enough main memory to hold 15 Mbytes and the overflow data used disk swap space. A larger primary memory resulted in expected performance. When transmitting the 1 Mbyte file using the file-based data transfer, we made sure that the data was read from the disk and not from the disk cache. If we did not preclude the use of the disk cache when transmitting a 1 Mbyte file, the throughput results would falsely show better performance.
6.3 Buffer Size Experiments were run with a buffer size (amount of the data passed, per call, from the application to the protocol stack) varying from 40 to 4096 bytes because the MTU is limited to 4096 in the API (this value was imposed by the SunOS STREAMS). Multiples of 512 bytes were chosen, as well as multiples of (n * 48 + 40) and (n * 48 + 40 + 1), with n being the number of full ATM cells varying from 0 to 84. At the AAL5 layer, an 8 byte trailer is added to the data and, if necessary, enough padding to make it an integral multiple of 48. Next the data is segmented into 48 byte ATM payloads and placed into ATM cells. Following this procedure, the least amount of overhead is incurred when no padding is needed. This implies that the data arriving at the AAL5 layer is a multiple of 48 bytes + 40 bytes. Therefore, all ATM cells except the last cell will contain 48 bytes of data. The last cell will contain 40 bytes of data and 8 bytes of trailer information. The worst case is when one more data byte is added resulting in an extra ATM cell being transmitted. As shown in Figures 4 and 6, our measured throughput for all the network programs increased asymptotically with the buffer size. This result has already been shown with TCP and UDP over IP in various other studies [9] [14], and is still true when using the Fore Systems’ API/ATM. This shows how important it is to choose an optimum buffer size, since a one byte difference may yield as much as a 10% degradation in the effective throughput. The saw tooth shape of the performance curves shows that local performance maximums (peaks) occur when the buffer size is an exact multiple of ATM cells, while local performance minimums (valleys) occur when that buffer size is increased by one byte, yielding as much as a 10% difference in performance. The actual shape of the NetPerf curve in Figure 4 also exhibited a saw tooth shape, but for illustration purposes we only plotted the upper envelope of the curve. The application buffer size influences the overall time spent in the protocol stack due to operating sys-
tems and protocol manipulations of the data. Some of the additional work required can be absorbed by operations done mostly in hardware on the ATM interface board and by unused network capacity for additional cells.
6.4 TCP/IP vs. API In both memory and file-based transfer programs we have run experiments using TCP/IP/ATM with a default 8Kbyte window, instead of the Fore Systems’ API/ATM, in order to quantify the difference between this API and a reliable protocol (see Figures 4 and 6). The results show that TCP/IP/ATM was up to 40 Mb/s slower than API/ATM for a memory-based data transfer and up to 10 Mb/s slower than API/ATM for a filebased data transfer. This was mostly due to the overhead caused by TCP layer routines for functions such as checksumming and data copying (see [12]) as well as delays due to flow control and acknowledgements.
6.5 AAL Type Since AAL5 was developed to be more efficient than AAL 3/4 [16], we wanted to measure the difference in throughput between these two. Experiments revealed no measurable difference in throughput for the two ATM Adaptation layers, even though we measured more cells transmitted with AAL3/4 than with AAL5 (AAL5 puts 48 bytes of data per cell while AAL3/4 only puts 44 bytes). We assume that the time difference to process and send the cells using AAL5 or AAL3/4 was actually absorbed by the faster underlying structure (the network and the ATM interface board where the AAL processing is done mostly in hardware), since no measurable difference is seen at the application level. This differs somewhat from the ”Buffer Size” experiments, where different amounts of data are passed between the application and the ATM interface board. In this case the same amount of data is transfered between the application and the ATM interface boards, independent of the AAL.
6.6 Switch Influence We found that the switch does not have any effect on the throughput, since the results are statistically the same with and without it. This was determined by running our ”Buffer Size” experiment with the switch connecting source and destination machines and rerunning it again with the switch bypassed, that is just a direct fiber connection between source and destination machines. The Fore Systems’ ASX-100 aggregate cell-switching capacity is rated at 2.5 Gb/s when supporting its maximum 16 ATM connections each running at up to 155 Mb/s. Due to such high capacity switching, compared to our architectural limits, we were unsuccessful in saturating even a single incoming link of the cell-switch.
6.7 Cell Loss Since receiving takes longer than sending, as shown by [14] [9], continuous transmission between equivalent processors can overwhelm the receiver [2] unless the underlying network and its protocols introduce delays either through flow control or reduced bandwidth. All of our experiments showed no cell loss in the switch or the client. The switch, as expected, is fast enough to handle the traffic from the client, and
the sending side most likely has ”built-in” flow control between the device driver and the ATM board. Because of the extra disk access delays incurred in the client by the file-based transfer, no cell loss occurred. However, values up to 39% were measured with the memorybased data transfer (Figure 7). Incoming cells are lost at the ATM Adaptation layer. Hand optimization of the application code, which resulted in speeding-up the receiving code, leads to no cell loss. Thus this cell loss is due to the extra processing time required by the receiver. This extra time is distributed among the ATM board, its associated driver and the application program. But the amount of extra processing time at these different locations are not known. By speeding-up the application receiving program, we were able to reduce the extra receiving time below the threshold which causes cell loss. Obviously for any real application this approach of optimizing code would be insufficient to eliminate cell loss. Thus some forms of flow control or rate limiter is required.
7 Conclusions We studied the application level throughput performance on a local area ATM network using commercially available ATM products from Fore Systems. Performance measurements were obtained using programs developed in-house, plus a public domain network benchmark program called NetPerf. We determined that communications throughput was insensitive to the amount of data to transmit. Although we also measured insensivity to AAL5 vs. AAL3/4 and the use of an ATM switch, we fell this was due to the fact that our machines were not fast enough to load the underlying network so that these parameters could be stressed. If faster machines were used, one would expect AAL5 to yield better performance than AAL3/4, but still no measureable effect from the switch. We were able to highlight throughput increasing asymptotically with the amount of data passed to the protocol stack by the application, and following a saw tooth structure curve. Our measurements also showed that the asymptotic throughput of TCP/IP/ATM was much lower than API/ATM, although our previous studies indicate this is due largely to the implementation of the protocol rather than the protocol itself. Although the available bandwidth was 100 Mb/s (limited by the output of the ATM interface board), it was not possible for our SUNs to move data that quickly. Our measurements showed the architecture limits of our machines to be about 75 Mb/s for internal data movement. Therefore relative to the architecture limits of our machines, a maximum measured application bandwidth of about 62 Mb/s is approximately 83% of that limit. In contrast a measured throughput of 21 Mb/s representing 28% of the architecture limit was realized when an inefficient combination of network/protocol parameters are applied (TCP/IP/ATM with an 8 Kbyte window). For disk-based data transfer, we measured an architectural limit of about 29 Mb/s. A maximum measured throughput of about 25 Mb/s is approximately 86% of that architecture limit, while an inefficient combination of parameters achieved only 15 Mb/s, approximately 52% of that limit. Because it has not been standardized yet, Quality of Service (QoS) has not been im-
plemented, nor has multiplexing of AAL connections. Therefore, measuring the effect of bandwidth reservation, multiplexed connection, and various other tuning optimizations on throughput could not be conducted.
8 Glossary AAL: ATM Adaptation Layer API: Application Program Interface ASAP: Application Service Access Point ATM: Asynchronous Transfer Mode BSD: Berkeley Software Distribution CRC: Cyclic Redundancy Check DMA: Direct Memory Access FDDI: Fiber Distributed Data Interface Gb/s: Giga bits per second IP: Internet Protocol ITU: International Telecommunications Union LAN: Local Area Network Mb/s: Mega bits per second MIPS: Mega Instructions Per Second MTU: Maximum Transfer Unit NIST: National Institute of Standards and Technology OSI: Open Systems Interconnection PDU: Protocol Data Unit PVM: Parallel Virtual Machine RISC: Reduced Instruction Set Computer RPC: Remote Procedure Call SONET: Synchronous Optical NETwork TAXI: Transparent Asynchronous X(transmitter/receiver) Interface TCP: Transmission Control Protocol VCI: Virtual Channel Identifier VPI: Virtual Path Identifier
References [1] The ATM Forum, ATM User-Network Interface Specification, PTR Prentice Hall, Version 3.0. 1993. [2] Brustoloni, J., Bershad, B., Simple Protocol Processing for High-Bandwidth Low-Latency Networking, CMU-CS-93-132., March 1992. [3] Comer, D., E., Internetworking with TCP/IP, Principles, Protocols, and Architectures, Volume 1, Prentice Hall, 1991. [4] Cypher, D., Wakid, S., Standardization for ATM and Related B-ISDN Technologies, StandardView, Vol. 1, No. 1, September 1993, pp. 40-47.
[5] Fore Systems, Inc., ForeRunner ASX-100 ATM Switch - Documentation Overview, 1994. [6] Fore Systems, Inc., SBA-100 SBus ATM Computer Interface - User’s Manual, 1994. [7] Fore Systems, Inc., 200-Series ATM Adapter - Design and Architecture, January 1994. [8] Hewlett Packard., NetPerf: A Network Performance Benchmark, Revision 1.9alpha, Information Networks Division, HP. August 1, 1994. [9] Luckenbach, T., Ruppelt, R., Schulz, F., Performance Experiments within Local ATM Networks, GMD-FOKUS, Berlin, Germany. 1994. [10] ITU-T I.362 BISDN ATM AdaptationLayer (AAL), Functional Description, 1992. [11] Mengjou, L., Hsiehn J., Du, H., C., Thomas, J., P., MacDonald., J., A., Distributed Network Computing over Local ATM Networks, Computer Science Department University of Minnesota, Computer Science Department Technical Report, TR-94-17. [12] Mink, A., Fouquet, Y. A., Wakid, S., Hardware Measurement Techniques for High-Speed Networks, Journal of High Speed Networks, Vol. 3, No. 2, 1994, pp. 187-207. [13] Minoli,D., Vitella, M., ATM and Cell Relay Service for Corporate Environments, McGraw-Hill Series on Computer Communications, 1994. [14] Moldeklev, K., Klovning, E., Kure, O., TCP/IP Behavior in a High-Speed Local ATM Network Environment, 19th Conference on Local Computer Networks, Mineapolis MN, 1994, pp. 176-185. [15] ISO DIS 9314-3, FDDI PMD, 1993. [16] Partridge, C., Gigabit Networking, Addison-Wesley Professional Series, 1994.
SUN SPARC 10 86.1 MIPS
FDDI/TAXI INTERFACE OPTICAL FIBER 100 Mbps
SBA 200
OPTICAL FIBER 100 Mbps
SBA 200
FDDI/TAXI INTERFACE
SUN SPARC 10 86.1 MIPS
FIBER BYPASSING SWITCH FOR BACK-TO-BACK TESTS
ASX-100 ATM SWITCH
Figure 1: NIST ATM Testbed.
APPLICATION SOCKET INTERFACE UDP
ATM-API INTERFACE
TCP ATM-API
IP
ATM SIGNALLING
DEVICE DRIVER
AAL ATM
Figure 2: Fore Software Environment.
Client
Server
OPEN (file) COPY (file to memory) CLOSE (file) CONNECT (server) gettimeofday () WHILE (data_sent < size_of_file) { ATM_SEND( dev, buffer, size ) } gettimeofday () DISCONNECT (server)
ACCEPT (client connection) WHILE (anything to read) { ATM_RECV( dev, buffer, size ) } CLOSE (client)
Figure 3: Memory-based Data Transfer.
Transfer Through ATM Network - AAL5 (Memory-based Data Transfer ) 80 Internal Memory Copy 70 Netperf 60
Throughput (Mbit/s)
1, 5 and 10 Mbytes 50
15 Mbytes
40
30
TCP/IP
20
10 Note: The curves for 1, 5 and 10 Mbytes overlay each other 0
1000
2000 3000 Buffer Size or MTU (BYTES)
4000
Figure 4: Memory-based Data Transfer Using AAL 5.
5000
Client
Server
OPEN (file) CONNECT (server) gettimeofday () WHILE ( READ(file, buffer, size) ) { ATM_SEND( dev, buffer, size ) } gettimeofday () DISCONNECT (server) CLOSE (file)
ACCEPT (client connection) WHILE (anything to read) { ATM_RECV( dev, buffer, size ) } CLOSE (client)
Figure 5: File-based Data Transfer.
Transfer Through ATM Network - AAL5 (File-based Data Transfer) 80
70
Throughput (Mbit/s)
60
50
40
30
Internal Disk Read 1, 5, 10, 15 Mbytes
20
TCP/IP 10 Note: The curves for 1, 5, 10, 15 Mbytes overlay each other 0
1000
2000 3000 Buffer Size or MTU (BYTES)
4000
Figure 6: File-based Data Transfer Using AAL 5.
5000
(File Size (Bytes))
Buffer Size (Bytes) 2048
3072
4096
1 MByte
7.46 %
6.17 %
23.89 %
5 MBytes
20.45 %
23.26 %
37.27 %
10 MBytes
21.77 %
25.65 %
39.04%
Figure 7: Cell Loss on receiving side for Memory-based Data Transfer at AAL5 and AAL3/4 layers.
