An Investigation of Network Support for Grid Middleware and

ICICS-PCM 2003 15-18 December 2003 Singapore

1B4.7

An Investigation of Network Support for Grid Middleware and Applications Y F Wong1; C Y Wong2; L H Ngoh2; W C Wong2 1

Computer Center, National University of Singapore, Singapore 2 Institute for Infocomm Research (I2R), A*Star, Singapore

Abstract Owing to the rapidly growing interests and excitement in distributed computing and Grid-enabled applications, the facilitating middleware layer has become a center of attention in recent years. At the same time, rapid advances in networking technologies such as the Resilient Packet Ring (RPR) have evolved to carry all kinds of information including data, voice, multimedia and now, grid applications. However, little studies have been made to discover the interactions between grid middleware and the network layer. This paper therefore investigates the degree of which grid middleware design takes advantage of the underlying networking resources and features.

1. Introduction While a lot of attention is devoted to middleware research and vast resources have also been allocated to networking improvements, it is suspected that middleware and networking remains very disparate and non-interacting. Popular middlewares such as the Globus toolkit [6] are often developed without much consideration of underlying network features and primarily focuses on immediate operational issues. While new technologies such as the Resilient Packet Ring (RPR) [1] technology, the Packet over SDH/SONET (POS) [2] and the Gigabit Ethernet (GE) [3] provide new features including fault recovery, Qualityof-Service (QoS), multicasting, etc., these are seldom taken full advantage of. The motivation for this paper is therefore to investigate the degree of interaction between these two layers. While networking advancements have provided rich and wide ranging set of features for use and implementation, it is interesting and important to find out to what extent, the upper grid middleware layer, is making good use of them to achieve its purpose of facilitating grid computing. This paper first discuses a relatively young optical networking technology – RPR and subsequently analyses the commonly used, open-source middleware – The Globus Toolkit. All experiments are conducted on a real network testbed – The Kent Ridge Advanced Network (KRAN).

0-7803-8185-8/03/$17.00 © 2003 IEEE

2 KRAN testbed construction Cluster X 100Mbps TX

RX TX TX

100Mbps

RX

RX

OC-48 RPR Ring max 45km circumference TX RX Cluster Z

TX

RX TX

RX

100Mbps

Cluster Y

Figure 2.1. Testbed setup The core network testbed, KRAN, consists of three Cisco 10720 internet routers located at three different locations spanning a 4km circumference. Optical links or fibre drums are deployed to simulate a wider area network up to a 45km circumference. The network runs the Resilient Packet Ring (RPR) technology with core bandwidth 2.488Gbps. Figure 2.1 shows the attachment of three clusters (X, Y and Z) to the three routers at Fast Ethernet speed of 100Mbps. Each cluster is only a single Pentium IV, 2Ghz processor with 256MB RAM PC running Red Hat Linux 7.2, kernel version 2.4.18-3. Both Globus 2.0 [7] and 2.2 [8] are installed in each PC. For our purpose, it is sufficient to have one PC per cluster for the moment, as we are not evaluating the computational efficiency of grid computing. Rather, we are more interested in the understanding of the performance of individual middleware utilities available in the package, and this is best understood with a simpler straightforward setup.

3 Experimental Results For network testings (section 3.1), results are based on the papers in [4, 5], which are also experimented on the same KRAN testbed ensuring consistency. For middleware testing, two utilities are of particular interest – globusrun and globus-url-copy. The former is a job execution utility, while the latter is a file transfer utility between Globus machines. Both Globus 2.0 and 2.2 versions are tested. All delays are computed using the Linux time command. At the middleware level, since program execution or file transfer delays are of the order of seconds and minutes, this method gives us enough accuracy and precision. Moreover, we would like to define delays and throughputs from a middleware user perspective. As such, delays and throughputs are calculated from the time the middleware user runs the utility, to the time it fully completes successfully with the return of the Linux prompt on the console. Middleware experiments are repeated until consistent results are obtained so that a meaningful average could be taken. In most circumstances, 3 to 5 repetitions are conducted. Some experiments are repeated up to 10 times.

3.1 Resilient Packet Ring (RPR) The Resilient Packet Ring (RPR) technology caters to time-sensitive applications by offering an Intelligent Protection Switching (IPS) [1], which “self-heals” a network in the event of a fibre fault, promising to restore the circuit within 50ms. Experiments conducted support this claim [4]. Fault recovery time is about 4.5ms. FS (bytes) 64 128 1518 Loading (G) 0.8 1.5 2.4 L (ms) 0.3 9.0 0.3 9.0 0.4 60 Table 3.1.2: RPR Latency, L, Stress Test for varying frame sizes, FS (bytes) FS (bytes) 64 96 128 192 512 1518 T (%) 33.2 45.9 59.9 84.5 93.7 93.7 Table 3.1.3: RPR Throughput , T, Stress Test for varying frame sizes, FS (bytes) Loading (% of line rate) < 96% 102% 106% High priority packets 3.97ms 4.05ms ≈345µs Low priority packets ≈352µs 239ms 264ms Table 3.1.4: RPR Latency QoS Test To understand the full limitations and perhaps even bottlenecks of the underlying network layer, KRAN is also subjected to stress loading up to 108% of the maximum core bandwidth (i.e. 108% of OC-48). Table 3.1.2 shows the partial latency results when loading is increased

gradually from 0Gbps (0%) to 2.6Gbps (108%), where traffic is forced to move from cluster X and Z to cluster Y in clockwise direction. For small frame sizes (less than 192 bytes), a substantial jump in latency from 300µs to about 9ms is detected when loading is as early as about 0.8Gbps (33% of total core bandwidth). For larger frame sizes, a substantial latency jump only occurs during network congestion around 2.4Gbps. This presents one limitation of the network for small frame sizes due to the processing limitation of the router processor. Throughput experiments were also conducted on KRAN. Throughput defines the maximum percentage loading of the network bandwidth before loss occurs. Table 3.1.3 reveals for large frame sizes, throughput may be as high as 93.7%. However, a frame size of 64 bytes may only achieve a low throughput of 33.2%. Again, packet frame size used is important to guarantee good performance. QoS has been a widely researched topic in networking and an important network aspect to test. RPR offers two queues at its MAC layer so that high priority packets may be mapped to the high queue and low priority packets to the low queue. With this configuration, Table 3.1.4 shows that packet differentiation only occurs during network congestion at around 104% of OC-48. During congestion, both packet types experiences larger delays but low priority packets have substantially poorer performance (by as much as 100 times poorer). The same trend can be said for QoS frame loss although not graphically presented in this paper. Interested readers may refer to [4]. For RPR multicast, multicast packets are sourcestripped and may be bandwidth inefficient. However, since all nodes are arranged in a ring in RPR, the mechanism is much simplified. It is important to point out that although results presented are only for RPR, many other networking protocols such as GE also offer similar network features [4] and is not limited to RPR.

3.2 Globus-url-copy (GUC) Globus-url-copy (GUC) is a secure file transfer utility between Globus hosts. It offers features such as thirdparty file transfer, TCP window adjustments, block size tuning and even the degree of parallelism. This section presents each of these results for both Globus 2.0 and Globus 2.2. Different file sizes from as small as an empty file to as large as a 1GB single file have been tested. Data files of the desired size are generated using the command [9]: • [user@clusterX user]$ dd if=/dev/urandom of=filerand.out count=n where (512 x n) = file length in bytes. The extent of cooperation between GUC and the network layer will be summarized towards the end of this section.

TCP-window

Files of varying file length (from 0 bytes to 1GB) are transferred from Cluster X to Cluster Y by tuning the TCP window size from 4kB to 256kB. The transfer delays for both Globus 2.2 (GT2.2) and 2.0 (GT2.0) are measured. The command used involves the –tcp-bs switch: • [user@clusterX user]$ time globus-url-copy –tcp-bs n file:///home/user/filerand.out gsiftp://clusterY/home/user/filerand.out where n = {4096, 8192, 16384, 32768, 65536, 131072, 262144}. Figure 3.2.1.1 reveals the behaviour of Globus 2.2 GUC results. Note that for all file transfers, there is a consistent overhead of about 80s (calculated from the transfer of an empty file). This includes all the connection times, handshaking and authentication before the actual file transfer begins. As the file sizes increase, the effects of different TCP windows become more obvious. Generally, the larger the window size, the better is the delay. This is only obvious for 4kB window size. Since the core network (at OC-48 or 2.4Gbps) experiences no congestion, the larger the TCP window, the quicker the transfer. Globus 2.0 reveals the same trend although not graphically presented in this paper. Overhead delay for Globus 2.0 is similarly 80s.

Ave Delay (s)

250

Globus 2.2: Globus-url-copy file transfer delays against file sizes (with various TCP window sizes)

200 150 100 50 0

0 4096 65536

1M

10M 50M 100M 500M File Sizes (Bytes) 8192 131072

16384 262144

1G 32768

Figure 3.2.1.1. Globus 2.2: File transfer delays for various file sizes across varying TCP window sizes Figure 3.2.1.2 shows that the file transfer throughput of Globus 2.2 reaches only about 47% of line rate for the largest TCP window sizes. The main reason is the long handshaking and authentication periods that affected actual throughput results. Results also show that about 64% is overhead time for transferring a 500MB file and 45% is overhead time for a 1GB file for Globus. Disconnect time is almost negligible, although at times, the utility hangs

after completion of file transfer. Further, Globus 2.0 results indicate no significant difference from the 2.2 version. Globus-url-copy file transfer thriughput against various TCP window sizes for a 1GB file

Throughput as a % of line rate

3.2.1

50 40 30 20 10 0

4k

8k

16k

32k

64k 128k 256k

TCP Window Sizes (bytes) Globus 2.0

Globus 2.2

Figure 3.2.1.2. File transfer throughput for a 1GB file with varying TCP window sizes 3.2.2

Block Size

Globus offers another parameter that can be tuned for better file transfer performance – the block size. Again, files of varying file length (from 0 bytes to 1GB) are transferred from Cluster X to Cluster Y by tuning the block size from 4kB to 256kB. The transfer delays for both Globus 2.2 and 2.0 are measured. The command used involves the –bs switch: • [user@clusterX user]$ time globus-url-copy –bs n file:///home/user/filerand.out gsiftp://clusterY/home/uesr/filerand.out where n = {4096, 8192, 16384, 32768, 65536, 131072, 262144}. The immediate trend is the larger the file size, the poorer the delay for all block sizes. This is consistent since larger files take longer duration to transfer. However, there is no difference in the delay and throughput performance results by using different block sizes. This trend is the same for both Globus versions. The supposed overheads for processing smaller block sizes are largely missing for our testbed equipment. 3.2.3

Parallelism

In file transfers, multiple TCP streams may be opened for transferring the same file. This often promises quicker file transfer of the same contents. In this experiment, files of varying file length (from 0 bytes to 1GB) are transferred from Cluster X to Cluster Y by tuning the parallelism option from 1 to 128. The transfer delays for both Globus

TS 1 2 >2 GT2.2 172.7s 129.3s 127.9s GT2.0 173.6s 131.2s 129.8s Table 3.2.3.1. File transfer for varying TCP str≈eams (TS) or parallelism for a 1GB file 3.2.4

Transfer delay comparison of SCP / FTP / Globus-url-copy over various file sizes 250.1 200.1 150.1 100.1

Comparison of GUC with SCP, FTP and IPERF

Although it is interesting to know that results thus far indicate that Globus 2.2 does not necessarily outperform its predecessor version of 2.0, it may be even more meaningful to compare Globus against other major conventional file transfer methods. These would include FTP and SCP. IPERF [10] is a network performance measurement tool that is capable of reporting bandwidth, delay, jitter and packet loss. Its results are also included so that network delays and throughputs can be compared with corresponding results obtained at the middleware layer.

50.1 0.1 0

Third Party File Transfers

Third-party file transfers allow a node Z to command the transfer of a file from node X to node Y for instance. In this experiment, cluster Z commands the file transfer of varying file length (from 0 bytes to 1GB) from Cluster X to Cluster Y. The transfer delays for both Globus 2.2 and 2.0 are measured. The command used is: • [user@clusterZ user]$ time globus-url-copy gsiftp://clusterX/home/user/filerand.out gsiftp://clusterY/home/uesr/filerand.out Experiments involving TCP window, block size and parallelism tuning are repeated. Results indicate that third party file transfer delays and throughputs are largely similar to results of the normal GUC results for both gloibus 2.0 and 2.2. 3.2.5

Files of varying file lengths from 0 bytes to 1GB are transferred and the results are presented in Figure 3.2.5.1. IPERF results are the best. This is largely because it measures delays at the network layer with minimum networking overheads and free from middleware and complex authentication overheads. In fact, Table 3.2.5.2 reveals that its maximum throughput for a 1GB file is about 84.7Mbps, quite close to line rate of the FE card of 100Mbps. FTP delay performance closely mirrors that of IPERF. This is expected since FTP sits just on top of TCP without means of secure transfer mechanisms. FTP throughput is about 82.7Mbps.

Delays (seconds)

2.2 and 2.0 are measured. The command used involves the –p switch: • [user@clusterX user]$ time globus-url-copy –p n file:///home/user/filerand.out gsiftp://clusterY/home/uesr/filerand.out where n = {1, 2, 4, 8, 16, 32, 64, 128}. Contrary to common belief, better performance in terms of file transfer delays for both Globus versions for more than two simultaneous streams were not observed (Table 3.2.3.1). For a single TCP stream, the delay is about 173s for a 1GB file. For two simultaneous streams, delay is approximately reduced by a quarter to about 130s. In fact, the actual file transfer time is halved but the handshaking and authentication overheads of about 80s remain. This appears encouraging at first, but when more than two streams are employed, there is only a very slight delay improvement after that.

SCP iperf

1M

10M 50M 100M 500M File size (bytes) FTP globus 2.0

1G

globus 2.2

Figure 3.2.5.1. Comparison of transfer delays for GT2.2 GUC, GT2.0 GUC, FTP, SCP and IPERF SCP FTP GT2.2 IPERF GT2.0 T (%) 39.4 82.7 46.8 84.7 46.9 Table 3.2.5.2. Comparison of transfer throughputs, T, as a percentage of the line card rate of 100Mbps, for GT2.2 GUC, GT2.0 GUC, FTP, SCP and IPERF Not surprisingly, Globus results are one of the worst. Previous results actually hinted to this outcome. Throughput is barely 47Mbps due to connection overheads, handshaking and overheads. This represents a serious problem because Globus imposes heavy penalties on file transfer performance. SCP is also built-in with security and its performance is similarly not as ideal as FTP or IPERF. SCP outperforms Globus most of the time but its performance degrades substantially for large file sizes such as 500MB or 1GB. 3.2.6

Relationship of GUC with the underlying network layer technology

The major features of GUC are TCP-windowing, block size adjustments, parallelism, and third-party file transfers. TCP window, block size and parallelism settings

are GUC’s way of improving data transfer throughputs. However, from the results discovered, the best TCP window, block size and parallelism options merely offer a maximum of about 47Mbps throughput for both versions. Results do not come close to IPERF or FTP’s performance of about 83 Mbps. Again, GUC does not have features that effectively operate with the network layer. QoS and multicast technologies are clearly not visible in the utility, which may be an important aspect for grid applications that are wide ranging and may even be time-critical. GUC does not extend its performance and current set of capabilities by incorporating network features.

3.3 Globusrun (GR) The globusrun (GR) utility commands the execution of a remote program on a remote host to operate on a certain set of data. A quick investigation reveals that globusrun did not incorporate QoS features in its options (globusrun –help), for both Globus versions. Multicast features are also not implemented with globusrun. With a lack of these features, globusrun may not be ideal in handling certain applications. For instance, imagine realtime financial databases located on multiple remote machines. A client may require fast execution of his buy or sell orders across the network to multiple servers. These time-sensitive commands will certainly require higher priority than other normal commands.

4. Conclusion Network results (RPR) have been presented to identify its features, capabilities and limitations. These include rapid fault recovery techniques, rich QoS provisioning, simple Multicast mechanism and limitation on small frame lengths. Middleware results (Globus toolkit 2.0 and 2.2) are also presented to highlight its features, options and capability. These include two commonly used utilities globusrun and globus-url-copy. Both middleware utilities failed to make good use of the underlying network resources and features to extend its current capabilities, such as QoS provisioning or multicast. Although talks have been around for implementing QoS to Globus, little has been done and they do not have the focus of having a common QoS platform or both layers for example. Further, the Globus middleware added heavy penalties to the overall file transfer performance from a user perspective. This is therefore a huge concern. Future work for further investigation of the network and middleware layers and subsequently improve on the performance of current middleware utilities will therefore be of value.

5. Acknowledgements This Kent Ridge Advanced Network (KRAN) project, is funded by the Agency for Science, Technology and Research (A*STAR), under the Singapore Advanced Research and Education Network (SingAREN) programme [11] and in collaboration with Cisco Systems and SCS Networks Pte Ltd. We extend our thanks to A*STAR for sponsoring the project. We also like to express our gratitude to the KRAN steering committee members for their valuable comments and suggestions.

References [1] “Dynamic Packet Transport Technology and Performance”, Cisco Systems White Paper, http://www.cisco.com/en/US/netsol/ns110/ns221/ns223/ns2 26/networking_solutions_positioning_paper09186a00800a5 a3a.html, 2000. [2] Cisco Systems White Paper, “Cisco' s Packet over SONET/SDH (POS) Technology Support; Mission Accomplished”, http://www.cisco.com/warp/public/cc/pd/rt/12000/tech/posd h_wp.htm, Jul 2000. [3] IEEE P802.3z Gigabit Task http://grouper.ieee.org/groups/802/3/z/, Nov 2001.

Force,

[4] Y. F. Wong, C.Y. Wong, L.H. Ngoh and W.C. Wong, “Operation, Management and Performance Issues of LAN Technologies Applied to WAN Architecture”, IEEE Symposium on Computers and Communications (ISCC), pp 1379-1386, Jun 2003. [5] Y. F. Wong, C.Y. Wong, L.H. Ngoh and W.C. Wong, “Performance Comparison of Resilient Packet Ring (RPR), Packet over SDH/SONET (POS) and Gigabit Ethernet (GE) for network design”, International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS), pp 680-687, Jul 2003. [6] The Globus Project, http://www.globus.org, Apr 2003. [7] The Globus Project, “The Globus Toolkit 2.0 download page”, http://www.globus.org/gt2/install/download.html, 9 Aug 2002. [8] The Globus Project, “The Globus Toolkit 2.2 download page”, http://www.globus.org/gt2.2/download.html, 18 Mar 2003. [9] CERN, “Globus FTP: evaluation test”, http://www.to.infn.it/grid/gridftp/GlobusFTP-CERN190901.ppt, Sep 2001. [10] A. Tirumala, F. Qin, J. Dugan, J. Ferguson and K. Gibbs, “IPERF”, http://dast.nlanr.net/Projects/Iperf, Mar 2003 [11] Singapore Advanced Research and Education Network (SingAREN), http://www.singaren.net.sg, 2003.