POND: The Power of Zone Overlapping in DHT Networks - CiteSeerX

International Conference on Networking, Architecture, and Storage

POND: The Power of Zone Overlapping in DHT Networks Zhiyong Xu Suffolk University [email protected]

Jizhong Han Chinese Academy of Science [email protected]

Laxmi Bhuyan University of California, Riverside [email protected]

TRY [7], Chord [8], and TAPESTRY [9] used distributed hash tables (DHT) for this purpose. One major goal of P2P model is to achieve better load balancing than C/S model. Theoretically, DHT-based systems can achieve this. In ideal environment, the peers are identical and they are evenly distributed on the name space. Each peer is given the responsibility to manage data structure and file information in a zone with nearly equal size on the name space. Each zone maintains almost the same number of objects, and each object has the identical number of accesses. Thus, system workload is evenly distributed on every peer. If system has n peers, each peer will process exactly 1/n of the whole system workload. In such a scenario, DHT algorithms can also achieve very good routing performance. With the well defined routing data structures, a routing process is guaranteed to be finished within log(n) steps. This characteristic is especially important for large-scale DHT applications. However, such an ideal model is un-achievable in reality. Due to the skewed distribution of computer resources, zone sizes, file sizes and file access frequencies, the workload can not be well balanced on the peers [10]. For example, in a large scale DHT system, the sizes of corresponding zones of two peers may vary in the order of O(nlogn). Thus, the peer with a larger zone may have to take much more workload than another peer with a smaller zone. For various peers, the difference of the processing power could be in the order of two or four magnitudes. Thus, the processing capabilities to handle requests vary a lot. In addition, the access frequencies of objects are highly skewed and change from time to time. In a certain period, a file f1 could be requested by thousands or even millions of peers, while another file f2 have no service demand at all. However, it is possible that, one month later, there are a fast increasing service requests for f2 and no any further requests on f1. Clearly, the diverse and dynamic natures of P2P systems result in seriously load imbalance problem, and hurt the overall system performance. Figure 1 shows a sample DHT system. It has 8 peers. The white small circles represent the ideal locations of the peers on the name space. The black circles are the actual locations. Rectangles are used to refer objects. A larger rectangle means higher access frequency. Obviously, with such a skewed distribution of files, zones, and peers, it is almost impossible to achieve the optimal load balancing performance. In this paper, we propose a novel and efficient algorithm —- POND, which uses zone overlapping to achieve better load balancing in DHT based P2P systems. POND divides the peers into two categories: Main Peer (MP) and Shadow Peer (SP). Un-

Abstract— Distributed Hash Table (DHT) based algorithms have been proposed to solve the routing issues in large-scale Peer-to-Peer (P2P) applications. DHT systems create an elegant Peer-toPeer service model and work perfect in a homogeneous environment. Workload are evenly distributed on every participating peer and system is supposed to achieve the optimal performance. However, in the real world, the complicated scenarios such as peers’ capability heterogeneity, diverse file popularities, and mismatching problem between physical network structure and logical routing data structure, jeopardize the efficiency of DHT infrastructure and may result in serious load imbalance problem. In this paper, we propose ——- POND, a novel and efficient DHT algorithm to relieve the above problem. In POND, peers are divided into two types: Main Peers (MP) and Shadow Peers (SP). As other DHT systems, POND splits the name space into several disjointed zones. However, unlike other systems which only have one peer in charge of each zone, in POND, we use zone overlapping technique. In each zone, a MP and several SPs are used for services. Any of them is capable to handle service requests fallen into this zone. When a new request is coming, the peer with the minimal load has the high possibility to be chosen to provide the service. Thus, the risk of overload problem is reduced. With the introduction of shadow peers, POND also provides better failure tolerance and relieves the hotspot problem. Moreover, POND can improve the routing performance by reducing the average number of routing hops per request. We conduct extensive simulation experiments and the results show POND is superior to other DHT algorithms.

I. I NTRODUCTION In P2P architecture, to provide the service, system workload is distributed, not only on a small number of powerful servers, but on all the participant peers. P2P architecture attracted a lot of attentions from both academia and industry due to its fascinating characteristics such as self-autonomous, fault tolerant and high computer resource utilization. A lot of P2P file sharing applications appeared [1], [2], [3], [4], [5]. To design efficient, high performance P2P systems, a lot of technical issues, such as routing, file retrieving, resource management, security, and load balance, etc. have to be addressed. Structured P2P systems are being considered as the optimal solutions for the above problems. For example, CAN [6], PAS-

978-0-7695-3187-8/08 $25.00 © 2008 IEEE DOI 10.1109/NAS.2008.56

81

1 0 1 0 0 1 1 0

0

1 0 1 0

000 111 000 111 000 111 111 000

1 0 1 0

000 1 11 11 00

n−1

using SHA-1 hash algorithm with the predefined seeds. Clearly, these peerids could be quite different, and their corresponding zones are always spread all across the name space. Second, virtual server algorithms achieve load rebalancing during system running time by dynamically adjusting the amount of workload on each peer. This is done by changing the number of virtual servers. In case a physical peer is in danger of overloaded because of the heavy demands on its virtual servers, some of them could be migrated to other light-loaded peers. Such a strategy can provide flexible load balancing scheme by only touching a portion of the workload on the current peer. Thus, it became very popular in DHT systems.

): Name Space

ideal Peerids for peers Real Peerids

11 00 11 00

0 1 1 0 1 0 000 111 111 000 000 111 000 111

[0, 2

11 00 1111 0000 0000 1111 0000 1111 0000 1111 111 000 000 111 000 111 000 111

1 0 1 0 1 0 1 0

1 0 0 1

11 00 11 00 111 000 00 000 11 111 000 111 000 111

1 0

11 00

11 00 00 11 1111 0000 0000 1111 0000 1111 1111 0000

files with different access frequencies

B. Problems

Fig. 1. Peers’ and objects’ distributions in DHT systems

Although virtual server approach has been proved that it can significantly improve the load balancing performance, the performance gain comes with the high cost. It has severe drawbacks: 1) It increases the routing latency. Since each peer has multiple virtual servers, the average size of a zone managed by a virtual peer is much smaller than a zone managed by a physical peer in the original DHT systems. Thus, when building routing infrastructure, more distinguished peers will appear in different entries of a routing table. Furthermore, to satisfy a routing request, a peer may have to contact more intermediate virtual servers (which are likely on different physical peers), and this results in a larger average number of routing hops per request. 2) It also increases the routing table maintenance overhead. If a physical peer has c virtual servers, it has to keep c different routing tables for each virtual server. That means, to keep the state of the routing information upto-date, more ”ping-pong” messages have to be transmitted to check the availability of any peers who appeared in these routing tables. 3) It has high migration cost. First, it must find a suitable candidate peer for migration. Then, it has to make data copy to the chosen peer. For routing services, it is possible, but it could have certain overhead. For example, the size of a torrent file could be tens or hundreds of KBs. if there are many torrent files have to be migrated, the communication overhead could be high. For file retrieving services, it is almost impossible for migration, since the size of files are much bigger. 4) Another serious issue is, even if we are able to migrate a heavy loaded virtual server from an overloaded peer to another light loaded peer, it may cause that peer to be overloaded as well. In a highly dynamic environment, virtual servers might move back and forth on different physical peers, and result in virtual server thrashing problem. Clearly, to some extend, the virtual server approach is useful. However, its accomplishing drawbacks prevent it to achieve the optimal load balancing performance. A better solution is in great need. We propose POND to relieve these problems.

like the previous DHT algorithms which only use a single peer to manage each zone, in POND, except a MP which manages the routing data structures and information, multiple SPs also maintains the same information in this zone. The MP and all the SPs share the same zone. The routing or file retrieving requests which fallen into this zone can be handled by any of these peers (we can always choose the peer with the lightest outstanding workload). To reduce the risk of overloaded peers, system always balances the workload to be processed on each of these peers. With such a strategy, we can achieve better load balancing performance than virtual server solutions. We can also achieve better fault tolerance and avoid the single point of failure problem since multiple candidates are available. With the zone overlapping strategy, another benefit we can get is the reduced average number of outstanding peers in routing tables, the amount of hello messages transmitted between peers to check “alive” status are greatly decreased. Thus, we can achieve lower routing table maintenance overhead. Furthermore, we find that POND algorithm is able to improve the routing performance because of the reduced average number of routing hops per request. The rest of the paper is organized as follows: In Section II, we discuss the virtual server solutions and the existing problems in these solutions. In Section III, we describe the detailed system design, load balancing algorithm and peer join/leave operations. In Section IV, we demonstrate the simulation environment and conduct the performance evaluation and comparison of POND with other DHT algorithms. In Section V, we discuss some related works. Finally, in Section VI, we conclude the paper and give out our future work.

II. T HE CURRENT

SOLUTION :

V IRTUAL S ERVERS

A. Virtual Servers The load imbalance problem in DHT-based P2P systems has been discovered and studied since the appearance of these systems. Different solutions are proposed to attack this issue in various ways. A bunch of research projects have [11], [12], [13] proposed dynamic load balancing mechanisms. In these algorithms, virtual servers are used. Virtual server approaches achieve load balancing through two strategies: First, when a physical peer joins the DHT system, a set of virtual servers are created. The number of virtual servers is determined by its capability (The capability of a peer is a general term to represent computer resource, such as the computation power, storage space, network bandwidth, etc.). Each virtual server has a corresponding peerid and owns a small unique zone on the name space. The peerids of these virtual servers are also generated

III. S YSTEM D ESIGN In this section, we describe the detailed system design. To make the description more clear, we use Chord as the baseline algorithm. POND creates its own routing infrastructure upon Chord. One benefit of building POND on top of another DHT algorithm is that it can use the existing solutions for routing information creation and maintenance, peer join/leave operations,

82

etc., which makes it a perfect add-on on current systems. POND algorithm can also be applied to other DHT systems such as PASTRY, or CAN as well.

Physical Peer

Peerid

Zone

A

0

(224,0]

B

64

(32,64]

A. Main Peer (MP) and Shadow Peer (SP)

C

128

(96,128]

A unique feature of POND design is the zone overlapping (Sharing). In current DHT based systems, for a specific zone, there’s only one corresponding peer. For example, in Chord, a peer could host several peerids (one for each virtual server). The corresponding zone for each peerid starts from the peerid of the most adjacent peerid (smaller, count clockwise direction, this peerid could be resided on another physical peer) exclusive and ends on its own peerid inclusive. Because of the usage of collision free algorithms to generate peerids, the peerids of a physical peer spread all across the name space. Clearly, the zones owned by each physical peer are disjointed. The zone next to the current one on the clockwise direction is very likely to be located on the third physical peer. Just like original DHT design, in virtual server approach, for any point on the name space, there’s only one physical peer has the responsibility. In POND, we use another approach, we lose this limitation by allowing multiple peers to exist for any zone. We divide the peers into two types: Main Peer (MP) and Shadow Peer (SP). Zones are determined by the peerids of MPs. MPs take the major maintenance responsibilities for every zone on the name space. For each MP, the corresponding zone starts from the peerid of the peer who is the numerically closest to it (count clockwise) exclusive to its own peerid inclusive. Just like other DHT systems, the zones managed by MPs have no overlapping at all. However, for each zone, except this unique MP, several SPs are allowed to share the same zone and each takes a portion of workload in this zone. To be a SP of a specific zone, the peerid of a peer must be fallen within the range of this zone. In POND, for any request whose objectid fallen into this zone can be satisfied by the MP or any of the SPs. Compare to previous approaches, more efficient load balancing algorithms can be adopted. The detailed strategy will be described later.

D

192

(160,192]

E

32

(0,32]

F

96

(64,96]

G

160

(128,160]

H

224

(192,224]

Fig. 2. A sample Chord system, no virtual servers used

Physical Peer

VS 1

VS 2

A

(248,0]

(64,72]

B

(56,64]

(128,136] (200,208] (48,56]

C

(120,128] (192,200] (40,48]

D

(184,192] (32,40]

(104,112] (176,184]

E

(24,32]

(96,104]

(168,176] (240,248]

F

(88,96]

(160,168] (232,240] (16,24]

G

(152,160] (224,232] (8,16]

(80,88]

H

(216,224] (0,8]

(144,152]

VS 3

VS 4

(136,144] (208,216]

(72,80]

(112,120]

Fig. 3. A sample Chord system, with four virtual servers on each peer

virtual servers used. It also has eight physical peers. However, the whole name space is divided into 32 zones. Each peer hosts four virtual servers. For example, the zones managed by virtual servers on A are (248,0], (64,72], (136, 144] and (208,216], the corresponding peerids for these virtual peers are 0, 72, 144 and 216. On the contrast, POND uses a different approach. Here, we create a one-to-many relationship between zones and peers. For each zone, multiple peers exist (one MP and 0 to K SPs, K is the maximal number of shadow peers allowed). Figure 4 shows a sample POND system. It has eight physical peers as well, and the size of the name space is also 256. Only four zones are generated, with two peers are located in each zone. One of which is claimed as MP, and another is claimed as SP. For example, zone (192,0] has a MP: A (peerid 0) and a SP: H (peerid 224). The boundaries of a zone are determined by the peerids of the main peers only. For example, the boundaries of zone (192,0] is determined by the peerids of the main peers A (0) and D (192). The peerid of the shadow peer(s) could appear in any position within this zone. In this example, the peerid of the only one shadow peer in this zone is H, whose peerid is 224. When a request for an object whose identifier is fallen into (192,0], either peer A or peer H can be used to serve the request. For shadow peers, although its peerid is not on the boundaries of the zone, it is treated as a shadow of the peerid of the main peer. When it is to be added into any routing tables, its responsible zone is considered the same as a MP.

B. Zone Generation Because of the introduction of shadow peers, the zone infrastructure in POND is different from other Chord systems. We first describe the zone generation in Chord with no virtual servers, then describe the virtual server solution, and finally discuss the mechanism used in POND. Figure 2 shows a sample Chord system without virtual servers used. The size of the name space is 256. It has eight physical peers: A, B, C, D, E, F, G and H. The peerids for these peers are shown as well. Since no virtual servers are used, here, a one-to-one mapping relationship is created between zones and peers. Each peer only has a single peerid and one corresponding zone. For any point (identifier) on the name space, it is managed by one and only one physical peer. Thus, a request for a specific identifier goes to the only peer who is responsible for it. In virtual server approach, a many-to-one relationship is generated between zones and peers. Since each peer has multiple virtual servers, it also owns multiple zones. For each individual virtual server, a unique peerid is assigned, and the zone managed by it is the smallest movable unit. One thing we need to point out is: there’s still a one-to-one mapping relationship exist between a point and a virtual server (physical peer). For any identifier on the name space, it must belong to one and only one zone. All requests related to it have to be serviced by the virtual server who owns it. Figure 3 shows a sample Chord system with

C. Routing Data Structure In POND, each peer maintains a routing table and a neighbor table. As other DHT systems, in neighbor tables, the IP address

83

Zone

(192,0]

(0,64]

(64,128]

(128,192]

Area

Entry

MP

0 (A)

64 (B)

128 (C)

192 (D)

(0,1]

8 (H)

SP

224 (H)

32 (E)

96 (F)

160 (G)

(1,3]

8 (H)

(3,7]

8 (H)

(7,15]

8 (H)

Fig. 4. Main Peer and Shadow Peer in A sample POND system

Area

Entry

(15,31]

16 (G)

(0,1]

32 (E)

(31,63]

32 (E)

(1,3]

32 (E)

(63,127]

64 (B)

(3,7]

32 (E)

(127,255]

128 (C)

(7,15]

32 (E)

(15,31]

32 (E)

(31,63]

32 (E)

(63,127]

64 (B)

Area

Entry

(127,255]

128 (C)

(0,1]

64.32

(1,3]

64.32

(3,7]

64.32

(7,15]

64.32

(15,31]

64.32

(31,63]

64.32

(63,127]

128.128

(127,255]

128.128

Fig. 6. A sample routing table on peer A in Chord, virtual server used

Fig. 5. The sample routing table on peer A in Chord, no virtual servers used

information of several topological close peers is recorded. For the routing tables, a slight different strategy has to be adopted due to the introduction of shadow peers. Again, we use Chord as the baseline system to explain routing data structure used in POND. Figure 5 shows the content of the routing table on peer A (peerid 0) in original Chord system with no virtual servers used. In each entry (we use span to denote the area represented in each entry, thereafter), we record the IP address information of the Peer whose peerid is the numerically closet to the start of that span. Figure 6 shows the content of the routing table on peer A with the peerid 0 in Chord system with virtual server used. In each entry, we record the IP address information of the peer (who has one of its virtual server peerid closest to the start of that span). Since the physical peer A has 4 virtual servers with the peerids 0, 72, 144 and 216, a routing table has to be created for each of the other three virtual servers as well. Figure 7 shows the routing table on a main peer A in POND. With the introduction of the shadow peers, the rule to create the entries has to be modified slightly. This is done in two steps. First, for each entry, when we look for the peer whose peerid is the numerically closest to the start of that span, only the peerids of MPs will be considered. After the main peer is determined, in the second step, at the time when we make the decision which peer should be added, not only the MP but also any of the SPs located in the same zone as this MP can be chosen. As shown in Figure 7, for the first 6 spans, instead of the main peer B (peerid 64), the shadow peer E (32) is chosen and added to these entries. For the last two spans, the main peer C (128) is chosen. As we can see from the above example, unlike other DHT systems, in POND, when building routing tables, multiple candidates are available for each entry. We can always choose the best peer according to some certain criteria, such as the peer is the topologically closest (can be measured using the mechanism introduced in HIERAS [14]), or it has the largest storage space, fastest processing power, etc. It gives POND maximal flexibility to provide load balancing. For shadow peers, two different routing data structure generation mechanisms can be applied. First, it defines the spans in

Fig. 7. The sample routing table on peer A in POND

its routing table just according to its own peerid, the first span starts next to its peerid (on clockwise direction) with size 1, and other spans follow with sizes doubled each time. When choosing the candidate peer for a certain entry, the same strategy used above can be applied. The second mode is much simpler, the peerid of a shadow peer is treated as pseudo id, and it copies all the routing data structures from the main peer in this zone. A SP acts as a backup peer of the MP. Except the above tables, in POND, each peer also keeps a zone information table. This table is used to store the start and end points of the zone it belongs to, the IP address and peerid information of the main peer as well as all the shadow peers in this zone. As a new peer is added, the zone information table on each peer has to be modified. As soon as the peer leaves the system, the tables on the remaining alive peers also need to be updated. In current POND design, the MP makes modification and notifies all the SPs for any changes.

D. Routing Algorithm The routing process in POND is the same as Chord. When a peer requests an object, it sends out a routing request. The request is forwarded by consulting the routing table on each intermediate peers, and the response will be send back when it reaches the peer who stores the location information of the requested object. In POND, a routing request might be sent to a shadow peer instead of the main peer. However, the request still can be forwarded correctly and finally arrive the destination.

84

E. Peer Join Operations

n: the new coming peer M: the current MP K: the MP whose peerid is before M Procedure Peer Join(n) { id=SHA-1(n) find the current main peer M which is responsible for this id if (split) { choose a peer T to be the new MP divide the current zone into two zone1=(id(K), id(T)] create routing data structure on T using Chord copy the related information from M to T zone2=(id(T),id(M)] modify data structure and information on M } else { T is added as a SP in zone table on M copy the routing data structure and information from M to T } }

Like other DHT systems, we assume that we know at least one peer in the system when a new peer wants to join. Figure 8 describes the peer join algorithm in POND. It has two steps. First, a unique identifier (peerid) is generated using a collision free hash function such as SHA-1 based on some unique information of this physical peer (such as Ethernet card address). Second, a peer join message (with the peerid of the newly joined peer) is sent to the overlay network. Eventually, this message will reach a peer (the MP or any SP) who is in charge of the zone, in which the new generated peerid is fallen in. At this time, POND determines if the coming peer will be a MP of a new zone or just a SP of the current zone. If we make the decision that the new peer should be a shadow peer, then no zone split operation is necessary. The procedure is straightforward. The main peer records the new peer as a shadow peer in its zone information table, notifies all the other shadow peers, and copy the routing data structures such as routing table, zone information table to this new peer. If the new peer chooses to build its routing tables by itself, the same mechanism presented in Chord [8] can be used. Next, the message about the addition of this new shadow peer could be spread across the overlay network, for any peer who originally record the information of this MP or another shadow peer (of this zone) in their routing tables can change to this new peer. This process is not done in a hurry and can be done gradually during the system normal process time (same mechanism used in Chord can be used). If we decide to generate a new MP, then the split of the current zone is necessary. First, we choose a peer (previously a shadow peer in this zone or the newly joined peer) as the new main peer. It becomes the MP of the new zone which starts from the peerid of the previous main peer (count clockwise) exclusive, and ends at its own peerid inclusive. This new MP has to build its own routing data structures (this can be done using the peer join algorithm in Chord algorithm). All the shadow peers whose peerids are within this new zone become its shadow peers. The original MP copies the related information (such as file location information of files whose objectids fallen into this new zone) to this new MP. The new MP then generates or updates the content in its own data structures, and makes copies on shadow peers. The new zone creation process finishes. The second zone ends at the peerid of the original main peer. It manages all the remaining shadow peers. Here, the original MP and SP have almost all the information of the original bigger zone. Actually, no data copies are necessary. The peers only need to change the start and ending points of the zone. If any peer receives any requests for routing or file location information now owned by another zone, it has to notify the intermediate peers along the path about the change, and forward the request to any peer in the new zone as well. (This can be done with the same strategy in Chord algorithm)

Fig. 8. Peer Join Algorithm

other peers in this zone. (same as the node leaving mechanism described in Chord). If a main peer leaves, and it is the only peer in this zone, there’s no any shadow peers exist, then this zone will be combined with the nearby zones, and the main peer of that zone will take its responsibility. The operation is almost the same as Chord. If there’s at least one shadow peer exist, one shadow peer will be elected to become the main peer, and replace the functionalities of the dead main peer. In this case, the zones will be adjusted. This new main peer will be in charge of the zone starts from the peerid of the previous MP (clockwise), and ends at its own peerid. This can be done with a peer leave operation combined with a peer join operation in Chord. If any other shadow peers located within this zone, they will become the shadow peers of this new zone. The MP previously next to it will increase the size of its corresponding zone. Now, it starts from the peerid of the new MP and ends at its own peerid. The data structures including routing and file information within this added zone can be easily obtained from any of the above peers. One problem in POND and other DHT systems is: When a peer leaves, other peers which have the failed peer in one of entries in their routing tables might be affected. Like Chord, the lazy update mechanism can be used. When a request uses the invalid entry, it will result in a time out, and a new route will be created to provide the service. The detailed mechanism can be found in Chord.

F. Peer Leave/Failure Procedure In a large-scale DHT based P2P system, peers could fail or leave at any time. In POND, in each zone, peers send hello messages periodically to verify the availabilities of other members in this zone. This is the same mechanism we can use for routing table maintenance. MP leave/failures and SP leave/failures are treated differently. If a shadow peer failure occurs or it leaves the system, the MP in the corresponding zone just delete it from the zone information table and notifies other SPs. Any routing request previously send to that peer will eventually be solved by

G. Load Balancing Algorithm In POND, since each zone has multiple service candidates to be chosen, a better load balancing strategy can be achieved. For a routing request, when it reaches the corresponding zone into which their objectids are fallen, POND is capable to find out the best peer to serve this request.

85

each day’s traces together with the temporal ordering information preserved. The total size of all files in these traces varies from 16.75GB to 28.34GB, and the median file size varies from 7.62KB to 18.31KB with the maximal file size is 224MB and the minimal file size is 0. We use Chord-v to denote Chord algorithm with virtual server used. We assume each physical peer contains 4 virtual servers. In reality, this number may be even bigger. We use Chord-nv to denote Chord algorithm with no virtual server used. Each physical peer only has one unique peerid. POND-1 is used to represent POND algorithm with 1 shadow peers allowed. POND-3, POND-5 and POND-7 are used for 3, 5 and 7 shadow peers allowed, respectively. In our experiments, we assume every peer in a zone can deal with any requests within that zone. At any time, we know exactly the amount of workload on each peer, and try to distribute the workload as even as possible. In reality, this is not necessary. We only need to keep the workload below a certain valve. Another thing we want to point out is: In our experiments, we fix the number of shadow peers in each peer. For example, in POND-7, each zone has exactly 1 MP and 7 SPs. In reality, the number of shadow peers in each zone could vary.

The simplest mechanism is always to choose the peer who has the lowest outstanding workload in this zone to serve the new coming request. For a specific zone, when a peer (the MP or any of the SPs) receives a routing or file retrieving request, it forwards the request to the peer with the least load. However, in order to do that, each peer has to know the current workload on each peer. This phenomenon might incur a lot of exchanging messages. We can modify this approach to reduce the synchronization overhead. Each peer notifies other peers its outstanding workload periodically, or if the amount of workload changed a certain amount, it notifies others. In POND, we propose the third approach as well. We define two valve values: Vh and Vl . Vh is used to represent the workload condition. If the current workload (can be represented as resource consumption requirements on CPU power, network bandwidth, storage space, etc.) on a peer reaches Vh , it will notify all the other peers in this zone that it has a high load, and will not accept any further requests forwarded to it. All the other peers in this zone will note Vl as highly loaded, and avoid forwarding requests to it. Vl is to denote the lower end valve. After a while, if the workload on a highly loaded peer is reduced to Vl , it will notify all the other peers in this zone, that it is available to serve forwarding requests again. For each peer, if its current workload is above Vh , and it still receive external requests, it will forward these requests to other peers in this zone. If its current workload is between Vl and Vh , it randomly chooses forward the request to other peers or serve the request by itself. If the workload is lower than Vl , it serves any requests it receives and never forward to other peers. The third approach does not achieve the best load balancing as the previous two, but it compromise the maintenance overhead and the performance, and can be considered as the feasible and optimal solution.

B. Load Balancing In the first set of experiments, we measure and compare the workload distribution in POND and Chord. We order all the peers according to the amount of workloads they served. The peers which serve the maximal number of requests are placed on the top, while the peers serve very few requests are placed on the bottom. Figure 9 shows the ratios of the average workloads taken on the top 1% peers and the bottom 1% peers in different sized TS network. As we can observe from the results, Chord-nv has the worst load balancing performance. In the 1000-peer network, the workload carried on the top 1% peers is almost 47 times higher than the workload on the bottom 1% peers. While in POND algorithms, this number varies from 6.50 to 26.58. Clearly, the introduction of the shadow peers make system more powerful to balance the workload. As more shadow peers are allowed, more performance improvement can be achieved. Chord-v algorithm also has very good load balancing property. Its performance is comparable with POND-3 configuration. As the number of peers in the network increases, the ratio in all algorithms rises. However, Chord-nv algorithm increases the fastest. In the 10000-peer network, the ratio is almost 100. Clearly, with the reduced zone sizes, each peer has less objectids to manage. However, the skewed file access pattern makes the accesses on the hot files are concentrated on a smaller percentage of peers, and make the load imbalance problem even worse. All the other algorithms increase much slower than Chord-nv. This proves the importance of using proper load balancing technique in DHT network. We also observe, in larger networks, Chord-v does not work as well as POND-3. We believe the reason is the virtual server migration algorithm. System always migrate the highest loaded virtual server from one physical peer to another. Because of the highest skewed file access frequency distribution, after the migration, the workload on the original peer may decrease a lot while the workload on the other peer may increase a lot. Figure 10 shows the ratio of the average workload on the top 5% peers divided by the average workload on the bottom 5% peers. As we can observe from the results, Chord-nv still has the worst load balancing performance. The ratio varies from 12.76 in the 1000-peer TS network to 30.25 in the 10000-peer TS network. POND systems have much better performance, in

IV. P ERFORMANCE S TUDY We conduct extensive simulation experiments to evaluate and compare the load balancing and routing performance in POND with Chord. In this section, we first describe the experimental environment used in our simulations, and then present our simulation results.

A. Experimental Setup In our simulations, we choose GT-ITM Transit-Stub (TS model) as the primary network topology model. TS is an internetwork topology model proposed by E. Zegura in [15]. It is a 2-layer internetwork model, a TS network is composed of interconnected transit and stub domains. Transit domains function more like Internet backbone while stub domains work more like local area networks. TS model reflects the hierarchical character of internetwork and was widely used by network research work. In our simulations, we vary the number of peers in the system from 1000 to 10000. We use Chord as the underlying DHT algorithm to compare with POND. Although many commercial P2P systems are running on Internet, few P2P systems traces are available. Due to the lack of appropriate P2P application traces, we use the web proxy logs obtained from the National Laboratory for Applied Network Research (NLANR) as the P2P workload traces in our simulation. This method has been widely used in many other research projects [16] [17]. The trace data we use are collected from three individual servers pa, pb and uc between July 8, 2004 and July 14, 2004. We also tested other traces, but the results do not change. To generate enough traffic, we combine

86

pb trace

pa trace 120

70

Chord-nv

Chord-v

Chord-v

60

POND-1

100

50

POND-5

POND-5 POND-7

POND-7

The Ratio

The Ratio

POND-1 POND-3

POND-3

80

Chord-nv

60

40 30

40 20

20

10

0

0 1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

1000

Number of Peers

3000

4000

5000

6000

7000

8000

9000

10000

Number of Peers

Fig. 9. The ratio of the workload carried on top 1% peers and bottom 1% peers, TS model, pa trace

pa trace 35

2000

Fig. 11. The ratio of the workload on top 5% loaded peers/ bottom 5% loaded peers, TS model, pb trace

of hot files in pb trace are much lower than those in pa trace.

Chord-nv Chord-v

30

V. R ELATED W ORKS

POND-1 POND-3

25

POND-5

The Ratio

POND-7

20

15

10

5

0 1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Number of Peers

Fig. 10. The ratio of the workload carried on top 5% peers and bottom 5% peers, TS model, pa trace

POND-1 (which we add one shadow peer), the ratio decreases sharply to 7.23. As more shadow peers are allowed, the ratios decrease even more. While for all algorithms, as the size of the network increases, the ratios increase. However, Chordnv increases much faster than POND algorithms. The ratio in the 10000-peer network is about 137% more than that in the 1000-peer network. While for POND-1, POND-3, POND-5 and POND-7, these numbers are 76.8%, 67.7%, 35.2% and 14.0%. This result proves that we could achieve more performance gain in larger scale DHT systems by using POND algorithms. The more shadow peers are allowed, the more benefits we can get. On the other hand, more shadow peers will increase the maintenance overhead such as data and routing structure copies, we should carefully choose a suitable number of shadow peers in order to achieve the optimal overall performance. From Figure 10, we also find Chord-v has good load balancing performance, which is comparable with POND-3. As the size of the network increases, POND-3 can do a better job than Chord-v. We already explained the reason in the previous section. For verification purpose, we also test other traces. Figure 11 shows the ratio comparison results for pb trace. As we can observe from the result, POND systems have better performance than Chord-nv algorithm. Compare to Figure 9, the absolute values here are smaller. This is because of the different file access frequency distribution in pb trace. The access frequencies

87

A bunch of research papers [11], [12], [13] proposed dynamic load balancing algorithms to relieve the load imbalance in Chord. In all these approaches, virtual servers are used, and the peer heterogeneity are well considered to achieve a better address-space distribution. System dynamically monitors the workload distribution, and moves virtual servers from overloaded peers to light-loaded peers. However, the usage of virtual servers greatly increases the amount of routing metadata needed on each peer and causes more maintenance overhead. Furthermore, peers’ topological characters are not taken into account. On the other hand, even with the perfect address-space assignment, which means the sum of segments on all peers are proportional to their capabilities, the above systems may still not be able to achieve the optimal load balancing. This is due to the highly skewed file access behaviors. The fileid of a hot file may fall into a zone owned by a weak peer, and it is not capable to deal with the high demands for this file. It is also very likely that many files which are seldom used fall in a zone owned by a powerful peer. The computer resources on the powerful peer are wasted. The system workload is imbalanced, and the performance is limited. In [14], [18], the topological information was used to reduce the routing latency as well as the user retrieving overhead However, load balancing issue is not considered. In this paper, we adopt this idea, and use topological information to improve DHT system load balancing performance. Zhu and Hu [19] also attacked the load balancing problem by utilizing topological information. System migrate the workload from an overloaded peer to a nearby light-loaded peer and relieves the load imbalance problem. However, it still uses virtual servers, the high maintenance overheads reduced its benefits significantly.

VI. C ONCLUSION AND F UTURE W ORK In this paper, we propose POND, a new load balancing algorithm to compromise the maintenance overhead and load balancing performance. POND uses zone overlapping technique. Unlike the previous DHT approaches, which use only one peer to manage a certain area on the name space, POND uses larger zone mechanism and creates a many-to-one relationship between peers and zones. Such a strategy can significantly reduce the probabilities of the overloaded peers. It also reduces

the routing infrastructure maintenance overhead since it reduces the amount of outstanding peers in routing tables. Furthermore, it can also reduce the routing overhead (measured in the average number of hops per routing request). Our simulation results show POND is superior to previous DHT load balancing algorithms.

VII.

[18] Z. Xu, X. He, and L. Bhuyan, “Efficient File Sharing Strategy in DHT-based P2P Systems,” in Proceedings of the 24th IEEE International Performance, Computing, and Communications Conference(IPCCC’05), (Phoenix, AZ), April 2005. [19] Y. Zhu and Y. Hu, “Towards Efficient Load Balancing in Structured P2P Systems,” in Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)), (Santa Fe, NM), April 2004. [20] C. Jin, Q. Chen, and S. Jamin, “Inet: Internet topology generator.” Report CSE-TR443-00, Department of EECS, University of Michigan, 2000. [21] A. Medina, A. Lakhina, I. Matta, and J. Byers, “Brite: An approach to universal topology generation,” in Proceedings of the International Workshop on Modeling, Analysis and Simulation of Computer and Telecommunications Systems (MASCOTS’01), Cincinnati, OH, Aug. 2001. [22] N. Spring, L. Peterson, A. Bavier, and V. Pai, “Using planetlab for network research: myths, realities, and best practices,” SIGOPS Oper. Syst. Rev., vol. 40, pp. 17–24, January 2006.

ACKNOWLEDGEMENT

This research has been supported by NSF Grant CNS0509207, 0509440 and National Basic Research Program of China (973 Program) under Grant No. 2004CB318202. We would also like to thank the anonymous reviewers for their valuable comments.

R EFERENCES [1] [2] [3] [4] [5] [6] [7]

[8] [9] [10] [11]

[12]

[13]

[14]

[15]

[16]

[17]

Gnutella, “http://www.gnutella.wego.com.” SETI@HOME, “http://setiathome.ssl.berkeley.edu/.” Napster, “http://www.napster.com.” Edonkey, “http://www.edonkey2000.net/.” BitTorrent, “http://www.bittorrent.com/.” S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker, “A scalable content addressable network.” Technical Report, TR00-010, U.C.Berkeley, CA, 2000. A. I. T. Rowstron and P. Druschel, “Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems,” in Proceedings of the 18th IFIP/ACM International Conference on Distributed Systems Platforms (Middleware’01), (Heidelberg, Germany), pp. 329–350, Nov. 2001. I. Stoica, R. Morris, D. Karger, M. Kaashoek, and H. Balakrishnan, “Chord: A scalable peer-to-peer lookup service for internet applications.” Technical Report TR-819, MIT., Mar. 2001. B. Zhao, J. Kubiatowicz, and A. Joseph, “Tapestry: An infrastructure for fault-tolerant widearea location and routing.” Technical Report UCB/CSD-01-1141, U.C.Berkeley, CA, 2001. V. King and J. Saia, “Choosing a random peer,” in Annual ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing(PODC’04), (Newfoundland, Canada), July 2004. B. Godfrey, K. Lakshminarayanan, S. Surana, R. M. Karp, and I. Stoica, “Load balancing in dynamic structured p2p systems.,” in Proceedings of The Annual Joint Conference of the IEEE Computer and Communications Societies(INFOCOM’04), (New York, NY), June 2004. J. Ledlie and M. Seltzer, “Distributed, secure load balancing with skew, heterogeneity, and churn,” in Proceedings of The Annual Joint Conference of the IEEE Computer and Communications Societies(INFOCOM’04), (New York, NY), June 2004. A. Rao, K. Lakshminarayanan, S. Surana, R. M. Karp, and I. Stoica, “Load balancing in structured p2p systems.,” in Proceedings of the 2nd International Workshop on Peer-to-Peer Systems (IPTPS’03), (Berkeley, CA), pp. 68–79, 2003. Z. Xu, R. Min, and Y. Hu, “HIERAS: A DHT-Based Hierarchical Peer-to-Peer Routing Algorithm,” in the Proceedings of the 2003 International Conference on Parallel Processing (ICPP’03), (Kaohsiung, Taiwan, ROC), October 2003. E. W. Zegura, K. L. Calvert, and S. Bhattacharjee, “How to model an internetwork,” in Proceedings of the IEEE Conference on Computer Communication(INFOCOM’96), (San Francisco, CA), pp. 594–602, Mar. 1996. A. Rowstron and P. Druschel, “Storage Management and Caching in PAST, A Large-Scale, Persistent Peer-to-peer Storage Utility,” in Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP’01), (Banff, Alberta, Canada), pp. 188– 201, Oct. 2001. S. Ratnasamy, M. Handley, R. Karp, and S. Shenker, “Topologically-Aware Overlay Construction and Server Selection,” in Proceedings of The Annual Joint Conference of the IEEE Computer and Communications Societies(INFOCOM’02), (New York, NY), June 2002.

88