EFFICIENT ROUTING IN NON-UNIFORM DHTs FOR RANGE QUERY SUPPORT * Maha Abdallah LIP6, Université Paris 6 8, rue du Capitaine Scott 75015 Paris, France
Eliya Buyukkaya LIP6, Université Paris 6 8, rue du Capitaine Scott 75015 Paris, France
[email protected]
[email protected]
ABSTRACT By randomly mapping items and nodes to a common address space, most P2P index structures based on DHTs provide only exact match data lookups. This compromises their use in data-oriented applications where more advanced query facilities, such as range queries, are needed. We have previously proposed a P2P index structure that refines the Chord DHT by mapping data items to the Chord address space in an order-preserving way. Load balancing of skewed data is then achieved deterministically by pulling nodes to zones where too many data items reside. However, based on Chord's underlying routing infrastructure, our previous solution can no longer ensure logarithmic routing cost when the node distribution over the address space becomes highly skewed. In this paper, we push our previous study one step further and propose a simple yet efficient refinement to Chord's routing component. By offering nodes more flexibility in the choice of their neighbors, our solution ensures logarithmic routing cost, with high probability. Most importantly, our solution is dynamic in the sense that it adapts to any node distribution, including highly adversarial ones. KEY WORDS Peer-to-Peer Systems, Distributed Hash Tables (DHTs), Range Queries, Routing, Load-balancing.
1. Introduction In the last few years, peer-to-peer (P2P) computing has become a key paradigm for structuring large-scale distributed systems. In P2P systems, nodes (called peers) organize themselves into an overlay network in which they all play an equal role and act as both clients and servers, thus enabling the aggregation and sharing of ∗
This work is supported by the “SemWeb” project of the ACI Masses de Données program, and by the “Respire” project of the ARA MDMSA program.
world-wide distributed resources in a cost-effective manner. At the heart of any P2P system is the issue of data lookup and search. In this regard, initial P2P systems suffered from serious scalability problems either due to the use of a central index server [18] or due to the flooding of the network with request messages, thus inducing a prohibitive communication cost [11]. These problems have motivated several research groups to focus on the design of what became the standard approach to scalability in P2P research: distributed hash tables (DHTs) [20, 21, 8, 22, 6]. Simply stated, DHT systems implement a hash function that maps any given item to a specific node in the system: every data item is associated with a unique key, and every node in the system is responsible of a set of keys; based on its key, an item (or a pointer to the node storing the item) is routed to and stored at the node responsible of that key; key-based queries are then directly routed to that same node. The immense success of DHTs is mainly due to the scalability guarantees they offer in terms of lookup latency, and that without requiring a node to know about all other nodes in the system. This is indeed crucial in P2P settings where a global system view cannot be assumed. In a system of N nodes, most DHTs require every node to keep information about only O(log N) other nodes, called neighbors; a lookup is then performed in O(log N) overlay hops by routing the request to the neighbor that is "closest" to the destination node. 1.1
Range Queries vs. Efficient Routing
A critical issue in DHT systems is the mapping of items to nodes in a load-balanced way. This is usually achieved using a uniform hash function that randomly maps items and nodes to a common key space. Randomization removes data skew by ensuring a uniform distribution of items over the key space, and achieves load balancing by having every node be responsible of a balanced portion of the common space.
While hashing schemes are very efficient for load balancing, they are seriously deficient in one regard: they only support exact-match lookups. Indeed, hashing destroys ordering among data items, making range query processing rather impossible. This compromises their use in a wide variety of distributed applications, such as P2P databases, in which range searches are crucial. To overcome this limitation, we have recently proposed an indexing mechanism that builds on the DHT technology to provide high scalability while enabling efficient range query processing [1]. Our solution, designed in the context of the Chord DHT [21], achieves this goal by replacing the DHT hashing scheme with an order-preserving function for mapping items to the DHT key space. Load balancing is then performed by moving nodes around the key space depending on the system's load and data skew. Although adapted to arbitrary item distributions, our load-balancing technique might however lead to situations where the distribution of nodes over the key space becomes highly skewed. In this case, logarithmic routing cost can no longer be guaranteed based on Chord's underlying routing infrastructure. 1.2
Our Contribution
In this paper, we push our previous study one step further and refine our indexing scheme so as to get the best of the three worlds, namely range query support, load-balancing, and scalable routing in arbitrary node distributions. Most importantly, we identify, for the first time, worst case scenarios that might arise in a number of applications, and in which DHT routing complexity grows linearly in the number of nodes. We then solve this problem and propose a simple yet efficient refinement to the Chord's original routing component. By allowing nodes more flexibility in building their routing tables, our solution is dynamic and provides logarithmic routing guarantees under arbitrary node distributions, including adversarial ones. Finally, although designed in the context of Chord, our solution seems to be applicable to a broader range of DHT overlays. The rest of the paper is organized as follows. Section 2 defines the underlying system model and gives some background on the Chord DHT as well as our orderpreserving variant. Section 3 discusses routing issues in detail, defines worst-case node distribution scenarios, and describes our solution for efficient routing in arbitrary non-uniform settings. Section 4 validates our proposition and presents simulation results of its performance. Section 5 discusses related works. Finally, section 6 concludes this work and points out some of our future perspectives.
2. System Model As stated earlier, our indexing mechanism can be seen as a refinement of the Chord DHT overlay so that efficient range query processing can be supported under arbitrary node distributions. In the following, we will use the terms "node" and "peer" interchangeably.
Nodes Keys
m
n1
2 -1 0 1 2
3
4
+2 +4
n8
n2
+8
n7
n3
+16
keys of n7 +32
n6
n4 n5
Figure 1. A Chord identifier space
2.1 Original Chord In Chord, nodes and items are randomly mapped to a m circular m-bit key space K = {0, …, 2 1} using SHA-1 [7] as a base hash function. A node is mapped to the key space by hashing its IP address, while a data item is mapped to the same key space by hashing the item itself. The image of a node under the hash function is also called its identifier. Keys are ordered in a ring, and a data item is assigned to the first node whose identifier is equal to or follows the data item's key in the key space. A successor (resp. predecessor) of a node n is defined as the first node that succeeds (resp. precedes) n on the key ring. In this setting, a node needs to maintain routing information about only O(log N) other nodes, called neighbors. The set of neighbors of a node n contains all nodes managing keys succeeding n by 2i-1 in the key space, where 1 ≤ i ≤ m. A lookup is then resolved in only O(log N) messages, where a request is passed from one neighbor to another until the destination node managing the requested item is reached. Figure 1 shows a Chord ring with m = 6. The system is composed of 8 nodes storing 15 keys. Node n1 has three neighbors, n2, n3, and n5, that it uses to route lookups to destination nodes. ―
2.2 Order-preserving Chord The major benefit of uniform hashing in Chord, i.e., loadbalancing, comes however at a price since hashing scrambles the items over the key space, thereby destroying items' order and making any range search impossible. To deal with this problem, we have previously proposed in [1] an indexing mechanism that builds on the Chord overlay by replacing Chord's item hashing by any order-preserving function to map items to the key space. In order to deal with load imbalance in case of data skew where some nodes would have significantly more items landing in their ring interval than other nodes, we extended the Chord infrastructure with a mechanism technique that achieves load-balancing by dynamically moving nodes around the ring depending on the key distribution such that every node has no more than its fare share from the system.
key k n6
key k
Nodes
n10 n11 n1 n n9 n7 8 6th …
Nodes
n1
n11
n2 3rd
n10
3rd n5 2nd n4
n3
2nd
n9
1st hop
1st hop
n4 n8 n3
n5
n7 n2
n6
lookup (k)
Figure 2(a). Chord's routing in a nonUniform node distribution
The basic idea is to ensure that the load li for every node ni does not cross some threshold load L (i.e., li ≤
L ) [1]. This is achieved deterministically by having every node ni regularly checks its current load li. If li > L , ni tries to shed part of its load to its predecessor (resp. successor) nj, whenever (li+lj) /2 ≤ L . This is done by having nj (resp. ni) moves clockwise (resp. counterclockwise) around the ring by increasing (resp. decreasing) its identifier and takes (resp. sheds) over (li−lj) /2 items from ni's interval. This leads to both nodes having (li+lj) /2 items. If, on the other hand, (li+lj) /2 > L , then ni contacts a distant node, say nk, such that lk+lk+1 ≤ L . This condition is crucial to ensure that migrating node nk to a different zone in the ring does not leave nk's successor overloaded. On ni's request, nk leaves the system by handing its own items to its successor node, and rejoins the system by positioning itself on the ring such that it takes in charge half of ni's load. An important question is then how an overloaded node ni (i.e., li > L ) can discover an underloaded node nk (i.e., lk +l k+1 ≤ L ). The solution we proposed in [1] is deterministic and is based on the underlying DHT infrastructure to maintain a distributed index, whereby an underloaded node nk advertises itself through the DHT using its load to generate it hashing key (cf. [1] for details). This results in underloaded nodes being advertised and discovered in only O(log N) messages.
3. Routing in non-uniform DHTs A key component in DHTs is the list of neighbors, called routing table, that each node maintains so that logarithmic routing cost is achieved. In Chord, the table of a node n contains the identity (i.e., the IP address as well as the ring identifier) of log N neighbors that are halfway from n on the ring, quarter of the way, eighth of the way, and so forth. More precisely, the ith entry of the table of a node n
lookup (k)
Figure 2(b). Chord's routing in a uniform node distribution
contains the identity of the first node that succeeds n by at least 2i-1 on the ring, where 1 ≤ i ≤ m. To lookup a key k, a node forwards the query to that neighbor with the largest identifier not exceeding k. 3.1 Problem statement The power-of-two structure of the routing table ensures that each routing hop cuts the ring distance to the destination at least by half. Given that the basic hypothesis underlying Chord is a uniform distribution of nodes over the ring, the routing table structure also ensures that each routing hop crosses at least half of the remaining nodes to destination, allowing a lookup to be resolved in O(log N) hops. While a uniform node distribution is guaranteed in Chord's original infrastructure, this hypothesis does not always hold for our order-preserving variant. Indeed, by eliminating item hashing, non-uniform distributions of items over the ring are likely to occur. Since in our case the node distribution mirrors the item distribution for load-balancing purposes, non-uniform node distributions can naturally arise. In particular, if the node distribution becomes highly skewed, Chord's original routing component on which our order-preserving solution is based can no longer provide logarithmic routing guarantees. 3.2 Adversarial scenarios To illustrate, consider the non-uniform node distribution depicted in Figure 2(a). The system is composed of 11 nodes. Based on Chord's original routing infrastructure, node n2 initiates a lookup request for key k that lies on the ring interval managed by node n1. Since the largest identifier not exceeding k in n2's routing table is that of n4, the query is forwarded to node n4. In turn, n4 determines the largest identifier in its routing table not exceeding k and forwards the query to node n6. This query forwarding continues along the same protocol until it reaches the destination node n1. Most importantly, we observe that while each routing hop indeed cuts the remaining ring
Neighbors created by Chord
key k n n1 n5 n6 7
Neighbors added by successor2
… 6th
n4
rd
3
key k n6 n7 n1 n5 rd n4
2nd 3
1st hop
2nd
n3
n3 1st hop
n2
Figure 3. Non-uniform node distribution: worst-case scenario
distance to key k by half, it clearly does not cross half of the remaining nodes to the destination node n1. This obviously results in routing performance degradation, whereby the lookup request for key k requires 6 routing hops, as opposed to 3 hops had the distribution been uniform for the same number of nodes (Figure 2(b)). Through a closer look at the above example, we identify, for the first time, worst-case scenarios in which Chord's original routing component requires a number of messages that is linear in the number of nodes in the system. Typically, consider the node distribution depicted in Figure 3. The system is composed of 7 nodes distributed over an m-bit identifier ring following a geometric i -1 sequence ai = a r , where the scale factor a=2m-1, the common ratio r = ½ , and i ≥ 1 . More precisely, the ring distance between nodes defines a geometric sequence such that the identifier of node ni+1, Id(ni+1) = Id(ni) +a i, wh er e ai = 2 m-1 ×
(½)
i -1
n2
lookup (k)
and i ≥ 1 .
In this setting, a lookup request initiated by a node, say n2, for key k will visit every node on the ring between n2 and the manager of k, node n1. This clearly results in a prohibitive cost where a key lookup requires a number of routing hops that grows linearly with the number of nodes in the system. One might however wonder whether such scenarios are likely to arise in practice. We argue that they do arise in a broad range of data applications. Recall that in our order-preserving Chord, the node distribution mirrors the item distribution, meaning that the above described worst-case scenarios involve data items that follow a geometric sequence. Geometric sequences are actually very common in a number of applications, such as statistical physics, sensor networks, and software reliability engineering applications, just to name a few. This stresses the need for a DHT routing infrastructure capable of dealing with such adversarial node distributions, thereby broadening DHTs' applicability to a wide range of data sets.
lookup (k)
Figure 4. Augmented routing infrastructure: adding neighbors using successor2
3.3 Design rationale A careful analysis of the scenarios presented in Figure 2(a) and Figure 3 gives an insight into the problem. As stated earlier in this section, when Chord's original routing component is used in non-uniform distributions, each hop no longer crosses half of the remaining nodes. This is due to the fact that in Chord, neighbors are chosen independently of the node distribution, which indeed makes sense in uniformly distributed settings. This suggests that in non-uniform settings, the routing infrastructure should be designed with the node distribution in mind if logarithmic routing cost is to be maintained. Towards this objective, the difficulty one is faced with is that knowledge of the density of nodes over various ring intervals cannot be assumed in large-scale environments. To overcome the need for a density function, we focus on the major consequence of nonuniformity on Chord's routing component, which translates by an important number of entries in the routing table pointing to the same node. This clearly reduces the number of neighbors of a node (
