Content Networks: Distributed Routing Decisions in ...

Content Networks: Distributed Routing Decisions in Presence of Repeated Queries Mojca Ciglariþ, Tone Vidmar, Matej Trampuš, Matjaž Panþur University of Ljubljana Faculty of Computer and Information Science Tržaška cesta 25, 1000 Ljubljana, Slovenia {mojca.ciglaric|tone.vidmar|matej.trampus|matjaz.pancur}@fri.uni-lj.si

Abstract Content networks are overlay networks, enabling access to distributed contents on centralized servers or individual computers. Since flooding-based routing scheme features poor scalability, we present a modification, which reduces the total network traffic while retaining the original efficiency. In choosy routing, each node, while passing an answer, remembers where it came from. Subsequently repeated queries about the same content are forwarded only to one neighbor. This way, the network learns effective routes. The simulations on several topology types have shown the expected behavior, with up to three-fold reduction in the overall query traffic. Keywords. Content network, content-based routing, flooding, peer-to-peer networking.

1 Introduction and Related Work In recent years, many types of networks applications have appeared that can be regarded as content networks. Content networks are overlay networks on top of IP networks that enable access to the distributed contents located not only on centralized servers but also on individual computers within as well as outside of a certain enterprise. Their common feature is support to the content addressing and routing, which means that the messages (typically user queries) are routed among network nodes on the basis of their contents rather than on the basis of the destination address. In flooding-based content networks, if a receiving network node cannot answer the query, it forwards it to all of its neighbor nodes. Although flooding guarantees

the answer will be found in the lowest possible number of hops, it causes significant network overhead, often causes congestions and features very limited scalability. In the paper, we present a routing modification, which reduces the total network traffic while it retains the success rate, i.e. the number of successful queries and the average response time. Our method (called choosy routing) expects each node to remember where the answers passed back to the query source came from and route the subsequent identical queries the same way. The simulations on several types of network topologies have shown that choosy routing exhibits the expected behavior, with about one third lower overall query hops and the same success rate. Taxonomy of the content networks is explained by Kung and Wu in [1]. They classify content networks regarding to the type of the content aggregation (semantic and syntactic/none) and the content placement (contentsensitive or content-oblivious). Most widely deployed content networks today belong to the category of syntactic content-oblivious networks, which we study in the present paper. We consider networks where the contents are not aggregated (grouped either according to their meaning or according to some artificial function, for example through hash calculation) and the placement is content-oblivious, so the content can be at any node without regard to its meaning. This way, a high level of local autonomy can be achieved (for example, an end user can control the placement of her or his files). Several popular file sharing and distributed storage applications belong to this category [3-5]. Flooding-based routing is known by its limited scalability and large overhead and is therefore used only where it is absolutely necessary. In content-oblivious networks, the content can reside on any network node and the system has to find or learn the way to reach it somehow.

0-7695-1926-1/03/$17.00 (C) 2003 IEEE

A

A

A

A

C

B

B

B

B

Fig. 1. Flooding mechanism. A query is originated by A, an answer is found at B. Peers who have seen the query are shaded, thin lines represent peer-to-peer (application-level) connections, thick lines represent query hops (routes taken), thick arrows represent answer hops. An obvious way is to introduce a centralized server or a small set of servers, where each node should register and report about its contents [5, 6]. A node requesting the contents should first send a request to the server, wait for the reply with the location of the result and then contact the given node. However such an indexing approach is very vulnerable - if the server crashes for any reason, the system ceases to function. Centralized indexing is also unwanted in the systems where anonymity is of high importance. On the other hand, fully distributed approach is highly fault-tolerant. The contents can still be placed everywhere without regard to the system topology or node location, while a node requesting the contents may advertise its query to its neighbors. The node possessing the requested contents sends back the answer message, while all other nodes forward the query to their neighbors. Since all the nodes perform same functions and have same privileges and obligations, we call them peers. Accordingly, such a network is a peer-to-peer network [4, 7]. Several research projects deal with finding the contents in a peer-to-peer system, but most of them dictate specific network structure and very formal query structure since they only operate with content identifiers [8-10]. Several factors (not related to content network) cause certain queries to be repeated by same or distinct network nodes over and over. Such queries, when repetitively flooded over possibly the same routes, represent high amount of unnecessary traffic. The quantity of repeated queries, i.e. queries requesting the same content, and their effect on the network behavior is sketched in [2].

2 Distributed Decision Making In order to reduce the unnecessary query traffic we designed a routing scheme where network nodes

remember which queries were successful. Actually, they do not remember the queries but rather the answers they are to pass back to the query originator. Although routing implementation can be adjusted according to the problem domain, let us assume we study a distributed storage system where all kinds of files are stored. Files are described by their name, type, size, date and possibly a few keywords. Let a query represent a search string or condition that is matched against the available file information. A node originating the query initially broadcasts the query to all of its neighbor nodes. Each of the receiving nodes evaluates the query against their local database and if there are no matches, they in turn broadcast it to all of their neighbors. When a matching file is found, the node returns an answer message with its contact information and the desired file information (metadata). Answer messages travel back over the same row of nodes as the original query. This is the basic flooding algorithm that is robust and fail-safe. The nodes do not bring any routing decisions. In our suggested routing, we could call it choosy routing, before forwarding the answer message, each intermediate node stores locally the matching file information (metadata) and the identification of the neighbor node that passed the answer. When the next query arrives, each node checks it against their local database and against the stored information. • If a match is found in a local database, an answer message is generated and sent back towards the query originator. • If a match is found within the stored information gathered from the forwarded answers, a query is sent only to the neighbor who has passed the matching answer. This way all the intermediate nodes contribute to the construction of the optimal route in a distributed fashion.

0-7695-1926-1/03/$17.00 (C) 2003 IEEE

A

D

A

D

A

D

A

D

C

B

B

B

C

B

Fig. 2. Improved routing mechanism – the choosy routing. Let us assume a situation from Fig. 1 has happened earlier. When D floods a new query, C knows the answer came from B and therefore forwards the query only to B. A knows the answer came from C and forwards the query to C. The peers who have not seen the previous answer would flood the query nevertheless. The size of local storage would depend of the system type, i.e. on the size of metadata to be stored and of the number of distinct answers that could be passed by the node before the metadata became obsolete. In [2], the author shows that a few megabytes would suffice for substantial caching of Gnutella queries. Although it can not be directly compared with a choosy routing, it can give us an approximate idea about the storage size. Figures 1 and 2 illustrate the difference between basic flooding and choosy routing. Basic flooding is demonstrated on Fig. 1. We can see the query made 9 hops before the answer was found. If the same query were executed again, it would be forwarded to the same nodes and would make 9 hops again before the answer was found. In addition, it would further be forwarded over the network by the nodes lying outside the way to the answering node until its lifetime expired. Figure 2 assumes that the query was already flooded over the network before as Fig. 1 shows. The nodes have remembered the promising routes and now they forward queries only to the nodes that gave an answer before. Instead of 9 query hops, the network only experiences 4 hops now. Despite the promising example a few more issues have to be addressed before we implement the choosy mechanism. Network nodes are highly autonomous and are allowed to leave the network arbitrarily. What to do when a perspective neighbor becomes unavailable? We suggest three possibilities: 1. Users could issue an explicit request to flood the query after the same query remained unanswered when executed for the first time. However users are selfish by nature: they care more about getting what they want then about the overall system performance. It is

questionable whether we can believe they would not misuse the power of flooding request. 2. The second possibility is to burden the network nodes with more accounting: a node forwarding the query to only one of its neighbors should wait for the expected answer and if it did not arrive within a reasonable amount of time, the node should flood the query. Such a solution is basically equivalent to flooding (the same set of nodes would receive the query), only it would take roughly twice longer since the choosy routing attempt was executed first. 3. The third possibility is similar to the second in that after an unsuccessful attempt of choosy routing is detected flooding happens. But the node who triggers the flood is the one who notices its prospective neighbor is no longer accessible. In that case, flooding starts farther form the source node and if the query is to reach a comparable number of network nodes as with the basic flooding, its lifetime has to be prolonged with regard to the number of hops it made until that moment. We think that the third option is the best since it requires fewer resources from the node as the second one. The node only has to check the neighbor connection before it forwards the query over it. However we have to stress this solution does not solve the problem of content migration. If a target node removes some previously accessible contents, subsequent queries matching the removed contents would still be routed to this node. It is up to the target node to either re-route or flood the misdirected queries. The choice among the three solutions listed above is largely dependent on the system structure, users trustworthiness and above all on the level of system stability. In a quickly changing system with highly transient resources, it is pointless even to think about

0-7695-1926-1/03/$17.00 (C) 2003 IEEE

storing routing information because it would become obsolete before its first use. Choosy routing is only to be used in a relatively stable system where the effort put in finding better routes can be paid off with a lower number of message transfers and consequently faster average response times.

3 Simulation We performed the routing simulation on several artificially generated topologies. Graph topologies with small world and power-law properties are reported [11,12] acceptable for simulating peer-to-peer and other self-organizing networks. Additionally we generated a few regular lattices and random topologies in order to compare their topological properties. In the next subsection, we briefly describe the background of small world and power-law networks.

3.1 Small world A small-world property in graphs was studied and described by Watts and Strogatz [13] and its relevance to P2P networks is discussed in [11-15]. It was shown that many large networks occurring in nature, technology or sociology exhibit small world properties. Each one of us has some small world experience: talking with a complete stranger at the party, we discover we have friends or relatives that know each other. In mathematical terms, small-world graphs exhibit small average shortest path among all pairs of nodes, sometimes called also characteristic path length, and are highly clustered. Clustering means that many of the neighbors of a certain node also know each other or are connected to each other. The measure for clustering is C, the so-called clustering coefficient, representing the ratio of number of existing connections among the node's neighbours to all possible connections among them (full graph), averaged over all the nodes. Typically, the clustering coefficient C in small world graphs is by an order of magnitude larger then C in a random graph with a comparable number of nodes, connections and average node degree. Intuitively we can say the flooding-based mechanisms are not very well suited for small world networks: since more than a few neighborhood connections exist in the graph, flooding causes unnecessary high message replication since each node receives the same message from several or even all of its neighbors.

3.2 Power-law The problem of overlay network topology simulation at the application level is closely related to modeling of the Internet topology on the network level. Faloutsos et al. [16] have identified the existence of mathematical rules in the form of power laws in the Internet topology. Power laws are believed to hold in various natural networks and recent studies [11,12] have shown that also P2P networks exhibit power-law properties. Adamic et al. [12] study exploitation of power law properties in P2P networks. As the Faloutsos et al. have shown, the power laws are formulated as

f ( x) ∝ x β

(1) The power-law exponent is obtained by performing a linear regression on f(x), drawn on the log-log graph. Out of the four identified power laws, one is especially interesting for our topology generation. It concerns the out degree of a node (the number of connections to other nodes): the fraction of the nodes with a given degree is proportionally related to the degree of the node – this is the so-called out-degree exponent. Other power laws describe: rank-exponent (we sort the nodes in decreasing order of their out degree and the out degree is proportional to the rank of the node), hopplot exponent (the total number of pairs of nodes within a given number of hops) and eigen exponent (the sorted eigenvalues of a graph are proportional to their order in a sorted sequence).

3.3 Network Topology Generation We represent a network topology by a set of nodes (vertices) V and a set of connections (edges) E among them: G = (2) Bu and Towsley [17] discuss weaknesses of several power-law topology generators and suggest a new generator, which is superior over other known generators in matching power-law exponent and the clustering properties. If Π(di) denotes the probability of choosing the node i when connecting a given node to another node, and d i denotes the degree of the node i, we define linear preference model as (3) Π( di) = di / Σ j d j . It has been reported that in small world networks, new nodes tend to connect to the nodes with a higher degree with stronger preference than described by linear preference model. A generalized linear preference (GLP) model includes a tunable parameter β that tells us how strong is the preference for connecting to nodes with higher degree: (4) Π( di) = ( di - β ) / Σ j ( dj - β )

0-7695-1926-1/03/$17.00 (C) 2003 IEEE

3.4 Simulation results The simulation perform as follows. The data about the topologies, nodes, connections, shortest paths, data residing on nodes, queries, results, etc. is stored in the Oracle 9i database. The topology generation and simulation of query execution is written in PL/SQL, and results (query hops, answers found, answer hops, etc.) are immediately inserted into database tables. The network topology analysis is written in Java since it is faster and convenient for a bit more ambitious algorithms implementation. Table 1 shows the average topological properties of GLP-created topologies and compares them with a random topology and with regular lattice, where each node is connected with three of its closest neighbors on each side. A random topology is created by adding the nodes one by one. Each node being added is connected to a randomly picked existing node.

Answers found FLOODING CHOOSY

1 312 1 080

All query hops

All answer hops

5 075 3 872

3 125 2 513

GLP RANDOM GRAPH LATTICE (3)

Average node degree

Clustering coefficient

When the desired number of nodes is generated, we generate additional connections by randomly picking two distinct nodes from the graph until the desired average node degree is reached. A lattice is created by generating a ring with the desired number of nodes and after that, connecting each node with k nearest nodes on both sides. In our case, k was 3.

Characteristic path length

With smaller β, where β ∈ (- ∞, 1), there is less preference for connecting to the higher degree nodes. The model starts with m0 vertices, connected by m 0 -1 edges. At each time step, one of the following two actions are taken: 1. With probability p, we add m ≤ m0 new links. 2. With probability 1 - p, we add a new node with m new links. It can be shown mathematically that the GLP model produces power-law distribution of the node out-degree. In the next subsection, it is verified that the generated topologies exhibit small world properties: short characteristic path length, comparable to random graphs, and high clustering coefficient, comparable to regular lattices.

2.59

0.531

5.91

2.61

0.077

5.88

6.17

0.600

6.00

Table 1. Topological properties for three types of generated topologies: generalized linear preference (GLP) as described in section 3.3, a random graph and a lattice. We may check whether or not the GLP graphs satisfy the small world properties: the characteristic path is short, just like the characteristic path in random graphs. The clustering coefficient is an order of magnitude larger than the one in random graphs. It is almost equal to the one in regular graph (lattice) where nearly all neighbor connections exist. Since node degree is a parameter of topology generation, the average degrees are very close in all the topologies. Thus, we can confirm a GLP topology exhibits the small world properties.

Avg. hops per query

Avg. hops per answer

Total of query and answer hops

2.38 2.32

8 200 6 385

38.72 50.75

Table 2. Average numbers of message transfers after the simulated run of 100 queries on the topology of 100 nodes with high data replication. In both cases, 97% of queries received at least one answer.

FLOODING CHOOSY

Answers found

All query hops

All answer hops

11 444 3 914

360 445 121 429

45 684 16 429

Avg. hops per query

Avg. hops per answer

Total of query and answer hops

2.36 2.41

406 129 137 858

3 604.45 1 214.29

Table 3. Average numbers of message transfers after the simulated run of 100 queries on the topology of 1500 nodes with moderate data replication. In the first case, 97% of queries received at least one answer, and 95% in the second.

0-7695-1926-1/03/$17.00 (C) 2003 IEEE

There are 100 distinct data items in our simulated system. In the first case (100 nodes topology), each node has from 1 to 20 distinct data items, while in the second case (1500 nodes topology) each node has exactly 3 data items. Each time step a new query is generated at a randomly picked node, while all other queries and answers are passed one hop further. The data items are queried according their popularity, following a Zipf distribution, as described in [2]. Popularity is an artificial measure, evaluating the share of queries asking about a particular content. The placement of data item and the number of replicas are not related to its popularity. In our simulation, the metrics for network delay is a number of message hops. While this is a convenient assumption when evaluating a simulation result, in a real implementation we should replace it with real peer connections delays, measured while passing other messages over. Table 2 and Table 3 show the simulation results. Our goal was to reduce the number of transferred messages and we can see that the total number of message hops achieved with choosy routing is considerably lower than the one with flooding. The mechanism still should not be regarded as scalable, however it brings an inexpensive way to reduce congestion problems. Note that the average number of answer hops remained unchanged, which implies the end user receives the answer in at least comparable time, if not faster due to the reduced network traffic and thus better network response times.

4 Conclusions and Further Work In the paper, we addressed basic issues regarding content networks and peer-to-peer systems, especially those related to the distributed search redundancy in presence of repetitive queries asking about the same contents. As a means to reduce the amount of unnecessarily replicated query messages, we suggested a choosy routing - relatively simple improvement to the basic flooding algorithm. We discussed system properties that encourage or discourage the use of choosy routing and simulated its behaviour on the artificially modeled topologies. To mirror the genuine routing behavior, our future work includes simulation of longer runs on larger network topologies. We believe that in a longer run, a network would have more time to configure itself properly and thus additional traffic reduction should emerge. The simulations might be performed on other topology types (random, lattice) in order to evaluate the relationship between routing behavior and topological properties. Further, network stability and content transience should be studied and incorporated into the simulation properly.

Thus we would be able to set the lifetime of locally stored data properly. Another improvement of the routing mechanism could be exchange of the stored metadata among the neighbor nodes. Acknowledgement. This work represents a part of the first author's doctoral research at the Faculty of Computer Science, University of Ljubljana, Slovenia.

References 1.

2. 3. 4. 5. 6. 7. 8. 9. 10. 11.

12. 13. 14.

15. 16.

17.

Kung, H. T., Wu, C. H.: Content Networks: Taxonomy and New Approaches, to appear in The Internet as a Largescale Complex System, K. Park and W. Willinger (eds.), Oxford university Press, Santa Fe Institute Series, 2002. Sripanidkulchai, K.: The popularity of Gnutella queries and its implications on scalability, http://www-2.cs.cmu.edu/ ~kunwadee/research/p2p/ gnutella.html Rohrs, C.: Query Routing for the Gnutella Network, LimeWire LLC, May 16, 2002, http://www.limewire.com/ developer/query_routing/keyword%20routing.htm. Gnutella, http://gnutella.wego.com Freenet (2002), http://freenetproject.org/cgi-bin/twiki/ view/Main/WebHome Napster (2001), http://www.napster.com Oram, A. (ed.): Peer-to-Peer: Harnessing the Power of Disruptive Technologies (Sebastopol, CA: O'Reilly & Associates, 2001). Kubiatowitz, J., et al.: Oceanstore: An Architecture for Global-scale Persistent Storage. In ASPLOS 2002. Ratnasamy, S., Francis, P., Handley, M., Karp, R., Shenker, S.: A Scalable Content-addressable Network. In ACM SIGCOMM, August 2002. Rowstron, A., Druschel, P.: Pastry: Scalable, Distributed Object Location and Routing for Large-Scale Peer-to-peer Systems. In Middleware, 2001. Jovanovic, M. A., Anexstein, F. S., Berman, K. A.: Modeling Peer-to-Peer Network Topologies through 'SmallWorld' Models and Power Laws, in Proc. IX. Telecommunications Forum TELFOR 2001, Belgrade, 2001. Adamic, L., Lukose, R., Puniyani, A., Huberman, B.: Search in Power-Law Networks, Physical Review E, Vol.64, 46135 (2001). Watts, D. J., Strogatz, S. H.: Collective dynamics of SmallWorld Networks, Nature, 393 (June), 1998, 440-442. Shiode, N., Batty, M.: Power Law Distributions in Real and Virtual Worlds, ISOC2002, http://www.isoc.org/inet2000/ cdproceedings/2a/2a_2.htm. Hong, T.: Performance, in A. Oram (ed.), Peer-to-Peer: Harnessing the Power of Disruptive Technologies (Sebastopol, CA: O'Reilly & Associates, 2001). Faloutsos, M., Faloutsos, P., Faloutsos, C.: On Power-Law Relationships of the Internet Topology, in L. Chapin (ed.), Proc. Conf. Applications, Technologies, Architecture and Protocols for Computer Communications (New York: ACM, 1999), 251– 262. Bu, T., Towsley, D.: On Distinguishing between Internet Power Law Topology Generators, in Proc. INFOCOM 2002.

0-7695-1926-1/03/$17.00 (C) 2003 IEEE