Internet Topology Discovery Through mrinfo Probing - IP Networking Lab

1

Internet Topology Discovery Through mrinfo Probing Jean-Jacques Pansiot∗ , Pascal Mérindol† , Benoit Donnet† , Olivier Bonaventure† ∗ Université de Strasbourg, Strasbourg – France † Université catholique de Louvain, Louvain-la-Neuve – Belgium

Abstract— Active and passive measurements for topology discovery have known an impressive growth during the last decade. While a lot of work has been done regarding inter-domain topology discovery and modeling, only a few papers raise the question of how to extract intra-domain topologies from measurements results. In this paper, based on a large dataset collected with mrinfo, a multicast tool that silently discovers all interfaces of a router, we propose a method to retrieve intra-domain topologies. The main challenge is to assign an AS number to a border router whose IP addresses are not mapped to the same AS. Our algorithm is based on probabilistic and empirical IP allocation rules. The goal of our pool of rules is to converge to a consistent router to AS mapping. We show that our router to AS algorithm is efficient in more than 99% of the cases. Furthermore, with mrinfo, point-topoint links can be distinguished from multiple links attached to a switch, providing an accurate view of the collected topologies. We provide a set of large topologies in various formats. In addition, we experimentally demonstrate that the node degree distribution is strongly impacted by the presence of layer-2 (L2) hardware, such as switch.

I. I NTRODUCTION Internet topology discovery has been an extensive subject of research during the past decade [1], [2]. While topology information can be retrieved from passive monitoring (using, for instance, BGP dumps in the case of the AS level topology), router level topology is usually obtained from active measurements using traceroute. Nevertheless, if traceroute has been largely deployed those last years, it comes with some important drawbacks. Traceroute provides a partial view of the network as it is routing dependent. For instance, backup links (high IGP weighted links for intra-domain and low local preference links for interdomain) will rarely be captured. This situation is illustrated in Fig. 1 where traceroutes (dashed arrows) are run on a topology with non uniform weights (Fig. 1(a)). Whatever the traceroute source (Fig. 1(b) to Fig. 1(g) give the shortest path trees (SPT) from all source) and whatever the destination, links R3 ↔ R2 , R4 ↔ R2 , and R4 ↔ R5 will never be discovered except possibly if the destination is the address of an end point of the link. Furthermore, traceroute might miss some router interfaces and the alias resolution problem is difficult to solve [3]. This leads thus to an incomplete view of the network. Obtaining complete intra-domain topologies is a daunting task, requiring extensive probing campaigns [4]. Recently, we have used mrinfo [5], a management multicast tool, in order to collect topology information [6].

mrinfo has the advantage of sweeping out many of the limitations of traceroute as it is able to silently discover all interfaces of a router. However, it requires multicast being enable within ISPs’ networks and no filtering policies, limiting so its applicability range. On the one hand, only IPv4 multicast enabled routers reply to mrinfo . On the other hand, some ISPs filter the IGMP messages used by mrinfo (they do not propagate them). In this paper, we use the mrinfo dataset [7] to extract intra-domain topologies. Obtaining real data concerning intradomain topologies is of the highest importance. Indeed, it allows one to study actual network characteristics (e.g, degree distribution, network connectivity, . . . ) to obtain insights on the way operators build their network. Further, real topologies are crucial inputs for network simulations, allowing so to consider complex and actual scenarios instead of using less realistic ones. Finally, by modeling their characteristics, it can contribute to build better topology generators. The contributions of this paper are multiple. We first describe how to extract intra-domain topologies from raw mrinfo data. While it is pretty easy to map IP addresses to an autonomous system number (ASN), the challenge is to mark the boundary of a given autonomous system (AS). In such a context, it is necessary to assign the right ASN to an AS border router (ASBR). In this paper, we provide an efficient algorithm, called router-to-AS mapping, for solving this issue. We evaluate our algorithm and show that it is able to assign an AS to a router in more than 99.5% of the cases. In addition, an interesting feature of mrinfo is that point-to-point links between routers may be distinguished from multiple links attached to a switch. On average, we discover that roughly 11% of the nodes in probed networks are actually switches connecting routers. As depicted in Sec. III, this is a fundamental issue to correctly analyze network characteristics. Second, based on our router-to-AS mapping, we provide a set of intra-domain topologies under various formats. Our set of topologies is composed of three kind of networks: Tier-1 (such as Sprint), Transit networks (such as the TDC Data Network), and Stub networks (such as U NINETT)1 . Further, based on a set of ASes generated using our routerto-AS mapping, we tackle the problem of potential bias in node degree distribution [8], [9] induced by the presence of layer-2 (L2) hardware. The bias introduced by L2 is illustrated on Fig. 2. Fig. 2(a) shows the actual topology where four 1 See

http://inl.info.ucl.ac.be/content/mrinfo.

2

R3

R1

R3

R4

1 1 3

4

R6

3

3

R6

3

R1

1

3

1

1 1

R3 1

R6

3

R1

1

1

R3 1

3

R6

3

1

1

R1

1

1

4

3 1

1 R5

R2

(d) SPT rooted on R3

R6

3 1

1 R5

R2

R4

1

1 4

R5

(c) SPT rooted on R2

R4

1

1 4

1 R2

(b) SPT rooted on R1 R4

R6

3

1

R5

R2

(a) Topology

R4 1

4

1 R5

R3

3

4

1

1 R2

R1

1 1

1

R1

1

1

R3

R4

1 1

1

(e) SPT rooted on R4 R3

R5

R2

(f) SPT rooted on R5

R4

1 1

1

R1

3

4

R6

3 1

1 1 R5

R2

(g) SPT rooted on R6 Fig. 1.

Weighted topology, traceroute, and missing interfaces/links

R1

R1

AS1 1.1/16

R1

R5 3 .2.

1.1

R4

R2

R4

R2 R6

switch 1.1.2.2

R3

R3

(a) Actual topology with L2

(b) Logical topology as seen by traceroute

Fig. 2. vision

R0

.1.1 .2 1.1 1.1.1 1.1 .0. 1 1.1 .0. 2

Fig. 3.

.1 1.1.2

AS2 2.2/16 2.2.3.1 .4.1

2.2 2.2.1.1

R2

2.2.3.2 .2 2.2.4

2.2.1.2

R3 2.2.2.1 2.2.2.2 R4

1.1.0.2 [version 12.4] 1.1.0.2 → 1.1.0.1 [1/0/pim/querier] 1.1.2.1 → 1.1.2.2 [1/0/pim/querier] 1.1.2.1 → 1.1.2.3 [1/0/pim/querier] 2.2.4.1 → 2.2.4.2 [1/0/pim/querier] 2.2.1.1 → 2.2.1.2 [1/0/pim/querier]

mrinfo example

From star to clique: the effect of L2 hardware using a logical L3

routers are interconnected through a switch. We notice that each router on Fig. 2(a) has a degree of 1. On the contrary, Fig. 2(b) provides a view of the same topology as it can be seen through a traceroute exploration: since the L2 hardware is not discovered (even not suggested), each router has a logical degree of 3. In this paper, we also tackle this problem by demonstrating the potential bias on node degree distribution induced by the presence of L2 hardware. Our work in this paper is based on a large dataset collected by mrinfo [5]. mrinfo is a tool that silently discovers all interfaces belonging to an IPv4 multicast enabled router and the IP addresses of its neighbors. In contrast to traceroute probing, it is possible to infer the presence of L2 hardware between routers. Using our routerto-AS mapping technique, we are able to analyze the influence of L2 hardware according to each AS. We deeply analyze several characteristics describing the L2 influence depending on the AS considered. We also provide keys to interpret the interaction between L2 and L3 nodes in order to model the bias of a L3 probing scheme such as a

traceroute campaign. Our results show that although each AS is topologically engineered in a different way, in all cases, the impact of L2 hardware is really significant. For instance, the tail of the node degree distribution is much more heavy using an L2 agnostic view than distinguishing the two layers. This result is partially explained by the fact that large access routers have a huge number of connections involving L2: there exists a strong correlation between the router point-to-point degree and the number of its L2 neighbors. Our results indicate that L2 hardware must be carefully taken into account when analyzing the topology properties. The remainder of this paper is organized as follows: Sec. II discusses how we collect topology data using mrinfo; Sec. III explains our router-to-AS mapping; Sec. IV evaluates our router-to-AS mapping algorithm; Sec. VI demonstrates the impact of L2 hardware on the node degree distribution; Sec. VII positions our work regarding the state of the art; finally, Sec. VIII concludes this paper by summarizing its main achievements and discussing further work.

3

II. C OLLECTION M ETHODOLOGY AND DATASET A. mrinfo In the late 1980s, the developers of IP Multicast designed the MBone, an overlay network composed of tunnels that interconnected workstations running an implementation of the Distance Vector Multicast Routing Protocol [10] (DVMRP). Several tools have been developed to monitor and debug the MBone [11]. Most of these tools have been deprecated with the replacement of DVMRP by the Protocol Independent Multicast (PIM) family of multicast routing protocols with one notable exception : mrinfo [5]. mrinfo messages use the Internet Group Management Protocol (IGMP, [12]). IGMP was initially designed to allow hosts to report their active multicast groups to a multicast router on their LAN. Most IGMP messages are sent with a Time-to-Live of 1. However, DVMRP has defined two special types of IGMP messages that can be used to monitor routers [10]. Although current IPv4 multicast routers do not use DVMRP anymore, they still support these special IGMP messages. Upon reception of an IGMP ASK NEIGHBORS message, an IPv4 multicast router will reply by sending an IGMP NEIGHBORS REPLY message that provides the list of all its local interfaces with some information about their state. Cisco and Juniper routers also report in the IGMP NEIGHBORS REPLY message the version of their operating system. Fig. 3 shows an example of the usage of mrinfo to query the router R2 (1.1.0.2 is the responding interface of R2 ). mrinfo reports that this router is directly connected to R0 (through interface 1.1.0.1) and to two AS border routers (ASBRs) R3 (through the interface 2.2.4.2) and R4 (through interface 2.2.1.2). We can also notice that R2 is connected to routers R5 and R6 through a switch because its interface 1.1.2.1 appears twice in the reply of R2 . This information is obtained by sending a single IGMP message. In practice, mrinfo provides similar information as a show command on the router’s command line interface. Compared to traceroute and its variants, mrinfo has both drawbacks and advantages. The main drawback of mrinfo is that it can only be used on routers having IPv4 multicast activated. IPv4 multicast is not always enabled in IP networks, but thanks to the deployment of video or television services that rely on IP multicast, more and more ISP networks have enabled multicast. IGMP, like ICMP which is used by traceroute, can be disabled or rate limited on routers. More and more operators filter or rate limit ICMP, but fortunately for our measurements not all of them are aware of this special usage of IGMP. The main advantage of mrinfo is that, in a single IGMP reply, a router lists all its multicast enabled interfaces, their IP addresses, and the IP addresses of its neighbor routers. Thus, mrinfo does not suffer from the alias resolution problems affecting traceroute [3]. Second, all multicast enabled links of a responding router are captured, even if the IGP weight of a link is high and no data packet are usually forwarded through it. Furthermore, the IGMP monitoring load is very small compared to traceroute. Indeed, with mrinfo it is possible to collect the topology of a multicast enabled network by

sending a single packet to each router. Standard traceroute is only able to discover a single path from the source to the destination. To discover more topology information, both the number of destinations and vantage points must be increase. However, standard systems that extensively trace the Internet, such as the recent Archipelago [13], mostly rely on a small set of sources (roughly 20 machines spread around the world). Paris Traceroute [14] is able to discover load balancing routers, as well as the set of paths joining those load balancing routers. However, this works mostly for intra-domain routers, BGP ECMP load balancing feature not being widely deployed. mrinfo is able to discover all links between routers from a single source if domains authorize multicast. In particular, mrinfo is able to report backup links inside and between domains. Intra-domain backup links are links configured with a high IGP weight. They are not used to forward data packets and thus cannot be discovered by traceroute. Interdomain backup links can be configured by using different types of BGP policies. A frequent policy is to attach a low BGP local-preference attribute to the routes learned from a backup link. For instance, on Fig. 3, if the link between R2 and R3 has a low local preference, it will never be discovered by traceroute-like exploration but mrinfo will report it.

B. Recursive Probing Our approach in probing the network with mrinfo is recursive and we call such a probing scheme mrinfo-rec. Initially, mrinfo-rec is fed with a single IP address corresponding to the first router attached to the mrinfo-rec vantage point. mrinfo-rec probes this router and recursively applies its probing mechanism on all the collected IP addresses. These recursive queries stop at unresponsive routers or when all discovered routers have been queried. The same process is run every day. It is worth to notice that an address not replying to an mrinfo probe during a given day will not be queried the days after except if it appears again in a list of captured addresses. To illustrate this behavior, let us apply it on the topology depicted in Fig. 3. mrinfo-rec receives, as input, the IP address of router R0 . From R0 , mrinfo-rec collects a set of neighbor IP addresses, i.e., {1.1.1.2, 1.1.0.2}. For all IP addresses in this set that were not previously probed, mrinfo-rec sends an IGMP ASK NEIGHBORS message and, if the probed router replied, it again runs through the set of neighbor IP addresses collected. This recursive probing scheme comes with the strong advantage of being very easy to setup as it initially requires only a single IP address as input. However, it has the drawback of limiting the mrinfo-rec probing spectrum. If the address set specified as input for a given day corresponds to non responding routers (because, simply, the ISP filters IGMP messages), mrinfo-rec will not be able to discover any topological information. To overcome this limitation, the set of responding router addresses of a given day is given as input of mrinfo-rec the next day.

4

(a) Interfaces Fig. 4.

(b) Routers

(c) ASes

General view on data collected by mrinfo

C. General View on the Dataset Since May 1st , 2004, we have been collecting topology data from a host located at the University of Strasbourg, France. In this paper, we consider the data collected until the end of December 2008. The entire dataset is publicly available [7] and has been collected using mrinfo-rec, as described in Sec. II-B. Note that the number of replies drastically decreased since November 2008 (during this month, there is almost no data at all and later three times less routers replied to mrinfo-rec). Indeed, this is due to the filtering policies of some providers. In particular, we notice that the research network GEANT blocks all IGMP request since this date. During the whole probing period, on average, mrinfo-rec was able to daily discover roughly 10,000 different routers while scanning 100,000 interfaces. For the remainder of this paper, we remove interfaces with nonpublicly routable IP addresses. The addresses that we consider as non-publicly routable are the special-use IPv4 addresses described in RFC 3330 [15]. Specifically, we eliminate the private IP address blocks 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. We also remove the loopback address block 127.0.0.0/8 and the 0.0.0.0 address. On average, 25% of the interfaces collected by mrinfo-rec are categorized as non-publicly routable. We also remove all multicast tunnel interfaces detected with the keyword ‘tunnel” in the mrinfo raw output. Fig. 4 provides general statistics on the mrinfo-rec dataset collected between 2004 and 2008. Regarding interfaces (Fig. 4(a)) and routers (Fig. 4(b)), the best period was between 2006 and 2007, when we identify roughly 110.000 interfaces and 10.000 routers. On the contrary, the global vision of mrinfo-rec seems to increase with time as the number of identified ASes (Fig. 4(c)) continues to grow with time. We roughly identify between 400 and 650 different ASes every day of mrinfo-rec probing and more than 850 distinct ASes during the whole probing period. Note that these numbers correspond to the IP to AS mapping performed on the whole set of discovered interfaces. In practice, half of those AS do not actually respond to mrinfo-rec, the average number of responding AS (at least one of its router replies) is approximatively 300.

Fig. 5.

IP-AS mapping issues over time

Indeed, although multicast enabled, some routers filter and/or do not repond to mrinfo requests. In particular, we can notice that the number of responding routers has drastically decreased since November 2008 because the GEANT network decides to filter mrinfo probes. We observe that more than one third of mrinfo-rec replies were collected through this network before this date. Although GEANT has been multicast enabled, since the begining of 2004, its administrators decided to deactivate routers compatibility with remote mrinfo requests, and finally decides to block all mrinfo requests since November 2008 (mrinfo requests are no more propagated by GEANT). This choice leads to the drastic reduction of responding routers shown in Fig. 4. Note that, in the future, we intend to bypass this kind of limitations due to filtering policies by using several vantage points. Fig. 4(c) is related to the IP-to-AS mapping. The IP-to-AS mapping is performed using the last BGP table dump of each day from Routeviews [16]. Note that we use a /24 prefix to perform the IP-to-AS mapping such that we assume that no /24 prefix is shared between several ASes. Fig. 5 shows issues encountered when performing this mapping. In particular, we found two types of issue: the mapping was impossible (curve labeled unmapped on Fig. 5) and multiple origined ASes [17] (curve labeled moas). While the proportion of unmapped IP addresses is relatively marginal (0.05% on average, sometimes even equal to zero), the problem of MOAS is more frequent (between 2 and 3% of IP addresses discovered each day by

5

R2

R4 AS1

AS1

R1

AS2

AS2

AS2

AS1 AS2

AS1

AS1

R6

AS2

AS2

AS2

R1

R7

R2

AS1

R5

AS2

AS2 R3

R4

AS2 R6

AS3

R5

R3

R7 Fig. 7.

AS3

Shared Addressing Space case on R1

mrinfo ), although it tends to decrease over time but with a larger variation, specially in 2008. Fig. 6(a) provides the cumulative distribution of the number of multiple originated ASes (horizontal axis). Note that this analysis has been performed on the whole period of probing. In the vast majority of MOAS cases, only two AS numbers are mapped to a given IP address (93%). The maximum number of ASes is 13 but this case is very rare (0.0008%). MOAS cases can be generated by configuration errors or standard anycast technique (see [17] for more details), but it is difficult to quantify each of the two subsets. Note that we have consider the announcement of an AS set as a particular MOAS case. An AS set is a normal procedure used to aggregate a list of several prefixes belonging to distinct domains in a single annoucement.list of several distinct prefixes belonging to distinct domains. However, the number of AS set cases that we found is equal or less than 1. Fig. 6(b) shows the cumulative distribution of the difference between the occurrences of two ASes for /24 prefixes generating MOAS. To obtain this difference, for a given /24 prefix, we consider the most frequent AS and the less frequent one during the whole period of probing and calculate the difference between these two. We believe this is reasonable because, as highlighted in Fig 6(a), the vast majority of IP addresses involving MOAS contain only two different ASes. We see on Fig. 6(b) that, in most of the cases (i.e., 88%), the difference is less or equal than two. Thus, most of those MOAS cases seems to be stable during the four probing years. We can argue that the majority of MOAS comes from standard configurations such as anycast. However, we cannot use this result to reduce the number of MOAS cases by choosing the most frequent AS between the two originated.

Fig. 8.

Neighborhood empirical rule, N

both directions of this link belong to AS2 . In practice, two cases must be considered. First, there exists ASBRs where all IP addresses do not belong to the same AS. In such a case, identifying them as ASBR is straightforward but assigning them an ASN is more difficult. This situation is illustrated in Fig. 7 where router R1 has two interfaces mapped to AS1 while the remaining two interfaces are mapped to AS2 . The resolution of this Shared Addressing Space case allows one to perform the router-to-AS mapping. Therefore, it is possible to extract intra-AS topologies without falsely cutting between ASes. We denote SAS the subset of ASBRs falling into the Shared Addressing Space case. Second, there are ASBRs where all IP addresses are mapped to the same ASN. If the router-to-AS mapping is obvious, identifying those routers as ASBRs is a different ball game: their detection essentially relies on their relationships with ASBRs belonging to SAS. This is illustrated in Fig. 7 with routers R4 and R5 as all their interfaces are mapped to AS2 . If R1 is correctly assigned to AS1 , then R4 and R5 are ASBRs mapped to the AS corresponding to the address space of links R1 ↔R4 and R1 ↔R5 . This issue is thus trivial if the SAS case has been previously correctly solved. At this point, it is already worth to notice (see Sec. IV for further details) that the vast majority (almost 90%) of routers were mapped directly to the right ASN because they do not belong to the SAS set. Thus, approximatively, only 10% of routers discovered in our dataset must be mapped by our router-to-AS algorithm.

III. ROUTER - TO -AS M APPING If it is easy to determine the ASN of a core router (each IP address of a router is mapped to the same ASN2 ), the challenge is to accurately identify a router as an ASBR and assign it the right ASN. Fig. 7 and 8 illustrate the basics of our routerto-AS algorithm. The label given on each link is the result of the IP-to-AS mapping (we assume that IP addresses on each directed link are necessarily mapped to the same AS, otherwise we ignore them). Regarding to the link between R3 and R4 in Fig. 8, it means that IP addresses involved in 2 Note that there exists some particular cases to distinguish core from border routers that are partially described in Sec. III-D.

A. Router to AS algorithm Our router-to-AS algorithm is based on two types of rules: probabilistic and empirical rules. The main idea behind our algorithm is to quickly converge to a single and consistent mapping for each router. For that purpose, our algorithm verifies the consistency of the results returned by each rule. We start by assigning a candidate ASN to each router. This is done using our first probabilistic rule (called “global election”, or elec). It works as follows: each router is mapped to the ASN assigned to the largest number of its IP addresses (using the IP addresses apparition order for tie-breaking equal-

6

(a) ASes list size distribution Fig. 6.

(b) modification

MOAS statistics

ity cases).3 Let Sr be the set of the occurrences of each IP-toAS mapping computed on addresses belonging to the router r. If r is initially mapped to AS n, it means that n appears max(Sr ) times in the IP-to-AS mapping of r. We associate a confidence level to the ASN mapped to r in such a way: c(r) = 1 −

max(Sr \ {max(Sr )}) . max(Sr )

(1)

Closer to one, higher the confidence in the mapping. Note that r ∈ SAS if and only if c(r) < 1. Looking at Fig. 7, we compute c(R4 ) = 1 whereas c(R1 ) = 0, we know that R1 is an ASBR because it belongs to the set SAS, but we have to figure out if it belongs to AS1 or AS2 and respectively if R2 , R3 or R4 , R5 are ASBRs. In contrast, in Fig. 8, c(R3 ) = 0.5 meaning that R3 seems to belong to AS1 . The elec rule is the primary block of all our analysis: other rules try to confirm or infirm it when c < 1. We use a threshold 0 < α < 1 to decide whether other rules can overwrite the candidate assignment (the result of elec). For all routers r ∈ SAS, if a given rule is not in concordance with the elec rule (i.e., the ASN returned by this rule differs from the candidate one given by elec), we select the ASN returned by the tested rule only if c(r) ≤ α. Otherwise, we ignore the result provided by the tested rule. In practice we choose α = 0.54 . Other rules are applied to every router r such that c(r) < 1 and their result may be: - leave the assigned ASN unchanged but perform c(r) ← 1 if the rule confirms the elec assignment. - change the assigned ASN and perform c(r) ← 1 if the rule contradicts the elec assignment with c(r) ≤ α. - nothing if the rule contradicts the elec assignment with c(r) > α. The second probabilistic rule relies on the detection of LAN interfaces. The lan rule concerns a subset of x → 0.0.0.0 interfaces. We remove “down” and “disabled” interfaces using a regular expression (those words appear between brackets in 3 The elec rule uses the order of IP address apparition in the list given by a mrinfo reply to tie-break equality cases. Thus, a router having two interfaces mapped to ASX and ASY respectively will be mapped by default to ASX if this ASN appears before ASY in the mrinfo list of interfaces reply. 4 It implies that there are at least two more IP addresses mapped to the candidate ASN compared to any other ASN.

mrinfo raw output). The remaining x → 0.0.0.0 interfaces usually describe leaf LAN without transit and multicast capabilities. If we assume that x is mapped to the right ASN n, the router-to-AS mapping is then easy to perform. To discard x → 0.0.0.0 describing transit inter-domain LAN interfaces without multicast capabilities, we remove x → 0.0.0.0 interfaces if there does not exist another routable interface whose resulting mapping is n. With this procedure5 , we try to remove interfaces that are connected to a non-PIM compliant router through a layer-2 switch. To refine our leaf LAN detection, we define four combinations of criteria based on the logical combination of the two following conditions: • (i) – the responding IP corresponds to a specific interface called pseudo-loopback, • (ii) – all LAN interfaces are mapped to the same AS (LAN consistency). If at least one of the conditions (i) or (ii) is true and returns AS n, then we assign AS n to r if AS(r) = n ∧ c(r) > 0.5. The pseudo-loopback assignment relies on a particular type of router reply: the IP x which responds to the mrinfo request (the IP assigned to the router) is listed as x → 0.0.0.0 in the output of its interfaces list. Then, this case means that x is a loopback interface, because x is not involved in the forwarding plane. In addition, as previously mentioned, we only consider a x → 0.0.0.0 interface as a pseudo-LAN if there exists another interface y → z attached to the same router such that x and y are mapped to the same AS and such that z 6= 0.0.0.0. Hence, it implies that we consider pseudo LAN interfaces only if the AS of x is multicast enabled. More details are given in Sec. III-B. As already mentioned, we also use empirical rules. The first empirical rule takes into account the loopback interface of a router (lb rule). When configuring a loopback interface, an ISP is supposed to use an address belonging to its own IP address space. Thus, identifying loopback interfaces using their DNS name and performing a standard IP-to-AS mapping on this address resolve the router-to-AS mapping. Moreover, we also use an empirical rule consisting in a neighborhood analysis (N rule): we assume that inter-domain 5 The last assumption we make is that a multicast enabled ASBR takes advantage of all its inter-domain connections.

7

links are mapped to the address space of one of the AS it interconnects6 . This rule can be applied when a SAS is mapped to a given AS thanks to another rule. For instance, in Fig. 8, let us assume that R3 is mapped to AS1 with the lb rule, then R4 necessarily belongs to AS2 and iteratively R7 is then mapped to AS3 . In practice, we apply this rule iteratively until no more new AS assignment is recorded and use it at each step of the router-to-AS mapping process. As already mentioned by claffy et al. [13], a provider generally allocates IP addresses from its own address space to its customers links (c2p rule). From a per link perspective, the c2p rule works as follow: for each interface a → b of a router r mapped to the ASX , let us consider that the b interface is attached to a router z mapped to ASY . Then, the interface a → b verifies the c2p rule if: (i) ASX is a customer of ASY , and if the mapping of the link (a → b) returns ASY . (ii) ASX is a provider of ASY , and if the mapping of the link (a → b) returns ASX . In a simple case, this means that if two routers denoted R1 and R2 are connected through an interdomain link mapped to ASx , and such that R1 also uses the address space allocated by another ASy (y 6= x) while ASy is a customer of the ASx , then R1 is mapped to ASy (and R2 to ASx ). In Fig 7, if we know that AS1 is a customer of AS2 , the c2p rule allows us to map R1 to AS1 . To perform such a relationship mapping, we use the AS ranking data set provided by C AIDA [18]. We verify the consistency of this rule according to different levels of assignment confidence produced with the elec rule. For each confidence level 0.8 ≥ α ≥ 0, we compute that 70% of c2p/p2c relations between ASes verify this rule of IP allocation. In practice, we use the c2p rule only if there are two ASes involved in the equality case, e.g., ASX or ASY . For any router r such that c(r) = 0, we assign r to the customer AS of the relation between ASX and ASY . This is our penultimate rule, so that it tie-breaks remaining equality cases when the rule can be used (c2p or p2c relationships between involved ASes). Finally, to perform the router-to-AS mapping we need a global order between our pool of rules to characterize the confidence we attribute to each of them. This order is depicted in Fig. 9. The notation Hβ stands for a β-confident assignment. Depending on the confidence threshold 0 < β < 1, if c(r) > β (for a given router r ∈ SAS mapped to n with the elec rule), Hβ maps definitively r to n by attributing to r a confidence level of 1. In order to take advantage of AS assignments produced by the decreasing level of confidence of our set of rules, we apply the neighboring rule after the application of each other rule. Sec. IV describes the consistency of the mapping we obtain using this ordered set of rules while the three following subsections detail some interesting features of mrinfo and of our analysis in general. 6 This is typically the case for direct peering or provider-customer links but it is subject to further verification for IXPs

B. mrinfo Raw Output and Missing Links and Routers Obviously, mrinfo-rec cannot discover all the Internet. Where is the boundary between the explored part and the un-explored one? First of all, mrinfo gives information on interfaces where IPv4 multicast is enabled. If IPv4 multicast is not activated on an interface, this interface does not show up at all, even if routers are reachable in unicast through this interface. If multicast is activated on some interface with address a, then one or several lines a → x appear in the mrinfo output. If a PIM router having an address b is connected through a, then a line a → b shows up in the mrinfo raw output. However if there are IP routers connected through this interface, but multicast is not enabled on them, then the output of mrinfo is a → 0.0.0.0 at least for Ethernet and point to point interfaces. To validate this behavior we have performed several tests on cisco routers using ios 12.4. Therefore a line a → 0.0.0.0 may have 2 meanings: either a is an interface to a local LAN without any router, or a is an interface to other routers without IPv4 multicast. Therefore the boundary between multicast enabled networks and non multicast enabled networks is basically invisible. Note that the Internet exploration by mrinfo-rec may also stop at routers that do not reply to mrinfo queries, although they have multicast enabled. In this case, we see a line a → b in some router output, but b dos not respond to mrinfo queries. The boundary between explored and unexplored cuts through the link a → b.

C. DNS Names Using DNS names we can also capture some useful information about peering links. For instance, if the DNS resolution of a link gives us two DNS names that belong to different domains, we can deduce that the link is an inter-domain one even if IP addresses are mapped to the same ASes. For instance, we can deduce that a router is an ASBR even if it does not belong to the set SAS. We can use DNS name in the same way that we perform the IP-to-AS mapping. The difference is that the DNS-to-AS mapping is subject to more manual configuration error and is not always provided in mrinfo-rec replies. However, the main interest is that the suffix of the DNS name given on the two sides of a link can differ (on the contrary to IP addresses that are mapped to the same AS). Hence, DNS name can provide complementary information compared to the usual IP-to-AS mapping. Indeed, generally, IP addresses on both sides of a link are mapped to the same AS, but it is not necessarily the case with DNS names. A concrete example is the inter-domain link 209.244.160.154 ↔ 209.244.160.153 between Sprint and Level 3 (ASN1239 and ASN 3356). The DNS name resolution gives us the following translation: sl-bb20-nyc.sprintlink.net ↔ so-6-2.core1.NewYork1.Level3.net, we notice that although the two IP addresses on both side of the link are in the same /30 (this address space belongs to Sprint), the DNS names resolution allows us to determine the actual nature of this link.

8

elec

Fig. 9.

lb

N0

lan

N1

Hβ ∧ Ni

c2p

N11

Global order for router-to-AS mapping rules

In addition, using a regular expression such as “border” or “core” on DNS names, we can also capture some interesting information about the mapping of each router. As future work, we envision to use DNS information to tie-break cases that are not easily solved with our router-to-AS algorithm. For instance, when other rules contradict the elec rule more than once. However, a DNS parser resolving such cases is not trivial to develop because there is no standardized rules to attribute DNS names. Each ISP can follow its own guideline.

r: a→b c→d e (asy-loopback.net) → f Fig. 10.

Overwriting rule - a case study

Fig. 11.

The efficiency of the elec

D. Particular Cases and ASBR/Core Nodes Identification There exist many specific cases that can generate false ASN assignments. The use of a threshold α > 0 allows one to eliminate assignments producing strong inconsistency with the elec rule. A case that we encountered relatively frequently is the use of IP addresses from the space of an AS provider in the core of its customer AS. Thus, in practice, the boundary between ASes is not always as clean as expected. As an example, we found that the prefix 194.199.0.0/16 is announced via BGP as part of AS2200 (the Renater7 transit AS). However with mrinfo-rec we found that two routers of AS2259 (Osiris, a stub AS, client of AS2200) are interconnected to each other by a link in 194.199.215.0/24. One could wrongly conclude that these two routers are ASBR connecting AS2200 to AS2259 (with possibly other ASBR from AS2200 not showing up). However this is not the case. Actually 194.199.215.0/24 is sort of an isolated part of the larger prefix 194.199.0.0/16, reachable only through AS2259. Note that our AS extraction algorithm determines the nature of a router (e.g., ASBR or core router) according to its neighborhood. In other terms, a router mapped to ASX is an ASBR if, and only if, one of its neighbor belongs to ASY with X 6= Y . Moreover, the neighboring N rule may produce false assignment when the IP address of a link connecting two ASes is not taken among the address space of one of the involved ASes. Apparently, this case is not so rare, but unfortunately we cannot quantify accurately the number of occurrence of such an issue. We decide to discard changes not verifying the threshold α in order to increase consistency. A rule can overwrite the choice of the global election, elec, only if the number of interfaces mapped to the candidate ASN is less or equal than two times the number of occurrence of the ASN returned by the elec rule. In particular, if a router r has three interfaces, two mapped to ASX and one mapped to ASY . A rule such as lb can change the ASN assigned to r. Let us consider the situation depicted in Fig. 10. If a and c are mapped to the same ASX whereas e is mapped 7 Renater and Osiris are french research network. mrinfo-rec queries always goes through those two ASes to probe the remainder of routers.

to another ASY , then if the DNS name of e indicates a loopback configuration, we assign r to the ASY . Note that the vast majority of complex SAS cases involved routers with a low number of interfaces. This observation lead us to set our threshold to α = 0.5. IV. ROUTER - TO -AS M APPING E VALUATION A. Router-to-AS Mapping Efficiency From our four years daily dataset, we arbitrarily select the largest mrinfo-rec file of each month, leading to 56 files. We evaluate our router-to-AS algorithm on those files. Fig. 11 provides the cumulative distribution of the confidence level c (the horizontal axis) assigned during the first step (elec) of the router-to-AS mapping. The first observation is that, on average, 90% of the routers have addresses advertised by a single AS. We identify that roughly only 1.5% of the routers have a confidence level equal to 0, whereas more than 95% of the routers have a confidence level higher than 0.5. According to our threshold α = 0.5, only 5% of the router assignments are really problematic. Fig. 12 shows the cumulative distribution of the required number of rules (the horizontal axis) to converge to a positive decision, i.e., c = 1, in the router-to-AS mapping algorithm. On this figure, we observe that the decision process converges quickly: more than half of the SAS cases are treated after the lan probabilistic rule (i.e., 4th rule). This means that a large subset of the problematic cases are treated at the beginning of

9

Fig. 12.

Router-to-AS mapping convergence time

Fig. 13.

A closer look at each step

our set of rules which are ordered depending on the confidence we attribute to each of them. At the end of the process, using the N11 rule, we can notice that only 0.46% of the routers remain unmapped. Fig. 13 gives a more detailed overview of the actions taken during the convergence of our algorithm. Each point represents the mean over the 56 files we consider. We determined 95% confidence intervals for the mean but those intervals are typically too tight to appear on Fig. 13. The number of changes recorded during our router-to-AS algorithm works as follows: if the change computed on a router r is inconsistent (c(r) > α) with the elec rule we do not record it; but we increment the counter each time an inconsistency is produced by a given rule. The use of this particular counter means that on a given router r, we count all the inconsistencies generated by one or another rule. Thus, the number of routers being an issue for the mapping is lower than 6.4% of the cases changec>0.5 showed in Fig. 13 (The sum of the three curves is greater than 100% because of routers producing inconsistencies). We can retrieve the number of routers producing one or more inconsistencies by computing 100 − |conf irmation| − |changec≤0.5 |. Indeed, these two operations (confirmation and changec≤0.5 ) are executed at most once per router. The actual proportion of routers producing inconsistent cases is 5.5% of the SAS set, in other words only 0.55% on the whole data set. We can see that in most cases, our pool of rules confirms the candidate assignment of the global election, elec: at the

end of the process, 88.9% of the candidate assignments are confirmed by one or another rule. We also count the number of contradictions produced by our set of rules against elec, and divide them in two categories according to our α threshold. We notice that on average, 12% of the AS assignments are concerned, and mainly when c > 0.5 (6.4% compared to 5.6% when c ≤ 0.5). However, using our threshold α = 0.5, we only effectively record the 5.6% of changes that are not strongly inconsistent (c ≤ 0.5) to stay consistent with the elec rule. We have also noticed that fewer than 6% of routers in the SAS set are subject to real inconsistencies. Note that the interactions between our pool of rules must be carefully interpreted because of its complexity. In particular, we can focus on the correlation between the neighboring rule Nj and the previously applied rule. For instance, the number of cases falling in each categories (confirm, changec>0.5 and changec≤0.5 ) during a given step is correlated with previously applied rules. In particular, there is a strong correlation between the Nj rule and the last one applied. Indeed, the initialization of this rule is fed by the changes and confirmations produced by the last rule used. Thus, the number of confirmations, changec>0.5 and changec≤0.5 produced by the Nj rule is dependent on the changes (and so on their reliability) produced by the last rule applied. In Fig. 13, we show that the number of potential changec>0.5 required by N1 results both from the rule itself (and its associated bias) and from the lan rule wrong assignments. Indeed, the lan rule is the rule that is the most popular in the sense that lan can be applied on a large number of routers falling in the SAS set. Hence, false entries generated with lan induce wrong entries generated by N1 . To summarize, we have seen that 90% of the routers are assigned, directly at the first step of the algorithm, to an ASN with the highest level of confidence. The remaining 10% of routers belongs to SAS and represent thus critical cases. Our algorithm is able to quickly solve (after the 4th step) the vast majority of those cases. At the end of the process, only 0.46% of the routers remain unmapped. In addition, there are 5.5% of the SAS set assignments that are subject to further validations (for example using DNS names), because their AS assignments produce inconsistencies with the elec rule. Thus, only 1% of assignments are problematic. B. Point-to-Point Links and Switches In addition to our router-to-AS algorithm we also provide a way to distinguish point-to-point links from switch interconnections. This point is of the highest importance since it provides more information on the real network connectivity. Using traceroute-like probing, switches are not easily detectable and this bias leads to produce false interpretations: a set of nodes may appear to be fully meshed whereas they are actually connected through a simple switch. Identifying switches in mrinfo output is straightforward as it is enough to capture outgoing IP addresses appearing several times on the same router (see interface 1.1.0.3 on router R2 in Fig. 3). Note that a switch inter-connection discovered with mrinfo-rec can hide a switch cascade, i.e., several switches

10

AS

Tier-1

Transit

Stub

Sprint (ASN1239) Level 3 (ASN3356) N TTC -G IN (ASN2914) GlobalCrossing (ASN3549) Alternet (ASN701) T DC data network (ASN3292) TelecomItalia (ASN3269) IUnet (ASN1267) D FN -I PX -W IN (ASN680) JANETUK (ASN786) CiscoSystemsEU (ASN109) U NINETT (ASN224) G ARRItalian (ASN137) Wisconsin (ASN59)

Max 678 286 194 188 99 1116 931 571 366 336 250 243 153 64

Routers Mean 636 ±24 249 ±11 142 ±19 87 ±11 67 ±9 627 ±115 589 ±102 369 ±64 265 ±21 241 ±20 72 ±20 196 ±12 115 ±7 44 ±6

Max 68 170 48 3 13 152 86 305 84 23 5 6 9 6

Switches Mean 62 ±2 121 ±8 22 ±3 0 ±0 2 ±1 82 ±16 53 ±9 161 ±28 44 ±8 15 ±1 3 ±1 4 ±1 6 ±1 2 ±1

TABLE I I NTRA - DOMAIN TOPOLOGIES DIMENSIONS (# ROUTERS & # SWITCHES )

(a) switch and routers Fig. 15. switches

(b) core vs. border Fig. 14.

Switches and routers proportion over the 56 weeks

might be connected together. It can also hide some other type of level-2 inter-connections. Moreover, when possible, we verify that all routers connected through a switch share the same vision of the inter-connection (e.g., one IP address pointing towards the same set of addresses). As previously mentioned (see Fig. 3), replies collected with mrinfo-rec allow us to easily discover switch pseudonodes and extract them from our raw data. Fig. 14 provides the distribution of switches and routers over the 56 weeks. On average, we identify that 11% of nodes discovered in the network are switches (or cascade of switches), while

Proportion of routers connected through point-to-point links and

the remaining 89% are actual routers. In practice, we can discover only switches connecting at least three routers, but this is the interesting case in order to find multiple level-3 connections using the same level-2 adjacency. Note that the same distribution occurs when we distinguish inter-domain from intra-domain connections. Only 1% of the whole set of discovered nodes are inter-domain switches whereas 9% of them are inter-domain routers (the entry/exit points for mrinfo-rec probes). This distribution is not uniform among the topologies that we extract from the mrinfo-rec dataset. This is an important feature of our algorithm as it provides a more accurate view of the topology compared to traceroutelike probing. Fig. 15 depicts the proportion of routers connected through point-to-point links and switches. Roughly, we can see that 20% of the routers are interconnected through switches. Therefore, this proportion is strongly significant and we cannot ignore this kind of connectivity. Moreover, this is necessarily a lower bound because there may exists router pairs connected through layer-2 switches that we cannot distinguish from point to point links. Our switches analysis highlights the drawback of traceroute or layer-3 probing. Indeed, if one is interested in physical diversity analysis, this bias may lead to false conclusions: the network may appear much more connected than it really is.

11

C. Topologies

(a) point-to-point

Based on a four years daily dataset, we select fourteen different ASes belonging to three different hierarchical levels (Tier-1, Transit, and Stub ASes) and obtain, for each of these ASes, one intra-domain topology per month. The topologies selection process is based, among other, on the volume of data collected during the four years of probing. First, we select the days where the maximum volume of data has been collected during each month, and then, according to these specific days, we select the biggest topologies fallen in each category (in terms of data collected). These topologies are freely available under various formats.8 The choice of those ASes is also based on their frequency of apparition in the whole mrinfo dataset, their dimensions (e.g, #nodes, #edges), and their number of connected parts (as well as their respective dimensions). Note that we have not discarded small non connected components. Table I provides an insight of the fourteen ASes size in terms of number of nodes discovered (i.e., switches and routers). We provide the maximum and the mean of the number of routers and switches (with the confidence interval) over the 56 studied weeks. V. C ONNECTED C OMPONENTS

(b) switches Fig. 16.

Prefix length distribution

Note that there exists a small subset of point-to-point links where IP addresses of both extremities are not in the same /24 or, even worse, they are not mapped to the same AS. We simply discard links falling within the second category (and so on in the first) because we argue that they are pseudo tunnel interfaces even if mrinfo-rec does not discover them as tunnel (e.g., the word “tunnel” does not appear between brackets in the mrinfo raw output). These links are not interesting in the sense that they are not involved in the actual and physical underlying topology. We compute the distribution of the size of the minimal subnet mask covering the addresses of the two end points of a link. Fig. 16(a) shows that more than 99% of point-to-point links uses a subnet mask smaller than /24 and more than 95% use a subnet mask smaller than /28. In the same way (look at Fig. 16(b)), we also plot the distribution of the minimal subnet mask covering all IP addresses attached to each switch. We can notice that in the vast majority of the cases (roughly 90%), the size of the prefixes is very small: usually /31 or /30 for point-to-point links and smaller than /24 for switches. In a small number of cases the minimal subnet mask may yield a prefix shorter that /24 or even /16. This is probably an anomaly, possibly explained by the use of a tunneling or a layer 2 WAN technologies (ATM, frame relay...). However, the small occurrence of this observation reenforces the credibility of our analysis because this distribution seems realistic and in concordance with our analysis.

Fig. 17 shows the evolution of connected components over the 56 weeks for our selected networks. Each of the three plots is composed of two parts. The left vertical axis provides the size of each connected component and must be read in conjunction with stacked bars. The right vertical axis gives the quantity of connected component plotted in a stacked bar and must be read with the line. Studying how connected components in a given ISP evolve over time allow us to evaluate how accurate is the mrinfo vision of the network and how mrinfo probes are processed by the network. Fig. 17 shows the evolution of connected components for three ISPs, at different level (Tier-1, Transit, and Stub). For a tier-1 ISP (Fig. 17(a)) the mrinfo-rec coverage degrades with time. While at the beginning we were able to capture a single large connected component (more than 700 connected nodes), the number of connected components starts exploding in 2008: up to 38 connected components, a lot of them being made of only 2 nodes. This situation might be explained in two steps. First, the filtering policies of ISPs (the tier-1 itself, or ISPs along the path) has evolved and IGMP packets are more frequently discarded. Second, an hardware update in the tier-1 network. We know that C ISCO recommends to restraint some multicast capabilities. Most recent and expensive routers, such as the crs ones, seems by default to turn down the reply to IGMP ASK NEIGHBORS message. The situation for a Transit network (Fig. 17(b)) seems to be quite stable over time, except two peaks in the number of connected components seen. Those peaks might be explained by temporary configurations somewhere in the network leading to a strong filtering of IGMP packets. 8 See


12

(a) tier-1 (AS1239 - Sprint) Fig. 17.

(b) transit (AS1267 - IUnet)

(c) stub (AS137 - GarrItalian)

Connected components evolution over time

The Stub network (Fig. 17(c)) is roughly disconnected over time but the number of components is reasonably low (between three and five, on average). However, we notice the presence of a reasonably large connected component (around 100 nodes). VI. H OW L2 HARDWARE IMPACTS THE L3

Second, the validation can be done “theoretically” based on three rules: •

NODE DEGREE

DISTRIBUTION

In the remainder of the paper, we focus on the relative difference between an L3 and an L2 vision of the network. We do not claim to provide the actual degree distribution: our goal is rather to emphasize on the difference in the apparent connectivity when adopting a given vision of the network. Hence, using the same set of IP addresses, we construct two graphs, one corresponding to a traceroute like view and the other to the mrinfo inferred view. Obviously, our analysis is based on a partial view of the multicast Internet graph. Furthermore, we restrict the two constructed topologies to their connected and responding components, i.e., we remove all links not connected to a router responding to mrinfo so that we guarantee a set of active and existing links and routers. We assume that this partial vision does not impact the relative comparison although it may introduce some non coherent results in an absolute analysis, e.g., a large proportion of leaf routers.

•

•

A. Layer-2 Inference Technique In the analysis provided in this paper, L2 hardware inference is critical. In Sec. II, we have seen how an L2 hardware presence is suggested from raw mrinfo output. Here, we check whether this inference technique is accurate and what kind of technology it seems to reveal. Indeed, it may exist several kind of L2 hardware such ATM clouds or a simple broadcast switches. In our analysis, we particularly focus on the latter. There are several ways to validate our L2 inference technique. First, one could use small testbeds and access networks whose administrators give us the physical map. We performed such tests in our local AS in Strasbourg and found that our inference is correct in revealing switches.

symmetry rule. We verify the view of each router involved in the interconnection. For instance, on Fig. 3, we are able to detect that router R2 is connected to R5 and R6 through an L2 hardware. When probing R5 and R6 with mrinfo, the L2 interconnection to R2 must also be visible in the mrinfo output. Such a symmetry rule suggests the detection of “broadcast L2 hardware” such as a switch. querier rule. In a normal case, only one router per interconnection must be tagged as the IGMP “querier” (i.e., it wins the querier election on the subnet). In the case depicted in Fig. 3, as interface 1.1.2.1 is tagged as “querier”, interfaces 1.1.2.3 of R5 and 1.1.2.2 of R6 should not be referenced as such. subnet mask rule. We verify the validity of the minimum subnet mask covering all IPs involved in the connection. Fig. 16 provides the cumulative distribution of subnet length for point-to-point and point-to-multipoint connections. We notice that in the vast majority of the cases (roughly 90%), the size of the prefix is very small: usually smaller than /24 for point-to-multipoint and using a /31 or /30 prefix for point-to-point links. Fig. 16 shows that more than 99% of point-to-point links (plain line) uses a subnet mask smaller than /24 and more than 95% uses a subnet mask smaller than /28. In a small number of cases the minimal subnet mask may yield a prefix shorter that /24 or even /16. This is probably an anomaly, possibly explained by the use of a tunneling or a L2 WAN technologies (ATM, frame relay, . . . ). We consider a /16 threshold for the minimal subnet mask rule applied to an L2 hardware.

Note that the symmetry and the querier rule need at least two L3 nodes connected to the same L2 hardware to work. Based on those three rules, we decide to classify a given L2 inference in one of those three categories: •

coherent. In this case, the three rules are verified and we are sure that our L2 inference technique is accurate and suggests a switch. This is the best case.

13

Notation S R S N =R S succ(a) r(x) s(x) t(x) = r(x) + s(x) p(x) b(x) r ′ (x)

Meaning set of L2 nodes set of L3 nodes set of nodes (including L2 inferred ones) set of neighbors of node a number of L3 nodes of degree x number of L2 nodes of degree x number of nodes of degree x number of L3 nodes having a degree of x and at least one neighbor in S number of L3 nodes having x neighbors in S number of routers having a logical L3 degree of x in the L3 graph TABLE II

Fig. 18. L2 inference rule verification with a /16 threshold for the subnet mask rule

incoherent. In this case, at least one of the three rules is not verified. This category gathers L2 technologies that we are not able to accurately reveal. • incomplete. In this case, the L2 component is only seen by a single L3 hardware (the symmetry and querier rules cannot be applied). Fig. 18 shows how our L2 inference technique verifies the three prescribed rules based on the three cases. We did the evaluation on the 56 selected months. Over the 56 selected months, on average, we have the following results: 90.16% of the L2 inferred nodes are coherent and suggest broadcast hardware such as switches. 5.84% of the inferred nodes are incomplete, meaning that a low 4% are incoherent. Thus, we notice that in the vast majority of the cases, our L2 inference mechanism seems to be verified and coherent. In the remainder of our analysis, we decide to ignore L2 components falling into incoherent or incomplete categories. Note that those problematic cases are not uniformly distributed across the domains and time: for example, we notice that a large proportion of those case comes from AS680 and AS5400 and seems to progressively and slightly disappear. We argue that those cases generally imply “L2 circuits technologies” such as ATM. In the following, we highlight the drawback resulting from an L3 probing such as traceroute. Indeed, we demonstrate that a bias may arise due to a restricted vision not taking into account L2 technologies: the network may appear much more connected than it really is. •

B. Notations Table II provides basic definitions and notations used in the remainder of this paper. In our analysis, the notion of degree is related to the L2 graph. In contrast, we use the term logical layer-3 degree to refer to the number of edges seen by an L3 tool such as traceroute. A node n has a logical L3 degree of x if and only if degr (n) + degs (n) = x.

(2)

Furthermore, we consider the IP multigraph issue: the degree of a node is the number of neighbor IP addresses.

N OTATION SUMMARY

Typically, an L2 node refers to a broadcast hardware such as a switch, while an L3 node corresponds to an IP router. In an L2 aware graph, there exist two types of connection: L3 point-to-point connections that are correctly interpreted by traceroute, and L2 point-to-multipoint connections that are incorrectly seen as multiple point-to-point connections using traceroute. Our goal is to measure the importance of this false interpretation using the node degree distribution as an indicator to understand the difference between the L3 and L2 graphs. Note that one may also use other indicators such as the clustering coefficient, the assortativity, or a path diversity indicator. Notations degr (a) and degs (a) respectively refer to the L3 point-to-point degree of node a, and to the L2 point-tomultipoint degree of a such that: degs (a) =

X

deg(v) − 1.

(3)

∀v∈succ(a) | v∈S

Eqn. 3 is the key point of our analysis: L2 point-tomultipoint connections between a and its neighbors are seen as full meshes using an L2 agnostic view, whereas using mrinfo, we can infer stars centered on L2 nodes instead of cliques. When parsing our dataset, we generate an L2 aware topology containing inferred L2 nodes and links used to reveal the underlying topology. The set of nodes in this L2 inferred graph is denoted N . Except considering the use of technologies such as V LAN, we can assume that this vision describes the L2 topology. Let us assume an L2 node connected to m routers (2m directed links in the L2 graph), then it induces m×(m−1) directed links in the L3 graph. With the use of V LAN, let us now consider k V LANs deployed on a L2 node connecting m routers (at most 2km directed links in the L2 graph instead of 2m in the physical topology), then, in the worst case, it can induce km × (m − 1) directed links in the L3 graph. Note that we provide a lower bound on the bias resulting from a L3 graph when considering the physical topology: 2m < 2km ≤ km × (m − 1), ∀k > 1, m ≥ 3. Moreover, our analysis is conservative because mrinfo underestimate the number of

14

Fig. 19.

Node Degree Distribution Comparison

are translated to full mesh connections. Consequently, it also means that the shape parameter used to model the power law distribution of degree and the degree of connectivity in general are strongly biased if one wants to consider a physical view of a given network (which is generally the case when measuring path diversity characteristics for example). A first attempt to model the high degree shifting of an L3 view is to consider that a router connected to an L2 node “will increase” its L3 logical degree by the number of neighbors seen through that L2 node. With a naive model considering that, in the vast majority of cases, L2 nodes interconnect leaf routers (e.g., routers of degree 1, so that distributions p(x) and b(x) -see Table II- are meaningless), we may obtain:  r(x) + s(x + 1) × x if x > 1, X r′ (x) ≈ r(1) − (4) deg(s) otherwise.  s∈S

active IP interfaces: the probing view is restricted to multicast enabled links and mrinfo replies may be fragmented and then only partially interpreted. See Sec. V for details. C. Global Vision In this section, we focus on the node degree distribution considering L2 hardware. In particular, we would like to know how L2 components might impact the degree distribution. The purpose of our study is to: (1) explain why there exist nodes with very high degree in a L3 graph, and (2) highlight the huge difference between graphs describing L2 and L3 connections. Using our L2 inference technique, we can easily compute r(x) (i.e., the number of L3 nodes of degree x - see Table II) and s(x) (i.e., the number of L2 nodes of degree x - see Table II). In contrast, with an agnostic view of L2 such as with traceroute, the global degree distribution is purely logical: we can only compute r′ (x) (i.e., the number of routers having a logical L3 degree of x - see Table II). Hence, using our mrinfo dataset, we can easily emphasize the difference between t(x) (i.e., the sum of r(x) and s(x) - see Table II) and r′ (x) because we are able to compute all key distributions. Fig. 19 illustrates the difference between those two distributions.9 The horizontal axis gives the degree, while the vertical axis shows the pdf. The shaded area represents the t(x) distribution while the line refers to the r′ (x) distribution. Note that the maximum degree for r′ x(x) is 115 but it is too tight (0.01%) to appear on the plot. We can notice that the degree distribution is strongly impacted using only a L3 perspective: the L2 agnostic distribution has a tail much more heavy than the one considering both L2 and L3 nodes. For example, using the L2 view (t(x)), we observe that there do not exist routers having a degree greater than 4510 whereas the L3 views (r′ (x)) suggests routers having degree greater than 80. This simply means that there exists much more nodes of large degree when L2 interconnections 9 For the sake of clarity, we select only one of our 56 global topologies, the one being the most connected (with the maximum number of edges). However, note that the general behavior is roughly identical across our set of global topologies. 10 And it probably implies the use of VLANs.

However, we can easily notice that this model is not sufficient to explain the heavy tail of the L3 distribution. Indeed, we can observe that obtaining such an amount of nodes having such a high degree requires point-to-multipoint connections through multiple L2 nodes. We will detail this issue in the next section considering a closer look at each key distribution (e.g., the L2 nodes degree distribution and the number of L2 neighbors) and according to a given domain. D. Domain Specificity In Sec. VI-C, we have seen that the impact of L2 hardware on a characteristic such that the node degree distribution is considerable. This analysis was done on the whole Internet, without considering the potential particularities of domains. In this section, we refine our analysis, focusing on several domains of interest. In addition, we want to know whether there exist some common behaviors between domains. We first discuss our methodology in extracting and selecting particular domains from the raw mrinfo dataset (Sec. VID.1) and, next, analyze the impact of L2 hardware on the node degree distribution of selected domains (Sec. VI-D.2). 1) Methodology: For each of the 56 selected mrinfo files, we extract four particular intra-domain topologies. This extraction is made possible with our router-to-AS mapping algorithm (see Sec. III). It comes for the router-to-AS mapping execution a set of intra-domain topologies that can be classified as follows: Tier1 (such as Sprint), Transit (such as TDC), and Stub networks (such as UNINETT). Those one-per-month topologies are freely available in various formats.11 In the remaining of the paper, we only consider four ASes taken from the set of extracted topologies, two being Tier1 (AS1239 and AS3356) and two being Transit (AS1267 and AS3269). We select those ASes as they are the largest topologies in our dataset. Furthermore, they are very well connected and seem more impacted by the presence of L2 compared to the global vision provided by the whole Internet (see Tab. III). Note that we remove inter-domain links from 11 See


15

Level Global Tier-1 Transit

ASN 1239 3356 1267 3269

Name Sprint Level-3 IUnet Telecom Italia

|R| 8,670 ±448 649 ±7 254 ±6 517 ±16 822 ±28

|S| 1,100 63 123 225 73

α ±63 ±0 ±7 ±10 ±2

0.38 0.58 0.67 0.77 0.69

±0.01 ±0.01 ±0.03 ±0.04 ±0.02

β − L3 0.69 ±0.01 0.70 ±0.01 0.87 ±0.01 0.85 ±0.03 0.95 ±0.01

β − L2 0.34 ±0.01 0.34 ±0.01 0.69 ±0.01 0.74 ±0.03 0.67 ±0.01

TABLE III S ELECTED AS ES GLOBAL OVERVIEW

those topologies reducing so the degree of connectivity of those ASes. Figures for other ASes are in appendix of this report. Table III provides an overview of the amount of routers and L2 hardware in our dataset for the selected ASes. Values have been averaged over the 56 considered mrinfo files. The confidence intervals are also given. In addition, Table III provides three ratios: (α) the proportion of routers connected to at least one L2 node in the L2 graph, and (β −L3 and β − L2) the proportion of edges using an L2 hardware. Note that β − L3 refers to the quantity of links using an L2 point-to-multipoint hardware in the L3 graph whereas β − L3 refers to the same quantity in the L2 graph. For the remainder of the analysis and for the sake of clarity, we select only one probing day per AS according to the size of the largest connected component inside each AS. Sec. V detailed this process: intuitively, we simply extract the best topology for each AS. 2) Experimental results: First, as shown in Table III, the proportion of L2 hardware is generally quite significant. On the whole dataset, approximatively 11.2% of nodes are inferred as L2 nodes. It is worth to notice that this proportion is not uniformly distributed across ASes. For instance, some large ASes, such as AS1267 or AS3356, have |S| ≈ 0.3 × |N |. This high proportion can obviously lead to an important difference when considering the connectivity degree in L2 and L3 graphs. However, another aspect impacts this potential difference: if the degree of L2 nodes is high, it means that a large number of routers are involved in L2 connections. For example, we can notice that, although less than 10% of nodes in AS3269 are L2 hardware, about 97% of connections between routers modeled by a L3 graph actually implies L2 hardware. This phenomenon is due to the fact that L2 nodes interconnect a large number of routers. Note that the number of links is much more important in the L3 graph than in the L2 one when most of L2 nodes involves more than three routers. In the fashion of Fig 19, Fig. 20 provides a comparison of the node degree distribution when L2 nodes are taken into account (t(x) - see Table II) and when L2 nodes are ignored by an L3 only vision (r′ (x)) for the four considered ASes. The line labeled “correlation” will be discussed later (see Sec. VIE). As mentioned in Sec. VI-C, we notice a great shifting in the distribution of large degrees. However, we observe that this difference significantly varies from one AS to another. For example, Level 3 is much more impacted than Sprint. In all cases, we can notice that the model given in Eqn. 4 cannot describe such a degree shifting. For instance, even for

Sprint, the maximum router degree in the topology is 21 (see Fig. 21(a)). And if we assume that this router is connected to the largest L2 node of degree 23 (see Fig. 21(b)), we may obtain a L3 logical degree of 21+23−1 = 45 whereas it exists a router having 47 neighbors in the L3 topology. Note that the Sprint topology given in the Rocketfuel dataset [4] presents approximatively the same symptomatic distribution as the L3 one (r′ (x)) given in Fig. 20(a). The bias resulting from an L2 agnostic view obviously depends on the s(x) distribution, but also on the way L2 and L3 nodes are interconnected. Indeed, we can notice that the S is not the only factor explaining this bias. Fig. 20 ratio N shows completely different t(x) and r′ (x) distributions even for topologies with less than 10% of L2 hardware, such as Sprint and TDC Data. This means that if one tries to model an L2 aware degree distribution with a power law, one must carefully consider the shape parameter to use. In practice, two other distributions could emphasize the influence of L2 hardware: p(x) (i.e., the number of L3 nodes n having a degree of x | degs (n) > 0 - see Table II), and b(x) (i.e., number of L3 nodes having x L2 neighbors - see Table II). Fig. 21(a), Fig. 21(c), Fig. 21(d), and Fig. 21(b) illustrate some key behaviors useful to understand such a difference between t(x) and r′ (x). First of all, we can notice among our set of selected ASes illustrated on Fig. 21 that there is no significant common engineering way to construct a network. It highly depends on economics and technologies factors not captured with classical models. Compared to the global analysis depicted in Fig. 19, we can highlight the difficulty to provide a model capturing the behavior of each domain specifically. Second, focusing on common properties, we can observe that r and p are slightly different distributions: keeping in mind that most of routers counted in the r distribution also fall in p, we can emphasize the fact that peak values in r becomes still more important in p. For example, on AS1239, it means that globally 40% of routers have a degree of 1 while focusing on routers connected to L2 hardware, this proportion grows to almost 70%. In practice, on AS1239, we can conclude that the majority of routers connected to L2 nodes are leaf nodes. However, this observation is highly topology dependent. The only common characteristic being the increase of the same peak values for each AS. Those results explain why the difference between t and r′ distributions shown in Fig. 20 is so high for most representative degree values. Fig. 21(b) and Fig. 21(d) plot the degree distribution of L2 nodes, s(x), and the distribution of number of L2 neighbors for routers, b(x). First, we can notice on Fig. 21(d) that the

16

(a) AS 1239: Sprint Fig. 20.

(c) AS 3269: Telecom Italia

(d) AS 3356: Level-3

(c) Routers having L2 neighbors

(d) Number of L2 neighbors

Degree distribution comparison for our set of selected ASes

(a) L3 node degree distribution Fig. 21.

(b) AS 1267: IUnet

(b) L2 node degree distribution

Keys distributions for our set of selected ASes

power law modeling the L2 nodes distribution is again highly dependent on the topology and not equivalent to the one modeling r(x) for each given topology. The maximum degree of s(x) plays a critical role in the absolute difference between t(x) and r′ (x) but Fig. 21(c) highlights the higher importance of b(x). First, we can see on Fig. 21(d) that most of L3 nodes are connected to L2 hardware. Only 10% for AS3356 and 40% for AS1239 of routers are pure point-to-point routers not using any L2 technology. Moreover, we observe that, for topologies such as AS1267 and AS3269, most of routers are not connected to a single L2 node but to multiple ones: two for AS3269 and six for AS3356. Generally speaking, the number of L2 nodes neighbor is an important factor explaining the heavy tail distribution of r′ (x). In practice, it means that a significant subset of routers are connected to several high degree L2 gear. Surprisingly, this experimental analysis shows that, for topologies such as AS3356 and AS3269, more than half of routers are connected to more than two L2 nodes. We also notice that there exists routers connected to more than 20 L2 nodes in AS1267: in that case, it probably means that those routers are involved in V LANs crossing one or several switches. Fig. 22 illustrates two practical cases leading to the large r′ (x) bias. Those scenarios are common in our dataset and may explain why the impact of L2 is so significant: (a) M AN access routers are generally connected to multiple metropolitan locations to distribute the access. To reduce the cost and facilitate the deployment, the use of L2 hardware is very common. (b) The use of redundant access connections for load balanc-

Internet Internet

(a) access Fig. 22.

(b) redundancy

Examples of L2 configuration

ing purpose can lead to an increase of this effect. On Fig. 22(a), the Internet access router has a logical L3 degree of nine whereas it has only three physical links. On Fig. 22(a), the two access routers are logically connected to each other routers in order to ensure load balancing and resiliency. Note that a further analysis is necessary to understand the difference in terms of L2 usage between core and access part of a network. Indeed, the pure backbone part is probably differently impacted than the border of a network. To mitigate this effect, we need to characterize the core and access parts in order to divide networks in multiple areas. The use of traceroute probes may help us to perform such a mapping. However, this is a huge challenge that is not in the scope of this paper. E. Why such a great shifting in the node degree distribution? The goal of this section is to model relationships between L2 and L3 key distributions explaining such an heavy tail

17



(b) AS 1267: IUnet



Dependence between b (number of L2 neighbors) and s (L2 degree distribution) distributions



Dependence between p (routers having L2 neighbors) and b (number of L2 neighbors) distributions


(b) AS 1267: IUnet

(b) AS 1267: IUnet



Dependence between p (routers having L2 neighbors) and s (L2 degree distribution) distributions

distribution when considering a L3 view. We want to focus on the correlation between L2 and L3 hardware that generates nodes having a very high degree. In Sec. VI-D, we have seen that the degree distribution of L2 nodes (s(x)) and the L2 neighbors distribution (b(x)) are the first impacting factors. 1) L2 and L3 Correlations: In order to provide an accurate model, correlations between keys distributions should also be considered. For routers involved in L2 interconnections, we denote p their maximum degree and b their maximum number of L2 neighbors. Using r(x), p(x) and normalizing b(x) and s(x) distributions, we are able to provide a probabilistic model. For that purpose, we also need to quantify the nature of the dependence between p(x), b(x) and s(x) distributions. Fig. 23 shows the normalized p(x) × b(x) dependence distribution. It allows us to understand the relationship between p and b distributions. Obviously, b(x) depends on p(x) because a router cannot have more L2 neighbors than its total degree,

as indicated by the empty triangle in the upper left part of Fig. 23. We further notice that large degree routers generally have a larger portion their neighborhood being L2 nodes. We thus observe a strong correlation between those two distributions meaning that b(x) is not uniformly distributed among routers and their degree: a router having a large number of neighbors has a high likelihood to be connected to a large number of L2 nodes. This correlation is quite linear on most of the AS that we analyzed. We do not find any other linear dependency between other relationships: Figs. 24 and 25 do not suggest any kind of obvious dependence relations. However, we cannot argue that there does not exist any kind of non linear dependance. Figs. 24 and 25 may also help to understand some specific behaviours seen in Fig. 20. Potentially, for example if there exists some particular relations between p(x) and s(x), it may explain several shifting details in the degree distribution. In

18

this section, we investigate whether the p(x)×b(x) dependence seems sufficient to explain the large tail of the L3 degree distribution. To solve this question, we decide to only model the linear correlation between p(x) and b(x). 2) Model: In this section, we only model the relation between the number of L2 neighbors and the total number of neighbors of a given router. Our goal is to highlight the fact that this relation conditions the L3 degree distribution. Our model works as follows: considering a router having i neighbors with j being L2 ones (j ≤ i), its L3 logical degree may become x = i − j + η where η denotes the sum of the degree (minus one) of its L2 neighbors. Because we do not model any relations between s and the other distributions, we consider that the degree of its L2 neighbors only depends on s(x). Generalizing this idea, the set of routers having a L3 logical degree of x is composed by all routers having a lower degree that may reach this degree thanks to their L2 neighbors. Our model is probabilistic is the sense that we consider the probability that a router having a degree of i connected to a set of j L2 nodes will increase its L3 logical degree by a certain amount, x − i, to appear directly connected to x neighbors. In the following, the notation bp(i, j) refers to the probability to find a router having a degree of i and j L2 neighbors (0 < j ≤ i). In other words, bp(i, j) corresponds to the distribution function given in Fig. 23. Hence, our probabilistic model verifies: ½ r(x) + χ(x) − p(x) if x > 1 r′ (x) ≈ (5) r(x) − p(x) otherwise. with χ(x) referring to the probability that L3 nodes with a lower degree than x reach a degree of x because of their connections to L2 nodes, such that: min(x,p+1)−1

χ(x) =

X



p(i) ×

i=1



min(i,b)

X j=1

bp(i, j) × cij (x) (6)

cij (x)

where denotes the sum of all possible product combinations between a set of j elements {s(a1 ), . . . , s(ak ), . . . , s(aj )}, with 2 < ak ≤ x − i − j + 3 ∀k ∈ [1, j], and such that: j X

ak

= x − i + 2j

k=1

= ω.

(7)

We denote δ the set containing ak ’s j-tuple, δ = [3, x − i − j + 3]j . Thus, it comes:

cij (x) =

X ∀j-tuple{a1 ,...,aj }∈δ |

j X

j Y

s(ak )

(8)

k=1

ak = ω

k=1

Note that in practice we do not have to explore all possible combinations in the space δ to perform the computation of cij (x). Indeed, we do not care about the degree apparition order

because cij (x) is a product, so the j terms are permutable. Let us denote ∆ the set of j-tuple {a1 , . . . , aj } ∈ δ verifying Eqn. 7. In practice, we then just need to compute:

cij (x) =

j Y

X

∀{a1 ,...,al ,...,aj }∈∆ k=1 such thatal ≤al+1 ∀0

Internet Topology Discovery Through mrinfo Probing - IP Networking Lab

Internet Topology Discovery Through mrinfo Probing - IP Networking Lab

Suggest Documents

internet topology discovery - IP Networking Lab - UniversitÃ© catholique

Notes in pdf - IP Networking Lab

Automating the iBGP organization in large IP ... - IP Networking Lab

Observing the Evolution of Internet AS Topology - Internet Research Lab

A Complete IP Network Topology Discovery ... - Prof. Dr. Arshad Ali

MERLIN: MEasure the Router Level of the INternet - IP Networking Lab

MERLIN: MEasure the Router Level of the INternet - IP Networking Lab

Scalable Route Selection for IPv6 Multihomed Sites - IP Networking Lab

BGP Add-Paths : The Scaling/Performance ... - IP Networking Lab - UCL

A Comparative Study of Path Performance Metrics ... - IP Networking Lab

Implementation and evaluation of the Shim6 ... - IP Networking Lab

Safe Updates of Hybrid SDN Networks - IP Networking Lab

The Paths Towards IPv6 Multihoming - IP Networking Lab - UniversitÃ©

Understanding the topological properties of ... - IP Networking Lab

Safe Updates of Hybrid SDN Networks - IP Networking Lab

Seamless Network-Wide IGP Migrations - IP Networking Lab - UCL

WiFi Roaming: Legal Implications and Security ... - IP Networking Lab

Open issues in Interdomain Routing: a survey - IP Networking Lab

Authenticated Wireless Roaming via Tunnels - IP Networking Lab

CSIS 4222 Computer Networking Lab 3 – IP Addresses and DHCP

Preventing the Unnecessary Propagation of BGP ... - IP Networking Lab

Computer Networks : Protocols and Practice - IP Networking Lab

IP-Cameras-Networking-FAQ_EN_R1

IP networking - GBV