NTS6: IPv6 based Network Topology Service System of CERNET2 Lin Cong, Bo Yang, Yang Chen, Guohan Lu, Beixing Deng, Xing Li, Ye Wang (Department of Electronic Engineering, Tsinghua University, Beijing 100084, China)
[email protected] Abstract In this paper, we propose NTS6, an IPv6 based Network Topology Service system, which provides a Query and Response Service of Network Topology Information between any pair of hosts accessing IPv6 Internet via CERNET2. The NTS6 system periodically collects the latest OSPF, BGP and SNMP information of CERNET2. Based on these information, NTS6 provides Network Topology Information such as Net Distance, Available Bandwidth and Path Packet Loss in response to users’ query. The real-world experiment on CERNET2 confirms the accuracy of our system. Also, NTS6 now performs well on large deployed Maze P2P File-Sharing System. We believe NTS6 will be a powerful middleware for other applications.
1. Introduction P2P overlay networks are popular solutions for largescale distributed network services and applications, such as application layer multicast, P2P multimedia streaming, file sharing and wide-area storage. If we do not consider the network locality in the construction of the P2P overlay, hosts nearby in the overlay network may actually be far away in the underlying network. This may waste too much network resource and, therefore, degrade performance significantly. One important aspect of constructing an efficient overlay network is how to exploit network locality in the underlying network [1]. There are mainly two schemes for the construction of locality-aware P2P overlay. One is measurement based, such as Meridian [2]. This scheme requires lots of end-to-end measurements, and its scalability is questionable. The other is coordinate based, such as GNP [3] and Vivaldi [4]. But the availability of the network coordinate system has not been well tested. Rather that there is not yet a large-scale network coordinate system according with the real network condition.
We propose a new-style approach to handle this problem using network-layer information. Considering the exactitude and integrity of ISP's network topology information, it should be an accurate and deployable solution. We obtain routing, bandwidth and traffic information from Next Generation China Education and Research Network (CERNET2) ISP directly. Through analyzing and utilizing these information, we propose the NTS6, an IPv6 based Network Topology Service system, which provides a Query and Response Service of Network Topology Information between any pair of hosts accessing IPv6 Internet via CERNET2. Our major contribution of this work could be summarized as follows. We present the design and implementation of NTS6 system and deploy it on the real IPv6 Internet. We evaluate the performance of our NTS6 system by real-world experiment on CERNET2. Our NTS6 system is used by largely-deployed Maze P2P File-Sharing System for more than seven months and performs well. The rest of this paper is organized as follows. First we introduce the CERNET2 in Section 2. Then we present the design and implementation of our NTS6 system in Section 3 and evaluate its performance in Section 4. We introduce the application of our NTS6 system in the large deployed Maze P2P File-Sharing System in Section 5. We conclude the whole paper with Section 6 and present our acknowledgement in Section 7.
2. Overview of the Next Generation Education and Research Network in China (CERNET2) CERNET2 is one of the largest next generation Internet backbones and the core network of the China Next Generation Internet (CNGI) Project as well as the only nationwide academic network. Also it is the world's largest native IPv6 backbone so far. CERNET2 connects 200 universities and 100+ Research Institutes at 1Gbps-10Gbps. CERNET2
backbone adopting the native IPv6 protocol provides a rich experimental environment for the next generation Internet technique. CERNET2 also partly introduces the advanced IPv6 routers developed by local company. It will become the most important infrastructure for deployment of advanced network technique and killing applications of the next generation Internet, and it will be a big promotion of China’s next generation Internet development. The routing control of CERNET2 bases on some new generation network protocols. The Interior Gateway Protocol of CERNET2 backbone network is OSPFv3 [5] [6], the External Gateway Protocol between the CERNET2 backbone AS and the CERNET2 regional ASes connected to the backbone is BGP4+ [7]. The network management protocol of the CERNET2 backbone network is SNMPv2c [8].
parses route entries into AS Number field, Prefix field and NextHop field. 3.1.3. SNMP Information Collecting and Processing The data collecting host sends SNMP requests to backbone network routers, and obtains SNMP data from the special MIB database. Then it derives the available bandwidth and Path Packet Loss on the shortest path (3.1.1) between each pair of backbone network routers. Set the packet loss rate of the i th segment on the shortest path as di , and define the Path Packet Loss as :
D = 1 − ∏ (1 − di ) . i
3. The Design and Implementation of NTS6 System NTS6 system includes two subsystems. One is Network Topology Information Collecting and Processing subsystem. The other is Network Topology Information Query and Response subsystem.
3.1. Network Topology Information Collecting and Processing Subsystem The NTS6 system collects latest OSPF, BGP and SNMP information on CENTET2 backbone network every 15 minutes. It obtains the topology and route state of CERNET2 backbone from OSPF information, the IPv6 address prefixes of the ASes connected to CERNET2 from BGP information, and the backbone network link state such as bandwidth and packet loss rate from SNMP information. 3.1.1. OSPF Information Collecting and Processing The data collecting host builds an adjacency relationship with a core router on CERNET2 backbone network, and collects all the LSAs (Link State Advertisement) of OSPF on CERNET2. Then it calculates the shortest path and OSPF metric between each pair of backbone Advertising Routers on CERNET2 using Dijkstra algorithm. 3.1.2. BGP Information Collecting and Processing The data collecting host builds a peer relationship with a core router on CERNET2 backbone network, then obtains the BGP information from CERNET2, and
Figure 1. CERNET2 Network Topology
Figure 1 shows the network topology of CERNET2 and the geographical locations of the CERNET2 backbone and regional ASes.
3.2. Network Topology Information Query and Response Subsystem The NTS6 system provides users with service of querying the Network Topology Information between any pair of IPv6 hosts (A and B). The Network
Topology Information is a three-tuple (Net Distance, Available Bandwidth, Path Packet Loss) , includes Net Distance that denotes whether the special IPv6 addresses A and B belong to the same AS and the OSPF metric on backbone network between A and B, the minimal available bandwidth (Available Bandwidth) and the Path Packet Loss (Path Packet Loss) on the shortest path between A and B. 3.2.1. Network Topology Information Query There are four steps in Network Topology Information Query. Figure 2 shows this four-step procedure.
matches A and B, then records its Advertising Router field separately. If Advertising Router of A and B are the same, which means they connect to CERNET2 through the same backbone network router, NTS6 will skip Step 3 to Step 4. Step3. OSPF Shortest Path Query According to the result from OSPF information processing, NTS6 calculates the shortest path between the backbone network routers through which A and B connect to CENRET2, also obtains the OSPF metric. Step4. SNMP Information Query and NTS6 Information Creation If Step 3 has been executed, there must be a shortest path between A and B. Then query the Available Bandwidth, Path Packet Loss from SNMP information, and set Net Distance equal to OSPF metric on this shortest path. Otherwise, because the A and B belong to the same AS or connect to CERNET2 through the same backbone network router, NTS6 will consider that the Available Bandwidth between A and B is infinity, the Path Packet Loss and Net Distance are all zero. Finally, create the three-tuple NTS6 information (Net Distance, Available Bandwidth, Path Packet Loss) and send it back to users. Figure 3 describes the query process of NTS6 system.
Figure 2. Frame of Network Topology Information Query
Step1. BGP Information Query Figure 3. Query Process of NTS6 System
NTS6 searches the BGP IPv6 Prefix field, fetches out the entries of BGP routing table that matches A and B, and then records its AS Number field and NextHop field. If AS number of A and B are the same, which means they are located in the same AS and Net Distance should be set to 0, NTS6 will skip the next three steps.
3.2.2. Network Topology Information Response
Step2. Backbone Access Router Query NTS6 searches the Prefix field of AS External LSA and Intra Area Prefix LSA in OSPF routing information, fetches out the Prefix field that matches the NextHop field in the BGP information which
Figure 4. The Procedure of XML-RPC Method
NTS6 system answers network topology information query by XML-RPC method. XML-RPC is a message transmission method based on HTTP protocol which can deliver a complex data structure easily. Figure 4 [9] denotes the procedure of the XML-RPC method.
4. Performance System
Evaluation
of
NTS6
Since the NTS6 System was put into practice in July 2006, the Net Distance on CERNET2 has been stable. The Available Bandwidth between most pairs of IPv6 addresses in different ASes on CERNET2 backbone network is stably periodic. The Path Packet Loss of every shortest path is always zero, illustrating the fact that the packet loss rate on CERNET2 backbone link is always zero.
4.1. Peering Selection based on NTS6 System A typical application of NTS service is for the end hosts to perform peer/server selection based on the NTS6 information. Firstly, the end host queries the NTS6 service about its NTS6 information to several other hosts. When it gets the response, it usually ranks the hosts based on the NTS6 information and then tries the hosts in proper order beginning from the one with the highest rank. When such ranks reflect the actual end-to-end performance accurately, the end host then gets good network performance without the overhead of performing the end-to-end measurement. In this section, we design a straightforward procedure to cluster and rank the hosts based on the NTS6 information and then compare the ranks with the end-to-end network performance measured by the end hosts. We setup an IPv6 test-bed composed of 12 end hosts distributed among six different sites on the CERNET2 (see Table 1), two hosts per site. We cluster and rank the hosts based on the NTS6 information (Net Distance, Available Bandwidth, Path Packet Loss) as follows: 1. When the (Net Distance)ij is zero, classify the two hosts i and j to a same class. 2. For each Class i, we rank other classes based on (Net Distance)ij. If two classes have equal (Net Distance)ij to Class i, we then rank them based on (Available Bandwidth)ij accordingly, and finally based on (Path Packet Loss)ij when (Available Bandwidth)ij is still the same. Meanwhile, we measure the end-to-end TCP throughput Bulk Transfer Capacity (BTC) [10] and RTT using Iperf [11] and ping6 between every pair of the 12 hosts. We ran Iperf for 100 seconds and took 80 RTT samples on each path.
Table 1. Six Institutes in Our Experiment
TSCS TSEE PKU SCUT ZJU HUST
Computer Science Department of Tsinghua University Electronic Engineering Department of Tsinghua University Peking University South China University of Technology Zhejiang University Huazhong University of Science and Technology
Table 2 shows the result of clustering and ranking. The 12 hosts are clustered to five classes. Each row shows the ranks from one host to the total five classes. Note that the hosts from TSEE and PKU are clustered into a same Class, despite the different universities they are from. However, although TSEE and TSCS are both in Tsinghua University, they are clustered into different classes showing that the Network Topology Information may not match exactly with the geographical topology. Table 2. Clustering and Ranking of Class based on NTS6 Information
Class TSCS
TSEE &PKU
SCUT
ZJU
HUST
TSCS
1st
2nd
4th
5th
3rd
TSEE &PKU
2nd
1st
4th
5th
3rd
SCUT
3rd
4th
1st
5th
2nd
ZJU
3rd
4th
5th
1st
2nd
HUST
2nd
3rd
4th
5th
1st
Class
Figure 5. End-to-end BTC VS. NTS6 ranks
Figure 6. RTT VS. NTS6 ranks
Figure 7. The User Number of NTS6 system
Figure 5 compares the ranking result with the endto-end BTC. Each curve corresponds to one row in Table 2. As shown in this figure, the BTC decreases when the rank increases. Figure 6 compares the ranking result with average RTT between a pair of classes. As shown in this figure, the RTT increases as the rank increases. This is likely to be true as we first rank the hosts based on the topology distance (Net Distance)ij between two hosts, and larger (Net Distance)ij likely leads to fewer BTC and larger RTT. Both two figures show that the NTS6 ranks agree well with the actual end-to-end performance and suggest that we can use NTS6 information to perform peer/server selection.
Figure 7 denotes the user number of NTS6 system from December 1st, 2006 to January 7th, 2007. Compared with IPv4 users, the number of IPv6 users is still under development.
5. Application of NTS6 in Maze System Maze [12] [13] [14] is a P2P File-Sharing system which is developed and deployed by the Computer Science Department of Peking University. The structure of Maze is similar to Napster, with a centralized, cluster-based search engine, but is additionally outfitted with a social network of peers. Maze is a largely deployed file sharing system with more than 1.4 million registered users. It has an average of 30,000 peers online at any time, and roughly 13TB of data exchanged daily. Recently Maze releases an IPv6 version and provides P2P file-sharing service for IPv6 users. During the collaboration in China New Generation Internet (CNGI) Project, now NTS6 provides network topology service for all the users accessing the Maze File-Sharing system through IPv6. According to our observation, NTS6 performs very stably and effectively for more than seven months, and responses about twenty thousand queries coming from more than eight thousand individual IPv6 users distributing around China.
Figure 8. The Distribution of Access IPv6 Addresses
Figure 8 denotes the distribution of these individual IPv6 users. The number under every city means the summation of the accessing IPv6 addresses belonging to that city.
6. Conclusion
This paper presents the design and implementation of NTS6, an IPv6 based Network Topology Service System of CERNET2. According to our real-world experiment on CERNET2, the actual end-to-end measurement indicates that the NTS6 system can reflect the actual data transmission performance by clustering the IPv6 addresses to Classes and then ranking them. Now NTS6 system has been used in the large deployed Maze P2P File-Sharing System and performs very stably and effectively. We now focus on applying our NTS6 system on more kinds of large scale P2P applications, such as the P2P multimedia streaming system, overlay routing and etc.
7. Acknowledgement This work is supported by the National Science Foundation of China Project (No.60473087) and the China New Generation Internet Project (No.CNGI-0412-2A).
8. Reference [1] Xin Yan Zhang, Qian Zhang, Zhensheng Zhang, etc. A Construction of Locality-Aware Overlay Network: mOverlay and Its Performance. IEEE Journal on Selected Areas in Communications, Vol.22, No.1, Pages 18-28, 2004. [2] B. Wong, A. Slivkins, and E. G. Sirer. Meridian: A Lightweight Network Location Service without Virtual Coordinates. In Proceedings of ACM SIGCOMM’05, August 2005. [3] T. S. E. Ng and H. Zhang. Predicting Internet Network Distance with Coordinates-Based Approaches. In Proceedings of IEEE INFOCOM’02, June 2002.
[4] F.Dabek, R. Cox, F. Kaashoek, and R. Morris. Vivaldi: A Decentralized Network Coordinate System. In Proceedings of ACM SIGCOMM’04, August 2004. [5] J. Moy. RFC 2328: OSPF Version 2. IETF 1998. [6] R.Coltun, D.Ferguson, J.Moy. RFC 2740: OSPF for IPv6. IETF 1999. [7] Y.Rekhter, T.Li, S.Hares. RFC 4271: A Border Gateway Protocol 4 (BGP-4). IETF 2006. [8] D.Harrington, R.Presuhn, B.Wijnen. RFC 3411: An Architecture for Describing SNMP Management Frameworks. IETF 2002. [9] XML-RPC. http://www.xmlrpc.com/. [10] Matt Mathis and Mark Allman, RFC 3148: A Framework for Defining Empirical Bulk Transfer Capacity Metrics. IETF 2001. [11] Iperf. http://dast.nlanr.net/Projects/Iperf/. [12] Maze Project Homepage. http://maze.pku.edu.cn. [13] Mao Yang, Zheng Zhang, Xiaoming Li, Yafei Dai. An Empirical Study of Free-Riding Behavior in the Maze P2P File-Sharing System. In Proc. of IPTPS’05, Feb. 2005. [14] Qiao Lian, Yu Peng, Mao Yang. Robust Incentives via Multilevel Tit-for-tat. In Proc. of IPTPS’06, Feb. 2006.
