Implementation and Performance Evaluation of Diffserv Enabled MPLS Testbed for Linux using Past Information of Queue Dibyajyoti Guha, S.S. Pathak G.S.Sanyal School of Telecommunications, Indian Institute of Technology Kharagpur, India-721302. Email:
[email protected],
[email protected] Abstract—The advent of Multi Protocol Label Switching (MPLS) has reach beyond the original idea behind the development of MPLS. The original idea behind MPLS was to facilitate fast packet switching using 20 bit MPLS Label header. Presently Traffic Engineering has drawn considerable major interest of the researcher. In this paper we describe an algorithm to use multiple Link Switching Path (LSP) for MPLS which is implemented in GNU Linux under link overload condition. This work has been done using MPLS-Linux GNU project. This algorithm collects the statistics of Hierarchical Token Bucket (HTB) queueing discipline inside Linux Advanced Routing and Traffic Control (LARTC) subsystem. HTB allows the sharing of unused allocated bandwidth among the leaf classes under the same parent class of queue defined by the administrator of Network. Our algorithm will split the low priority traffic over multiple LSPs during overload and fallback to single LSP in the absence of overload. A Kernel module has been developed to consistently check the bandwidth usage and HTB buffer health to feed the information to our algorithm and communiacting another user space program to trigger the traffic split. The communication between Kernel and user space program is done by Netlink socket. As a process of modifying the exisitng Kernel code we have also verified the interoperability of Differentiated Services Code Point (DSCP) in IP and Experimental bits (EXP) in MPLS. An experimental testbed has been setup using Linux and performance metrics such as delay, packet loss has been analyzed for MPLS with and without traffic splitting on overload condition on this testbed.
I. I NTRODUCTION The original idea behind MPLS was to ensure fast packet switching with 20 bits Label between Layer 2 and Layer 3. But the increasing demand of network bandwidth by application like VoIP, Video streaming, P2P data sharing, with diversified traffic characteristics has forced Internet Engineering Task Force to draft Differentiated Service Code Point (DSCP) [1]. Although 3 bits were designed for IP Type of Service (TOS) with the same goal but it was insufficient. DSCP is used in IP header to mark the packets differentially for delay (voice) or loss (data) priority traffic. Very few layer 2 protocol were using TOS, later 3 bits field called Experimental bits (EXP) were developed in MPLS header to maintain the mapping from DSCP in IP to MPLS [7]. EXP header plays an important role to do bandwidth partitioning when multiple MPLS flows are sharing one link. The edge router in the MPLS domain encapsulates(decapsulates) the IP datagram in the MPLS header. MPLS has Label edge router(LER) and
Label switch router(LSR). LERs attach(detach) the MPLS header before the IP header at the ingress(egress) node while sending data into(out) MPLS domain. LSRs are responsible for switching the Label in the MPLS header from a precomputed switching table for each hop. The switching table contains input labels, input interfaces port and corresponding out labels, out ports and next hop. At ingress node, incoming IP packets are processed and labels applied according to precomputed table which is populated using Label distribution protocol. The LSR reads the MPLS packets,applies policies of QoS and forwards to the next hop. Computing intensive analysis happens only at ingress nodes. At egress node MPLS label is stripped and IP packet is delivered to the next hop. The basic framework for attaching(detaching) 4 bytes MPLS header between IP and Ethernet header on the ingress(egress) node LERs and mapping of DSCP to EXP is developed by GNU Project MPLS-Linux [13]. The incoming traffic is divided into different class according to their Service Level Agreement (SLA) by using Netfilter iptables [6]. The packet is translated into MPLS and Next Hop Label Forwarding Entry (NHLFE) is chosen from precomputed Forward Information Base (FIB) table. The packet is handed over to the LARTC [2] for traffic shaping purpose before it is sent to the driver specific Kernel subsystem for transmitting over the outgoing interface. We have used HTB as queueing discipline on the output links [3]. Our algorithm fetches the information from the HTB on a regular interval and decides when to split the traffic and send it over least utilized LSP. Once the traffic resumes to the normal rate the algorithm fallback to the single LSP mode. We have avoided to run a periodic probe message between ingress and egress LER in order to control QoS on a local granularity basis. The communication between Kernel and user space program is done by Netlink socket. Our algorithm is implemented as a separate Kernel module and communicates to a user space program to trigger a traffic split on MPLS Linux. Earlier works are based on NS2 simulation. This is an enhancement to develop one’s own Netfilter [6] MATCH and TARGET module to investigate packet and treat accordingly. Ingress node is doing everything. In this paper, we have used Debian 5 with Linux Kernel version 2.6.26-1-686 [5]. Key contribution of this paper is
1) Collecting statistics of HTB buffer and avoiding running a probe message between LER on a regular interval 2) Running the algorithm as a standalone module in the Kernel and communiacting to user space program. 3) To incorporate MATCH module in the Netfilter to choose different MPLS NHLFE based on DSCP value present in the ingress IP when the ingress bitrate has crossed a threshold. Rest of the paper is organized as follows: Section II describes Related Works. Section III describes the Analytical Model, Section IV describes design of MPLS Linux, Section V describes Experimental Testbed. Section VI describes Results. We conclude in Section VII. II. R ELATED W ORK A generalized comparative performance analysis based on OPNET simulation for MPLS and non MPLS is discussed in [16]. The integrated service based on ATM-MPLS and a comparative performance analysis of MPLS and non-MPLS over ATM network with OPNET simulation has been discussed in [14]. An adaptive resource allocation approach and its Linux testbed implementation for QoS enabled Virtual Private Networks (VPNs) is described in [15]. The authors of the above [15] paper using an open source RSVP-TE implementation complaint with 2.4.19 kernel version. They have proposed a traffic predicting mechanism assuming that the measurements comprise samples gathered at regularly spaced instants. The samples are themselves some function of the traffic rates in the interval between sampling instants. The GNU open source project [13] has made the basic framework of MPLS implementation including Label stacking(used for tunneling one LSP inside another LSP), and Experimental bits support RFC5462 recommended by IETF in 2009 [7]. Mitzenmacher has envisaged the concept of using delayed or old information which is described in [4] and [17]. It is used in a multi server environment where each user can join in one of the least loaded servers independent of previous user. The author has proposed a mechanism to share the load in a distributed manner instead of centralized manner. He has proposed a logical entity called bulletin boards which maintain separate queue for storing load information for each of the server. Each of the server updates the bulletin board in every regular interval. Each arriving packet lookup the bulletin board to decide which server to choose based on few strategies. If each packet choose the server independently and uniformly at random then each server has been modeled as M/M/1 queue. III. A NALYTICAL M ODEL The queue length evolution of the outgoing link can be modeled as M/M/1 queue for standard way of performance evaluation. But our algorithm is fetching the bandwidth usage statistics from HTB buffer in a regular interval basis, hence the number of packets served during that interval is not constant and it resembles the bulk service model with Poisson arrival distribution which is discussed by Kleinrock’s pioneer book [18]. Q1= total number of data packets. Q2= total
number of voice packets. Q1+Q2 = total number of packets allocated for both voice and data. λ1 = average arrival rate with Poisson distribution for data packet. λ2 = average arrival rate with Poisson distribution for voice packet. λ = λ1 +λ2 = total arrival rate. Probability of arriving one voice packet before the data packet arrives λλ2 . Pa = Acceptance probability for voice packets. QX 2 −1 λ1 λ2 Q1 + Q2 − 1 )k ( )Q1 +Q2 −1−k Pa = ( λ1 + λ2 λ1 + λ2 k k=0 (1) πk =Probability of having k number of voice packet in the queue. Φ(z) = Probability Generating function of steady state queue length. c = number of customer transmitted in one lookup interval and essential condition for finite queue length is λ < c. Pij = one step transition probability from πi to πj during one lookup interval, can be given by following equation : λ2 j , 0 ≦ i < c, j = 0, 1, 2, .... (2) Pij = e−λ2 j! Pij = e−λ2
λ2 j−i+c , i ≥ c, j = i − c, i − c + 1, .... (3) (j − i + c)!
πj = e−λ2
c+j c−1 X λ2 j X λ2 j−i+c πk e−λ2 πk + j! (j − i + c)!
Φ(z) = e−λ2 (1−z)
(4)
k=c
k=0
[
Pc−1
c k k=0 (z − z )πk ] c −λ (1−z) z −e 2
(5)
Finding the equilibrium probability is very difficult as such which is given in Kleinrock’s book [18]. We present a heuristic solution of πj for the sake of simplicity and reducing the computational complexity. We define αk to be k
αk =
λ2 X αk−j bj λ2 + µ j=0
(6)
j
where bj = e−λ2 λj!2 , and α0 = 1 − ρ. Equating πk with αk and putting the values of πk in the RHS of the well known equation (4), we can get a heuristic solution of πj . Our algorithm is explained in the diagram 1. It is called upon when FIB table is looked upon by MPLS module, our algorithm keeps an account of old queue information of the link usage so that it can trigger the traffic split feature once the bandwidth usage is higher than a threshold value which is user configurable. IV. D ESIGN
OF
MPLS IN L INUX I MPLEMENTATION
The Netfilter [6] is the most important part of the kernel space used to classify the packets, QoS and fair queuing. We have added a Netfilter module to check for DSCP value in the ingress IP packet and attach MPLS label based on DSCP. We have written a simple MATCH and TARGET module in kernel. The DSCP MATCH module is of type struct xt match. It is registered to the matching list using the function xt register match() and is passed as a parameter
xt dscp.o obj-$(CONFIG NETFILTER XT TARGET DSCP) + xt DSCP.o
=
The MPLS specific parameters are configured by command line terminal of Linux. Below is the configuration to choose nhlfe value 4 if the ingress IP is having dscp 10. iptables -t mangle -A FORWARD -m dscp –dscp 10 -j mpls –nhlfe 0x4 ip route add 172.16.10.0/24 via 10.0.1.2 mpls 0x4
V. E XPERIMENTAL T ESTBED Fig. 1.
Load Balance in IP Transmit path
to the function xt register match(). The MATCH module has a function ipv4 get dsfield(ip hdr(skb)) to extract the DSCP value of the IP header. All information pertaining to received frame is stored in the sk buff structure. ip hdr(skb) is used to obtain the buffer starting with IP header information. Similarly the TARGET module is of type struct xt target. It is registered to the TARGET list using the function xt register target() and is passed as a parameter to the function xt register target(). The target module is passed with the NHLFE value which is extracted from the user specified configurations. TARGET module invokes dst) which in turn set the mpls set nexthop2(nhlfe, skb appropriate next hop from the Forwarding Information Base (FIB) table. FIB table has the NHLFE [12] value and corresponding next hop IP address information. This table is populated from user specified configuration. mpls set nexthop2() is an important function which performs necessary jobs :
1) Populating the EXP bits of MPLS. 2) Finding out the next hop from FIB table and interact with the routing module. 3) Buffer operation of struct sk buff for aligning the header because MPLS header has to be inserted between Ethernet and IP header. 4) Setting the ether type to be 0x8847 so that Ethernet is carrying MPLS encapsulated packet in the payload of Ethernet. We have added MATCH and TARGET module as a standalone module in the file xt dscp.c and xt mpls.c. The compilation procedure is described below. The xt dscp.c,xt mpls.c file is put into net/netfilter/ and xt dscp.h,xt mpls.h is put into include/ linux/netfilter/. Then we edited net/netfilter/Makefile and add a rule for required MATCH and TARGET to be built. The rule can be described as below obj-$(CONFIG NETFILTER XT TARGET MPLS) + = xt mpls.o obj-$(CONFIG NETFILTER XT MATCH DSCP) + =
Network Topology is shown in 2. Source is UserA with IP address 172.16.10.10 and destination is UserB with IP address 172.16.20.20. MPLS Router1(MPLS Router3) is ingress(egress) LER,MPLS Router2 is LSR. Ingress router has two paths to reach the destination. First one is connected directly to the egress router,second one is connected via MPLS Router2. Both the links are 100 Mbps ethernet. Ingress router enables traffic splitting when the packet reception rate crosses a threshold of 2 mbps. Disabling the traffic splitting mechanism, all the VoIP and Data packet passes through the first path. Enabling traffic splitting that extracts the Data packets whose IP headers are marked with DSCP 30 to go over second path [1]. Hence the delay for VoIP has been given priority. We have used Open source GNU tool Distributed Internet Traffic Generator (D-ITG) [11] to originate and terminate the VoIP and TCP,UDP data traffic concurrently. The Ingress MPLS router splits the traffic over the two output interfaces eth0(data) and eth3(voip) while the input link eth1 usage exceeds the user defined threshold. Disabling the packet marking with dscp will cause all the traffic to go through output interface eth3. The inter-arrival duration of data packets is exponentially independent and identically distributed with average 1/1000, packet size is distributed uniformly from 100 to 800. VoIP is generated using G.711 codec. In 60 seconds a total of 40 number of voice and data traffic is generated. D-ITG inserts the timestamps of sending and receiving events for each packet in the log messages which can be extracted using D-ITG’s own log decoder. We have synchronized sender and receiver clocks using Network Time Protocol version 3 [8] . Tools used to analyze the traffic in our testbed are Bmon [9] and Ethereal [10]. The Bmon is a GNU tool used for monitoring the bandwidth of all the interfaces. The Ethereal is a GNU tool used as a sniffer and protocol analyzer. The following variables are used to calculate the propagation delay and mean R[i]= reception time of ith packet, S[i]= sending time of ith packet P[i]=propagation delay of ith packet = R[i]-S[i] D[i]=difference of propagation delay between the ith and i-1 th packet = P[i]-P[i-1] Avg Jitter = Σ |D[i]| number of packets transmitted
VI. R ESUTS
TABLE I P ERFORMANCE C OMPARISON BETWEEN ENABLING AND DISABLING TRAFFIC SPLIT
Comparative performance results is shown in the below table. The D-ITG’s own log decoder is used to produce the statistical details of the delay [11]. For example, in our display of results it contains the delay of both voice and data packets. It can be seen that average delay has increased with traffic splitting because the data packets has traversed a longer path. Figure 4 it can be seen that ingress interface eth1 is receiving the traffic at 9.6 MiB and going out of eth3 interface at 9.6 MiB while traffic splitting is disabled. An additional ftp connection is going on the reverse direction to verify the co-existence of bidirectional traffic is working perfectly. Figure 3 shows that traffic has been split over two output link eth0(for data) and eth3(for voice) for same destination but for different type of traffic. Figure 3 also shows that packet reception rate of 9.82 MiB on ingress eth1 interface is split over eth0 and eth3 at a transmission rate of 5.44 MiB (eth0) and at 4.47 MiB (eth3). Figure 5 Ethereal output is displayed. It confirms that MPLS packet headers are successfully decoded by Ethereal and DSCP [1] values has been mapped to MPLS EXP [7] properly. It also assures that insertion of MPLS header between Ethernet and IP headers comply with the standard [7]. One of the significant improvement is that the standard deviation of delay is less with enabling traffic splitting feature. Reduction of transmission latency is not significantly visible for MPLS over non MPLS in our simple two hop topology. The difference may be visible when there are large number of routing entires in the IP routing table. This is attributed to the fact that MPLS provides switching based on 20 bit MPLS label instead of 32 bit IP address. Table I shows a comparison of performance between enabling and disabling traffic split condition for voice traffic alone. In the background data traffic is flowing through for both the condition. The traffic generation process did not alter during the course of experiment of disabling and enabling traffic split feature. The only change happened during the course of experiment is enabling and disabling traffic split feature. This will help us to measure the difference of average or standard deviation in delay due to traffic split feature. A total of 40 voice calls have been generated for 10 seconds duration, we see mean delay of 3.33799(ms) and 3.32143(ms), standard deviation for delay of 0.03859 and 0.04511, while enabling and disabling traffic split feature, respectively. It is visible that enabling traffic split feature gives a better quality of transmission. The Figures 6 and 7 show the delay of each voice packet in one particular flow while traffic splitting enabled and disabled respectively. X-axis contains the sequence number of received packets and Y-axis contains the propagation delay faced by each packet. The value and number of delay peaks both are considerably reduced in Figure 7. The variance of the delay is observed to be less in Figure 7 than that of Figure 6.
P arameters Number of flows Total Time (s) Total packets Minimum Delay(ms) Maximum Delay(ms) Average Delay(ms) Average Jitter(ms) Delay Standard Deviation(ms) Bytes Received Packets dropped
Fig. 2.
VII.
W ithQoS 40 10.542425 219900 3.29475 4.02484 3.33799 0.00504 0.03859 91611857 18(0.01 )
W ithoutQoS 40 10.295128 219847 3.27470 4.92036 3.32143 0.00517 0.04511 91652938 17(0.01 )
Network Topology
CONCLUSION
In this paper we have verified the procedures to split the traffic on MPLS Linux in user defined congestion environment. We have also investigated the interoperability between DSCP in IP headers and EXP in MPLS headers using a testbed developed in our laboratory. A design for choosing NHLFE based on DSCP in ingress IP does not significantly affect the existing design of DSCP to EXP mapping. Presently static routes have been used in the LERs and overall bandwidth consumption is considered for congestion condition. Future works are 1) We need to integrate our Linux Kernel with Quagga and Zebra Linux in order to incorporate dynamic routing on the LERs. 2) To monitor the bandwidth usage by each dscp [1] value on LERs and exp [7] value on LSPs. 3) Dynamic partitioning the total bandwidth among all the dscp(or exp) values to provide provide flexible bandwidth sharing for each class of traffic. We need to explore and modify different option of the Linux Advanced Routing and Traffic Control. R EFERENCES [1] Black D.,Carlson M., Davies E.,Wang W., ”An Architecture for Differentiated Service,” IETF, Request for Comments: 2475, Dec 1998 [2] Thomas G.,Gregory M.,et al, ”Linux Advanced Routing Traffic Control”, GNU Open Source,Jan 2000. [3] Martin Devera, ”Hierarchical Token Bucket”, ”Linux Routing and Traffic Control”, GNU Open Source, Jan 2004. [4] Mitzenmacher M., ”Load balancing and density dependent jump Markov processes,” Foundations of Computer Science, 1996. Proceedings., 37th Annual Symposium on , vol., no., pp.213-222, 14-16 Oct 1996.
Fig. 3.
With Splitting BMON Output
Fig. 5.
Ethereal Output
3.55 Without QoS VoIP
Propagation Delay (ms)
3.5
3.45
3.4
3.35
3.3
3.25 0
Fig. 4.
100
200
300
400
500
600
700
Sequence Number pf Packets
800
900
1000
Without Splitting BMON Output Fig. 6.
[5] Linus Torvald, “The Linux kernel”, Jan 1995. [6] Marc B.,Rusty R.,James M., ”Packet mangling and IP Firewall”, GNU Linux, Jan 2000. [7] Acreo R.,Asati,Andersson L., ”Multiprotocol Label Switching (MPLS) Label Stack Entry:EXP Field Renamed to Traffic Class Field”, IETF, Request for Comments: 5462, Feb 2009. [8] David L. Mills, ”Network Time Protocol version 3”, IETF RFC 1305, March 1992. [9] Thomas Graf, ”Portable bandwidth monitor and rate estimator”, GNU Linux, Feb 1998. [10] Gilbert R.,Gerald C., ”Traffic Analyzer and Sniffer”, GNU Linux, Jan 1999. [11] Alessio B.,Stefano A.,Antonio P., ”Distributed Internet Traffic Generator”, GNU Linux, 2004. [12] A. Viswanathan,E. Rosen and R. Callon, ”Multiprotocol Label Switching Architecture”, ”IETF RFC 3031”, Jan 2001. [13] James Leu, “MPLS for Linux”, GNU Linux, Jan 2003. [14] Anchao Cheng; Xin Yao; Lianguo Duan; , ”Design and Implementation of Integrated Services Switch Simulation Model Based on ATM-MPLS,” Computer Science and Software Engineering, 2008 International Conference on , vol.2, no., pp.223-226, 12-14 Dec. 2008. [15] Jia, Y.; Guerrero, M.L.; Kabranov, O.; Makrakis, D.; Barbosa, L.O.; , ”Design and testbed implementation of adaptive MPLS-DiffServ enabled virtual private networks,” Electrical and Computer Engineering, 2003. IEEE CCECE 2003. Canadian Conference on , vol.2, no., pp. 965- 968 vol.2, 4-7 May 2003. [16] Porwal, M.K.; Yadav, A.; Charhate, S.V.; , ”Traffic Analysis of MPLS and Non MPLS Network including MPLS Signaling Protocols and Traffic Distribution in OSPF and MPLS,” Emerging Trends in Engineering and Technology, 2008. ICETET ’08. First International Conference on , vol., no., pp.187-192, 16-18 July 2008. [17] Mitzenmacher M., ”How useful is old information?,” Parallel and Distributed Systems, IEEE Transactions on , vol.11, no.1, pp.6-20, Jan 2000. [18] Leonard Kleinrock. 1975. Theory, Volume 1, Queueing Systems. WileyInterscience.
VIII.
Without Splitting VoIP
ACKNOWLEDGEMENT
The authors are grateful to Vodafone Essar-IIT Centre of Excellence in Telecommunications (VEICET) project for funding the work.
3.65
With QoS VoIP
Propagation Delay (ms)
3.6
3.55
3.5
3.45
3.4
3.35
3.3
3.25 0
100
200
300
400
500
600
700
Sequence Number of Packets
Fig. 7.
With Splitting VoIP
800
900
1000
