Design and Implementation of a Monitoring System Based on IPFIX Protocol Farzaneh Fatemipour Department of Computer Engineering, Azad University of Mashhad Mashhad, Iran
[email protected] Abstract Today measuring traffic is an essential tool for controlling and engineering communication networks and must provide useful information about flow states for network administrator. In this paper we present a measurement system for monitoring IP networks based on IPFIX (IP Flow Information eXport Protocol). This passive monitoring system can provide network operators with detailed information about the traffic traversing a network. Packet traces can provide information about the performance experienced by flows and other characteristics of the traffic, such as packet sizes, packet loss, one way delay and delay variation. We used the well-known database MySQL to store and retrieve data. The meter/exporter and collector process were implemented in the Linux operating system environment. The proposed monitoring system was tested in a campus network. Keywords- Monitoring, Measurement, IPFIX Protocol, Traffic Flows, One-Way Delay, One-Way Packet Loss, Jitter
1. Introduction Quality of service monitoring is becoming crucial to service providers for providing QoS-based services and service assurance and for managing network resources. In addition, users require network performance statistics, as network performance has a direct impact on perceived quality of the application viewed by the users. A measurement system can provide network operators with detailed information about the traffic traversing a link. For example, computing traffic volumes by source and destination IP addresses allows operators to identify the end hosts responsible for bulk of load on the network. The measured traces can be analyzed to compute the throughput for individual TCP connections or source-destination pairs. To measure QoS parameters of a network there are two methods; active measurement and passive measurement.
The Third Advanced International Conference on Telecommunications (AICT'07) 0-7695-2843-0/07 $20.00 © 2007
Mohammad Hossien Yaghmaee Department of Computer Engineering, Ferdowsi University of Mashhad Mashhad, Iran
[email protected] With active measurement the QoS factors are measured by sending and receiving test packets that emulate actual user data. Some parameters such as delay, delay variation and loss are usually measured by this method. However, injecting synthetic packets to the network may cause congestion and as the measurement is not based on the actual user data, the results may not be too accurate. Passive measurements use the actual data traversing the network to measure the QoS parameters. This method does not force additional load to the network and as it is working with actual data, the results are more precise. The proposed monitoring system performs passive measurement to calculate one way delay, delay variation (jitter) and one way packet loss. There are number of working groups in the Internet Engineering Task Force (IETF) related to measurements and monitoring such as Remote MONitoring (RMON) in RFC2819, IP Performance Metrics (IPPM) in RFC2330, Real Time Flow Measurement (RTFM) [1], IP Flow Information eXport Protocol (IPFIX) in RFC3917 and packet sampling (PSAMP) [2]. These working groups are defining metrics, developing a common IP traffic flow measurement technology and specifying a standard set of capabilities for sampling packets. The IP Flow Information Export (IPFIX) working group has been chartered to select a protocol for the export of flow information from traffic-observing devices (such as routers or dedicated probes). Based on the requirements defined in RFC3917, there exist different protocols such as NetFlow v9 [3], Diameter [4], CRANE [5], IPDR [6] and LFAP [7], which were evaluated as candidates to provide the basis for the future IPFIX protocol [8]. The results of the evaluation show that NetFlow v9 provides the best basis for it. There are numerous monitoring tools, such as the RIPE Test Traffic Measurement (TTM) [9], NetFlow from Cisco, SFlow, NIMI (National Internet Measurement Infrastructure) [10], Network Analysis Infrastructure (NAI) [11], cflowd, RTG highperformance SNMP statistics monitoring system, Skitter, NeTraMet [12], CoralReef, and Beluga of CAIDA (Cooperative Association for Internet Data Analysis) [13], etc.
There has also been some work on monitoring and measurements at inter-domain level, by European research projects (IST). The objective of the ISTINTERMON [14] project has been to develop an integrated inter-domain QoS monitoring, analysis and modeling system to be used in the multi-domain Internet infrastructure for the purpose of planning, operational control and optimization. The MESCAL [15] project proposed solutions for the deployment and delivery of inter-domain Quality of Service across the Internet. MESCAL was partially funded by the EU as part of the IST Program, a major theme of research and technological development within the Fifth Framework. In this paper, we present a QoS monitoring system based on the IPFIX protocol architecture. The proposed monitoring system consists of Metering, Exporting and Collecting processes and can measure one way delay, one way packet loss, and jitter for individual flows. The remainder of this paper is organized as follows: Section 2 gives a short description of IPFIX protocol. Section 3 describes the details of the proposed system and its components. In section 4 the database used in the monitoring system is described. Section 5 shows the test results and finally section 6 concludes the paper.
2. IPFIX Protocol The IP Information eXport (IPFIX) protocol defines a format and protocol for the export of flow information from routers or measurement probes. IPFIX is a general data transport protocol that is easily extensible to suit the needs of different applications. Figure1 shows the architecture of IPFIX protocol.
• Defining common notion of an “IP flow” and specifying a data model and a transport mapping for flow and exporter attributes; • Having a standardized IPFIX which is nearly compatible to Cisco NetFlow v9; • Selecting a (congestion-aware) transport protocol by which IP flow information can be transferred in a timely fashion. The IPFIX protocol is not complex, which ensures simple implementation. IPFIX messages can be transferred using SCTP, TCP or UDP as bearer protocols. The main task of the exporter is to take the measurement data from one or more metering processes and to send the IPFIX messages to data collectors. The exporter has to take care that the templates are sent prior to the related data records. The IPFIX collector has to maintain a list of sources and per source a list of templates to decode incoming data templates. Because of the template feature of IPFIX the collector does not need any knowledge of the transferred data. All information needed to decode all kinds of data is transferred via template records. In the scope of passive QoS measurements, there is often the need to exchange and export measurement data in a finer granularity than per flow. One typical application is the passive one way delay measurement. The IPFIX protocol is designed for exporting per flow information. The IETF standardization team developed an extension of the IPFIX protocol that also allows exporting per packet information (draftboschi-exportperpktinfo-02.txt) and proposed the extension to the IPFIX working group. The proposed measurement system uses this extension of the IPFIX protocol.
3. Proposed Monitoring System
Figure 1. Overall architecture of IPFIX protocol The protocol is flexible in both flow key and flow export. The flow key defines the properties used to select flows and can be defined depending on the application needs. Flow information is exported using flow data records and the information contained in those records can be defined using template records. A template ID uniquely identifies each template record and provides the binding between template and data records. Some goals and targets of IPFIX are listed bellow: • Standardizing current practice in flow data export;
The Third Advanced International Conference on Telecommunications (AICT'07) 0-7695-2843-0/07 $20.00 © 2007
In this section we describe the proposed monitoring system. The overall architecture of the system is similar to the overall architecture of IPFIX protocol shown in Figure 1. The metering processes are implemented as Linux/PC boxes. To measure one way delay and one way loss, we need two measurement points that are time synchronized with each other or with an external point. We call the Source and Destination Measurement Points SMP and DMP, respectively. Both measurement points should assign a timestamp and a unique label to each packet they observe from network interfaces. The collector brings together the measured data from these two points and calculates the delay, loss and jitter. We use Libpcap [16] for obtaining packets from network interfaces. Libpcap provides a copy of packets and a timestamp that the kernel puts on reception of a new packet from the network interface. To generate a unique ID for the packets, we use BOB hash function
[17] which provides low collision probability at a small computational cost [18].
3.1. Meter-Exporter Process After obtaining packets by Libpcap, a hash function is calculated over the header’s fields and payload of the packet. In [19] the packet’s header fields are divided into three categories: • Variable fields such as TTL, ToS and Header Checksum which change along the packet's path; • Low entropy fields such as Version, Header Length and Protocol which are invariant for a given packet along a path, but have little or no variation from packet to packet; • High entropy fields such as Src/Dst IP addresses, Identification, Flags and Fragment Offset, which are invariant for a given packet along a path and can also vary greatly between packets. The fields that hash function is calculated over them can be chosen from high entropy fields. To select a packet, we compute BOB hash function over Total Length, Identification, Source/Destination IP Addresses fields and 12 byte of TCP/UDP header. We use these 24 bytes as an input key to the hash function. After computing the hash function over the fields, the result is compared to the selection range. If the packet is selected, another hash function is used to produce a label as an identifier for the packet. The input key of this function is as the same as the previous one. Source and destination measurement points store the label and timestamp of the sampled packet for exporting. Additional information about the sampled packet (such as its length and its source and destination IP addresses) is required for distinguishing flows in collector. We collect this additional information once per packet in SMP. Figure 2 shows the structures used to store the information in each end point of the measurement. Destination Export Structure Label
TimeStamp
a) Destination Measurement Point Structure Source Export Structure Label
TimeStamp
IP Version
IP Tos
IP Protocol
Src & Dst Port
IP Src & Dst Address b) Source Measurement Point Structure
Figure 2. The structure used in meters
The Third Advanced International Conference on Telecommunications (AICT'07) 0-7695-2843-0/07 $20.00 © 2007
In both cases the gathered data is sent to a buffer to be exported. After the buffer gets full or a timer is expired, exporting process builds an IPFIX message and exports it to the collector.
3.2. Collecting Process IPFIX messages are decoded using templates that have been negotiated between the exporter and the collector. Since distinguishing flows at metering process cause complexity, flow definition and separation is done at collector. To collect and summarize the information of labels, we use the structure shown in Figure 3. Label Structure Label
Timer
IP Version
IP Tos
IP Protocol
Packet Length
Src & Dst Port
Number of Packets
IP Src & Dst Address First TimeStamp
Last TimeStamp
Figure 3. The structure used in the collector As shown in Figure 3 a Timer is assigned to each label l. This Timer shows the maximum time to wait for receiving another packet from label l in the collector. The Timer associated with label l is (re)started every time a label l is arrived at the collector. The Timer is calculated as follows [20]: (1) Timer = Tb + Tm + maxe∈ E T(e) Where: • Tb is the maximum time the label is delayed within the label buffer at the exporter process; • Tm is an upper bound of the time it takes a measurement packet generated by a router to reach the collector process; • T(e) is the delay of link e; • E is the links set of the network. If the Timer is expired and no packet is received, the label is assumed to be finished. Then it is deleted from the list. After deletion, a function is called to compare this label with flow definitions defined by different users. A flow is defined as a set of rules to measure traffic with common properties such as TCP source/destination ports. When a new rule is defined, a new task is added to the system that can produce several flows. For example, consider the rules shown in Figure 4. The rule in Figure 4-a defines a single TCP flow that is traversing from host 192.168.1.1 to host 192.168.2.1. The rule in Figure 4-b defines two or more flows that
their source and destination addresses are the same as the previous rule, but the transport protocol can be TCP/UDP/ICMP/…. This rule will produce several flows all of which belong to the same task.
IPFIX Messages
IP version = 4 Src-Addr = 192.168.1.1 Dst-Addr = 192.168.2.1 Transport = Categorize (b)
Figure 4. An example of flow definition rules When comparing the expired label to flow rules, if any match is found, the flow information will be updated. If the label is detected to be lost, the loss rate of the flow is updated. The other parameters that can be updated are Bytes of the flow, Number of packets of the flow, Mean/Max/Min delay, Mean/Max/Min jitter.
4. Database Design In this section we explain the storage method of measured data. We use the well-known database MySQL to store and retrieve data. We divide the data in among three tables. The first table, called USERS is responsible for storing the user information. Each user has a unique ID which is used by TASK table to refer to him/her. Second table is called TASKS table and stores the information about defined tasks of the system. Each row of the TASK table refers to a specific task and may have several related rows in the FLOW table. Each task has a unique TaskID that is used by FLOW table for linking each flow to its corresponding task. Start and end time of each task is stored in this table. The other fields of the table specify the flow definition conditions that cannot be set as "Categorize". The transport and Type of Service (TOS) conditions that are specific for each flow are stored in FLOW table. Depending on the flow definition, some fields may have valid values or not. In FLOW table the TaskID-FlowID pair is the primary key. Calculated statistics about each flow are other fields of this table.
5. Test Results The functionality of the proposed system is tested in a network shown in Figure 5. The SMP is located in Lab1 and DMP is located in Lab2. As our goal is to measure the traffic traversing between these two points, we define a task like the one in figure 4-b. The task shows packets that are traversing from SMP in Lab1 to DMP in Lab2. These packets are categorized into three different flows based on their transport protocol type. We performed a one-hour test and the aggregation time for packet loss and delay variation was set to 20
The Third Advanced International Conference on Telecommunications (AICT'07) 0-7695-2843-0/07 $20.00 © 2007
Measured Data
Database Collector
Campus Network Lab 1
Lab 2
SMP DMP Figure 5. Test network topology Table 1 shows the measured statistics for each flow which are stored in MYSQL database. The minimum, maximum and average of One-Way-Delay and Delay Variation, the overall One-Way-Packet Loss and number of packets and bytes are the parameters that are calculated during measurement period and are stored in database. Table 1. Statistic Results of the Test Flow Statistics
ICMP
TCP
Mean Delay (msec)
1.22
0.53
0.94
Max Delay (msec)
1.43
0.84
1.135
Min Delay (msec)
UDP
1.06
0.36
0.774
Mean Jitter (msec)
0.0045
0.0323
0.0017
Max Jitter (msec)
0.056
0.13
0.035
Min Jitter (msec)
0
0
0
0.00%
0.00%
0.00%
Packet Loss Number of Packets Number of Bytes
5638
11710
2546
2411664
983640
112024
In Figures 6-14 the test results are shown. Figures 6, 7 and 8 show the measurement results for ICMP flow between SMP and DMP. 10 8 Delay (msec)
IP version = 4 Transport = TCP Src-Addr = 192.168.1.1 Dst-Addr = 192.168.2.1 (a)
seconds. Since packet loss was zero for all flows, we exclude the packet loss diagram from figures.
6 4 2 0 0
1000
2000
3000
4000
time (sec)
Figure 6. One way delay vs. time for ICMP flow
20000 Number of Bytes (byte)
Delay Variation (msec)
0.2
0.15
0.1
0.05
16000 12000 8000 4000 0
0 0
1000
2000
3000
0
4000
600
1200
Figure 7. Delay variation vs. time for ICMP flow
3000
3600
Figures 12, 13 and 14 show the measurement results for UDP flow between SMP and DMP.
8000 6000
10
4000
8
2000 0 0
600
1200
1800
2400
3000
3600
6 4 2
time (sec)
Figure 8. Delay variation vs. time for ICMP flow Figures 9, 10 and 11 show the measurement results for TCP flow between SMP and DMP.
0 0
1000
2000
3000
4000
time (sec)
Figure 12. One way delay vs. time for UDP flow 0.2 Delay Variation (msec)
10 8 Delay (msec)
2400
Figure 11. Delay variation vs. time for TCP flow
Dalay (msec)
Number of Bytes (byte)
10000
6 4 2
0.15
0.1
0.05
0
0 0
1000
2000
3000
0
4000
1000
2000
3000
4000
time (sec)
time (sec)
Figure 9. One way delay vs. time for TCP flow
Figure 13. Delay variation vs. time for UDP flow 1000 Number of Bytes (byte)
0.2 Delay Variation (msec)
1800 time (sec)
time (sec)
0.15
0.1
0.05
0
800 600 400 200 0
0
1000
2000
3000
4000
time (sec)
Figure 10. Delay variation vs. time for TCP flow
0
600
1200
1800
2400
3000
3600
time (sec)
Figure 14. Delay variation vs. time for UDP flow
6. Conclusion and Future Work In this paper we presented a passive QoS measurement and monitoring system based on IPFIX protocol. To measure the traffic, we need two measurement points that assign synchronized timestamps to the packets. These measurement points
The Third Advanced International Conference on Telecommunications (AICT'07) 0-7695-2843-0/07 $20.00 © 2007
select packets by calculating a hash function over some fields of the IP packet’s header. Each meter assigns a label (ID) and a timestamp to the selected packet. The collector then analyzes the measured data and computes the QoS factors such as one way delay/ loss and delay variation. Simple processing at meters is one of the desirable properties of this system which makes it suitable for using in the network routers with high speed links. Another desirable property of the proposed monitoring system is its capability of passive measuring of the QoS parameters. Measuring one way delay/loss and jitter is usually done by active measurement methods. In active measuring, some test packets are sent to destination. As it was mentioned before, active method has some disadvantages like forcing additional load to the network. We can use the proposed system to measure per link/ hop network traffic. As meters select packets by calculating a hash function, if one packet is selected in one router it also will be selected at every other routers in its path. So it is possible to run the source and destination meters at the ingress and egress routers, respectively. The system has the capability to store the timestamps which are reported by all routers in the path. After path termination, it is possible to calculate the QoS factors for each link in the traversed path. The information which is obtained by the proposed monitoring system can be used by the Traffic Engineering (TE) subsystem. Furthermore, it is possible to design an Intrusion Detection System (IDS) to detect any possible attack in the network. As attack packets have special traffic pattern, the produced traces can be used to detect any type of attacks such as Distributed Denial of Service (DDOS).
[5] Zhang, K. and E. Elkin, "XACCT's Common Reliable Accounting for Network Element (CRANE) Protocol Specification Version 1.0", RFC 3423, November 2002. [6] Meyer, J., "Reliable Streaming Internet Protocol Detail Records", Work in Progress, August 2002. [7] Calato, P. and M. MacFaden, "Light-weight Flow Accounting Protocol Data Definition Specification Version 5.0", July 2002. [8] S. Leinen, "Evaluation of Candidate Protocols for IP Flow Information Export (IPFIX)", RFC 3955, October 2004. [9] Test Traffic Measurements (TTM) project of RIPE (Réseaux IP Européens) Network Coordination Centre (NCC), for more information visit: http://www.ripe.net/ttm/. [10] V. Paxson, J. Mahdavi, A. Adams, and M. Mathis “An Architecture for Large-Scale Internet Measurement” IEEE Communications Magazine, vol. 36 no. 8, pp. 48-54, August 1998. [11] Network Analysis Infrastructure (NAI) projects of National Laboratory for Applied Network Research (NLANR), for more information visit: http://mna.nlanr.net/infrastructure.html. [12] NeTraMet measurement tool, available at: ftp://ftp.auckland.ac.nz/pub/iawg/NeTraMet [13] Cooperative Association for Internet Data Analysis (CAIDA) projects, for more information visit http://www.caida.org [14] IST INTERMON project, www.ist-intermon.org [15] IST MESCAL project, www.mescal.org
7. References
[16] TCPDUMP public repository, http://www.tcpdump.org/
[1] Real-time Traffic Flow Measurement, IETF WG RTFM, for more information visit: http://www2.auckland.ac.nz/net/internet/rtfm.
[17] Bob Jenkins’ hash function web page, paper published in Dr Dobb’s journal, http://burtleburtle.net/bob/hash/doobs.html
[2] T.Zseby, M. Molina, F. Raspall, N.Duffield, “Sampling and Filtering Techniques for IP Packet Selection” Work in progress, Feb. 2005 , Available at: http://www.ietf.org/internet-drafts/draft-ietf-psamp-sampletech-06.txt
[18] M. Molina, S.Niccolini, N.G.Duffield: "A Comparative Experimental Study of Hash Functions Applied to Packet Sampling", International Teletraffic Congress (ITC-19), Beijing, August 2005.
[3] NetFlow, for more information visit: www.cisco.com/warp/public/732/netflow/index.html. [4] Calhoun, P., Loughney, J., Guttman, E., Zorn, G., and J. Arkko, "Diameter Base Protocol", RFC 3588, September 2003.
The Third Advanced International Conference on Telecommunications (AICT'07) 0-7695-2843-0/07 $20.00 © 2007
[19] N. G. Duffield, M. Grossglauser, “Trajectory Sampling for Direct Traffic Observation”, IEEE/ACM Transactions on Networking, 9(3):280-292, June 2001 [20] N.G. Duffield, A. Gerber, M. Grossglauser,``Trajectory Engine: A Backend for Trajectory Sampling'', IEEE Network Operations and Management Symposium 2002, (NOMS 2002) Florence, Italy, April 15-19, 2002.
