On the Accuracy and Overhead of Cisco Sampled NetFlow - CiteSeerX

1

On the Accuracy and Overhead of Cisco Sampled NetFlow Baek-Young Choi and Supratik Bhattacharyya

Abstract— Traffic measurement and monitoring are an important first step for network management and traffic engineering. With high-speed Internet backbone links, efficient and effective packet sampling are not only desirable, but also increasingly becoming a necessity. The Sampled NetFlow [10] is a Cisco router’s traffic measurement functionality with static packet sampling for high speed links. Since the utility of sampling depends on the accuracy and economy of measurement, it is important to understand sampling error and measurement overhead. In this paper, we first discuss fundamental limitations of sampling techniques used in the Sampled NetFlow. We assess the accuracy of the Sampled NetFlow by comparing its output with complete packet traces [8] from an operational router. We also show the overheads involved in the Sampled NetFlow. We find that the Sampled NetFlow performs correctly without incurring dramatic overhead during our experiments. However, a care should be taken in its use, since the overhead is linearly proportional to the number of flows recored.

I. I NTRODUCTION Traffic measurement and monitoring serve as the basis for a wide range of IP network operations, management and engineering tasks. Particularly, flow-level measurement is required for applications such as traffic profiling, usage-based accounting, traffic engineering, traffic matrix, and QoS monitoring. With today’s and future high-speed (e.g., Gbps or Tbps) links, monitoring every packet traversing a measurement point may no longer be feasible. Furthermore, the large number of flows observed on today’s high-speed links introduces scalability issues. Capturing every packet and recording statistics of evert flow require too much processing capacity, cache memory, and I/O and network bandwidth, in order to update, store, and export flow records. Packet sampling has been suggested as a scalable alternative to address this problem. Both the Internet IETF (Internet Engineering Task Force) working groups, IPFIX (Internet Protocol Flow Information eXport) [7] and PSAMP (Packet Sampling) [9], have recommended the use of packet sampling. Static sampling method such as “1 out of k” is being used by Cisco for high-speed backbone routers [10]. The fundamental questions regarding sampling are its accuracy and economy. An inaccurate packet sampling not only defeats the purpose of traffic measurement and monitoring, but worse, can lead to wrong decisions by network operators. Particularly, when it comes to accounting, users would not make monetary commitment based on erroneous and unreliable data. Efficiency of packet sampling is also an important concern. Excessive oversampling should also be avoided for the measurement solution to be scalable. The current network management and engineering exercise drives the need of traffic measurement. Sampled NetFlow is

the primary traffic measurement tool built in routers deployed in many ISP networks such as Sprint, and it is the standard of IPFIX Working Group in IETF. Therefore, it is important to understand the accuracy of estimation and overhead of measurement involved in Sampled NetFlow. Few studies have been done regarding NetFlow related issues. [12] studies how much (full) NetFlow loses or gains information due to the flow-level granularity of traffic summary, compared to fine grained packet level and coarse grained SNMP report. In particular, bidirectional TCP connection summaries are constructed from two unidirectional NetFlow statistics and compared with the ones from packet traces collected by tcpdump. Also byte count summary of NetFlow is compared with SNMP byte count. They found that TCP connection summaries can be recovered fairly accurately with their proposed methodology. Also, total load (byte count) computed by NetFlow was close to SNMP statistics unless aggregation time period is not too small. In this study, we address the issue of the accuracy of Sampled NetFlow by comparing the statistics from packet traces captures at the same time from an operation router. We find that the Sampled NetFlow performs correctly without significant overhead under the load of our experiment. However, the amount of samples increases proportionally with amount of traffic which could be of an operational concern during peak periods. We also compared the systematic and random samplings as both are employed in the Sampled NetFlow, and our experiment indicates that their performances are similar. To the best of our knowledge, this is the first effort to measure and assess the overhead and accuracy of Sampled NetFlow with packet traces from an operational router. The remainder of the report is organized as follows. In Section II we discuss theoretical aspects of sampling techniques used Sampled NetFlow. We present our experimental setup and the evaluation results using network traffic traces in Section III. We summarize our findings in Section IV. II. T HEORETICAL L IMITATIONS OF S AMPLING T ECHNIQUES In this section we first give background on how traffic is estimated using samples, and discuss on the optimal number of samples for a certain accuracy. We then discuss the limitation of accuracies with systematic and random sampling techniques. A. Traffic estimators with samples With sampled packets, various actual (population) parameters can be estimated such as population mean (mean packet

2

TABLE I

B. Optimal Number of Samples

N OTATION . Notation m n V Vˆ pf pˆf nf mf m ˆf vf vˆf Sf

Explanation total number of arriving packets total number of sampled packets actual total byte count estimated total byte count actual proportion of flow f in packet counts estimated proportion of flow f in packet counts (r.v.) number of sampled packets of flow f (r.v.) total number of packets of flow f estimated packet count of flow f (r.v.) actual byte count of flow f estimated byte count of flow f squared coefficient of variation of packet sizes of a flow f

size), total (i.e., total load), and proportion (i.e., flow size). Assume that there are m packets arriving in a measurement interval, and let Xi be the size of the ith packet. Suppose we ˆj , sample n, 1 ≤ n ≤ m, packets out of the m packets. Let X j = 1, 2, . . . , n, denote the size of the jth sampled packet. In order to estimate mean packet size µ, the average of sampled packet sizes µ ˆ are used as follows: Pn ˆ j=1 Xj µ ˆ= (1) n Let usPdenote the traffic load of a measurement interval with m V = i=i Xi . Then the traffic load V can be estimated by Vˆ using the samples, where Vˆ is given by n

mX ˆ Xj Vˆ = n j=1

(2)

Flow estimation is a bit complicated than the mean packet size or total load estimation. Consider a time interval that contains packets of a flow f . Let mf packets belong to flow f out of total m packets. Then the estimated packet count of the flow, m ˆ f is computed by m ˆf = m·

nf = m · pˆf n

(3)

where pˆf is the sample proportion1 of flow f . Then, the actual byte count of a flow f is expressed as follows: v f = mf µf = mpf µf

(4)

where µf is the actual average packet size of flow f . Similarly the estimated flow byte count vˆf is vˆf = m ˆ fµ ˆf = mˆ pf µ ˆf

(5)

where µ ˆf is the estimated average packet size of flow f . Notice that two levels of uncertainties are involved for flow byte count estimation, namely the estimations of flow proportion and flow average packet size. 1 A proportion may be considered to be a special case of the mean where a variable Y takes on only the values 0 and 1.

We briefly discuss the optimal number of samples for a given accuracy in the context of total load as well as flow size. Intuitively, the accuracy of estimation corresponds to the variability of object being measured. Consider a case where all packet sizes are the same for instance. Then only one sample is enough to perfectly estimate mean packet size or total byte count (if exact sampling rate is known). However, when the of packet sizes are variable, as in the case of current Internet packets, more packet samples are needed to achieve a reasonable accuracy of estimation. we want to bound the relative error of total load Suppose Vˆ −V V within a prescribed error tolerance level given by two parameters {η, ε} (0 < η, ε < 1), i.e., ) ( Vˆ − V (6) Pr > ε ≤ η. V

From the central limit theorem of random samples2 [2], as the sample size n → ∞, the average of sampled data approaches the population mean, regardless of distribution of population. Thus (6) can be rewritten as follows: ( Vˆ − V Pr V

)   √  εµ n ≤ η, >ε ≈2 1−Φ σ

(7)

where µ and σ are, respectively, the population mean and standard deviation of the packet size distribution in a block, and Φ(·) is the cumulative distribution function (c.d.f) of the standard normal distribution (i.e.,N (0, 1)). Hence, to satisfy the given error tolerance level, the required number of packet samples must satisfy  −1 2 Φ (1 − η/2) σ ∗ n≥n = = zp · S (8) · ε µ  −1 2 where zp = Φ (1−η/2) and S = (σ/µ)2 is the squared ε coefficient of variance (SCV ) of the packet size distribution in a block. Eq. (8) concisely relates the minimum number of packet samples to the estimation accuracy and the variability in packet sizes. In particular, it states the minimum required number of packet samples, n∗ , is linearly proportional to the squared coefficient of variance, S, of the packet size distribution in a block. Now, let us consider the flow size estimation problem. Namely,  f   f  m vˆ − v f ˆ − mf > ε ≤ η (9) P r > ε ≤ η, P r vf mf where pf ≥ pθ for flow f . Eq. (9) can be rewritten as follows: ) ( mpˆf − mpf Pr >ε ≈2 1−Φ mpf

p

p

√ !! pf nε

(1 − pf )

≤η (10)

2 The requirement that samples be i.i.d (independent and identically distributed) is achieved by random sampling from the same population.

3

By solving the inequality in Eq. (10) with respect to n, we can derive the minimum required number of samples n∗,p to estimate flow packet count within the given error tolerance level    1 − pf ∗,p (11) n ≥ n = zp · pf

to 0, systematic sampling is roughly equivalent to simple random sampling. When the population is randomly ordered, systematic sampling will give us a precision approximately equivalent to that obtainable by simple random sampling.

Similarly, the required number of samples for flow byte count estimation can be obtained as,    1 − pf + S f n ≥ n∗,b,f = zp · (12) pf

where S f = (σ f /µf )2 is the squared coefficient of variation (SCV) of packet sizes of flow f . More details and the choice of sampling rate can be found in [4], [5]. In the next subsections, we discuss on the accuaracy of systematic and random sampling techniques with a fixed number of samples, and illustrate how traffic characteristics can impact on the accuracy. C. Systematic sampling: Original Sampled NetFlow

Systematic sampling is a popular and primary sampling design employed in Cisco routers [10] due to its simplicity and low overhead. In general, 1-out-of-k systematic sampling involves random selection of one element from the first k elements, and selection of every kth element thereafter requiring only one random number generation and a counter. We discuss the accuracy of a sampling technique with the variance of estimation, since it tells how reliable the estimation is. For the rest of this section, we simplify the discussion of the variance to a mean estimation. The performance of systematic sampling can be explained with the concept of correlation between samples of an experiment (sample set). The variance of sample mean using systematic sampling is given by [13], [11] 2 ˆsys,n ) = σ [1 + (n − 1)ρ] V ar(X (13) n where ρ is a measure of the correlation between pairs of samples within the same systematic sample.

ρ=

E(Xij − µ)(Xij 0 − µ) E(Xij − µ)

(14)

1 where, − (n−1) ≤ ρ ≤ 1, i = 1, . . . , k and j, j 0 = 1, . . . , n, 0 j 6= j . Thus, a theoretical accuracy of systematic sampling is not practically assessable, as the knowledge on all k systematic samples is necessary to calculate the variance of systematic samples. Eq. (13) also shows that when ρ is positive, the estimator is not consistent, since the accuracy is not increased with large n. If ρ is close to 1, then the variability of elements within the sample set is too small compared to variability among possible sample sets, and systematic sampling will yield a larger variance than using simple random sampling. If ρ is negative, then systematic sampling may perform better than simple random sampling. The correlation may be negative if variability of elements within a systematic sample set tends to be larger than among systematic sample sets. For ρ close

Fig. 1.

Systematic sampling on periodic population

Figure 1 illustrates extreme performance of systematic sampling from the same population. Suppose one try to estimate a population mean with a systematic sampling. For the periodic traffic shown in Figure 1, when k(= m/n) is the same as the period, the value of the sample is the same for all samples in any possible set of samples, thus, an increase of sample size would not increase the accuracy at all. Meanwhile, with k = 4(> 3), it always gives the exact population mean with smaller number of samples which is better than random sampling. Therefore, randomness in samples is important to assess the accuracy, and to avoid extreme performance. Recall that systematic sampling is first employed due to its simplicity and low overhead compared to random sampling which involves the overheads of random number generation per - each packet or per - samples. However, random number generation is not considered costly, in modern routers. In fact, most of routers already have the feature implemented for a mechanism such as RED (Random Early Detection) or newer versions of Sampled NetFlow. D. Random Sampling: New Sampled NetFlow Static random sampling selects packet randomly with a fixed sampling probability. Random sampling is known to provide unbiased estimators for the population mean, total, and proportion, in that their expectations are equal to the values of population (e.g., E(ˆ µ) = µ, E(ˆ pf ) = pf ). The technique is also consistent, since the estimation of stratified random sampling approaches the population parameter as the number of samples increases. i.e., pˆf → pf as n → ∞ (or m). Efficiency of a sampling describes how closely a sampling distribution is concentrated around the value of the population (population parameter). For consistent estimators, efficiency can be measured by the variance, where a smaller variance is preferred. An estimator of a smaller variance would give more accurate estimation, given the same number of samples. Mean square error (MSE) is a frequently used metric to compare estimators. Let X be a random variable of the population (in ˆ be the estimated mean of the population. (In general) and X case of proportion, X takes on 1 if a packet belongs to flow f and 0 otherwise.) As in the following equation, the variance

4

itself becomes MSE for an unbiased estimator. ˆ = E(X ˆ − µ) = V ar(X) ˆ + bias M SE(X) 2

2

(15)

Thus it is important to study variance carefully. The variance mean estimation of simple random sampling with n samples can be derived as follows: Pn ˆ Xi σ2 var (ˆ µ) = i=1 = (16) n n The variances of total load (Vˆ ) or flow byte count estimation (ˆ v f ) can be found by using the results of a variance of mean estimation (ˆ pf or µ ˆf ). For example, V ar(m ˆ f ) = V ar(mˆ pf ) = m2 V ar(ˆ pf )

(17)

Particularly, the variance of relative error can be derived as below: ! Pn ˆ ! Vˆ − V i=1 Xi = var var V nµ  2 S 1 · n · σ2 = = (18) nµ n

where S (= σ 2 /µ2 ) is a squared coefficient of variation of packet sizes. III. S AMPLED N ET F LOW E VALUATION A. Experiment Data

In order to study the accuracy and overhead of the Sampled NetFlow, we turned on the Sampled NetFlow in an operational router, and collected its export statistics and router resource usage. The operational router of our study is a Cisco GSR 12000 series router. The router has a 200 Mhz MIPS R5000 family processor with 512 MB of memory. This router type and configuration is common across the Sprint IP network. There are two OC-48 links, one OC-12 link and three OC-3 links in the router. We turned on the Sampled NetFlow on all of the interfaces with a sampling frequency of 1-in-250. NetFlow is currently supported on all interfaces and Cisco IOS platforms collecting unidirectional flows of inbound traffic. It works with Cisco Express Forwarding (CEF) [1] and is not a part of switching path. NetFlow export packets are mostly 1500 bytes. One packet can typically contain 20-50 flow records. NetFlow version 5 is used in our experiment. The export format includes flow information of packet count, byte count, start time, end time, input interface, output interface, type of service, TCP flags, protocol, source IP address, destination IP address, source port, destination port, next hop address, source AS number, destination AS number, source prefix mask, and destination prefix mask. NetFlow report statistics directly from the raw sample packet, scaling up to the sampling rate is user’s task. We have processed the Sampled NetFlow data with arts tool [3]. While we collect the Sampled NetFlow statistics from the router, we also passively capture packets from the six links of the router using IPMON systems [8] which consist of optical splitters and packet capturing cards [6]. The entire packet traces allow us to evaluate the accuracy of the Sampled

NetFlow. Our monitoring systems capture the first 44 bytes of every IP packet, and timestamp each of them with a GPS (Global Positioning System) synchronized clock. The granularity of the timestamp is sub-microsecond. Among the six links traces of the router, in this report we demonstrate the results from a OC-3 links of the router. The results are similar with other interfaces. Furthermore, we also collect router resource usages such as router CPU and memory usages using router interactive commands, so as to assess the overhead of the Sampled NetFlow on the router. We collect the router’s resource usage statistics via the command line interface (CLI). We use the ’show memory’ and ’show process cpu’ commands every 30 minutes 3 to obtain the memory and CPU utilization. The first command gives the available memory and the consumed memory. The second command gives the overall CPU utilization averaged over three different time intervals: 5 seconds, 1 minute and 5 minutes. So the 5 seconds is the minimum interval of router CPU utilization one can measure. B. Overhead of the Sampled NetFlow We first examine the overhead of the Sampled NetFlow in terms of the router resource usage and the amount of export. Router resource usage may impact on other performance of routers and the amount of export affects the bandwidth used by measurement data as well as the storage and processing overhead on its analysis system. We observe CPU utilization and memory usage, as they are the main router processor resources. As shown in Figure 2, the router CPU utilizations exhibit larger fluctuations at time scale of 5 seconds often up to or nearly 100%, even though 1 minute and 5 minute exponentially averaged CPU utilization wave over a small range. However, we find that those spikes are not due to NetFlow processing and ’BGP Scanner’ is the main process that takes the most CPU cycles among the process list. Figure 3 shows the snapshot of the router’s memory usage every 30 minutes. Unlike CPU utilization, the router’s memory usage stays fairly constant for all the measurements. Also, we have found that most of the memory is used by routing (BGP, ISIS) related processes. The bandwidth consumption by the Sampled NetFlow export is plotted against the number of flows in Figure 4. The time series of the number of UDP NetFlow export packets and the number of flows exported are plotted in Figure 5, and Figure 6, respectively. Note that bandwidth usage and the number of export packets are proportional to the number of flows. Even though flow export takes less than 10M b most of times, occasionally large amount of export packets may be generated, as the distinctive outlier in the Figures. Thus, large number of flows could potentially impact seriously on resources related to the (Sampled) NetFlow export. In such a case, the impact might be mitigated by aggregating 7-tuple flows into bigger granularities before export. However, the feature of aggregation before export is only supported in the 8 or higher versions of NetFlow. 3 We limited the frequency so as not to minimize the impact of router operation.

5

100

170

5 sec 1 min 5 min

90

169.5

80

169 memory usage (MB)

CPU load (%)

70 60 50 40 30

168.5 168 167.5 167 166.5

20

166

10

165.5

0

165

06:00 09:00 12:00 15:00 18:00 21:00 00:00 03:00 time

Fig. 2.

06:00 09:00 12:00 15:00 18:00 21:00 00:00 03:00 time

Router CPU utilization

Fig. 3.

Router memory usage 6

60

5

2.5

7

x 10

6

40

30

20

5 1.5

3

1

2 1

1

2

3 4 No. flows

5

6

7 6

0 0

50

x 10

Bandwidth used by the Sampled NetFlow Fig. 5. packet

100 time series

150

200

0 0

50

100 time series

150

200

Number of the Sampled NetFlow export Fig. 6. The number of flows reported by the Sampled NetFlow

C. Correctness of the Sampled NetFlow Router features (also any system functionality) may not perform exactly as it is claimed due to implementational issues. We measure how close the Sampled NetFlow sampling performs as it is supposed to. The Sampled NetFlow in our study is configured with 1-in-250 systematic sampling. We investigate if the actual sampling frequency is indeed 1-in-250. For a given interface (link) and a time interval, we first compute total number of sampled packets and the corresponding start time and end time in unit of seconds from the NetFlow statistics. Then, with the IPMON packet trace from the corresponding link, we compute exact total number of packets during the time interval reported by NetFlow. The ratio of total packets and number of samples yields the sampling frequency (k of ’1-in-k’ sampling).4 There was be a difference between the router timestamp used in NetFlow statistics and GPS timestamp used in IPMON packet traces. Therefore, we have performed a clock shift correction with which it gives a minimum deviation of sampling frequency. A caveat is that since the second is the finest time granularity provided by NetFlow, the error of as much as one second 4 Even

4

0.5

10

Fig. 4. export

2

No. flows

No. NetFlow export packets

NetFlow export BW (Mb)

50

0 0

x 10

though sampling frequency refers to ’1-in-k’, we use ’k’ for simplicity when it does not cause confusion.

(interval boundary) could be caused by our measurement. In Figure 7, we plot the number of samples, the number of total packets, and sampling frequency against on the same time intervals. The time interval is 0.2 hour (900 seconds). The top two plots exhibits that the number of samples is proportional to the total number of packets, as it should be the case of a static sampling. The bottom most plot shows that sampling frequency is fairly close to the configuration, except the first point. The standard deviation of the sampling frequency is 3.64. The small value of the total packet count of the first interval is likely due to the fact that the packet traces starts in the middle of a second rather than right beginning of a second. After removing the outlier, the standard deviation becomes down to less than 1 (exactly 0.64). A small variation can be caused by various factors such as sub-second level clock shift and a statistical variation. 5 Therefore, we conclude the Sampled NetFlow conforms to the frequency configuration.

5 For instance, consider an interval in which the total number of packets is larger than 250 and less than 500. Then systematic sampling will collect one or two samples depending on the initial seed.

6

x 10

random systematic

0.8 0 6 0 x 10 6

20

40

60

80

100

4

2 0 300

k (1−in−k)

1 0.9

20

40

60

80

100

cumulative probability

No. total packets

No. samples

4

5

0.7 0.6 0.5 0.4 0.3 0.2

250 200 0

Fig. 7.

0.1 20

40 60 Time interval (unit: ~ 900 sec)

80

100

No. samples, No. total packets, sampling frequency vs. time

D. Performance of systematic sampling vs. random sampling Now we compare the performance of systematic sampling and random sampling, empirically. Original Sampled NetFlow only employed systematic sampling and it is still the common use of the Sampled NetFlow. In Section II we have shown that the performance bound of systematic sampling can not be assessed unless the periodicity (or correlation) of packets is known. Practically, in high aggregation links (such as backbone links), one may assume packet arrivals are uncorrelated or random. To validate the assumption, we perform systematic sampling (with sampling frequency 1-in250) and static random sampling (with sampling probability equals to 0.004 (=1/250)) on the packet trace, and compare their relative error in estimating total load. We conducted the experiment over 200 times in order to assess their performance statistically. In Figure 8 the cumulative empirical probability is plotted with its relative error. It indicates that the performance of systematic and static random sampling is not distinguishable in practice. We believe it is true in most of backbone links where the degree of multiplexing of flows is high. IV. S UMMARY Packet sampling is a must in today’s high speed links to reduce the measurement overhead. NetFlow is the primary traffic measurement tool built in Cisco routers and the IETF standard. However, its performance and feasibility study should proceed before its wide deployment. In this paper, we first discussed theoretical limitations of sampling techniques used in Sampled NetFlow. It has fundamental limitations due to static sampling frequency. As the accuracy of estimation depends on the characteristics of traffic as well as the number of samples, static sampling does not always ensure the accuracy of estimation. Furthermore, the amount of samples increases proportionally with amount of traffic which could be of an operational concern during peak periods. Thus, for the measurement scheme to be accurate, scalable, and robust, an automatic adaptation of sampling frequency

0 0

Fig. 8.

0.005

0.01 relative error

0.015

0.02

The number of the Sampled NetFlow export packet

would be a desirable improvement for Sampled NetFlow. We have assessed the overhead and validated the correctness of Sampled NetFlow. We find that the Sampled NetFlow performs correctly without significant overhead under the load of our experiment. However, a care should be taken in its use, since the overhead is indeed linearly proportional to the number of flows recored. We have also compared the performance of systematic and random sampling as implemented in different Sampled NetFlow versions. Even though the theoretic performance of systematic sampling is not provable nor boundable without the knowledge on the correlation of traffic, our experiments show that the performances of the both are similar. R EFERENCES [1] Cisco Express Forwarding. http://www.cisco.com. [2] D. Berry and B. Lindgren. Statistics theory and Methods. Duxbury Press, ITP, 2nd edition, 1996. [3] David Bolen and licensed to CAIDA (The Cooperative Association for Internet Data Analysis). ARTS. http://www.caida.org/tools/utilities/arts. [4] B.-Y. Choi, J. Park, and Z.-L. Zhang. Adaptive Packet Sampling for Total Load Estimation. In IEEE International Conference on Communications (ICC’03), Anchorage, Alaska, May 2003. [5] B.-Y. Choi, J. Park, and Z.-L. Zhang. Adaptive Packet Sampling for Accurate and Scalable Flow Measurement. In to appear in the Proceedings of IEEE Global Internet Symposium (Globecom’04), Dallas, TX, Nov/Dec 2004. [6] DAG. DAGMON SONET network interface. http://www.endace.com. [7] IPFIX. Internet Engineering Task Force, IP Flow Information Export Working Group. http://www.ietf.org/html.charters/ipfix-charter.html. [8] Sprint ATL IPMon project. http://ipmon.sprint.com. [9] PSAMP. Internet Engineering Task Force Packet Sampling Working Group. https://ops.ietf.org/lists/psamp. [10] Cisco Sampled NetFlow. http://www.cisco.com. [11] T. R. Scheaffer, W. Mendenhall, and R. Ott. Elementary Survey Sampling. Duxbury Press, 5th edition, 1995. [12] R. Sommer and A. Feldmann. NetFlow: Information Loss or Win. In ACM SIGCOMM Internet Measurement Workshop, Marseille, France, November 2002. [13] T. Yamane. Elementary Sampling Theory. Prentice-Hall, Inc., 1967.