Inline Measurements: A Native Measurement Technique for IPv6 Networks D. P. Pezaros1, Student Member, IEEE, D. Hutchison1, Member, IEEE, R. D. Gardner2, Member, IEEE, F. J. Garcia2, J. S. Sventek3, Member, IEEE 1
Communications and Distributed Systems, Computing Dept, Lancaster University, Lancaster, LA1 4YR, UK. 2 Agilent Laboratories Scotland, Agilent Technologies, Edinburgh, EH30 9TG, UK. 3 Networked Systems Measurement & Control Group, Computing Science Dept., Univ. of Glasgow, Glasgow, G12 8QQ, UK. E-mail: {dp, dh}@comp.lancs.ac.uk, {frankie_garcia, robert_gardner}@agilent.com,
[email protected] Abstract – The ability to measure, monitor and control network traffic is becoming of crucial importance as the traditionally best-effort Internet Protocol is increasingly used to transport aggregated, time-critical data flows alongside other heterogeneous traffic. Existing service measurements fall into two main categories: active and passive. This paper introduces another complementary technique called ‘inline measurements’ that makes use of the extendible features of the emerging IPv6 protocol. Through the exploitation of native IPv6 extension headers, measurement triggers and measurement data may be carried in the same packets as the payload data itself, providing a high level of probability that the behaviour of the real user traffic flow is being observed. The paper also presents the results from a dynamically configurable prototype implementation in which end-to-end, one-way delay and delay variation of real-time video streams have thus far been measured, as well as the ‘goodput’ of services operating on top of reliable transport. Index Terms – active, passive, measurement, monitoring, performance, quality of service, service management, IPv6, extension headers, delay measurement, real-time traffic.
I. INTRODUCTION The Internet Protocol (IP) is emerging as the ubiquitous, universal convergence layer in the gradual marriage of telephony networks with data communications networks. The result is the increasing aggregation of multi-service traffic on IP networks that operate, by nature, according to the ‘best-effort’ paradigm. Within this environment supplementary mechanisms are required to ensure the IP network is able to deliver the appropriate quality of service (QoS). This is particularly important for high revenue generating, time-critical data flows, such as voice calls [1, 2, 3]. Various QoS techniques are being developed, such as DiffServ [4] and IntServ/RSVP [5] or overlay models involving MPLS, ATM and Frame Relay, which are able to provide a range of QoS levels that can be categorised in terms of synchronicity, latency, jitter, loss and throughput. However, key to the success of delivering good quality of service is the ability to measure and monitor the service performance provided by the network and provide appropriate, timely feedback to maintain QoS levels.
At the lower layers, measurement and monitoring allow network operators to gauge the real-time health of the network through, for example, detecting traffic congestion and delay, measuring throughput, checking link / path availability, verifying routing/forwarding mechanisms and calculating error rates. Measurement and monitoring enable responsive traffic engineering, providing a path towards automated provisioning and optimisation within the network. At the service layer, network service providers use measurement and monitoring to check if service level agreements are being met, to monitor individual end-user service quality, to provide billing, policing and fraud detection, to determine traffic levels for short and longer term provisioning and thus avoid the expense of over-provisioning. QoS measurement techniques fall under two main categories: active and passive [6, 7]. Active measurements are performed by injecting traffic with known characteristics into the network at one point and observing the same traffic at another point and are often used to test specific attributes of a service, such as available bandwidth and one-way delay. On the other hand, passive measurements involve observing real traffic on a link without disruption to the service, often employing filtering and event search mechanisms to continuously maintain and update attribute counters. Passive measurements are commonly used for onepoint packet filtering and counting measurements. Figure 1 shows a general measurement structure that may apply to network management, in which active or passive measurements may be performed, together with feedback to control the network and adapt the incoming traffic. Actuators IN PU TS
M E A SU R ED SYS TE M
O UTP UTS Sensors M easurem ent Traffic
C ontrol
M easurem ent & C ontrol A pplication E xternal S ystem s
Figure 1: General System Measurement Structure
In this paper, a complementary measurement technique for data networks is introduced that exploits the new, enhanced features of the IPv6 protocol [8]. Named ‘inline measurements’, this technique represents an alternative to active and passive measurements and is particularly suited to directly and transparently assessing the performance properties experienced by real user traffic at the IP layer. The term ‘inline’ implies that measurement invoking triggers and the measurement data are piggybacked onto real user packets using the extendible features of IPv6. Having potentially low overhead and minor impact on the network traffic, this service-oriented technique measures, with a high level of probability, the real user experience and it is equally applicable to measuring certain aspects of aggregate flows as it is to particular applications or protocols. This paper is organised as follows: Section II presents the IPv6 inline measurement technique, outlines various strengths and weaknesses compared to alternative approaches and discusses appropriate application areas. Section III describes one-way delay measurements carried out using IPv6 inline measurements. Section IV concludes the paper, describing some future work that is intended in this area. II. INLINE MEASUREMENTS A. General description of the technique The inline measurement technique requires that measurement data and triggering mechanisms be piggybacked onto real user packets and it may be viewed in part as a hybrid of active and passive measurement approaches – see Figure 2. Measurement trigger and/or data added
Payload
Measurement is triggered and/or data removed
Hdr
the same path as untagged user traffic. Thus an accurate reflection of the characteristics of the real user traffic flows can be provided, with the proviso that, for appropriately selected packets, the marginal increase in the header length does not change how the packet is treated on its journey through the network and the traffic is not singled out for special treatment by the network. There is also a small additional systematic processing delay. These issues are detailed in subsection C. B. Implementation via IPv6 Extension Headers Native Internet Protocol version 6 (IPv6) extension headers can be conveniently used to implement inline measurements [8]. IPv6 has a common, 40-octet, fixed-sized header that holds the addresses and potential QoS capability – see Figure 3. Data for other functionality is implemented via a daisy-chain of optional extension headers positioned before the payload that hold small data structures called ‘options’. In contrast to IPv4, IPv6 extension headers and their options are only processed where necessary in the network and there are specific rules relating to the order in which they must be processed. The IPv6 standard defines a small number of extension headers such as Hop-By-Hop Options, Destination Options, Routing, Fragment, Authentication and Encapsulating Security Payload. However, other extension headers and options can be defined for dedicated purposes – e.g. those within recent Mobile IPv6 IETF drafts. 0
4
12
16
32
Version Traffic Class
Flow Label
Payload Length
IPv6 fixed sized header
Extension Header(s)
Next Header
Hop Limit
Source Address
Destination Address Next Header
Hdr Ext Len
Next Header
Hdr Ext Len
Figure 2: Inline Measurement Technique
Inline measurements are intrinsically multi-point measurements whereby packets are tagged with information at one point in the network and this information is augmented, retrieved and/or observed at a point or points elsewhere. As a result, the complex task of correlating measurements from multiple points in the network is circumvented, as it is entirely certain that the same packet has been observed at any chosen point in the network. Similarly, because any added measurement data will be piggybacked onto real user traffic, it will, with a high degree of probability, receive the same treatment and follow
Data
Data
Figure 3: IPv6 header structure
It is proposed that the Destination Options extension header be used to carry additional, bespoke options to convey measurement data such as timestamps, counters, trace information or other associated measurement system traffic. The Destination Options extension header is a simple header that is used to hold standalone options to be processed by a packet’s destination and is thus ideal for per-
forming end-to-end inline service measurements. The Destination Options extension header is illustrated in Figure 4. Next Header Next Header (1 octet)
Hdr HdrExt ExtLen Len (1 octet)
Options TLV-encoded Options (variable number of octets)
Figure 4: Destination options extension header
The next header field contains a code identifying the subsequent header or higher-layer protocol payload type. The Hdr Ext Len field holds the length, in octets, of destination options header excluding the Next Header Field. The options field is of variable length and contains destination options encoded as type-length-value (TLV) tuples, which represent a suitable format for the transportation of opaque objects – see Figure 5. Options also have a type and a length. They are processed in sequence and, by setting bits in the type field, may be skipped or an error issued if unrecognised by a processing entity. O ption T ype 1 octet
O ption Length
D ata
1 octet
varia ble num b er of octets
Figure 5: TLV-encoded option structure
Using a combination of destination options and appropriate measurement infrastructure, it is possible to selectively add data to real user traffic, which can be detected and processed elsewhere in the network making use of the IPv6 stack. The level of processing will be determined by the options carried and may involve operations such as incrementing counters, adding timestamp annotations, extracting and caching packet data. Note that with the addition of a routing header, it would even be possible to target specific ‘destinations’ en-route to enable the implementation of more detailed service measurements as the user traffic crosses pertinent points. A particularly elegant approach to inline measurement implementation is to use fixed or temporary extensions to the existing IPv6 protocol stack, the approach used to produce results for this paper. Temporary extensions can be achieved using dynamically loadable modules with appropriate hooks in the kernel stack, allowing the extension header processing rules defined within the IPv6 standard to be maintained. Alternatively, packets could be suitably instrumented and observed via separate hardware/firmware elements that may exist inside interface cards or just outside, attached as a separate plug-in module. A full description of implementation methods including a discussion on appropriate active networking techniques is outside the scope of this paper.
C. Comparison with other techniques The inline measurement technique has benefits and drawbacks and it should thus be considered as another potentially useful approach to measurement that may complement existing solutions. Just like the active and passive measurement approaches, they are not applicable for every type of measurement. A comparison of inline, active and passive measurement techniques is given in Table 1, in which ‘+’ / ‘-’ represent perceived advantages / disadvantages respectively. The main advantages of inline measurements are minimal additional load on the network, ability to measure the ‘real user experience’, two-point measurement capability without requirement for data shipping and correlation, the ability to perform policy-based measurements on a dynamically deployable basis, and a suitably wide variety of measurement application areas. The main disadvantages of inline measurements are their intrusive nature, requirement for an accommodating protocol, security and privacy concerns since some packets must be modified, and that they can only operate when there is traffic available. One of the main difficulties with two-point passive measurements is the need to correlate samples collected at two distinct observation points to yield one-way flow measures. Guaranteeing that both observation points trigger on the same packet and the subsequent data correlation are challenging tasks and shipping data can consume significant amounts of network bandwidth. Besides, for some metrics it is exceptionally difficult to see how one can measure them passively (e.g. route taken by packets) without scalability and complexity arising as significant restraining factors. A particular problem with active measurements is that performance properties measured by active techniques do not necessarily reflect the behaviour experienced by operational network traffic since synthetic traffic may not have exactly the same statistical properties and may be treated differently if too easily distinguishable. The additional traffic unavoidably impacts the network and may itself be a factor in measuring an otherwise poorer performance. Also, periodic sampling and packet injection used by some techniques may fail to observe periodic network events [9]. D. Appropriate application areas Inline measurements are well-suited to end-to-end, twopoint and multi-point measurements such as packet tracing, one-way and round-trip delay, and path loss. For example, tagging various types of signalling traffic may facilitate service level agreement verification and/or application troubleshooting. Moreover, the same inline measurement framework can perform passive type one-point measurements that involve filtering and classifying passing packet flows and maintaining various attribute counts, particularly useful for network performance monitoring and troubleshooting.
TABLE 1: SUMMARY OF BENEFITS AND DRAWBACKS OF PASSIVE, ACTIVE AND INLINE MEASUREMENT TECHNIQUES Aspect/Property
Active Measurements
Impact on network (Measurement process)
- Intrusive: Generates additional load which ++ Non-intrusive: No impact on network competes for resources
+ Intrusive: Marginal load increase and minor delay might be incurred
+ Load generated at one end point
- Load generated at one or both ends
+ Load generated at one end point
Confidence
- Artificially injected traffic used to infer/predict experience of real traffic - Test traffic may be treated differently - Injected traffic affects performance
+ Measures real user traffic
+ Measures real user traffic - Possibility that instrumented traffic is distinguishable and treated differently
Controllability
+ Can test any traffic, path, method of sampling, protocol, etc. – at any time.
- Can only measure available traffic
Security/Privacy issues
+ Private, injected traffic + Real data not examined
- Observing real traffic
-- Observation and modification of real traffic
Scalability issues
+ Can be dynamically deployed on a per interface basis + Can inject a chosen amount of traffic
- Probes per interface at ingress & egress - Full packet capture is not scalable + Can use filtering and sampling
+ Can be dynamically deployed on a per node or per interface basis + Can use filtering and sampling
+ Correlation not required Complexity and Processing - Non-trivial generation of statistically representative test patterns
- Correlation of large quantities of data from ingress and egress is computationally intensive and doesn’t scale well
+ No correlation
Two-point measurements: Quality of Service testing, such as available bandwidth, trip delay, and packet loss.
One-point measurements: packet filtering and counting to obtain traffic type, source / destination etc.
Multi-point, policy-based measurements, active troubleshooting, packet loss, delay, tracing, routing, packet / flow foot printing.
Impact on network (Measurement data)
Major application areas
Passive Measurements
- Eavesdropping not possible Other comments
- Requires substantial expertise to produce meaningful test patterns
+ Eavesdropping possible
Adopting a more active approach, inline measurements lend themselves well to policy-based, event—condition— action operations, where the occurrence of certain packets triggers measurement or other network management activity, such as billing and accounting functions. III. ONE-WAY DELAY MEASUREMENT
IPv6 router
shaper Private Network
IPv6 Network
Private Network
Access Point
Router
University IPv4 Backbone
red
IPv6/IPv4 Backbone Network
To demonstrate the feasibility and applicability of the inline measurement technique, various measurement experiments have been performed on different traffic types in an IPv6 network. Owing to space limitations, this paper describes only one such scenario: measurement of one-way delay for streamed UDP video traffic across the IPv6 measurement test bed at Lancaster University – see Figure 6.
JANET UK Academic Network
Time servers
blue W ireless node
Router
Figure 6: Lancaster University Measurement Test bed
Inline Measurements
- Can only measure available traffic - Requires an accommodating protocol
- Statistical sampling and filtering
+ Eavesdropping possible - Not applicable to all traffic types (e.g. real-time, max MTU traffic)
A. Framework Overview With a strong bias to placing intelligence in networking nodes where and when required, Application Level Active Network (ALAN) concepts were used in the development of a core set of experimental measurement application frameworks [10]. ALANs comprise a set of classes and components that together provide measurement system services while adhering to strict a object, organisational and collaboration model [11] and handle initialisation, control, co-ordination, and scheduling of tests in a distributed network. ALANs also provide capabilities for collection, wireformat representation and streaming of measurement samples to registered measurement data ‘consumers’. A more detailed description of the measurement framework is outside the scope of this paper. Underpinning the design philosophy is the notion of telemetry modules which are the basic components that instrument nodes to facilitate inline measurement techniques through the addition, modification and removal of data in the extension headers, as well as other supporting functions such as the storage, retrieval, correlation and forwarding of measurement-related data. These modules can be dynamically created and remotely configured, managed and controlled. B. One-way Delay Telemetry Module A one-way delay measurement typically requires that at least two different telemetry modules be operating simultaneously. On loading, each module hooks into the Linux kernel protocol stack, allowing packets to be modified as
required. Figure 7 shows the typical operation of the oneway delay modules at a source and destination node of a measured path.
1. Select Packet
One way delay destination module
Configuration from user space
3. Add Timestamp 4. Insert Ext Hdr
To user space
3. Add to output message queue
2. Create Ext Hdr
Options carried in packets to destination
one-way delay / ms
One way delay source module
IPPM working group. Figures 9 and 10 show the one-way delay and jitter, respectively, measured for a video stream from the “wireless node” to “Blue”.
2. Add Timestamp 1. Filter
Packet Flow
Figure 7: One-way delay source and destination modules
(Total option size: 22 octets)
Pointer
Option type
Option data len
Flags
(Reserved)
Overflow
tim e / s
Figure 9: One-way delay of video stream from “wireless node” to “Blue”
delay variation / ms
The source module selects a packet based on the filter and sampling criteria that were uploaded to the module in advance. Subsequently a 22-octet ‘one-way delay option’ (see Figure 8) is generated, timestamped and appended to the existing or a newly created destination options header. The destination module looks for appropriately instrumented IPv6 packets and adds the second time-stamp. The data in the IPv6, extension and transport layer headers then join a FIFO message queue, accessible from user space. If required, the extension header/option may be stripped from the packet returning the packet to its original form.
Source timestamp: seconds Source timestamp: microseconds Destination timestamp: seconds
tim e / s
Figure 10: IPDV of video stream from “wireless node” to “Blue”
Destination timestamp: microseconds 4 octets
Figure 8: One-way delay TLV-encoded option
Source and destination points are clock synchronised using an NTP or a GPS clock feed, depending on the degree of accuracy required. The open-source VideoLAN server/client pair was used to stream MPEG video traffic between the different nodes in the test bed across both wired and wireless (IEEE 802.11b) Ethernet links [12]. In the generation of all the following results, no packet sampling was performed and the filter was configured to let UDP packets through only. C. Measurement Results Inline measurements were used successfully to measure the one-way delay and jitter of a UDP video stream across the test network. The one-way delay of packets was calculated as the difference between the source and destination timestamps recorded in the one-way delay options of each packet. Jitter was calculated as the difference between the one-way delays of consecutive packets, i.e. the Inter-Packet Delay Variation (IPDV), as defined in a draft of the IETF
The instantaneous one-way delay for this video stream varied from 8 to 89 ms and the average delay was 20.97 ms. The IPDV varied from -68 to 58 ms and -0.00049 ms was its average value. Figures 11 and 12 depict a more interesting situation that arose during another video streaming session. There is a huge increase in the delay from less than 100 milliseconds up to more than 3 seconds, followed by an even more sudden decrease back to the flow’s normal values. This fluctuation was tracked down to configuration changes taking place at the time of the experiment within the infrastructure of the wireless network. In this instance, the one-way delay varied from 5 to 3138 ms and the average value was 148.16 ms. IPDV assumed values between -192 and 29 with an average of 0.000443 ms. As expected, experiments conducted solely over the two wired private networks revealed no surprising or remarkable results. The one-way delay always assumed values between one and two milliseconds and the jitter varied between -1 and 1. 22 octets of overhead were appended to each UDP/IP packet of original length 1326 octets, representing an over-
one-way delay / ms
all efficiency of 98.4%, a figure that could be substantially improved using packet sampling. For example, sampling 1 in 20 packets would give 99.9% efficiency, but the certainty of the results would be correspondingly poorer.
The design of a prototype implementation has been briefly described, showing how the technique can be provisionally realised and exploited to measure one-way delay of a real-time UDP video stream operating over both wired and wireless IPv6 topologies. Further work will concentrate on exploiting filtering and sampling mechanisms to address scalability and overhead issues while assessing the applicability of inline measurements for particular application domains and network operations such as mobility management. ACKNOWLEDGEMENTS
tim e / s
delay variation / ms
Figure 11: One-way delay of video stream from “Blue” to “wireless node”
tim e / s
Figure 12: IPDV of a video stream from “Blue” to the “wireless node”
Note that the measurements take into account clock drifts in each system from the appropriate NTP server. While running, NTP improves the clock’s accuracy, and as the NTP polling intervals increase (i.e. from the default values of 64 to 1024 s), the clock drifts tend towards a fairly static and stable value. IV. CONCLUSIONS AND FUTURE WORK This paper introduced and demonstrated an inline measurement technique for assessing the performance properties of application flows across the Internet. This measurement approach exploits the concept of IPv6 extension headers to instrument portions of the network traffic, thus giving high probability that the measurements reflect the service experienced by real user data. Indeed, the technique could conceivably be integrated into the IPv6 stack as standard. The characteristics of inline measurements and the relative benefits and drawbacks over traditional measurement techniques have been discussed, and the scenarios and assumptions under which inline measurements should operate have also been highlighted.
We are grateful to Agilent Technologies for the support of Dimitris Pezaros’ work through an industrial fellowship. Thanks also to Stefan Schmid, Steven Simpson, Joe Finney, Andrew Scott and Barry Rowlingson at Lancaster University. REFERENCES [1] H. G. Hegering, S. Abeck, B. Neumair, “Integrated Management of Networked Systems”, Kaufmann, 1998. [2] V. Paxson, “Towards a Framework for Defining Internet Performance Metrics”, Proceedings of INET’96, Montreal, Canada, June 1996. [3] K. Claffy, H. Braun, G. Polyzos, “A Parameterizable Methodology For Internet Traffic Flow Profiling”, IEEE JSAC, Special Issue on the Global Internet, 1995. [4] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang and W. Weiss, “An Architecture for Differentiated Services”, RFC 2475, December 1998. [5] R. Braden, D. Clark and S. Shenker “Integrated Services in the Internet Architecture: an Overview”, RFC 1633, June 1994. [6] C. Fraleigh, et. al., “Design and Deployment of a Passive Monitoring Infrastructure”, Passive and Active Measurement Workshop (PAM2001), Amsterdam, April 2001. [7] Georgatos, F. Gruber, et. al., “Providing Active Measurements as a Regular Service for ISP's”, Passive and Active Measurement Workshop (PAM2001), Amsterdam, April 2001. [8] S. Deering, R. Hinden, “Internet Protocol Version 6 (IPv6) Specification”, RFC 2460, December 1998. [9] W. Matthews, L. Cottrell, “The PingER project: Active Internet Performance Monitoring for the HENP Community”, IEEE Communications Magazine, May 2000 [10] D. Govoni, “Java Application Frameworks”, John Wiley & Sons, ISBN 0-471-32930-4, 1999. [11] M. Fry and A. Ghosh, “Application level active networking”, Computer Networks, 31 (7), pp.655-667, 1999. [12] VideoLAN, [Online], “STREAMING: Overview of the VideoLAN streaming solution”, Available: http://www.videolan.org/streaming, 18th April 2004 [date accessed]
