In-line Service Measurements: Exploiting IPv6

In-line Service Measurements: Exploiting IPv6 Extension Headers D.P. Pezaros‡, F.J. Garcia†, R.D. Gardner†, D. Hutchison‡, J.S. Sventek§ ‡

†

Communications and Distributed Systems Computing Department Lancaster University Lancaster, LA1 4YR, UK. {dp, dh}@comp.lancs.ac.uk

Communications Solutions Department Agilent Laboratories Scotland Agilent Technologies Edinburgh, EH30 9TG, UK. {frankie_garcia, robert_gardner}@agilent.com

§

Networked Systems Measurement and Control Group Department of Computing Science University of Glasgow Glasgow, G12 8QQ, UK. [email protected]

Paper ID Number: E00-628544224 Total number of pages: 13 focused on providing fault management support for the network, and usually adopt a link or device-centric view in order to trace the sources of and reasons behind service degradation. More recent measurement infrastructures specialise either in measuring the path properties of special-purpose traffic (active measurements) or in correlating and analysing one-point, link traffic (passive measurements and standalone monitoring tools). The Internet Protocol (IP) is evolving as the universal transport mechanism from the continuing convergence of telephony with data communications, where there is increasing aggregation of multi-service traffic onto IP networks carrying various equivalence classes of network flows. The different Quality of Service (QoS) requirements and different sensitivities to potential service degradation of these equivalence classes of traffic make timely and accurate measurement of actual network flow QoS essential. Measurements revealing the real service experienced by user traffic can prove valuable for long and short term network design decisions, dynamic traffic engineering, as well as for Service Level Agreement (SLA) negotiation and policing, and advanced network and service management. In this paper, a new IPv6 measurement technique, named ‘in-line measurements’, is introduced that is able to assess the performance properties experienced by real user traffic transparently at the IP layer. The term ‘inline’ implies that measurement triggers, which invoke measurement activity, and the measurement data are piggybacked onto real user packets. With low overhead and minimal impact on the network traffic, the technique provides a high level of probability that the real user

Abstract -- The ability to measure, monitor and control the service quality experienced by network traffic is becoming increasingly important as multiple traffic types are aggregated onto IP networks. This paper introduces a novel measurement technique for assessing performance metrics (e.g. one-way packet loss, delay, delay variation, and ‘goodput’) of IPv6 network flows. By exploiting native IPv6 extension headers, measurement triggers and measurement data are carried in the same packets as the payload data itself, providing a high level of probability that the behaviour of the real user traffic flows is being observed. A prototype implementation of this technique has been constructed and used to measure numerous properties of different application flows, over both wireline and wireless IPv6 environments. End-to-end, oneway delay and delay variation of real-time video streams have been measured, as well as the goodput of services operating on top of reliable transport. This measurement technique can be the basis for low-overhead, scalable, transparent and reliable measurement of individual and aggregate network flows that can be dynamically deployed where and when required in a multi-service IP environment. Keywords: active/passive measurements, performance metrics, Internet Protocol version 6 (IPv6), extension headers, one-way delay/jitter/loss.

A. INTRODUCTION As the Internet grows larger and becomes increasingly heterogeneous, measuring the real-time properties and the service quality experienced by user traffic becomes immensely challenging, and moreover, of critical importance. Traditional measurement techniques have

1

experience is being measured and it is equally applicable to measuring aggregate flows as it is to particular applications or protocols. Additionally, it is well-suited to dynamic deployment and presents a scalable implementation. By adopting a service-oriented approach, in-line measurements can form the basis of end-to-end measurement instrumentation that is able to reveal the effects of the network behaviour on traffic flows. The technique is facilitated by the steady introduction of IPv6 in research and academic networks, and the established need for its widespread deployment, owing to the consumption of the IP address space primarily driven by the growth in mobile and wireless markets requiring IP connectivity. In-line measurements also address the challenge of avoiding misleading results, either due to under-estimating or over-estimating network performance. Furthermore, by avoiding the need for correlation of measurement samples from numerous locations, the transfer of measurement data is kept to a minimum and in-line measurements can therefore reduce the consumption of network and computational resources. This paper is organised as follows: Section B describes the pros and cons of existing measurement techniques and infrastructures, discussing their strengths and weaknesses. In Section C, in-line measurements are introduced and the motivation and design decisions are presented. Section D and E describe the prototype design and implementation. Section F discusses the results taken over different experimental, IPv6 configurations. Finally, Section G contains concluding remarks and indicates some of the future directions of this work.

I. Passive Measurements Passive measurement techniques, as the name would suggest, observe, characterise and analyse real traffic on a link without disruption to the service [5, 6, 7]. They are particularly useful in gathering measurements of real user traffic at a single observation point in the network but may also be employed for two-point measurements by employing additional correlation techniques. Passive measurement techniques typically run state machines that provide initial traffic filtering and then search for particular events using pattern-matching techniques [8, 9] whereupon various counters are updated. The collected data is then made available to interested third parties through either pull (SNMP, CMIP based products) or push models. Some passive monitoring techniques that focus on addressing QoS performance related issues, such as route caching, resource reservation and usage-based accounting can be found in [10, 11]. However, as network speeds continue to increase, the amount of measurement data, which often needs to be transferred across the monitored links, is substantial and the consequential difficulty in targeting specific services, identifying packets belonging to the same flows and inferring knowledge about them makes it difficult to prove that contractual agreements are being met [12]. Besides, for some metrics it is exceedingly difficult to see how one can measure them passively (e.g. route taken by packets) without scalability and complexity arising as significant restraining factors. One of the difficulties with two-point passive measurements is the need to correlate samples collected at two distinct observation points to yield one-way flow measures. Guaranteeing that both observation points trigger on the same packet and subsequent data correlation are challenging tasks and shipping the data can consume significant amounts of network bandwidth. In addition, lengthy standardisation processes governing the implementation of the various counters used by passive monitoring products deter open deployment and experimentation.

B. MEASUREMENT TECHNIQUES Measurements are collected in operational networks for a number of reasons. From an operator point of view this includes installation/in-service troubleshooting & diagnostics, protocol analysis and conformance testing, real-time performance monitoring, traffic engineering, service level agreement validation and long term network planning [1], as well as to provide an empirical validation of models assessing Internet protocol performance [2, 3, 4]. From a service provider point of view, measurements can provide a competitive advantage by monitoring service quality and proving that contractual agreements are being fulfilled, reducing the reliance on over-provisioning, determining traffic trends for longer-term capacity planning and providing billing, accounting, and fraud detection functions. Existing measurement infrastructures fall into two main categories, namely passive and active.

II. Active Techniques Active techniques rely on injecting additional traffic with known characteristics into the network to test particular attributes of a service [13]. Active measurements are two-point measurements and techniques adopting this approach focus on measuring the properties of end-to-end network paths between instrumented systems, by injecting packets at one end and retrieving them at the other. Active measurement architectures have their origins in packet-probing studies that have produced some of the tools used today to measure network performance [14].

2

In-line measurements are intrinsically multi-point measurements whereby packets are tagged with information at one point in the network and this information is retrieved and/or observed elsewhere. Since it is the user traffic itself which carries the measurement and triggering mechanism, it is guaranteed that the same packet has been observed at both ends. Similarly, because any added data will be piggybacked onto real user traffic we can assume with a high degree of probability that it will receive the same treatment and follow the same path as the real user traffic. Unlike active measurements consisting of injected packets, twopoint in-line measurement results will more accurately reflect the real characteristics of traffic flows with only a small additional systematic processing delay and marginally larger overall packet length compared to unaffected packets. This makes the reasonable assumption that, for appropriately selected packets, the marginal increase in the packet header length does not change how the packet is treated on its journey through the network. Moreover, in contrast to two-point passive measurements, correlation of data from path endpoints is not necessary, reducing the complexity of the measurement system, potentially reducing the amount of measurement data that must be shipped across the network and speeding up the availability of the measurement results. In fact, the total amount of additional traffic transported across the network for measurement purposes can be kept to a minimum. Also, in-line measurements lend themselves to implementation according to some of the characteristics of Application Level Active Network (ALAN) programming [22], being deployable as and when required in the network on a dynamically loadable/removable basis, as will be later described.

Being mostly based on variations of the ping and traceroute programs, they try to measure the quality properties of end-to-end (and inter-domain) paths, by sending Internet Control Message Protocol (ICMP) packets [15, 16], or variations [17], and implementing metrics defined within Internet Engineering Task Force (IETF)’s IP Performance Metrics (IPPM) Working Group [18]. Injected variable and fixed-size UDP packets are also being used to measure one-way properties such as delay and loss [19, 1]. A particular problem with active measurements is that performance properties measured by active techniques are that experienced by the synthetic traffic injected in the network, and do not necessarily reflect the behaviour experienced by operational network traffic. For example ICMP messages may be processed and routed differently to operational network traffic. Some architectures have been designed to use dedicated protocols (e.g. IP Measurement Protocol – IPMP) for injected traffic [20], whose purpose is to overcome the protocol-specific treatment experienced by certain protocols (e.g. ICMP). However, these protocols can still receive different treatment from the network since they remain uniquely identifiable and can be filtered in routers along the path. Furthermore, using UDP packets for synthetic traffic can also lead to inconsistencies between the observed and the actual performance, mainly due to the bandwidth starvation caused to TCP flows by the “unresponsive” UDP bursts [21], or even due to specific policies possibly applied to UDP traffic. Finally, the additional traffic associated with active measurements obviously impacts the network and may itself be a factor in measuring a poorer performance than the network would otherwise deliver. Also, periodic sampling and packet injection used by some architectures may fail to observe periodic network events [15].

I. IPv6 Enhancements Native features of Internet Protocol version 6 (IPv6) can be conveniently used to implement in-line measurements, potentially being fully integrated into the protocol stack operation using IPv6 extension headers [23]. IPv6 has a common, 40-octet, fixed-sized protocol header (see Figure 1) that mainly deals with addressing and has QoS capability, while the rest of the functionality is implemented via a set of optional extensions headers containing data structures called ‘options’. The IPv6 protocol removes the complexity of options processing in IPv4, which was required at every node en-route to a destination. Instead, the specialised extension headers, which are inserted between the fixed size IPv6 header and the upper layer header in a packet, are only processed where necessary. A small number of

C. IN-LINE MEASUREMENT TECHNIQUE The acquisition of low-overhead, scalable, accurate, service-oriented measurements of network performance has led to the investigation of mechanisms that can circumvent some of the inherent difficulties of active and passive techniques. Indeed, the in-line measurement technique may be viewed as a hybrid of the beneficial characteristics of active and passive measurement approaches, whereby the measurement data and triggering mechanisms are piggybacked onto real user packets. Not unsurprisingly, however, there are tradeoffs too and consequently some drawbacks to using this technique. In-line techniques should thus be considered as another potentially favourable approach to measurement rather than as an omnipotent answer.

3

these extension headers and corresponding options have already been defined in the IPv6 standard [23] such as Hop-By-Hop Options, Destination Options, Routing, Fragment, Authentication and Encapsulating Security Payload. However, others are currently being defined for dedicated purposes – e.g. within recent Mobile IPv6 IETF drafts. 0

4

12

16

Next Header

Options

Figure 2: Destination options extension header

The Next Header field contains a value that identifies the type of the subsequent header, which could either be another extension header or higher-layer protocol such as UDP or TCP. The Hdr Ext Len field gives the total length in octets of the destination options header excluding the 8 bits representing the Next Header Field. The options field is a variable length field in which options are encoded as type-length-value (TLV) tuples, which represent a suitable format for the transportation of opaque objects – see Figure 3. The type identifies the type of option. The length indicates the length of the option data field in octets, while the value represents the option specific data.

32

Version Traffic Class

Flow Label

Payload Length

Next Header

Hop Limit

IPv6 Source Address

fixed sized header

Destination Address

Next Header

Ext. Hdr(s)

Next Header

Hdr Ext Len

Hdr Ext Len

Hdr Ext Len

Option Type

Option Length

Data

1 octet

1 octet

variable number of octets

Figure 3: Options in the form of TLV tuples

Data

Options are processed in sequence and the option type also has two bits that specify the action to be taken by an IPv6 node when it does not recognise a particular option in the sequence – e.g. skip the option, discard the packet or discard the packet and send an ICMP error message to the originator of the packet. Another bit in the type field is used to indicate whether the option data value may change en-route to its destination. Padding options also exist to help with alignment issues when constructing a destination options header to ensure that it represents a multiple of 8 octets in length.

Data

Figure 1: IPv6 Header showing extension headers

Apart from the special Hop-by-Hop options header, each of the different IPv6 extension headers is only examined at the destination address in the main IPv6 header. Each header points to the start of the next forming a kind of daisy chain using the Next Header field. Each header in the chain is processed in sequence and the rules and semantics of each header determines whether the receiving node will proceed to the next header or not. While any of these extension headers could be used in the development of secure in-line measurement methods, this paper describes the use of Destination Options extension headers only.

III. IPv6 Extension Headers & Inline Measurements In this paper, it is proposed that in-line measurements be implemented by exploiting the IPv6 extensions headers, in particular the Destination Options extension header. Extension headers can be used within a measurement infrastructure to carry measurement information in the form of TLV-encoded options, such as timestamps, counters and trace information as well as other associated measurement system traffic. Although not an extension header, the 20-bit experimental Flow Label field within the IPv6 header could also be used to carry measurement data. Previous suggested uses for this label have included the marking of sequences of packets for which special treatment is required by intermediate nodes on the delivery path - for example, to support non-default QoS or real-time services. This may become redundant now that the

II. Destination Options Header The Destination Options extension header is a simple header that can be used in two ways. It can either hold standalone options to be processed by a packet’s destination or it can be used in conjunction with another extension header. The extension header actually carries the optional data to be processed. The defined header is illustrated in Figure 2.

4

traffic class field has been standardised and hence this label may be put to better use. Using a combination of destination options, routing header and if necessary the flow label, it is possible to selectively add data to real user traffic which will be detected and processed, where instrumented, by the IPv6 protocol layer. The level of processing will be determined by the options carried and in most cases this may simply involve incrementing some counters, adding a timestamp annotation, extracting some packet data and dumping it into a cache with timestamp annotations and various counts, or triggering a full packet capture. Destination options on their own can easily be used to perform end-to-end in-line service measurements in an IPv6 network cloud. With the addition of a routing header, it is possible to target specific nodes en-route to enable the implementation of more detailed service measurements as the user traffic crosses crucial points of a network cloud. It would also be possible to use some of the flow-label bits to easily identify and trigger measurement and monitoring behaviour as the user traffic with the in-line data visits nodes en-route to its destination. In terms of extensions to the existing IPv6 protocol stack, it is necessary to develop appropriate functionality to process the measurement options and perform the appropriate action at the visited node, while maintaining the semantic and syntactical header extension processing rules defined within the IPv6 standard [23]. The insertion of the extension headers into real user traffic for the purpose of measurement and monitoring can be dynamically controlled depending on a particular management application requirement. Thus not all user packets need to carry this data. Selection can be based on application and sampling can also be effectively used. It is important to ensure that the addition of measurement data does not become intrusive and therefore adversely affect the traffic being measured. For example, if the addition of measurement data were to cause the packet to exceed the link maximum transmission unit (MTU) then packet fragmentation and reassembly would occur and any resulting measurements might be highly questionable, especially in relation to delay. Also, the addition of measurement data to time- or length-sensitive packets, such as Voice-over-IP packets, may have detrimental consequences and, in any case, be difficult because of header compression techniques. It can be concluded therefore that in-line measurements are not necessarily appropriate for all types of traffic. In the next section, a working prototype in-line measurement framework is described and specific Destination options are defined that facilitate various traffic measurements.

D. PROTOTYPE DESIGN With a strong bias to placing intelligence in networking nodes where and when required, ALAN concepts have been used in the development of a core set of application frameworks. These frameworks are made up of a set of classes and components that together provide services for a particular domain while adhering to an object, organisational and collaboration model [24]. The main aim is to develop intelligent Operational Support Systems. Without going into any specific detail, for the purpose of brevity and maintaining the scope of this paper, these frameworks handle the initialisation, control, co-ordination, and scheduling of tests in a distributed network. They also provide capabilities for collection, wire-format representation and streaming of measurement samples to registered consumers. Underpinning this design philosophy is the notion of telemetry modules which are the basic components employed in the prototype to instrument nodes to facilitate in-line measurement techniques through the addition, modification and removal of data in the extension headers. These modules are remotely configured, managed and controlled.

I. Telemetry Modules At a system level the required functionality for performing delay and other measurements can be implemented, for example, by using dynamically loadable modules to provide additional processing logic for the manipulation of packet extensions in headers and other supporting functions such as the storage, retrieval, correlation and forwarding of measurement-related data. By modularising the set of monitoring and measurement tasks it is possible to dynamically load only those modules that are needed at a particular time and then unload them once they are no longer required. The loadable modules may be remotely delivered to the nodes, loaded and configured and, whilst in use, effectively become an integral, embedded part of the nodes operating software. Minimisation of the actively used processing logic in this way can reduce memory usage, speed up processing time, limit circuit-board space requirements, simplify designs and reduce overall subsystem complexity. This modular approach also lends itself well to a remote, distributed implementation as the modules can be freely inserted and removed around the network as required, providing a potential for dynamicallyconfigurable, localised processing sites and correlation entities. Another significant advantage of the embedded module approach is that it is not necessary physically to connect into the electrical or optical cables comprising the links between routers in order to monitor passing data. The embedded module approach instead makes

5

modify extension headers as an integral part of the kernel’s implementation of the network protocol stack rather than having to explicitly construct entire packets in an external user-space process. Processing IPv6 extension headers in user space, rather than as part of the operating system’s protocol stack implementation, may not be as elegant a solution and may result in poorer performance. However it does not require knowledge of operating system kernel programming techniques and therefore may be simpler to implement. In addition it avoids possible problems with operating system security and integrity which may conflict with security policies of an organisation operating the routers or other nodes in question.

use of spare programmable logic or processing capability within the routers or other network devices, providing a more integrated, inherently powerful solution. Upgrades involve delivering new modules to nodes (for example over the network itself), which can either be directly loaded on delivery or be temporarily stored on some form of local media (e.g. hard disc storage) for later use. Network Element

Physical medium e.g. copper, optical fibre

Line I/F

Line I/F

Line I/F

Line I/F

Line I/F

Line I/F

Line I/F

Line I/F

E. LINUX-BASED PROTOTYPE IMPLEMENTATION A prototype implementation of this design philosophy has been realised on Linux systems running kernel version 2.4.x.; telemetry modules are implemented as dynamically loadable kernel modules that can be linked with a running Linux kernel at runtime. Each node in this prototype that is intended for the deployment of telemetry modules runs a test scheduler. The primary functions of this test scheduler include:

Controller (Processor, memory, firmware) Kernel space (if present) User program space (if present)

ƒ Checking the validity of test descriptions ƒ Managing local resources and implementing local policy ƒ Running device-specific scripts for loading and unloading modules ƒ Creating virtual devices for configuring and obtaining results from installed telemetry modules ƒ Adding and removing consumers to running tests

Figure 4: Abstract model of a network element

Figure 4 shows an illustrative architecture of a single network element with a number of line interfaces and a controller comprising a processor, memory and program software or firmware. The figure illustrates three different example integration points where, depending on the design of the network element, dynamically loadable modules can be accommodated:

Communication with test schedulers is achieved via Java RMI and they in turn use JNI for managing the checking, loading, configuration and unloading of telemetry modules onto the running Linux kernel. Only the telemetry modules are implemented in C. All other application framework components are implemented in Java. To run a test on a particular node, a test description object is remotely downloaded to the particular test scheduler. In addition to telemetry module and identity information, this description also contains configuration details that include the following two objects:

ƒ The module may be loaded onto a line interface as dynamically reconfigurable hardware (e.g. using field programmable gate arrays – FPGAs), as software or as a hybrid combination of both options. ƒ The modules may exist as loadable software in the software operating system ‘kernel’ or equivalent for the controller; ƒ The modules may exist as loadable applications in ‘user’ space provided by the controller’s operating system. Integration as a loadable software module assumes a suitably compliant operating system, e.g. Loadable Kernel Modules (LKMs) under the Linux operating system [25]. These typically provide better processing performance compared to applications executing in user space, and can easily be configured to add, remove and

ƒ Filter: The filter object is used as a packet classifier to describe the traffic of interest to the telemetry module at an ingress node. Once configured with the appropriate filter, the loaded module will perform its telemetry functions when it observes

6

the time elapsed since 0000 hrs on 1st January 1970 Universal Co-ordinated Time (UTC). ƒ Destination timestamp – Two 32-bit unsigned integers indicating time of receipt of the packet at the interface of the node where the extension header is detected.

traffic characterised by the filter. Presently, the parameters used to describe a filter include source and destination IPv6 addresses, defined IP protocol numbers, source and destination port numbers, traffic class and flow labels. ƒ Sampling: The sampling object, like the filter object, is used to configure the installed telemetry module at an ingress node. Essentially, once a filter has been configured, the sampling mechanism determines whether the module should act on all the observed packets, on 1-in-N packets or at a specific temporal sampling rate. This technique is particularly useful when analysing aggregate or continuous media flows.

The Option Type identifier in the header is set to a value allocated to identify “one-way end-to-end delay measurement”. In this prototype, this octet has been set to a value 001000012 (33 in decimal), also ensuring that the option is skipped if it is an unrecognised option and indicating that the data may change en-route. This option can be used to measure one-way delay within a section of a network, such as between ingress and egress points of a packet flow across the boundaries of a section under the control of a single operator. However, this would involve modification of the standard processing procedures for Destination options or a new extension header to be defined. Note also that the option is equally applicable to “end-to-end” measurements of one-way delay, such as from a server (e.g. serving website) to a client (e.g. a wireless connected personal digital assistant (PDA) running a web browser application).

A number of telemetry modules have been implemented to include one-way delay, response time and packet tracing functionality. These modules are presently being evaluated and experimented with, while others, such as a loss module are in the design phase. The results that follow have been gathered using the one-way delay telemetry module.

I. One-way Delay Option Format For the measurement of delay, a Destination Options extension header is conveniently used to carry the required information. Figure 5 shows the format of the TLV-encoded option for measuring one-way delay.

Pointer

Overflow

Option type

Option data len

Flags

(Reserved)

II. One-way Delay Telemetry Module To perform a one-way delay measurement typically requires at least two different telemetry modules to be operating simultaneously. Figure 6 shows the typical operation of the one-way delay modules at a source and destination node of an Internet path.

Source timestamp: seconds Source timestamp: microseconds Destination timestamp: seconds Destination timestamp: microseconds

Figure 5: One-way delay TLV-encoded option

ƒ Pointer – 8-bit unsigned integer. Used to indicate the location of the next unused slot in the option data, i.e. for storage of a timestamp. ƒ Overflow – 8-bit unsigned integer. Used to indicate if an attempt is made to store more timestamps than there are slots to accommodate them. ƒ Flags – Octet comprising eight binary flags, for example for indicating the nature of data stored elsewhere in the option data fields. ƒ Reserved – A zero-valued octet included for alignment purposes ƒ Source timestamp – Two 32-bit unsigned integers indicating time of forwarding of the packet from the interface of the node where the extension header or option was inserted. The two integers represent the seconds and microseconds portions respectively of

Figure 6: One-way delay source and destination modules in operation

The source module selects a packet based on the filter and sampling criteria which are set using input/output control (ioctl system) calls to the loaded module. If the filter selection criteria are satisfied and the packet passes the sampling algorithm the one-way

7

connected in a wired Ethernet topology via a public and two private IPv6 networks. Red and Blue are two PCs configured to act as a source-destination pair and to send/receive traffic through both their public and private network interfaces. Shaper is a powerful PC that currently performs IP forwarding on all its interfaces and acts as a router for the two private networks. It also takes part as an endsystem in experiments run over the wireless network. Finally, a laptop equipped with an IEEE 802.11b network interface card stands as the mobile wireless node of the experimental set-up, and it is being used as an end-system for the core of the experiments that run over the wireless infrastructure, forcing traffic to span two operational IPv6 networks.

delay option is generated, time-stamped and either appended to an existing destination options header or one is created for it. The destination module looks for IPv6 packets whose protocol (next header) field is set to the value 60 indicating a Destination Options header [23]. Once found, it processes the option with value 33 by adding the second time-stamp. The IPv6 header, extension header, and transport header (e.g. TCP, UDP, ICMP, etc.) are then added to a FIFO message queue, which can be accessed via an ioctl system call to the module. Newly arriving messages cause those at the head of the queue to be discarded if the queue buffer is already full. If required, the extension header may also be stripped from the packet, or the particular option removed, returning the packet to its original form. These one-way delay telemetry modules can easily be exploited to measure bi-directional properties of Internet paths, by combining two uni-directional deployments as in Figure 7. Both source and destination modules can be run on a pair of nodes simultaneously.

Figure 7: Bi-directional delay source and destination modules

F.

EXPERIMENTAL RESULTS

An experimental measurement testbed has been set up at Lancaster University to demonstrate the operation of the in-line measurements prototype. Representative types of traffic have been chosen to run over an IPv6 network. Various indicative properties have been measured and computed for traffic operating over connection-oriented and connectionless transports, using the prototype oneway delay option. End-to-end one-way delay and InterPacket Delay Variation (IPDV) have been measured for video streaming on top of UDP and application goodput has been measured for Telnet, SSH and FTP sessions. The IPv6 Measurement Testbed (Figure 8) comprises a number of real PCs and virtual machines, each installed with SuSE Linux 8.0 PRO, running IPv6enabled 2.4.19 kernels. The three physical machines are

Figure 8: Lancaster University Measurement Testbed and IPv6 connectivity

All the machines run the NTP daemon and obtain their system time via the network. Shaper periodically polls three different stratum 1 NTP servers to obtain global time. The rest of the systems have Shaper configured as their stratum 2 server and obtain their time locally. No external time references/servers are used since synchronisation between the machines is the crucial element, not absolute time. The traffic for the experiments was generated using existing, open-source application software. In the generation of all results that follow, no packet sampling was performed. Filtering was applied only on the

8

transport protocol (i.e. all TCP or all UDP traffic) and the rest of the attributes (source/destination addresses, ports) were ignored for the purposes of filtering.

60

Inter-Packet Delay Variation (Jitter) Vs. T ime (Video/UDP Stream)

40

I. UDP Measurements

Instant One-way Delay(msecs)

80

80

100

60

80

100

120

The instantaneous one-way delay for this video stream varied from 8 to 89 milliseconds and the average delay was 20.97 milliseconds. IPDV varied from -68 to 58 and its average value was -0.00049. Figure 11 and Figure 12 show a more interesting situation that arose during a video streaming session (part of the same set of experiments) from Blue to the wireless node, again over the wireless network. There is a huge increase in the delay indications from less than 100 milliseconds up to more than 3 seconds, followed by an even more sudden decrease back to the flow’s normal values. This fluctuation possibly occurred due to configuration changes taking place at the time of the experiment, at the infrastructure of the wireless network. The one-way delay varied from 5 to 3138 milliseconds and the average value was 148.16 milliseconds. IPDV took values from -192 to 29 with an average of 0.000443. As expected, experiments over the two private networks revealed no particularly interesting results. The one-way delay always assumed values between one and two milliseconds and hence the jitter varied between -1 and 1. The measurements take into account clock drifts in each system from the appropriate NTP server. While running, NTP improves the clock’s accuracy, and as the polling intervals increase (e.g. 1024 seconds), the clock drifts tend towards a fairly static and stable value.

60

60

40

Figure 10: Inter-Packet Delay Variation (IPDV) of a video stream from the wireless node to “Blue”.

40

40

20

Time(sec)

20

20

0 0

One-way Delay over T ime (Video/UDP Stream)

0

-20 -60

-40

Delay Variation

20

Streamed video data was monitored over both the wired and wireless topologies, using the in-line measurement techniques described previously. The main focus was on the properties of the wireless portion of the network, since the two private Ethernet networks constitute a relatively delay-free topology and therefore useful for time synchronisation purposes. The VideoLAN server/client pair [26] was used to stream MPEG video between the different nodes in the testbed. One-way delay was measured as the difference between the timestamps recorded at the source and the destination of the packet, respectively, carried within every TLV-encoded option. Jitter was measured as the difference between the delay indications of two successive packets, i.e. the Inter-Packet Delay Variation (IPDV), as defined in a draft of the IETF IPPM working group. Figure 9 and Figure 10 show the one-way delay and jitter, respectively, measured for a video stream from the wireless node to Blue.

120

Time(sec)

Figure 9: One-way delay of a video stream from the wireless node to “Blue”.

9

dictionaries also contained statistics pertinent to each TCP session. As aforementioned, the monitored sessions consisted of SSH, Telnet and FTP. All the running protocol stacks within the testbed adhered to TCP with the NewReno and SACK extensions – see RFCs 793, 1122 and 2001. Figure 13 shows the output of the TCP conversation dictionary for the three applications, running across the two private, Ethernet networks. Application sessions are identified by their well-known service ports; the client ports, conversation set-up times and packets belonging to each flow can then be deduced. Conversation set-up time is measured as the time between a SYN packet sent from the client and a SYN+ACK packet received from the server. The “Completeness” of a flow signifies that at least one FIN packet, sent in either direction, was observed. Figure 11: One-way delay of a video stream from “Blue” to the wireless node

Server:: 2001:630:80:7091:0:0:0:1 Client:: 2001:630:80:7092:0:0:0:1

(RED) (BLUE)

Service Port:: 21 Port:: 3260 State-> Packets #: 73 Complete: true Conversation Set up time: 674 (µsec) Service Port:: 23 Port:: 3257 State-> Packets #: 815 Complete: true Conversation Set up time: 783 (µsec) Service Port:: 22 Port:: 3258 State-> Packets #: 738 Complete: true Conversation Set up time: 690 (µsec)

Figure 13: Conversation Dictionary output for the three measured TCP applications over the wired topology

One-way delay is a less interesting property for TCP traffic, since the protocol itself adjusts its transmission rate based on round trip indications, and because applications running on top of the reliable transport usually have no real-time requirements. A more critical property of the protocol is the goodput experienced by the different TCP flows. Goodput is defined as the number of packets received per unit time in each direction, as opposed to throughput, which measures the number of packets sent, regardless of their eventual fate [2]. The number of bytes of pure application/payload data (excluding bytes up to and including the TCP header) that have been received in both directions over time was measured. This computation was based on the annotations collected from each instrumented packet, which included the total packet length disregarding the length of the Destination Options header.

Figure 12: Inter-Packet Delay Variation (IPDV) of a video stream from “Blue” to the wireless node

II. TCP Measurements The TCP results were collected by running bi-directional delay source and destination modules as previously illustrated in Figure 7. The data in the respective FIFO queues of each destination module was collected in capture files which were post-processed and analysed using a combination of purposely built tools and other freely available tools. From these capture files, conversation dictionaries were built based on source/destination addresses and port numbers. These

10

Figure 14, Figure 15 and Figure 16 show the application goodput for the three services, in bytes per millisecond, as measured over the wired topology, across the two private networks, from Red to Blue. Dots indicate values for packets from the client to the server, and ‘x-points’ represent values for packets from the server to the client. Goodput has also been measured over the wireless IPv6 network by having instrumented data running from Red to the wireless node. Figure 17 through to Figure 20 show the relevant conversation dictionary outputs and the goodput results (in bytes per millisecond), respectively.

400 300 0

100

200

Goodput(bytes/msecs)

500

600

Goodput Vs. T ime (SSH Stream)

0

20

40

60

80

100

120

Time(sec)

(RED) Server:: 2001:630:80:7090:0:0:0:10 Client:: 2001:630:80:1010:202:2dff:fe66:a145 (Wireless Node)

Figure 14: Goodput measured for a SSH flow, from “Red” to “Blue” over the wired topology

Service Port:: 21 Port:: 32791 State-> Packets #: 110 Complete: true Conversation Set up time: 3205 (µsec)

70

Goodput Vs. T ime (T elnet Stream)

50 40

Service Port:: 22 Port:: 32790 State-> Packets #: 638 Complete: true Conversation Set up time: 3166 (µsec)

30

Goodput(bytes/msecs)

60

Service Port:: 23 Port:: 32789 State-> Packets #: 751 Complete: true Conversation Set up time: 3102 (µsec)

0

50

100

Figure 17: Conversation Dictionary output for the three measured TCP applications over the wireless networks.

150

Time(sec)

Figure 15: Goodput measured for a Telnet flow, from “Red” to “Blue” over the wired topology

Goodput Vs. T ime (SSH Stream)

100 50

50 40

0

30

Goodput(bytes/msecs)

60

Goodput(bytes/msecs)

70

150

Goodput Vs. T ime (FT P Stream)

0 0

10

20

30

40

50

10

20

30

40

50

60

70

60

Time(sec)

Time(sec)

Figure 16: Goodput measured for a FTP flow, from “Red” to “Blue” over the wired topology

Figure 18: Goodput measured for a SSH flow, from “Red” to wireless node over the wireless IPv6 networks

11

For example, for the FTP experiments, adding measurement options to the data packets could cause MTU violation, since the protocol maximises its performance (throughput) by sending the maximum possible packets sizes for data. Therefore, the telemetry modules only instrumented the FTP control traffic to compute properties such as conversation set-up time and number of packets exchanged. In-line measurement results were compared against the outputs from a network analyser tool [27] running on Shaper that recorded all data passing through the network. This comparison showed that the in-line technique was indeed instrumenting data from the control channels of the FTP connection, and left data traffic bursts from the (binary) data channels to be exchanged without interruption. The FTP results illustrated in this paper therefore exhibit the properties of the control channel (Port 21) of the FTP master server.

50 40 30 10

20

Goodput(bytes/msecs)

60

70

Goodput Vs. T ime (T elnet Stream)

0

20

40

60

80

100

120

Time(sec)

G. CONCLUSIONS AND FUTURE WORK

Figure 19: Goodput measured for a Telnet flow, from “Red” to wireless node over the wireless IPv6 networks

This paper introduced and demonstrated a novel in-line measurements technique for assessing the performance properties of application flows across the Internet. This measurement approach exploits the concept of IPv6 extension headers to instrument portions of the network traffic, thus almost guaranteeing that the measurements reflect the service experienced by real user data. The characteristics of in-line measurements and the relative benefits over traditional measurement techniques have been discussed, and the scenarios and assumptions under which they should operate have also been highlighted. The design of a prototype implementation has been described showing how the technique can be provisionally realised and exploited to measure unidirectional and bi-directional traffic properties. Initial measurement results have been presented, documenting numerous properties of real-time data and applications operating on top of reliable transport (TCP), over both wired and wireless IPv6 topologies. Further work will concentrate on how to exploit filtering and sampling mechanisms to address scalability and overhead issues while assessing the applicability of in-line measurements for particular application domains and network operations. Further measurement header TLV-encoded options will be defined and implemented to expand the set of the measured performance properties. Destination Options headers have been used quite liberally at times during our investigations and processing has even been employed at intermediate nodes along the delivery path, which is strictly not permitted. (Note that the experimental results have not employed this method). To overcome this limitation, new extension headers could be defined for

40 30 10

20

Goodput(bytes/msecs)

50

60

Goodput Vs. T ime (FT P Stream)

0

20

40

60

80

100

120

Time(sec)

Figure 20: Goodput measured for a FTP flow, from “Red” to wireless node over the wireless IPv6 networks

As previously stated, it is not envisaged that in-line measurements should instrument and be carried along with every single packet across the network. Rather, they should serve as an additional measurement tool deployed where and when required to perform service-oriented measurements along Internet paths. For this reason, target application domains need to be carefully addressed and evaluated.

12

[8]

measurement purposes but it would then be necessary for nodes which are not involved in the measurement process to detect and process the relevant extension headers, even though they would ignore them in normal network operation. Such extension headers would also be easily identifiable and equipment could readily be designed to treat such packets in a favoured but atypical manner, potentially defeating the purpose of the measurements. By using the Destination Options extension header this risk is minimised, as there is nothing to distinguish the measurement purpose of the header. These issues are subject to further investigation.

[9] [10]

[11] [12]

ACKNOWLEDGEMENTS We are grateful to Agilent Technologies for the support of Dimitris Pezaros’ work through an industrial fellowship. Thanks are due to Stefan Schmid for his invaluable support with IPv6 network configuration, and to Steven Simpson for commenting and advising on the packetlevel programming. Joe Finney and Andrew Scott provided useful comments, influencing the network design decisions. The authors are indebted to Barry Rowlingson for providing recommendations and advice on the use of statistical packages.

[13] [14]

[15]

[16]

REFERENCES [1]

[2] [3]

[4] [5] [6]

[7]

[17]

Georgatos, F., Gruber, F., Karrenberg, D., Santcroos, M., Susanj, A., Uijterwaal, H., Wilhelm, R., Providing Active Measurements as a Regular Service for ISP's, Passive and Active Measurement Workshop (PAM2001), Amsterdam, NL, April 2001 Padhye, J., Firoiu, V., Towsley, D., Kurose, J., Modeling TCP Throughput: A Simple Model and its Empirical Validation, ACM SIGCOMM 1998 Mathis M., Semske J., Mahdavi J., Ott, T., The Macroscopic Behavior Of The Tcp Congestion Avoidance Algorithm. Computer Communication Review, 27(3), July 1997. Ott, T., Kemperman, J., Mathis, M., The stationary behavior of ideal TCP congestion avoidance, IEEE INFOCOM'99, New York, 1999 Apsidorf, J., OC3MON: Flexible, Affordable, High Performance Statistics Collection, INET’97, June, 1997 Claffy, K., C., Miller, G., Thompson, K., The Nature Of The Beast: Recent Traffic Measurements From An Internet Backbone, INET’98, Geneva, Switcherland, July 1998 Fraleigh, C., Diot, C., Lyles, B., Moon, S., Owezarski, P., Papagiannaki, D., Tobagi, F., Design and Deployment of a Passive Monitoring Infrastructure, Passive and Active Measurement Workshop (PAM2001), April 2001

[18] [19] [20] [21] [22] [23] [24] [25] [26] [27]

13

Hegering, H-G., Abeck, S., Neumair, B., Integrated Management of Networked Systems, Morgan Kaufmann, 1998 Cisco IOS NetFlow, http://www.cisco.com/warp/public/732/Tech/nmp/ind ex.shtml Claffy, K., Braun, H., and G. Polyzos, A Parameterizable Methodology For Internet Traffic Flow Profiling, IEEE Journal on Selected Areas in Communications, Special Issue on the Global Internet, 1995. Brownlee, N. Mills, C. and G. Ruth, Traffic Flow Measurement: Architecture, January 1997, RFC 2063 Jain, R., Routhier, S., Packet Trains – Measurements and a New Model for Computer Network Traffic, IEEE Journal on Selected Areas in Communications, Vol.4, No.6, September 1986, pp. 986-995 Paxson, V., Towards a Framework for Defining Internet Performance Metrics, Proceedings of INET’96, Montreal, Canada, June 1996 Downey, A., B., Using Pathchar to estimate Internet Link Characteristics, Proceedings of ACM SIGCOMM’99, Cambridge, MA, pp. 241-250, September 1999 Matthews, W., Cottrell, L., The PingER project: Active Internet Performance Monitoring for the HENP Community, IEEE Communications Magazine, May 2000 The Surveyor Project Homepage, http://www.advanced.org/surveyor/ Claffy, K., C., Polyzos, G., C., Braun, H., Measurement Considerations for Assessing Unidirectional Latencies, Internetworking, Research and Experience, 4, John Wiley & Sons, 1993, 121 132 IETF IPPM Working Group, http://www.ietf.org/html.charters/ippm-charter.html RIPE NCC Test Traffic Measurements, http://www.ripe.net/ripencc/memservices/ttm/index.html Active Measurement Project (AMP) Homepage, http://watt.nlanr.net//active/intro.html Floyd, S., Fall, K., Promoting the Use of End-to-End Congestion Control in the Internet, IEEE/ACM Transaction on Networking, May 1999 Fry, M. and Ghosh, A., “Application Level Active Networking”, Computer Networks, 31 (7), 1999, pp.655-667. Deering, S., Hinden, R., Internet Protocol Version 6 (IPv6) Specification, IETF, IPNG Working Group, RFC 2460, December 1998 Govoni, D., “Java Application Frameworks”, John Wiley & Sons, ISBN 0-471-32930-4 Pomerantz, O., Linux Kernel Module Programming Guide, April 1999 VideoLAN Open Source Streaming Solution Homepage, http://www.videolan.org The Ethereal Network Analyzer, http://www.ethereal.com/