Data Plane Timestamping

Time and Timestamping in Softwarized Environments Tal Mizrahi, Yoram Moses Technion – Israel Institute of Technology

April 2016

The First Synchronized Clocks

Railway switch

The TimedSDN Project

2

Networks: Communications vs. Trains

SDN Controller

Train Control Center

Network switch

Railway switch The TimedSDN Project

3

Timed

at a Glance Controller

A protocol allowing the controller to schedule network updates

Synchronized clocks

Switches The TimedSDN Project

4

Accurate Synchronization: It’s Already Here Precision Time Protocol (PTP) [IEEE 1588 2008] ~1μsec accuracy

Mobile backhaul

Industrial automation

But what about Software Defined Networks?

Power substations

Open Networking Foundation (ONF) SDN product directory

Financial applications

~70%

~70% of the SDNenabled silicons have native PTP support.

China Mobile: over 1,000,000 PTP-enabled base stations https://mailarchive.ietf.org/arch/attach/tictoc/pdfsY1ADO.pdf The TimedSDN Project

5

Timed

at a Glance Controller

Challenge: A protocol allowing the controller to schedule network updates

Synchronized clocks

Switches The TimedSDN Project

6

The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization

Project

Flow Swaps TIME4 [INFOCOM, ‘16]

Multi-phase updates [SOSR, ‘15]

Data Plane Timestamping [SWFAN, ‘16]

ONECLOCK to Rule Them All [NOMS, ‘16]

OpenFlow Scheduled Bundles

NETCONF Time Capability

[INFOCOM ’16, OpenFlow 1.5]

[NOMS ’16, RFC7758]

TIMEFLIP

ONECLOCK

[INFOCOM ‘15]

[NOMS ‘16]

REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project

7

The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization

Project

Flow Swaps TIME4 [INFOCOM, ‘16]

Multi-phase updates [SOSR, ‘15]

Data Plane Timestamping [SWFAN, ‘16]

ONECLOCK to Rule Them All [NOMS, ‘16]

OpenFlow Scheduled Bundles

NETCONF Time Capability

[INFOCOM ’16, OpenFlow 1.5]

[NOMS ’16, RFC7758]

TIMEFLIP

ONECLOCK

[INFOCOM ‘15]

[NOMS ‘16]

REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project

8

Time is Optimal for Flow Swaps

[INFOCOM ‘16]

A key benefit of SDN: Dynamic path allocation based on network load

old new

o1

...

controller o2

The TimedSDN Project

9

Time is Optimal for Flow Swaps

[INFOCOM ‘16]

A key benefit of SDN: Dynamic path allocation based on network load If O1 , O2 are not updated at the same time…

old new

Temporary congestion.

o1

...

controller o2

O2 is updated first. The TimedSDN Project

10

Time is Optimal for Flow Swaps

[INFOCOM ‘16]

A key benefit of SDN: Dynamic path allocation based on network load

TIME4

old new

If O1 , O2 are updated at the same time: No congestion!

o1

...

controller o2

The TimedSDN Project

11

The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization

Project

Flow Swaps TIME4 [INFOCOM, ‘16]

Multi-phase updates [SOSR, ‘15]

Data Plane Timestamping [SWFAN, ‘16]

ONECLOCK to Rule Them All [NOMS, ‘16]

OpenFlow Scheduled Bundles

NETCONF Time Capability

[INFOCOM ’16, OpenFlow 1.5]

[NOMS ’16, RFC7758]

TIMEFLIP

ONECLOCK

[INFOCOM ‘15]

[NOMS ‘16]

REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project

12

Example: Consistent Path Update before after

Sequential update approaches: [Dionysus ‘14] [zUpdate ‘13] [Vanbever et al. ‘11] [Francois et al. ’07]

0. 1. 2. 3.

The ‘before’ configuration. Controller updates S1. Controller updates S2. Controller updates S3.

X

S3

S4

S3

S4

S2

S1

S2

S1

0

wait…

S3

S4

S2

S1

1

2 The TimedSDN Project

wait…

S3

S4

S2

S1

S3

S4

S2

S1

3 13

Simultaneous Updates? En-route packets run into a ‘black hole’. Not consistent!

S3

S4

S3

S4

S3

S4

S3

S4

S2

S1

S2

S1

S2

S1

S2

S1

1

2 The TimedSDN Project

3 14

Timed Multi-phase Consistent Updates [SOSR ‘16] - The controller sends timed update messages to S1, S2, S3. - Scheduled updates occur at times T1, T2, T3.

Controller does not need to wait between steps!

S3

S4

S3

S4

S3

S4

S3

S4

S2

S1

S2

S1

S2

S1

S2

S1

T1

The TimedSDN Project

T2

T3

15

The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization

Project

Flow Swaps TIME4 [INFOCOM, ‘16]

Multi-phase updates [SOSR, ‘15]

Data Plane Timestamping [SWFAN, ‘16]

ONECLOCK to Rule Them All [NOMS, ‘16]

OpenFlow Scheduled Bundles

NETCONF Time Capability

[INFOCOM ’16, OpenFlow 1.5]

[NOMS ’16, RFC7758]

TIMEFLIP

ONECLOCK

[INFOCOM ‘15]

[NOMS ‘16]

REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project

16

Using Timestamps instead of The advantages of timestamping are well-known, e.g., [Lamport ‘78]. Current work: Data Plane Timestamping (DPT) in SDN. ingress switch

Timestamp is • Pushed by ingress switch. • Removed by egress switch.

egress switch

VNF

X

External Network

vSwitch

Timestamp can be used in processing of intermediate devices. The TimedSDN Project

Timestamped Domain 17

DPT Header Example: NSH Encapsulation • All traffic is encapsulated with an NSH. • NSH may include a timestamp [NSH-Timestamp]. [NSH-Timestamp] is under discussion in the SFC working group of the IETF.

ingress switch

egress switch

VNF

External Network

vSwitch

Timestamped Domain • •

NSH is used for Service Function Chaining (SFC). Defined by the SFC working group of the IETF.

Network Service Header (NSH) encapsulation

[NSH-Timestamp] R. Browne, A. Chilikin, B. Ryan, T. Mizrahi, and Y. Moses, “Network service header timestamping,” draft-browne-sfc-nsh-timestamp, work in progress, IETF, 2015. The TimedSDN Project

18

Timestamps Ranges SDN switch (OpenFlow / P4) matchaction procedure SDN Switch

SDN Switch DPT Processing

Flow Table

field1 field2 field3

Flow Table

…

packet header

action

timestamp field2 field3 field4 … packet header

action

Each Eachbit bithas hasaternary ternaryvalue: value:00//11//**

Timestamp-based ranges [TimeFlip, INFOCOM ‘15] Extremal range (T≥T0)

T0 0

Periodic range

... timestamp value

0

The TimedSDN Project

... timestamp value 19

Timestamp Format Network Time Protocol (NTP) timestamp format: Time.Sec

Time.Frac

Seconds

Second Fraction

32 bits

32 bits

The TimedSDN Project

20

Timestamps Ranges: Examples

packet header

01 1

SDN Switch

SDN Switch

DPT Processing

DPT Processing

Flow Table

Flow Table

timestamp field1 field2 field3 … action timestamp field1 field2 field3 …

*…* *…*

*…* *…*

Time.Sec

Time.Frac

timestamp field2 field3 field4 … packet header

231>T≥230 T≥231

*…*

Time.Sec

0

*…*

Time.Frac

Periodic range

Extremal range: T≥230 230

1

action

... timestamp value

0

... timestamp value

1 second The TimedSDN Project

21

DPT Use Case 1: Network Telemetry Delay measurement Loss measurement

S1

S2

T1

Time

T2

One-way delay: T2-T1 Obviously DPT can be used for delay measurement. How can DPT be used for loss measurement? The TimedSDN Project

22

Color-based Loss Measurement • [Coloring]: All data packets from S1 to S2 include a 1-bit ‘color’. • Measurement is performed separately for each block. S1 inserts 1-bit color into packet header.

S1 CS11

S1 toggles color.

S1 and S2 periodically export counters to central point, e.g., SDN controller. Controller can compute: Packet loss = (CS12-CS11)-(CR12-CR11)

CR11 CS01 CR01

CS12

Time

S1 and S2 maintain per-color counters.

S2

CR12

[Coloring] M. Chen, L. Zheng, G. Mirsky, G. Fioccola, and T. Mizrahi, “IP Flow Performance Measurement Framework,” draft-chen-ippm-coloringThe TimedSDN Project based-ipfpm-framework, work in progress, 2016.

23

Color-based Delay Measurement Delay measurement:

S1

S2

T1

Time

T2

T3 T4

Controller can compute: Two-way delay = (T4-T1)-(T3-T2) The TimedSDN Project

24

Color-based Measurement

Q: How does an SDN switch toggle the color? A: Controller periodically updates S1 coloring policy... Overhead on the controller…

S1 CS11

S2 CR11

CS01 CR01

Time

CS12

The TimedSDN Project

CR12

25

DPT-based Coloring • Choose 1 bit from the timestamp. • Use this bit as the color. • Each switch has two match-action rules: • Timestamp bit=0 • Timestamp bit=1

‘1’

• Each block: a constant time interval. • The controller is not involved in the color toggling. 

The TimedSDN Project

Time.Sec

Time.Frac

Seconds

Second Fraction

color ‘0’ ‘1’

‘0’

timestamp ... value

26

DPT-based Coloring DPT allows for both delay and loss measurement. No other fields are necessary! ‘1’

Time.Sec

Time.Frac

Seconds

Second Fraction

color ‘0’ ‘1’

‘0’

timestamp ... value

A 1-bit timestamp is sufficient!

DPT-based coloring reduces the load on controller.

The TimedSDN Project

27

DPT Use Case 2: Consistent Updates Multicast tree update • • •

Two-phase update [Reitblatt et al. ’12]. S2 attaches version tags to packets. • ‘before’ / ‘after’. • Indicate distribution tree. Guarantees per-packet consistency.

The TimedSDN Project

S3

S4

S2

S1

before after

28

Two-Phase Consistent Updates 1. Controller sends ‘after’ configuration to S1,S2 (subject to ‘after’ version tag). 2. Controller updates S2 to start sending ‘after’ version tag.

S3

S4

S3

S4

S3

S4

S2

S1

S2

S1

S2

S1

1 The TimedSDN Project

2 29

DPT-based One-Phase Consistent Updates 1. Controller sends ‘after’ configuration to S1,S2 (subject to Timestamp≥Tthreshold). 2. Controller updates S2 to start sending ‘after’ version tag. Switches automatically start using the ‘after’ path when Timestamp≥Tthreshold

The DPT is the version tag. S3

S4

S3

S4

S2

S1

S2

S1

DPT

S3

S4

S2

S1

1 The TimedSDN Project

30

DPT-based One-Phase Consistent Updates DPT reduces the load on controller.

DPT reduces the management overhead of per-flow version tags.

The TimedSDN Project

31

DPT Use Case 3: Elephant Flow Load Balancing

src

20

S3 10

S1

10 A 20 20 S2 S5 dst B 10 S 10 4

20 Gbps elephant flow. Stateless load balancing: - Equal Cost Multiple Paths (ECMP)  50% utilization. - Random Packet Spraying (RPS) [Dixit et al. ‘13]. Stateful methods, e.g., Round Robin. The TimedSDN Project

32

DPT-based Load Balancing Push DPT to all packets.

src

20

S1

S3 10

10 A 20 20 S2 S5 dst B 10 S 10 4

20 Gbps elephant flow.

DPT-based load balancing:

...

A

B

A

B

... timestamp value

toggle interval The TimedSDN Project

33

DPT-based Load Balancing

src

20

S3 10

S1

10 A 20 20 S2 S5 dst B 10 S 10 4

How does the controller compute the toggle interval? • L = maximal packet length. ... A B • BW = total bandwidth of the two links. • Toggle interval = L / BW Toggle interval Independent of the flow rate ! The TimedSDN Project

A

B

... timestamp value 34

Goodput of DPT-based Load Balancing • Load balancing using 1-bit from the DPT. • DPT: higher goodput than RPS / ECMP.

A

Time.Sec

Time.Frac

Seconds

Second Fraction

B

A

B

Toggle interval = L / BW  1 msec

timestamp ... value

A B The TimedSDN Project

35

The Why do we need time in SDN? Scheduling protocols Accurate scheduling methods SDN Clock synchronization

Project

Flow Swaps TIME4 [INFOCOM, ‘16]

Multi-phase updates [SOSR, ‘15]

Data Plane Timestamping [SWFAN, ‘16]

ONECLOCK to Rule Them All [NOMS, ‘16]

OpenFlow Scheduled Bundles

NETCONF Time Capability

[INFOCOM ’16, OpenFlow 1.5]

[NOMS ’16, RFC7758]

TIMEFLIP

ONECLOCK

[INFOCOM ‘15]

[NOMS ‘16]

REVERSEPTP [HotSDN ‘14, ISPCS ‘14] The TimedSDN Project

36

OneClock to Rule Them All

[NOMS ‘16]

Many networked applications require time - Sensors - Actuators - IoT devices

- Switches - Routers - Toasters Client

Client

T T Server 1 Server 2

...

T Server n

T T Server 1 Server 2

Coordinated operation

...

T Server n

Coordinated snapshot The TimedSDN Project

37

Summary

Summary Controller A protocol allowing the controller to schedule network updates

Synchronized clocks

Switches

The Timed

Project

http://tx.technion.ac.il/~dew/TimedSDN.html The TimedSDN Project

39

Thanks!

References [1]

T. Mizrahi, Y. Moses, “Software Defined Networks: It’s About Time", IEEE INFOCOM, 2016.

[2]

T. Mizrahi, Y. Moses, “OneClock to Rule Them All: Using Time in Networked Applications”, IEEE/IFIP Network Operations and Management Symposium (NOMS), 2016.

[3]

T. Mizrahi, E. Saat, Y. Moses, "Timed Consistent Network Updates", ACM SIGCOMM Symposium on SDN Research (SOSR), 2015. (http://tx.technion.ac.il/~dew/TimedConsistentSOSR.pdf)

[4]

T. Mizrahi, O. Rottenstreich, Y. Moses, "TimeFlip: Scheduling Network Updates with Timestamp-based TCAM Ranges", IEEE INFOCOM, 2015. (http://tx.technion.ac.il/~dew/TimeFlipINFOCOM.pdf)

[5]

T. Mizrahi, Y. Moses, "Time-based Updates in Software Defined Networks", the second workshop on hot topics in software defined networks (HotSDN), 2013. (http://tx.technion.ac.il/~dew/TimeSDN.pdf)

[6]

T. Mizrahi, Y. Moses, “Using ReversePTP to Distribute Time in Software Defined Networks”, International IEEE Symposium on Precision Clock Synchronization for Measurement, Control and Communication, (ISPCS), 2014. (http://tx.technion.ac.il/~dew/ISPCSReversePTPFinal.pdf)

[7]

T. Mizrahi, Y. Moses, "ReversePTP: A Software Defined Networking Approach to Clock Synchronization", in the third workshop on hot topics in software defined networks (HotSDN), 2014. (http://tx.technion.ac.il/~dew/ReversePTPHotSDN.pdf)

[8]

T. Mizrahi, Y. Moses, "Time4: Time for SDN", arXiv preprint arXiv:1505.03421, 2016. (http://tx.technion.ac.il/~dew/Time4TR.pdf)

[9]

Open Networking Foundation, OpenFlow switch specification, Version 1.5.0, 2015. (https://www.opennetworking.org/images/stories/downloads/sdn-resources/onf-specifications/openflow/openflow-switch-v1.5.0.noipr.pdf)

[10]

Open Networking Foundation, OpenFlow extensions 1.3.x package 2, 2015. (https://www.opennetworking.org/images/stories/downloads/sdn-resources/onf-specifications/openflow/openflow-extensions-1.3.x-pack2-noipr.zip) The TimedSDN Project

41