Distributed Operating Systems: Theoretical Foundations

Distributed Operating Systems: Theoretical Foundations Neeraj Mittal Department of Computer Science The University of Texas at Dallas

[email protected]

Department of Computer Science, The University of Texas at Dallas

CS 6378: Advanced Operating Systems – Chapter 5: Theoretical Foundations

Slide 1/44

Two Important Characteristics Inherent Limitations

■

Absence of Global Clock ◆ there is no common notion of time

❖ Two Important Characteristics ❖ Absence of Global Clock ❖ Absence of Shared Memory Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 2/44

Two Important Characteristics Inherent Limitations

■

Absence of Global Clock ◆ there is no common notion of time

❖ Two Important Characteristics ❖ Absence of Global Clock ❖ Absence of Shared Memory Ordering of Events Abstract Clocks

■

Absence of Shared Memory ◆ no process has up-to-date knowledge about the system

Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 2/44

Absence of Global Clock Inherent Limitations

■

Different processes may have different notions of time




Slide 3/44


■


❖ Two Important Characteristics ❖ Absence of Global Clock ❖ Absence of Shared Memory Ordering of Events

I invented AIDS cure!

I invented AIDS cure! Abstract Clocks Ordering of Messages State of a Distributed System Monitoring a Distributed System

Earth


Mars


Slide 3/44


■





Abstract Clocks Ordering of Messages

Who did it first?

State of a Distributed System Monitoring a Distributed System

Patent Officer

Earth


Mars


Slide 3/44


■






Who did it first?


Patent Officer

Earth

Mars

Problem: How do we order events on different processes? Department of Computer Science, The University of Texas at Dallas


Slide 3/44

Absence of Shared Memory Inherent Limitations

■

A process does not know current state of other processes




Slide 4/44


■



0 Harry Bob



Slide 4/44


■



0 Harry Bob



Slide 4/44


■


❖ Two Important Characteristics ❖ Absence of Global Clock ❖ Absence of Shared Memory Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System

What is Harry doing at present?

Monitoring a Distributed System

0 Harry Bob



Slide 4/44


■


❖ Two Important Characteristics ❖ Absence of Global Clock ❖ Absence of Shared Memory Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System

What is Harry doing at present?


0 Harry Bob

Problem: How do we obtain a coherent view of the system?



Slide 4/44

When is it possible to order two events? Inherent Limitations

Three cases:

Ordering of Events

1. Events executed on the same process:

❖ When is it possible to order two events? ❖ Happened-Before Relation ❖ Concurrent Events Abstract Clocks Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 5/44


Three cases:

Ordering of Events

1. Events executed on the same process: ■ if e and f are events on the same process and e occurred before f , then e happened-before f

❖ When is it possible to order two events? ❖ Happened-Before Relation ❖ Concurrent Events Abstract Clocks Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 5/44


Three cases:

Ordering of Events


❖ When is it possible to order two events? ❖ Happened-Before Relation ❖ Concurrent Events Abstract Clocks Ordering of Messages State of a Distributed System

2. Communication events of the same message:




Slide 5/44


Three cases:

Ordering of Events



2. Communication events of the same message: ■ if e is the send event of a message and f is the receive event of the same message, then e happened-before f




Slide 5/44


Three cases:

Ordering of Events





3. Events related by transitivity:



Slide 5/44


Three cases:

Ordering of Events





3. Events related by transitivity: ■ if event e happened-before event g and event g happened-before event f , then e happened-before f



Slide 5/44

Happened-Before Relation Inherent Limitations

■

Happened-before relation is denoted by →

Ordering of Events ❖ When is it possible to order two events? ❖ Happened-Before Relation ❖ Concurrent Events Abstract Clocks Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 6/44


■ ■

Happened-before relation is denoted by → Illustration:

Ordering of Events ❖ When is it possible to order two events? ❖ Happened-Before Relation ❖ Concurrent Events

a

b

c

Abstract Clocks

P1

Ordering of Messages

d

P2

e

State of a Distributed System

f


P3 g


h

i


Slide 6/44


■ ■



a

b

c

Abstract Clocks

P1


d

P2

e


f


P3 g

◆

h

i

Events on the same process: examples: a → b, a → c, d → f



Slide 6/44


■ ■



a

b

c

Abstract Clocks

P1


d

P2

e


f


P3 g

◆

◆

h

i

Events on the same process: examples: a → b, a → c, d → f Events of the same message: examples: b → e, f → i



Slide 6/44


■ ■



a

b

c

Abstract Clocks

P1


d

P2

e


f


P3 g

◆

◆

◆

h

i

Events on the same process: examples: a → b, a → c, d → f Events of the same message: examples: b → e, f → i Transitivity: examples: a → e, a → i, e → i



Slide 6/44

Concurrent Events Inherent Limitations

■

Events not related by happened-before relation




Slide 7/44


■ ■

Events not related by happened-before relation Concurrency relation is denoted by k




Slide 7/44


■ ■ ■

Events not related by happened-before relation Concurrency relation is denoted by k Illustration:

Ordering of Events ❖ When is it possible to order two events? ❖ Happened-Before Relation ❖ Concurrent Events Abstract Clocks

a

b

c


P1


d

P2

e


f

P3 g


h

i


Slide 7/44


■ ■ ■



a

b

c


P1


d

P2

e


f

P3 g

◆

h

i

Examples: a k d, d k h, c k e



Slide 7/44


■ ■ ■



a

b

c


P1


d

P2

e


f

P3 g

◆ ■

h

i

Examples: a k d, d k h, c k e

Concurrency relation is not transitive: example: a k d and d k c but a ∦ c



Slide 7/44

Different Kinds of Clocks Inherent Limitations

■

Logical Clocks ◆ used to totally order all events

Ordering of Events Abstract Clocks ❖ Different Kinds of Clocks ❖ Logical Clock ❖ Implementing Logical Clock ❖ Implementing Logical Clock: An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock ❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 8/44


■


Ordering of Events Abstract Clocks ❖ Different Kinds of Clocks ❖ Logical Clock

■

Vector Clocks ◆ used to track happened-before relation

❖ Implementing Logical Clock ❖ Implementing Logical Clock: An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock ❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 8/44


■



■


❖ Implementing Logical Clock ❖ Implementing Logical Clock: An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock

■

Matrix Clocks ◆ used to track what other processes know about other processes

❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 8/44


■



■


❖ Implementing Logical Clock ❖ Implementing Logical Clock: An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock

■

Matrix Clocks ◆ used to track what other processes know about other processes

❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages State of a Distributed System Monitoring a Distributed

■

Direct Dependency Clocks ◆ used to track direct causal dependencies



System

Slide 8/44

Logical Clock Inherent Limitations

■

Implements the notion of virtual time




Slide 9/44


■


Ordering of Events Abstract Clocks

■

Can be used to totally order all events

❖ Different Kinds of Clocks ❖ Logical Clock ❖ Implementing Logical Clock ❖ Implementing Logical Clock: An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock ❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 9/44


■



■

■

Can be used to totally order all events Assigns timestamp to each event in a way that is consistent with the happened-before relation: e → f ⇒ C(e) < C(f )

❖ Different Kinds of Clocks ❖ Logical Clock ❖ Implementing Logical Clock ❖ Implementing Logical Clock: An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock ❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages

C(e): timestamp for event e C(f ): timestamp for event f




Slide 9/44

Implementing Logical Clock Inherent Limitations

■

Each process has a local scalar clock, initialized to zero ◆ Ci denotes the local clock of process Pi




Slide 10/44


■

■

Each process has a local scalar clock, initialized to zero ◆ Ci denotes the local clock of process Pi Action depends on the type of the event:




Slide 10/44


■


■

Action depends on the type of the event:

■

Protocol for process Pi :




Slide 10/44


■


■


■

Protocol for process Pi : ◆

On executing an interval event: Ci := Ci + 1




Slide 10/44


■


■


■


◆

On executing an interval event: Ci := Ci + 1 On sending a message m: Ci := Ci + 1 piggyback Ci on m




Slide 10/44


■


■


■


◆

◆

On executing an interval event: Ci := Ci + 1 On sending a message m: Ci := Ci + 1 piggyback Ci on m


On receiving a message m: let tm be the timestamp piggybacked on m Ci := max{Ci , tm } + 1



Slide 10/44

Implementing Logical Clock: An Illustration Inherent Limitations Ordering of Events Abstract Clocks ❖ Different Kinds of Clocks ❖ Logical Clock

P1

a

❖ Implementing Logical Clock ❖ Implementing Logical Clock:

1

An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock ❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock

P2

Ordering of Messages State of a Distributed System

P3

Monitoring a Distributed

1

a: internal event


System

C(a) = 1


Slide 11/44


P1

a


1

An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock

b

❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock

P2 1


c


P3


1

System

b and c: internal events


C(b) = 1 and C(c) = 1


Slide 11/44


a P1

1

d


2

An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock

2

❖ Comparing Two Vectors ❖ Implementing Vector Clock

b


P2 1


c


P3


1

d: send event


System

C(d) = 2


Slide 11/44


a P1

1

d


2


2


b

e

1

3


P2


c


P3


1

e: receive event


System

C(e) = 3


Slide 11/44


a P1

1

d


2


3

2

4


b

e

1

3


P2 4

4


c


P3


1


2

5


System

Slide 11/44

Limitation of Logical Clock Inherent Limitations

■

Logical clock cannot be used to determine whether two events are concurrent




Slide 12/44

Limitation of Logical Clock Inherent Limitations

■

Logical clock cannot be used to determine whether two events are concurrent

Ordering of Events Abstract Clocks ❖ Different Kinds of Clocks ❖ Logical Clock ❖ Implementing Logical Clock ❖ Implementing Logical Clock:

e k f does not imply C(e) = C(f )

An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock ❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 12/44

Vector Clock Inherent Limitations

■

Captures the happened-before relation




Slide 13/44

Vector Clock Inherent Limitations

■

Captures the happened-before relation


■

Assigns timestamp to each event such that: e → f ⇐⇒ C(e) < C(f ) C(e): timestamp for event e C(f ): timestamp for event f

❖ Different Kinds of Clocks ❖ Logical Clock ❖ Implementing Logical Clock ❖ Implementing Logical Clock: An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock ❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 13/44

Comparing Two Vectors Inherent Limitations

■

Vectors are compared component-wise:




Slide 14/44


■



◆

Equality:

❖ Different Kinds of Clocks ❖ Logical Clock

V =V

0

iff h∀i : V [i] = V [i]i 0

❖ Implementing Logical Clock ❖ Implementing Logical Clock: An Illustration ❖ Limitation of Logical Clock ❖ Vector Clock ❖ Comparing Two Vectors ❖ Implementing Vector Clock ❖ Implementing Vector Clock: An Illustration ❖ Properties of Vector Clock Ordering of Messages State of a Distributed System Monitoring a Distributed System



Slide 14/44


■



◆

Equality:

❖ Different Kinds of Clocks ❖ Logical Clock

V =V ◆

0


iff h∀i : V [i] = V [i]i 0


Less Than: V V w

Ordering of Events Abstract Clocks Ordering of Messages ❖ Ordering of Messages ❖ Useful Notations ❖ Causal Delivery of Messages ❖ A Causally Ordered Delivery Protocol ❖ The BSS Protocol ❖ The BSS Protocol: An Illustration ❖ Another Causally Ordered Delivery Protocol ❖ When to Deliver a Message? ❖ The SES Protocol ❖ The SES Protocol (Continued) ❖ The SES Protocol: An Illustration State of a Distributed System Monitoring a Distributed System



Slide 25/44

When to Deliver a Message? Inherent Limitations

■

A message m destined for process Pi can be delivered if: ◆

for every message w that causally precedes m and destined for Pi : Vi > V w

Ordering of Events Abstract Clocks Ordering of Messages ❖ Ordering of Messages ❖ Useful Notations ❖ Causal Delivery of Messages ❖ A Causally Ordered Delivery Protocol ❖ The BSS Protocol ❖ The BSS Protocol: An

or, equivalently

Illustration ❖ Another Causally Ordered Delivery Protocol ❖ When to Deliver a Message? ❖ The SES Protocol ❖ The SES Protocol (Continued) ❖ The SES Protocol: An Illustration State of a Distributed System Monitoring a Distributed System



Slide 25/44

When to Deliver a Message? Inherent Limitations

■

A message m destined for process Pi can be delivered if: ◆

for every message w that causally precedes m and destined for Pi : Vi > V w

Ordering of Events Abstract Clocks Ordering of Messages ❖ Ordering of Messages ❖ Useful Notations ❖ Causal Delivery of Messages ❖ A Causally Ordered Delivery Protocol ❖ The BSS Protocol ❖ The BSS Protocol: An

or, equivalently

Illustration ❖ Another Causally Ordered Delivery Protocol ❖ When to Deliver a Message? ❖ The SES Protocol ❖ The SES Protocol (Continued) ❖ The SES Protocol: An

◆

let past(m, j) denote the set of all messages that causally precede m and are destined for process Pj : Vi >


max

w∈past(m,i)

{Vw }


Illustration State of a Distributed System Monitoring a Distributed System

Slide 25/44

The SES Protocol Inherent Limitations

■

Each process maintains a list of tuples with at most one tuple for every other process ◆

let DLi denote the list for process Pi




Slide 26/44


■


■


The list is piggybacked on every message a process sends ◆

let DLm denote the list for message m




Slide 26/44


■


■



let DLm denote the list for message m ■

if past(m, j) = ∅, then DLm does not contain a tuple for process Pj




Slide 26/44


■


■



let DLm denote the list for message m ■

■

if past(m, j) = ∅, then DLm does not contain a tuple for process Pj Otherwise, the tuple for process Pj is given by:

Ordering of Events Abstract Clocks Ordering of Messages ❖ Ordering of Messages ❖ Useful Notations ❖ Causal Delivery of Messages ❖ A Causally Ordered Delivery Protocol ❖ The BSS Protocol ❖ The BSS Protocol: An Illustration ❖ Another Causally Ordered Delivery Protocol ❖ When to Deliver a Message? ❖ The SES Protocol ❖ The SES Protocol (Continued) ❖ The SES Protocol: An Illustration State of a Distributed System

(j,


max

w∈past(m,j)

{Vw })



Slide 26/44

The SES Protocol (Continued) Inherent Limitations

■





Slide 27/44


■



◆

On sending a message m to process Pj : piggyback DLi on m remove entry for Pj from DLi , if any add (j, Vm ) to DLi

Ordering of Messages ❖ Ordering of Messages ❖ Useful Notations ❖ Causal Delivery of Messages ❖ A Causally Ordered Delivery Protocol ❖ The BSS Protocol ❖ The BSS Protocol: An Illustration ❖ Another Causally Ordered Delivery Protocol ❖ When to Deliver a Message? ❖ The SES Protocol ❖ The SES Protocol (Continued) ❖ The SES Protocol: An Illustration State of a Distributed System Monitoring a Distributed System



Slide 27/44


■



◆

◆

On sending a message m to process Pj : piggyback DLi on m remove entry for Pj from DLi , if any add (j, Vm ) to DLi On arrival of a message m from process Pj : if DLm does not contain any tuple for Pi then deliver m else let (j, V ) be the tuple for Pi in DLm deliver m once Vi > V endif




Slide 27/44


■



◆

◆

◆

On sending a message m to process Pj : piggyback DLi on m remove entry for Pj from DLi , if any add (j, Vm ) to DLi On arrival of a message m from process Pj : if DLm does not contain any tuple for Pi then deliver m else let (j, V ) be the tuple for Pi in DLm deliver m once Vi > V endif


On delivery of a message m: merge DLi and DLm



Slide 27/44

The SES Protocol: An Illustration Inherent Limitations Ordering of Events Abstract Clocks

a P1

Ordering of Messages ❖ Ordering of Messages ❖ Useful Notations

{}

❖ Causal Delivery of Messages ❖ A Causally Ordered Delivery

0 0 0

Protocol ❖ The BSS Protocol ❖ The BSS Protocol: An Illustration ❖ Another Causally Ordered Delivery Protocol ❖ When to Deliver a Message? ❖ The SES Protocol

P2 {}

❖ The SES Protocol (Continued) ❖ The SES Protocol: An Illustration

0 0 0


P3 {} 0 0 0


1 0 1


Slide 28/44


a

P1

Ordering of Messages ❖ Ordering of Messages ❖ Useful Notations

1 0 0

{} {(3, )} 0 0 0

1 0 0


1 0 0

Protocol ❖ The BSS Protocol ❖ The BSS Protocol: An

{}

Illustration ❖ Another Causally Ordered Delivery Protocol ❖ When to Deliver a Message? ❖ The SES Protocol

P2 {}


0 0 0


P3 {} 0 0 0


1 0 1


Slide 28/44


a

P1

1 0 0

{} {(3, )} 0 0 0

1 0 0

b Ordering of Messages

{(3,

1 0 0

), (2,

2 0 0

)}

2 0 0

2 0 0

1 0 0

c

{(3,

1 0 0

❖ Ordering of Messages ❖ Useful Notations ❖ Causal Delivery of Messages ❖ A Causally Ordered Delivery

)}


{}


P2 {}


0 0 0


P3 {} 0 0 0


1 0 1


Slide 28/44


a

P1

1 0 0

{} {(3, )} 0 0 0

1 0 0


{(3,

1 0 0

), (2,

2 0 0

)}

2 0 0

2 0 0

P2 {} 0 0 0

1 0 0

c

{(3,

1 0 0

❖ Ordering of Messages ❖ Useful Notations ❖ Causal Delivery of Messages ❖ A Causally Ordered Delivery

)}


{}


1

{(3, 0 )} 0


2 1 0


P3 {} 0 0 0


1 0 1


Slide 28/44


a

P1

1 0 0

{} {(3, )} 0 0 0

1 0 0


{(3,

1 0 0

), (2,

2 0 0

)}

2 0 0

2 0 0

P2 {} 0 0 0

1 0 0

c 1

1 0 0

{(3,

❖ Ordering of Messages ❖ Useful Notations


{} d


2

{(3, 0 )} {(3, 2 ))} 0 0 2 1 0


)}

2 2 0

1 2 0 )} 2 {(3, 0

0

❖ The SES Protocol (Continued) ❖ The SES Protocol: An Illustration State of a Distributed System Monitoring a Distributed System

e P3 {} 0 0 0


1 0 1


Slide 28/44


a

P1

1 0 0

{} {(3, )} 0 0 0

1 0 0


{(3,

1 0 0

), (2,

2 0 0

)}

2 0 0

2 0 0

P2 {} 0 0 0

1 0 0

c 1

1 0 0

{(3,



{} d


2

{(3, 0 )} {(3, 2 ))} 0 0 2 1 0


)}

2 2 0

1 2 0 )} 2 {(3, 0

0

❖ The SES Protocol (Continued) ❖ The SES Protocol: An Illustration State of a Distributed System Monitoring a Distributed

e P3 {} 0 0 0


g

System

{} 1 0 1


Slide 28/44


a

P1

1 0 0

{} {(3, )} 0 0 0

1 0 0


{(3,

1 0 0

), (2,

2 0 0

)}

2 0 0

2 0 0

P2 {} 0 0 0

1 0 0

c 1

1 0 0

{(3,



{} d


2

{(3, 0 )} {(3, 2 ))} 0 0 2 1 0


)}

2 2 0

1 2 0 )} 2 {(3, 0

0

❖ The SES Protocol (Continued) ❖ The SES Protocol: An Illustration State of a Distributed System Monitoring a Distributed

P3 {} 0 0 0


g

f

{}

{}

1 0 1

System

2 2 2


Slide 28/44

State of a Distributed System Inherent Limitations

■

State of a distributed system is a collection of states of all its processes and channels:

Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System ❖ State of a Distributed System ❖ Events and Local States ❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol ❖ The CL Protocol (Continued) ❖ The CL Protocol: An Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 29/44


■

State of a distributed system is a collection of states of all its processes and channels: ◆

a process state is given by the values of all variables on the process




Slide 29/44


■


◆




a channel state is given by the set of messages in transit in the channel

❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before

State of a Distributed System ❖ State of a Distributed System ❖ Events and Local States

Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol ❖ The CL Protocol (Continued) ❖ The CL Protocol: An Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 29/44


■


◆






■

can be determined by examining states of the two processes it connects





Slide 29/44


■


◆

Abstract Clocks





■

■

Ordering of Events


State of a process is called local state or local snapshot ◆

the textbook refers to local state as cut event

◆

cut event is not same as event





Slide 29/44


■


◆

■

Abstract Clocks





■

■

Ordering of Events


State of a process is called local state or local snapshot ◆

the textbook refers to local state as cut event

◆

cut event is not same as event



State of a distributed system is called global state or global snapshot



Slide 29/44

Events and Local States Inherent Limitations

■

Processes change their states by executing events




Slide 30/44


■



■

Example:


x=0

x=2

x=3

P1 a y=1

b y=2

❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent

y=3

P2 c

❖ State of a Distributed System ❖ Events and Local States

d

Global Snapshot ❖ The CL Protocol ❖ The CL Protocol (Continued) ❖ The CL Protocol: An Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 30/44


■



■

Example:


x=0

x=2

x=3

P1 a y=1

b y=2

◆

y=3 d

Process P1 changes its state from x = 0 to x = 2 on executing event a



P2 c




Slide 30/44


■



■

Example:


x=0

x=2

x=3

P1 a y=1

b y=2

◆

◆

y=3 d

Process P1 changes its state from x = 0 to x = 2 on executing event a Process P2 changes its state from y = 2 to y = 3 on executing event d



P2 c




Slide 30/44

When is a Global State Meaningful? Inherent Limitations

■

Example:


x=0

x=2

x=3 Ordering of Messages

P1 a

b


y=1

y=2

y=3

P2 c

d

❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol ❖ The CL Protocol (Continued) ❖ The CL Protocol: An Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 31/44


■

Example:


x=0

x=2


P1 a

b


y=1

y=2

y=3

P2 c

d

❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol

◆

Is it possible for x to be 0 and y to be 3 at the same time?

❖ The CL Protocol (Continued) ❖ The CL Protocol: An Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 31/44


■

Example:


x=0

x=2


P1 s

a

t

b

u


y=1

y=2

y=3

P2 v

c

w

d

x

❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol

◆ ◆

Is it possible for x to be 0 and y to be 3 at the same time?

❖ The CL Protocol (Continued) ❖ The CL Protocol: An

Does {s, x} form a meaningful global state?

Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 31/44

Revisiting Happened-Before Relation Inherent Limitations

■

Typically, happened-before relation is defined on events




Slide 32/44


■



■

The relation can be extended to local states as follows:

Ordering of Messages State of a Distributed System ❖ State of a Distributed System ❖ Events and Local States ❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol ❖ The CL Protocol (Continued) ❖ The CL Protocol: An Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 32/44


■



■

The relation can be extended to local states as follows: ◆

let s be a local state of process Pi




Slide 32/44


■



■

The relation can be extended to local states as follows: ◆ ◆

let s be a local state of process Pi let t be a local state of process Pj




Slide 32/44


■



■

The relation can be extended to local states as follows: ◆ ◆ ◆

let s be a local state of process Pi let t be a local state of process Pj s → t if there exist events e and f such that:




Slide 32/44


■



■


let s be a local state of process Pi let t be a local state of process Pj s → t if there exist events e and f such that: 1. Pi executed e after s,




Slide 32/44


■



■





2. Pj executed f before t, and


❖ State of a Distributed System ❖ Events and Local States ❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol

Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 32/44


■



■





2. Pj executed f before t, and


3. e → f

Illustration ❖ The CL Protocol: An Illustration (Continued)

❖ State of a Distributed System ❖ Events and Local States ❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol




Slide 32/44

A Consistent Global State Inherent Limitations

■

For a global state G, let G[i] refer to the local state of process Pi in G




Slide 33/44


■


Ordering of Events Abstract Clocks Ordering of Messages

■

A global state G is meaningful or consistent if ∀i, j : i 6= j : (G[i] 9 G[j]) ∧ (G[j] 9 G[i])

State of a Distributed System ❖ State of a Distributed System ❖ Events and Local States ❖ When is a Global State Meaningful? ❖ Revisiting Happened-Before Relation ❖ A Consistent Global State ❖ Recording a Consistent Global Snapshot ❖ The CL Protocol ❖ The CL Protocol (Continued) ❖ The CL Protocol: An Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed System



Slide 33/44


■



■

A global state G is meaningful or consistent if ∀i, j : i 6= j : G[i] k G[j]




Slide 33/44

Recording a Consistent Global Snapshot Inherent Limitations

■

Proposed by Chandy and Lamport (CL)




Slide 34/44


■ ■

Proposed by Chandy and Lamport (CL) Assumptions and Features:


◆

channels satisfy first-in-first-out (FIFO) property




Slide 34/44


■ ■



◆ ◆

channels satisfy first-in-first-out (FIFO) property channels are not required to be bidirectional




Slide 34/44


■ ■



◆ ◆ ◆

channels satisfy first-in-first-out (FIFO) property channels are not required to be bidirectional communication topology may not be fully connected




Slide 34/44


■ ■



◆ ◆ ◆ ■


Messages exchanged by the underlying computation (whose snapshot is being recorded) are called application messages




Slide 34/44


■ ■



◆ ◆ ◆ ■

■


Messages exchanged by the underlying computation (whose snapshot is being recorded) are called application messages Messages exchanged by the snapshot algorithm are called control messages




Slide 34/44

The CL Protocol Inherent Limitations

■

Initially: all processes are colored blue




Slide 35/44


■ ■

Initially: all processes are colored blue Eventually: all processes become red




Slide 35/44


■ ■



■

On changing color from blue to red: record local snapshot send a marker message along all outgoing channels




Slide 35/44


■ ■



■

■

On changing color from blue to red: record local snapshot send a marker message along all outgoing channels On receiving marker message along incoming channel C: if color is blue then change color from blue to red endif record state of channel C as application messages received along C since turning red




Slide 35/44


■ ■



■

■

■



Any process can initiate the snapshot protocol by spontaneously changing its color from blue to red



Slide 35/44


■ ■



■

■

■



Any process can initiate the snapshot protocol by spontaneously changing its color from blue to red ◆ there can be multiple initiators of the snapshot protocol



Slide 35/44

The CL Protocol (Continued) Inherent Limitations

■

Global Snapshot: local snapshots of processes just after they turn red




Slide 36/44


■

■

Global Snapshot: local snapshots of processes just after they turn red In-Transit Messages: blue application messages received by processes after they have turned red




Slide 36/44


■

■

■

Global Snapshot: local snapshots of processes just after they turn red In-Transit Messages: blue application messages received by processes after they have turned red Why is the global snapshot consistent?




Slide 36/44


■

■

■

Global Snapshot: local snapshots of processes just after they turn red In-Transit Messages: blue application messages received by processes after they have turned red Why is the global snapshot consistent? ◆ Assume an application message has the same color as its sender




Slide 36/44


■

■

■

Global Snapshot: local snapshots of processes just after they turn red In-Transit Messages: blue application messages received by processes after they have turned red Why is the global snapshot consistent? ◆ Assume an application message has the same color as its sender ◆ Can a blue process receive a red application message?




Slide 36/44

The CL Protocol: An Illustration Inherent Limitations

■ ■

Three processes: P1 , P2 and P3 Five channels: C1,2 , C1,3 , C2,1 , C2,3 and C3,1

Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System ❖ State of a Distributed System ❖ Events and Local States

P1


C1,3

C1,2

Global Snapshot ❖ The CL Protocol

C3,1 C2,1


P3

P2

Relation ❖ A Consistent Global State ❖ Recording a Consistent

Illustration ❖ The CL Protocol: An Illustration (Continued) Monitoring a Distributed

C2,3



System

Slide 37/44

The CL Protocol: An Illustration (Continued) Inherent Limitations Ordering of Events

P1


m5


m2

m4


m3 P2


m1

Global Snapshot ❖ The CL Protocol ❖ The CL Protocol (Continued) ❖ The CL Protocol: An Illustration ❖ The CL Protocol: An Illustration (Continued)

P3


1. All processes are blue



Slide 38/44


P1


m5


m2

m4


m3 P2


m1


P3


1. P1 turns red 2. P1 sends a marker message along C1,2 and C1,3



Slide 38/44


P1


m5


m2

m4


m3 P2


m1


P3


1. P2 receives the marker message along C1,2 and turns red 2. P2 sends a marker message along C2,1 and C2,3 3. P2 records the state of C1,2 as ∅



Slide 38/44


P1


m5


m2

m4


m3 P2


m1


P3


1. P3 receives the marker message along C1,3 and turns red 2. P3 sends a marker message along C3,1 3. P3 records the state of C1,3 as ∅



Slide 38/44


P1


m5


m2

m4


m3 P2


m1


P3


1. P1 receives the marker message along C2,1 2. P1 records the state of C2,1 as {m4 , m5 }



Slide 38/44


P1


m5


m2

m4


m3 P2


m1


P3


1. P3 receives the marker message along C2,3 2. P3 records the state of C2,3 as {m1 }



Slide 38/44


P1


m5


m2

m4


m3 P2


m1


P3


1. P1 receives the marker message along C3,1 2. P1 records the state of C3,1 as {m2 , m3 }



Slide 38/44

A Subclass of Distributed Computation Inherent Limitations

■

Many distributed computations obey the following paradigm:

Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System Monitoring a Distributed System ❖ A Subclass of Distributed Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm ❖ Huang’s Algorithm (Continued) ❖ Huang’s Algorithm: An Illustration



Slide 39/44


■

Many distributed computations obey the following paradigm: ◆

A process is either in active state or passive state




Slide 39/44


■

Many distributed computations obey the following paradigm: ◆ ◆

A process is either in active state or passive state A process can send an application message only when it is active




Slide 39/44


■


◆

A process is either in active state or passive state A process can send an application message only when it is active An active process can become passive at any time




Slide 39/44


■


◆ ◆

A process is either in active state or passive state A process can send an application message only when it is active An active process can become passive at any time A passive process, on receiving an application message, becomes active




Slide 39/44


■


◆ ◆

■



Intuitively:



Slide 39/44


■


◆ ◆

■



Intuitively: ◆

if a process is active, then it is doing some work



Slide 39/44


■


◆ ◆

■



Intuitively: ◆ ◆

if a process is active, then it is doing some work if process is passive, then it is idle



Slide 39/44


■


◆ ◆

■



Intuitively: ◆ ◆ ◆

if a process is active, then it is doing some work if process is passive, then it is idle an active process uses an application message to send a part of its work to another process



Slide 39/44

Distributed Computation: An Illustration Inherent Limitations

P1

Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System Monitoring a Distributed System ❖ A Subclass of Distributed Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm

P2

❖ Huang’s Algorithm (Continued) ❖ Huang’s Algorithm: An Illustration

P3



Slide 40/44


P1


m1

State of a Distributed System Monitoring a Distributed System ❖ A Subclass of Distributed Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm

P2


P3



Slide 40/44


P1


m1

State of a Distributed System Monitoring a Distributed System ❖ A Subclass of Distributed

P2

Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm ❖ Huang’s Algorithm (Continued) ❖ Huang’s Algorithm: An Illustration

P3



Slide 40/44


P1


m1


P2


m2

P3



Slide 40/44


P1


m1


P2


P3


m2


Slide 40/44


P1


m1


P2

Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm

m2

P3


m3



Slide 40/44


P1


m1

State of a Distributed System Monitoring a Distributed

m4


P2

System ❖ A Subclass of Distributed

m2

P3


m3



Slide 40/44


P1


m1


m4


P2


m2

P3


m3



Slide 40/44


P1


m1


P2

m4

m3



m2

P3



Slide 40/44


P1


m1


P2

m4

m3



m2

P3



Slide 40/44

Termination Detection Inherent Limitations

■

To detect if the computation has finished doing all the work




Slide 41/44


■

To detect if the computation has finished doing all the work ◆

all processes have become passive, and




Slide 41/44


■

To detect if the computation has finished doing all the work ◆ ◆

all processes have become passive, and all channels have become empty




Slide 41/44


■



Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System Monitoring a Distributed

■

Different types of computations:

System ❖ A Subclass of Distributed Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm ❖ Huang’s Algorithm (Continued) ❖ Huang’s Algorithm: An Illustration



Slide 41/44


■




■

Different types of computations: ◆

diffusing: only one process is active in the beginning




Slide 41/44


■




■

Different types of computations: ◆ ◆

diffusing: only one process is active in the beginning non-diffusing: any subset of processes can be active in the beginning




Slide 41/44


■




■

Different types of computations: ◆ ◆

diffusing: only one process is active in the beginning non-diffusing: any subset of processes can be active in the beginning ■ no process knows which processes are active and which processes are passive




Slide 41/44

Huang’s Algorithm Inherent Limitations

■

Assumption: computation is diffusing




Slide 42/44


■

Assumption: computation is diffusing ◆

the initially active process is called the coordinator




Slide 42/44


■


the initially active process is called the coordinator ■ coordinator is responsible for detecting termination




Slide 42/44


■



Ordering of Events Abstract Clocks Ordering of Messages State of a Distributed System

■


Initially:




Slide 42/44


■




■


Initially: ◆


coordinator has a weight of 1




Slide 42/44


■




■


Initially: ◆ ◆


coordinator has a weight of 1 all other processes have a weight of 0




Slide 42/44


■




■

◆ ◆ ■


Initially:




Invariants:




Slide 42/44


■




■

◆ ◆ ■


Initially:




Invariants: ◆


total amount of weight in the system is 1



Slide 42/44


■




■

◆ ◆ ■


Initially:




Invariants: ◆ ◆


total amount of weight in the system is 1 a non-coordinator process has a non-zero weight if and only if it is active



Slide 42/44


■




■

◆ ◆ ■


Initially:




Invariants: ◆ ◆

◆


total amount of weight in the system is 1 a non-coordinator process has a non-zero weight if and only if it is active a channel has a non-zero weight if and only if it is non-empty



Slide 42/44


■




■

◆ ◆ ■


Initially:




Invariants: ◆ ◆

◆


total amount of weight in the system is 1 a non-coordinator process has a non-zero weight if and only if it is active a channel has a non-zero weight if and only if it is non-empty ■ weight of a channel is the sum of the weight of all its messages



Slide 42/44

Huang’s Algorithm (Continued) Inherent Limitations

■

Actions:




Slide 43/44


■

Actions:


◆

On sending an application message:

Ordering of Messages State of a Distributed System Monitoring a Distributed System ❖ A Subclass of Distributed Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm ❖ Huang’s Algorithm (Continued) ❖ Huang’s Algorithm: An Illustration



Slide 43/44


■

Actions:


◆

On sending an application message: ■ send half of its weight along with the message




Slide 43/44


■

Actions:


◆

◆

On sending an application message: ■ send half of its weight along with the message On receiving an application message:




Slide 43/44


■

Actions:


◆

◆

On sending an application message: ■ send half of its weight along with the message On receiving an application message: ■ add the weight of the message to the current weight




Slide 43/44


■

Actions:


◆

◆

◆

On sending an application message: ■ send half of its weight along with the message On receiving an application message: ■ add the weight of the message to the current weight On becoming passive:




Slide 43/44


■

Actions:


◆

◆

◆

On sending an application message: ■ send half of its weight along with the message On receiving an application message: ■ add the weight of the message to the current weight On becoming passive: ■ send the current weight to the coodrinator




Slide 43/44


■

Actions:


◆

◆

◆

■



Coordinator announces termination once:



Slide 43/44


■

Actions:


◆

◆

◆

■



Coordinator announces termination once: ◆

it has become passive and



Slide 43/44


■

Actions:


◆

◆

◆

■



Coordinator announces termination once: ◆ ◆

it has become passive and it has collected all the weight



Slide 43/44

Huang’s Algorithm: An Illustration Inherent Limitations Ordering of Events Abstract Clocks

1

P1

Ordering of Messages State of a Distributed System Monitoring a Distributed System ❖ A Subclass of Distributed Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm

0


P2

0 P3



Slide 44/44


P1

1 2


1


m1( 12 )

System ❖ A Subclass of Distributed Computation ❖ Distributed Computation: An Illustration ❖ Termination Detection ❖ Huang’s Algorithm

0


P2

0 P3



Slide 44/44


P1

1 2


1


m1( 12 )

0


1 2


P2


0 P3



Slide 44/44


P1

1 2


1


m1( 12 )

0

1 2


1 4


P2


m2( 14 )

0 P3



Slide 44/44

Huang’s Algorithm: An Illustration Inherent Limitations Ordering of Events Abstract Clocks Ordering of Messages

P1

1 2

1


m1( 12 )

0

1 2


1 4


P2


m2( 14 )

P3


1 4

0


Slide 44/44


P1

1 2

1


m1( 12 )

0

1 2


1 4


P2


m2( 14 )

P3


1 4

1 8

0

m3( 18 )


Slide 44/44


P1

1 2

1


m1( 12 )

0

1 2

1 4


3 8


P2

1 m4( 16 )

m2( 14 )

P3


1 4

0

m3( 18 ) 1 8

1 16


Slide 44/44


9 16


P1

1 2

1


m1( 12 )

0

1 2

1 4


3 8


P2

1 m4( 16 )

m2( 14 )

P3


1 4

0

m3( 18 ) 1 8

1 16


Slide 44/44


9 16

15 16


P1

1 2

1


m1( 12 )

1 m4( 16 )


3 8 1 4

3 8

P2

1 2

0

0 1 16

m2( 14 )


1 8

1 16

0

P3

1 4

0

m3( 18 )



Slide 44/44


9 16

15 16

1


P1

1 2

1


m1( 12 )

1 m4( 16 )


3 8 1 4

3 8

P2

1 2

0

0 1 16

m2( 14 )


1 8

1 16

0

P3

1 4

0

m3( 18 )



Slide 44/44

Distributed Operating Systems: Theoretical Foundations

Distributed Operating Systems: Theoretical Foundations

Suggest Documents

• Advanced Operating Systems (Distributed Operating Systems ...

• Advanced Operating Systems (Distributed Operating Systems ...

Distributed Operating Systems

Distributed Operating Systems

Distributed Operating Systems

Distributed Operating Systems

Lecture 7: Distributed Operating Systems

Middleware and Distributed Systems Introduction - Operating Systems ...

Distributed Operating Systems - Department of Computer Science ...

b010evccfa-Distributed-Operating-Systems-Paperback-September.pdf

Survey: Distributed Operating Systems Introduction 1 Network ...

Causality Considerations in Distributed, Persistent Operating Systems

Middleware and Distributed Systems System Models - Operating ...

Generalized Load Sharing for Distributed Operating Systems

DISTRIBUTED OPERATING SYSTEMS Andrew S. Tanenbaum 1 ...

Service Modeling in Distributed Operating Systems - The Distributed ...

IV. Parallel Operating Systems - Distributed Systems Group / INESC ...

Theoretical and Historical Foundations

Theoretical Foundations of Leadership

Modern Operating Systems Modern Operating Systems

Operating Systems

Operating Systems

Operating Systems

Operating Systems