Optimistic David
Parallelization
F. Bacon,
University Robert
of Communicating
of California, E. Strom,
Berkeley
IBM
T.J.
dfb(lcs.berkeley.
edu,
Abstract
and
IBM
Watson
Research
non-interference
using the framework
for managing communications,
probability, anteed. cation
and aborting
program.
and process
PutLine,
of guarded
copying
Machinery.
without
fee
and
sequential
specific
e 1991
ACM
lines until
calls all out-
delays
are long
relative
such an implementation
X cannot
be parallelized
is inefficient.
by conventional
of the dependency code from
to the speed of com-
between
the previous
Process
static
analysis
each call and the
call.
processes
otherwise,
of tha
code.
If the continuation the process
makes fur-
is once again
split,
with the effect that a series of two-way calls is converted into a series of one-way sends. This transformation is termed call streaming. It is extremely valuable when bandwidth
of
this
mate[rial
is high but round-trip
delays are long.
Every optimistic protocol relies on a guess. In the case of call streaming, the guess is that the first thread’s call (1) will be serviced before the second thread’s call, and (2) will return auccesafully. Some systems, such as
ie
appear,
and
Association
or to republish,
notice
is given
the X-window
for Computing requires
formance
a fee
permission. 0-89791-390-6/91
return
calls to PutLine,
if not, they must be
all or part
its date
ie by permission
and/or
the line
X repeatedly
We contrast our
that the copiee are not made or distributed for advantage, the ACM oopyright notice and the
To copy
Y
threads. The first thread makes the call and waita for the return, while the second executes the continuation after the call, assuming that the call completed with a
are run in parallel;
publication
of whether
Process
putation,
return
run sequentially. granted provided direct commercial
Process
It takes a single line
an indication
successfully.
guar-
is an appli-
manager.
PutLine.
passing it successive output
because
has been extensively studied. Existing approaches use static analysis [2, 5] to prove that two sequential code segments do not interfere. If this can be proven,, then
copy
Y a window
X
We can improve performance by transforming process X into an optimistic execution containing two parallel
of how to parallelize
to
be statically
process
syshigh
put has been delivered or until it receives an unsuccessful return code. In a straightforward implementation, process X waits after each call for a return from Y before making the next call. If Y is remote, and commu-
ther
segments
it cannot
suppose
a service named
successful The problem
to prove.
even though
was displayed
thread creation, committing,
in the transformed
difficult
per call and returns
Introduction
of the
Center
For instance,
exports
approach with related work.
that
Center
However, many circumstances arise in distributed tems where non-interference can be guessed with
cornputatiow, in which each computation is tagged with the set of guesses on which it depends. We present an algorithm
title
Research
[email protected]
nication We analyze the problem
Permission
Watson
processes not known at compile-time, and because the non-determinism introduced by communication makes
Such a transformation is valuable in the case where S1 and SZ cannot be para.llelized by static analysis either because S2 reads a value from S1 or because S1 and S2 each interact with an external process. The optimistic transformation works under a weaker set of conditions: (1) if the value Sz reads from S1 can usually, but not always, be correctly guessed ahead of time, and (2) if S1 and S2 interact with an external process, conflicts which violate the ordering of S1 and % are possible but rare. Practical applications of this approach include executing the likely outcome of a test in pwdlel with making the test, and converting sequences of calls into streams of asynchronous sends.
b&h
T.J.
Processes
This technique is rarely applicable to a process in a distributed system because it may communicate with other
We present a transparent program transformation which converts a sequential execution of S1; SZ by a procesl; in a multiprocess environment into an optimistic parallel execution of SI and S2.
1
Sequential
/0004/0155...$1
.50
155
protocol,
by providing
trade
off correctness
an asynchronous
for per-
send-baaed
in-
terface, and requiring the user to handle asynchronous notification of errors. In contrast, in our approach we abort the second thread and undo all its side effects if ei-
ever, the processes are compiled and instantiated separately. This means that there is no a pm”ori ordering of events between processes. Instead, there is a partial or-
ther of our guesses was wrong.
This way our results
correct
or wrong,
der of events between processes which is dynamically determined by their communication. These assumptions
whether
we usually
we guess right
guess right,
we still
obtain
are
and provided
are typical
a performance
of distributed
systems.
improvement. This
paper
presents
an optimistic
protocol
Wlthln ments
for trans-
a process there is a sequence of program segS1; S2. Both S1 and S2 may include interac-
tions with other processes in the system. We assume the that there is some mechanism by which the compiler is told that it is desirable to parallelize S1 and S2. This mechanism could be programmer supplied pragmas, run-time profiling, static analysis, or a combina-
forming sequential code into parallel code. We use the call streaming transformation as a running example. However, our approach is applicable to parallelizing arbitrary sequential constructs S1; S2 in the case where
tion of these methods. ●
S2 depends upon S1. Static analysis will indicate that the two process fragments are not parallelizable,
either
because
for
some execution
path
We further assume that either S2 uses no values defined without in S1 (the only ‘guess” is that S1 terminates
S2
reads a value from S1, or else because S1 and S2 each interact with some external process, such as a server in a distributed ●
what
system.
with
S2), or that
of S1 with
collection
Transformations
[10] and is similar
ery [9]: we guess that
conflicts
of a process
occur.
will
not
When
Recov-
We illustrate
serial segments
a conflict
complex
occurs,
Formulation
and
to a database
that
a somewhat
than
more
we gave in the
server (process
by calling
process
Y),
which
actually
Z, the network
writes
filesystem
OK is usually
true.
Figure 1 illustrates the code of the original program segment. Figure 2 illustrates a typical execution in an unoptimized implementation, using a time-line diagram.
Hermes [8]. A feasible target environment is the Mach operating system [1], which supports both parallelism and distribution. program.
with
of call streaming
server. S2 then makes a direct call to Write to the filesystem server. The only value passed from S1 to S2 is the variable OK, which indicates whether the call successfully updated the database. The compiler has been told that it is desirable to parallelize S1 and S2, and
Example
as a sequential
our transformation
example
the value
We assume a high-level model in which independent sequential processes communicate by message passing or by making inter-process calls, as in CSP [3], Ada [12], or
Each process is written
el + ez.
introduction. Suppose that process X is a client containing two segments S1 and S2. S1 is an Update call
we roll back the processes that depend on the incorrect guess and restart them, running the computation in the original serial order.
2
is a
on Optimistic
to Optimistic
between
A trace
In this implementation,
events. Specifically, if el is an event in the execution of S1 and e2 is an event in the execution of S2, then we must preserve
work
program.
events.
by our mechanism. We preserve both the data values of each input and output event, and the ‘happens before” relationship [7], which we denote by ‘+n, between these
the guess is incorrect. If a bad guess is made, the program still runs correctly, but the average performance will be worse because of excessive rollbacks. on earlier
of the original
of observable
the observable events are the messages sent and received by all computations except those which are “aborted”
cution speed. The transformed computation completes earlier whenever the guess is correct, and later whenever
is baaed
S1 to S2.
Our transformation generates a parallel execution of S1 and S2. To be correct, our implementation must pre-
of success.
serve the traces
solution
has been told
in S1 and used in S2.
a high
Our program transformation is transparent. That is, the programmer makes no change to the source code, and sees no change in the program behavior, except for exe-
Our
the compiler
to guess for values defined
These values are said to be passed from
There is a way to guess the result probability
interfering
There First,
How-
156
are three
obstacles
to parallelizing
there is a data dependency
S1 and
on the boolean
S2:
value
x
/*
S1
*/
/*
S2
*/
OK = Update(Item,Value); if OK Urite(File,’’Di.d
Figurel:
OK between
Program
analysis
Segment
S1 and S2.
Z are independent
call
Y */
/*
call
z */
processes
another-and
X,
I!,
and
inter-procedural
calls is not possible.
on communication
Finally,
delays, X’s call to Z could
arrive before Y’s call to Z, which happens-before relationship.
clearly
Wewillovercome
trackingdependen-
these problemsby
violates
the
ciesdynamically, byexploiting theguessthat OKistrue, and by guessing that the messages arrive at Z in the correct order. X into
Given
these assumptions,
two parallel
threads:
Figure 2: No Call Streaming. Processes X, Y, and Z communicate via calls and returns (labeled C and R).
we can transform
one to execute
end of S1, and one to execute
up to the
Sa and its continuation.
3 Figure
3 illustrates
a successful
(non-aborting)
pendency tracking and will happens-before relationship logical
ordering
of events within
ordering
the dotted
between
line,
and
original
events.
The example above provides the motivation for optimistic parallelization and gives a rough idea of our approach. We now provide the theoretical framework in which we solve the problem and show how we use this framework to create an optimistic algorithm.
each thread,
(2) the
depicted
ordering
The execution
3.1
by
in Figure
2, and is therefore
Our
in Fig-
solution
is based on applying
the theory
of Op-
a correct
work for deriving optimistic distributed protocols from conservative protocols. In this framework, a process
If the call to Update we call this
(right-hand)
Transformations
timistic Transformations [10] to the case of time faults and value faults. Optimistic Transformations is a frame-
implementation.
at all,
Optimistic
between
the same events in the same order as the
execution
Execution
below. The by: (1) the
the two threads
(3) the causal
send events and receive ure 3 generates
be explained is determined
Optimistic
execu-
tion of the transformed process X. The two threiads are overlapped in real time, whereas they were sequential in Figure 2. The dotted arrow pointing backwards in real time indicates that the last event of the execution of S1 logically precedes the first event in the execution of S2; the labels in braces are used for dynamic de-
physical
z
Y
to Be Parallelizled
Second,
ofone
across the RPC
depending
it”);
/*
thread
The parallelized
returns
false
a value ~auli,
or fails
may choose at some point
to return
is aborted.
thread
will
also be aborted
to make a guess, and initiate
a computation baaed on this guess. For each guess, there is some stable predicate pn which confirms the guess —
and the parallelized
that is, if pn is true, then the computation Xn based on thh guess is correct. In this case, pn is said to be the
if an exe-
commit guard of z~. At any given time, the predicate ~ is either known to be true, known to be false, or in doubt. While pn is in doubt, there is a live computation to compute pn. Therefore, any commit guard predicate
cution such as the one in Figure 4 occurs. In this case, the call from X waa received by process Z before the call from Y. This contradicts the logical order in which the return from Z to Y precedes the return from Y to X, which precedes X’s call to Z, An examination of the diagram shows that there is a cycle in the ‘happens before” relationship. Such a cycle is a violation of
is eventually
causality,
(possibly
discovered
At any given time,
and is called a time fault.
157
empty)
to be either
a computation set of identifiers
true or false.
x~ is associated of in-doubt
with a
predicates
{PI, . . . . Pk). T~ commit
To prepare for rollback, a process may take a state checkpoint at each point prior to acquiring a new commit guard predicate. This approach is used in Time
This set is called the commit guard set. guard predicate of Xn is the conjunction
Ai=lpi” empty
If the commit guard set of a computation is then the commit guard predicate is vacuously
true—this
means that
the validity
Warp [4]. Alternatively, a process may take less frequent checkpoints, and log input messages, restoring the state
of the computation
by resuming
is independent of any guesses. Such a computation is said to be committed. A computation with a non-empty commit
guard
computation by rolling
set is called eventually
optimistic.
either
back to an earlier
For each in-doubt cess which
will
hood of pi. The predicate
pi, there must exist compute
the truth
is said to commit
from This
the checkpoint
approach
and replaying
is used in Optimistic
ery. A process may also rollback
optimistic
by inverting
logged Recovthe steps
of its computation. The particular technique used for rollback is a performance tuning decision and does not affect the correctness of the transformation.
or else aborts
state.
predicate eventually
An
commits
messages.
a pro-
or false-
when it re-
3.2
Algorithm
Description
solves to true, and abort when it resolves to false. When an in-doubt
predicate
computations are notified,
containing pi in their commit and must abort. If a predicate
putation’s
commit
pi is committed
guard
or aborted,
set commits,
is removed from the set. When becomes empty, the computation
all We now apply the above framework rithm for Optimistic Parallelization.
guard set in a com-
that
the commit commits.
a new commit
the union of its previous mit
guard
guards
track
set of the
guard
commit
When we run Sa without waiting for S1 to complete, are making two optimistic guesses:
set
set consisting
no value fault
●
no time-fault
●
of
ternal observers
to see possibly
fault, ... +
e. is in S2 and ek in S1. The existence chain implies that eo (in Sz ) precedes which contradicts the logical semantics
there
is
ek where of such a
ek (in S1 ), S1; S2.
These guesses constitute the commit guard predicate associated with the parallel execution of S2. If S1 completes without violating either of these guesses, then the predicate
is committe~
otherwise
it is aborted.
since we do not allow exincorrect
outputs.
When
back.
its exter-
aborts,
it discards
nal messages and rolls back.
External
input
messages
always have an empty aborted.
set, since they
are never
guard
el +
compu-
messages of rolling
a computation
For each time
commit
a computation commits, it releases its external and discards any state it created for purposes When
and passes
way,
Messages to or from processes that can not be rolled back, such as workstation displays, printers, and systems not participating in our protocol, are called ezternai messages. External messages sent by a guarded must be buffered,
occurs.
a causal chain of events eo +
tations.
computation
S1 terminates,
between
In this
dependencies
occurs:
we
the guessed values to S2, and
guard set and the com-
message.
the causal
an algo-
predicate guard
Each message carries with it a tag containing the commit guard set of the computation which sent the message. When a computation receives such a tagged message, it acquires
to create
If the predicate
is committed,
without
the possibility
commit
guard
execution
of rollback
predicates).
of S2 proceeds
(unless there are other
A control
message called
a
commit message is sent to other processes which depend upon the predicate which committed; this may enable them to commit aa well. If the predicate is aborted, S2 is rolled back and re-executed. Additionally, all its side effects are undone. A control message called an abo?’t message is sent to other processes which depend upon the predicate which aborted; this may cause further rollbacks.
To implement optimistic transformations, the compiler generates the guarded computations and the comput~ tions which check the guesses. The run-time environment encodes each guess, maintains a commit guard for each current in-doubt computation, propagates commit guards on messages exchanged between processes, dktributes control messages notifying other processes that a guess has committed or aborted, and maintains the ability to roll back state.
Provided that the overhead of our method is small and the relative frequency of aborts is sufficiently low, we will achieve a performance gain by parallelizing S1 and
S2. Let
158
So; S1; S2 be the
code of a sequential
process
X
x
) J
I
● .-’
xl
{}
)
,,.H
{}
{xl}
● ’
, : : :
: : :
C2{} —
: :::1
‘2 + E4
4
(}
,-: T
J--
: :;&
!r
B [xl)
c
{xl}
R
b
{X. 11
B
}
{Xli ):
,.”
R
I
: :
●
RI{)
A
A cl{
{Xl}
.,’
3
:
D)
c
Figure 3: Successful Optimistic Call Streaming. Commit guard sets are indicated next to the process time
Fimre 4: Aborted Optimistic call Streaming. ~1 is + {z 1}, indicating a-cycle in aborted at D since {xl}
lines and on the message identifiers,
the happens-before in Figure & The
that
and points
o:f in-
A, 1?, etc. The dashed line indicates
terest are labelled
A precedes Z1 in logical
relationship. The results are shown cycle includes C3, Ra, R1, and the
dashed line.
time.
The two threads the state, except
which is to be transformed by running S1 and Sa in parallel. We call the boundary between SO and S1 the fork point and the boundary between S1 and S2 the
will that
each run on a separate copy of in the right thread, we start by
setting each vi to the guessed value bi. If there is no antidependency between S1 and S2 — that is, there is
join point. Without loss of generality, does not itself contain a computation
assume that S1 which is being
no variable read by S1 and overwritten by Sz — then is unnecessary. This is the case in our call the COpy
parallelized.
passed from
streaming
Let {vi}
to S2 and let {bi} {hi}
depend
be the variables
be the values guessed for {vi}.
S1
After
only on the state at the fork point.
the fork,
parallel, When execution
reaches the fork point,
we shall label
*1.
The name Z1 will
(1) as an eveni!, Z1 denotes
the join
X have an empty commit of X is guarded by {x1}, commit
denote
asserting
that
the left
thread
will
guard
thread
in is
guard set. The right thread and when it sends C3, this
set is appended
receives C3 at point
point
to the message; when Z
B, its commit
guard
to {z1}. This guard is also sent with not change the guard at point C.
between computations S1 and Sz in process X; (2) as a commit guard predicate, z 1 denotes the commit guard predicate
of S1 and S2 proceed of Sz in the right
process receiving such messages will inherit the commit guard al. In Figure 3, Y, Z, and the left thread of
ecut es S1; the right thread executes S2. The left thread has the same commit guard set as So. The right thread has a commit guard set containing an additional predicate which
the execution
but the computation
guarded by xl. Any messages sent from the right thread will be annotated with the commit guard name z 1. Any
the transformed
program executes a fork, splitting the computation into two parallel threads. By convention, the left thread ex-
two things:
example.
The
set is changed
R3, but this does
complete
with no value fault and with no time fault. It is possible that the right thread contains further fork points, in which case there will be a right-branching forking struc-
When the left thread completes, a check is made that the expected values {bi} were assigned to {vi}. If the
ture. In general, c~ will name the n~~ join point event in X and will encode the predicate that the nth forked
The right thread and any causal dependents are rolled back immediately, and an abort message is propagated to any processes which depend upon z 1. In Figure 3,
computation fault.
will complete
without
a value fault
check fails, predicate
or time
x 1 is aborted
thb would mean rolling
159
due to a value fault.
Z back to point
B and aborting
Y 10
x ‘1
{)
~{xl}
C3{xl
.
m.,
: : .
cl{
:
4
~%-t “... -----
point D logically precedes the beginning of the right thread, we have xl -+ Zl, meaning that a time fault has occurred. Z must be rolled back to point A, Y must
: : : : .:
(xl ~ ...........
“-.. . ----(} -- . . . . . .
I
}
R2{ }
I
back to point
aborted,
aa is shown in Figure
There
\
i R3{ ) a T
is another
possibility,
case we can commit ... ......... ‘----------Figure
5: Abort
Z must re-read
the right
Normal Computation/Messages A~fied Compunction/Messages
we can abort
Control Messages
we will
and Re-execution.
Note
message C2 after rolling
thread
that
back.
namely
Zl,
message and learn
that
have detected
of the fork.
-
duration
The timeout
then Z1 can commit
is set at
guarantees
x 1 precedes
in which
case
a precedence
gm, in which
a cycle and can abort
case
z 1.
fault
due to non-termination
3.3
demonstrate
nates with
erwise live computation
set, then there
and
In any protocol
are
no uncommitted forks in its causal past, and xl can be committed. This is the case at point A in Figure 3: R1 has an empty guard set, and so the left thread terminates with an empty guard set and commits.
a cycle and aborts.
Liveness
at least one dependency which goes backwards in real time—that is, one resulting from a fork. If S1 termiguard
Z1 commits,
Figure 7 illustrates the laat case. We begin the same way, but process X later learns from process Z that Z1 precedes Z1. Thus process X detects
is
If the left thread terminates with no value fault, we check for a time fault by examining the commit guard set at the termination point. Every time fault includes
commit
Later
also.
that
predicate xl aborts in case S1 diverges. If the timeout expires, the left thread continues, but predicate x 1 is
an empty
receive
Figure 6 illustrates the first case. In this example, both process X and process Z have been optimistically parallelized. The return message to process X depends upon
process
of X.
aborted, and a value assumed.
S1’s commit
an earlier guess 21 in process Z. Process X informs other processes which depend upon guess x 1 that the
of implementation-defined
the beginning
that
Z1, (2) gm will abort, or (3) we will
Z1 guess precedes the Z1 guess. A timeout
of X is
that the event gm is in the past of S1, and that S1 can commit only if gm commits. But we do not yet know whether xl can commit. We will notify other processes that gm precedes xl by sending a new type of control message called a precedence message. Eventually, one of three things will happen: (1) gm will commit, in which
C3{}
—
thread
5.
guard does not include xl, but it does include some other guard gm from another process. NOW we know
*
RI{ }
B, and the right
be rolled
?..,
T
I
at Z before
propagates it to the left thread of X at point C. As a result, Z1 is in the causal past of point D. But since
xl)
i
4: because C3 arrived
C2, xl is in the guard set of R2. This causes xl to be added to the guard set of Y at point B, and RI then
..~ .,.
Xl} ‘, ... .... .. ..-.
B-
D in Figure
at point
R2{
R (Xl}
A (xl)
R3{Xl }
b
~.m . ... ... ... .. ..i...-’-”-{( D: .. . . ----
~
. .
}
.. : : ~**,,,......””**”’””* : : :
)
● ..m. mm. . . . . . . ..m...
:.......................
If the guard set at the end of the left thread is nonempty, there may be a time fault. Recall that the commit guard set includes all the uncommitted guards in the causal paat. If xl belongs to that set, then we know there is a time fault, because this means that S1’s execution causally follows the S2 thread. This is the case
that
employing
aborts,
it is not possible by rolling
it is necessary to deadlock back infinitely
to
an othoften.
Our algorithm is live provided: (1) we assume a finite number of processes in the system; (2) we impose a limit L specifying the maximum number of times the same
160
y
x ‘1
{}1
z
w
‘1
{)]
1(1
1{}
xl )
Figure
6: Successful
and returns.
Parallelization
At point
sent to X and Y, which its computation.
of Two Threads.
One-way
C the message PRECEDENCE(Z1 ,{z1}) both
commit
their
computations.
message sends (itfl is sent to Z.
Finally,
z.
(}
‘1
)
I
M1{z1)
r
:Zl) C*{}
: ,
•Z1 }
:
1!
+ :
M2{ X1)
● :
R2{
E
&--
is
then commits
()
... .. #
.“
i : : : :
to W, which
{)1
,.. : : : :
are used aa well as calls
D the message COMMIT(Z1)
X sends COMMIT($l)
x
B (Zl
and itf2)
At point
.“ ,.”
x1}
D {xl]
: ,.“
c
Fg
Fimre 7: Aborted ParaHelization of Two Threads. At point C {z1} --i {xl} and at point F {xl} left threads send PRECEDENCE messages to one another, and discover the cycle z 1 + Z1 + z 1. X to Z and W, and Z sends ABORT(Z1 ) to X and Y. As a result, Y rolls back to point A, W rolls and X and Z both abort their right-hand threacls. The cycle includes M2, R2, Ml, RI, and both
161
+ {z1}. The two sends ABORT(31 ) back to point ~, dashed lines.
computation this limit
will
be re-executed dependency in finite
be re-executed
is exceeded,
that
optimistically;
particular
pessimistically;
(3) commit,
messages are guaranteed
plies. If it is the receipt of a message, then the message is from a partial computation with a maximum of
when
computation
will
abort,
k optimistic computation
and
to be propagated
decisions. But we know that this partial is live, so we will produce the same results
as the pessimistic
time.
algorithm
by retrying
the computa-
tion from the read up to L times, and then blocking the receive operation, accepting only committed messages. Theorem
1 Subject
to the above conditions,
an opti-
❑
mistic parallelization of a distributed system will yield the same pantial tTaces as the pessimistic computation.
We first show that icate is committed Every
commit
fork.
The
timeout
guard
left
completes
will
or will
with
pletes without fault.
predicate
thread
period,
4
every individual commit guard predor aborted in a finite period of time: xi is associated
either
time
a value fault, a value fault,
complete
out.
before
If it times
ai is aborted. then there
with
4.1
a or
denoted
may be a time
processes
is a time fault, there exists a chain .zk . . . + such that Xi d ~j + Y, Z,...
Predicate
Zi will be aborted
ym will have already
fore the fork
~m+k, or ym ~
then
eventually
Zi will
y~+k.
receive
Xi.
successfully
4.1.1
be-
depend
upon
message.
If
y~+k.
Therefore
or else commit
a finite
number
Process
are The
be denoted
Xn; the guess that
no time
fault
and no value fault.
Index
messages will
retains
a current
of its relative
thread
state number
indez,
a pair
con-
n, and an inter-
val number Interval which is incremented every time a message is received which introduces a new dependency.
the
The process stores its largest as MaxThTead.
4.1.2 Similarly, if there is no time fault, then either antecedent will abort, causing X to be aborted nite time,
with
State
Each thread sisting
messages will propagate through a chain which never includes the same process more than once. Since there are finitely many processes, this chain must eventually terminate.
through
identifiers
such as X and Y.
If y~ haa aborted,
an abort
y~ ~ Yna+k, then any dependency message for y~ received by process Y can be immediately propagated to the processes which
notation:
case letters,
gave rise to that fork will also be denoted Zn. The guess is that the corresponding left thread will complete
in the
aborted
by upper
nth fork of process x will
of
chain have terminated or timed out and messages have traveled a number of hops equal to the length of this chain. If any process appears more than once, say y~ then either
Structures
We use the following
If it com-
If there
and y~+~,
Data
the
out
after all left threads
Implementation
Commit
Guard
assigned
thread
identifier
Set
some in fiEach
be propagated
thread
it depends
of processes and will eventually
maintains upon.
a set Guard
For the initial
reach process X.
Guard
We now show liveness of the transformation by induction on the number n of antecedent optimistic decisions prior to any particular optimistic decision of a computation:
for all processes upon instance, {x3, x5, y9}.
of guesses that
thread
of a process,
= 0.
The commit
guard set contains which
all uncommitted
guesses
the process depends.
For
decision must be a fork. in a finite time. If it has
A thread may depend upon many guesses by the same process, particularly if an optimization like call streaming is applied repeatedly. We would like to take advan-
we simply refuse to not committed after L repetitions, fork, guaranteeing that there will be no abort.
tage of the fact that a dependence upon X5 implies a dependence upon all previous committed threads of z,
Suppose that our transformation is live for n < k. Then if the k + 18* decision is a fork, the above reasoning ap-
and only store one guess per process in Guard. However, since Z3 may abort before or after 35 has been forked, this would make it impossible to decide how to
If n = O, then this optimistic It will either commit or abort
162
respond
to the message ABORT (Z3).
4.1.5
The solution is to represent the guess identifiers as lpairs, consisting of an incarnation number along with a thread identifier. The incarnation number is incremented every
as a pair
Xi,n where
and n the index
guess of each of these
number
incarnation
staA table records the index associated
a given event.
For example,
whether
of the thread.
the
aborted,
history
records
or is unknown.
Since most guesses are assumed to commit, thk should be implemented aa a sparse vector with missing elements
An with
aasumed to be commits.
The incarnation start tathe set of antecedents of if incarnation
processes,
it has committed,
i is the
incarnation
the start of each incarnation. ble can be used to determine
Histories
Each process maintains commit information about each of the processes with which it communicates. For each
time the process aborts one of its own threads, and the thread index is reset to the index of the abort thread. Each guess is represented
Commit
simplicity. history
2 of process
We write
entry
We will
History[z3]
treat
it as a vector
to denote
for
the commit
for guess 3 of process x.
X begins at event 3, then the guess X2,4 is known to be preceded by Xl,l, X1,2, and X2,3, but not by X1,3. Receipt of C2,3 can also be taken as an implicit abort of z 1,3. Because the incarnation start table recordk the abort
history
Events
of each process, only the most recent guess
from each process needs to be maintained guard
4.2
4.2.1
in the commit
Fork
set.
Incarnation numbers and their implementation scribed in [9]; we will describe the algorithm incarnation they should
A foFk operation begins a new thread. A timer is set for the compiler-determined timeout interval. The
are dewithout
numbers for simplicity of explication, be used in any real implementation.
state is copied into each new thread;
but
lnteFval 4.1.3
Rollback
Points
For each guess g E GuaFd, the thread
guess, denoted then
we roll
we retain
became
by RoUbacks[g].
the state
dependent
which
introduced
is just
the dependency
Zn, so the new state index
from
the receive operation
the creating
thread.
upon g.
For
each thread
Graph
we maintain
a commit
graph,
CDG,
GuaFd
as leaf nodes.
which
is a DAG When
knows the set of dependencies it broadcasts to its commit
with
a thread
before, which of out-of-order
will substantially message arrivals
4. Also, since the section of the process
the
of the state for the right-hand
dependency members
terminates
for the following
thread.
of and
thmead,
4.2.2
Send
graph. When a message is sent, the GuaFd is appended to it. The entire CDG could be appended as well. This would allow earlier commit processing by the receiver at the expense of larger messages and more processing per message arrival.
For instance, if GuaFd = {xl, y7} and the control message PREGEDENCE(=l, {Z5}) arrives, then the edge Z5 -+ xl is added to the CDG. If a cycle is detected in the CDG,
the
between the fork and join points is simply waiting for the return of the call, it is not necessary to make a copy
them and they are added by each receiver dependency
vector is copied
s[xn] is assigned
As an implementation issue, note that in the case of call streaming the fork can be performed afteT the call has
such as in Figure
Dependency
Rollback.
value (n, O).
been sent, rather than decrease the likelihood Commit
n of the new
is (n, O).
GuaTd and the guess Xn. The Rollbacks
preceding
Note that if the rollback point is in a previous thread, then this (and all intervening threads) are aborted.
4.1.4
is incremented
number
The History and (7DG are copied from the creating thread. The GuaFd is the union of the creating thread’s
that
If that guess is aborted,
before
MaxThFead
the relative
to determine
index
upon
back to the end of the interval
Rollbacks[g], that
first
is set to O and
by one and becomes thread
at which
then the compiler-
determined predictor function is applied the initial state of the right thread.
then some guess must be aborted.
163
4.2.3
Message
introduce
Arrival
new dependencies. Newguards
When a message arrives at a process, it is first checked that the message is not an orphan. The message is an
then
orphan
ing that
if ~~~@ard
Orphan
: History[g]
= Guar&
new dependencies
mented
– Guardx
have been
a new interval
# 0
introduced,
haa begun.
Interval
and for each g c Newgucmds,
the Guard and the CDG, Index = (Thread, Interval),
= aborted.
If
meanis incre-
g is added
to
and Rollbacks[g] is set to the new state index.
messages are discarded. Note that
From a correctness
point
introduces
of view, a message can be de-
if several messages are available, the fewest new dependencies
the one that
(that
is, the one
livered to an arbitrary thread of the process, but we will often have information available which allows us to
for whkh lNewguardsl is smallest) should be chosen. ThkI minimizes the chance that receiving the message
optimize
will lead to an aborted
the delivery
decision.
As an optimization, if we know message (does not cause the the ified), then we do not need to aborted, we just remove it from
If we have some knowledge that the message was intended for a particular thread, for example if this is the return of a call, we can check that the message does not depend upon some future thread. If it does, then we know that the future thread must be aborted, since a return
which
must be in its past has interacted
with
that
this is purely
an optimization
which
When
If the message passes these checks, to any thread type,
provided
the future
which that
is waiting
then
the message does not
of the thread.
Generally,
cases
we deliver
for a message
When a join committable, known.
it
of this
depend
First
on
the message should
= {Z5, 113}, and the process is running
ends and follows
there it,
is a
the two
occurs, guess x.+k may be immediately immediately abortable, or remain un-
the verifier
function
is sent, and thread
its left
is executed
to check that
n is consistent
with
the
the state
n + k. If the verifier message ABORT(Zn+~ )
n+ k and any dependent
threads
are
terminated. Otherwise, if Guar& = @then Zn+k does not depend upon any uncommitted guesses, and can therefore be committed. The control message COMMIT (~n+k) is sent,
and preferably to Z6. Delivering it to Z4 would cause an abort because the end of X4 precedes the beginning of X5, and X5 would therefore depend upon itself.
thread
n is terminated.
If Guar& # 0, then Zn depends upon guesses whose outcome is unknown. In this case, the message PRECEDENCE(Xn ,Guard) is sent, and thread n waits for other COMMIT, ABORT, and PRECEDENCE messages until it has enough information to decide to commit or
Receive
When a receive is executed one of the messages made available by the message arrival processing described above is received. If no message is available, the thread
abort
blocks. When
X
that was used to begin thread fails ~n+k is aborted: a control
thread plus three forks with guesses X4, X5, and x& then the message can only be delivered to threads X5 and x&
4.2.4
Xn of process
state at the end of thread
probably be delivered to the earliest possible thread. For instance, if a message arrives for process a with its Guard
a thread
thread %+k running that logically threads are said to execute a join.
waiting for the return and abort. For call streaming this optimization is straightforward since returns are easy to calls; in other
Join
allows
early detection of aborts: if we do not check the message, then the caller thread will eventually time out
match up with their corresponding this may be more difficult.
that a message is a read process state to be modput it in Rollbacks; if the CDG.
it. 4.2.5
Note
state.
xn.
So far we have described these control messages as simply being ‘sent”. They could either be sent by broada message m is received
by process
X,
cast or by explicitly
it may
164
sending
them
to processes
which
If an edge added to the CDG creates a cycle, then a time fault haa been detected. All threads in the cycle
are known to depend on the guard in question (this information could be recorded during message send processing). The former should work well in a local-area network quently.
where the threads are created relatively The latter would be more appropriate
are aborted.
infrein a
wide-area network or when the number of threads created is large; the control messages could then be piggybacked on data messages being For simplicity
of explication,
5
exchanged. we assume
that
control
messages are broadcast. The broadcast itself need not be reliable, since the loss of a message will not cause incorrect commits. However, the broadcast must ble live in the sense that if repeated broadcasts are made, a message will eventually be delivered.
Related
Jefferson
[4] first
a model
in which
Receive
individual analogous
Commit
When the message COMMIT(Zn) is updated so that History[xn] moved from
Any
threads
informed,
and CDG.
each Guard
of Xn are also removed also have committed. that
is received, = commit.
are waiting
and may commit
Any
the CDG
from
Receive
the history z~ is re-
to Time
in a Time
to complete
Warp
a join
are
is received,
of Guard
a = min{rlg
virtual
is the
earliest
state
will be sent. that
follow
Receive
order
or else the server
depends
in a cycle in the
since this would
upon
require
the sending
process
process and any other
senders.
In
contrast, the parallel simulation application for which Time Warp was originally designed can use the simula-
A T = Rol&zcks[g]} that
result
roll
relation.
dent of the receiving
interval
would
will
order of We only
to decide upon a receive time for the message indepen-
tion time for events to provide
z~,
and we roll back to that point, possibly aborting some guarded threads, including threads waiting to complete a join.
4.2.8
time
In a distributed or loosely coupled parallel system, processes are compiled and instantiated independently. It is not feasible to impose a total ordering upon the com-
Then c Abortset
each
the history
putations, Xn in the CDG.
simulation,
There is nothing of a process and
call a server then the server must process the calls in the
= abort. Each thread is if x~ c CDG, itmust be rolled
the members
or-
as well.
roll back if our execution
thread
denote
distributed
back. In our approach, there is only a partial events, which is determined during execution.
message for that
to
are totally
Most importantly however is that Time Warp imposes a single, totally ordered global virtual time throughout the system. Every event must be assigned both a send time and a receive time by the application. If two clients
as a result.
When the message ABORT(Z~) is updated so that HistoTy[z.]
Let Abortset
Warp
process runs sequentially. to our guessing the future
happens-before
back, and an abort
of time-faults
predecessors
since they must
Abort
then checked in turn:
the concept
the events of a system
executing it in parallel with the present. However, there is no reason in principle why this could not be applied
assigned
4.2.7
applied
dered. Like Jefferson’s Time Warp implementation, our approach is based on optimism and rollback. The two approaches differ in two ways. Firstly,
4.2.6
Work
the total
order.
The ParaTran system [11] transparently parallelizes Scheme code using Time Warp. It replaces variables by stacks to support rollback, and automatically generates the timestamps required by Time Warp based upon the sequential
Precedence
dependencies
of the program.
As in Time
When the message PREcEDENCE(z~,Gua?’d) is received, we set History [zfi] = unknown. The CDG of each
Warp, there is a global total ordering of events as opposed to our partial ordering. The ParaTran methodology applies to a single sequential program, not a network of multiple sequential processes. In addition, there
thread is updated: for each g E Guard, the edge g d is added if either g or z~ is a node of the CDG.
is no provision for increasing mistically guessed values.
x~
165
parallelism
by using opti-
UNIX
Concurrency control techniques, such as optimistic concurrency control[6], also guess that confllcts do not occur and abort However,
computations
when conflicts
there are two differences
ness requirement
for optimistic
development.
Useniz
Conference
In Proceedings (July
of the Summer
1986).
are detected.
between
[2] ALLEN,
the correct-
parallelization
lation
and the
R., AND KENNEDY, of Fortran
!l’hansactions
correctness requirement for concurrency control: (1) In concurrency control, all transactions must be serializable, whereas in optimistic parallelization, each individ-
[3] HOARE, C. Prentice-Hall 1985.
Automatic
to vector
on Programming
tems 9, 4 (October
ual process is separately serializable but the system as a whole is partially ordered; (2) In concurrency control, any serializable ordering is legal, whereas in optimistic parallelization only orderings which are consistent with
K.
programs
trans-
form.
Languages
ACM
and Sys-
1987).
Communicating International,
Sequential Processes. Englewood Cliffs, N.J.,
Virtual time. Transactions [4] JEFFERSON, D. R. on Programming Languages and Systems 7, 3 (July
S1; Sz are legal.
1985), 404-425. [5] KUCK,
6
Conclusions
Because
our
[6] KUNG,
are transparent,
Trans.
and familiar
statements,
which
ming languages explicitly
programming
idioms,
we can
managing
low-level
The
particular
application
of our relevant
works in which network latency ingly large fraction of the total tributed
our algorithm
B.,
graphs and compiler
AND
ROBINSON,
J.
for
concurrency
control.
Syst. 6, 2 (June clocks,
T.
On op-
ACM
1981), 213-226.
and the ordering
system.
Commun.
of
ACM
1978), 558-565.
mes: A Language tice Hall,
January
for Distributed
Computing.
Pren-
1991.
[9] STROM, R. E., AND YEMINI, recovery in distributed systems.
for
to
high-speed
call
will become an increastime to complete a dis-
has the transparency
S. A.
Optimistic
ACM Transactions
Systems 3, 3 (August
1985), 204–226.
[10] STROM, R. E., AND YEMINI, S. A. Synthesizing distributed and parallel programs through optimistic transformations. In Current Advances
net-
in Distributed Computing and Communications, Y. Yemini, Ed. Computer Science Press, Rockville,
computation.
In summary,
LEASURE,
Program-
primitives
algorithm to
D.,
[8] STROM, R. E., BACON, D. F., GOLDBERG, A., LOWRY, A., YELLIN, D., AND YEMINI, S. A. 17e~-
concurrency.
is particularly
T.,
Database
on Computer streaming
M. Dependence
methods
21, 7 (July
such as call
are easy to reason about.
need not contain
PADUA,
[7] LAMPORT, L. Time, events in a distributed
achieve the performance benefits of complex procotols such as call streaming without requiring programmers to rewrite code when moving from a centralized to a distributed system. We allow programmers to use straightforward
H.
timistic
S1 terminates.
transformations
R.,
optimization. In Proceedings of the 8th ACM Conference on Principles of Programming Languages (January 1981), ACM.
S1 II Sz. It is not necessary to prove statically that S1 and S2 do not interfere; our algorithm allows them to run in parallel and if a conflict is detected, aborts S2 it after
KUHN,
AND WOLFE,
We have given an algorithm which is capable of transparently transforming a serial program S1; Sz into
and re-runs
D.,
MD,
prop-
erties of conventional parallelizing optimization. However it is applicable to systems of distributed processes that are either not subject to static analysis or which are not considered parallelizable by static analysis.
1987, pp. 234-256.
M. [11] TINKER, P., AND KATZ, Parallel tion of sequential Scheme with ParaTran. and Functional Programming Conference
execuIn Lisp (1988),
pp. 28–39. [12] UNITED STATES DEPARTMENT OF DEFENSE. Reference Manual for the Ada Programming Language,
References [1] ACETTA, GOLUB, YOUNG,
ANSI/MIL-STD-1815-1983 M., D., M.
BARON,
R,,
BOLOSKY,
RASHID, R., TEVANIAN, A., Mach: A new kernel foundation
W., AND for
166
cd., February
1983.
