Specification and Implementation of Replicated List: The Jupiter ...

TECHNICAL REPORT, AUGUST 2017

1

Specification and Implementation of Replicated List: The Jupiter Protocol Revisited

arXiv:1708.04754v1 [cs.DC] 16 Aug 2017

Hengfeng Wei, Yu Huang, and Jian Lu Abstract—The replicated list object has been frequently used to model the core functionality (e.g., insertion, deletion, and read) of replicated collaborative text editing systems. In this paper we revisit the specification and implementation of a replicated list object, specifically the weak list specification proposed recently by Attiya et al. and the Jupiter protocol designed in the 1990s. We prove that Jupiter indeed satisfies the weak list specification, solving the conjecture of Attiya et al. To address the mismatch between the global property of weak list specification and the local views each replica maintains in the Jupiter protocol, we propose the CSS (Compact State-space) Jupiter protocol, which at a high level, maintains only a single novel n-ary ordered state-space for a client/server system with n clients. By contrast, the original Jupiter protocol, we call the CSCW protocol, needs to maintain 2n 2D state-spaces where replica states are dispersed. We first show that the CSS protocol and the CSCW protocol are equivalent in the sense that their behaviors are the same under the same schedule of operations/messages. Then, we prove that the CSS protocol satisfies the weak list specification. Index Terms—Collaborative editing systems, Replicated list object, Weak/Strong list specification, Jupiter, Operational transformation

F

1

I NTRODUCTION

C

O llaborative text editing systems, like Apache (Google) Wave [1], CodoxWord add-on to Microsoft Office [2], Gobby [3], or the Jupiter system [4] allows multiple users to concurrently edit the same document. To achieve high availability and low latency, these systems often replicate the document at several replicas. Furthermore, they often employ the optimistic replication techniques [5], allowing replicas to respond to user operations immediately without any communication with others, and then asynchronously communicate updates to others using message passing. The replicated list object has been frequently used to model the core functionality (e.g., insertion, deletion, and read) of replicated collaborative text editing systems [4], [6]. As a consequence, a number of protocols in the literature for collaborative text editing systems embody an implementation of a replicated list object as the core, using techniques such as OT (Operational Transformations) [4], [6], [7], [8], [9], [10], [11] or CRDT (Conflict-free Replicated Data Types) [12], [13], [14], [15], [16]. Among OT-based protocols, the Jupiter protocol designed in the 1990s [4] 1 is of particularly importance because it is foundational and original. For instance, the Jupiter protocol is the basis for Apache Wave [17]. Despite many protocols for a replicated list object mentioned above, there have been little formal specification of it. The most frequently used formal specification of a replicated list object is the convergence property [6], which requires the final list objects (representing documents) at all replicas be identical after executing the same collection of user operations. In 2016, Attiya et al. [16] proposed two new formal specifications of a replicated list object, namely the strong

•

The authors are with the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu 210023, China. E-mail: {hfwei, yuhuang, lj}@nju.edu.cn

1. The Jupiter system is a multimedia collaboration system. We focus on its collaborative text editing part in this paper.

list specification and the weak list specification. The strong list specification ensures that ordering relative to deleted elements hold even after the deletion, while the weak list specification only requires the ordering between elements that are not deleted to be consistent across the system. Attiya et al. [16] have proved that a variant of an existing RGA (Replicated Growable Array) protocol [15], which is a CRDT, satisfies the strong list specification. Meanwhile, they conjectured that the Jupiter protocol [4], [11], which uses the OT technique, satisfies the weak list specification. In this paper we revisit the specification and implementation of a replicated list object, specifically the weak list specification and the Jupiter protocol. One of the main contributions of this work is to prove that Jupiter indeed satisfies the weak list specification, solving the conjecture of [16]. There are two challenges to solve. The first one is that the system model and terminology used in prior work on Jupiter do not fit the framework for specifying replicated data types [18] adopted by Attiya et al [16]. Another major challenge lies in the mismatch between the global property of weak list specification and the local views each replica maintains in the Jupiter protocol. As mentioned above, the weak list specification specifies some global property over the states of the replicated list object at all time. However, in Jupiter (specifically, the CSCW Jupiter protocol [11] we consider in this work), these states are dispersed in multiple 2D state-spaces maintained locally at individual replicas. To address these issues, we make the following contributions: •

(Sections 2, 3, and 4) We review three formal specifications of the replicated list object (i.e., convergence property, strong list specification, and weak list specification) and the Jupiter protocol in the same formal framework for specifying replicated data types. In particular, we propose a framework for describing Jupiter (or generally, OT-based) protocols, which separates schedule of operations/messages from replica behaviors in an


•

•

•

2

execution. We compare the CSCW protocol (reviewed in Section 5) and the CSS protocol (discussed below) in this framework. (Section 6) We propose the CSS (Compact StateSpace) Jupiter protocol which uses a novel data structure called n-ary ordered state-space. Proposition 6.6 presents a remarkable advantage of the CSS protocol that at a high level, for a client/server system with n clients, the CSS protocol maintains only a single nary ordered state-space; hence the name “Compact”. Each replica (being the server or a client) behavior corresponds to a path going through the state-space. By contrast, the CSCW protocol needs to maintain 2n 2D state-spaces. Based on the proposition above, we conclude that the CSS protocol satisfies the convergence property. (Section 7) We prove that the CSS protocol is equivalent to the CSCW protocol in the sense that the behaviors of corresponding replicas in two protocols are the same under the same schedule of operations/messages. The CSCW protocol is slightly optimized in implementation by eliminating redundant OTs at clients, which has unfortunately obscured the similarities among clients. (Section 8) We prove that the CSS protocol satisfies the weak list specification (Section 8.2). Facilitated by the compactness of the CSS protocol, we are able to focus on a single n-ary ordered state-space which provides a global view of all possible replica states. By contrast, in the CSCW protocol, states are dispersed in multiple 2D state-spaces, leading to a mismatch between global properties such as the weak list specification. In this proof, we have studied several key properties about state pairs and paths in the n-ary ordered state-space. We also show that Jupiter does not satisfies the strong list specification (Section 8.1), by showing a counterexample adapted from an example scenario of [16].

R EPLICATED DATA T YPES

2 • • •

σ0 ∈ Σ is the initial state; E is a set of possible events; and ∆ : Σ × E → Σ is a (partial) transition function.

State transitions determined by ∆ are local steps of a replica, describing how it interacts with the following three kinds of events from users and other replicas. • do(op, v): a user invokes an operation op on the replicated object and immediately receives a response v from the replica. We leave the users unspecified and also say that the replica generates the operation op.; • send(m): the replica sends a message m to some replicas; and • receive(m): the replica receives a message m. Definition 2.2 (Protocol). A protocol is collection R of replicas. 2.1.2 Executions An execution of a protocol R is an interleaving of a sequence of events occurring at each replica in R. In other words,

Definition 2.3 (Execution). An execution of a protocol R is a (possibly infinite) sequence of events occurring at replicas in R.

We denote by R(e) the replica at which an event e occurs. For an execution α, we denote by e ≺α e0 (or e ≺ e0 if the context is clear) that e precedes e0 in α. We consider only well-formed executions. Definition 2.4 (Well-formed Execution). An execution α is well-formed if for every replica R = (Σ, σ0 , ∆) ∈ R, • the subsequence of events he1 , e2 , . . .i at R, denoted α|R , is well-formed, namely there is a sequence of states hσ1 , σ2 , . . .i, such that σi = ∆(σi−1 , ei ) for all i; and • every receive(m) event at R is preceded by a send(m) event in α. We are also interested in the behaviors (and states) of individual replicas in an execution.

In this section we describe the system model for highlyavailable replicated data stores and a formal framework for specifying replicated data types, following Burckhardt et al. [18] and Attiya et al. [16], [19].

Definition 2.5 (Replica Behavior). Let α be a well-formed execution. The behavior of replica R is a (possibly infinite) sequence of the form:

2.1

where he1 , e2 , . . .i = α|R and σi = ∆(σi−1 , ei ) for all i.

System Model

A highly-available replicated data store is modeled as a message-passing system, consisting of replicas that receive and respond to user operations on the replicated objects, and communicate updates to each other with messages. To be highly-available, the replicas are required to respond to user operations immediately, without any communication with other replicas, using the optimistic replication techniques [5]. This makes the system tolerant of high network latency and network partitions. 2.1.1 Replicas We model a replica as a state machine. Definition 2.1 (Replica). A replica is a state machine R = (Σ, σ0 , E, ∆), where • Σ is a set of states;

σ0 , e1 , σ1 , e2 , . . . A replica state in an execution can be represented by the event sequence the replica has processed. Definition 2.6 (Replica State Representation). Let α be a well-formed execution. A replica state σ ∈ Σ of replica R is represented by the events in a prefix of α|R , he1 , e2 , . . .i it has processed. Specifically, • The initial state is σ0 = ∅; and • We denote by σi (i ≥ 1) the replica state at the end of he1 , e2 , . . . , ei i. We define the happens-before relation [20] on events in an execution, which is used to derive a causal relation on user operations (i.e., do events) in Definition 4.1. hb

Definition 2.7 (Happens-before Relation ‘−→‘). Let α be an execution. Event e ∈ α happens before e0 ∈ α, denoted

TECHNICAL REPORT, AUGUST 2017 hbα

3

hb

e −−→ e0 (or e −→ e0 if the context is clear), if one of the following conditions holds: 0 0 • Thread of execution: R(e) = R(e ) ∧ e ≺α e . 0 • Message delivery: e = send(m) ∧ e = receive(m). hb 00 00 00 hb 0 • Transitivity: ∃e ∈ α : e −→ e ∧ e −→ e .

Definition 2.11 (Compliance). An execution α of a protocol R complies with an abstract execution A = (H, vis) if

The happens-before relation is a (strict) partial order which can be extended to total orders consistent with it. This is used to derive a total ordering relation between user operations in Definition 4.3.

Definition 2.12 (Satisfaction ‘|=‘). A protocol R satisfies a specification S , denoted R |= S , if every execution of R complies with some abstract execution in S .

tb

Definition 2.8 (Totally-before Relation ‘−→‘ Consistent hb with Happens-before Relation ‘−→‘). Let α be an executbα tion. A relation on events in α, denoted e −−→ e0 for e, e0 ∈ α tb 0 (or e − → e if the context is clear), is a totally-before relation hb consistent with the happens-before relation ‘−→‘ on events in α if and only if • •

tb

tb

Total: e − → e0 or e0 − → e for all e, e0 ∈ α. hb tb 0 Consistent: e −→ e =⇒ e − → e0 for all e, e0 ∈ α.

2.1.3

Network Model

To ensure that every operation eventually propagates to all the replicas, we require the network to be sufficiently connected, namely every pending message is eventually sent and every message sent is eventually delivered to the designated replicas. We refer the interested readers to [16] for a formal definition. 2.2

Specifying Replicated Objects

Following the framework by Burckhardt et al. [18] for specifying replicated data types, we specify the return value of an operation based on operations that are visible to it, and specify a replicated object by a set of abstract executions which record the operations invoked by the users and visibility relations between them. Definition 2.9 (Abstract Execution). An abstract execution is a pair A = (H, vis), where •

•

H is a sequence of do events, and vis ⊆ H × H is an acyclic visibility relation such that

1) if e1 precedes e2 in H and R(e1 ) = R(e2 ), then vis e1 −→ e2 ; vis 2) if e1 −→ e2 , then e1 precedes e2 in H ; and vis vis 3) vis is transitive: if e1 −→ e2 and e2 −→ e3 , then vis e1 −→ e3 .

An abstract execution A0 = (H 0 , vis0 ) is a prefix of abstract execution A = (H, vis) if: 1) H 0 is a prefix of H ; and 2) vis0 = vis ∩ (H 0 × H 0 ). Definition 2.10 (Specification). A specification S of a replicated object is a prefix-closed set of abstract executions, namely if A ∈ S , then A0 ∈ S for each prefix A0 of A. A protocol correctly implements a specification when the return values of operations that it produces in any (concrete) execution can be justified by some abstract execution allowed by the specification.

∀R ∈ R : H|R = α|do R,

where α|do R denotes the subsequence of do events by replica R in α.

3

R EPLICATED L IST S PECIFICATION

The replicated list object has been frequently used to model the core functinonality of replicated collaborative text editing systems [4], [6]. In this section we specify what a replicated list object is and review three formal specifications for it ever presented in the literature, namely convergence property [6], strong list specification [16], and weak list specification [16]. 3.1

Replicated List

A replicated list object supports three user operations: • I NS (a, p) for a ∈ U (U for some universe) and p ∈ N: inserts the element a at position p in the list (starting from 0) and returns the updated list. We assume that all inserted elements to be unique, which can be done by attaching replica identifiers and sequence numbers. As a consequence, there is a one-to-one correspondence between inserted elements and insert operations. • D EL (a, p) for a ∈ U and p ∈ N: deletes the element a at position p and returns the updated list 2 . • R EAD : returns the contents of the list. We denote by elems(A) the set of all elements inserted into the list in an abstract execution A = (H, vis): elems(A) = {a | do(I NS(a, ), ) ∈ H}. 3.2

Convergence Property

We adopt the convergence property given in [16] which requires that two R EAD operations that observe the same set of list updates (i.e., I NS and D EL operations) return the same response. 3 Definition 3.1 (Convergence Property ‘Acp ’). An abstract execution A = (H, vis) belongs to the convergence property Acp if and only if for every pair of R EAD events e1 = do(R EAD, w1 , a01 , . . . , am 1 ) and e2 = do( R EAD , w2 , a02 , . . . , an2 ) (where aji ∈ elems(A)), it holds that −1 vis−1 I NS,D EL (e1 ) = visI NS,D EL (e2 ) =⇒ w1 = w2 , where vis−1 I NS,D EL (e) denotes the set of list updates visible to e: 0 0 vis vis−1 I NS,D EL (e) = e ∈ H | e −→ e ∧ e0 = do(I NS, ) ∨ e0 = do(D EL, ) . 2. The operation D EL(a, p) includes as parameters both the element a and the position p, because on the one hand, OT is performed based on positions and on the other hand, condition 1a) of the strong/weak list specification defined in Definitions 3.2 and 3.3 explicitly refers to the element to be deleted. 3. This is called the strong convergence property in [21] and is stronger than the original convergence property considered in [6].


3.3

The Strong List Specification

The strong list specification ensures that orderings relative to deleted elements hold even after the deletion. Definition 3.2 (Strong List Specification ‘Astrong ’). An abstract execution A = (H, vis) belongs to the strong list specification Astrong if and only if there is a relation lo ⊆ elems(A) × elems(A), called the list order, such that: 1) Each event e = do(op, w) ∈ H returns a sequence of elements w = a0 . . . an−1 , where ai ∈ elems(A), such that a) w contains exactly the elements visible to e that have been inserted, but not deleted:

∀a.a ∈ w ⇐⇒ (do(I NS(a, ), ) ≤vis e) ¬(do(D EL(a, ), ) ≤vis e). b) The list order is consistent with the order of the elements:

∀i, j.(i < j) =⇒ (ai , aj ) ∈ lo. c) Elements are inserted at the specified position:

op = I NS(a, k) =⇒ a = amin{k,n−1} . 2) The list order lo is transitive, irreflexive, and total, and thus determines the order of all insert operations in the execution. 3.4

The Weak List Specification

The weak list specification only requires the ordering between elements that are not deleted to be consistent across the system. Definition 3.3 (Weak List Specification ‘Aweak ’). An abstract execution A = (H, vis) belongs to the weak list specification Aweak if and only if there is a relation lo ⊆ elems(A) × elems(A), called the list order, such that: 1) Condition 1) in Definition 3.2 is satisfied. 2) The list order lo is irreflexive and, for all events e = do(op, w) ∈ H , it is transitive and total on {a | a ∈ w}.

4

P RELIMINARIES : O PERATIONAL T RANSFORMA TION AND J UPITER P ROTOCOLS

In this section we first review the ideas of OT [6] and especially context-based OT [10]. Then, we propose a framework for describing Jupiter (or generally, OT-based) protocols, in which different protocols can be compared. The terminology has been adapted to fit the framework for specifying replicated data types reviewed in Section 2. 4.1

Ordering on User Operations

We start with the basic concept of ordering relations on user operations. In the following, let α be an execution and A = (H, vis) an abstract execution such that H is the subsequence of all do events (corresponding to user operations) in α. Definition 4.1 (Causal Relation ‘→’). Event e ∈ H is H causally before event e0 ∈ H , denoted e −→ e0 (or e → − e0 hbα if the context is clear), if and only if e −−→ e0 in α.

4

Definition 4.2 (Concurrent Relation ‘k’). Events e, e0 ∈ H are concurrent, denoted e kH e0 (or e k e0 if the context is H H clear), if and only if neither e −→ e0 nor e0 −→ e. Many major OT systems, including the Jupiter system, rely on a total ordering relation on user operations to control operation execution and transformation orders [10], [11]. Definition 4.3 (Total Ordering Relation ‘⇒‘). Event e ∈ H is before event e0 ∈ H in the total ordering relation ‘⇒‘, H denoted e = ⇒ e0 for e, e0 ∈ H (or e = ⇒ e0 if the context is tbα 0 clear), if and only if e −−→ e . tb

Remark. Since totally-before relation ‘− →‘ is defined to hb be consistent with happens-before relation ‘−→‘, which in turn determines the causal relation ‘→ − ‘, the total ordering relation ‘= ⇒‘ defined above is consistent with causal relation. 4.2

Operational Transformation

OT [6] is the core of many protocols for collaborative text editing systems, including the Jupiter system [4]. The basic idea of OT is to execute any local operation immediately and to transform a remote operation to take into account the concurrent operations previously executed. We use a scenario from [22] to illustrate the motivation for OT (Example 4.1) and the effects of it (Example 4.2). Example 4.1 (OT). Consider a collaborative text editing system consisting of two replicas R1 and R2 that initially hold the same list “efecte”; see Figure 1a. Suppose that user 1 invokes o1 = I NS(f, 1) at R1 and concurrently user 2 invokes o2 = D EL(e, 5) at R2 . After being executed locally, each operation is sent to the other replica. Without OT, the states of two replicas diverge, ending up with “effece” at R1 and “effect” at R2 . The OT of transforming o1 with o2 is expressed by the function o01 = OT (o1 , o2 ). We also write

(o01 , o02 ) = OT (o1 , o2 ) to denote both o01 = OT (o1 , o2 ) and o02 = OT (o2 , o1 ). To ensure the convergence property, the OT function is required to satisfy CP1 (Convergence Property 1) [6]: 4 Definition 4.4 (Convergence Property 1 (CP1)). Given two operations o1 and o2 defined on the state σ , if OT (o1 , o2 ) = (o01 , o02 ), then the following holds:

σ; o1 ; o02 = σ; o2 ; o01 , which means that the same state is obtained by applying o1 and o02 in sequence, and applying o2 and o01 in sequence, on the same initial state σ . An OT satisfying CP1 is often depicted as a “commutative diagram” such as Figure 1c. 4. There is another convergence property called CP2 [23] which, however, we are not concerned with in this work.


5

R1 efecte

R2 efecte

R1 efecte

R2 efecte

o1 = Ins(f, 1)

o2 = Del(e, 5)

o1 = Ins(f, 1)

o2 = Del(e, 5)

σ o1

effecte

efect

effecte

efect

o′2 = Del(e, 5)

o′1 = Ins(f, 1)

o′2 = Del(e, 6)

o′1 = Ins(f, 1)

effece

effect

effect

effect

(a) Motivation for OT.

(b) Effects of OT.

o2

o′ 2

′

o1

(c) OT satisfying CP1.

Fig. 1. Illustrations of OT.

Example 4.2 (OT Continued). As illustrated in Figure 1b, with the OT of OT o1 = I NS(f, 1), o2 = D EL( , 5) = o01 = I NS(f, 1), o02 = D EL( , 6) ,

o2 = D EL( , 5) is transformed to o02 = D EL( , 6) at R2 , taking into account the fact that o1 has inserted an element at position 1. Meanwhile, o1 remains the same. As a consequence, two replicas converge to the same list “effect”. The OT functions satisfying CP1 for collaborative text editing systems (i.e., for a replicated list object) can be found in [6], [22]. 4.3

Context-based Operational Transformation

CP1 for OT (o1 , o2 ) requires that o1 and o2 be defined on the same state σ , which is called the context of them [10]. To be more precise, we first describe the representation of replica states and then define the concept of operation contexts. There are two kinds of operations in an execution of an OT-based protocol: the original user operations and the transformed ones by OTs. Thus, the replica state defined in Definition 2.6 can be represented as follows. Definition 4.5 (Replica State Representation: Definition 2.6 Revisited). Let α be a well-formed execution. A replica state σ ∈ Σ of replica R is represented by the prefix subsequence of operations of α|R , he1 , e2 , . . .i it has processed. Specifically, • The initial state is σ0 = ∅; and • After processing the (i + 1)-st operation oi+1 , the new replica state is σi+1 = σi ∪ {org(oi+1 )} 5 , where org(o) denotes the original form of o. Each operation is associated with a context which determines with which and how it is transformed. Definition 4.6 (Operation Context). Let α be an execution. The context of an operation o in α, denoted C(o), is the replica state on which o is defined: 1) for an original operation o, C(o) = σ , where σ is the replica state from which o is generated; 5. We use the set-theoretic notations here, because the ordering of the operations in σ does not matter.

2) for a transformed operation o0 = OT (o, ox ), C(o0 ) = C(o) ∪ {org(ox )}.

Following [11], we denote by o1 {o2 } that o2 is in the context of o1 . Thus, the OT of transforming o1 with o2 can also be written as o1 {o2 } = OT (o1 , o2 ). Remark. By definition, operations in a context are all original (user) operations. Causal and total ordering relations are also defined on original operations. For notational convenience, we assume an automatic conversion, from o to org(o), whenever necessary. 4.4

A Framework for Jupiter Protocols

Jupiter adopts a centralized client/server architecture. Collaborating clients are connected to a central server via TCP connections. All communication channels, from clients to the server or in the other direction, are FIFO. For convenience, we assume that the server does not generate operations [11], [16]. It mainly serializes operations and propagates them from one client to others. This work contains two Jupiter protocols. Section 5 reviews the existing Jupiter protocol described in the CSCW paper [11] that we call the CSCW Jupiter protocol. Section 6 proposes a new Jupiter protocol called the CSS Jupiter protocol. To facilitate the comparison of these two protocols later on, we describe them in a uniform framework which separates the schedule of messages/operations from replica behaviors (Definition 2.5) in an execution of a protocol. A schedule is concerned with only the temporal aspect of an execution. Formally, Definition 4.7 (Schedule). Let α be an execution. The schedule of α is α with the arguments of each event erased. A schedule is depicted as a “space-time diagram” such as Figure 2. In both Jupiter protocols, we will see that it is the causal and total orderings (Definitions 4.1 and 4.3) on user operations in a schedule that determine which OTs are performed and in what order, which in turn, together with the OT functions, determine the behavior of each replica.

5

T HE CSCW P ROTOCOL

The CSCW Jupiter protocol [11] is a complete multi-client description of the two-way synchronization protocol first


S

C1

6

C2

C3

Remote Processing

Suppose client ci receives a remote operation o{L1 } from the server. It

1 1

2 3 2 1 1 3

5.2.3

2

1) searches its state-space DSSci for the state σ that matches the context of o{L1 }, and saves o{L1 } at σ along the global dimension; 2) transforms o{L1 } with a sequence of operations, denoted L2 , consisting of operations along the local dimension from σ to the final state of DSSci ; this transformation can be expressed as

2 3

OT (o{L1 }, L2 ) = (o{L1 , L2 }, L2 {o});

3

3) executes o{L1 , L2 }. Fig. 2. An schedule consisting of 3 pairwise concurrent operations.

5.3

Properties of the CSCW Protocol

proposed in the Jupiter paper [4]. The original protocol has only illustrated the single-client case.

The following lemma characterizes the operation sequence L1 in server processing.

5.1

Lemma 5.1 (CSCW OT Sequence [11]). In the CSCW protocol, the operation sequence L1 with which an operation o transforms at the server

Data Structure: 2D State-spaces

Suppose that the client/server system includes n clients. The CSCW protocol maintains 2n(= n + n) 2D state-spaces, each of which consists of a local dimension and a global dimension. Each client ci maintains a state-space, denoted DSSci , with the local dimension for operations generated by the client and the global dimension for operations generated by other clients. The server maintains n 2D state-spaces, one for each client. The state-space for client ci , denoted DSSsi , consists of the local dimension for operations from client ci and the global dimension for operations from other clients.

• •

6

is empty; or consists of all operations that are totally ordered before (Definition 4.3) and concurrent with (Definition 4.2) o. Furthermore, all operations in L1 are totally ordered.

J UPITER M ADE C OMPACT: T HE CSS P ROTO -

COL

5.2

Operation Processing

Following [11], we describe the CSCW protocol in three parts. 5.2.1 Local Processing When a client generates an operation o, it 1) executes o immediately and locally; 2) saves o along the local dimension of its 2D state-space; and then 3) propagates o to the server. 5.2.2 Server Processing Suppose the server receives an operation o from client ci . It 1) searches the state-space DSSsi for the state σ that matches the context of o, and saves o at σ along the local dimension; 2) transforms o with a sequence of operations, denoted L1 , consisting of operations along the global dimension from σ to the final state of DSSsi ; this transformation can be expressed as

OT (o, L1 ) = (o{L1 }, L1 {o}); 3) executes o{L1 }; 6 4) for each j 6= i, saves o{L1 } at the end of the global dimension of DSSsj ; 5) propagates o{L1 } to other clients. 6. We assume that the server also holds a replicated list object.

We now propose an alternative Jupiter protocol, called the CSS (Compact State-Space) protocol. We show, in Section 6.4 that the CSS protocol is compact in the sense that it essentially maintains only one state-space which compactly captures the information about OTs. This, together with CP1 (Definition 4.4) implies that the CSS protocol satisfies the convergence property. Section 7 proves that the CSS protocol is equivalent to the CSCW protocol, which is slightly optimized in implementation by eliminating redundant OTs at clients.

6.1

Data Structure: n-ary Ordered State-spaces

For a client/server system including n clients, the CSS protocol maintains (1 + n) n-ary ordered state-spaces, one per replica (CSSs for the server and CSSci for client ci ). A state σ is represented by the set of operations the replica has already processed. Initially, all state-spaces contains only the initial state σ0 = {0}. A transition between two states is labeled with the (original or transformed) operation involved. Unlike in a 2D state-space, a state in an n-ary statespace may have more than 2 child states. Furthermore, these child states (or the transitions to them) are totally ordered, according to the total order among operations established by the server.


7

Algorithm 1 OTs along the leftmost transitions in the CSS protocol; see Examples 6.1 and 6.2.

3

σ1 2

1

3

4

σ12 3

4

4

6.2

2

L{o}[k] in

their appropriate order

L

1

2

2: o{L} ← o 3: for k ← 1 . . m do 4: (o{L}, L{o}[k]) ← OT (o{L}, ok ) 5: add the transitions labeled with o{L} and

1

L ← ho1 , o2 , . . . , om i . operation sequence along the leftmost transitions; it may be empty

2

1:

σ0

σ124 3

Uniform Operation Processing

In the CSS protocol, the local processing, the server processing, and the remote processing proceed in a uniform way. Specifically, when a replica “meets” an operation o (it can be the case of a client generating o, the server receiving o from some client or a client receiving o from the server), it 1) searches the state-space for the state σ that matches the context of o, and saves o at σ along the transition of the right order among all the transitions from σ ; 2) transforms o with a sequence of operations, denoted L, consisting of operations along the leftmost (for the moment) transitions from σ to the final state of the state-space; this transformation can be expressed as

OT (o, L) = (o{L}, L{o}); This is done by repeatedly applying OTs along L as shown in Algorithm 1. The key difference from the OTs in the CSCW protocol is that new transitions generated at each step of OT should be arranged in their appropriate order among all the transitions from the same state in the n-ary ordered state-space (Line 5); see Example 6.1 below. 3) executes o{L}; 4) (Optional) in the case of the server receiving o from a client, the server then propagates o to other clients; 7 in the case of a client generating o, the client then propagates o to the server. Remark. In the case of a client generating an operation, step 2) above can be omitted. Thus, the local processing of the CSS protocol is the same with that of the CSCW protocol. Example 6.1 (OTs in the CSS protocol (Algorithm 1)). Consider a schedule consisting of four operations with causal relations: o3 k (o1 k o2 ) → o4 , and total ordering relations: o1 ⇒ o2 ⇒ o3 ⇒ o4 . Figure 3 shows the n-ary state-space (indicated by the solid lines) at some client when it receives a remote operation o3 (indicated by the dashed line at the top). According to Algorithm 1, the operation sequence L with which o3 needs to transform is L = ho1 , o2 , o4 i, corresponding to the leftmost transitions from the matching state σ0 to the final state of the present state-space (Line 1). The transformation procedure is iterative (Line 3). First, OT (o3 , o1 ) is performed (Line 4), obtaining a new transition labeled with o3 {1} from state σ1 . Because o2 ⇒ o3 ⇒ o4 , the new transition is added between those labeled with o2 {1} 7. Unlike in the CSCW protocol, the operations redirected by the server in the CSS protocol are the original ones.

Fig. 3. Illustration of OTs in the CSS protocol described in Algorithm 1.

and o3 {1} respectively (Line 5). Then, OT (o3 {1}, o2 {1}) is performed, and the new transition labeled with o3 {1, 2} is added to the left of that labeled with o4 {1, 2}. Finally, OT (o3 {1, 2}, o4 {1, 2}) is performed. In the examples and proofs below, we often need to keep track of the construction of the state-spaces of Jupiter protocols. Therefore, for each state-space, we introduce a superscript k to refer to the state-space after the k -th step, counting from 0. For instance, the state-space CSSci after the k -th step maintained by client ci in the CSS protocol is denoted by CSSkci . Similarly, the state-space DSSsi after the k -th step maintained by the server for client ci in the CSCW protocol is denoted by DSSksi . We use the notations without superscripts to denote the state-spaces under construction, namely the state-spaces at present. Example 6.2 (Illustration of the CSS Protocol). Figure 4 illustrates the CSS protocol under the schedule of Figure 2. In the following, we elaborate on the behavior of client c3 ; see Figure 4d. Before receiving any remote operations from the server, client c3 first generates an operation o3 . Client c3 executes o3 immediately, stores it at state σ0 = {0} along a new transition labeled with the operation o3 , and propagates it to the server. Next, client c3 receives operation o1 from the server. The context of o1 is σ0 , the same with that of o3 . Furthermore, o1 is not totally ordered before o3 (since o1 is not aware of o3 at the server). Therefore, o1 is saved at σ0 along the transition which is totally ordered before (i.e., on the left of in figure) the transition from σ0 to σ3 = {3}. Then o1 is transformed against o3 , represented as OT (o1 , o3 ) = (o1 {o3 }, o3 {o1 }). The transformed operation o1 {o3 } is executed at client c3 , leading to the state σ13 = {1, 3}. Finally, client c3 receives operation o2 from the server. First, we note that in the CSS protocol, the server propagates the original operation o2 instead of the transformed one o2 {o1 }. The context of o2 is {0} (by contrast, the context of o2 {o1 } is {1}). Furthermore, o2 is totally ordered between o1 and o3 (since o2 is aware of o1 but not aware of o3 at the server). Therefore, o2 is saved at σ0 along the transition which is totally ordered between the transitions to σ1 and to


0

CSS1s

0

0

CSS1c1

0

0

CSS0c2

0

0

CSS1c2

0

CSS0c3 CSS1c3 3

2

CSS0c1

1

1

CSS0s

8

2

1

2

3

1

0

CSS2c1

0

3

2 2

13

1

2 2

1

12

13

13

2

(c) Client c2 (CSSc2 ).

1

2 2

1

3

3

12

13 3

3

123

123

(b) Client c1 (CSSc1 ).

3

3

3

3

2

1

3

3

12

123

(a) Server s (CSSs ).

1

1

1

12

2 2

1

1

1

CSS3c3 3

2

3

3

13

0

3

2

2 1

12

CSS3c2

0

3

2

1

3

2

CSS3c1

0

3

2

1

0

2 3

12

12

CSS3s

1

1

1

1

12

2

1

2 2

1

1

1

1

2 2

2

1

2

1

2

1

CSS2c3

0

2

1

0

CSS2c2 1

CSS2s

123

(d) Client c3 (CSSc3 ).

Fig. 4. Illustration of the CSS protocol under the schedule in Figure 2. The order in which each replica constructs its state-space is indicated as a path; see the thick lines.

σ3 . According to Algorithm 1, o2 is then transformed with the operation sequence consisting of o1 and o3 {o1 }, producing the transformed operation o2 {o1 , o3 }. The transformed operation o2 {o1 , o3 } is executed at client c3 , leading to the state σ123 = {1, 2, 3}. 6.3

Properties of the CSS Protocol

We proceed to prove several properties of the CSS protocol, which will be used to establish the compactness of the CSS protocol (Section 6.4) and the equivalence of the CSS protocol and the CSCWprotocol (Section 7). We begins with two properties about n-ary ordered statespaces. Lemma 6.1 (n-ary Transitions). Each state in an n-ary ordered state-space has at most n child states. Proof: This lemma holds because the operations associated with the transitions from the same state are pairwise concurrent, and there are n clients in total. Lemma 6.2 (n-ary Order). Suppose that state σ in an n-ary state-space has k ≥ 3 child states, denoted σ1 , σ2 , . . . , σk in total order, with transitions (or operations associated) o1 , o2 , . . . , ok , respectively. There exist no scenarios in the construction of the state-space where s has k 0 ≥ 2 child states, none of which, however, is o1 . Proof: This is due to the client/server model of the system and the FIFO communication between clients and the server. The “n-ary order” lemma, together with the “leftmost transition“ rule used in the CSS protocol, implies that all OTs defined on the same state involve the leftmost (i.e., smallest in total order) operation in all transitions from that state. We then consider paths through the state-space. Lemma 6.3 (Simple Path). Let Pσ1 σ2 be a path of length l (l = 0 if σ1 = σ2 ) from state σ1 to state σ2 in the n-ary

ordered state-space CSSs at the server. Then, there are no duplicate operations (when converted to their original forms) along the path Pσ1 σ2 . Formally, the cardinality of the set of operations Oσ1 σ2 along the path Pσ1 σ2 is l. We call such a path a simple path. Proof: Due to the specific structure of OTs (Figure 1c), all the transitions associated with the same operation are “parallel” in CSSs . They cannot be in the same path. Finally, we characterize the operation sequence with which a new operation is transformed at the server in the CSS protocol. According to Algorithm 1, it is determined by the leftmost transitions from the matching state. Lemma 6.4 (Leftmost Transitions). Let O = ho1 , o2 , . . . , om i be the operation sequence the server has processed in total order ‘⇒’. For any state σ in CSSs , the path P along the leftmost transitions from it to the final state • •

is empty, if σ is the final state of CSSs ; or, consists of the operations of O \ σ in their total order.

Proof: By mathematical induction on the operation sequence O the server processes. Base Case: O = hi. CSSs contains only the initial state σ0 = {0}, and the leftmost transition from σ0 is empty. Inductive Hypothesis: Suppose that the lemma holds for Ok = ho1 , o2 , . . . , ok i. Inductive Step: Consider Ok+1 = ho1 , o2 , . . . , ok , ok+1 i. Suppose that the matching state of operation ok+1 is σ . We distinguish between σ being the final state of CSSks , denoted σk , or not. C ASE 1: σ = σk . According to Algorithm 1, the statespace CSSk+1 is obtained by extending CSSks with a new s transition from σk labeled with ok+1 . Thus, each leftmost transitions path in CSSks is extended by the transition labeled with ok+1 , meeting the second condition of the lemma in CSSk+1 . In addition, the leftmost transition from the final s state of CSSk+1 is empty, meeting the first condition. s


9

k

Ss

CS

the following proposition shows that all the (n + 1) statespaces are the same. Therefore, the essence of the CSS protocol can be captured by the single n-ary ordered statespace at the server. As an important corollary, the CSS protocol satisfies the convergence property (Definition 3.1).

=

ox

Lk

σ

Proposition 6.6 (1 + n ⇒ 1). In the CSS protocol, the replicas (being either the server or a client) that have processed the same set of operations have the same n-ary ordered state-space.

Ok

ok+1

\ =

ox

O

oy

k

\

σ

σ ok+

L

k

1

σk ok 1

+

oy

σk+1

Fig. 5. Illustration of C ASE 2 (σ 6= σk ) of the proof for Lemma 6.4.

C ASE 2: σ 6= σk . According to Algorithm 1, the server transforms ok+1 with the operation sequence, denoted Lk , along the leftmost transitions from σ to the final state σk with final state of CSSks , obtaining the state-space CSSk+1 s σk+1 . By inductive hypothesis, Lk consists of the operations in Ok \ σ in their total order. To prove that the lemma holds for CSSk+1 , we need to check that (See Figure 5): s 1) It holds for old states in CSSks . Each leftmost transitions path from states in CSSks is extended by the transition labeled with ok+1 , meeting the second condition of the lemma in CSSk+1 . s 2) It holds for new states in CSSk+1 \ CSSks . This is because s these new states form a path along which the corresponding operation sequence is exactly Lk . Lemma 6.4 immediately implies that Lemma 6.5 (CSS OT Sequence). In the CSS protocol, the operation sequence L with which an operation o transforms at the server • is empty; or, • consists of all operations that are totally ordered before (Definition 4.3) and concurrent with (Definition 4.2) o. Furthermore, all operations in L are totally ordered. Proof: We show that if L is not empty, then 1) All operations in L are totally ordered before o. This holds because operation o is the last one in the total order ‘⇒‘. 2) All operations in L are concurrent with o. By contradiction. Suppose that o0 ∈ L is not concurrent with o. Then it must be the case that o0 → − o and thus o0 ∈ / L. 3) L consists of all the operations satisfying 2) and 3) and all operations in L are totally ordered. This is due to Lemma 6.4.

Remark. In the CSS protocol, all replicas will eventually receive and process all operations. Therefore, Proposition 6.6 implies that all the (n + 1) state-spaces maintained by the server and n clients are eventually the same, though the construction orders may differ replica by replica. Example 6.3 (Illustration of Proposition 6.6). As shown in Figure 4, the final n-ary ordered state-spaces at all replicas are the same. Moreover, Figure 4 indicates the order in which each replica constructs the same state-space, by depicting their behaviors as different paths going through it. Proof: By mathematical induction on the number of operations in the Jupiter system. Because all operations are serialized at the server, we proceed by mathematical induction on the operation sequence O = ho1 , o2 , . . . , om i the server processes in total order. Base Case. O = ho1 i. There is only one operation in the Jupiter system. When all replicas have eventually processed this operation, they obviously have the same state-space. Formally, CSS1s = CSS1ci , ∀1 ≤ i ≤ n. Inductive Hypothesis. O = ho1 , o2 , . . . , ok i. Suppose that when all replicas have eventually processed all the k operations, they have the same state-space. Formally, CSSks = CSSkci ,

∀1 ≤ i ≤ n.

Inductive Step. O = ho1 , o2 , . . . , ok+1 i. Suppose that the (k+1)-st operation ok+1 processed at the server is generated by client cj . We shall prove that for any client ci , when it has eventually processed all these (k + 1) operations, it has the same state-space as the server. Formally, CSSk+1 = CSSk+1 s ci ,

∀1 ≤ i ≤ n.

In the following, we distinguish client cj that generates ok+1 from other clients. C ASE 1: i 6= j . The state-space CSSk+1 at the server is s obtained by applying the (k + 1)-st operation ok+1 to CSSks , denoted by CSSk+1 = ok+1 ⊗ CSSks . s

Since the communication is FIFO, the state-space CSSk+1 at ci client ci is obtained by applying the remote operation ok+1 to CSSkci , denoted by CSSk+1 = ok+1 ⊗ CSSkci . ci According to the inductive hypothesis,

6.4 The CSS Protocol is Compact and Satisfies the Convergence Property Although the CSS protocol maintains (n + 1) n-ary ordered state-spaces for a client/server system including n clients,

CSSks = CSSkci ,

i 6= j.

Therefore, we have CSSk+1 = CSSk+1 s ci .


10

C ASE 2: i = j . Now we consider client cj that generates cj c c cj the operation ok+1 . Let σk+1 , ho1j , o2j , . . . , ok+1 i, 8 a pers mutation of σk+1 , O = ho1 , o2 , . . . , ok+1 i, be the operation sequence executed at client cj . The operation ok+1 may not be the last one executed at client cj . Instead, suppose ok+1 is the l-th (1 ≤ l ≤ k + 1) operation executed at client cj , c namely ol j ≡ ok+1 . cj c into three The operation ol j splits the sequence σk+1 cj parts: the subsequence σ1,l−1 consisting of the first (l − 1) c operations, the subsequence σl,lj containing the operation cj cj ol ≡ ok+1 only, and the subsequence σl+1,k+1 consisting of the last (k − l + 1) operations. We formally denote this by c

c

c

j j j . ◦ ok+1 ◦ σl+1,k+1 = σ1,l−1 σk+1

c

j We remark that all operations in σl+1,k+1 are concurrent with ok+1 , because they are generated by other clients than cj before ok+1 reaches these clients, and ok+1 is generated before they reach ok+1 ’s local site (i.e., cj ). Furthermore, due cj to the FIFO communication, the operations in σl+1,k+1 are totally ordered. cj cj cj c c c , ol+1 , . . . , ok+1 i be the opLet σkj , ho1j , o2j , . . . , ol−1 cj eration sequence obtained by deleting ol (i.e., ok+1 ) from cj σk+1 , namely

c

c

c

j j . ◦ σl+1,k+1 σkj = σ1,l−1

with which ok+1 transforms consists of exactly the (possibly cj : transformed) operations in σl+1,k+1 c

c

c

c

CSSk+1 cj

=

cj σk+1

⊗

CSS0cj ,

is the same with the state-space CSSk+1 at the server cons structed by applying the (k + 1)-st operation ok+1 to CSSks , namely CSSk+1 = ok+1 ⊗ CSSks . s According to the inductive hypothesis, CSSks would be the same with the state space CSSkcj constructed at client cj if it c had processed σkj in sequence. Formally, c

CSSks = CSSkcj (, σkj ⊗ CSS0cj ). Therefore, it suffices to prove that the state-space CSSk+1 at cj client cj constructed by executing c

c

c

j j j σk+1 = σ1,l−1 ◦ ok+1 ◦ σl+1,k+1

c

c

c

c

c

c

c

c

c

c

We now consider the construction of CSSk+1 by executcj cj in three stages, corresponding to scenario (1). 1) At ing σk+1 the beginning, it grows as CSSkcj does when executing the cj . 2) Next, the operation ok+1 common subsequence σ1,l−1 is generated at client cj . According to (the local processing mechanism of) the CSS protocol, the state-space grows by cj along a new transition. saving ok+1 at the final state σ1,l−1 cj 3) Then, the sequence σl+1,k+1 of remote operations are cj processed at client cj . Each operation in σl+1,k+1 , when executed in sequence, not only “simulates” the growth of CSSkcj , but also completes one step of the iterative operational transformations of ok+1 with the sequence L mentioned above when applying ok+1 to CSSkcj ; We take cj as an example the case of the first operation ol+1 . After transforming with some subsequence of operations (which cj cj may be empty) in σ1,l−1 , operation ol+1 is transformed as cj cj cj cj ol+1 {σ1,l−1 }. At that time, ol+1 {σ1,l−1 } is then transformed cj with ok+1 {σ1,l−1 }, which is also performed when applying ok+1 to CSSkcj : c

=

c

c

j j j }) }, ok+1 {σ1,l−1 {σ1,l−1 OT (ol+1

cj cj ol+1 {σ1,l−1

cj cj ◦ ok+1 }, ok+1 {σ1,l−1 ◦ ol+1 } . c

j in sequence, we obtain As it goes on, after executing σk+1 a state-space same with that obtained by applying ok+1 to CSSkcj . Proposition 6.6, together with the fact that OTs for list satisfy CP1 (Definition 4.4), immediately implies that

Theorem 6.7 (CSS |= Acp ). The CSS protocol satisfies the convergence property Acp (Definition 3.1). Example 6.4 (Another Illustration of the CSS Protocol). To compare with the CSCW protocol [11], Figure 6 illustrates the CSS protocols under the slightly more involved schedule in Figure 2 of [11]. A great advantage of the CSS protocol over the CSCW protocol is that it captures all the OTs in a single n-ary ordered state-space, which facilitates the proof that Jupiter satisfies the weak list specification.

(1)

in sequence would be the same with the state-space constructed at client cj if it had processed j j σkj ◦ ok+1 = σ1,l−1 ◦ σl+1,k+1 ◦ ok+1

c

j j j j j j j ol+3 {σ1,l−1 ◦ ol+1 ◦ ol+2 }, ok+1 {σ1,l−1 ◦ ol+1 ◦ . . . ◦ okj }.

c

Thus, σkj is a permutation of σks , ho1 , o2 , . . . , ok i. at In the following, we prove that the state-space CSSk+1 cj cj client cj constructed by executing σk+1 in sequence, namely

c

j j j j j L : ol+1 {σ1,l−1 }, ol+2 {σ1,l−1 ◦ ol+1 }, . . . ,

(2)

in sequence. We first consider the state-space obtained by applying c ok+1 to CSSkcj (which is obtained after executing σkj ) at client cj , corresponding to scenario (2). The matching state cj of ok+1 is σ1,l−1 . According to Lemma 6.5 and the inductive hypothesis that CSSks = CSSkcj , the operation sequence L 8. We abuse the symbol ‘σ ’ for representing states to denote operation sequences. This is reasonable because replica states are defined by the operations a replica has processed (Definition 4.5).

7

J UPITER P ROTOCOLS E STABLISHED E QUIVA -

LENT

In this section we establish the equivalence of the CSS protocol and the CSCW protocol. Specifically, we aim to prove that Theorem 7.1 (Equivalence Theorem). The behaviors of corresponding replicas in the CSS protocol and the CSCW protocol are the same under the same schedule. Proof: We show the equivalence from perspectives of both the server and clients. Specifically, we prove that 1) the behaviors of the servers are the same (Section 7.1) and that 2) the behaviors of each pair of corresponding clients are the same (Section 7.2).


11

C1

S

C2

That is, the state-space CSSs at the server in the CSS protocol equals the union of all state-spaces DSSsi maintained at the server for each client ci in the CSCW protocol.

C3

1 2

1

3

1

1

4

Proof: By mathematical induction on the operation sequence O the server processes. Base Case. k = 1. According to the protocols, it is obviously that CSS1s = DSS1sc(o ) .

2 2

3

1

3

Inductive Hypothesis. Suppose that (∗) holds for k :

2 3

CSSks =

4 4

i=k [

DSSisc(o ) . i

i=1

4

Inductive Step. We shall prove that (∗) holds for (k + 1): CSSk+1 = s

(a) The schedule in Figure 2 of [11]. C2

i=k+1 [

DSSisc(o ) . i

i=1

C1

C3

By inductive hypothesis, we shall prove that

0

CSSk+1 \ CSSks = DSSk+1 s sc(o

1

2

k+1 )

1

In other words, the OTs for ok+1 performed by the servers in both protocols are the same. This holds due to the following two reasons. First, under the same schedule, the matching state of ok+1 in DSSksc(o ) is the same with that in

2 2

3

1

4

23

14

k+1

1

3

CSSks determined by its context C(ok+1 ) (or causal relation). Second, according to Lemmas 5.1 and 6.5, the operation sequences with which ok+1 is transformed are the same in both protocols. From Proposition 7.2, it immediately follows that

2

12

4

123

.

124 4

3

1234

Theorem 7.3 (Equivalence of Server). Under the same schedule, the behaviors of the servers in the CSCW protocol and the CSS protocol are the same.

(b) Illustration of the CSS protocol. Fig. 6. Illustration of the CSS protocol under the schedule in Figure 2 of [11].

7.2 7.1

The Servers Established Equivalent

It follows from Lemmas 5.1 and 6.5 that the CSS protocol performs the same OTs as the CSCW protocol at the server. Stated differently, the state-spaces at the server are the same. Formally, Proposition 7.2 (n ↔ 1). Suppose that under the same schedule, the server has processed a sequence of m operations, denoted O = ho1 , o2 , . . . , om i, in total order. We have, for all 1 ≤ k ≤ m, CSSks

=

i=k [ i=1

DSSisc(o ) i

(∗)

where c(oi ) denotes the client that generates the operation oi . Remark. Since in the CSCW protocol the OT for an operation generated by client ci is only performed once on the state-space DSSsi , we have CSSks =

i=k [ i=1

DSSisc(o

i)

=

[

j=k [

DSSjsc , i

ci ∈c(O) j=1

where c(O) = hc(o1 ), c(o2 ), . . . , c(om )i.

The Clients Established Equivalent

In this section we prove that the behaviors of each pair of corresponding clients of both protocols are the same. The CSCW protocol can be regarded as an optimized variant of the CSS protocol in that it eliminates redundant OTs at clients, by letting the server redirect the transformed operations rather than the original ones. Proposition 7.4 (1 ↔ 1). Under the same schedule, DSSkci is a part of CSSkci , for all 1 ≤ i ≤ n and k ≥ 1. Formally, DSSkci ⊆ CSSkci .

(∗)

Proof: By mathematical induction on the operation sequence Oci = hoc1i , oc2i , . . . , ocmi i the client ci processes. Base case. k = 1, namely, Oci = hoc1i i. No matter whether oc1i is generated by client ci or is a remote operation propagated to client ci , it obviously holds that DSS1ci = CSS1ci . Inductive Hypothesis. Suppose Oci = hoc1i , oc2i , . . . , ocki i and (∗) holds for k : DSSkci ⊆ CSSkci .


12

Inductive Step. Client ci executes the (k + 1)-st operation i . We shall prove that (∗) holds for (k + 1): ock+1

S ϵ 1

1

Del(x, 0) 2

Ins(a, 0)

4 3 3 2 2 4

4

4

3

(a) A schedule adapted from Figure 1 of [16]. S, C1

C2 C3

0 ϵ 1: Ins(x, 0)

1 x

(a ,

0)

2:

123 a

8 J UPITER P ROTOCOLS AND R EPLICATED L IST S PECIFICATION

4:

13 ax

,0 )

( el D

1)

14 xb

4: In s(b, 0 )

D

In s

1) x,

4: I ns ( b,

2:

3:

0)

el (x

s( a,

124 b 1)

12 ϵ

Proof: Notice that no matter whether the operation ocki is generated by client ci or is a remote operation propagated to client ci , the final transformed operations (maybe unchanged) executed at ci in both protocols are the same.

In

D

In

s( b, 0)

s( a,

2:

In

3:

)

,0

(x el

3:

Theorem 7.5 (Equivalence of Clients). Under the same schedule, the behaviors of each pair of corresponding clients in the CSCW protocol and in the CSS protocol are the same.

1234 ba

We examine the CSS Jupiter protocol (and the CSCW Jupiter protocol) from the perspectives of strong/weak replicated list specification. Specifically, Section 8.1 illustrates that the CSS protocol does not satisfy the strong list specification by means of a counterexample. Section 8.2 proves that the CSS protocol satisfies the weak list specification.

(b) The n-ary ordered state-space produced by the CSS protocol. As illustrated, the server and client c1 go along the path 0 → 1 → 12 → 123 → 1234, client c2 goes along the path 0 → 1 → 13 → 123 → 1234, and client c3 goes along the path 0 → 1 → 14 → 124 → 1234. (We assume that client with a larger id has a higher priority.)

Jupiter Violates the Strong List Specification

Proof: Figure 7 shows a counterexample, including a schedule adapted from Figure 1 of [16] and the n-ary ordered state-space produced by the CSS protocol. Initially, the replicated list is empty, namely w = . The fact that the strong list specification is not satisfied by this execution follows from: • The list w13 in state σ13 = {1, 3} of client c2 is ax; • The list w14 in state σ14 = {1, 4} of client c3 is xb;

Ins(b, 1)

3

i OT (ock+1 , L)

Theorem 8.1 (CSS 6|= Astrong ). The CSS protocol (or equivalently, the CSCW protocol) does not satisfy the strong list specification Astrong (Definition 3.2).

1

2

performed at the server in both protocols are also performed at client ci in the CSS protocol. By contrast, in the CSCW i protocol, the transformed operation ock+1 {L} is propagated to client ci , where the set of OTs performed is a subset of i those involved in OT (ock+1 , L). Given the inductive hypothk k esis DSSci ⊆ CSSci , we conclude that DSSk+1 ⊆ CSSk+1 ci ci . Following similar lines of the proof for Proposition 7.4, we conclude that

8.1

C3 ϵ

1

i We distinguish two cases between ock+1 being generated by client ci or a remote operation propagated to client ci . i is generated by client ci . C ASE 1: The operation ock+1 k+1 The new state-space DSSci (resp. CSSk+1 ci ) is obtained by i saving ock+1 at the final state of the previous state-space DSSkci (resp. CSSkci ). Since DSSkci ⊆ CSSkci (by the inductive ⊆ CSSk+1 hypothesis), we conclude that DSSk+1 ci . ci ci C ASE 2: The operation ok+1 is a remote operation propagated to client ci . Due to Lemmas 5.1 and 6.5, the operation i sequences L with which ock+1 is transformed at the server in both protocols are the same. Since the communication is i FIFO, when ock+1 is received by client ci , all the (original) operations totally ordered before it have already been in CSSkci . According to Proposition 6.6, the OTs involved in

=

C2 ϵ

Ins(x, 0)

DSSk+1 ⊆ CSSk+1 ci ci .

ci i (ok+1 {L}, L{ock+1 })

C1 ϵ

Fig. 7. A counterexample showing that Jupiter does not satisfy the strong list specification.

• •

The list w1234 in the same final state σ1234 = {1, 2, 3, 4} of all replicas is ba; According to condition 1) of the strong list specification, the list order lo would have to be lo = {(a, x), (x, b), (b, a)};


C1 abc

o1

Jupiter Satisfies the Weak List Specification

Proof: We prove both directions by contradiction. lo

“⇐” (if): Suppose by contradiction that a − → a for some a ∈ elems(H). According to Lemma 6.3, a list state w contains no duplicate elements. Therefore, there exist two lo list states such that for some element b, a − → b (namely lo a appears before b) in one state and b − → a (namely b appears before a) in the other. However, this contradicts the assumption that all list states are pairwise compatible. “⇒“ (only if): Suppose by contradiction that two states w1 and w2 are incompatible. That is, they have two common elements a and b such that a appears b in, say, w1 and b lo lo appears a in w2 . Thus, both a − → b and b − → a hold. Since lo lo is transitive on w1 (or w2 ), we have a − → a, contradicting the assumption that lo is irreflexive.

aybc

1

1)

2

3

o2 {1} = Del(b, 1)

}= {2

x s(

In

o3 {2} = Ins(y, 1)

o1

ayc

s(x

,1

)

axc 12 o3 {1, 2} = Ins(y, 2)

axyc 123

ac

) ,2 (b el o1 {3} = Ins(x, 3) D

) ,1

}= {3 o3 {1 }= o2

23

Ins

(y , 1)

13 aybxc

o2 {1, 3} = Del(b, 2)

)

Lemma 8.3 (Inreflexivity). Let α be an execution and A = (H, vis) the abstract execution constructed from α as above. The list order lo based on α is irreflexive if and only if the list states (i.e., returned lists) in A are pairwise compatible.

s( y,

,2

Definition 8.2 (State Compatibility). Two list states w1 and w2 are compatible, if and only if for any two common elements a and b of w1 and w2 , their relative orderings are the same in w1 and w2 .

In

s(x

By definition, 1) lo is transitive and total on {a | a ∈ w} for all events e = do(op, w) ∈ H ; and 2) lo satisfies 1b) of Definition 3.3. The irreflexivity of lo can be rephrased in terms of the pairwise state compatibility property.

=

o2 = Del(b, 1)

In

Definition 8.1 (List Order ‘lo’). Let α be an execution. For lo a, b ∈ elems(A), a − → b if and only if there exists an event e ∈ α with returned list w such that a appears before b in w.

o3

2)

}= {3 o1

Proof: For each concrete execution α of the CSS protocol, we construct the abstract execution A = (H, vis) with vis = → (Definition 4.1), as [16] does. We then prove the conditions of Definition 3.3 in the order 1a), 1c), 1b), and 2). Condition 1a) holds due to the FIFO communication mechanism of the CSS protocol and the property of OTs that when transformed, the effect of an insertion (resp. deletion) operation I NS(a, p) (resp. D EL(a, p)) remains unchanged, namely to insert (resp. delete) the element a (possibly at a different position than p). Condition 1c) follows directly from the local processing mechanism of the CSS protocol. To show that A = (H, vis) belongs to Aweak , we define the list order relation lo in Definition 8.1 below, and then prove that lo satisfies conditions 1b) and 2) of Definition 3.3.

x, s(

In

Theorem 8.2 (CSS |= Aweak ). The CSS protocol (or equivalently, the CSCW protocol) satisfies the weak list specification Aweak (Definition 3.3).

abxc

=

In

}=

One of the main contributions of this work is as follows.

C2

0

{2 ,3

8.2

However, the list order lo above contains a cycle, contradicting condition 2) of the strong list specification.

o1

•

13

123 ayxc

Fig. 8. A “counterexample” to Lemmas 8.4, 8.5, and 8.6 produced by an incorrect protocol; see Example 8.1. (Adapted from Figure 1 of [11].)

In the remainder of this section, we prove that all list states are pairwise compatible, which concludes the proof for Theorem 8.2. By Proposition 6.6, a single n-ary ordered state-space at any replica contains all possible states that each client goes through. In other words, in the CSS protocol, the nary ordered state-space provides a global view of all client states. By contrast, in the CSCW protocol, these client states are dispersed in multiple 2D state-spaces. It is the global view that makes the CSS protocol more suitable for analyzing properties involving states of different clients. In the following, we focus on the state-space CSSs at the server. We proceed to prove several key properties about (a pair of) states and paths of the state-space CSSs , which serve as building blocks for the proof of the main result (Theorem 8.7). In addition, we illustrate these properties using a running “counterexample” produced by an incorrect OT-based protocol. Example 8.1 (A Running “Counterexample”). Consider three pairwise concurrent operations in a collaborative editing system, namely o1 = I NS(x, 2), o2 = D EL(b, 1), and o3 = I NS(y, 1), invoked on the same initial list state “abc”; see Figure 8. Suppose that in some execution of an incorrect OT-based protocol, client C1 has executed o1 , o3 {1} (after performing OT (o3 , o1 )), and o2 {1, 3}, and client C2 has executed o2 , o3 {2}, and o1 {2, 3} in sequence, as shown by the dashed lines in the figure. This execution satisfies neither the convergence properties nor the weak list specification. Lemma 8.4 (LCA). In the CSS protocol, every pair of states in the n-ary ordered state-space CSSs at the server has a unique LCA (Lowest Common Ancestor). Proof: By mathematical induction on the operation sequence processed in total order at the server. Base Case. Initially, the state-space CSS0s contains only the single initial state σ0 = {0}. The lemma obviously holds.


14

σ′

L

CS

s

Sk

σ0

σ

σL′

ok

σβ

σα

+1

o

o

o

o

σβ′

σα′

σL σ1 σ2

CS

S∆

“extension ladder” structure

σ1

σ2 o

o

(a) C ASE 2.1: σα − → σα0 and σβ − → σβ0 are in the same “extension ladder” structure. Fig. 9. Illustration of proof for Lemma 8.4: states σ and σ 0 are two 0 }. incomparable common ancestors of σ1 and σ 00 = σL , min{σL , σL

CSS∆ ,

CSSk+1 s

\

o

σγ

p

o

te

o

o

σα′

CSSks

be the extra part of CSSk+1 obtained by applying s OT (ok+1 , L) to CSSks ; see Figure 9. We need to verify that 1) every pair of states in CSS∆ has a unique LCA; and 2) every pair of states consisting of one state in CSSks and the other in CSS∆ has a unique LCA. The former claim obviously holds because all states in CSS∆ are in a path. We prove the latter by contradiction. Let σ1 be any state in CSSks and σ2 any state in CSS∆ ; see Figure 9. Clearly, the initial state σ0 = {0} is a common ancestor of σ1 and σ2 . Suppose by contradiction that there are two LCAs, denoted σ and σ 0 , of σ1 and σ2 in CSSks (they cannot be in CSS∆ ). Note that any path from σ or σ 0 (or generally, from any state in CSSks ) to σ2 passes through some state in L. Let 0 σL (resp. σL ) be the last state in L in the path from σ 0 0 (resp. σ ) to σ2 . Let σ 00 = min{σL , σL } be the first state 0 0 of σL and σL along L. Then, σ and σ are two incomparable common ancestors of σ1 and σ 00 . This, however, contradicts the inductive hypothesis. Example 8.2 (LCA). Although the proof for Lemma 8.4 does not directly refer to the “leftmost transitions” rule of the CSS protocol, it relies on this rule in a fundamental way: The reason that we are allowed to consider only the single state-space CSSs at the server is that thanks to the “leftmost transitions” rule, all state-spaces of replicas are eventually the same (Proposition 6.6). If we simply took the union of all state-spaces at clients without the guarantee of Proposition 6.6, Lemma 8.4 may not necessarily hold. For example, the state-space in Figure 8 is the union of

σ0

la

dd e

r”

σγ′

“s

Inductive Hypothesis. Suppose that the server has processed k operations and that every pair of states in the statespace CSSks has a unique LCA. Inductive Step. The server has processed the (k + 1)-st operation ok+1 . We shall prove that every pair of states in has a unique LCA. Let the state-space CSSk+1 s

st

ru

ct

ur e

o

o

o

σβ′

o

σα o

σβ σ1 o

σ2 o

o

(b) C ASE 2.2: σα − → σα0 and σβ − → σβ0 are in a “step ladder” structure. Fig. 10. Illustrations of C ASE 2 of the proof for Lemma 8.5.

state-spaces of clients C1 and C2 . In this state-space, both states {1} and {2, 3} are common ancestors of the two states {1, 2, 3} at the bottom, and they are incomparable. The following lemma shows that the two paths (not necessarily unique) to a pair of states in CSSs respectively from their LCA are disjoint in the operations they involve. Lemma 8.5 (Disjoint Paths). Let σ0 be the unique LCA of a pair of states σ1 and σ2 in the n-ary ordered state-space CSSs at the server (denoted σ0 = LCA(σ1 , σ2 )). Then, the set of operations Oσ0 σ1 along a simple path (Lemma 6.3) Pσ0 σ1 is disjoint from the set of operations Oσ0 σ2 along a simple path Pσ0 σ2 . Proof: We distinguish two cases according to whether σ1 and σ2 are in the same path or not. C ASE 1: σ1 and σ2 are in the same path. This lemma obviously holds because either Oσ0 σ1 or Oσ0 σ2 is empty. C ASE 2: σ1 and σ2 are not in the same path. In this case, we


15

prove this lemma by contradiction. Suppose that

o ∈ Oσ0

σ1

∩ Oσ0

σ2 ,

where o can be either original or transformed. As illustrated in Figure 10, the paths Pσ0 σ1 and Pσ0 σ2 are now:

Pσ 0

σ1

= Pσ

0

σ1

Pσ 0

σ2

= Pσ

σα − →σα0

0

σβ − →σβ0

σ2

o

o

, .

In the following, we derive a contradiction that σ0 is not the unique LCA of σ1 and σ2 . We consider two cases according o o to how the transitions σα − → σα0 and σβ − → σβ0 are related via OTs in CSSs . o o C ASE 2.1: σα − → σα0 and σβ − → σβ0 are in the same “extension ladder” structure of OTs. Without loss of generality, 0 we assume that σβ0 is reachable from σα ; as illustrated 0 in Figure 10a. In this case, σα is a lower common ancestor of σ1 and σ2 than σ0 . This contradicts the condition LCA(σ1 , σ2 ) = σ0 . o o C ASE 2.2: σα − → σα0 and σβ − → σβ0 are in a “step ladder” 9 structure of OTs. Because all the transitions labeled with the same operation o are constructed directly or indirectly from the OTs involving the original operation org(o), there exists o some transition σγ − → σγ0 that is in the same “extension o o → σβ0 ; as ladder” with σα − → σα0 as well as with σβ − 0 illustrated in Figure 10b. In this case, σγ is a common ancestor of σ1 and σ2 other than σ0 . This contradicts the condition LCA(σ1 , σ2 ) = σ0 . Example 8.3 (Disjoint Paths). In Figure 8, the paths from state {1} to the two states {1, 2, 3} at the bottom are not disjoint in operations they involve. This is because {1} is not the unique LCA of those two states. We now present a sufficient condition for two states being compatible. Lemma 8.6 (Compatible Paths). Let Pσ0 σ1 (resp. Pσ0 σ2 ) be a path from state σ0 to state σ1 (resp. σ2 ) in the state-space CSSs . If both Pσ0 σ1 and Pσ0 σ2 are simple paths and they are disjoint, then σ1 and σ2 are compatible. Proof: We prove a stronger statement that every pair of states consisting of one state in Oσ0 σ1 and the other in Oσ0 σ2 are compatible, by mathematical induction on the length l of the path Pσ0 σ2 . To this end, we first show that Claim (One-step Compatibility). Suppose that states σ and σ 0 are compatible. Let σ 00 be the next state of σ 0 along the transition labeled with operation o which does not correspond to any element of the list in state σ . Then, σ and σ 00 are compatible. Proof. Let C(σ, σ 0 ) be the set of common elements of states σ and σ 0 and C(σ, σ 00 ) of states σ and σ 00 . Since o does not correspond to any element of the list in state σ , C(σ, σ 00 ) is a subset of C(σ, σ 0 ). Furthermore, the total ordering of elements in C(σ, σ 00 ) is consistent with that in C(σ, σ 0 ). Base Case. l = 0. Pσ0 σ2 contains only the state σ0 . We shall prove that σ0 is compatible with every state along Pσ0 σ1 . This can be done by mathematical induction on 9. This cannot happen in 2D state-spaces.

the length of Pσ0 σ1 with the claim above and the fact that Pσ0 σ1 is a simple path. Inductive Hypothesis. Suppose that this lemma holds when the length of Pσ0 σ2 is l ≥ 1. Inductive Step. We shall prove that the (l + 1)-st state, denoted σl+1 , of Pσ0 σ2 is compatible with every state along Pσ0 σ1 . This can be done by mathematical induction on the length of Pσ0 σ1 with the claim above, the fact that Pσ0 σ2 is a simple path (for σ0 and σl+1 being compatible), and the fact that Pσ0 σ1 and Pσ0 σ2 are disjoint. Remark. The proof for Lemma 8.6 implicitly relies on the fact that OTs preserve the types (I NS or D EL) of operations transformed. 10 Example 8.4 (Compatible Paths). In Figure 8, the two states {1, 2, 3} at the bottom are incompatible. Neither are the states {1, 3} and {1, 2, 3} on the bottom left. Finally, the pairwise state compatibility property below, which has been shown to be equivalent to the irreflexivity of the list order lo (Lemma 8.3), concludes the proof for Theorem 8.2. Theorem 8.7 (Pairwise State Compatibility Property). Every pair of states in the state-space CSSs are compatible. Proof: Consider a pair of states σ1 and σ2 . The arguments are as follows. • By Lemma 8.4, they have a unique LCA, denoted σ0 . • By Lemma 6.3, Pσ0 σ1 and Pσ0 σ2 are simple paths. • By Lemma 8.5, Pσ0 σ1 and Pσ0 σ2 are disjoint. • By Lemma 8.6, states σ1 and σ2 are compatible.

9

R ELATED W ORK

Convergence is the main property for implementing a highly available replicated list object [6], [11]. Since the seminal work [6] in 1989, a number of protocols that aim to achieve convergence have been proposed, based on the techniques of OT [4], [6], [7], [8], [9], [10], [11] or CRDT [12], [13], [14], [15], [16]. The OT-based protocols can be classified according to whether they use a central coordinator or are fully distributed, as reviewed in [11]. Jupiter [4], [11], Google Wave [1], and NICE [9] are based on a central server for serializing and propagating operations. TIBOT [24], COT [10], and TIBOT 2.0 [11] rely on a distributed scheme to totally order operations. In addition, SOCT3 and SOCT4 [8] use a central sequencer, which delivers continuously growing timestamps to totally order operations. The CRDT-based protocols utilize operation commutativity to ensure convergence [21]. WooT [12] maintains a partial list order and ensures convergence by using a monotonic linear extension function. TreeDoc [13] uses a binary tree to maintain the total order between position identifiers and keeps deleted elements as tombstones. Logoot [14] eliminates tombstones in TreeDoc by using a position identifier based on a list of integers. In RGA [15] and a variant of it in [16], a replica state can be viewed as a tree (which is not 10. Lemma 8.6 still holds when taking into account the exceptional cases that operations are transformed to be N OP, the idle operation [22].


necessarily a binary tree), and the list order is defined by a deterministic pre-order traversal of the tree. In 2016, Attiya et al. [16] propose two new formal specifications of a replicated list object, namely strong list specification and weak list specification. They prove that a variant of the existing RGA protocol satisfies the strong list specification. They also conjecture that Jupiter satisfies the weak list specification.

10

C ONCLUSION AND F UTURE W ORK

This paper revisits the specification and implementation of the replicated list object, specifically the weak list specification proposed recently by Attiya et al. [16] and the CSCW Jupiter protocol [4], [11] designed in the 1990s. We propose the CSS Jupiter protocol based on a novel data structure called n-ary ordered state-space, show the equivalence of it and the CSCW protocol, and finally prove that they satisfy the weak list specification. Jupiter was originally proposed for ensuring the convergence property [4], which is weaker than the weak list specification [16]. This motivates us to design protocols that satisfies the convergence property but not the weak list specification. Furthermore, we plan to study the lower bound on the worst-case metadata overhead of protocols implementing the convergence property. We believe that the n-ary ordered state-space data structure is of its own interest in OT-based protocols. We plan to explore it in two ways. First, we consider extending the CSS protocol to a distributed setting, by integrating the compact n-ary ordered state-space with a distributed scheme to totally order operations. Second, we study how to algebraically manipulate and reason about n-ary ordered state-spaces.

ACKNOWLEDGMENTS This work is supported by the National 973 Program of China (2015CB352202) and the Fundamental Research Funds for the Central Universities (020214380031).

R EFERENCES [1] [2] [3] [4]

[5] [6]

[7]

[8]

[9]

“Apache wave,” https://incubator.apache.org/wave/. “Codoxword,” http://www.codoxware.com/. “Gobby,” https://gobby.github.io/. D. A. Nichols, P. Curtis, M. Dixon, and J. Lamping, “High-latency, low-bandwidth windowing in the jupiter collaboration system,” in Proceedings of the 8th Annual ACM Symposium on User Interface and Software Technology, ser. UIST ’95. ACM, 1995, pp. 111–120. Y. Saito and M. Shapiro, “Optimistic replication,” ACM Comput. Surv., vol. 37, no. 1, pp. 42–81, Mar. 2005. C. A. Ellis and S. J. Gibbs, “Concurrency control in groupware systems,” in Proceedings of the 1989 ACM SIGMOD International Conference on Management of Data, ser. SIGMOD ’89. ACM, 1989, pp. 399–407. C. Sun and C. Ellis, “Operational transformation in real-time group editors: Issues, algorithms, and achievements,” in Proceedings of the 1998 ACM Conference on Computer Supported Cooperative Work, ser. CSCW ’98. ACM, 1998, pp. 59–68. N. Vidot, M. Cart, J. Ferri´e, and M. Suleiman, “Copies convergence in a distributed real-time collaborative environment,” in Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, ser. CSCW ’00. ACM, 2000, pp. 171–180. H. Shen and C. Sun, “Flexible notification for collaborative systems,” in Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work, ser. CSCW ’02. ACM, 2002, pp. 77–86.

16

[10] D. Sun and C. Sun, “Context-based operational transformation in distributed collaborative editing systems,” IEEE Trans. Parallel Distrib. Syst., vol. 20, no. 10, pp. 1454–1470, Oct. 2009. [11] Y. Xu, C. Sun, and M. Li, “Achieving convergence in operational transformation: Conditions, mechanisms and systems,” in Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work, ser. CSCW ’14. ACM, 2014, pp. 505–518. [12] G. Oster, P. Urso, P. Molli, and A. Imine, “Data consistency for p2p collaborative editing,” in Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work, ser. CSCW ’06. ACM, 2006, pp. 259–268. [13] N. Preguica, J. M. Marques, M. Shapiro, and M. Letia, “A commutative replicated data type for cooperative editing,” in Proceedings of the 2009 29th IEEE International Conference on Distributed Computing Systems, ser. ICDCS ’09. IEEE Computer Society, 2009, pp. 395–403. [14] S. Weiss, P. Urso, and P. Molli, “Logoot: A scalable optimistic replication algorithm for collaborative editing on p2p networks,” in Proceedings of the 2009 29th IEEE International Conference on Distributed Computing Systems, ser. ICDCS ’09. IEEE Computer Society, 2009, pp. 404–412. [15] H.-G. Roh, M. Jeon, J.-S. Kim, and J. Lee, “Replicated abstract data types: Building blocks for collaborative applications,” J. Parallel Distrib. Comput., vol. 71, no. 3, pp. 354–368, Mar. 2011. [16] H. Attiya, S. Burckhardt, A. Gotsman, A. Morrison, H. Yang, and M. Zawirski, “Specification and complexity of collaborative text editing,” in Proceedings of the 2016 ACM Symposium on Principles of Distributed Computing, ser. PODC ’16. ACM, 2016, pp. 259–268. [17] D. Wang, A. Mah, and S. Lassen, “Google wave operational transformation,” https://svn.apache.org/repos/asf/incubator/wave/ whitepapers/operational-transform/operational-transform.html, accessed: August 2017. [18] S. Burckhardt, A. Gotsman, H. Yang, and M. Zawirski, “Replicated data types: Specification, verification, optimality,” in Proceedings of the 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, ser. POPL ’14. ACM, 2014, pp. 271–284. [19] H. Attiya, F. Ellen, and A. Morrison, “Limitations of highlyavailable eventually-consistent data stores,” in Proceedings of the 2015 ACM Symposium on Principles of Distributed Computing, ser. PODC ’15. ACM, 2015, pp. 385–394. [20] L. Lamport, “Time, clocks, and the ordering of events in a distributed system,” Commun. ACM, vol. 21, no. 7, pp. 558–565, Jul. 1978. [21] M. Shapiro, N. Preguic¸a, C. Baquero, and M. Zawirski, “Conflictfree replicated data types,” in Proceedings of the 13th International Conference on Stabilization, Safety, and Security of Distributed Systems, ser. SSS’11, 2011, pp. 386–400. [22] A. Imine, M. Rusinowitch, G. Oster, and P. Molli, “Formal design and verification of operational transformation algorithms for copies convergence,” Theor. Comput. Sci., vol. 351, no. 2, pp. 167– 183, Feb. 2006. [23] A. Prakash and M. J. Knister, “A framework for undoing actions in collaborative systems,” ACM Trans. Comput.-Hum. Interact., vol. 1, no. 4, pp. 295–330, Dec. 1994. [24] R. Li, D. Li, and C. Sun, “A time interval based consistency control algorithm for interactive groupware applications,” in Proceedings of the Parallel and Distributed Systems, Tenth International Conference, ser. ICPADS ’04. IEEE Computer Society, 2004, pp. 429–438.