Group Communication Protocol in Transaction-Oriented Applications Tomoya Enokido, Takayuki Tachikawa, and Makoto Takizawa Dept. of Computers and Systems Engineering Tokyo Denki University Email feno, tachi,
[email protected] Abstract In group communications, larger computation and communication are consumed to causally order all the messages transmitted in the network. Transactions in clients manipulate objects in servers by sending read and write requests to the servers. In this paper, we define significant messages by using the relation among the transactions. We newly propose an object vector to causally order only the significant messages. The scheme of the object vector is invariant in the change of the group membership. We also show a TBCO (transaction-based causally ordered) protocol which adopts the object vector, by which the number of messages to be causally ordered are reduced.
1. Introduction In distributed applications, a group of multiple objects are cooperating. Here, the messages have to be reliably delivered to the destination objects in the group. Many group protocols[2, 7, 8, 9] are discussed to causally order messages transmitted in the network. However, the group protocol implies O(n) to O(n2 ) computation and communication overhead. Cheriton[3] points out that it is meaningless to causally order all the messages transmitted from the application point of view. Agrawal[5] discusses a rollback algorithm where only processes computing significant requests are rolled back if the senders of the significant requests are rolled back. Tachikawa and Takizawa[13] define the object-based causality among messages based on the conflicting relation among operations supported by the objects in the presence of the nested invocation of operations. Raynal[11] discusses a way to relex the traditional messagebased causality based on the write-semantics for a group of the replicas. In this paper, we introduce a transaction[1] concept to define a causality among messages which are significant for the applications. A transaction T in a client processor issues read and write requests to server processors. Thus,
the subtransactions of T are computed in the servers. Data read from an object is written to another object if read and write are computed in a transaction. A message m1 may carry information in m2 if a transaction sends m1 after receiving or sending m2 . Thus, the messages received and sent by one transaction are related. Suppose one transaction T1 sends m2 after T2 receives m1 in the same processor. m1 and m2 are not related unless T1 and T2 manipulate same objects. In order to define messages to be causally ordered, we have to consider what subtransactions send and receive messages. Hence, the group is required to be composed of multiple subtransactions and servers. In this paper, we define how read and write messages are causally related in a context of transactions. In addition, each object is replicated in order to increase the reliability, availability, and performance of the system. We define significant messages to be causally ordered based on transaction concept. In most group protocols, messages are causally ordered by using the vector clock[6]. Each time transactions are randomly initiated and terminated, the size of the vector clock is required to change. In this paper, we newly propose an object vector whose size is given by the number of objects. We discuss a TBCO (transaction-based causally ordered) protocol which uses the object vector to causally order significant messages. In the TBCO protocol, significant messages can be delivered earlier without waiting for insignificant messages preceding the significant messages. We present a system model in section 2. In section 3, we define significant messages. In section 4, we present the object vector and the TBCO protocol. In section 5, we evaluate the TBCO protocol.
2. System Model A system is composed of processors p1 , : : :, pn (n 1) interconnected by the communication network. Each object oi is stored in one processor. In the communication network, messages may be lost and may be delivered out of order. On receipt of a request message m with an operation opi , oi computes opi . On completion of opi , oi sends back the
response message to the sender of m. We assume that each oi is replicated. Each replica is allocated in a processor. Let oti denote a replica of oi in a processor pt . A transaction Ti in one client processor ps issues read and write requests to the server processors to manipulate replicas of an objects oi . A processor sends a read request to one replica. pt sends back the response with the data derived. ps sends write requests to all the replicas of x. That is, the read-one write-all principle is adopted. Here, op1 !Ti op2 iff an operation op1 precedes op2 in Ti . Ti commits (ci ) or aborts (ai ) at the end of Ti . Ti commits if all the operations invoked by Ti complete successfully. If at least one operation fails in Ti , Ti aborts. Thus, Ti is an atomic sequence of read and write operations[1]. Let T be a set f T1 , : : :, Tm g of transactions in the system. Let opit (x) denote an operation op of Ti on a replica x in pt . op is r(read) or w(write). A subsequence of operations of Ti on the replicas in a processor pt is named a subtransaction Tit of Ti in pt . op1 !Tit op2 iff op1 precedes op2 in Tit . The interleaved computation of subtransactions T1t , : : :, Tmt in pt is a local history Ht of pt . op1 precedes op2 in pt ( op1 !Ht op2 ) iff op1 is computed before op2 in pt . A global history H of T is a collection of the local histories H1 , : : :, Hn . op1 precedes op2 in H op1 !H op2 iff op1 !Ht op2 , op1 !Ti op2 , or op1 !H op3 !H op2 for some op3 . H has to be serializable[1] to keep the replicas consistent. In order to make H serializable, the conflicting operations, i.e. write and read or write and write on the same object have to be computed in the same order as some serial history. Each Ti is given a unique identifer t(Ti ) in order to totally order the transactions. In the time stamp ordering (TO) concurrency control[1], every pair of conflicting operations are computed in the order of the transaction identifers, i.e. time stamp. That is, if an operation opi of Ti conflicts with opj of Tj in pt and t(Ti) < t(Tj ), opi is computed before opj in pt . In the two-phase locking[1], the replicas are locked before they are manipulated.
3. Significant Messages A processor pt may not receive messages or may receive messages out of order. If pt loses a message m sent by pu , pu is required to resend m to pt . After pt receives m, pt has to wait for all messages causally preceding m. [Example] Suppose wt1 (xt ), ws1 (xs ), and wu1 (xu ) are initiated by a request message m1 , rt2 (xt ) is by m2 , ws3 (xs ), wt3 (xt ), and wu3 (xu ) are by m3 , and ru4 (xu ) is by m4 as shown in Figure 1. wt3 (xt ) is computed after rt2 (xt ) in pt . Here, suppose pu receives m3 before m1 . In the traditional group protocols, pu has to wait for m1 without delivering m3 . That is, pu has to compute ws1 (xs ), wu3 (xu ), and ru4 (xu ) in this sequence. ru4 (xu ) reads x from wu3 (xu ) but not from ws1 (xs ). Here, suppose wu1 (xu ), wu3 (xu ), and ru4 (xu ) are still
stored in the receipt queue RQu of pu due to the communication delay although the operations complete in ps and pt . Since ru4 (xu ) reads from wu3 (xu ), wu1 (xu ) does not need to be computed, i.e. ws1 (xs ) can be omitted. 2
ws1 (x s )
p u
p t
p s m1
t w t1 (x )
u w u1 (x )
m2 2 r t (x t )
m3 ws (x s ) 3
wt3 (x t ) m4
m3
wu3 (x u) r u4 (x u)
Figure 1. Significant messages.
[Example] Suppose pt receives read requests rti , rtj , and
rtk from transactions Ti , Tj , and Tk , respectively. pt reads a value from the replica x and sends back the response with data each time pt receives a read. Here, pt computes read operations on x three times. If pt reads x once and sends back the response to Ti , Tj , and Tk , the number of operations computed can be reduced. That is, rtj (x) and rtk can be removed from RQt if rti is computed. 2
The first example shows that some write requests can be omitted. Suppose wti (x) !H wtj (x). If there is no read rtk (x) such that wti (x) !H rtk (x) !H wtj (x), pt dosej not need to receive wti (x). This means pt can compute wt (x) without waiting for wti (x). pt can further reject wti (x) if pt receives wti (x) after computing wtj (x). The second example shows some read requests can be omitted. We discuss what messages can be omitted. [Definition] A message m is insignificant iff (1) if m = rti (x), there is some read rtj (x) in RQt such that rtj (x) precedes rti (x) and there is no write wtk (x) between rtj (x) and rti (x) in RQt . (2) if m = wti (x), there is some write wtj (x) in RQt such that wti (x) precedes wtj (x) and there is no read rtk (x) between wti (x) and wtj (x) in RQt . 2 In Figure 1, suppose pu receives m1 = wu1 (xu ), m3 = 3 wu (xu ), and m4 = ru4 (xu ) in RQu . Here, m3 and m4 are significant in RQu but m1 is insignificant. Insignificant messages can be omitted from RQu . [Theorem] Let Rt be a sequence of messages in the receipt queue RQt of pt . Let Rt0 be a subsequence of Rt which includes only the significant messages. Here, the state of pt and the sequence of output values of read requests obtained
by computing the operations in Rt0 are the same as Rt .
2
4. TBCO Protocol 4.1. Object vector Let sndt [m] and rcvt [m] denote a sending and receipt events of a message m in a processor pt , respectively. Lamport[4] defines that an event e1 happens before e2 ( e1 ! e2 ) iff (1) e1 precedes e2 in a processor pt , i.e. e1 !Ht e2 . (2) e1 = snds [m] and e2 = rcvt [m], or (3) e1 ! e3 ! e2 for some event e3 . A message m1 causally precedes m2 ( m1 ! m2 ) iff snds [m1 ] ! sndt [m2 ][2]. pt has to receive m1 before m2 if m1 ! m2 . The vector clock[6] V = hV1 , : : :, Vn i is used to causally order the messages received. Each Vu is initially 0. pt increments Vt by one each time pt sends a message m. m carries the vector clock m.V ( = V ). On receipt of m, pt changes V as Vu = max( Vu , m.Vu ) for u = 1, : : :, n and u 6= t. m1 ! m2 iff m1 .V < m2 .V . If the membership of the group changes, all the processors have to be synchronized[2, 3, 4, 5, 6, 13]. We consider the conflicting and precedence relations among transactions to order only significant messages. That is, a group is considered to be composed of subtransactions and servers. The membership of the group changes each time transactions are initiated and terminated. If the vector clock is used, the group has to be frequently resynchronized. Here, the vector clock can be used to causally order messages sent by processors but not by transactions. In this paper, we propose an object vector to causally order the significant messages sent by the transactions. Each transaction Ti is given a unique identifier t(Ti ) satisfying the following properties. (1) If Ti starts before Tj in a processor, t(Ti ) < t(Tj ). (2) If a processor pt initiates Ti after receiving a message from Tj , t(Ti ) > t(Tj ).
The transaction identifier is given by the linear clock[4]. pt manipulates a variable tid showing the linear clock whose initial value is 0. pt manipulates tid as follows :
(1) tid := tid + 1 if a transaction is initiated in pt . (2) On receipt of a message from Tj , tid := max(tid, tid(Tj )).
When Ti is initiated in pt , t(Ti ) is given a concatenation of tid and the processor number pno(pt ) of pt . Here, let tid(Ti) denote tid given to Ti . For every pair of Ti and Tj initiated at pt and pu , tid(Ti ) > tid(Tj ) if (1) t(Ti ) > t(Tj ) or (2) t(Ti ) = t(Tj ) and pno(pt ) > pno(pu ). Each event e is given an event number no(e) as follows : (1) If e1 is the initial event of Ti , no(e1 ) = 0. (2) If e2 is write and e1 ! e2 in Ti , no(e1 ) < no(e2 ).
(3) If e2 is read and there is no write event e3 such that e1 !Ti e3 !Ti e2, no(e1) = no(e2 ).
noi is incremented by one each time a write event e occurs in Ti . no(e) := noi . Each event e in Ti is given a global event number tno(e) as the concatenation of t(Ti ) and no(e). Every replica oja of oa has the same version number v (oja ) = v (oa ). v (oja ) is updated to be tno(w) each time w(oja ) is computed. Here, if tno(op) > v (oja ), op can be computed. Otherwise, op aborts because op is obsolete. 4.2. Message transmission and receipt A message m is composed of the following fields :
m:src = sender processor of m, i.e. ps . m:dst = set of destination processors. m:op = type op of message, i.e. read, write, commit, abort, or response. m:tno = global event number hm:t, m:noi, i.e. tno(op). m:o = object to be manipulated by m:op. m:V = object vector hV1 , : : :, Vu i. m:SQ = vector of sequence numbers hsq1 , : : :, sqn i. m:d = data. If m:op = read, m:dst denotes one processor which has a replica oia of oa . If m:op = write, m:dst shows all the processors which have the replicas of oa . If m is a response message of a request message m, m .tno is m.tno. A processor ps constructs a request message m with oph from Th as follows : m.tno = h m.t, m.no i := h t(Th), no(oph ) i; m:op := oph ; m:o := oa ; m:src := ps ; m:Va := Va (a = 1, : : :, u ); sqt := sqt + 1 for every pt in m:dst; m:sqj := sqj (j = 1, : : :, n); ps manipulates variables sq1 , : : :, sqn . Each time ps sends a message to pt , sqt is incremented by one. Then, ps sends m to every destination pt in m:dst. pt can detect a gap between messages sent by ps by checking the sequence number sqt , i.e. messages lost or unexpectedly delayed. pt manipulates variables rsr1 , : : :, rsqu , to receive message. On receipt of a message m from Ts in ps , there is no gap if m:sqt = rsqs . If m:sqt > rsqs , there is a gap m sent from ps where m:sqt > m :sqt rsqs . m is correctly received by pt if pt receives every message m where m .sqt < m.sqt . That is, pt receives every message which ps sends to ps before m. pt enqueues m in the receipt queue RQt. pt manipulates the object vector V as Va = max(Va , m:Va ) for a = 1, : : : u. When a replica ota is manipulated by m.op, pt sends back a response m of m to ps . If m.op = read, m carries data derived from ota in m .d and m the version number v(oa ) of ota . m :t := m:t; m :no := m:no; m :o := oa ; 0
0
0
0
0
0
0
0
0
0
0
0
0
m :op := response; m :Va := v(oa ); Ts has a vector of variables V = h V1 , : : :, Vu i where each Va is initially 0. On receipt of the response m from pt , Ts manipulates V , i.e. Va : = m .Va (= v(oja )) if Va < m .Va . Otherwise, Ts aborts. 0
0
0
0
0
4.3. Message delivery Messages in a receipt queue RQt are ordered according to the ordering rule. [Ordering rule] A message m1 precedes m2 (m1 ) m2 ) (1) if m1 :V m2 :V , (2) otherwise (m1 :V = m2 :V ), m1 :t < m2 :t. 2 p ps p u t < 0, 0 > < 0, 0 > < 0, 0 > TID =1s w(x) w(y) < 0, 2v1 > < 0, 0 > TID =1s r s (y)
m1
< 0, 2v1 >
Res
>
m2 w s (x)
m1
(1)
TID = 1u m
m1 .V
< m 2 .V
TID =2u m1
m2
w u(x)
(2)
m1 .t
< m 2 .t
Figure 2. Message ordering.
[Example] In Figure 2, h , i shows an object vector where
and are the values of variables Vx and Vy showing the versions of objects x and y , respectively. Initially h 0, 0 i in every processor. x and y are in every processor in Figure 2(1). A transaction T1s is computed in ps where t(T1s ) = 1s where s is the processor number of ps and 1 is the local linear clock. T1s first sends a write request m1 with w(x) and then sends m2 with w(y ) to ps , pt , and pu . The global event number tno(w(x)) = 1s0. m1 .V = h 1s0, 0 i and m1 .t = 1s. Then, w(y ) is computed in ps , pt , and pu . Here, tno(w(y)) = 1s1. T1s sends m2 with m2 .V = h 1s0, 1s1 i and m2 .t = 1s. pt receives m2 after m1 but pu receives m1 after m2 . Since m1 .V (= h 1s0, 0 i) < m2 .V (= h 1s0, 1s1 i), m1 ) m2 by the ordering rule. pt and pu receive m1 and m2 in “)”. After w(x) is computed, h 1s0, 0 i in every processor. h 1s0, 1s1 i in every processor after w(y ). In Figure 2(2), T2u is computed after T1u in pu . x is in every processor but y is in ps and pt . Before T1s is started, v(y) is assumed to be 2v1. T1s reads y and gets the version number 2v 1 of y from pt . T1s and T2u send m1 with ws (x) and m2 with wu (x) to every processor. tno(wt (x)) = 1s1 in
T1s and tno(wu (x)) = 2u0. Hence, m1 .V = h 1s1, 2v1 i and m2 .V = h 2u0, 0 i in T2u . m1 .V and m2 .V are uncomparable but m1 .t (= 1s) < m2 .t (= 2u). Hence, m1 ) m2 . 2 [Theorem] m1 causally precedes m2 (m1 ! m2 ) if m1 precedes m2 in the ordering rule ( m1 ) m2 ). [Proof] We assume that m1 ) m2 but m1 6! m2 . In Figure 2 (1), m1 ) m2 . Since m1 is sent before m2 , m1 ! m2 . It contradicts the assumption. 2 Here, the messages in RQt are ordered in “ )” by the ordering rule. m is a top message in RQt iff there is no message m1 in RQt such that m1 ) m. Each top message m1 in RQt still cannot be delivered because pt may have not yet received some message m2 causally preceding m1 due to the unexpected delay. We discuss what messages in
RQt can be delivered. [Definition] Let m be a message sent by pu to pt and stored in the receipt queue RQt . m is stable in pt iff (1) there is a message m1 from pu in RQt where m1 :sqt = m:sqt + 1, and (2) pt correctly receives a message m1 in RQt from every pu where m !m1 . 2 The top message m in RQt can be delivered if m is stable, because every message causally preceding m from the transaction point of view is surely delivered in RQt . [Definition] A top message m in RQt is ready in pt if pt computes no operation conflicting with m.op in a replica m.o. 2 In addition, only significant messages in RQt are delivered from RQt by the following procedure in order to reduce time for delivering messages.
[Delivery procedure] While each top in RQt is stable and ready, fm is dequeued from RQt ; m is delivered if m is significant, otherwise m is neglectedg. 2 [Example] In Figure 3, two transactions T1s and T2s are
computed in ps , and T3t and T4t are in pt . T1s , T2s , and T3t issue write requests w1s (x), w2s (x), and w3t (x). w1s (x) arrives at neither pt nor pu . w2s (x) does not arrive at pu . Since w1s (x) and w2s (x) are insignificant in pu , they are omitted. 2 If pt receives read requests r1 (x), : : :, rh (x) (h > 1). pt computes r(x) once and then sends the response to all the source processors of r1 (x), : : :, rm (x). Thus, the number of operations computed can be reduced. Each pt has variables D1 , : : :, Du where each Da shows the version number of the object oa to deliver messages. Each time a message m is delivered, Da := m:t if m:o = oa and m.t Da . Here, if m:t < Da , m is omitted. If some pu sends no message to pt , the messages in RQt cannot be stable. In order to resolve this problem, each pu sends a message without data to pt if pu had not sent any message to pt for some predetermined time units. is defined to be proportional to delay time between the processors. pt considers that pt loses a message from pu if
w 3t (x)
r 4t (x)
3t
p t
w1s(x)
ps
w 2s(x)
1s
r 2s(x)
Res
4t
Res
2s time
: omit
: transactions
Figure 3. Omission of write operations.
(1) pt does not receive any message from pu for time units or (2) pt detects a gap in the receipt sequence of messages. If pt detects that m is lost, pt requires pu to resend m. pu considers that pt may lose m unless pu receives the receipt confirmation of m from pt in 2 after pu sends m to pt .
5. Evaluation In the TBCO protocol, messages are ordered by the object vector whose size is the number u of objects in the system. On the other hand, the size of the vector clock is given by the number of processes, i.e. servers and subtransactions in the group. Messages sent and received by a transaction are related but messages exchanged among transactions may not be related. Hence, it is critical to consider a group to be composed of subtransactions to causally order only significant messages. Each time transactions are initiated and terminated, the scheme of the vector clock has to change, i.e. subtransactions and servers have to be resynchronized. Since the object vector only depends on the number of the objects, the protocol can be adopted to transaction-oriented applications.
Traditional
1.00
TBCO
Ratio of operations
0.90 0.80
sages transmitted in the network. We evaluate the TBCO protocol in terms of the number of messages causally ordered compared with the traditional message-based protocol. The TBCO protocol is realized in threads of Ultra Sparc stations super server 6400 with 10 Ultra Sparcs. Each processor is bound to one Ultra Sparc. TCP[14] is used to exchange messages among the processors. Each processor randomly creates transactions each of which randomly manipulates objects by read and write. There are three objects which are fully replicated in all the processors. Each processor randomly initiates totally twenty transactions each of which issues ten arbitrary kinds of operations on arbitrary objects. Figure 4 shows the total number of requests computed in the objects for the ratio of write requests issued by the transactions for three processors. The more write requests are issued, the more messages are sent to the replicas. The write ratio 1.0 means that only write requests are issued and 0.0 shows that only read requests are issued. The figure shows that the more write messages are issued, the more messages can be omitted. For example, only about 68% of the messages transmitted in the messagebased protocol are transmitted in the TBCO protocol in case that all the messages are writes, i.e. the write ratio is 1.0 while 50% for the write ratio 0.4 and 90% for the write ratio 0.0.
Traditional TBCO
w:r = 9:1
1.00
Ratio of delivered messages
pu
0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00
1
0.70
2
3
4
5
6
7
8
9
10
Number of processors
0.60
Figure 5. Ratio of messages.
0.50 0.40 0.30 0.00
0.20
0.40
0.60
0.80
1.00
write / (write + read)
Figure 4. Ratio of operations. The TBCO protocol does not causally order all the mes-
In Figures 5 and 6, the vertical axis indicates the ratio of the number of messages delivered to the number of messages transmitted in case of ten processors in the traditional message-based protocol. The dotted line shows TBCO and the solid line indicates the traditional message-based protocol. Figures 5 and 6 show cases that 90% and 50% of the operations issued by each transaction are writes, respectively. These figures show the number of messages delivered
Ratio of delivered messages
1.00
[4] Lamport, L., “Time, Clocks, and the Ordering of Events in a Distributed System,” Comm. ACM, Vol.21, No.7, pp.558–565, 1978.
Traditional TBCO
w:r = 5:5
[5] Leong, H. V. and Agrawal, D., “Using Message Semantics to Reduce Rollback in Optimistic Message Logging Recovery Schemes,” Proc. of IEEE ICDCS14, pp.227?234, 1994.
0.90 0.80 0.70 0.60
[6] Mattern, F., “Virtual Time and Global States of Distributed Systems,” Parallel and Distributed Algorithms (Cosnard, M. and Quinton, P. eds.), North-Holland, pp.215?226, 1989.
0.50 0.40 0.30 0.20 0.10 0.00
1
2
3
4
5
6
7
8
9
10
Number of processors
Figure 6. Ratio of messages.
can be reduced by the TBCO protocol. About 30% and 20% of messages transmitted in the network are reduced for the write ratios 90% and 50%, respectively.
6. Concluding Remarks This paper has discussed what messages have to be causally ordered in replicated objects with read and write from the application point of view. We have proposed the novel object vector for causally preceding messages based on the transaction concept. In the TBCO protocol, only the messages to be meaningfully preceded for the applications can be causally ordered. We have also discussed a way for omitting messages which are not significant for the applications. We have shown the TBCO protocol implies fewer number of operations computed than the protocols which causally order all the messages transmitted in the network.
References [1] Bernstein, P. A., Hadzilacos, V. and Goodman, N., “Concurrency Control and Recovery in Database Systems,” Addison-Wesley, pp.25?45, 1987. [2] Birman, K., “Lightweight Causal and Atomic Group Multicast,” ACM Trans. on Computer Systems, pp.272?290, 1991. [3] Cheriton, D. R. and Skeen, D., “Understanding the Limitations of Causally and Totally Ordered Communication,” Proc. of the ACM SIGOPS’93, pp.44?57, 1993.
[7] Melliar-Smith, P. M., Moser, L. E., and Agrawala, V., “Broadcast Protocols for Distributed Systems,” IEEE Trans. on Parallel and Distributed Systems, Vol.1, No.1, pp.17–25, 1990. [8] Nakamura, A. and Takizawa, M., “Reliable Broadcast Protocol for Selectively Ordering PDUs,” Proc. of IEEE ICDCS-11, pp.239–246, 1991. [9] Nakamura, A., Tachikawa, T., and Takizawa, M., “Causally Ordering Broadcast Protocol,” Proc. of IEEE ICDCS-14, pp.48?55, 1994. [10] Object Management Group Inc.,“The Common Object Request Broker Architecture and Specification,” Rev. 2.0, 1995. [11] Raynal, M. and Ahamad, M., “Exploiting Write Semantics in Implementing Partially Replicated Causal Objects,” IRISA Research Report, PI-1080, 1997. [12] Tachikawa, T. and Takizawa, M., “Distributed Protocol for Selective Intra-group Communication,” Proc. of IEEE ICNP-95, pp.234?241, 1995. [13] Tachikawa, T. and Takizawa, M., “Object-Based Message Ordering in Group Communication,” Proc. of the IEEE 3rd Int’l Workshop on Object-oriented Realtime Dependable Systems (WORDS’97), pp. 315?322, 1997. [14] “Transmission Control Protocol,” RFC793, pp. 1–26, 1981.
