tion processes do not mind if some packets are lost and ... e.g. teleconference and virtual university [9]. .... We make the following assumptions on the stream.
Multimedia Intra-Group Communication Protocol Takayuki Tachikawa
and
Makoto Takizawa
Dept. of Computers and Systems Engineering Tokyo Denki University Ishizaka, Hatoyama, Hiki-gun, Saitama 350-03, JAPAN E-mail
ftachi,
g
taki @takilab.k.dendai.ac.jp
Abstract
In distributed applications, a group of multiple application processes have to send and receive multimedia messages by using the high-speed network. The multimedia message at the application level is decomposed into multiple smaller packets which are transmitted in the communication system. In this paper, we discuss the atomic and ordered delivery of messages at the application level rather than the system level. In some multimedia applications, the application processes do not mind if some packets are lost and in what order packets from di erent processes are received. The application process speci es the minimum receipt ratio " ( 1) showing how many percentages of whole data in each message the destination processes have to be received at least. The communication system delivers the packets to the destinations in the group so as to satisfy the receipt ratio ". The protocol is based on the fully distributed control scheme, i.e. no master controller. 1
Introduction
By using the high-speed networks [1], various distributed multimedia applications have been realized, e.g. teleconference and virtual university [9]. Here, a group of multiple processes are cooperated and multimedia data like video and voice are exchanged among the processes in the group. In the group communication, each application process either delivers messages to all the destinations in the group or none of them, i.e. atomic delivery . The application messages are decomposed into smaller packets which are transmitted in the network. In the high-speed networks [1], the processes may fail to receive packets due to bu er overrun and the packets may be lost due to the congestion in the network. According to the advances of VLSI and software technologies, processes can be considered to be more reliable than the packet loss. Thus, the packet loss is considered as the major failure in the network. In the presence of packet loss, the messages have to be delivered to all the destinations. In addition to providing the atomic delivery, the group communication protocol has to provide the application processes with the causally ordered (CO) delivery of messages. In the CO service [3, 18], messages are delivered according to the Lamport's happen -before precedence relation [11] among the sending events of the messages. That is, if a message a is sent
after receiving b, every common destination process of and b receives a after b. Thus, the CO service takes into account how each process looks at the order of sending and receipt events in another process. The CO service is required in distributed applications like fault-tolerant systems [26] and CSCW [6]. In the distributed applications, multimedia data like video and voice are exchanged among the processes. The application processes transmit messages including multimedia data to the destinations in the group. The system process in the communication system takes the message from the application process and decomposes it to smaller packets which are units of data transmission in the network. The system process sends and receives the packets by using the highspeed network. The system process may receive the packets out of order and may not receive the packets. The group communication protocols presented so far [3, 5, 8, 10, 15{18, 21] have discussed how to support the atomic and ordered delivery of packets while the application processes may not require the system processes to support such strict service for all packets. The application processes may not care loss of some packets in a message. In this paper, the application process speci es how many percentages " ( 1) of data included in each message have to be received by each destination. If the system process could receive packets at a larger receipt ratio than " even if it does not receive some packets, it is allowed to deliver them to the application processes and does not require the sender to send the packets lost again. In addition, the application processes are interested only in the receipt order of messages rather than packets. Hence, the messages are delivered to the application processes in a causal order. The overhead of the system processes can be reduced because the processes do not consume the computation to order the packets and recover from the packet loss. We would like to present a group communication protocol named an "-CO which supports the atomic and causally ordered delivery of messages at the presence of packet loss given the receipt ratio constraint ". In section 2, we model the communication system. In section 3, we discuss a concept of causal ordering. In section 4, we present the data transmission procedure. Finally, we present the evaluation of the "-CO protocol in section 5.
a
2
Model
The distributed system is composed of multiple computers interconnected by the high-speed network. The distributed application is realized by the cooperation of a group G of multiple application processes A1 , ..., An (n 2). For example, multiple agent processes are communicated in the teleconference. Each Ai sends and receives messages with the other application processes in the group G through the service access point (SAP) Ci supported by the underlying communication system. The communication system supports the atomic and ordered delivery of messages for the application processes by using the high-speed network.
1 11
A1
application layer system layer network layer
6C
1
?
1 11
S1
6N ?
1
Ai
6Ci ?
Si
6Ni ?
111
111
high-speed network
An
application process
6Cn k system SAP ? system
Sn
process
6Nn ? k network SAP
Figure 1: System model A system process Si in the communication system takes a message sent by Ai at Ci and decomposes it into packets which are units of data transmission in the network. On receipt of packets, Si assembles them into a message and then delivers it to Ai . In Figure 2, an application process Ai sends a message a into Aj in a group. Si takes a, decomposes a into three packets a1 , a2 , and a3 , and sends them to Sj by using the network. On receipt of a1 , a2 , and a3 , Sj assembles them into the message a and delivers a to Aj . A message b sent by Aj is delivered into Ai in the same way as a. A sequence of packets decomposed from a message m is referred to as stream of m. h a1 , a2 , a3 i is a stream of a in Figure 2. Ai
?
a
Si
a1 Sj
a2
a3
R R R a
Aj
6 b b b 1
?
6b
2
time
Figure 2: Messages and packets In the high-speed network, packets may be lost due to bu er overrun and congestion. Hence, the system
processes have to support an ordered delivery of messages in the presence of packet loss. For two packets ai and aj decomposed from a message a, ai depends on aj (written as ai j= aj ) i ai cannot be delivered to the application process if aj is not. For example, suppose that the key for deciphering ai is included in aj . Unless aj is received, the application process cannot accept ai because ai cannot be deciphered, i.e. ai j= aj . ai and aj are independent i neither ai j= aj nor aj j= ai . ai precedes aj (ai ` aj ) i ai has to be delivered to the application process before aj . We make the following assumptions on the stream h a1, ..., ah i of message a. [Assumptions] For every pair of packets ai and aj , (1) ai and aj are independent, and (2) either ai ` aj or aj ` ai . 2 That is, the packets in the stream is totally ordered. 3
Ordered Delivery
The communication system supports a group of application processes A1 , ..., An with the ordered delivery of messages. Let si[m] and ri [m] denote the sending and receipt events of a message m in an application process Ai , respectively. Here, \!" denotes the happen -before relation [11] among the events. The communication system supports the causally ordered delivery of messages if the following rule is satis ed: [Happen-before relation] For every pair of events e1 and e2 , e1 causally precedes e2 (e1 ! e2 ) i (1) e1 happens before e2 in Ai , (2) for some (not necessarily di erent) Ai and Aj , there exists some message a such that e1 = si [a] and e2 = rj [a], or (3) for some event e3 , e1 ! e3 ! e2 . 2 Suppose that a process Ai sends a message a to Aj and Ak , and Aj sends b after receiving a. Here, Ak has to receive a before b because si [a] ! sj [b]. The communication system has to deliver messages in the causal order. [Causally ordered delivery] For every two messages a and b, a is delivered before b in Ai (ri[a] ! rj [b]) if si [a] ! sj [b]. 2 In the multimedia applications, messages include large volume of data. The messages are decomposed into packets, i.e. units of data transmission in the network. Here, let ij (a) denote sequence of packets of a which Aj receives from Ai . Since the packets may lost in the network, ij (a) a. [Full delivery] If Ai sends a message a to Aj , the communication system delivers Aj the same content as a, i.e. ij (a) = a. 2 In the conventional communication system, messages sent by Ai are either delivered or not, i.e. messages have to be fully accepted. In multimedia applications, a full delivery may not be required. For example, even if some part of voice message is lost in the network, the destination process can accept the message. [Partial delivery] If Ai sends a message a to Aj , the communication system delivers ij (a) to Aj , i.e.
a. 2 Here, let jmj denote the size of a message m. A receipt ratio ij (m) is de ned to be jij (m)j / jmj, i.e. how many percentages of data in m to be delivered to Aj . [-delivery] The communication system partially delivers ij (a) to Aj if ij (a) for a given constant ( 1). 2 Ai gives the minimum receipt ratio to the system process Si when Ai would send a message a. Even if Si does not receive some packets of a, Si noti es Ai of the acceptance of a if Si could receive more packets of a than ". In the MPEG, suppose that B-frame b is sent after I-frame a. For a, " is required to be 1, i.e. the whole I-frame has to be received, and for b, " 1. In Figure 3, the application process Ai sends a message a, and the system process Si decomposes a into a stream h a1 , ..., a5 i. Si sends the packets to Sj , and Sj receives a1 , a2 , a3 , and a5 but not a4 . ij (a) = h a1 , a2 , a3 , a5 i and ij (a) = jij (a)j/jaj = 4/5 = 0.8. Suppose that Ai requires Si to send a with "i = 0.6. Since ij (a) = 0.8 > "i , Sj delivers ij (a) to Aj . If "i = 0.9, Sj cannot deliver ij (a) to Aj and requires Si to send a4 again. ij (a)
a1 Si Sj
a2
a3
R R R a1
a2
a4
R2 a3
a5
-
R a5
time
Figure 3: Receipt of stream Next, let us consider the ordered relation among the packets. [Correct order] For every pair of streams of a and b sent to Si , Si delivers correctly a and b i (1) Si constructs a stream of a from the packets, where the packets are ordered in the sending order, and (2) Si delivers the stream of a to Ai after b. 2 (1) means that Si has to receive packets from each Sj in the sending order. From condition (2), Si can deliver a stream to Ai independently of other messages. In the communication system, an event can be considered to be atomic, i.e. two events cannot occur simultaneously in the same process. At the application level, it takes longer time for an event to occur because messages are much longer than the packets. Hence, multiple events may be interleaved, e.g. processes are sending messages while receiving. In Figure 4(1), the application process Aj sends a to Ah and Ak . After receiving a, Ah sends b to Ak (a b). In Figure 4(2), Ah sends b before receiving a. Ak receives a and b. Here, Ak does not need to consider the causally ordered delivery of a and b since a and b are causally coincident (a k b). Figure 5 shows the data transmission among three system processes Sj , Sh , and Sk . Sj sends a stream h a1 , a2 i of a and Sk sends h b1 , b2 , b3 i of b. In Figure 5(1), Sh
Aj
Ah
Ak
Aj
a
Ah a
s
b
b
time
? ~?
?
Ak
(1)
?
s w ?
?
(2)
Figure 4: Causally ordered delivery (Apl.) sends the rst packet b1 of b after receiving the last packet at of a. Sk receives a2 before b1 . Figure 4(1) shows the data transmission among Aj , Ah , and Ak for Figure 5(1). Ah sends b to Ak after receiving a and Ak receives b after a, a b. Next, in Figure 5(2), Sh starts to send the stream of b before receiving all the packets in a. Figure 4(2) shows the application level data transmission of Figure 5(2). Since Ah sends b not after receiving a, a k b in Ak . In Figure 5(2), suppose that the communication system supports the causally ordered delivery of packets. Here, suppose that Sk receives the packets in a sequence < a1 b1 a2 b2 b3 ]. Sk assembles the stream of packets received into the message and then delivers it to Ak . Sk may deliver a before b because a1 is received before b1 . If so, the causal ordering of packets of a and b except a1 and b1 is meaningless for the application process. In this paper, we would like to discuss how to causally deliver messages at the application level rather than the system level in order to reduce the processing overhead of the communication system. Sj
Sh a1 a2
j j
Sk
z z b b z b z ? z? 1
2
time
?
3
(1)
Sj
Sh a1
z z j z b b z ? ? z? a2
j
Sk
b1
2 3
(2)
Figure 5: Causally ordered delivery (Sys.) Next, we would like to discuss a case that some packet is lost. In Figure 6, Sj sends Sh a stream h a1 , a2 , a3 , a4 , a5 i of a message a. In Figure 6(1), Sh does not receive a5 , and Sh starts to send Sk a stream h b1 , b2 i of b before delivering a to the application process
k
!
!
Ah . Here, a b since neither sj [a] sh [b] nor sh [b] sj [a]. In Figure 6(2), Sh does not receive a2 . Suppose that the receipt ratio jh (a) gets greater than " in Sh on receipt of a5 . Sh delivers jh (a) = a1 , a3 , a4 , a5 to Ah . Then, Sh takes b from Ah and sends b1 and b2 to Sk. Here, sj [a] sh [b] since rh [a] sh [b]. Hence, a b.
!
Sj
Sh
z z z z 2z
a1 a2 a3 a4 a5
time
h !
?
Sk
Sj
b1
(1)
Sh
z 2z z z z
a1 a2 a3 a4 a5
z ? z?
i
?
Sk
b1
z ? z?
(2)
Figure 6: Causally ordered delivery We assume that the communication network supports the system processes with the following services: (1) the system processes can receive packets sent by each process in the sending order, i.e. ri [a1 ] ! ri [a2 ] if sj [a1 ] ! sj [a2 ], and (2) the system processes may not receive some packets. The service supported by the high-speed network is referred to as multi -channel (MC) service. This is a model of high-speed networks like the ATM networks, where the processes can receive packets in the sending order while the processes may lose some of them due to the bu er overrun and congestion. In this paper, we would like to discuss an "-CO protocol which supports the causally ordered and "delivery of messages for a group of application processes by using the MC network. 4
"-CO Protocol
We would like to present the "-CO protocol which supports a group G = h S1 , ..., Sn i. We assume that every message is sent to all the application processes in G , i.e. we do not consider selective group communication discussed in [16, 21]. 4.1
Variables
The system process Si manipulates two data units, i.e. packets and streams of packets. On taking a message a from the application process Ai , Si decomposes a into a stream h a1 , ..., am i of a and sends the packets a1 , ..., am in the network. Si receives the packets and makes them into a stream. Then, Si delivers the stream to Ai. Each stream is identi ed by unique stream number (ST N ) and each packet in the stream is identi ed by unique packet number (P SQ). Each time Si takes a message and makes a stream for the message, and sends a packet in the stream, ST N and
P SQ are incremented by one, respectively. Si manipulates the following variables to send and receive the streams (for j , k = 1, ..., n). " = minimum receipt ratio given by Ai . ST N = sequence number of stream which Si expects to send next or is sending at present. REQj = sequence number of stream from Sj which Si has accepted most recently to ". DLVj = sequence number of stream from Sj which Si has delivered most recently to Ai. ALjk = sequence number of stream which Si knows Sj accepted from Sk . A stream whose packets are being sent by Si is referred to as current . Si manipulates the following variables to send and receive packets (for j = 1, ..., n). P SQ = sequence number of packet which Si expects to send next in the current stream. P RQj = sequence number of packet which Si expects to receive next from Sj . LSTj = number of packets lost in a current stream being received from Sj . BU Fj = available bu er size of Sj which Si knows. Here, let minALj denote minimum in ALj 1 , ..., ALjn (j = 1, ..., n). Let minBU F be minimum in BU F1 , ..., BU Fn . Let a be a message sent by Ai whose stream is h a1 , ..., ah i. Each packet ak of a has the following elds (k = 1, ..., h). SRC = system process which sends ak , i.e. Si . ST N = stream sequence number of a. P SQ = packet sequence number of ak . The top packet a1 of a has the following additional elds for the stream of a (j = 1, ..., n). P total = number of packets in a, i.e. jaj =h. ACKj = sequence number (STN) of stream which Si has accepted most recently from Sj . DCKj = STN of stream from Sj which Si has delivered most recently to Ai . BU F = available bu er size of Si . 4.2
Transmission
Application process Ai issues a message a to the system process Si . Si takes a and sends the stream h a1 , ..., ah i of a by using the MC service. Here, let W be a maximum window size and H be a maximum number of packets in a stream, i.e. h H . Let f be the maximum packet size. Si has to have bu er to store at least O(n) packets [22]. Each Si sends a packet ak by the following data transmission procedure if the following ow condition holds. Here, enqueue(L; a) means that a message a is enqueued into a queue L. send(a) means that a packet a is sent to all the system processes S1 , ..., Sn by using the MC network. Si has a sending queue SLi to store packets which Si sends. Each time Si starts to send a stream, ST N is incremented by one. Each time Si sends a packet in the stream, P SQ is incremented by
one. The top packet includes the information of the stream, i.e. P total, ACKj , DCKj , and BU F (j = 1, ..., n). [Flow condition] minALi STN < minALi + min (W , minBUF / (H 2 f 2 n)). 2
f
[Data transmission procedure] for k = 1 to h ak :STN := STN ; ak :PSQ := PSQ ; PSQ := PSQ if k = 1,
f
f
:= h; := REQ j (j = 1,...,n); a1 :DCK j := DLV j (j = 1,...,n); a1 :BUF := available bu er size of Si ;g enqueue (SLi , ak ); if the ow condition holds, send (ak ); else waits; g STN := STN + 1; DLV i := DLV i +1; g 2 Acceptance of packet
Let ak be a packet sent by Sj . On receipt of ak , Si accepts ak by the acceptance procedure and stores ak into the receipt queue RRLij .
[Packet acceptance(PAC) condition] ak :STN = REQ j + 1. [Packet acceptance(PAC) procedure] LSTj := LSTj + ( ak :PSQ PSQ j ); PSQ j := ak :SPQ + 1; if k = 1, ALhj := a1 :ACKh (h = 1, ..., n); BUF j := ak :BUF ; enqueue (RRLij , ak ); In RRLij , the packets from Sj are stored in the sending order. On receipt of ak , if ak .P SQ > P SQj , Si nds that Si has not received a packet ah where ak .P SQ > ah .P SQ P SQj . Here, Si does not yet require Sj to retransmit ah and accept ak . The number of packets lost is accumulated in LSTj . If LSTj = a (= a1 .P total) > 1 ", Si requires Sj to resend
2
0
f
f
g g2
jj
0
the packets lost. The top packet a1 in the stream of a includes the additional information on a. Hence, on receipt of a1 , the following procedure is invoked for a1 . Suppose that Si does not receive a1 from Sj . Since a1 includes additional elds P total, ACK1 , ..., ACKn , DCK1 , ..., DCKn , and BU F , Si cannot accept a. That is, ak j= a1 (k 2). Hence, if Si nds the loss of a1 , Si requires Sj to send a1 again. Another way is that a1 can be sent more than one time. In order to prevent a from su ering from the burst failure, the replicas of a1 are sent not continuously. Even if one replica of a1 is lost, Si can receive another replica of a1 . 4.4
Acceptance of stream
0
2 f
g2
+ 1;
a1 :Ptotal a1 :ACK j
4.3
[Acceptance(ACC) condition] (a1 :P total LSTj )/ a1 :P total " [Acceptance(ACC) procedure] If the ACC condition holds for the packets in RRLij , the packets in RRLij are assembled into a stream a; enqueue (CRLij , a): REQj := a1 :ST N ; P RQj := 1; LSTj := 0; The stream ij (a) accepted from Sj is enqueued into a queue CRLij . Here, the stream number of ij (a) and
Here, let the stream of a sent by Sj be h a1 , ..., ah i. If Si accepts ah of a or accepts no packet for some time units, Si decides whether to accept the stream. If Si accepts more packets of a than " 2 h, Si accepts ij (a), i.e. packets in a which Si has accepted. If the following ACC condition holds for a, Si accepts a by the ACC procedure.
acceptance con rmation to Sk are stored in a1 .ST N and a1 .DCKk (k = 1, ..., n). Unless the ACC condition holds for a, Si requires Sj to send again the packets which Si has lost. Sj sends an RET(retransmission) packet to Sj which carries the list LSTj of ST N s of the packets which Si has lost. On receipt of the RET packet from Sj , Sj sends the packets to Si again. On receipt of the packets retransmitted, Si enqueues them into RRLij and invokes the ACC procedure again. If the streams satisfy the ACC, they are inserted into CRLi in . 4.5
Causally ordered and atomic delivery
Suppose that Si accepts a stream ij (a) of a message a sent by Sj . The streams in CRLi are ordered in by the following condition. [Causality(C) condition] For every pair of streams a and b, a b i (1) a:ST N < b:ST N if a:SRC = b:SRC , (2) a:ST N b:DCKj if a:SRC (= Sj ) 6= b:SRC . 2 If Si sends a and b, a is sent before b (sj [a] ! sj [b]) from the condition (1). If not, rj [a] ! sk [b] from the condition (2). Hence, sj [a] ! sk [b] (b1 :SRC = Sk ), i.e. a b at the application level. [Causality Ordering (C) procedure] accepted is inserted into CRLi in
ij (a)
f according
to the C condition. 2 Si has to know that all the system processes have accepted a. If a satis es the following acknowledgment (ACK) condition, Si considers a as accepted by all the system processes, and delivers a to Ai by the following procedure. deliver (a) means that Si delivers a to Ai . Here, a is referred to as acknowledged in G . dequeue(CRLi ; a) means that the top a of CRLi is removed.
[Acknowledgment (ACK) condition] (where a:SRC = Sj ). [Passing message (P) procedure] while (the stream of the top a (where a:SRC = Sj ) of CRLi satis es the ACK condition) dequeue (CRLi , a); deliver(a): if j = i DLVj := a:SEQ ; [Theorem 5.1] Suppose that a stream a is delivered to Ai. For every stream b, b is delivered to Ai before
a:SEQ < minALj
6
f
gg2
2
f
if b a. Suppose that a:SRC = Sk , b:SRC = Sj , and b is not acknowledged and a is acknowledged by Si . Since a is acknowledged, Si has accepted some stream ch from every Sh where a:STN < ch :ACK k . That is, Si accepts ck such that sj [b] ! sh [ch ] for every Sh . If Si does not receive b, Si nds it on receipt of ch using a
[Proof ]
the failure condition. Since ch is not accepted, a is not acknowledged. If Si accepts b, b is acknowledged. It contradicts the assumption. 2 [Example 5.1] Figure 7 shows the data transmission in a group supported by three system processes S1 , S2 , and S3 . The application processes A1 , A2 , and A3 supported by S1 , S2 , and S3 send messages a and d, b and e, and c, respectively. Here, a.ST N = 4, d.ST N = 5, b.ST N = 1, e.ST N = 2, and c.ST N = 10. Suppose that initially, REQ1 = 4, REQ2 = 1, and REQ3 = 10 in every process. First, S1 takes a from A1 and sends the stream h a1 , ..., a5 i. S2 accepts all the packets in the stream while S3 does not accept a4 . Suppose that A3 requires that " be 0.7. Since 13 (a) = 4 / 5 = 0.8 > 0.7, S3 accepts the stream 13 (a) = h a1 , a2 , a3, a5 i. The packets are stored in RRLj 1 (j = 1, 2, 3). Here, for a message a sent by Si , let (a)j denote the stream ij (a) accepted by Sj . On receipt of a 5, Sj constructs the stream (a)j from the packets in RRLj 1 . Sj enqueues (a)j to CRLj (j = 1, 2, 3). After accepting (a)i , S2 sends h b1 b2 b3 i, and S1 and S2 accept the packets. b1 carries the acceptance con rmation of (a)2 and the stream (b)i is enqueued into RRLi2 (i = 1, 2). S3 cannot construct (b)3 since S3 does not accept b2 and 23 (b) = 2 / 3 < 0.7. S3 requires S2 to send b2 again. The stream h c1 c2 i is sent by S3 . c1 and c2 are accepted by all the processes and (c)i is enqueued into CRLi (i = 1, 2, 3). On receipt of d1 which carries the acceptance con rmation of (a)i , AL in Si changes as follows (i = 1, 2, 3), " # 5 5 5 AL = 2 1 1 : 10 10 10 Here, since (a)i .ST N < minAL1 = min( AL11 , AL21 , AL31 ) = 5, (a)i is acknowledged and delivered to Ai by using the P procedure. That is, (a)i is dequeued from CRLi . On receipt of streams of d and e, h d1 d2 d3 i and h e1 e2 i are accepted, and (d)i and (e)i are constructed by Si since every process accepts all the packets in d and e. By using the C procedure, (d)i and (e)i are enqueued into CRLi in every process. (b)i precedes (e)i in CRLi since (b)i (e)i (i = 1, 2). On receipt of b2 retransmitted, S3 accepts (b)3 and enqueues it into CRL3 since 23 (b) = 1 > 0.7. Then, S3 enqueues (b)3 into CRL3 such that (b)3 precedes (e)3 in CRL3 since (b)3 (e)3 by using the C procedure. CRL3 is manipulated as follows:
2
(1) (2) (3) (4)
< < <
