quired in the world-wide multimedia applications. Here, let us consider a world-wide teleconference among ve processes K, U, O, S, and H at Keele in UK, UCLA ...
Flexible Wide-area Group Communication Protocols { International Experiments { Takayuki Tachikawa
Hiroaki Higaki
Makoto Takizawa
Tokyo Denki University Mario Gerla
Ming T. Liu
Misbah Deen
UCLA
Ohio State University
Keele University
Abstract In distributed applications, a group of multiple processes is required to be cooperated by exchanging multimedia data. In addition, world-wide distributed applications are being realized by using the Internet. The traditional group communication protocols assume that every pair of processes support the same and xed delay time and reliability level. In world-wide multimedia applications, the assumption does not hold. We discuss high-speed protocols which can change the ways for distributing messages to multiple destinations and retransmitting messages to processes losing the messages in the wide-area group in the change of the delay and reliability. We present the evaluation of the protocols in the world-wide environment.
1 Introduction
In distributed applications like teleconferences, a group of multiple processes is rst established and then the processes in the group are cooperated. Group communication protocols support a group of the processes with the reliable and ordered delivery of messages to multiple destinations. ISIS(CBCAST)[4] and others[16, 23] support the causally ordered delivery. ISIS(ABCAST)[4], Ameoba[12], Trans/Total[15], Rampart[17] and others[5, 21] support the totally ordered delivery. The group communication protocols discussed so far assume that every pair of processes support the same and xed delay time and the same level of reliability. That is, only processes in a local area are cooperated. Group communication among multiple processes distributed in a wide area is required in the world-wide multimedia applications. Here, let us consider a world-wide teleconference among ve processes K , U , O, S , and H at Keele
in UK, UCLA and Ohio State Univ., and Tohoku Univ. at Sendai, and Tokyo Denki Univ. at Hatoyama, Japan, respectively. In the Internet, it takes about 60[msec] to propagate a message within Japan while taking about 240[msec] between Tokyo and Europe. Over than 10% of messages are lost between Japan and Europe while less than 1% are lost within Japan. Thus, it is essential to consider a wide-area group of processes where the delay times and reliability levels between the processes are signi cantly di erent[8, 9, 11]. In the wide-area group, the time for delivering messages to the destinations is dominated by the longest delay between the processes. For example, if S sends a message m to H and K , S has to wait for the response from K after having received the response from H . Next, suppose that K sends another message m to H and S , respectively. If S loses m, S requires the sender K to resend m. The delay time between S and K is about four times longer than S and H . If the destination H resends m, the delay time for delivering m can be reduced. A wide-area group G can be decomposed into disjoint subgroups G1 ; : : : ; G (sg 2)[8, 24]. Holbrook, et. al.[9] present a way where each subgroup has a message log to retransmit messages. The protocols[8, 9, 11, 24] are discussed to reduce the number of messages in large-scale groups. Jones, et. al.[11] discuss the saturation protocol where the sender sends multiple replicas of a message m to the destinations. In this paper, we discuss the destination replication where the destinations forward m to the other destinations on receipt of m. In sections 2 and 3, we present a system model and the measurement of the delay time and message loss ratio in the network. In section 4, we discuss ways to deliver messages in a wide-area group. In section 5, we present protocols in a wide-area group. In section 6, we present an evaluation of the protocols. sg
AP 1
Application
Group Communication Protocol
TP 1
Transport
AP i
AP n
GCP packet NP1
Network
TP n
TP i ICMP packet NPi
Internet Protocol
NPn
Figure 1: Distributed system 2 System Model
averages of and " , respectively. T P is nearer to than to T P if < . Here, we assume that = and " = " for every pair of T P and T P . ij
A distributed system is composed of application, layers as shown in Figure 1. A group of n( 2) application processes AP1 ; : : : ; AP are communicated by using the underlying group communication service supported by transport processes T P1 ; : : : ; T P . A group G = fT P1 ; : : : ; T P g of the transport processes is considered to support each pair of processes T P and T P with a logical channel. Data units transmitted at the transport layer are packets. T P sends a packet to T P by the channel. The network layer provides the IP (Internet Protocol) service for the transport layer. That is, IP packets may be lost, out of order, and duplicated. The cooperation of the processes at the transport layer is coordinated by group communication (GC) and group management (GM) protocols. The GC protocol rst establishes a group G and then reliably and causally[4] delivers packets to the processes in G. The GM protocol is used for monitoring and managing the membership of G. AP requests T P to send an application stream s . T P decomposes s into packets, and sends them to the destinations in G. The destination T P reassembles the packets into a stream s , and delivers s to AP . Packets decomposed from a stream are messages. Let dst(m) be a set of destination processes of a message m in G. A transport process T P has to know the delay time and message loss ratio " with each T P in G. In the GM protocol, T P requests periodically the network layer to transmit two kinds of ICMP (Internet Control Message Protocol) packets to all the processes in G: \Timestamp" and \Timestamp Reply". T P can know when \Timestamp" sent by T P is received by T P and when \Timestamp Reply" received by T P is sent by T P , i.e. round trip time. T P calculates by using the time information. In addition, the GM protocol monitors the ratio " of packets lost between each pair of T P and T P . Here, and " show the transport, and network
T Pj ij ji
ij
k
ij
ij
i
ik
ji
i
j
n
n
n
i
j
i
j
i
i
i
i
i
j
j
j
j
i
ij
ij
j
i
i
i
j
i
j
i
ij
ij
i
j
ij
ij
3
Network Measurement
In the world-wide Internet environment including processes H , S , U , O, and K in the Unix workstations, we measure the delay time and the message loss ratio " for every pair of processes T P and T P . First, the delay times among the processes H at Hatoyama, S at Sendai, U at UCLA, and K at Keele are measured. H sends 5,000 ICMP echo packets to S , U , and K . Each destination process monitors the delay time of each packet received. Here, R (t) shows the ratio of the packets, for which it takes t time units to arrive at the destination T P from T P , to the total number of packets transmitted, i.e. 5,000. In Figure 2, Sendai, UCLA, and Keele show R (t), R (t), and R (t), respectively. The longer the distance is, the more uctuating the receipt ratio is. For example, 80% of packets arrive at S from HRwithin 50[msec] after the fastest packet arrives, i.e. 3080 R (t)dt = 0:8. Here, it takes about 30[msec] for the fastest packet to arrive at S . On the other hand, 80% of packets arrive at U and K within 85[msec] and 137[msec] after the fastest packet arrives, respectively. If a process T P does not receive the receipt con rmation from T P in some time units after sending a message m, T P considers that m is lost. If the timeout period is 100[msec] (= 2 2 50[msec]), about 20% of packets are considered to be lost between H and S . In order to receive more than 80% of packets transmitted between H and U and between H and K , the timeout periods have to be larger than 170[msec] (= 2 2 85[msec]) and 274[msec] (= 2 2 137[msec]), respectively. The statistics of message loss ratios for S , U , and K are obtained as shown in Table 1. Here, MIN and ij
ij
i
j
ij
j
i
HS
HU
HK
HS
i
j
i
message loss ratio and delay get larger, the destination retransmission is adopted.
0.05
Sendai
Delay a day
0.04
600
500
0.03
UCLA
400 Delay(RTT) [msec]
Receipt ratio R(t)
H-K H-U H-S S-K S-U U-K
0.02
0.01
300
200
Keele 0 0
100
200
300 Delay t [msec]
400
500
100
Figure 2: Message receipt ratio v.s. delay.
0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Time [hour(JST)]
MAX shows the minimum and maximum delay time, respectively. For example, it takes about 119.5[msec] to deliver the fastest packet from RH to U . Only 1% of the messages are lost for S , i.e. 30 R (t)dt = 1:0. However, 8.3% and 11.7% of the packets are lost for U and K , respectively. It takes averagely 241.4[msec] to deliver a message from H to K while even the fastest packet takes 164.5[msec]. Table 1 shows that the longer the distance is, the more messages are lost.
Figure 3: Delay (RTT) in a day.
1
HS
Next, we measure how the delay time and message loss ratio are changed from hour to hour in a day. Figures 3 and 4 show the delay time and message loss ratio for each hour. For example, less than 5% of the packets are lost from 7 o'clock to 17 o'clock while more packets are lost before 7 o'clock and after 17 o'clock between H and K . In addition, Figures 3 and 4 show that the longer the distance is, i.e. more routers are hopped, the bigger the variances of the delay time and message loss ratio are. Hence, each process is required to change a transmission and retransmission way to support the shorter delay time and the higher reliability according to the changes of delay time and message loss ratio. For example, the sender retransmission is adopted if the message loss ratio is smaller. If the
H-K H-U H-S S-K S-U U-K
40
Loss ratio [%]
Table 1: Delay[msec] & lost[%]. host MIN AVRG MAX LOST S 30.4 60.4 756.3 1.0 U 119.5 157.2 532.4 8.3 K 164.5 241.4 2733.6 11.7
Loss ratio a day 50
30
20
10
0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Time [hour(JST)]
Figure 4: Loss-ratio in a day. Finally, we measure how reliably the destination can receive a message m by sending multiple replicas of m. H at Hatoyama sends n replicas of m to S , U , and K . Table 2 shows ratios at which each destination receives at least one replica of m for n = 1; 2; 3. For example, K at Keele almost surely receives m if H sends three replicas of m to K . K loses more than 10% of the messages if H sends only one replica of m
m
m. U at UCLA can surely receive m if H sends two replicas of m to U . That is, H does not need to send more than two replicas.
T P . On receipt of all the con rmations, T P informs all the destinations of the reliable receipt of m. Totally 3s messages are transmitted and it takes three rounds. In the distributed way[21, 23], every destination T P sends the con rmation of m to all the destinations and T P on receipt of m. If each T P receives all the con rmations, T P reliably receives m. Here, O(s2 ) messages are transmitted and it takes two rounds. In [23], only O(s) messages are transmitted by adopting the piggy back and the deferred con rmation. i
i
j
Table 2: Loss ratio [%] by replication. m 1 2 3 S 99.08 99.96 99.96 U 89.74 99.16 99.84 K 88.36 98.20 99.84
i
n
j
j
TP2
TP2 TP1
4 Reliable Delivery
In the group communication, each process T P sends a message m to multiple destination processes in a group G = fT P1 ; : : : ; T P g. Here, let s be the number of the destinations of m, i.e. s = jdst(m)j. There are two points to be discussed to realize the reliable delivery of m in G: 1) how to deliver m to the destinations, and 2) how to deliver the receipt con rmation of m to T P and the destinations. There are direct and hierarchical delivery ways [Figure 5]. In the direct delivery, T P sends m directly to all the destinations. In the hierarchical one, T P sends m to a subset of the destinations. On receipt of m from T P , T P forwards m to another subset of destinations. For the latter, tree-based routing algorithms[5, 7, 10] are discussed so far. Another example is to decompose G into disjoint smaller subgroups G1 ; : : : ; G (sg 2)[24]. Each G has one coordinator process C . T P sends m to every coordinator C and then C forwards m to all the destinations in G . i
i
i
j
sg
j
j
j
i
j
j
TP4 TP2 TP1
TP3
TP1
TP2 TP3
TP4 (1) direct
(2) hierarchical
TP5
A distributed protocol[16] supports the direct delivery and the distributed con rmation. ISIS[4] adopts the direct delivery and the decentralized con rmation. These protocols are referred to as decentralized. 4.2 Retransmission
In the underlying network, messages are lost due to bu er overruns, unexpected delay, and congestion. Hence, the processes have to recover from the message loss. Let us consider a group G = fH; U; O; K g. Suppose that H sends a message m to U , O, and K , and O loses m. In the traditional protocols, H retransmits m to O and it takes 2 . In another way, U forwards m to O on behalf of H . Here, it takes only 2 (< 2 ) so that the delay time is reduced. That is, retransmission from a process T P is preferable for a destination process T P if = 1 (1+ " )=(1 0 " ) is the minimum when k = j . 1) Sender retransmission: The sender T P retransmits m to T P if < for every destination T P of m [Figure 7(1)]. 2) Destination retransmission: Some destination T P forwards m to T P if < [Figure 7(2)]. HO
UO
HO
j
TP7
Figure 5: Delivery ways. There are two ways to deliver the con rmation [Figure 6]. In the decentralized way[4], the destinations send back the receipt con rmations of m to the sender
R ki
i
ki
ki
ki
i
R ij
j
R kj
k
k
TP6
(2) distributed
Figure 6: Con rmation ways.
i
n
TP3 TP4
TP4 (1) decentralized
4.1 Transmission and con rmation
i
TP1
TP3
j
R kj
R ij
4.3 Replication
If a sender T P sends multiple replicas of m to a destination T P , T P can more surely receive one replica. Another way is that another destination T P forwards m to T P while T P directly sends m to T P . T P receives two replicas of m from T P and T P . For example, U sends m to O on receipt of m from H while i
j
j
k
j
i
j
i
k
j
TP1
TP2
TP1
TP3
TP2
x
x
ij
time (1) sender
to receive at least one replica of m forwarded by is 1(1+" )=(10" )+ 1(1+" )=(10 ). The cost C is n 1 + n 1 . The probability 5 that T P receives at least one replica of m is (1 0 " )+ " (1 0 " )(1 0" ). The destination replication is more ecient than the sender replication if , 5 5 , and C C . For the time critical applications, the destination replication can be adopted if even if C C . T P is selected to forward m to T P if is the minimum among the destinations of m. Here, no sender replication is adopted if n = 1 and n = 1. Let " be a maximum allowable loss ratio in the application. If " " , T P does not need to send multiple replicas of m to T P . If " > " , T P has to send n (> 1) replicas of m whether or not " . T P retransmits m. It is required that " Hence, n log" =log" if " " ; otherwise, n = 1. For example, n = 3 if " = 0:01 and " = 0:2. We can consider a way that T P sends m to T P , T P sends m to T P , and nally T P sends m to T P . However we assume that there is only one process T P forwarding a replica of m to T P . T Pj T Pk ik "kj nkj
TP3
ik
Figure 7: Retransmission.
ik
ikj
ikj nij
nik
ik
kj
ik
ij
nij
ik
RR ikj
nik
kj
kj
ij
i
j
k
j
k
j
k
l
j
TPk
TPl
ikj
RR ikj
ij
ikj
RR ikj
j
ij
ik
max
ij
max
i
j
ij
max
i
ij
ij
i
ij
max
ij
ij
ij
nij
max
max
ij
max
ij
i
l
l
k
k
j
k
j
j
i
TPi
nkj
ij
k
directly sends m to O. O can receive m from U even if m sent by H is lost. 1) Sender replication: T P sends multiple replicas of m to T P . 2) Destination replication: A destination T P receiving m forwards m to T P . The sender replication is similar to the saturation protocol[11]. There are two kinds of the destination replication. First, one destination T P sends one, possibly multiple replicas of m to T P on receipt of m [Figure 8(1)]. Secondly, multiple destination processes, say T P and T P , send the replicas to T P [Figure 8(2)]. T P receives multiple replicas from T P and T P .
nkj
kj
ikj
RR ij
H
kj
j
RR ij
(2) destination
nik
TPj TPi
k
TPk
TPl
TPj
5
Protocols
We present a protocol for transmitting messages in the group G = fT P1 ; : : : ; T P g. n
time (2) multiple destinations
(1) one destination
5.1 Data transmission
Figure 8: Destination replication. In the continuous replication, T P sends continuously the replicas of m to T P . In the discrete replication, T P sends the succeeding replicas of m some time units after T P sends each replica of m. If the message loss occurs in a burst manner, the discrete replication has to be adopted. Suppose that T P sends n replicas of m to T P according to the sender replication. If T P receives no replica from T P , T P sends n replicas of m to T P again. The expected time for T P to receive at least one replica of m from T P is 1 (1+ " )=(1 0 ). The probability 5 that T P receives at least " one replica of m is 1 0 " . The cost C for T P to send m to T P is de ned to be n 1 . On the other hand, on receipt of at least one replica of m, a destination T P sends n replicas of m to T P according to the destination replication. The expected time for i
j
i
i
i
ij
j
j
i
i
ij
j
RR ij
j
ij
i
ij
nij
ij
j
k
ij nij
nij
j
ij
ij
kj
ij
i
ij
j
RR ikj
In the protocol, the messages are required to be causally ordered and the message losses have to be detected. The vector clock[14] is used in order to causally order the messages. Each message m carries a vector clock V C = hV C1 ; : : : ; V C i where V C is for each T P . Since the gap between messages, i.e. the message losses cannot be detected by the vector clock, the protocol detects the message losses by the sequence numbers of the messages. Each message m is sent to the destination processes, not necessarily all the processes in G. m is given a vector SEQ = hSEQ1; : : : ; SEQ i of sequence numbers. Each variable SEQ denotes a sequence number of a message for T P (j = 1; : : : ; n). Initially, SEQ = 0. Each time m is sent to T P , SEQ is incremented by one. Otherwise, SEQ is not changed. m:SEQ = SEQ for j = 1; : : : ; n. Here, suppose that T P sends a message m2 to T P after m1. If m2 :SEQ 0m1 :SEQ = 1, T P nds that there is no gap between m1 and m2 . If n
i
i
n
j
j
j
j
j
j
j
j
i
j
j
j
j
m2 :SEQ 0 m1 :SEQ = g (> 1), T P nds that T P loses g 0 1 messages between m1 and m2 . Each T P manipulates the variables V C1; : : : ; V C showing the vector clock to causally order the messages received. Initially, V C = 0 for every j . Each time T P sends a message m, V C is incremented by one and m:V C = V C for j = 1; : : : ; n. On receipt of m from T P , T P manipulates V C as V C = max(V C ; m:V C ) for j = 1; : : : ; n. On receipt of a message m from T P , T P stores m in the bu er RBUF. The messages in RBUF are causally ordered by using the vector clocks. For every pair of messages m1 and m2 , m1 causally precedes m2 i m1 :V C < m2 :V C , i.e. m1 :V C m2 :V C for every j and m1 :V C < m2:V C for at least one j [14]. T P manipulates variables REQ1 ; : : : ; REQ to receive messages. REQ shows that T P receives every message m from T P such that m:SEQ < REQ . On receipt of m from T P , T P knows that T P has received every message sent by T P where SEQ < REQ if m:SEQ = REQ . Then, REQ is incremented by one. If m:SEQ 6= REQ , T P detects that some messages from T P are lost. The messages sent are also stored for retransmission in the bu er SBUF in the sending order. In order to support the timeout mechanism, it is important to specify when the messages are sent. A variable T denotes a current local time in T P . T P stores the value of T in the eld T of a message m when T P sends m. Each message m is composed of the following elds: 1 m:SRC = process T P sending m. 1 m:DST = set of destination processes of m. 1 m:SEQ = hm:SEQ1 ; : : : ; m:SEQ i = vector of sequence numbers of m. 1 m:ACK = hm:ACK1; : : : ; m:ACK i = vector of sequence numbers of messages received. 1 m:V C = hm:V C1; : : : ; m:V C i = vector clock. 1 m:T = time when m is sent. 1 m:DT = data. Here, suppose that T P sends a data DATA to a subset of processes fT P 1; : : : ; T P g in G (1 l n). T P makes a following message m. m:SRC := T P ; m:DST := fT P 1 ; : : : ; T P g; SEQ := SEQ + 1 for every T P 2 m:DST ; m:SEQ := SEQ; m:REQ := REQ; V C := V C + 1; m:V C := V C ; m:T := T ; m:D := DATA; For each destination T P of m, T P sends m (> 1) replicas of m if " > " . Our protocol adopts the distributed con rmation. That is, on receipt of a message m from T P , T P sends the rej
j
j
j
i
n
j
i
i
j
j
j
j
i
j
j
j
i
j
j
j
j
i
n
j
i
j
i
j
j
i
i
j
j
i
i
j
j
i
j
i
j
i
i
i
i
i
i
n
n
n
i
i
i
ili
i
i
j
i
j
i
ili
j
i
i
j
i
ij
max
i
j
j
ceipt con rmation of m to not only T P but also every other destination. The con rmation information ACK = hACK1 ; : : : ; ACK i is carried by messages which T P sends after receiving m. Here, ACK denotes a sequence number of a message which T P expects to receive next from T P . Each time T P sends a message m, m:ACK = REQ for j = 1; : : : ; n. If T P sends con rmation messages each time T P receives a message, O(n2 ) messages are transmitted. T P adopts the deferred con rmation[23] to reduce the number of messages. Even if there is no data to send to T P , T P does not send the con rmation on receipt of m. T P sends a con rmation message of m to T P some predetermined time units after receiving m. That is, each message can carry the con rmation information of more messages received. i
n
i
j
i
j
j
i
j
i
i
i
j
i
i
j
5.2 Retransmission
If T P receives the con rmation of m from all the destinations, m has to be delivered in the causal order after all the messages causally preceding m are delivered. On receipt of m for T P , if m:SEQ REQ , m is accepted and stored in the bu er RBUF . Otherwise, T P knows that T P loses a message m from T P such that REQ m :SEQ m:SEQ . The top m of RBUF is delivered if some message received from every process causally following m exists in RBUF. Here, T P manipulates variables DCV = hDV C1; : : : ; DV C i denoting the vector clocks of the message which are most recently delivered. In the basic way to detect message loss, on receipt of a message m from T P , T P checks the sequence numbers of m. If a message m1 is detected to be lost, T P sends a NACK message to the destination process T P of m1 which is the nearest to T P . When T P receives the NACK of m from T P , T P retransmits or forwards m which is stored in SBUF or RBUF. The procedure is shown as follows. Here, receive(m) means that T P receives m. i
j
i
j
i
0
i
j
0
j
i
i
i
n
j
i
i
k
i
j
k
k
i
if (receive(m) == TRUE) if (m.SEQi < REQj ) send(NACK); else f REQj = m.SEQi ; deliver(m); send(ACK);
g
f
g
In our protocol, each destination T P receives the con rmation from the other destinations. Hence, T P i
i
considers that T P does not receive a message m unless T P receives the con rmation of m from T P . If T P is the nearest to T P , T P forwards m to T P . If T P receives the con rmation of m from T P that is the nearest to T P , T P does not forward m to T P because T P must forward m to T P .
= (
j
i
j
i
= (
j
i
k
j
k
i
)0 (
delivery(DL) delay T deliver T send reliable receipt(RR) delay T reliable receive T send detect(DT) delay T detect T send
j
i
)0 ( = (
). )0 (
).
).
j
i
Table 3: Delay [msec]. Protocols M receipt(R) 376 (1) delivery(DL) 383 reliable receipt (RR) 724 detect (DT) 386 (2) receipt (R) 393 delivery (DL) 394 reliable receipt (RR) 735
if (receive(m) == TRUE) f for each message m' in SBUF if ((m.ACKm0 :SRC < m'.SEQj ) && TPj is the nearest to TPi ) forward(m',TPj );
g
In a decentralized protocol, the sender T P resends a message m. T P detects a message loss by receiving a NACK or by the timeout scheme. i
i
if (receive(NACK) == TRUE || m.T+2*ij < T) resend(m,TPj);
6 Evaluation
The prototype of the protocol has been implemented to be a group G of ve UNIX processes in SPARC workstations, i.e. two (ktus0, ktus1 ) at Hatoyama, one (swan ) at Sendai, Japan, one (it sunshine) at UCLA, U.S., and one (nina ) at Keele, UK. Here, suppose that nina sends a message m to all the other processes in G. We consider two cases: there is no message loss and ktus0 loses m sent by it nina. We measure the delay time where the processes send messages of 1024[bytes] to all the processes in G. In the decentralized protocol (D) like ISIS, the sender nina retransmits m to ktus0. In our modi ed protocol (M), ktus1 nearest to ktus0 forwards m to ktus0 and nina does not retransmit. The following events occur in the process: send: m is sent by the original sender process. receive: m is received by the destination process. deliver: m is delivered to an application process. reliable receive: The sender process knows that m is received by all the destinations. detect: A destination process detects a loss of m by receiving a con rmation of m from another process. For each event e, let T (e) be time when e occurs. The following kinds of delays are obtained from the times measured: receipt(R) delay= T (receive ) 0 T (send ).
D 376 383 1128 762 1135 1139 1891
(1) of Table 3 indicates R, DL, and RR delays for D and M protocols in the rst non-loss case. The difference between R and DL shows time for the protocol processing. The di erence between R and RR shows time for exchanging the con rmation messages of m. (2) of Table 3 shows R, DT, DL, and RR delays in the presence of message losses. The di erence between DL and DT shows time for recovering from message loss by retransmission. For example, ktus1 forwards m to ktus0 in the protocol. The di erence between DT and DL shows how long it takes to retransmit m. As in Table 3, the processes can recover from message loss with the shorter delay in our protocol than the decentralized one. In addition, the delay time is almost the same as the non-loss case. In the wide-area group, each channel is di erent in the delay time and the loss ratio. Hence, messages can be delivered with shorter delay if they are transmitted through channels with the shorter delay and less loss ratio. 7
Concluding Remarks
It is critical to discuss how to realize the high-speed group communication of multimedia data in the worldwide area by using the Internet. Here, each logical channel between the processes has a di erent delay time and message loss ratio. In this paper, we have discussed wide-area group communication protocols to reduce the delay time of messages and improve the reliability in the wide-area group, i.e. destination retransmission and replication. We have shown that our protocol implies shorter delay time through the worldwide experiment than the decentralized protocols used in the conventional group communication protocols.
References
[1] Aiello, R., Pagani, E., and Rossi, G. P, \Causal Ordering in Reliable Group Communications," Proc of ACM SIGCOMM'93, 1993, pp. 106{115. [2] Amir, Y., Dolev, D., Kramer, S., and Malki, D., \Transis: A Communication Sub-System for High Availability," Proc. of the 22nd FTCS, 1993, pp. 76{84. [3] Amir, Y., Moser, L.E., Melliar-Smith, P.M. Agarwal, D.A., and Ciarfella, P., \The Totem SingleRing Ordering and Membership Protocol," ACM Trans. on Computer Systems, Vol. 13, No. 4, 1995, pp. 311{342. [4] Birman, K., Schiper, A., and Stephenson, P., \Lightweight Causal and Atomic Group Multicast," ACM Trans. on Computer Systems, Vol. 9, No. 3, 1991, pp. 272{314. [5] Chang, J.M. and Maxemchuk, N.F., \Reliable Broadcast Protocols," ACM Trans. on Computer Systems, Vol. 2, No. 3, 1984, pp. 251{273. [6] Florin, G. and Toinard, C., \A New Way to Design Causally and Totally Ordered Multicast Protocols," ACM Operating Systems Review, Vol. 26, No. 4, 1992, pp. 77{83. [7] Garcia-Molina, H. and Spauster, A., \Ordered and Reliable Multicast Communication," ACM Trans. on Computer Systems, Vol. 9, No. 3, 1991, pp. 242{271. [8] Hofmann, M., Braun, T., and Carle, G., \Multicast Communication in Large Scale Netwoks," Proc. of the 3rd HPCS, 1995. [9] Holbrook, H.W., Singhal, S.K., and Cheriton, D.R., \Log-Based Receiver-Reliable Multicast for Distributed Interactive Simulation," Proc. of ACM SIGCOMM'95, 1995, pp. 328{341. [10] Jia, X., \A Total Ordering Multicast Protocol Using Propagation Trees," IEEE Trans. on Parallel and Distributed Systems, Vol. 6, No. 6, 1995, pp. 617{627. [11] Jones, M., Sorensen, S., and Wilbur, S., \Protocol Design for Large Group Multicasting: The Message Distribution Protocol," Computer Communications, Vol. 14 No. 5, 1991, pp. 287{297.
[12] Kaashoek, M.F., Tanenbaum, A.S., Hummel, S.F., and Bal, H.E., \An Ecient Reliable Broadcast Protocol," ACM Operating Systems Review, Vol. 23, No. 4, 1989, pp. 5{19. [13] Liang, L., Chan, S.T., and Neufeld, G.W., \Process Groups and Group Communications: Classi cations and Requirements," IEEE Computer, Vol. 23, No. 2, 1990, pp. 56{66. [14] Mattern, F., \Virtual Time and Global States of Distributed Systems," Parallel and Distributed Algorithms (Cosnard, M. and Quinton, P. eds.), North{Holland, 1989, pp. 215{226. [15] Melliar-Smith, P.M., Moser, L.E., and Agrawala, V., \Broadcast Protocols for Distributed Systems," IEEE Trans. on Parallel and Distributed Systems, Vol. 1, No. 1, 1990, pp. 17{25. [16] Nakamura, A. and Takizawa, M., \Causally Ordering Broadcast Protocol," Proc of the 14th ICDCS, 1994, pp. 48{55. [17] Reiter, M.K., \The Rampart Toolkit for Building High-Integrity Services," Theory and Practice in Distributed Systems, LNCS 938, Springer-Verlag, 1995, pp. 99{110. [18] Peterson, L.L., Buchholz, N.C., and Ghemawat, S., \Preserving and Using Context Information in Interprocess Communication," ACM Trans. on Computer Systems, Vol. 7, No. 3, 1989, pp. 217{ 246. [19] Postel, J., \Internet Protocol," RFC-791, 1981. [20] Postel, J., \Internet Control Message Protocol," RFC-792, 1981. [21] Tachikawa, T. and Takizawa, M., \Selective Total Ordering Broadcast Protocol," Proc. of ICNP'94, 1994, pp. 212{219. [22] Tachikawa, T. and Takizawa, M., \Multimedia Intra-Group Communication Protocol," Proc of the 4th HPDC, 1995, pp. 180{187. [23] Tachikawa, T. and Takizawa, M., \Distributed Protocol for Selective Intra-group Communication," Proc. of ICNP'95, 1995, pp. 234{241. [24] Takamura, A., Takizawa, M., and Nakamura, A., \Group Communication Protocol for Large Group," Proc. of the 18th LCN, 1993, pp. 310{ 319.
