Semi-active replication of SNMP objects in agent

Semi-Active Replication of SNMP Objects in Agent Groups Applied for Fault Management

Elias Procópio Duarte Jr., Aldri L. dos Santos Federal University of Paraná, Dept. Informatics Caixa Postal 19081 Curitiba PR 81531-990 Brazil elias,aldri @inf.ufpr.br

Abstract It is often useful to examine MIB objects of a faulty agent in order to determine why it is faulty. This paper presents a new framework for semi-active replication of SNMP management objects in local area networks. The framework is based on groups of agents that communicate with each other using reliable multicast. A group of agents provides fault-tolerant object functionality. A SNMP service is proposed that allows replicated MIB objects of a faulty agent of a given group to be accessed through fault-free agents of that group. The presented framework allows the dynamic definition of agent groups, and management objects to be replicated in each group. A practical fault-tolerant tool for local area network fault management was implemented and is presented. The system employs SNMP agents that interact with a group communication tool. As an example, we show how the examination of TCPrelated objects of faulty agents have been used in the fault diagnosis process. The impact of replication on network performance is evaluated. Keywords Fault Management, Fault Tolerance, SNMP, Group Membership, Replication

1. Introduction Fault management is one of the key functions of integrated network management systems. The purpose of fault management is to locate, determine the causes and, if possible, correct network faults [1]. Different approaches have been proposed to accomplish this task, including the application of system-level diagnosis for faulttolerant network monitoring [2, 3], alarm correlation [4], and proactive fault detection [5], among others. In this work we present a new approach for fault management based on replicated SNMP (Simple Network Management Protocol) MIB (Management Information Base) objects. As MIB objects reflect the state of managed entities, in order to determine why a given network component has failed it is often useful to examine

0-7803-6719-7/01/$10.00 (c) 2001 IEEE

566

Session Sixteen

Fault Management II

566

SNMP GROUP TOOL

UDP

UDP

UDP

UDP

GROUP TOOL

GROUP TOOL

GROUP TOOL

SNMP

SNMP

SNMP

Figure 1: SNMP Agents interact with a group communication tool. the MIB of its crashed agent; i.e. the value of the MIB’s objects may help the diagnosis process. However, there is currently no standard way to examine MIB objects of crashed agents using SNMP. In this work we introduce a new SNMP service that allows MIB objects of crashed agents to be accessed through working agents. The new service is based on cooperating groups of SNMP agents defined with a group communication tool [6]. The group abstraction has been widely used to develop reliable distributed applications [7]. Each group is a collection of processes that usually communicate using reliable multicast. As shown in figure 1, in the new approach the group communication tool is located between SNMP and UDP, i.e. SNMP messages are sent via the group communication tool to a collection of agents; messages are encapsulated in UDP segments. Each group implements semiactive replication of selected objects. Using this approach it is possible to query and update replicated objects from any member of the group, however a leader agent is employed to determine the order in which objects are updated. A framework for defining SNMP agent groups and for keeping and accessing replicated objects is presented. A practical tool was built with NET-SNMP agents [8] that communicate using the Ensemble group communication toolkit [9]. As an example of the usage of the system we show how the examination of TCP-related objects have helped in the determination of causes of host failures. The system allowed the examination of replicas of TCP-related objects of a crashed agent which had suffered a TCP Syn flooding attack, leading to the diagnosis of the failure causes. An evaluation of the cost of object replication is also given. Previously published related work include [10], in which J. Schönwälder introduces an approach for defining SNMP agent groups that implement membership on top of SNMP; this is different from our approach in which the membership proto-

567

Semi-active Replication of SNMP Objects

567

col is a service used by SNMP. In that work a leader election algorithm is presented which employs multicast SNMP messages. Although object replication is not directly considered, the election algorithm could be used to implement passive replication, in which after the primary fails an election is run to determine the replica that will take the primary’s role. This is another difference from our approach, for semi-active replication does not require elections. Other related efforts include [11], in which K.H. Lee introduces a group communication protocol that provides ordered delivery of messages in a distributed network management system. In [12] a network administration tool based on a group communication toolkit is described, which provides solutions for maintaining consistent distributed name service configurations, and allows software distribution to a number of machines. The rest of this paper is organized as follows. Section 2 presents an overview of group membership and replication techniques. In section 3 we describe the new SNMP service, which allows the creation of groups of SNMP agents providing semiactive replication of selected objects. In section 4 a practical fault management system based on groups of SNMP agents that communicate with a group communication tool is presented. Section 5 considers the impact of replication on network performance. Conclusions and future work are in section 6.

2. Replication and Group Membership In this section we present an overview of two techniques that allow the development of reliable distributed systems: replication and group membership.

2.1 Replication Replication [13] is a useful tool which has been applied to introduce fault tolerance in both distributed systems and databases, where it also aims at improving performance [14, 15]. Replication is the process of making a replica (a copy) of resources, processes or data. Although it is an intuitive concept, replication requires sophisticated techniques for successful implementation in a distributed system [7]. It is necessary to guarantee the consistency of replicated data. Replication is usually classified in two different types: passive replication and active replication [16]. Passive replication is also known as the primary backup approach. Primary backup uses one replica, the primary, that plays a special role: it receives requests from client processes and returns responses. The other replicas, called backups, interact directly only with the primary replica, not the client processes. Whenever necessary, the primary sends update messages to the backups. When the primary fails, the backups must detect the failure and run a leader election algorithm in order to determine which one will become the next primary. This

568

Session Sixteen

Fault Management II

568

reconfiguration procedure is costly. The second type is called active replication, which is also called the state-machine approach. This replication technique is non-centralized, i.e. any replica can receive any update command from client processes. A client does not contact one particular replica, but addresses them as a group. The main advantage of active replication is its failure transparency, no matter how many replicas are faulty or fault-free, the data is accessed in the same way. However, a drawback of active replication is that it requires working replicas to receive the invocations of client processes in the same order, to guarantee the consistency of replicated data. Semi-active replication is a hybrid approach involving characteristics of both active and passive replication. In this approach, each replica processes every update command received from clients. Figure 2 shows that replicated data can be accessed through any member of the group. However, the order that update messages are received is established through a leader. Both clients and replicas send update requests to the leader, who processes the requests, reaching a new state and then forwards the same requests to all replicas, as shown in figure 3. query query

Group client query

replic query

leader

query

client

replic

replic

query

Figure 2: Semi-active replication: querying an object. Semi-active replication can be naturally applied to SNMP. Each agent serves as leader for its own objects. In other words, a leader is defined as the agent that would keep the object if it were not replicated. Consider a group of agents, which have part of their individual MIB’s replicated in the group. Although any group member can be queried to access any replicated object, if a replica receives an update command, it must forward this command to the proper leader. After validating the update command, the leader propagates the updated object value to all group members, which then update their local copies. Other hybrid approaches for replication exist, such as semi-passive replication [17].

569


569

replica update validate & run locally

client

update leader

group communication tool

update

replica

update

replica

update

replica

update

replica

Figure 3: Semi-active replication: updating the value of an object. The group abstraction, presented below, is an adequate framework for providing reliable multicast primitives required to implement replication in distributed systems.

2.2 Group Membership Reliable distributed applications must guarantee that its services are provided even when failures occur. One possible approach for constructing reliable applications consists in structuring the application into cooperating groups of processes which communicate using reliable multicast. All messages sent to the group are delivered to each group member that is not faulty. Group membership is a service that maintains consistent information about which members are functioning and which have failed at any moment. In order to allow this, membership service reports failures, recoveries or joining members, indicating changes into the membership [18, 19]. A group communication system usually provides reliable group multicast and membership service. Several group communication systems have been implemented and are available, such as Horus [20] and Ensemble [9]. Ensemble is a group communication toolkit which allows the development of fault-tolerant complex distributed applications. Ensemble provides a library of protocols which handles issues such as reliably sending and receiving messages, transferring process states, implementing security, detecting failures and managing reconfigurations in the system. We have used Ensemble as a group communication tool for implementing a reliable distributed SNMP-based network fault management tool which is described in section 4.

3. A Framework for Replication of Management Objects In this section we present a framework for a group of SNMP agents to support

570

Session Sixteen

Fault Management II

570

semi-active replication of management objects. A MIB portion is presented which consists of three dynamic tables. The core table provides means for group members to keep replicated objects, and allows a manager to access replicated objects through any agent of a given group. This table is automatically built from two other tables: the first contains the definition of multiple agent groups; the other contains the specification of objects that are replicated in each group. Specifying Group Members An agent group is a collection of agents that replicates a given set of objects. Table repMemberTable is a table used to define and keep group member information. Each entry contains the following core objects, with an index {groupId, memberAddress}: RepMemberEntry ::= SEQUENCE { groupId Unsigned32, memberAddress IpAddress }

agents. A group of agents consists of Consider a system consisting of agents, where . In order to allow the dynamic definition of multiple groups of agents in a given system, each group is assigned an identifier, called groupId. This identifier is an integer that is unique for a specific group. An entry keeps the IP address of each member in object memberAddress. Optionally, a non-standard port number might be specified for a member agent to listen to; such an agent could be used for replication purposes only. For example, consider a group of three agents: , , and , which have IP addresses 10.0.0.1, 10.0.0.2, and 10.0.0.3, respectively. The manager must obtain a unique groupId, in this example the group identifier is 100. The manager then selects one agent of the system in which to set three entries of table repMemberTable with these members IP addresses, and the group identifier. Table 1 illustrates repMemberTable for this example.

Group ID 100 100 100

Member Name

Member Address 10.0.0.1 10.0.0.2 10.0.0.3

Table 1: An example table of group members. Specifying Replicated Objects Besides specifying the members of a group, it is also necessary to define which objects are replicated in the group. Table repObjectTable is used to determine

571


571

which objects are replicated in a given group. Each entry contains the following objects, with an index {groupId,objectId}: RepObjectEntry ::= SEQUENCE { groupId Unsigned32, objectId OBJECT IDENTIFIER }

Each group, identified by groupId, keeps a set of replicated objects. Each entry of table repObjectTable contains the group identifier and the ASN.1 object identifier objectId. Consider the group defined above. Now the objects to be replicated in this group must be set by the manager in one agent of the system, as shown in table 2. Group ID 100 100 100 100

Object ID system.sysDescr.0 system.sysUpTime.0 tcpPassiveOpens.0 tcpCurrEstab.0

Table 2: An example table indicating replicated objects of a group. The manager should select objects that may help the diagnosis of failure causes. In this case, the objects selected are the description of the agent’s component, the amount of time elapsed since it was powered on, the number of TCP connection requests received by the agent’s host, and the number of TCP connections successfully established. Keeping and Accessing Replicated Objects In order to allow SNMP agents to keep replicated objects that SNMP managers can access through any member of a group, the following table, repGroupTable is defined. This table is automatically built and replicated throughout the group when the system is initialized. The index is {groupId,membAD,objectId}. RepGroupEntry ::= SEQUENCE { groupId Unsigned32, membAD IpAddress, objectId OBJECT IDENTIFIER, repObjectValue type varies according to the replicated object }

This table keeps the replicated objects in all members of all groups. A given agent only keeps replicated objects of groups in which it takes part. Each entry of this table keeps the value of a replicated object of a given agent. An entry consists of the group

572

Session Sixteen

Fault Management II

572

identifier, groupId, the address of the agent, membAD, the oid of the replicated object, objectId, and the value of object in that agent, repObjectValue. After the manager initializes at any agent the table of group members, RepMemberTable, and the table of replicated objects, repObjectTable, table repGroupTable is automatically created and initialized in all members of each group. This is the table that actually keeps the replicated objects. When the object named CreateNewGroup is set with the GroupId, both tables repMemberTable and repObjectTable are replicated in all group members. Table repGroupTable is generated and replicated in all members of the group. At pre-defined time intervals, each member of each group checks whether its own local objects that are replicated have been updated since the previous interval. In this case a message is reliably multicast to all group members so that they update their local replicas. The testing interval must be carefully chosen to allow objects in the replicas to be consistent, and avoid a high impact on network performance. An object repGroupInterval is used to set the checking time interval for each group. Alternatively, in the case of READ-WRITE replicated objects, whenever such an object is updated, the leader may run each set command locally and immediately send a multicast message to all replicas. In these cases, as there is no pre-defined checking interval, the cost of replicating objects may soar, because each time the object is updated a message is generated and sent to all members. A replica may also receive an update command directly from a client. In this case, the set command must be forwarded to the leader, which in turn will run the command locally and send the updated object value to the other group members. Continuing the example above, table repGroupTable is shown in table 3. This table is kept by all members of the group. 100 100 100 100

10.0.0.1 10.0.0.1 10.0.0.1 10.0.0.1

system.sysDescr.0 system.sysUpTime.0 tcp.tcpPassiveOpens.0 tcp.tcpCurrEstab.0

15346 50 10

100 100 100 100

10.0.0.2 10.0.0.2 10.0.0.2 10.0.0.2


12435 10465 25

100 100 100 100

10.0.0.3 10.0.0.3 10.0.0.3 10.0.0.3


65487 12 6

Table 3: An example table of replicated objects.

573


573

Now consider that agent has failed. The manager may then query either agent or agent in order to obtain the failed agent’s objects. For example, if the replica at agent is used, the following snmpget commands may be sent:

snmpget snmpget snmpget snmpget

agent= agent= agent= agent=

object=.100...0 object=.100...0 object=.100...0 object=.100...0

In this case, the commands are received by group member . Table repGroupTable is indexed by the tuple (groupId,membAD,objectId), in this case (100,B,tcp.tcpPassiveOpens.0) and is under subtree replic. Now consider that the response received is as shown below.

response: response: response: response:

object=replic.100.B.system.sysDescr.0 = B object=replic.100.B.system.sysUpTime.0 = 51234 object=replic.100.B.tcp.tcpPassiveOpens.0 = 1684 object=replic.100.B.tcp.tcpCurrEstab.0 = 25

This response may indicate the reasons agent has failed. Security and authentication mechanisms must be configured so that group members have permission to read and write replicated objects in the tables defined above.

4. A Practical Tool This section describes a practical tool based on the framework presented on the previous section that allows the examination of replicated objects of faulty agents. The tool is based on the public-domain NET-SNMP package [8] and Ensemble, a group communication toolkit [9]. The tool allows the creation of SNMP agent groups that support semi-active replication of selected objects. Agent groups are dynamic, in the sense that they can be created and destroyed; groups can also be updated, i.e. the selection of replicated objects in each group is not fixed. The tool is useful to examine MIB objects of faulty agents, and to query objects from the whole group using one command issued to one agent that is a group member. We describe how the system has been used to determine that a given faulty agent had suffered TCP-based attacks. A group of SNMP agents is a collection of agents that share replicated objects. Each agent must include the Replication-MIB introduced in section 3, including tables repMemberTable, repObjectTable, and repGroupTable. From the moment a group is initialized, objects are automatically replicated throughout group members. Semi-active replication is employed so that any group member may be queried at any time for any replicated object. We have used the Ensemble group communication toolkit to add reliable multicast capability to SNMP agents. Ensemble provides reliable communication to

574

Session Sixteen

Fault Management II

574

groups of processes, providing ordered channels between all members in a group. A service for managing multiple process groups is provided, called groupd which implements a group membership protocol [9]. This protocol allows all members of the group to detect failures and guarantees that they all see the same set of failures. Ensemble supports IP multicast, but also provides the gossip application which simulates low-bandwidth gossip broadcast for systems that do not have IP multicast. Each group member is an SNMP agent that is capable of receiving and handling messages that arrive from the group. A thread was added which parses each message, verifying whether the action can be committed. A query is processed in the standard way. A pre-defined checking interval, which is variable, was employed to check which objects were updated. The tool also allows a leader agent to send updating command to other group members after receiving a valid set command on a replicated object.

4.1 Detection of TCP SYN Flooding Attacks An experiment was executed in which a host was submit to a TCP Syn Flooding attack [22]. After a few seconds the machine was down. In order to examine the TCP MIB objects of the faulty agents, a replica was invoked. Replicated objects included tcpPassiveOpens, which gives the number of times TCP connections have made a direct transition to the SYN-RCVD state from the LISTEN state [21]. Another replicated object was tcpCurrEstab, the number of TCP connections for which the current state is either ESTABLISHED or CLOSE-WAIT. The value of the MIB objects left no doubt as to the cause of the attack, previous logs showed that usually tcpPassiveOpens was equal to zero, while after the failure it was equal to 85; object tcpPassiveOpens remained at the usual value, equal to 12. If the tool based on replicated objects were not available, it would have been impossible to examine the MIB of this machine when it was already faulty.

5. Replication Cost In this section we evaluate the cost of replication. It is not easy to analyze the overhead and tradeoffs represented in the semi-active replication approach because parameters like object replication frequency, group size, failure rates and failure correlations, among others, may vary widely from system to system. The overhead and tradeoffs depend also on network resources and load and especially on which objects are replicated. When replicated objects are updated, messages are sent to group members so that they can update their replicas. In order to present results that are as general as possible, i.e. results that are useful for widely different replicated objects and systems, we consider a large range of the frequency in which replicated objects are updated.

575


575

0.10 0.09

1 msg 10 msgs 50 msgs

Required Bandwidth (Mbps)

0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 0.5

0.6

0.7

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1 0.8 0.9 Frequency in which Replicated Objects are Updated (sec)

1.9

2.0

Figure 4: The impact of varying the updating frequency. The parameter we seek is the bandwidth required to send replication messages, in terms of bits transferred per second. In [23] a probabilistic evaluation of the impact of replication on network performance and replica consistency is presented. The impact of replication must be studied in order to avoid excessive consumption of resources [24]. The graph in figure 4 shows the required bandwidth considering that replication messages are sent in different frequencies, varying from one each 500 milliseconds to one each 2 seconds. The number of messages sent varies from 1 to 50, and the size of all messages is 100 bytes, i.e. each message carries one object. In another graph presented below we consider different message sizes. When only one message is sent per interval, the amount of bytes transferred per second remains under 2Kbps. As the number of messages increases to 10 per interval, up to 16 Kbps are required. With a group of 50 agents sending each one message per interval, the required bandwidth varies from around 30Kbps, when the interval is two seconds, to 80Kbps when those 50 messages are all sent in each 500 milliseconds. As the time interval increases, the required bandwidth remains under 30Kbps in all cases. On the other hand, as the time interval decreases, the required bandwidth does increase. However for small groups the required bandwidth remains low. The graph in figure 5 shows the bandwidth required when replication messages of different sizes are employed. These messages vary from 100 bytes to 1500 bytes. Replication messages are sent each in intervals of 1 second. The number of messages sent in each interval is either 1, 10, 50 and 100.

576

Session Sixteen

Fault Management II

576

1.2 1.1

Required Bandwidth (Mbps)

1.0

1 msg 10 msgs 50 msgs 100 msgs

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0.1

0.2

0.3

0.4

0.5

0.6 0.7 0.8 0.9 1.0 1.1 Length of Replicated Objects (Kbytes)

1.2

1.3

1.4

1.5

Figure 5: The impact of different message lengths. If the number of replication messages is less than 10 per interval, even when the message size is 1500 bytes, the required bandwidth is always below 100Kbps. However, considering 100 messages, each of 1500 bytes, sent each interval, they may require up to 1.2 Mbps. A conclusion is that the user needs to evaluate the impact of replication on network performance before setting actual replication parameters. Nevertheless, the cost of keeping small groups is low for proper checking intervals. As small groups are enough to allow objects to be accessed when an agent is faulty, semi-active replication can be considered feasible for most current systems.

6. Conclusion Network management systems are most needed during emergencies, when the network is partly non-operational. Fault management applications must be able to work correctly in the presence of network faults. In other words: these systems should be fault-tolerant. In this work, we presented a new SNMP service that allows the examination of MIB objects of crashed agents through working agents. The service is based on agent group membership that implements semi-active replication of selected SNMP MIB objects. Besides implementing fault-tolerant MIB objects, the proposed service may also be used to improve performance: the manager polls only one agent to get objects of various agents. A practical tool was presented, in which agents built with NET-SNMP communicate using the Ensemble group communication tool. We showed an example in which this tool was used to examine the MIB of a faulty host, allowing the manager

577


577

to discover that it failed due to a TCP syn flooding attack. The evaluation of the impact of agent groups on network performance shows that small groups are feasible in current systems. Future work includes developing a flexible approach for devices to automatically join and withdraw from groups, instead of the manual configuration of the current tool. A Java based interface is also being developed that allows a manager to query objects of both faulty and working agents through any member of a group.

References [1] A. Leinwand and K. F. Conroy, Network Management: A Practical Perspective, AddisonWesley, 1996. [2] E.P. Duarte Jr., and T. Nanya, “A Hierarchical Adaptive Distributed System-Level Diagnosis Algorithm,” IEEE Transactions on Computers, Vol.47, pp.34-45, No.1, Jan 1998. [3] E.P. Duarte Jr., F. Mansfield, T Nanya, and S Noguchi, “Non-Broadcast Network FaultMonitoring Based on System-Level Diagnosis,” Proceedings of the 5th IEEE/IFIP International Symposium on Integrated Network Management (IM’97), San Diego CA, 1997. [4] S. Kätker, and M. Paterok, “Fault Isolation and Event Correlation for Integrated Network Management,” Proceedings of the 5th IEEE/IFIP International Symposium on Integrated Network Management (IM’97), San Diego CA, 1997. [5] C. S. Hood, and C. Ji, “Proactive Network Fault Detection,” Proc. INFOCOM 97, 1997. [6] K. Birman, Building Reliable and Secure Network Applications, Prentice-Hall, 1996. [7] R. Guerraoui and A. Schiper, “Software-based Replication for Fault Tolerance,” IEEE Computer, Vol. 30, No. 4, pp. 68-74, April 1997. [8] The NET-SNMP Home Page, http://net-snmp.sourceforge.net. [9] M. G. Hayden, The Ensemble System, PhD Thesis, Cornell University, Ithaca, Jan. 1998. [10] J. Schönwälder, “Using Multicast-SNMP to Coordinate Distributed Management Agents,” IEEE Workshop on Systems Management, June 1996. [11] K.-H. Lee, “A Group Communication Protocol for Distributed Network Management Systems,” In Proc. ICCC 95, pp. 363-368, 1995. [12] D. Breitgand, Group Communication as an Infrastructure for Distributed Systems Management, Master Dissertation, Hebrew University of Jerusalem, June 1997. [13] R. Guerraoui and A. Schiper, “Fault-Tolerance by Replication in Distributed Systems,” International Conference on Reliable Software Technologies, Springer Verlag (LNCS), 1996. [14] M. Wiesmann, F. Pedone, A. Schiper, B. Kemme and G. Alonso, “Understanding Repli´ cation in Databases and Distributed Systems,” Technical Report SSC/1999/035, Ecole Polytechnique Fédérale de Lausa nne, Switzerland, September 1999. [15] L. E. Moser, P. M. Melliar-Smith, and P. Narasimhan, “A Fault Tolerance Framework for CORBA,” 29th Annual International Symposium on Fault-Tolerant Computing, 1999.

578

Session Sixteen

Fault Management II

578

[16] F. B. Schneider, “Implementing Fault-Tolerant Services Using The State Machine Approach: A Tutorial,” ACM Computing Surveys, Vol. 22, No. 4, pp. 299-319, Dec. 1990. [17] X. Défago, A. Schiper, and N. Sergent, “Semi-passive Replication,” In Proceedings of the 17th IEEE Symposium on Reliable Distributed Systems (SRDS), pp. 43-50, West Lafayette, IN, USA, Oct. 1998. [18] M. A. Hiltunem and R. D. Schlichting, “A Configurable Membership Service,” IEEE Transactions on Computers, Vol. 47, No. 5, May 1998. [19] S. Mishra, C. Fetzer and F. Cristian, “The Timewheel Group Membership Protocol,” Proceedings of the 3rd IEEE Workshop on Fault-Tolerant Parallel and Distributed Systems, Orlando, FL, April 1998. [20] R. V. Renesse, K. P. Birman and S. Maffeis, “Horus: A Flexible Group Communication System,” Communications of the ACM, Vol. 39, No. 4, pp. 76-83, April 1996. [21] K. McCloghrie, “SNMPv2 Management Information Base for the Transmission Control Protocol using SMIv2,” RFC 2012, IETF, November 1996. [22] R. Farrow, “TCP SYN Flooding Attacks and Remedies,” Network Computing Unix World, http://www.networkcomputing.com/unixworld/security/004/004.txt.html. [23] E. P. Duarte Jr., and A. L. Santos, “Evaluation of Network Fault Management Based on SNMP Agent Groups,” Proceedings of the International Workshop on Applied Reliable Group Communication, Phoenix, AZ, April 2001. [24] J. Holliday, D. Agrawal, and E. Abbadi, “The Performance of Database Replication with Group Multicast,” In Proc. FTCS 99, 1999.