Reliable Distributed Network Management by Replication Aldri L. dos Santos ∗, Elias P. Duarte Jr.†, and Glenn M. Keeni
‡
This paper presents a new clustering architecture for SNMP agents that supports semiactive replication of managed objects. A cluster of agents provides fault-tolerant object functionality: replicated managed objects of a crashed agent of a given cluster may be accessed through a peer cluster. The proposed architecture is structured in three layers. The lower layer corresponds to the managed objects at the network elements. The middle layer contains management entities called clusters that monitor and replicate managed objects. The upper layer allows the definition of management clusters as well as the relationship between clusters. A practical tool was implemented and is presented. The impact of replication on network performance is evaluated as well as a probabilistic analysis of replicated object consistency. Keywords: Distributed Management, Fault Management, Clustering, Data Replication, SNMP. ∗
Dept. of Computer Science, Federal University of Minas Gerais, BH-MG, Brazil,
[email protected] Dept. of Informatics, Federal University of Paran´ a, Curitiba-PR, Brazil,
[email protected] ‡ Cyber Solutions Inc., Aoba-ku Sendai-shi, Miyagi, Japan,
[email protected] †
1
Introduction
Fault management is one of the key functions of integrated network management systems [1]. The purpose of fault management is to locate, determine the causes and, if possible, correct network faults [2]. Different approaches have been proposed to accomplish this task, including the application of system-level diagnosis for fault-tolerant network monitoring [3, 4], alarm correlation [5], and proactive fault detection [6], among others. This work presents an agent clustering architecture for the Internet standard Simple Network Management Protocol (SNMP) [7] that supports semi-active replication [8] of managed objects. Although this architecture is presented taking into consideration the widely used SNMPv3 framework [9], it is a generic architecture that can be applied to any distributed management framework. Most network monitoring systems only allow the examination of managed objects of faultfree agents. Managed objects reflect the state of managed entities thus, in order to determine why a given network element is crashed or unreachable, it is often useful to examine the MIB (Management Information Base) of its crashed agent; i.e. the value of the MIB’s objects may help the manager to determine the reason the agent has crashed or become unreachable. A cluster of agents provides fault-tolerant object functionality: replicated managed objects of a crashed agent of a given cluster may be accessed through a peer cluster. Furthermore, the cluster behaves as a cache of managed objects that reduces the impact of monitoring on network performance. The proposed architecture is structured in three layers, as shown in figure 1. The lower layer corresponds to the managed objects at the network elements called Agent (Ag). The middle layer contains management entities called cluster manager (CM) that have the task of monitoring a set of managed objects and replicating those objects in other clusters. The upper layer allows the definition of management clusters as well as the relationship between clusters. Other distributed management systems and tools have been deployed based on the classical three-tier architecture [10].
Manager
CM
CM
CM Ag
Ag Ag
Cluster 1
Ag
Ag Ag
Ag Ag
Ag
Ag
Ag
Cluster N
Ag
Cluster 2
Figure 1: The proposed three-tier agent clustering architecture. The architecture allows different means of communication to be used to send instances of replicated objects among the clusters. We propose the use of a group communication protocol [11, 12, 13] for that purpose, due to the properties and services which can be offered by such a protocol, such as reliable multicast and group membership, among others. A practical tool was built with NET-SNMP agents [14] that communicate using the Ensemble group communication toolkit [13]. As an application example we show how the examination of replicas of TCP-related objects of a crashed agent which had suffered a TCP SYN-Flooding attack led to the determination of the fault causes. A probabilistic evaluation of the impact of replication on system performance versus replica consistency is also given. Previously published related work include [15, 16] which describe a two-tier architecture in which replicated objects are kept at the network elements themselves. In [17] J. Wei et al. describe a management system based on CORBA and the group communication tool Isis [11], deployed on a set of two management stations (NMS’s) and five managed network elements, on top of a WDM (Wavelength Division Multiplexing) network. In the system described, robustness of the management functionality is provided through the object group mechanism which actively replicates management software objects across the two management stations. In the event that one management station crashes, the other station takes over seamlessly. Except for the administrator front-end graphical interface, all other management objects are replicated and span across both management stations. This way,
crashing of a management process on one management station can be tolerated, without affecting the continuous operation of the management system. In [18] J. Sch¨onw¨alder introduces an approach for defining SNMP agent groups that implement membership on top of SNMP. In [19] K.H. Lee introduces a group communication protocol that provides ordered delivery of messages in a distributed network management system. In [20] a network administration tool based on a group communication toolkit is described, which provides solutions for maintaining consistent distributed name service configurations, and allows software distribution to a number of machines. The rest of this paper is organized as follows. Section 2 presents an overview of replication techniques and group membership. Section 3 describes the agent clustering architecture. Section 4 describes the SNMP framework for semi-active replication of managed objects in agent clusters. In section 5 a practical fault management tool based on the proposed architecture is presented. Section 6 considers the impact of replication on network performance. Conclusions and future work are in section 7.
2
Replication and Group Membership
In this section we present an overview of two techniques that allow the development of reliable distributed systems: replication and group membership.
2.1
Replication
Replication [21] is a useful tool which has been applied to introduce fault tolerance in both distributed systems and databases, where it also aims at improving performance [8, 22]. Replication is the process of making a replica (a copy) of resources, processes or data. Although it is an intuitive concept, replication requires sophisticated techniques for successful implementation in a distributed system [23]. It is necessary to guarantee the consistency of replicated data. Replication is usually classified in two different types: passive replication and active replication [24]. Passive replication is also known as the primary backup approach. Primary
backup uses one replica, the primary, that plays a special role: it receives requests from client processes and returns responses. The other replicas, called backups, interact directly only with the primary replica, not the client processes. Whenever necessary, the primary sends update messages to the backups. When the primary fails, the backups must detect the failure and run a leader election algorithm in order to determine which one will become the next primary. This reconfiguration procedure is costly. The second type is called active replication, which is also called the state-machine approach. This replication technique is non-centralized, i.e. any replica can receive any update command from client processes. A client does not contact one particular replica, but addresses them as a group. The main advantage of active replication is its failure transparency, no matter how many replicas are faulty or fault-free, the data is accessed in the same way. However, a drawback of active replication is that it requires working replicas to receive the invocations of client processes in the same order, to guarantee the consistency of replicated data. query query
Group
client query
replic query
leader
query
client
replic replic
query
Figure 2: Semi-active replication: querying an object. Semi-active replication is a hybrid approach involving characteristics of both active and passive replication. In this approach, each replica processes every update command received from clients. Figure 2 shows that replicated data can be accessed through any member of the group. However, the order in which update messages are received is established by a leader. Both clients and replicas send update requests to the leader, who processes the requests, reaching a new state and then forwards the same requests to all replicas, as shown in figure
3. update
client
replica validate & run locally
update
leader
group communication tool
update
replica
update
replica
update
replica
update
replica
Figure 3: Semi-active replication: updating an object. Semi-active replication can be naturally applied to SNMP. Each agent serves as leader for its own objects. In other words, a leader is defined as the only agent that would keep the object if it were not replicated. Consider a group of agents, which have part of their individual MIB’s replicated in the group. Although any group member can be queried to access any replicated object, if a replica receives an update command, it must forward this command to the proper leader. After validating the update command, the leader propagates the updated object value to all group members, which then update their local copies. Other hybrid approaches for replication exist, such as semi-passive replication [25]. The group abstraction, presented below, is an adequate framework for providing reliable multicast primitives required to implement replication in distributed systems.
2.2
Group Membership
Reliable distributed applications must guarantee that its services are provided even when failures occur. One possible approach for constructing reliable applications consists in structuring the application into cooperating groups of processes which communicate using reliable multicast. All messages sent to the group are delivered to each group member that is not faulty. Group membership is a service that maintains consistent information about which
members are functioning and which have failed at any moment. In order to allow this, membership service reports failures, recoveries or joining members, indicating changes into the membership [26, 27]. A group communication system usually provides reliable group multicast and membership service. Several group communication systems have been implemented and are available, such as Horus [12] and Ensemble [13]. Ensemble is a group communication toolkit which allows the development of fault-tolerant complex distributed applications. Ensemble provides a library of protocols which handles issues such as reliably sending and receiving messages, transferring process states, implementing security, detecting failures and managing reconfigurations in the system. We have used Ensemble as a group communication tool for implementing a reliable distributed SNMP-based network fault management tool which is described in section 5.
3
An Agent Clustering Architecture
Network management systems provide network monitoring and control [7]. The Simple Network Management Protocol version 3 (SNMPv3) is the Internet standard management architecture [9]. There is currently a large number of SNMP-based systems available, both commercial and in the public-domain. The agent clustering architecture presented below considers the SNMPv3 framework, given its huge popularity worldwide. Nevertheless, it is a generic architecture that can be applied to any other distributed management framework. An SNMPv3 system is composed of management entities which communicate using the management protocol. The architecture defines a Management Information Base (MIB) as a collection of related management objects which are kept in order to allow management applications to monitor and control the managed elements. SNMPv3 entities have traditionally been called managers and agents. A managed element contains an agent, which is a management entity that has access to management instrumentation. Each system has at least one Network Management Station, which runs at least one manager entity. Managers are collections of user-level applications, which provide services such as performance evaluation, fault diagnosis, and accounting, among others.
Distributed network management is widely recognized as a requirement for dealing with large, complex and dynamic networks [10]. The three-tier architecture has become popular for distributed management, in which mid-level distributed management entities are included between agents in the bottom level and managers in the top level. The distributed entities assume roles like executing scripts [28], monitoring events [29], scheduling actions [30] or reporting alarms [31], among others.
3.1
Managed Object Replication Based on Agent Clustering
It is often useful to examine the state of managed objects of a crashed or unreachable network element in order to determine why it has crashed or is unreachable. In current network monitoring systems, it is impossible to examine the MIB of such objects under such circumstances. We present below a three-tier clustering architecture for SNMP agents that supports semi-active replication of managed objects. A cluster of agents provides faulttolerant object functionality: replicated managed objects of a crashed agent of a given cluster may be accessed through a peer cluster. Agents are clustered for scalability reasons. Otherwise, every agent has to manage distributed replication, which could be expensive or even impossible depending on the agent’s resources. The clustering approach employed by the architecture is similar to the one introduced in [32] for reducing the number of message exchanges in mobile network management systems. In this work, we use agent clustering to support managed object replication. The replication technique used in the agent clustering architecture is semi-active replication [8, 16]. The proposed architecture is structured in three layers, as shown in figure 4 and is described below. The three layers are called the cluster member layer, the cluster layer, and the manager layer, from bottom to top. • The lower layer keeps management entities called cluster members and shown as (Ag) in figure4. These entities have a specified subset of their objects monitored and replicated in a cluster, as defined below. • The middle layer keeps management entities called cluster managers (CM). Each cluster
has the task of monitoring a subset of the managed objects in its cluster members and replicating those objects in other clusters. • The upper layer keeps Management Applications (MA) which define cluster managers, as well as the relationship among clusters.
MA
CLUSTER i
Ag
REPLICATION
Ag
CLUSTER MEMBERS
CLUSTER j
Ag
Ag
CLUSTER MEMBERS
Figure 4: The replication architecture. Agent clusters are defined at the manager layer. Clusters are entities of the middlelayer that monitor and replicate objects of the agents, which are at the bottom layer. The architecture allows different means of communication, such as group membership, to be used to send instances of replicated objects among the clusters.An agent can belong to multiple clusters. Clusters can be shared across management applications. The architecture does not put any restriction on how the agent groups are formed, the criteria for grouping a set of agents into a cluster depends on the set of monitored objects and the specific network’s monitoring policy.
3.2
Using the Clustering Architecture
Figure 5 shows an example in which three agent clusters are monitored by cluster managers CM1, CM2, and CM3. The first cluster manager, CM1, monitors the objects of the agents Ag1, Ag2, Ag3 and Ag4. The second, CM2, monitors the objects of the agents Ag5, Ag6, and Ag7; and CM3 monitors the objects of the agents Ag8, Ag9, Ag10, Ag11, and Ag12. Figure 5
shows the behavior of CM1. Besides monitoring the objects of its agents, and keeping replicas of those objects, CM1 also replicates those replicas in the other two peer cluster managers, as shown by the dotted lines in the figure. Considering the semi-active replication approach employed, CM1 is the leader for all replicated objects of its agent cluster.
CM1
CM3
CM2 Ag2
Ag1
Ag9 Ag8
Ag4
Ag3
Ag7
Ag5
Cluster 1
Ag11 Ag10
Ag12
Cluster 3
Ag6
Cluster 2 Ag
Agent Member
CM
Cluster Manager
Managed Objects
Figure 5: The operation of the clustering architecture. Figure 6 presents the steps executed by a manager to obtain objects monitored by cluster manager CM1. In this case, all cluster managers contain replicas of the objects monitored by all clusters. First, the manager makes a query to CM1. If no reply arrives, as in the example, the manager still has the option to obtain that information through the peer clusters. Next, the manager makes a query to CM2 and obtains the replicated objects from CM1. Manager
CM1 CM2 Ag2
Ag1 Ag3
Ag4
Cluster 1
Figure 6: Querying replicas.
CM3
4
An SNMP Framework for Object Replication in Agent Clusters
In this section we present an SNMP agent clustering framework that supports semi-active replication of management objects. The MIB that allows the definition and usage of clusters as well as replication is described. This MIB is divided in two groups: clusterDefinition and clusterReplica, as shown in figure 7 and described below. An example of the framework usage is given together with the description of the MIB objects. replicObjects clusterDefinition clusterTable memberTable repObjectTable peerTable clusterReplication replicaTable
Figure 7: Replic-MIB: basic structure. The clusterDefinition group consists of four tables that are used to define agent clusters: clusterTable, memberTable, repObjectTable, and peerTable. The clusterTable is located in a manager application and contains the definition of all agent clusters. The clusters are built automatically by the manager application. Each cluster contains the definition of its cluster members (memberTable), the specification of managed objects to be replicated (repObjectTable), and the definition of peer clusters (peerTable), i.e. those in which replicas are kept. The clusterReplication group consists of a single core table called replicaTable, which is kept by each cluster. This table is automatically built and keeps replicated managed objects from each agent member of a cluster as well as from its peer clusters. The description of the main portions of the MIB framework are given below. The complete MIB design is found in [33].
4.1
Defining Management Clusters
An SNMP manager application allows the definition of all agent clusters, their members, and the managed objects to be replicated. As shown described below, clusterTable is a table used to define and keep information required to create each agent cluster. The clusterIndex identifies an entry of table clusterTable. Object clusterID identifies a cluster. Object clusterAddressType describes the type of address in clusterAddress. This address identifies the cluster manager, i.e. the SNMP entity that monitors a set of agents. Object clusterMemberType and object clusterMember define respectively the type of address and the address of each member of a given cluster. Object clusterOID is the OID (the ASN.1 object identifier) of a managed object. Object clusterInstanceIndex defines the instance index of a managed object which is replicated. Object clusterRepClusterID keeps the identifier of the peer clusters. Object clusterName keeps information about the human manager responsible for the cluster. Object clusterDescr describes the purpose of the cluster, and the clusterStatus indicates the status of the cluster.
Index
ID
Address
Address
Type
Member
Member
OID
Type
Instancee
Rep
Index
CID
Name
Descr
Status
1
Ci
ipv4(1)
10.0.0.1
ipv4(1)
Mb1
ifInOctets
1
Cj
John
example
active(1)
2
Ci
ipv4(1)
10.0.0.1
ipv4(1)
Mb1
ifInOctets
2
Cj
John
example
active(1)
3
Ci
ipv4(1)
10.0.0.1
ipv4(1)
Mb2
ifInOctets
1
Cj
John
example
active(1)
4
Ci
ipv4(1)
10.0.0.1
ipv4(1)
Mb2
ifInOctets
2
Cj
John
example
active(1)
5
Cj
ipv4(1)
10.0.0.2
ipv4(1)
Mb3
ifInOctets
1
Ci
John
example
active(1)
6
Cj
ipv4(1)
10.0.0.2
ipv4(1)
Mb4
ifInOctets
1
Ci
John
example
active(1)
Table 1: An example cluster table as defined at the manager application level. Table 1 illustrates an example repClusterTable. This table defines a system consisting of two clusters: Ci and Cj, which have IP addresses 10.0.0.1 and 10.0.0.2, respectively. The type of address in each cluster is ipv4(1). Cluster Ci contains two members: M b1 and M b2, and monitors the instance indexes 1 and 2 of the if InOctets OID (the ASN.1 object identifier). Those managed objects are kept replicated in Ci and Cj, which is the peer cluster. Cluster Cj contains two members, M b3 and M b4, and only monitors instance index 1 of the if InOctets OID. The corresponding OID is replicated in Cj and Ci. if InOctets
OID keeps the number of octets that has arrived through a given network interface [7].
4.2
Specifying Cluster Members
An agent cluster is actually an agent that has the task of monitoring a subset of managed objects of a collection of agents called cluster members, and replicating selected objects in peer clusters. Table memberTable is used to define and keep cluster member information. Consider a cluster consisting of multiple members. Each member is assigned an identifier called cmIndex. This identifier is an integer, and is unique for a specific member. Objects cmAddressType and cmAddress define respectively the type of the address and the cluster member address. The security mechanism used to access an agent is defined by object cmSecurity while object cmStatus defines the status of an agent, i.e. if entry in the member table is complete. Table 2 illustrates repMemberTable for this example. Index
Address Type
Address
Security
Status
1
ipv4(1)
10.0.0.3 (Mb1)
community
active(1)
2
ipv4(1)
10.0.0.4 (Mb2)
community
active(1)
Table 2: An example member table in cluster Ci. Continuing the example above, consider cluster Ci. This cluster defines members M b1 and M b2, which have IP addresses 10.0.0.3 and 10.0.0.4, respectively. The type of address in each member is ipv4(1). Furthermore, each member employs community as the security mechanism.
4.3
Specifying Replicated Objects
Besides specifying the members of a cluster, it is also necessary to define which objects are replicated in the cluster. Table repObjectTable is used to determine which objects are replicated in a given cluster. Each entry of table repObjectTable contains an instance OID (the ASN.1 object identifier) and an instance index of a managed object, respectively roOID and roInstanceIndex. Object roInterval defines the time frequency on which the managed object is monitored
and replicated. Object roState determines whether the managed object is replicated in the cluster, and object roStatus indicates whether the entry information is complete. Table 3 illustrates repObjectTable used as example.
Index
OID
InstanceIndex
Interval
State
Status
1
ifInOctets
1
2 seconds
active(1)
active(1)
2
ifInOctets
2
2 seconds
active(1)
active(1)
Table 3: An example replicated object table in cluster Ci. Continuing the example above, consider cluster Ci. This cluster specifies two managed objects if InOctets.1 and if InOctets.2 to be monitored and replicated at a time interval of 2 seconds. The state of each of those managed objects are also active(1), i.e. the cluster Ci is monitoring and keeping replicas of these objects.
4.4
Specifying Peer Clusters
Besides monitoring objects of cluster members, each cluster has the task of keeping replicas of managed objects monitored by other clusters, called peer clusters. Table peerTable is used to define peer clusters that maintain replicas of the managed objects monitored by a given cluster. In order to identify a peer cluster in a given cluster, it is assigned an identifier called pcIndex. Each entry also keeps object pcAddress and pcAddressType which respectively keep the peer cluster address and its type. Object pcROTIndex indicates the index of an object in the replicated object table. This object is replicated in the peer cluster. Object pcStatus indicates whether the peer cluster entry is complete.
Index
Address Type
Address
ROTIndex
Status
1
ipv4(1)
10.0.0.2 (Cj)
1
active(1)
2
ipv4(1)
10.0.0.2 (Cj)
2
active(1)
Table 4: An example peer table in cluster Ci.
Continuing the example above, consider cluster Ci. As shown in table 4, this cluster defines Cj as its peer cluster. The address type of Cj is ipv4(1). The index of the managed objects defined in repObjectTable and replicated in Cj are 1 and 2, respectively.
4.5
Keeping and Accessing Replicated Objects
In order to allow SNMP clusters to keep replicated objects that SNMP managers can access through any peer cluster, table replicaTable is defined. This table is automatically built when the system is initialized. It keeps replicated object instances of all cluster members’ objects and peer cluster member objects. Each entry of table replicaTable contains the following information. Object repIndex indexes the table. Object repPeer keeps the address of the peer cluster from which replicas are obtained. Object repPeerType keeps the corresponding peer address type. Object repMember keeps the address of the agent (cluster member) from which replicas are obtained. Object repMemberType keeps the corresponding agent address type. Object repOID and object repInstanceIndex define respectively an instance OID and an instance index of a replicated managed object. Object repValue and repValueType define respectively the value of an instance of a replicated object and its data type. Object repTimeStamp contains the time elapsed since a value of an instance of a replicated object was last updated, and object repStatus indicates whether the replica entry is complete. After the manager activates the clusters, each cluster starts to monitor the specified managed objects of all cluster members. For each managed object a time interval is specified in order to determine the frequency in which the cluster monitors the object. At each time interval of each object the cluster polls all cluster members and checks whether the object values have been updated since the previous interval. The new values are then updated in table replicaTable. Next, the object instances are replicated at all clusters defined as peers for that object, they are maintained in table replicaTable. Thus, each replicaTable maintains a replica of the objects monitored by the local cluster and also replicas of the peer clusters. The time interval must be carefully chosen to allow objects in the clusters and peers to be consistent, and avoid a high impact on network performance.
Continuing the example above, table replicaTable is shown in table 5. This table is kept by all clusters. Table replicaTable is indexed by the tuple (repIndex,repPeer,repMember), and is under subtree clusterReplication.
Index
Address
Peer
Type
Member
Member
OID
Type
Instance
Value
Index
Value
TimeStamp
Status
Type
1
ipv4(1)
Ci
ipv4(1)
Mb1
ifInOctets
1
124
counter32(4)
0:00:35.24
active(1)
2
ipv4(1)
Ci
ipv4(1)
Mb1
ifInOctets
2
145
counter32(4)
0:00:36.83
active(1)
3
ipv4(1)
Ci
ipv4(1)
Mb2
ifInOctets
1
120
counter32(4)
0:00:37.89
active(1)
4
ipv4(1)
Ci
ipv4(1)
Mb2
ifInOctets
2
123
counter32(4)
0:00:38.89
active(1)
5
ipv4(1)
Cj
ipv4(1)
Mb3
ifInOctets
1
200
counter32(4)
0:00:32.17
active(1)
6
ipv4(1)
Cj
ipv4(1)
Mb4
ifInOctets
1
300
counter32(4)
0:00:33.77
active(1)
Table 5: An example replica table in cluster Ci. Still continuing the example above, now consider that agent M b1 has crashed. The manager then queries cluster manager (CM) in order to obtain the crashed agent’s objects. The following snmpget commands may be employed to access instances of objects ifInOctets of member M b1 of cluster Ci’s from any peer cluster CM. snmpget agent= object=.... snmpget agent= object=....
These commands are received by cluster manager CM, which can be either Ci or Cj, because objects are replicated in both clusters. No matter which cluster is the cluster manager (CM), the response received is as shown below. response: object=clusterReplication.Ci.Mb1.ifInOctets.1 = Counter32: 8282047 response: object=clusterReplication.Ci.Mb1.ifInOctets.2 = Counter32: 117834230
5
A Practical Tool
This section describes a practical tool based on the framework presented in the previous section that allows the examination of replicated objects of crashed or unreachable agents. The tool is based on the public-domain NET-SNMP package [14] and Ensemble, a group
communication toolkit [13]. The tool allows the creation of SNMP agent clusters that support semi-active replication of selected objects. Clusters are dynamic, in the sense that they can be created and destroyed at any time. We describe how the tool has been used to determine that a given crashed agent had suffered TCP-based attacks. SNMP GROUP TOOL
UDP
NETWORK
UDP
UDP
UDP
GROUP TOOL
GROUP TOOL
GROUP TOOL
SNMP
SNMP
SNMP
Figure 8: SNMP agents interact with a group communication tool. A cluster monitors a set of objects of a group of members. The system is based on cooperating groups of SNMP agents defined with a group communication tool [12]. Each group is a collection of processes that usually communicate using reliable multicast. As shown in figure 8, in the new approach the group communication tool is located between SNMP and UDP, i.e. SNMP messages are sent via the group communication tool to a collection of agents; messages are encapsulated in UDP segments. We have used the Ensemble group communication toolkit to add reliable multicast capability to SNMP agent clusters. Ensemble provides reliable communication to groups of processes, providing ordered channels between all peer clusters. A service for managing multiple clusters is provided, called mclusterd which implements a group membership protocol [13]. This protocol allows all peer clusters to detect failures and guarantees that they all see the same set of failures. Ensemble supports IP multicast, but also provides the gossip application which simulates low-bandwidth gossip broadcast for systems that do not have IP multicast. Considering the SNMP framework presented above, each cluster must include tables memberTable, repObjectTable, peerTable, and replicaTable. From the moment a clus-
ter is actived, objects are automatically monitored and replicated throughout peer clusters. Semi-active replication is employed among the peer clusters, each cluster manager is the leader for the replicated objects of its respective members. Each cluster is an SNMP agent that is capable of monitoring a collection of objects of an agent group, besides receiving and handling messages with replicated objects that arrive from the peer clusters. A thread was added which parses each message and updates the replicated objects. A query to an agent cluster is processed in the standard way. Furthermore, each replicated object in any agent cluster has a timestamp that keeps the information about how much time has elapsed since it was last updated. The tool allows any peer cluster to be queried in order to obtain the replicated objects of any agent that belongs to the cluster. Detection of TCP SYN-Flooding Attacks An experiment was executed in a local network composed by 26 machines based on different Intel Pentium and AMD K6 processors, running Linux and connected through a 100Mbps Ethernet LAN. The SNMP agent cluster was installed in two machines called genio and ork in order to monitor a subset of TCP managed objects. Each cluster monitored a set of 12 machines and replicated TCP MIB objects to the other cluster. A machine monitored by cluster genio was subjected to a TCP SYN-Flooding attack [34]; it crashed after a few seconds under the attack. In order to examine the TCP MIB objects of the crashed agent, cluster genio was invoked. Replicated objects included tcpPassiveOpens, which gives the number of times TCP connections have made a direct transition to the SYN-RCVD state from the LISTEN state [35]. Another replicated object was tcpCurrEstab, the number of TCP connections for which the current state is either ESTABLISHED or CLOSE-WAIT. The value of the MIB objects left no doubt as to the cause of the attack, previous logs showed that usually tcpPassiveOpens was equal to zero, while after the failure it was equal to 85; object tcpCurrEstab remained at the usual value, equal to 12. The same values obtained from cluster genio were obtained through the replica in cluster ork. If the tool based on replicated objects had not been available, it would have been impossible to examine the MIB of this machine after it
crashed.
6
Impact of Clustering on Network Performance
When replicated objects at a given cluster are updated, messages are sent to peer clusters so that they can update their replicas; thus replication has an impact on network performance given by the amount of network resources required to transmit replication messages. Furthermore, if replication messages are not sent frequently enough, replicas may become inconsistent, i.e. their object values may be different from the object values kept by the original cluster. We present a probabilistic evaluation of how inconsistent replicas can become considering different update intervals versus the impact on network performance. Standard Deviation 2.5 time units 0.10 0.09
0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 0.5
Object Update Interval Mean 6 Mean 7 Mean 8
0.9 Probability of Inconsistent Replicas
0.08 Required Bandwidth (Mbps)
1.0
Number of Messages 1 message 10 messages 50 messages
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
A: Frequency in which Replicated Objects are Updated (time units)
1.9
2.0
0
1
2
3
4
5
6
7
8
9
10
11
B: Replica Update Interval (time units)
Figure 9: The impact on network performance and on replica consistency. The replica update interval is the time interval in which a given cluster polls the cluster members and sends replica update messages, if necessary, to its peer clusters. The graph in figure 9-A shows the required bandwidth considering that replication messages are sent in different replica update intervals, varying from one each 500 milliseconds to one each 2 seconds. The number of messages sent varies from 1 to 50, and the size of all messages is 100 bytes, i.e. each message carries information about one object. When one message is sent per interval, the amount of bytes transferred per second remains under 2Kbps. As the number of messages increases to 10 per interval, up to 16 Kbps is
required. With a group of 50 clusters each sending one message per interval, the required bandwidth varies from around 20Kbps, when the interval is two seconds, to 80Kbps when those 50 messages are all sent in each 500 milliseconds. As the time interval increases, the required bandwidth remains under 30Kbps. On the other hand, as the time interval decreases, the required bandwidth does increase. Nevertheless, for small clusters the required bandwidth remains low considering current network technologies. Consider the graph in figure 9-B. This graph shows the impact of varying the replica update interval on replica consistency. We considered the Gaussian distribution for leader update intervals. The mean varies from 6 to 8 time units. The standard deviation is fixed at 2.5 time units. The graph gives the probability that replicas are inconsistent as the replica update interval varies. When the replica update interval is 1 time unit, the amount of replicas that will become inconsistent is under 5%. As the replica update interval increases to 4 time units, the inconsistency varies according to the mean, from 5% to 22%. If the replica update interval is equal to the mean leader update interval, then 50% of the replicas will be inconsistent. Finally, if the replica update interval is 11 time units, then from 88% to 97% of the replicas will be inconsistent. The proposed architecture can be used in any kind of system to monitor managed iobjects that behave according to widely varying patterns. The purpose of the probabilistic evaluation presented and the usage of the Gaussian distribution is to give an idea of how an implementation of the architecture will behave for a generic situation in which we assume that objects are updated according to the Gaussian distribution. Future work may also consider other distributions. In conclusion, in order to determine the replica update interval, the human network manager must evaluate the behavior of replicated objects. SNMP objects change their values in widely different intervals. A case by case statistical analysis is required to determine the best replica update interval for a given object. The manager must get enough samples of update intervals, computing the mean and standard deviation. Depending on these parameters and the performance impact evaluation, the replica update interval is set.
7
Conclusions
Network management systems are most needed during emergencies, when the network is partly non-operational. Fault management applications must be able to work correctly in the presence of network faults. In this work, we presented a new clustering architecture for SNMP agents that supports semi-active replication of managed objects. A cluster provides fault-tolerant object functionality: replicated managed objects of a crashed agent of a given cluster may be accessed through a peer cluster. Furthermore, the cluster behaves as a cache of managed objects that reduces the impact of monitoring on network performance. As this is a three-tier architecture, based on middle-layer agents which are responsible for monitoring and replicating objects, it can be expanded and used by any existing system based on SNMP. A practical tool was presented, in which agents communicate using the Ensemble group communication tool. We showed an example in which this tool was used to examine the MIB of a crashed network element, allowing the manager to discover that it has crashed due to a TCP SYN-Flooding attack. The evaluation of the impact of agent clustering on network performance shows that clustering is feasible in current systems. Future work includes developing a flexible approach for network elements to automatically join and withdraw from clusters, instead of the manual configuration of the current tool. A Java based interface is also being developed that allows a manager to query objects of both crashed and working agents through a cluster or peer cluster.
References [1] A. Leinwand and K. F. Conroy, Network Management: A Practical Perspective, AddisonWesley, 1996. [2] D. Medhi, and D. Tipper, Journal of Network and Systems Management: Special Issue on Fault Management in Communication Networks, Vol. 5, No. 2, June 1997. [3] E.P. Duarte Jr., and T. Nanya, “A Hierarchical Adaptive Distributed System-Level Diagnosis Algorithm,” IEEE Transactions on Computers, Vol.47, pp.34-45, No.1, Jan 1998.
[4] E.P. Duarte Jr., A. Weber, “A Distributed Network Connectivity Algorithm,” Proceedings of the 6th IEEE International Symposium on Autonomous Decentralized Systems (ISADS’2003), Pisa, Italy, 2003. [5] S. K¨atker, and M. Paterok, “Fault Isolation and Event Correlation for Integrated Network Management,” Proceedings of the 5th IEEE/IFIP International Symposium on Integrated Network Management (IM’97), San Diego CA, 1997. [6] C. S. Hood, and C. Ji, “Proactive Network Fault Detection,” Proc. INFOCOM 97, 1997. [7] W. Stallings, Snmp, Snmpv2, Snmpv3 and Rmon 1 and 2, Addison-Wesley, Reading, MA, 1999. [8] M. Wiesmann, F. Pedone, A. Schiper, B. Kemme and G. Alonso, “Understanding Repli´ cation in Databases and Distributed Systems,” Technical Report SSC/1999/035, Ecole Polytechnique F´ed´erale de Lausa nne, Switzerland, September 1999. [9] D. Harrington, R. Presuhn, and B. Wijnen, “An Architecture for Describing SNMP Management Frameworks,” RFC 2271, January 1998. [10] Distributed Management (DisMan) Charter http://www.ietf.org/html.charters/disman-charter.html [11] K. Birman, Building Reliable and Secure Network Applications, Prentice-Hall, 1996. [12] R. V. Renesse, K. P. Birman and S. Maffeis, “Horus: A Flexible Group Communication System,” Communications of the ACM, Vol. 39, No. 4, pp. 76-83, April 1996. [13] M. G. Hayden, The Ensemble System, PhD Thesis, Cornell University, Ithaca, Jan. 1998. [14] The NET-SNMP Home Page, http://net-snmp.sourceforge.net. [15] E.P. Duarte Jr. and Aldri L. dos Santos, “Semi-Active Replication of SNMP Objects in Agent Groups Applied for Fault Management,” Proceedings of the 7th IFIP/IEEE International Symposium on Integrated Network Management (IM’01), Seattle, May 2001. [16] E.P. Duarte Jr. and Aldri L. dos Santos, “Network Fault Management Based on SNMP Agent Groups,” Proceedings of the IEEE 21st International Conference on Distributed Computing Systems Workshops (ICDCS’2001), Workshop on Applied Reliable Group Communications, pp. 51-56, Mesa, Arizona, April 2001.
[17] J. Wei, C. Shen, B. J. Wilson, and M. J. Post, “Network Control and Management of a Reconfigurable WDM Network,” Proceedings of the Military Communications Conference (MILCOM’96), Mclean, Virginia, October 1996. [18] J. Sch¨onw¨alder, “Using Multicast-SNMP to Coordinate Distributed Management Agents,” IEEE Workshop on Systems Management, June 1996. [19] K.-H. Lee, “A Group Communication Protocol for Distributed Network Management Systems,” In Proc. ICCC 95, pp. 363-368, 1995. [20] D. Breitgand, Group Communication as an Infrastructure for Distributed Systems Management, Master Dissertation, Hebrew University of Jerusalem, June 1997. [21] R. Guerraoui and A. Schiper, “Fault-Tolerance by Replication in Distributed Systems,” International Conference on Reliable Software Technologies, Springer Verlag (LNCS), 1996. [22] L. E. Moser, P. M. Melliar-Smith, and P. Narasimhan, “A Fault Tolerance Framework for CORBA,” 29th Annual International Symposium on Fault-Tolerant Computing, 1999. [23] R. Guerraoui and A. Schiper, “Software-based Replication for Fault Tolerance,” IEEE Computer, Vol. 30, No. 4, pp. 68-74, April 1997. [24] F. B. Schneider, “Implementing Fault-Tolerant Services Using The State Machine Approach: A Tutorial,” ACM Computing Surveys, Vol. 22, No. 4, pp. 299-319, Dec. 1990. [25] X. D´efago, A. Schiper, and N. Sergent, “Semi-passive Replication,” In Proceedings of the 17th IEEE Symposium on Reliable Distributed Systems ( SRDS), pp. 43-50, West Lafayette, IN, USA, Oct. 1998. [26] M. A. Hiltunem and R. D. Schlichting, “A Configurable Membership Service,” IEEE Transactions on Computers, Vol. 47, No. 5, May 1998. [27] S. Mishra, C. Fetzer and F. Cristian, “The Timewheel Group Membership Protocol,” Proceedings of the 3rd IEEE Workshop on Fault-Tolerant Parallel and Distributed Systems, Orlando, FL, April 1998. [28] D. Levi and J. Sch¨onw¨alder, “Definitions of Managed Objects for the Delegation of Management Scripts,” RFC 3165, August 2001. [29] R. Kavasseri and B. Stewart, “Event MIB,” RFC 2981, October 2000. [30] D. Levi and J. Sch¨onw¨alder, “Definitions of Managed Objects for Scheduling Management Operations,” RFC 3231, January 2002.
[31] S. Chisholm and D. Romascanu, “Alarm MIB,” Working in Progress, December 2001. [32] W. Chen, N. Jain, and S. Singh, “ANMP: Ad Hoc Network Management Protocol,” IEEE Journal on Selected Areas in Communications, Vol. 17, No. 8, August 1999. [33] Aldri L. Santos, Elias P. Duarte Jr., and Glenn Mansfield, “A Clustering Architecture for Replicating Managed Objects,” Work in Progress, November 2001. Available at: http://www.rfc-editor.org/internet-drafts/draft-aldri-disman-replication-mib-00.txt [34] R. Farrow, “TCP SYN Flooding Attacks and Remedies,” Network Computing Unix World, http://www.networkcomputing.com/unixworld/security/004/004.txt.html. [35] K. McCloghrie, “SNMPv2 Management Information Base for the Transmission Control Protocol using SMIv2,” RFC 2012, IETF, November 1996.
Biographies Aldri L. dos Santos is a Ph.D. student at the Department of Computer Science of Federal University of Minas Gerais, Belo Horizonte, Brazil. Aldri received both his M.Sc. and B.Sc. in Informatics from Federal University of Parana, Curitiba, Brazil. From June to August 2001 he was an invited researcher at Cyber Solutions Inc., Sendai, Japan. Elias P. Duarte Jr. is an Associate Professor at the Department of Informatics of Federal University of Parana, Curitiba, Brazil. Elias received his Ph.D. in Computer Science from Tokyo Institute of Technology, Japan, 1997, M.Sc. in Telecommunications from the Polytechnical University of Madrid, Spain, 1991, and both a B.Sc. and M.Sc. in Computer Science from Federal University of Minas Gerais, Belo Horizonte, Brazil. He is a Researcher of the Brazilian National Research Counsil, CNPq, research interests include Network Management and Fault Tolerant Distributed Computing. Elias is a member of the IEEE and the Brazilian Computing Society (SBC). Glenn Mansfield Keeni is President and CEO of Cyber Solutions Inc. He is also a visiting researcher at Tohoku University. His areas interest are Internet security, management and applications. He is a member of the ACM and the IEEE.
