mib-2.system.sysContact, for example), every measurement for that OID will have a value of 0.0. 3. Notification Process. 3.1 Notifier Code. Run-time Parameters.
Predictive Notification of QoS Shortfall with the Network Weather Service Robert E. Busby, Jr.*
Daniel Andresen**
Mitchell L. Neilsen**
1. Introduction 1.1 The Network Weather Service The Network Weather Service (NWS) is a distributed resource monitoring system, originally developed by Rich Wolski, Neil Spring, and Jim Hayes of The University of California, San Diego as an aid in scheduling jobs in metacomputing environments [4]. The system tracks utilization of key system and network resources, and provides short-term predictions as to the future utilization of these resources. A scheduler process interested in a particular, critical resource can query NWS for a forecast of the utilization of that resource, and so select a host system that will likely provide the resource in sufficient quantities for the job to be scheduled. Currently, NWS measures the following resources: • • • •
CPU Availability TCP Call Setup Time TCP Network Latency TCP Network Bandwidth
A process known as a sensor collects resource utilization information. There are two types of sensors: passive and active. The activity of collecting information in a passive sensor does not impact the resource being monitored. An example of a passive sensor is the CPU Sensor, which determines CPU availability by parsing the output of the Unix uptime and vmstat utilities.1 In contrast, an active sensor performs a resource experiment, actively consuming a small amount of the resource it is monitoring in order to measure the overall availability of that resource. The Network Latency Sensor takes this approach, sending a small packet of data from one host to another to measure the time required. NWS uses an adaptive approach to forecasting. Several forecast methods are employed in parallel whenever a forecast is requested. For each forecast request, NWS chooses the two methods that have historically produced the lowest MSE (Mean Squared Error) and MAE (Mean Absolute Error) for that measurement. The client application is then free to select from between these two values, based upon the characteristics of the application. The distributed nature of NWS enhances not only its performance (by utilizing the computing resources of several systems), but also its fault tolerance and expandability. Processes may be replicated across systems *
AT&T Network Operations and Engineering, 12976 Hollenberg Drive, Bridgeton, MO 63044. Department of Computing and Information Sciences, 234 Nichols Hall, Kansas State University, Manhattan, KS 66506. ** Department of Computing and Information Sciences, 234 Nichols Hall, Kansas State University, Manhattan, KS 66506. 1 The CPU sensor actually contains both active and passive components. Primarily, NWS CPU experiments rely on passive measurements, by using utilities that simply examine data locations in the Unix kernel. **
1
to provide increased capacity as well as to eliminate single points of failure. A client process can tolerate an NWS process failure by switching to one of the failed process’s analogues. The modular nature of the system allows for the easy incorporation of new types of processes, for instance to provide a new forecasting method.
Goals In designing NWS, the researchers have attempted to develop a system with the following characteristics: •
Accurate: By employing adaptive forecasting strategies and allowing for the easy incorporation of new forecasting methodologies, NWS strives to provide the most accurate forecast possible.
•
Non-intrusive: NWS uses passive sensors whenever possible to reduce the impact of measuring the system. When active sensors are employed, great care is taken to assure that no more than one experiment be conducted concurrently utilizing the same resource.
•
Persistent: The system is designed to run continuously so as to be available as a general service to any client process at any time.
•
Ubiquitous: The designers of NWS intend for it to be a general-purpose service, with the ability to measure every resource that may be of interest to client applications. NWS is intended to run in as many operating environments as possible, and to be available from all sites within a network.
Components The Network Weather Service currently consists of four different types of component processes: •
Persistent State Server: Data that must be retained between invocations of the system are stored in primary and secondary memory by the Persistent State (or Memory) Server. The Memory Server is responsible for storing resource measurement data.
•
Name Server: The Name Server allows NWS component modules to register their services and network locations, and provides a method for client processes to dynamically locate these services. The Persistent State Server can perform the Name Server function, or a separate process may perform it, perhaps on a separate machine.
•
Network and CPU Sensor: This process performs the resource experiments and stores the results with the Memory Server. The sensors are organized in a hierarchy of cliques, which cooperate to reduce the contention that would be caused by simultaneous experiments.
•
Forecaster: The forecaster process allows client programs to retrieve resource measurement data and predictions.
1.2 Enhancements to Enable QoS Shortfall Notification While originally designed as a metacomputing scheduling aid, NWS has broader applications, particularly in the field of proactive system and network management. With currently developed applications making stronger and stronger demands on the resources of distributed systems, there is a need for these applications to be aware of and to react to changes in the “Network Weather” in such a manner that the services they provide are, at worst, gracefully degraded.
2
An example of an application that may take advantage of such a service is an audio teleconferencing service [1], which has stringent quality of service (QoS) requirements, particularly regarding network latency. Packets must be delivered from the server to the client machines at regular intervals, or the audio quality will suffer a defect known as “jitter”. The application may overcome an increase in latency by providing more information per packet, reducing the arrival frequency and thereby increasing the opportunity for queuing incoming packets at the client machine. To accomplish this, a reduction in audio quality might be initiated, reducing the size of the information as delivered to the client, enabling the server to deliver more information without increasing the bandwidth requirements (which, presumably, are already being used at capacity). The ideal situation would be for the server to recognize and react to these conditions before they can adversely impact the service. If successful, the users of the service may notice only a slight reduction in the fidelity of the delivered audio, which would be preferable to the fits and starts that are noticed by the users when packet delivery is delayed. This project attempts to enhance the Network Weather Service in a number of ways: •
Broader Metric Set: While NWS already provides tracking for a few very important metrics, there are many more that are currently being tracked by operating systems and user processes that may be of interest to client applications. A standards-based approach is needed to provide a method to harvest these and future metrics without the need for custom programming.
•
Asynchronous Notification: Requiring the application to constantly monitor a resource may be too much of a burden. A more convenient approach would be to provide a persistent service that client programs might utilize that would monitor the resource externally, and provide an interrupt to the client when a notable change is recognized.
•
Forecast and Trend Visualization: NWS clients consist not only of computer processes, but also human network managers and system operators. These humans need to visualize the metrics and forecasts provided by NWS. With the proper tools, human network managers can recognize and react to trends in resource utilization far more effectively than even the most sophisticated software. NWS should allow users to utilize the visualization tools that are already in the network management arsenal, as well as allow for the easy utilization of new tools that may be released in the future.
System Components •
SNMP Sensor: The Simple Network Management Protocol (SNMP) [3] is currently the most common network management protocol in use in the United States. Performance data from a wide variety of networking equipment and computer systems and applications are available from SNMP agents running on this equipment. The SNMP Sensor utilizes the SNMP protocol to retrieve information from any SNMP agent’s Management Information Base (MIB), and stores this information with the NWS Memory Server, allowing predictions to be made about a much wider variety of network and system metrics. By using a standard protocol, this sensor is able to gather information about network elements that NWS could not previously reason about. Previously, in order for NWS to gather data about a network element, that element had to be capable of running the NWS Network Sensor program, limiting measured entities to general-purpose computer systems running some variant of the Unix operating system. The SNMP Sensor, in contrast, is able to retrieve information from any network element that implements an SNMP agent, including non-Unix computer systems, routers, switches, hubs, and modems. Given that an SNMP agent is already present and gathering statistics on a network element, the SNMP Sensor serves as a reasonably non-intrusive measurement instrument, utilizing only the CPU resources
3
and network bandwidth necessary for the SNMP agent to respond to an SNMP Get request. This may provide a passive alternative to measuring values that previously required an active sensor. The SNMP Sensor is separately configured from the existing NWS Network Sensor, and it does not participate in the clique protocol. There may be any number of SNMP Sensors running in parallel on any number of NWS hosts. •
Notification Process: To facilitate asynchronous notification of changes in resource utilization, a separate Notification Process has been developed. This process accepts registration requests across the TCP/IP network from client applications. Client processes utilize a single-procedure interface library to pass information to the Notification Process. The Notification Process then spawns a background process that periodically retrieves NWS metrics on the client’s behalf. If the client-specified threshold for the metric is violated, the background process sends a small message to the client application. The background process can be configured by the client to repeat this message at regular intervals (known as the rearm interval) until the metric reaches a separate disarm threshold. At that time, a different message is delivered to the client process indicating that the alert condition has been cleared. To balance system and network load and to improve response time, any number of Notification Processes may be configured to run on a single machine, and any number of NWS machines may run Notification Processes.
•
NWSGraph Applet: To aid in the visualization of NWS forecasts, an HTML form, CGI program, and Java Applet have been developed to graphically display NWS measurements and forecast values. A separate server process, written in C, is utilized to retrieve the data from NWS and send it to the applet. Due to Java applet security constraints, the server process must run on the same machine as the HTTP server. However, the NWS Forecaster and Memory Server Processes may run on separate machines.
2. SNMP Sensor 2.1 SNMP Communication Library To facilitate code sharing of the SNMP routines, a separate SNMP library was developed. This library provides the methods necessary to encode information using the Basic Encoding Rules (BER) of SNMP’S Abstract Syntax Notation One (ASN.1), and to encapsulate this information into a properly formed SNMP Protocol Data Unit (PDU). Methods currently visible to external processes include: •
encode_var_bind: This procedure takes an Object Identifier (OID) and associated data, and produces an SNMP Sequence, which may be used in the Variable Bindings section of an SNMP PDU. Building these “varbinds” is the first step in sending either of the supported PDUs: TRAPs and GETs.
•
build_snmp_trap: This procedure allows a client application to send an event datagram, known as a Trap, to an SNMP management station. The Notification process defines two such traps; one for threshold violation and another for threshold disarm.
•
build_snmp_get: A client application may utilize this routine to build an SNMP Get request. This PDU requests particular information (identified in the varbinds section) from an SNMP agent.
4
2.2 SNMP Configuration Configuration File Format The SNMP Sensor can not be programmed with advanced knowledge of the types and locations of the metrics that it will be measuring. To provide this information, and to map NWS experiment tags to appropriate SNMP variables, a configuration file is read at startup. This is a text file, with each line describing a separate measurement. Each line consists of six fields, delimited by white space characters. Comment lines are defined in the traditional way, with a ‘#’ character in the first position. # Host # Name # -------buzzux buzzux buzzux2 test2 test3
community (64 chars) --------public public public City Country
period (secs) -----23 30 60 60 5
absolute| relative -------relative absolute absolute relative absolute
OID (64 chars) ---------------------------1.3.6.1.2.1.5.1.0 1.3.6.1.4.1.2021.7.3.1 1.3.6.1.4.1.1575.1.5.2.3.0 1.3.6.1.2.1.1.3.0 1.3.6.1.2.1.1.7.0
Exp. Tag (9 chars) --------icmpInMsg Load.1Min Load.1Min sysUpTime sysSvcs
Figure 1: Example SNMP Sensor Configuration File The following fields are defined: •
Host Name: This field identifies the IP host name of the network element that is to be measured.
•
Community String: SNMP agents authenticate requests with a simple password, known as the Community String. This field specifies the community string for the host under test.2 This field is case sensitive.
•
Period: This field defines the number of seconds to wait between measurements.
•
Data Treatment: This field may take one of two values: “relative” and “absolute”. Data that are defined as “relative” are stored differentially within NWS. In other words, before each measurement is stored, the prior measurement is subtracted so that the NWS stored value represents the change from the previous poll cycle. This treatment would be appropriate when measuring an SNMP COUNTER data type. In contrast, values declared to be “absolute” are stored without interpretation. This behavior corresponds to the SNMP GAUGE data type. This field is not case sensitive.
•
Object Identifier: This field identifies the OID of the metric at the SNMP agent, in dotted decimal notation. At this time, the SNMP Sensor is not able to parse symbolic OIDs.
•
Experiment Tag: This field specifies the NWS experiment tag that is used to identify the metric. When the SNMP Sensor registers this tag, it is concatenated with the string “.exp.SNMP” to identify the experiment as an SNMP experiment. The Memory Server, in turn, prepends the tag with the IP address of the system under test before writing the information to secondary storage. So, an SNMP experiment configured with a tag of “sysSvcs” performed on a host with IP address 100.135.150.254 will be stored as “100.135.150.254.sysSvcs.exp.SNMP”, and is retrieved from the forecaster as “sysSvcs.exp.SNMP”. To conform to NWS filename length limitations, the SNMP experiment tag is limited to nine characters or less. This field is case sensitive.
2
To protect the integrity of this information, care should be taken to properly secure the SNMP Sensor Configuration file so that unauthorized users are prevented from reading the community strings.
5
Each line in the SNMP configuration file produces an internal experiment data structure. These experiments are stored on a linked list in ascending order by poll period. This facilitates SNMP experiment scheduling, as described below.
2.3 SNMP Sensor Code Run-time Parameters The behavior of the SNMP Sensor may be controlled through the use of several run-time options, including: -a: Address - Specifies the IP address of the sensor’s host machine. This parameter is required if the host is configured with more than one IP address, and the user wishes to register itself as residing at an address other than the default address [2]. -c: Configuration File Name - This required parameter specifies the name of the SNMP Sensor configuration file. -e: Error File Name - Specifies the name of a file to which the sensor will log error messages. The default is stderr. -l: Log File Name - Specifies the name of a file to which the sensor will log informational messages. The default is stdout. -m: Memory (Persistent State) Server Host Name - The sensor uses this value to identify the Persistent State Process that stores the SNMP metrics. If not specified, the sensor expects this process to be running on its local host at the default memory port (configured at compile time in config-site.h). -n: Name Server Host Name - The sensor registers itself and its experiments with the NWS Name Server Process running on this host. The default name and port number are configured at compile time in config-site.h. -P: Password - The presence of this option causes the SNMP Sensor to query stdin for a password. This password must then match the password supplied with the kill message in order to stop the sensor. The default behavior is for the sensor to accept and process the kill message without a password. -t: Trace Option - If this option is specified, it causes the SNMP sensor to write a hexadecimal and ASCII dump to the log file of each SNMP packet sent or received by the sensor. -p: Port Number - This is a required parameter that specifies the TCP port upon which the sensor listens for incoming messages, most notably the “kill” request from the control_network_sensor program.
Initialization As the sensor begins execution, it first processes the run-time parameters. It then parses the configuration file, building the linked-list of experiment entries described above. The sensor registers itself with the Name Server and establishes a network connection with the Persistent State Server. It allocates a TCP socket with which to listen for sensor control messages. It allocates a UDP socket, bound to a random port number on the local host, which will be used to send SNMP Get requests and receive SNMP Get responses. It attempts to resolve each host name in the SNMP experiment list, invalidating any entries for which this operation fails. It then registers all of the valid experiment list entries with the Name Server, using the naming convention previously described. Finally, a child process is forked to perform the actual SNMP
6
experiments. The parent process then loops, listening for and processing incoming sensor control messages. During this loop, the parent process will reregister with the Name Server at periodic intervals. This reregistration serves as a heartbeat, so that the rest of the NWS system is aware of the continued health of the SNMP sensor.
Scheduling The child sensor process cycles through the entries on the experiment list, keeping track of the time required to examine them. As it examines each entry, the accumulated cycle time is subtracted from the amount of time the entry is to wait before its next poll cycle is initiated (this value is initialized to the experiment’s poll period). If this value falls below 1, an SNMP Get PDU is formed and processed according to the configuration in the experiment’s data structure. When complete, the elapsed time of the experiment is added to the total accumulated time, and the next entry is checked. As this polling is performed, the sensor takes note of the smallest wait value encountered on the list (adjusted by subsequent SNMP poll delays). When the cycle is complete, the sensor sleeps for the amount of time (if any) required by the experiment that will be performed next. With this algorithm, each SNMP experiment may have a different poll period, yet all will be initiated as close to the correct time as possible.3
SNMP Data Retrieval To retrieve an SNMP measurement, the child process builds an SNMP Get PDU, using the OID and Community String from the experiment list entry. It uses the current system timestamp in the Request ID field, and sends the PDU to the destination host using the well-known SNMP port (as returned by getservbyname()). It then waits for an SNMP Get Response to arrive at the socket, timing out after a configurable delay period.4 If a Get Response does arrive, the Request ID field is validated against the timestamp that was used in the Get Request. If they match, the value is extracted from the varbind section of the PDU and stored with the NWS Persistent State Server. NWS metrics are all stored as double-precision floating-point values. However, SNMP does not define a floating-point data type. If the SNMP Sensor finds a numeric data type (INTEGER, COUNTER, GAUGE, etc.) in the varbind area of the Get Response PDU, it extracts the number into a long int, and then type casts it into a double before it is stored. If, on the other hand, an OCTET-STRING is encountered, the SNMP Sensor extracts the string and uses atof() to produce an ANSI C double value for storage. Should the OCTET-STRING variable actually contain non-numeric data (as would be the case for mib-2.system.sysContact, for example), every measurement for that OID will have a value of 0.0.
3. Notification Process 3.1 Notifier Code Run-time Parameters The Notification Process accepts several options from the command line. All run-time parameters are optional. They include:
3
An SNMP experiment that takes more than one second to complete (especially one that times out without a response) may delay subsequent measurements in the cycle if subsequent experiments have wait values less then the amount of time required to perform the prior measurement. 4 This delay period is a symbolic constant in the SNMP experiment code. It is currently set to 60 seconds.
7
•
-p: Registration Port Number - This parameter specifies the TCP port number upon which the Notification Process will listen for registration requests. The default is configured at run-time in the file notifier.h.
•
-l: Log File Name - Specifies the name of a file to which the Notification Process will log informational messages. The default is stdout.
•
-e: Error Log File Name - Specifies the name of a file to which the Notification Process will log error messages. The default is stderr.
•
-t: Trace Option - The presence of this flag causes the Notification Process to write a hexadecimal and ASCII dump to the log file of each notification message as it is sent.
•
-d: Debug Option - This option turns on tracing, detailed diagnostic logging, suppresses forking of the child processes in response to registration requests, and queries stdin for each NWS metric and forecast value.
Initialization At startup, the Notification process validates the run-time parameters, and then opens a TCP socket upon which to listen for registration requests. When one comes in, it forks a child process to actually handle the registration protocol, and then to monitor the requested statistics.
Registration Handling The child process accepts the registration request, which consists of several unordered command/argument pairs, each terminated with a newline character. Custom line buffering code is used to allow each line to be extracted one at a time from the socket, and to limit the amount of time that the process will wait for each registration line. The timeout is currently set to 60 seconds. If no data are received within that amount of time, the registration is abandoned, and the child process exits.
Statistics Monitoring Upon successful registration notification, the child process will allocate a socket back to the client process (or, in the case of an SNMP notification type, to the Network Management Station). In the case of a TCP notification, it will periodically attempt to establish a connection with the client process. Next, the specified Forecast Server will be contacted, and the forecast request will be registered. A disarm level is computed as the sum of the threshold value and the delta value. The delta value serves two purposes. First, its sign indicates which side of the threshold value is to be considered the “safe” side, and which is to be considered the “warning” side. For example, too little CPU availability would be unacceptable for a process that needs to perform intensive numerical computations. Such a process might wish to be notified when CPU availability drops below 50%, and might wish to discontinue notifications when CPU availability rises above 70%. Such a process would register with a threshold value of 0.5 and a delta value of 0.2. In contrast, a process concerned about the speedy delivery of information might wish to be alerted when a network packet takes longer than 700 milliseconds to be delivered. Such a process might register for notification on "lt.exp.rich" (the TCP network latency experiment) with a threshold of 0.7 and a delta value of -0.2. The process would then receive a threshold "arm" message when latency was detected at or above 700 milliseconds, and would receive a "disarm" message when latency falls below 500 milliseconds.
8
The second purpose of the “delta value” value is to simply provide a buffer between the value that causes the threshold violation message and the value that disarms the warning. If we were to consider only the threshold value, then the Notification Process might flood the network with consecutive arm and disarm notifications when the target metric happens to consecutively fall just below and then just above the threshold value. The message frequency will, of course, be limited by the sample interval, but the idea is to avoid flip-flopping between a warning and safe condition. Next, the child process enters a loop, requesting the NWS metric at the specified polling interval. If either the current measurement or the NWS forecast value violate the specified threshold, an appropriate notification message is transmitted. If the client process had specified a non-zero repeat interval, then the message is retransmitted whenever that time period expires and the previous forecast request contains either a measurement or a forecast below the disarm level. When the child process receives a forecast request in which both the measurement and the forecast value are above the disarm level, a disarm message is sent. For SNMP notifications, the messages take the form of an SNMP trap. The Trap Enterprise is configurable at compile time5 as are the variable OIDs and the specific trap IDs. The current implementation uses two different traps. Specific trap ID 1 is the threshold violation trap, while specific trap ID 2 indicates that the metric has returned to the disarm level. 1998-12-13 11:30:17 buzzux.busby.org [155.111.106.24] Enterprise Specific Trap (1) Uptime: 0:00:00 enterprises.615.999.1601 = "cpu.avail.exp.rich" enterprises.615.999.1602 = " 0.484" enterprises.615.999.1603 = " 0.484" enterprises.615.999.1604 = " 0.500" 1998-12-13 11:31:17 buzzux.busby.org [155.111.106.24] Enterprise Specific Trap (1) Uptime: 0:00:00 enterprises.615.999.1601 = "cpu.avail.exp.rich" enterprises.615.999.1602 = " 0.484" enterprises.615.999.1603 = " 0.484" enterprises.615.999.1604 = " 0.500" 1998-12-13 11:32:17 buzzux.busby.org [155.111.106.24] Enterprise Specific Trap (1) Uptime: 0:00:00 enterprises.615.999.1601 = "cpu.avail.exp.rich" enterprises.615.999.1602 = " 0.655" enterprises.615.999.1603 = " 0.655" enterprises.615.999.1604 = " 0.500" 1998-12-13 11:32:57 buzzux.busby.org [155.111.106.24] Enterprise Specific Trap (2) Uptime: 0:00:00 enterprises.615.999.1601 = "cpu.avail.exp.rich" enterprises.615.999.1602 = " 0.701" enterprises.615.999.1603 = " 0.701" enterprises.615.999.1604 = " 0.500"
enterprises.615.999:
enterprises.615.999:
enterprises.615.999:
enterprises.615.999:
Figure 2: Example SNMP Notifications6
Figure 2 shows an example trap log from the UCD snmptrapd process running on a Linux machine that has received one threshold violation trap, followed by two rearm traps and one threshold disarm notification. The Notification Process generated these traps as directed by a process that registered for the experiment “cpu.avail.exp.rich” (CPU Availability) with a threshold value of .5 and a delta value of .2. The polling interval was set to 20 seconds, with a repeat interval of 60 seconds. TCP and UDP notifications take the form of a one-line, ASCII message terminated with a newline character. The threshold violation (arm) message has the following format:
5
The Enterprise is controlled by the symbolic constant NOTIFICATION_ENTERPRISE in notifier.c. It is currently set to the imaginary enterprise of “1.3.1.6.4.1.615.999”, which would be represented symbolically as “private.enterprises.ucsd.999”. 6 The IP address in the listing has been altered in order to preserve the test system’s privacy.
9
NWS-NA: V F T \n Where “NWS-NA” stands for “Network Weather Service, Notification Arm” and , , and are replaced with the appropriate values, formatted with sprintf() using a %10.3f format. The disarm message has a similar format: NWS-ND: V F T \n Where “NWS-ND” stands for “Network Weather Service, Notification Disarm”.
3.2 Interface Code A very small interface library is provided to allow Unix-based, ANSI C programs to register for notifications without concern for the details of the TCP-based communication with the Registration Process. The library contains a single procedure, RegisterForNotification(). This procedure returns an integer value: zero indicating failure, or non-zero indicating success. It takes the following parameters: •
char *notifierHost: The host name of the machine upon which the Notification Process is running.
•
unsigned short notifier_port: The registration port for the Notification Process.
•
short notification_type: Identifies the notification method as one of the following symbolic constants (defined in notifier.h): ♦ ♦ ♦
NOTIFICATION_SNMP NOTIFICATION_UDP NOTIFICATION_TCP
•
char *snmpTrapHost: The IP host name of the machine that should receive SNMP trap notifications. This parameter is ignored if the notification_type parameter is not equal to NOTIFICATION_SNMP.
•
unsigned short port: Identifies the TCP or UDP port number to which notification messages will be sent. In the case of SNMP notifications, specifying a port number of 0 causes the Notification Process to send traps to the well-known “snmp-trap” port (as returned by getservbyname()). For UDP and TCP notifications, a non-zero port number must be specified.
•
char *foreHost: The IP host name of the machine upon which the forecast host process is running. The Notification Process will retrieve forecast information from this process.
•
char *experiment: The NWS experiment tag to monitor.
•
char *foreSource: The IP host name of the experiment source machine to monitor.
•
char *foreDest: The IP host name of the experiment destination machine to monitor. Some experiments (such as CPU and SNMP experiments) have no destination machine. In this case, a NULL pointer may be supplied.
10
•
double threshold: The threshold value.
•
double fudge: The “delta value” value (described above).
•
unsigned int poll_interval: The number of seconds to pause between forecast retrievals.
•
unsigned int repeat_interval: The interval (in seconds) between the transmission of rearm messages while the threshold is in an armed state.7
•
int useMSE: A non-zero value here will cause the Notification Process to examine the MSE forecast. Zero causes the Notification Process to use the MAE forecast instead.
Client to Notifier Protocol In order to allow registration by processes on non-Unix platforms and/or written in languages other than ANSI C, a simple, ASCII-based registration protocol has been developed. The protocol uses TCP to provide flow control and guarantee ordered delivery of information. In this protocol, a notification is described using simple keyword/argument pairs. In general, the protocol is not case sensitive (the exception being the value for the optional community string). A newline character terminates each keyword/argument pair. The registration transaction begins with the keyword “Register”, and ends with the keyword “End”. Neither of these keywords takes an argument. Between these framing keywords, the remaining keywords may be supplied in any order. All numbers are transmitted in standard ASCII format, in any format appropriate for the Unix atoi() or atof() system calls. The protocol is described below. In this description, optional elements are surrounded by square brackets ( [ and ] ), variable items are surrounded by angle brackets ( < and > ), and curly brackets ( { and } ) surround choices separated by the pipe character ( ‘|’ ). All other elements are verbatim requirements. The protocol keywords are: • • • • • • • • • • • • • • •
Register Notification { SNMP | UDP | TCP } [ SNMPHost ] [ Community ] Port ForecastHost Experiment SourceHost [ DestHost ] Threshold FudgeFactor PollInterval [ RepeatInterval ] [ UseMSE ] END
7
In this implementation, the Notification Process actually divides the repeat_interval value by the poll_interval value to compute the number of cycles between rearm messages. As such, the repeat_interval is not strictly followed to the second unless it happens to be an exact multiple of the poll_interval.
11
The following rules apply concerning this protocol: • • • • • •
The default for community string is “public”. The SNMPHost keyword and value must be supplied when the Notification type is “SNMP”. Otherwise, the notification messages are always delivered to the registering process’s IP address. In the case of SNMP notifications, specifying a port value of 0 will cause the traps to be delivered to the well-known “snmp-trap” port (as returned by getservbyname()). For TCP or UDP notifications, a non-zero port number must be specified. The default threshold is 0. The default repeat interval is 0, meaning that the notifications will not be repeated at all. The MAE forecast is checked unless the keyword “UseMSE” is supplied.
4. NWSGraph Applet 4.1 Graph Request Form
Figure 3: NWS Graph Form as displayed by Netscape 4.5 under Windows 95. To provide an interface to the many run-time parameters of the nwsGraphServer and the NWSGraph applet, an HTML form has been developed. Figure 3 illustrates the form. The user is able to request one or two graphs to be displayed in the same browser window. Upon submission, the form invokes the nwsGrapher.pl CGI program, described below.
4.2 Graph Server CGI Launch Program (nwsGrapher.pl) The CGI program is written in Perl, and will accept either POST or GET methods from the HTML form. Execution begins by validating the parameters selected by the user. For each graph, an nwsGraphServer process is launched. The Perl program captures the output of this process (which will be a TCP port
12
number that will be required by the applet). Various other parameters are passed from the HTML form to the server, including the polling frequency and the forecast method. Once the servers are running, the CGI program builds an HTML page containing the appropriate titles and applet tags to produce the requested graphs. The port number from the nwsGraphServer process is passed to the applet in an applet parameter tag.
Site dependent modifications Two items in the Perl code may need to be modified before installing nwsGrapher.pl: • Location of Perl binary The first line of the file contains the shell directive, “#!/usr/local/bin/perl –w”. If this is not the correct path to the Perl binary, then this line must be changed. • Location of nwsGraphServer binary The second non-blank line of the program defines a variable to hold the path to the nwsGraphServer binary: $nwsGraphServer = "/opt/apache/share/cgi-bin/nwsGraphServer"; If this is not where the nwsGraphServer program was installed, then this line must also be changed.
4.3 Graph Server Run-time parameters •
-H: Forecaster Host Name: The IP host name of the machine upon which the forecast host process is running. The graph server will retrieve forecast information from this process. The default location is configured at compile time in the file config-site.h.
•
-P: Forecast Period: The number of seconds to pause between forecast retrievals. The default is 60 seconds.
•
-S: Source Host: The IP host name of the experiment source machine to monitor. This is a required parameter.
•
-D: Destination Host: The IP host name of the experiment destination machine to monitor. This is an optional parameter, since some experiments (such as CPU and SNMP experiments) have no destination machine.
•
-T: Experiment Tag: The NWS experiment tag to monitor. This parameter is required.
•
-M: Forecast Method: One of either “MSE” or “MAE”. The default is “MAE”.
13
•
-L: Log File Name: Specifies the name of a file to which the graph server will log informational messages. The default is stdout.8
•
-E: Error Log File Name: Specifies the name of a file to which the graph server will log error messages. The default is stderr.
Initialization The graph server starts by validating its run-time parameters. It then opens a TCP socket, binding it to any random TCP port number, as selected by the kernel. It uses getsockname() to determine the value of this port number, and prints the value on stdout. At this point, the program forks a child process and exits. It is the function of the child process to accept the TCP connection and proceed to supply numbers to the client program.
Child Process Upon startup, the child process immediately closes stdout and stderr, relying instead upon the log and error files (which, by default, are duplicates of stdout and stderr). It then waits up to five minutes for a connection to be established on the random TCP port socket. If no connection presents itself within that period of time, the child process exits. When a connection is established, the child process registers its forecast request with the specified Forecast Server. It then enters a loop, requesting a forecast, sending the two numbers (the measurement data, and the forecast value that was specified with the –M option) to the client program, then sleeping for the number of seconds specified with the –P option. The numbers are formatted in ASCII by sprintf() with a %10.3f format. If the send() call returns any error, the graph server assumes that the client program has disconnected, and the server exits.
Figure 4: Example output of NWSGraph Applet (under Netscape 4.5, Windows 95)
8
This parameter is not strictly required. However, if nwsGraphServer is being launched by the nwsGrapher.pl CGI program, then the –L option should be supplied to ensure proper function. If the graph server is allowed to write diagnostics to stdout, the Perl program will never see the close on stdout and therefore fail to process the port number from the parent process. The program will hang, failing to produce the HTML code necessary to display the applet.
14
4.4 NWSGraph Applet The y-axis is established initially based upon the yMin and yMax parameters. It extends itself appropriately whenever any of the plot points exceed the stored maximum and minimum values. The yaxis is labeled in ten divisions. The x-axis is never labeled. The maximum number of plots that can be displayed is computed at initialization based upon the width of the applet and the value of the plotWidth parameter. The x-axis autoscales until the number of plot points reaches the computed maximum. Then the x-axis becomes a sliding window, dropping older time-series values on the left as newer ones arrive on the right. The data and forecast values are plotted with lines. The data values are plotted first, followed by the forecast values. This implies that, if the user sees only a forecast plot, then the forecast and data values for that sample were equal. The forecast plot is drawn one unit to the right of the data value on the x-axis. This provides visual indication of the predictive nature of the data. On the far right side of the applet window are drawn two stacked-bar graphs, designed to resemble the output level meters on a modern stereo system. These “Hi-Fi LED” indicators represent the values of the latest measurement and forecast. Also present on these indicators are maximum and minimum value indicators for each metric. The colors of each of these components are selectable through run-time parameters.
Parameters To provide maximum control over the applet’s behavior, a wide variety of parameters are accepted at runtime: •
period: This parameter is parsed into a long. It specifies the number of milliseconds the applet should wait between plots. In other words, it controls the amount of time the applet waits before attempting to retrieve a new data point from the server. This value should equal the reporting period of the server process. The default for period is 1000 milliseconds.
•
plotWidth: This parameter is parsed into an int. It controls the number of pixels that the applet will retain between each plot point. This controls the density of the graph. When the applet can no longer add another plot point to the graph without reducing the number of pixels between points below plotWidth, then the X-axis changes from an autoscale to a sliding window, dropping older points to make room for newer ones.
•
yMin: Represented internally as a double, this parameter specifies the initial bottom range of the y-axis. It defaults to 0.0.
•
yMax: Represented internally as a double, this parameter specifies the initial top range of the yaxis. It defaults to 1.0.
•
dataColor: This parameter is parsed into a Java AWT Color name. It specifies the color of the graph plot for the measurement data. This parameter defaults to red.
•
foreColor: This parameter is parsed into a Java AWT Color name. It specifies the color of the graph plot for the forecast data. Blue is the default.
15
•
backColor: This Java AWT Color name specifies the desired color for the graph background. The default is “lightGray”.
•
gridColor: This Java AWT Color name specifies the color that the applet will use when drawing the graph’s numbers and tic marks. Black is the default.
•
maxColor: This Java AWT Color name specifies the color that the applet will use for the Maximum value indicator on the “Hi-Fi LED” graph component. By default, yellow is used.
•
minColor: This Java AWT Color name specifies the color that the applet will use for the Minimum value indicator on the “Hi-Fi LED” graph component. If not specified, the default of cyan is used.
•
port: Internally represented as an int, this parameter specifies the TCP port number upon which the applet will contact the nwsGraphServer process. The default is 1, but the applet will not function correctly unless the proper value is specified for this parameter.
5. Future Work The development of this enhancement to the Network Weather Service has presented several opportunities for future research. Now that we are able to harvest SNMP metrics and use NWS to reason about them, some careful evaluation should be performed to determine which standard MIB variables are of value in predicting resource utilization. It may be instructive to contrast and compare the results of current NWS measurement methods with the values that are retrieved using SNMP. This project was developed under NWS version 1.1.0.0. As the project was nearing completion, a new version of NWS, version 1.1.1.0, was released with many improvements to the data storage and retrieval APIs, as well as to the Name Server API. The components of this project should be updated to use these new interfaces. The incorporation of features from SNMP V2 and V3 should be evaluated, as well. The Notification Process currently does not participate in the standard NWS control protocol. It does not respond to a “Kill” request, nor does it provide a dynamic method for controlling log detail level, as the network sensor does. These features will be added in a future version. Similarly, the SNMP Sensor may be extended to dynamically receive configuration records over the network instead of relying solely upon the disk-based configuration file that it currently reads only at startup. The Notification Process currently has no way to recognize the death of a client unless it happens to be using the TCP notification method. Even then, the absence of the client will not be recognized until a message transmission is attempted. There is a need for a deregistration facility within the Notification process to allow a client to indicate its lack of continued interest in receiving NWS notifications. To supplement this capability, the server should perform periodic liveness testing on its clients so that it may recognize when clients fail in the absence of a deregistration request. To further enhance the ability to visualize NWS measurements, an SNMP agent will be developed to provide a method for SNMP enabled applications to retrieve NWS data. This will allow for easy integration of NWS into the fabric of existing SNMP-based network management applications without the need for custom programming. Finally, all SNMP enabled components should be provided with the capability to read and interpret ASN.1MIB files. This will allow for symbolic representation of SNMP OIDs, which are much easier for humans to deal with than are the long, dotted decimal versions.
16
References [1] I. Busse, B. Deffner, and H. Schulzrinne. Dynamic QoS control of multimedia applications based on RTP. Computer Communications, January, 1996. [2] J. Hayes. NWS User’s Guide. December, 1998. [3] M. Miller. Managing Internetworks with SNMP. M & T Books, 1993. [4] R. Wolski, N. Spring, and J. Hayes. The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing. December, 1998.
17
