Monitoring of Distributed Systems - CiteSeerX

University of London Imperial College of Science, Technology and Medicine Department of Computing

Monitoring of Distributed Systems

Masoud Mansouri-Samani

December 1995

Thesis submitted for the degree of Doctor of Philosophy of the University of London, and for the Diploma of the Imperial College of Science, Technology and Medicine.

Abstract

Monitoring is essential for obtaining the required information about the operation of distributed systems in order to make management decisions and control their behaviour. This thesis presents a generic model of monitoring based on the life-cycle of monitoring information which consists of four stages — generation, processing, dissemination and presentation. A generalised monitoring service for distributed systems can be constructed as a configuration of generic components which can perform the functionalities identified in the model. Based on the model a survey of the area is presented and some representative existing approaches are described in detail. The main contribution of this thesis is the support for a flexible and scalable distributed event monitoring service. In particular, this thesis presents features of a new declarative, interpreted and Generalised Event Monitoring language (GEM), used to program event monitors which can perform common processing activities such as filtering, composition and notification on event reports generated in a distributed system. The novelty of this work can be summarised as follows: It allows "on-the-fly" detection of composite events in the presence of variable communication delays and unordered delivery of messages. The notion of real-time has been tightly integrated into the language with built-in facilities to deal with delays in a flexible and user-defined manner. Many temporal constraints which would otherwise have been very difficult to express in a distributed environment can be specified. The interpreted nature of the language allows dynamic changes to the observed composite events. It is shown that a scalable, dynamic and distributed event processing service can be constructed as a configuration of multiple event monitors which can receive and interpret appropriate GEM scripts. This thesis describes the implementation of the event monitor and presents examples implemented in the prototype version of GEM.

i

Acknowledgements

I would like to use this opportunity to thank all the people who have helped me, in different ways, during my PhD. I owe a special debt of gratitude to my supervisor Dr. Morris Sloman, for his invaluable help and advice. The completion of this thesis would not have been possible without his thoughtful guidance, constructive suggestions and prompt responses. Thanks are due to all my friends and colleagues in the Distributed Software Engineering section for their friendship and support. In particular, I would like to thank Andrea, Anne, Bashar, Celso, Christos, Damian, Dimitra, Douglas, Emil, Fox, Keng, Mark, Naranker, Nikos, Pomme, Tracy and Ulf for providing such an interesting and stimulating environment to work in. I am deeply grateful to my close friend of many years, Kaveh, for many fruitful discussions and his unfailing support at different stages of my work. Thanks are also due to my friends Keith, Mohammad, Narmin and Nasrin for their moral support. Almost last but certainly not least, I am indebted to my beloved family in Iran, for their patience and unceasing encouragements during all these difficult years of separation. It is to them that I dedicate this thesis. Above all, I thank God for giving me the strength and guidance to accomplish this task. Finally, I gratefully acknowledge EPSRC and Hewlett-Packard Laboratories in Bristol for a CASE studentship award, as well as financial support from Esprit SysMan (7026) Projec.

ii

Table of Contents

1

Chapter 1 Introduction

1.1 Monitoring and Management ..........................................................................................1 1.2 Requirements for a Monitoring Service............................................................................3 1.2.1 Filtering & Correlation Capabilities .......................................................................3 1.2.2 Scalability .............................................................................................................5 1.2.3 Dynamic Changes .................................................................................................5 1.2.4 Intrusiveness..........................................................................................................5 1.2.5 Failures .................................................................................................................6 1.3 Contributions of Our Work..............................................................................................6 1.4 Thesis Outline..................................................................................................................7

Chapter 2 Generic Monitoring Model

8

2.1 Introduction ....................................................................................................................8 2.2 Basic Concepts and Terminology...................................................................................10 2.2.1 Event-driven vs. Time-driven monitoring ............................................................10 2.2.2 Classification of Events........................................................................................11 i) Internal vs. External Events ............................................................................11 ii) Primitive vs. Composite Events ......................................................................12 2.3 Generation of Monitoring Information..........................................................................12 2.3.1 Status Reporting..................................................................................................13 2.3.2 Event Detection and Reporting............................................................................14 i) Location of Event Detection..........................................................................14 ii) Time of Event Detection................................................................................14 iii) Event Report Format ....................................................................................14 2.3.3 Trace Generation.................................................................................................15 2.4 Processing of Monitoring Information...........................................................................17 2.4.1 Merging and Multiple Trace Generation..............................................................18 i) Merging of Monitoring Traces.......................................................................19 ii) Generating Multiple Traces ...........................................................................20 2.4.2 Validation of Monitoring Information.................................................................20 2.4.3 Model Updating..................................................................................................21 2.4.4 Combination of Monitoring Information.............................................................21

iii

2.5 2.6

2.7

2.8

2.4.5 Filtering of Monitoring Information....................................................................25 2.4.6 Analysis of Monitoring Information....................................................................26 Dissemination of Monitoring Information .....................................................................26 Presentation of Monitoring Information ........................................................................28 2.6.1 Display Approaches ............................................................................................28 i) Textual Data Presentation...............................................................................28 ii) Time Process Diagrams .................................................................................29 iii) Animation....................................................................................................31 2.6.2 Desirable User Interface Features.........................................................................33 i) Visualisation at Different Abstraction Levels...................................................33 ii) Placement of Monitoring Information...........................................................35 iii) Controlling the Time of Display...................................................................36 iv) Use of Multiple Views...................................................................................36 v) Visibility of Interactions ................................................................................36 Implementation Issues ...................................................................................................37 2.7.1 Intrusiveness of Monitoring Systems ...................................................................37 i) Hardware Monitors.........................................................................................38 ii) Software Monitors.........................................................................................38 iii) Hybrid Monitors ..........................................................................................40 2.7.2 Global State, Time and Ordering of Events..........................................................41 i) Physical Clock Synchronisation.....................................................................42 ii) Logical Clocks ..............................................................................................42 iii) Vector Clocks...............................................................................................43 iv) Global Snapshots..........................................................................................44 Conclusion ....................................................................................................................45

4 6

Chapter 3 Related Work

3.1 ZM4/SIMPLE................................................................................................................46 3.1.1 Generation of Events...........................................................................................46 i) ZM4 Architecture...........................................................................................46 ii) Format of Event Traces in SIMPLE...............................................................47 3.1.2 Processing of Events............................................................................................49 3.1.3 Presentation of Events .........................................................................................49 3.1.4 Discussion...........................................................................................................50 3.2 Meta..............................................................................................................................50 3.2.1 Generation of Monitoring information................................................................51 3.2.2 Processing Monitoring Information.....................................................................51 i) Updating a Data Model ..................................................................................51 ii) Combination of Monitoring Information.......................................................51 3.2.3 Implementation...................................................................................................52 3.2.4 Discussion...........................................................................................................52 3.3 EVEREST .....................................................................................................................53 3.3.1 Primitive Event Definition & Placement...............................................................54 3.3.2 Composite Event Definition & Notification .........................................................55

iv

3.3.3 Evaluation Assignment........................................................................................56 3.3.4 Moving Evaluation Responsibilities .....................................................................57 3.3.5 Discussion...........................................................................................................57 3.4 OSI Management Standards...........................................................................................58 3.4.1 OSI Management Approach................................................................................58 3.4.2 Generation of Monitoring Information................................................................59 3.4.3 Event Reporting Service ......................................................................................60 3.4.4 Log Service.........................................................................................................61 3.4.5 Processing of Management Information..............................................................62 3.4.6 Discussion...........................................................................................................62 3.5 CORBA .........................................................................................................................63 3.6 Active Databases............................................................................................................63 3.7 Conclusion & Other Related Work.................................................................................64

6 6

Chapter 4 GEM

4.1 Introduction ..................................................................................................................66 4.2 Event specification.........................................................................................................67 4.2.1 Primitive Event Specification...............................................................................67 i) Event Attributes..............................................................................................70 4.2.2 Composite Event Specification ............................................................................71 4.2.3 Guards ................................................................................................................72 i) Occurrence intervals.......................................................................................74 ii) Effective and Correct Use of Guards..............................................................75 4.3 GEM Scripts..................................................................................................................75 4.3.1 Event Declarations...............................................................................................75 4.3.2 Rule Definitions ..................................................................................................76 i) Detection Window ..........................................................................................77 ii) Event Expression...........................................................................................78 iii) Action Sequence ..........................................................................................78 4.3.3 Top level Commands...........................................................................................82 i) triggering events.............................................................................................82 ii) enable & disable commands..........................................................................83 iii) stop command.............................................................................................83 iv) ~ (remove) command ...................................................................................83 v) load command...............................................................................................83 vi) ? (query) command......................................................................................83 4.3.4 Selective Consumption of Events.........................................................................84 4.3.5 GEM Error Messages ..........................................................................................85 4.4 Examples of GEM Scripts..............................................................................................85 4.5 Conclusion ....................................................................................................................90

9 2

Chapter 5 Dealing with Delays

5.1 Introduction ..................................................................................................................92

v

5.2 5.3 5.4 5.5 5.6 5.7

Problem of Delays.........................................................................................................92 Uniform Delaying Technique........................................................................................94 Dealing with Delays in GEM..........................................................................................95 Calculating D and I........................................................................................................98 An Example ..................................................................................................................99 Conclusion ..................................................................................................................106

Chapter 6 Distributed Monitoring Service

107

6.1 Introduction ................................................................................................................107 6.2 Regis/Darwin environment ...........................................................................................108 6.2.1 The monit Component ......................................................................................108 6.2.2 Composite Components.....................................................................................109 6.2.3 Communication.................................................................................................111 6.2.4 Component Placement.......................................................................................111 6.3 Downloading GEM Scripts to Monitors .......................................................................112 6.4 Encoding and Decoding Event Reports........................................................................112 6.5 Event Dissemination ....................................................................................................113 6.6 Case Study — Distributed Filtering & Event Composition............................................115 6.7 Configuration of an Event Monitoring Service.............................................................124 6.8 Conclusion ..................................................................................................................125

Chapter 7 Monitor Implementation

126

7.1 Introduction ................................................................................................................126 7.2 Overall Structure of an Event Monitor .........................................................................126 7.2.1 GEM Interpreter................................................................................................127 7.2.2 Event Class Manager .........................................................................................127 7.2.3 Rule Manager....................................................................................................127 7.2.4 Timer................................................................................................................128 7.3 Event Detection — Mechanism & Algorithm...............................................................128 7.3.1 Event Evaluation Tree .......................................................................................129 7.3.2 Event Evaluation Nodes.....................................................................................129 7.3.3 Guard Evaluation Tree ......................................................................................130 7.3.4 Event Records ...................................................................................................131 7.3.5 Hierarchical Event History.................................................................................132 7.3.6 An Example of Composite Event Detection.......................................................133 7.3.7 Storage Requirement.........................................................................................135 7.3.8 Complexity of the Event Detection....................................................................136 7.4 Prototype Implementation ...........................................................................................136 7.5 Conclusion & Related Work.........................................................................................136

138

Chapter 8 Conclusion

8.1 Future Work.........................................................................................................140

vi

Appendices

142

Appendix A — GEM's Event Detection Algorithm ............................................................142 Appendix B — GEM Grammar .........................................................................................152 Appendix C — GEM Error Messages ................................................................................158

References

160

vii



A typical distributed system consists of a number of processes which run on different machines and cooperate to perform a common task. Processes coordinate their activities by sending messages over a communication network. Examples of such systems include process control, telecommunication and banking applications. Distributed systems offer a number of advantages compared to centralised systems. • reduced cost • higher reliability and availability • ease of modularity • improved performance • better response time • configuration flexibility • incremental growth As the use of distributed systems increases, the ability to monitor their behaviour becomes more important in all stages of their life-cycle, from the development stage to the day-today management. Monitoring these systems involves dynamic collection, processing and presentation of information concerning objects or software processes under scrutiny.

1.1 Monitoring and Management Monitoring is required for various purposes such as debugging, testing and program visualisation. It is also used for general management activities which have a more permanent and continuous nature. They include: i) performance and quality of service management ii) configuration management iii) fault management iv) security management, and v) accounting management

1

Chapter 1 Introduction A managed distributed system consists of a number of managed objects. A managed object (henceforth referred to just as object) is defined as any hardware or software component whose behaviour can be controlled by a management system. Each object’s interface encapsulates its state and behaviour and may hide internal details vital for performing management activities. Any internal information or operations needed externally must be made visible via this interface, which can be divided into two parts (Holden & Langsford 1989; Sloman 1987), as shown in figure 1.1: •

An operational interface which fulfils the main purpose of the service provided by the object (i.e., the normal information processing operations), and

•

A management interface which supports monitoring and control interactions with the management system.

Control Commands Status requests Monitoring Information

Management interface Managed Object

Information Processing Operations

Operational interface

Figure 1.1 A Managed Object The management interface supports three types of operations: • Receiving requests for status information which may be received periodically (i.e., polling) or on a random basis, • Generation of monitoring information regarding object behaviour and status in the form of event and status reports. Event reports are generated as a result of state changes or the execution of a particular operation. Status reports may be generated periodically or as a response to external requests, and • Receiving control commands specifying the actions performed on the object (e.g., stop) Monitoring reports are used by human or automated managers to make management decisions, resulting in control actions that may modify the behaviour of the managed systems, as shown in figure 1.21.

1

Note that a monitoring system itself would have to be managed and therefore the generic model of management shown below can be recursively applied to the components of the model itself.

2


Decision Making

Control

Monitoring

Control actions

Monitoring Information Managed System

Figure 1.2 Management Model Unlike monitoring which is generally a passive process, control actively changes the behaviour of the managed system. In this thesis we shall only focus on the monitoring aspect of management.

1.2 Requirements for a Monitoring Service There are a number of requirements which are common to all the management areas (i-v), identified earlier. Often the same monitoring data may be of interest to more than one area. For example, an event representing a machine failure may be required by both the fault manager and the configuration manager. This stipulates the need for a generalised monitoring service to satisfy the following overall common requirements:

1.2.1 Filtering & Correlation Capabilities In large distributed systems the quantity and frequency of the generated monitoring reports is such that it can quickly grow to consume the available processing, storage and communication resources. Additionally, from the view point of any one manager, a significant portion of this information may be either irrelevant or simply at the wrong level of abstraction. Furthermore the rate of generation of these reports may be too fast to be observed or analysed by the managers. A generalised monitoring service must provide support for filtering and correlation 2 which allow discarding and raising the level of abstraction of monitoring information,

2

In this thesis the terms correlation, combination and composition are used interchangeably.

3

Chapter 1 Introduction respectively. Filtering and correlation are complementary techniques which are essential for managing the high volume of monitoring information in large distributed systems 3. The task of correlating monitoring information is made very difficult due to certain inherent characteristics of distributed environments. The physical distribution of objects means that monitoring reports are generated by components on remote machines. These reports may be subject to variable and sometimes unbounded communication delays and can therefore be out of date and/or reach the observer in reverse temporal order. The notion of a global time is not readily available in distributed systems as each machine has its own local clock which may drift with respect to other clocks. The lack of a global clock coupled with communication delays make it difficult to correlate monitoring data and arrive at a consistent global view of system state. These problems are exacerbated as the size of the system under scrutiny grows (e.g., hundreds or thousands of objects) and affects the set of temporal constraints that can be expressed and detected. For example, a configuration manager must start a new server when an existing one fails and the automatic recovery system fails to bring it back up again within 10 minutes. Unordered delivery of events may result in the unnecessary creation of a new service. Centralised correlation techniques are unsuitable to deal with these problems. Correlation of monitoring information in the presence of delays is by far the most challenging task performed by the monitoring service. Managers should be able to specify filtering and correlation criteria, including various temporal constraints which are essential for many management activities in distributed systems. Clock synchronisation mechanisms are necessary so that a global time frame can be constructed in order to support the specification of temporal information which include: • Relative, periodic and absolute times, • Occurrence interval of an event, • Order of event occurrences, and • Time intervals between event occurrences. Managers should be able to include in their specifications: • any explicit knowledge about the maximum possible delays for bounded communication delays, or • a timeout facility representing the tolerated delays that the correlation process must be able to cope with for unbounded delays.

3

Most (if not all) network management systems rely on network surveillance and control operations which are based on event correlation.

4

Chapter 1 Introduction Different techniques and mechanisms can be used to support consistent detection of system-wide events or states, as described in section 2.7.2.

1.2.2 Scalability The monitoring service must be able to monitor large systems with hundreds or possibly thousands of objects (e.g., telecommunication systems). A number of systems use a centralised approach to filter and correlate monitoring reports generated from these distributed sources. The problem with this approach is that all reports have to be sent to the centralised monitoring server which can cause considerable network traffic in large systems. It would be best to perform the monitoring activities as close to the sources of this information as possible. This stipulates a distributed monitoring service, whose components are themselves physically spread throughout the system and cooperate in order to recognise events and states across a number of nodes. This reduces the amount of communication, storage and processing overheads and makes it easier to access the required information. Dissemination components are necessary to send the required reports to those who need them.

1.2.3 Dynamic Changes As systems evolve, it frequently becomes necessary to change them by adding new or removing existing machines and objects (e.g., as a result of failures or upgrades to the system). This, in turn, could necessitate a corresponding change or extension to the management functionality of the system. Therefore, the ability to change the configuration and the functionality of the distributed monitoring service dynamically, without having to bring the operation of the entire system to a halt, is highly desirable. This flexibility can be provided through support for dynamic reconfiguration and provision of components which can interpret monitoring specifications.

1.2.4 Intrusiveness Monitoring may affect the behaviour of the system under observation. This is called intrusiveness or the probe effect and means that the monitored and unmonitored systems may behave differently. This is particularly true when the application references external state or its behaviour is affected by race conditions. In general this intrusion or interference is undesirable and must be reduced or, when possible, completely eliminated. This can be achieved through suitable instrumentation techniques as described in section 2.7.1.

5


1.2.5 Failures The monitoring service should be able to tolerate partial failures which are a fact of life in a distributed environment. Both monitored and monitoring components may be subject to failures. This may result in erroneous or incomplete views of system behaviour and state. *

*

*

In this thesis we shall only address the first three requirements outlined above.

1.3 Contributions of Our Work In this thesis we shall present a generic model of monitoring (Mansouri-Samani & Sloman 1993; Mansouri-Samani & Sloman 1994) which extends the existing models to encompass all facets of the monitoring life cycle. It is used to characterise the monitoring process and identify the functionalities performed by a generalised monitoring service. It consists of four main stages; generation, processing, dissemination and presentation of monitoring information and has been used as a basis for a comprehensive survey of the area. It has also been used as an implementation model in the Esprit SysMan project (7026) (Debski & Janas 1994) which specifies a Generalised Monitoring Service (GMS) for distributed systems as a configuration of generic components which implement the functionalities identified in the model. We present an event detection algorithm and the associated mechanisms necessary to allow "on-the-fly" detection of "concurrent" composite events in the presence of delays and unordered delivery of messages. In an effort to facilitate event-driven monitoring and in particular on-the-fly filtering and composition of events in a distributed environment, we use the algorithm as part of a new declarative, interpreted and Generalised Event Monitoring language (GEM) (MansouriSamani & Sloman 1995). The notion of real-time has been tightly integrated into the language with built-in facilities to deal with delays in a flexible and user-defined manner. Many temporal constraints which would otherwise have been very difficult to express in a distributed environment can be specified. The language can be used to program generic event monitors, which receive event reports from different objects and perform the common event processing activities identified in the Model (e.g., filtering, composition, notification). This coupled with the interpreted nature of the language and the dynamic reconfiguration capabilities provided by the underlying Regis/Darwin environment allows dynamic changes to the monitoring specifications and the structure of the service without stopping the execution of the monitored system. GEM 6

Chapter 1 Introduction is used to support the specification of management obligation policies (Marriott et al. 1995). The initial version of GEM was used as a basis for the specification of a generalised monitoring language in the Esprit SysMan project (7026) (Debski & Janas 1994). It was also used in Hewlett-Packard Laboratories (Bristol), as the basis for an event correlation language for Telecommunication Management Network. We demonstrate that a scalable, dynamic and distributed event processing service can be constructed as a configuration of multiple event monitors (capable of interpreting appropriate GEM scripts), disseminators and other generic components.

1.4 Thesis Outline The remainder of the thesis is organised as follows: In the next chapter we will present our proposed generic monitoring model and will give examples of how the existing systems provide the functionalities identified in the model. Based on this model chapter 3 examines and compares some related work in detail. Chapter 4 presents the features of GEM together with examples of its use. Chapter 5 discusses the effects of delays on composite event detection and shows how the built-in features of the language can be used to deal with delays in a flexible and user-defined manner. Chapter 6 shows how a scalable and dynamically reconfigurable distributed event processing service can be constructed as a configuration of multiple event monitors which receive and interpret appropriate GEM scripts. Chapter 7 describes the implementation details of event monitors, the proposed algorithm and the associated detection mechanism. Finally the conclusion and a description of future work is presented in chapter 8. The appendices contain details of the event detection algorithm, the GEM grammar and error messages.

7



2.1 Introduction In this chapter we shall present a generic functional model of monitoring (Mansouri-Samani & Sloman 1993) which is derived from the Event Management Model of (Feldkuhn & Erickson 1989), with some changes and enhancements. We use the model as a basis for a survey of the existing monitoring approaches and will give examples of how these systems provide the functionalities identified in the model. It also provides a framework for deriving the facilities required for the design and construction of a generalised monitoring service for distributed systems. This model identifies the following four monitoring activities performed in a looselycoupled, object-based distributed system: • Generation: important events are detected and event and status reports are generated. These reports are used to construct traces or histories, which represent historical views of system activity. • Processing: a generalised monitoring service provides common processing functionalities such as merging of traces, validation, model updating, correlation and filtering of monitoring information. They convert the raw and low-level monitoring data to the required format and level of detail. • Dissemination: monitoring reports are distributed to users, managers or processing agents who require them. • Presentation: gathered and processed information is displayed to the users in an appropriate form. Implementation issues relating to the instrumentation and how clock synchronisation is achieved for event ordering, provide a fifth dimension for comparing monitoring systems. Figure 2.1 summarises the elements of the monitoring reference model presented in this chapter.

8


Generation of Monitoring Information •

Status reporting

•

Event detection and reporting

•

Trace generation

Processing of Monitoring Information •

Merging and multiple trace generation

•

Validation

•

Model updating

•

Combination

•

Filtering

•

Analysis

Dissemination of Monitoring Information •

Registration of subscribers to dissemination service

•

Specification of information selection criteria

Presentation of Monitoring Information •

Textual displays

•

Time process diagrams

•

Animation of events and status

•

User control of levels of abstraction

•

User control of information placement and time frame for updates

•

Multiple simultaneous views

•

Visibility of interaction message contents

Implementation Issues •

Special purpose hardware

•

Software probes

•

Time synchronisation for event ordering Figure 2.1 Elements of a Monitoring Reference Model

9


Many models have been proposed in order to describe the monitoring process. The Event Management Model of (Feldkuhn & Erickson 1989) covers the status reporting, event detection and reporting, trace generation (logging), filtering and dissemination elements of our model. Presentation and event processing functionalities such as combination, validation and merging of traces are not included in the model. The Event/Action Paradigm of (Marinescu et al. 1990) models the monitoring activities in a layered fashion. At first sight, the above four activities appear to be a layered model with generation as the lowest layer and presentation using the services of the lower layers. However a generalised monitoring system may need to perform these activities in various places and in different orders to meet specific monitoring requirements. For example, generated information may be directly displayed by an object without processing or dissemination. Monitoring reports which are distributed to particular mangers, could be reprocessed to generate new reports. Presentation of information may occur at many intermediate stages. For these reasons we present the monitoring model as a set of activities which can be combined as required in a generic monitoring service. Components of the monitoring service perform any combination of these activities in different order and at different stages.

2.2 Basic Concepts and Terminology In order to present the rest of the material in this chapter, we need to describe a number of concepts and terminology.

2.2.1 Event-driven vs. Time-driven monitoring Each object has associated with it a status, and a set of events (representing status changes and activities). The behaviour of an object can be defined and observed in terms of its status and events. The status of an object is a measure of its behaviour at a discrete point in time and is represented by a set of status variables contained within a status vector (Feldkuhn & Erickson 1989). These variables or attributes may be static (e.g., machine type) or time-varying (e.g., load). Static attributes may further be subdivided into those associated with permanent objects or temporary objects. An event is defined as an atomic entity that reflects a change in the status of an object. The status of an object has a duration in time, e.g., "process is idle" or "process running", whereas an event occurs instantaneously, e.g., "message sent" or "process started". Usually, the status of an object is changing continually and therefore the behaviour of the object is normally observed in terms of a distinguished subset of events, called events of interest. These reflect changes

10

Chapter 2 Generic Monitoring Model that are of significance to the management and therefore are generated when a pre-defined set of conditions are satisfied. Time-driven monitoring is based on acquiring periodic status information to provide an instantaneous view of the behaviour of an object or a group of objects. There is a direct relationship between the sampling rate and the amount of information generated. Eventdriven monitoring is based on obtaining information about occurrence of events of interest, which provide a dynamic view of system activity. The amount of generated and communicated monitoring data is reduced as only the information pertaining to activity of interest is collected and transmitted. For this reason it is the most common approach adopted in monitoring systems. These systems provide automatic recognition of events which means that the user need not explicitly collect and analyse the lower level details of system behaviour.

2.2.2 Classification of Events i) Internal vs. External Events In a distributed system three kinds of events can be identified (Bemmerl et al. 1990): a) A control flow event represents a control activity and is associated with a control thread. Such an event occurs when a process or the operating system reaches a previously defined statement. For example: • process P2 enters/leaves procedure Fred for the n th time. • the operating system enters the scheduler. b) A data flow event occurs when a status variable is changed or accessed. For example, • variable a of process P1 is assigned value X, • variable b in the third invocation of procedure Fred is read by process P2. Although such an event is caused by a specific control flow activity, it is not associated with any particular control thread. c) Process-level events show the creation and deletion of processes and the interactions and data flow between them. For example, • process P1 started, • process P1 sends message m to process P2, • the number of waiting processes in queue w is incremented by one. Control flow and data flow events can be referred to as internal events which are related to the local state of a component or object and are not visible outside it, unless explicitly made

11

Chapter 2 Generic Monitoring Model visible at the management or debugging interface. Such events are particularly useful for debugging purposes. Note that debug tools often create a new "interface" to make visible internal events and state which are not normally visible at either the operational or management interface to an object. Internal events thus violate the encapsulation of objects and are more appropriate for analysing the behaviour of a single component. Process-level events can be considered as external events, which represent the external behaviour of an object and its interactions with other objects. These events are of particular interest to management. A generalised monitoring system should allow us to observe a combination of these events. Many monitoring tools enable the users to specify and detect only process-level events such as interprocess communication, as these events do not violate the encapsulation of objects and are at the correct level of abstraction for analysing the behaviour of distributed systems (Bates 1995; Joyce et al. 1987; LeBlanc & Robbins 1985) .

ii) Primitive vs. Composite Events According to their level of abstraction events can also be classified, into primitive and composite (or combined or correlated) events. i) A primitive event signifies a simple change in the state of an object. ii) A composite event is defined as a combination or grouping of other primitive and composite events. This classification is useful for describing the global behaviour of a group of objects in terms of the local behaviour of every object in the group. Various languages are used for specifying combined events and states. This is described in more detail in section 2.4.4, on combination of monitoring information.

2.3 Generation of Monitoring Information Monitoring data is generated in the form of status and event reports (figure 2.2). A sequence of such reports is used to generate a monitoring trace. Status reporting, event detection and reporting, and trace generation are described below.

12


Monitoring Trace Trace Generation Monitoring Reports

Status Report Status Reporting Criteria

Event Report Event Detection & Reporting

Status Reporting

Event Detection & Reporting Criteria

Status Vector

Figure 2.2 Generation of Monitoring Reports and Traces

2.3.1 Status Reporting A status report contains a subset of values from the status vector and may include other related information e.g., time stamp and object identity. It represents the status at a specific instance in time and can be generated: i) Periodically: Reports are generated based on a pre-determined schedule. ii) On demand: A report is generated upon receiving a request (solicited reporting). Note that the request may itself be periodic (i.e., polling) or on a random basis. One example of periodic status reporting can be found in Conic environment (Magee 1989) where each node sends a configuration status report to a name server every 5 seconds. The OSI event management permits scheduling of event reports on both daily and weekly basis (ISO 1990c). The Clouds operating system (Dasgupta 1986) uses on demand status reporting scheme. Requests are called probes and every object or process has a predefined or user defined probe procedure which is executed whenever a probe is received. The probe handler sends a message back to the originator of the request, reporting the status condition of the process. Status reporting criteria will define which reporting scheme to use, what the sampling period is and the contents of each report.

13


2.3.2 Event Detection and Reporting Significant changes in the status of an object or a group of objects (i.e., events of interest) must be detectable. An event is said to have occurred when certain pre-defined conditions are satisfied (defined by event detection criteria). To detect events associated with an object certain software or hardware probes or sensors would have to be inserted and installed in the object. Insertion of such mechanisms is part of a process called instrumentation. Where, how and when event detection is carried out depends on the resources available for detection (e.g., dedicated hardware, communication channels) and the intrusiveness of the monitoring system. This intrusiveness or probe effect is the degree to which the observed system is perturbed by the act of monitoring 4, as discussed further in section 2.7.1.

i) Location of Event Detection Event detection may be internal to the object and is typically performed as a function of the object itself. For example, a function that updates the status vector may check the event detection criteria when it performs the update. Event detection may also be performed externally, for example by an external agent which receives status reports and detects changes in the state of the object.

ii) Time of Event Detection Detection of event occurrences may be immediate (real-time) or delayed (detected some time after the occurrence). For example, signals on the internal bus of a node may be monitored, using a hardware monitor, to detect any changes in the status of the node as and when they occur. This is particularly used for detection of events in time frames of milliseconds. Alternatively, status reports may be generated, stored and used to detect events at some later time.

iii) Event Report Format Once the occurrence of an event is detected, an event report is generated. It contains attributes such as the event identifier, type, priority, time of occurrence, the state of the object immediately before and after the occurrence of the event and other application specific status variables.

4

Here we only consider detection intrusion , associated with the recognition of events, and not action intrusion, associated with performing control actions on the system.

14

Chapter 2 Generic Monitoring Model Event and status reports may be generated in one or more stages. A preliminary report may be generated by an object, containing a minimum amount of monitoring information (e.g., object and event identifier). Such a report could then be sent to a different object which can generate a more complete report by adding further attributes such as time-stamps, event type or text messages. Obviously, the amount of information contained in a monitoring report depends on the requirements of users or clients who need them. The attributes assigned to each event or status may be of two kind: Independent or Dependent (Mohr 1990). Independent attributes are those primitive attributes which are assigned to all events and status, such as the time stamp, identity of the object, etc. Dependent attributes are assigned depending on the type of the event or status. For example, a configuration event report, representing a "create process" event may contain the identity of the created process. Event and status reporting criteria is used to determine what information to include within each report. The format and structure of a report may be fixed or variable. Obviously with a variable report structure some amount of filtering can be performed implicitly, where only the necessary information is included within a report, as explained in section 2.4.5.

2.3.3 Trace Generation In order to describe the dynamic behaviour of an object or a group of objects over a period of time, event and status reports are recorded in time order as monitoring traces. A complete trace contains all the monitoring reports generated by the system since the beginning of the monitoring session. A segmented trace is a sequence of reports collected during a certain period of time. It describes "a completely observed time interval" of the behaviour of the monitored object or system. A trace may be segmented due to overflow of a trace buffer, or deliberate halting of trace generation which results in the loss or absence of reports over a period of time. A trace may have a header giving some general information such as the start and end time stamps, the identity of the monitored object, its size and the identity of the program etc. Monitoring traces may be generated for various reasons: •

Archiving purposes and post-mortem analysis: Monitoring reports may be needed at some later stage for further processing, analysis and usage. They are stored as monitoring traces and examined by the user at a later time, possibly after the completion of the program. This is particularly important for debugging purposes (McDowell & Helmbold 1989). Archive traces are usually referred to as histories or logs. A logging service can be used to generate these traces in log files. Certain facilities or services can

15

Chapter 2 Generic Monitoring Model be used to browse or query these traces. Such traces can also be used to control a replay or re-execution of the program, allowing the reproduction of the erroneous computations. With a complete trace an individual process can be debugged in isolation, where the history provides the needed communication and simulates the environment of the process. •

Availability of resources: Other reasons for forming these traces may be lack of processing power to analyse and interpret the monitoring reports "on the fly"; or limited communication resources to send reports to an external processing agent, as and when they are generated. This is particularly true for real-time monitoring.

•

Speed of visualisation: Trace generation might be necessary when the rate at which monitoring information is received and displayed is too quick for the observer to follow. This is especially true for real-time observation and display of system activity. Special display tools can be used to control the speed at which information is presented to the user.

•

Transformation of the logical view of the system activity: It enables the construction of a global monitoring trace from local traces, which describe the behaviour of the system as a whole (see merging of monitoring traces — section 2.4.1 (i)). Multiple traces can be generated from a single trace, to reflect specialised or restricted views of the system activity (see generating multiple traces — section 2.4.1 (ii)). Lower level histories can be used to generate more meaningful higher level monitoring information, by using combination and filtering (see sections 2.4.4 and 2.4.5 respectively).

The design alternatives relating to trace generation include: •

Location of trace generation: Traces could be formed by objects which generated the monitoring reports, by intermediate monitoring objects when processing monitored information, or by the final user of such information.

•

Temporary vs. long-term traces: Temporary storage involves placing the monitoring reports in a temporary buffer before they are processed or transferred to another object for use. Long-term storage involves recording the reports in persistent log or history files by making use of a logging service.

•

Storage capacity: Obviously there is a limit to the amount of available storage space and therefore the size of a monitoring trace. Overwriting older records when the maximum storage size is reached can be appropriate if we are interested in the most recent behaviour of the system. Alternatively, new reports are discarded until more space is available by halting trace generation. Both these strategies may result in a segmented trace as some of the reports may be discarded.

•

Sophistication of trace generation: A simple scheme would store all the reports, in the arrival order, in a local trace-buffer of a monitoring object, without changing the 16

Chapter 2 Generic Monitoring Model contents of the stored information. This is usually used for temporary storage of monitoring data. A sophisticated scheme might use a logging service (which can be modelled as a managed object (LaBarre 1991)) for the long-term storage of reports together with some extra information, in a variety of formats, representations and orderings. It may allow the generation of multiple traces or logs, representing different logical views of system activity (see section 2.4.1 (ii)). Special log records could be generated, containing a log record identifier, logging time, information contained in the report to be logged and other related data. Various implicit or explicit filtering activities can be performed when such records are generated and stored in log files. A sophisticated trace generation mechanism may allow reports to be stored in occurrence order if an occurrence time stamp is available. This overcomes the problems of ordering according to arrival order which may lead to incorrect interleavings of monitoring reports and invalid observations, due to communication delays. Determining the order of monitoring reports is discussed in section 2.7.2. •

Access to the trace reports may be on demand by issuing an explicit request to the storage entity, or according to pre-determined conditions. On demand access enables the processing agent to receive the reports when it has the necessary resources to deal with them, or when the communication traffic is low. By using a scanning and selection service the user may be able to specify the particular reports required (e.g., generated by a particular object), the trace file from which they may be read or the number of reports needed. Pre-determined conditions may be used to transfer reports from a trace-buffer to a remote processing agent when the buffer is full, contains n elements, the communication load is below a certain threshold or the processing load is low (Van Riek & Tourancheau 1991). These strategies are particularly useful for reducing the communication overheads by sending reports in blocks and therefore economising on channel set-up time.

2.4 Processing of Monitoring Information In previous sections we discussed the steps necessary for generation of monitoring information. In this section we will consider some common processing activities that can be performed on this information. A monitoring service could provide certain functional units (as building blocks) which can be combined in different ways to suit the monitoring requirements. Figure 2.3, shows one possible combination. Note that these processing functionalities are often integrated and are performed in different places and at various stages.

17


Monitoring Trace

Multiple Trace Generation

Dissemination Monitoring Reports Filtering Criteria

Filtering

Model Updating

Combination

Monitoring Definitions

Validation Report

Validation

Validation Rules Global Monitoring Trace Merging

Monitoring Trace

Figure 2.3 Processing of monitoring reports

2.4.1 Merging and Multiple Trace Generation In order to provide different logical views of system activity over a period of time, monitoring traces may be constructed and ordered in various ways. The attributes of a report which can be used as selection criteria in determining how monitoring traces are processed include: • Generation or arrival timestamp, priority or type of the report • Identity, priority or type of the reporting entity • Identity or type of the managed object to which the report refers • Identity or type of the destination of the report

18

Chapter 2 Generic Monitoring Model Construction of monitoring traces is performed according to a trace specification, based on these factors; it specifies how the final traces are formed and what they will contain. Obviously, not including a report in a trace is equivalent to filtering out that report from the point of view of the users who access that trace. Monitoring traces may be generated from event or status reports as they arrive or from one or more already existing traces, as described in the following sections. P1

P2

Merging

P3

P5

Multiple Trace generation

Trace Specification

P4 Destination

P6

P7

Priority

Low

Medium

Type

Configuration

(a)

Fault

P8 High Security

(b)

Figure 2.4 Merging and Multiple Trace Generation

i) Merging of Monitoring Traces A trace segment containing all the reports related only to one object is called a local trace segment. A set of all the local trace segments from all the objects under scrutiny, over the same period of time, can form a global trace segment. One of the important activities in a monitoring system, as shown in figure 2.4 (a), is merging of several monitoring traces into one trace, using a scanning and selection service

19

Chapter 2 Generic Monitoring Model to generate a global trace segment, representing the global behaviour of the system in a particular interval. Merging could be an iterative operation. A trace generated in this way could itself be merged with others to generate a more general trace. The original traces may be discarded once merging is complete. Generating one global monitoring trace from several local traces is, in effect, imposing a linear (total) ordering on the event and status occurrences within the system. An ordered trace is simpler to understand and can be easier to work with but a linear stream may be misleading since it implies an ordering between every pair of event or status reports, even when they are completely unrelated. A partial ordering can more accurately reflect the behaviour of a distributed system (McDowell & Helmbold 1989). A general technique for obtaining the required partial ordering is described in (Fidge 1988). To achieve this, a vector of logical time stamps is associated with each event. By comparing the vectors, the ordering of events can be determined. This is described in more detail in section 2.7.2.

ii) Generating Multiple Traces As shown in figure 2.4 (b), a monitoring trace can be used to generate several other traces, representing various logical views of object or system activity. Selection of reports from a trace segment may be based on a combination of destination, priority, report type or other factors. For example, a trace could be generated with all high priority security events, so that they can be processed first and another trace with all the accounting events destined for a particular process. A trace generated in this way may be used to generate other traces. Obviously some duplication may be necessary because some reports may have to appear in several traces. For example, an event report may be of interest to two processing agents and therefore it is stored in each of their trace-buffers.

2.4.2 Validation of Monitoring Information Another important monitoring activity is performing validation and plausibility tests on monitoring information, to make sure that the system has been monitored correctly. This may be performed at different levels. When an individual report is received, its contents may be checked to see if they are valid. For example, whether the event id. represents one of the expected events, or the value of the time-stamp is valid. Invalid reports are discarded. Monitoring reports may also be validated in relation to one another. For example, to see whether two event reports in a trace satisfy an expected temporal ordering, or to check the validity of an event report against the current system status, before applying the status change to the model (see section 2.4.3 on model updating). The detection of invalid orderings may be followed by reordering or filtering of the reports. Validation is done according to certain validation rules, and a validation report may be generated. The

20

Chapter 2 Generic Monitoring Model SIMPLE monitoring system (Hofmann et al. 1992) performs some validation and plausibility tests on selected event reports and event traces.

2.4.3 Model Updating Valid monitoring information, may be used to maintain and update a representation or model of the current status of the system. In OSI terminology it is called the Management Information Base (MIB). This representation could be used by other users, managers or processing agents. For example, the configuration manager may periodically examine this data model to detect component failures. A data-oriented approach to monitoring communication networks in which a conceptual data-base model of the network is constructed and continuously updated to represent the current status of the network is presented in (Wolfson et al. 1991). Some approaches use historical and temporal databases in order to maintain both the current and also the historical behaviour of the system (Shim & Ramamoorthy 1990; Snodgrass 1988) . There are two general approaches to collecting MIB data. A dynamic approach in which only the information which is requested by the user is collected (e.g., (Snodgrass 1988)). User queries result in the automatic activation of relevant sensors in monitored objects and the collection of the required data. The advantage of this approach is that only the requested data is collected. This is particularly important when monitoring resources (processors, memory, etc.) are limited and / or there are many sensors present. The disadvantage is that the queries must be specified before the data is collected. The user, however, may not know in advance exactly what information is required. At the other extreme, a static approach is used in which the collection of data is independent of its use. All possible monitoring data must be collected and stored for potential access by users. This is done by permanently enabling all the sensors, or forcing each sensor to be enabled manually. This solves the problem associated with the previous approach but results in collection of large amounts of information that may not be used. In a distributed system, due to various factors such as communication delays and component failures, it is usually impossible to construct a model which could provide a truly up-to-date and consistent snapshot of the system status.

2.4.4 Combination of Monitoring Information Combination (also referred to as correlation or composition) of monitoring information is the process of increasing the level of abstraction of the data. In conjunction with filtering, it prevents the users of such information from being overwhelmed by the considerable volume of details present in all of the system's activity, so that they can observe the

21

Chapter 2 Generic Monitoring Model behaviour of the system at a desired level of detail. To do this, low-level primitive events and states can be processed and interpreted to give a higher-level view of the complex states and events which occur in the system. Combination is performed according to certain event and state definitions. These definitions specify new events and states based on primitive or other combined events and states. In other words, events and states at one abstraction level can be used to generate those at higher abstraction levels. The ability to combine monitoring information is particularly important in distributed systems that implement fault tolerance. Events generated by several sensors or probes with different failure modes can be combined to provide the necessary reliability and resilience against failures. For example, the expression: t-exceeded

=

t1-exceeded

OR

t2-exceeded

OR

t3-exceeded

with ti-exceeded representing the "temperature exceeded" event detected by sensor i, may be used to make sure that an event representing an increase in temperature above a predefined limit is detected even if two out of three independent sensors fail. Various languages have been developed which enable the user to define new events and states and combine monitoring information. In the State and Event Specification Language (SESL) (Holden 1989), a user can define two declarative statements: e WHEN elist

event e occurs when elist occurs

s EQUALS expr

state s will have the value of expr

An event list (elist) defines a pattern of occurrence of events, which is specified in terms of other event lists, events, low-level events (represented by an event notification message), and state changes. These are related by temporal operators => and ->, and the logical operators | and ! (see table 2.1). Each time a sequence of events occurs which matches the pattern, the event list also occurs.

22


elist1 => elist2

matches when elist2 occurs immediately after elist1

elist1 > elist2

matches every time elist2 occurs after elist1 has once occurred

elist1 -> elist2

matches when elist2 occurs sometime after elist1

elist1 | elist2

matches when either of the event lists occurs

!elist

matches only if elist did not occur

elist PROVIDED condition matches if condition is true at the time elist occurs Table 2.1 Example SESL operators for creating event lists Also various conditions and expressions can be defined using the operators listed in table 2.2. A condition is also an expression with a zero value representing true and a non-zero value representing false. $state

value of state defined by EQUALS

#event

number of times event occurred

State("string")

value of state string of monitoring activity

constant

a numerical constant

!expr -expr

unary operators

*/+-

binary arithmetic operators

< > = == !=

binary relational operators

&|

binary logical operators

Table 2.2

SESL operators for creating expressions and conditions

As an example, to check the availability of a service consisting of two servers, the user can write the following SESL script: busy

WHEN EVENT ("received request")

free

WHEN EVENT ("sending reply")

non_busy WHEN one_busy => free one_busy WHEN non_busy => busy | two_busy => free two_busy WHEN one_busy => busy

The first two SESL statements can be used to show when the server is busy and when it is free. Internal events free and busy are defined in terms of external events "received request" and "sending reply", respectively. The next three statements define non_busy, one_busy, 23

Chapter 2 Generic Monitoring Model and two_busy events. For example, non_busy is the internal event associated with the state transition where no server is being used, and is triggered by the event of 1 server becoming busy followed (without any other events) by the detection of a server becoming free. More details about SESL can be found in (Holden 1991). A specification language used for debugging and performance monitoring is described in (Lumpp et al. 1990). The user or programmer includes, with the application, a monitoring section that defines primitive and combined events. This is similar to the declaration of data structures and procedures for the application program. Examples of some event definitions are shown below: e1 ::= (xmitregister == 1) on node 0; e2 ::= (rcvregister == 2) on node 0; e3 ::= e1 && (waitregister == 1) on node 0; e4 ::= e2 && (waitregister == 1) on node 0; e5 ::= e3 && e4; e6 ::= (ptr > 0xE4000) on node 2 e7 ::= (touch(flag)) on node 1; e8 ::= (reach(label_1) on any node; e9 ::= (done_flag == TRUE) on all nodes;

Events e1-e6 are self-explanatory. They are defined in terms of counters such as rcvregister, xmitregister and waitregister, a pointer ptr and other events on specific nodes. In the definition of e7, touch() operator is used which specifies events that correspond to any change of a specified variable regardless of the value stored (e.g., flag). In the definition of e8, the operator reach() is used for tracing the flow of control of a thread. The user can place labels in appropriate sections of the application and define events that correspond to the program counter reaching those points (in this case label_1) during execution. The definition of an event can span more than one node in the target system (e.g., any node, all nodes, 5 nodes). The user can specify various temporal, relational and logical relationships between events. In another approach, (Wolfson et al. 1991) use a data manipulation language based on SQL, with some enhancements to specify primitive and combined events. Also the Event Definition Language (EDL) (Bates 1995) allows the user to define primitive and higher level events with various filtering constraints. Section 3.3 contains another example of such notations.

24


2.4.5 Filtering of Monitoring Information A typical distributed system may generate large amounts of monitoring information. This results in heavy usage of resources such as CPU and communication bandwidth for generation, collection, processing, and presentation of monitoring information. In addition, the users of the monitoring information may be overwhelmed with vast amounts of data which they are unable to comprehend. Filtering is the process of minimising the amount of monitoring data, so that users only receive the desired data and at a suitable level of detail relevant to their purposes. It is also needed for security, where certain users should not have access to particular monitoring information. Filtering functionality must be considered separately from the process of combination of monitoring information. Filtering generally involves discarding information, but combination permits both high level and low level views on the information i.e. the information may not be discarded. Filtering may be performed, explicitly or implicitly, in different places and at various stages: • Global filtering: Performed by discarding the monitoring reports or traces which do not satisfy global filtering criteria. This includes validation failures. • Reducing report contents: With a variable report structure and the use of a selection facility, a monitoring object could receive a report and generate a new one with only a subset of monitoring information contained in the old report. The old report could be discarded and the new one may be used or stored by the object itself or forwarded to another object. Obviously the best policy is to avoid generating unwanted or unnecessary information. For example this could be done by: • Controlling report contents: by using event and status reporting criteria at generation stage so that only the required information is included in each report. • Conditional generation: A monitoring report or trace is generated when certain predefined conditions are satisfied. For example, in periodic status report generation, by increasing the sampling period, the frequency and number of generated reports can be reduced. Also, report generation mechanisms can be activated or deactivated. This could be done at various levels of granularity. For example, the reporting mechanisms for all or individual events associated with a component may be deactivated. • Dissemination Filtering: Disseminating monitoring reports based on a subscriber / provider principle performs an implicit filtering function as selected reports are forwarded only to those subscribers who have requested them, as explained in section 2.5. A monitoring trace may be generated for each destination object. A report may not be discarded but simply placed in one trace-buffer and not the others. Obviously, not 25

Chapter 2 Generic Monitoring Model including a report in a trace would be equivalent to filtering it out from the point of view of the users who have access only to that trace. In general, it is better to perform the necessary filtering at an early stage to reduce the resource usage in subsequent stages. In all, the above cases, implicit or explicit filtering criteria may be based on the information contained within the reports (e.g., event type, time, priority, type) or external information such as previous status or events and the capacity to process each report. Some approaches provide a language in which various filtering criteria can be defined (e.g., FDL in the SIMPLE environment — section 3.1). Users of monitoring reports should be able to define their own filtering criteria.

2.4.6 Analysis of Monitoring Information Monitoring information can be analysed, for example, to determine average or mean variance values of particular status variables. Trend analysis is important for forecasting faults in components. Diagnosis of faults requires correlation of event reports. Some aspects of analysis are considered part of the presentation of information e.g. displaying information as histograms or graphs. In general, analysis is application specific so is not really considered part of a generalised monitoring service. It can range from very simple gathering of statistics to very sophisticated model based analysis. Some monitoring tools collect various statistics such as total CPU usage, ready, blocked, idle and busy times, number of messages or bytes sent, etc. (e.g., TMP (Wybranietz 1990)). In the SIMPLE environment (Mohr 1990), a commercial data analysis and graphics package, S, from AT&T, has been integrated for interactive and complex analysis of monitoring data. A more complex approach has been adopted in the Event Based Behavioural Abstraction approach (EBBA), which is a paradigm for high-level debugging of distributed systems (Bates 1995). The EBBA toolset allows the user to construct models of system behaviour in a top-down manner. These models reflect user understanding of the expected system behaviour, and are compared to the actual system activity represented by the monitoring information.

2.5 Dissemination of Monitoring Information Monitoring reports generated by the objects would have to be forwarded to different users of such information. The destination of such reports may be human users, managers, other monitoring objects or processing entities. Dissemination schemes range from very simple and fixed to very complex and specialised. An example of a fixed scheme is to broadcast all the reports to all the users. A complex and specialised dissemination scheme could be based on the subscription principle (Feldkuhn & Erickson 1989) as shown in figure 2.5.

26


Report Processors or Clients

Subscription Requests

Dissemination Unit

Report Selection Criteria Subscription Service Subscription List

Monitoring Report

Subscription Criteria

Figure 2.5 Dissemination of monitoring reports Clients of a monitoring service subscribe to receive the required status or event reports from the dissemination unit by registering themselves with the subscription service. Each client sends a subscription request indicating its identity, the list of reports and the frequency of the reports required. Subscription authorisation information, held by the service, is used to determine whether the client is an authorised user and what reports it is permitted to receive. Only authorised users are permitted to subscribe and are entered in the Subscription List maintained by the dissemination unit. Selection Criteria contained within the subscription request are used by the dissemination system to determine which reports and their contents should be sent to the clients. This provides implicit filtering as only the requested reports are forwarded. In CORBA, event channels, which are standard CORBA objects, are used to disseminate event data, based on event types, between multiple consumers (clients) and suppliers (servers) (OMG 1995). This is described in more detail in section 3.5.

27


2.6 Presentation of Monitoring Information Generated, collected, and processed monitoring information has to be presented to clients in a format which meets their specific application requirements. A suitable user interface should enable the user to specify how to display information as well as cope with: • large amounts of monitoring data generated by the system, • various levels of abstraction of such information, • the inherent parallelism in the system activity, represented by monitoring data, • the rate at which this information is produced and presented.

2.6.1 Display Approaches In this section, we shall describe different techniques that can be used to display information to the user, and we will conclude by considering useful features of some existing tools. Various presentation techniques have been used for displaying debugging data in parallel debugging systems (McDowell & Helmbold 1989). Similar techniques can be used in a generalised monitoring system to display different types of monitoring information relating to configuration, performance, security, accounting, etc. These approaches are described below:

i) Textual Data Presentation It is a common type of display with a simple text presentation of the monitoring information, which may involve highlighting or colour. Events may be displayed in their causal rather than temporal order as in Traveller (Manning 1987). The Jade monitoring system (Joyce et al. 1987) observes the process level events such as interprocess communication, creation and killing of a process. It provides a Text Console which displays one or two lines of textual output to describe the event (figure 2.6).

1. 2. 3. 4.

process1 enters the system process1 waits to receive a message from any process process2 enters the system process2 sends a message to process1 "connect" 5. process1 receives a message from process2 "connect" :

Figure 2.6 Example Jade Textual Display

28

Chapter 2 Generic Monitoring Model It shows the name of the process initiating the event, the event type, the name of the process that is the subject of the event, if any, on the first line. If the event is one in which processes communicate, the contents of the message are printed as the second line of the output. Appropriate indentation, highlighting and colouring can be used to increase the expressive power of visualisation and also to distinguish monitoring information at various levels of abstraction. Advantages of this technique are that no special devices are necessary to present monitoring data and that it is simple to convert such information to textual form. However, this technique is not enough to present parallel system activities. Nowadays, simple textual presentation is often used in combination with other more expressive techniques as outlined below.

ii) Time Process Diagrams The state of the parallel system is represented as a two dimensional diagram, with one axis representing the objects and the other representing time. It shows the current status of the system and the sequence of events that led to that status, and therefore can display patterns of behaviour over time. The unit of time may be the occurrence of an event or a period of real time. One or two characters can be used to represent each of the possible events associated with an object or group of objects. The advantage of using a time-process diagram is that monitoring information can be presented on a simple text screen. In Jade (Joyce et al. 1987), an event line console has been provided which displays the current state and history of each process in a compact form and, at the same time, defines the relative ordering of events, as shown in figure 2.7. The names of the processes and single-letter abbreviations are listed on the right-hand side. In the middle there is one row for each process representing the event line for that process. Each event line is divided into a number of event intervals separating adjacent events, and each interval displays an event; events are inserted at the right of the display and scroll to the left. The last event to scroll off the left-hand side is always displayed (to the left of the vertical bar) so that, if a process has had all of its events scrolled off of the middle section, the current status of a process is always available. Here, A dotted line "..." signifies that the process is blocked and a dashed line "---" means that it is executing.

29


Event Line Console Commands: Go Pause Step a 5 do create SigPro(...)

Control Program

Data Model Meta System Application Structure of controlled system Hardware and Operating System

Sensors and actuators

Figure 3.4 Meta Application Structure

Figure 3.5 Meta Function Layers

It enables the management functions to monitor and control the behaviour of the underlying application. Here, we shall concentrate on its monitoring aspect only. Figures 3.4 and 3.5

50

Chapter 3 Related Work show the model of a distributed application and the functional layering in the Meta system, respectively.

3.2.1 Generation of Monitoring information The programmer uses a set of built-in or user-defined software sensors and actuators to instrument the application and its environment, in order to expose its state for purposes of control. The value of a sensor can be polled at predefined intervals, accessed on demand or a watch can be set up that alerts the client when the sensor value satisfies some predicate.

3.2.2 Processing Monitoring Information i) Updating a Data Model The Meta system maintains an abstract data model of the structure of the application and the environment in which it is run, as a set of entities, connections and groupings of entities, described using the object-oriented data modelling facilities of a language called Lomita. The model is updated by new sensor readings and can be viewed as an object-oriented temporal database in which the managed system provides the data values.

ii) Combination of Monitoring Information To provide higher level views of the behaviour of the program in the data model Meta allows the combination of multiple sensors, or values of one sensor over a period of time in the form of derived sensors. Lomita provides simple arithmetic operations, and also functions for min, max, size and median etc. that operate over sets of values e.g., sensor load_ratio: real := SigPro.load / Machine.load sensor high_load: integer := max(history(load, 600))

where load_ratio is the derived sensor calculated from primitive sensors SigPro.load and Machine.load and high_load is a derived sensor representing the maximum load over the last 10 minutes (600 seconds). Using the data model, the programmer at the Policy Layer defines in a control program a set of Lomita policy rules of the form: when condition do action, which describe the intended behaviour of the system. The condition may be a complex predicate, expressed on the underlying data model, involving references to multiple (basic and derived) sensors, other functions and the usual arithmetic, relational and set operators. A condition is converted to a finite state automaton whose acceptance state results in the execution of the action. The action component is a sequence of actuator invocations and data model

51

Chapter 3 Related Work operations. The condition may also contain interval temporal logic expressions with operators shown in table 3.1:

During I always P true if and only if predicate P is true throughout time interval I. During I occurs P

true if and only if predicate P is true at some point within time interval I.

P until Q

Expresses the time interval beginning when predicate P next becomes true, until predicate Q subsequently becomes true.

P for T

Expresses the time interval beginning when predicate P next becomes true, until T seconds have passed. Table 3.1 Temporal operators in Lomita

3.2.3 Implementation A Lomita control program is a high-level language which supports a rich syntax of expressions over the objects in the data model. Lomita is compiled and is broken into a set of lower-level guarded commands in a language called NPL which provides simple postfix expressions over primitive sensors and actuators. NPL is tied to a specific process and must reference nonlocal sensors explicitly. It is not very convenient to read or write but is designed to impose little overhead in parsing and execution. An instrumented program is implemented with a Meta stub which contains an NPL interpreter and a cache of the values of sensors that are referenced in a locally-executed NPL command. Pieces of NPL code are distributed among the stubs5 and the guards / predicates are evaluated against the sensor cache. Meta is built on top of another toolkit, the ISIS distributed programming toolkit (Birman et al. 1988). In order to guarantee that a predicate referencing remote sensors is satisfied and detected in a consistent global state, Meta uses the atomic broadcast communication protocol provided by the ISIS which guarantees a global order on all messages.

3.2.4 Discussion The Meta system adopts a time-driven monitoring approach in which guarded commands specify predicates over the state of the components, exposed by the sensors. The Meta

5

Distribution is done in such a way as to minimise references to nonlocal sensors and actuators to improve response time.

52

Chapter 3 Related Work toolkits’s instrumentation routines are too cumbersome and costly to support fine granularity monitoring, such as rapidly changing values in an operating system kernel. Also, new sensor values can only be accessed by a stub when the application explicitly transfers control to it, by calling a predefined Meta library routine. This leads to some doubts as to the use of the polled sensors, as the required values may not be available at the end of the polling period. Guards may include interval temporal logic assertions but, due to the potentially unbounded latencies in the underlying UNIX and ISIS platforms, these assertions can only be viewed as approximate upper bounds. Certain temporal events and timing constraints and intervals can not be expressed in the command language (e.g., at(...), every(...), 10:30 to 5:30). Meta guarantees that predicates are satisfied and detected in a consistent global state — i.e., a state that could have occurred in the absence of synchronised clocks and is physicallyrealisable, though not necessarily actual system state. Meta’s reliance on the atomic broadcast communication protocol provided by the ISIS, means that non-ISIS communication (e.g., through files or the TCP protocol) may result in Meta constructing an inconsistent global state and hence invalid detections of predicates. Our approach in GEM differs from Meta in that we use an event-driven monitoring in which timestamps are included in event reports to perform event ordering and detection, rather than relying on the underlying communication system to guarantee total ordering. Various temporal events and constraints can be specified and the problem of delayed arrivals can be dealt with in a flexible and user-defined manner at the specification level through built-in features of the language. Also, in contrast to Meta, which uses a finite-state automaton to evaluate guards, we have adopted a tree-based approach (as described in chapter 7) that allows us to easily deal with delayed arrivals.

3.3 EVEREST The EVEnt REcognition teSTbed (EVEREST) (Spezialetti & Bernberg 1995), as its name suggests, is a testbed for studying various approaches to event recognition. Monitoring is performed in two separate phases. (i) The definition and pre-processing phase: Users define the behaviour to be recognised, the configuration of the monitoring system and the strategies to be used in recognising event occurrences for a particular monitoring session. These definitions are contained in a script file which is used for application instrumentation, runtime configuration of monitoring modules and event recognition.

53

Chapter 3 Related Work (ii) The monitoring and recognition phase: At this stage the following set of components are used for actively recognising event occurrences: • Central Communication Monitor (CCM): the central monitor which based on the user specifications in the script file and the runtime behaviour of the application, creates other monitors, manages the dynamic configuration of the monitoring system and assigns event evaluation responsibilities to them. • Component Monitor: an instrumented application program module. Instrumentation is done by using the user-supplied script file in the first phase. The component monitor captures primitive event occurrences, timestamps them and sends the relevant information to appropriate evaluation monitors for processing. • Evaluation Monitor: an evaluation monitor which is independent of the original application program and monitors events coming in from one or more component monitors. It is only created if: ° its existence has been specified in the script file, ° the associated component monitors have been created, ° the activity which it is responsible for analysing has occurred. • Output Module: a graphical user interface used for displaying event information with an option for logging event occurrences for future analysis. All monitors may be assigned event recognition responsibilities which may or may not involve other, possibly remote, monitors. The following sections describe the capabilities of EVEREST and the relevant monitoring commands.

3.3.1 Primitive Event Definition & Placement A primitive event definition takes the following form: primitive is [] []

type specifies the type of the event, with the given symbolic name, which may represent actions such as testing the value assigned to a variable, operations related to files, pipes, sockets and signalling, function entry and exit, and process creation and termination. variable specifies the variable name to be monitored, and functname allows monitoring to be restricted to a particular function. For example, primitive e1 is funcenter calculate // function calculate entered primitive e2 is funcenter ANY

// any function entered

primitive e3 is intval x main

// any changes of x in main

54

Chapter 3 Related Work Events are timestamped using the processor's clock (real time) or vector clock, as specified in the script file. Primitive events also carry the identification of the component monitor which generated them (consisting of its executable name, machine name and instance number). Some events may have an event specific value associated with them (e.g., e3) and are called value primitive events. The location of primitive events can be defined by a place command with the following format. place at

Where, name is the symbolic name of the event, procexe is the executable name of the process to be monitored, machine specifies the machine at which the process will be located and instance specifies the instance number of the process. Various wild cards can be used (EVERY, ALL, ANY, ...). For example, // monitor e1 in first instance of myprog on machine saturn place e1 at myprog saturn 1 // monitor e2 in every instance of every process on machine saturn place e2 at EVERY saturn EVERY // monitor e3 in every instance of myprog process on any machine place e3 at myprog ANY EVERY

3.3.2 Composite Event Definition & Notification Composite or high level events are defined as follows: highlevel []

The named event occurs when the application of the operator to the value of its operands yields TRUE. Two types of operators are defined. Event operators, shown in table 3.2, and value operators, which test event specific attribute values associated with an event.

55


e1 then e2

e1 occurs followed by e2

e1 shfl e2

e1 and e2 occur in any order

e1 or e2 e1 conc e2 e1 count exp e1 not e2 e1 notbw e2 e3

e1 or e2 occurs e1 and e2 occur concurrently e1 occurs exp number of times e1 occurs but not e2 e2 occurs followed by e3, no occurrence of e1 in between Table 3.2 Event Operators

In order to focus on a specific activity and be notified of the occurrence of a particular primitive or composite event, the user must enter the following command. goal

For example, highlevel e4 is e1 shfl e2 primitive e3 is intval x main highlevel e5 is e3 lt 8 // e5 occurs on e3 when new value of x < 8 goal e5

// just notify the occurrence of e5

3.3.3 Evaluation Assignment Event evaluation responsibilities can be assigned to monitors via the eval command: eval at

monitor specifies the monitor to which the evaluation of the named event is assigned. It may be a specific component monitor, the CCM, or the keywords em or cm to generically indicate an evaluation or component monitor. monmac is the machine name on which the monitor is located and instance is the monitor's instance number. Various wild cards can be used (e.g., ANY and EVERY). For example, // evaluate e1 at first instance of component monitor // myprog on jupiter eval e1 at myprog jupiter 1 // evaluate e2 at every component monitor in the system eval e2 at cm EVERY EVERY // evaluate e3 at evaluation monitor instance 1 on jupiter eval e3 at em jupiter 1

56


3.3.4 Moving Evaluation Responsibilities The responsibility for evaluating an event can be transferred from one monitor to another based on the occurrence of an event at runtime, by using a command of the form: if move to (|(CCM))

condname is an event which determines if the responsibility for recognising the named event should be reallocated. On occurrence of condname, the current monitor evaluating the event stops evaluation and sends the event state information to the specified monitor. The destination monitor may be CCM or the monitor exe, running on machine mac with instance number instance. For example, eval e1 at em saturn 1 e2 is exit place e2 at myprog saturn EVERY e3 is e2 count 4 eval e3 at em saturn 1 if e3 move e1 to em jupiter 1

3.3.5 Discussion In EVEREST, separating the definition of events from the definition of the monitoring configuration and evaluation strategy offers a significant amount of flexibility to the users. Event definitions can be made in an abstract fashion and mapped into different physical configurations at a later stage. Also, a variety of abstract and physical configurations can be specified for event recognition. The creation and configuration of the monitoring system is dynamically driven by the behaviour of the application and the directives of the strategy specified in the script file. The occurrence of primitive events generates event information and it is the need to evaluate this information which drives the configuration of the monitoring system (e.g., by initiating the creation of new evaluation monitors). This demand-driven monitoring capability and the ability to place event recognition responsibilities closer to the sources of events reduces both the interprocess communication and the delay in event detection, and increases the efficiency of the monitoring system. The following limitations of EVEREST can be identified. As we have seen, communication delays make the task of composite event detection very difficult. Although the system allows the task of event composition to be distributed among different monitors, as specified by the user, it is not certain how late-arriving events are handled by those monitors.

57

Chapter 3 Related Work Primitive events can only have one event dependent attribute which may be tested. This restricts the amount of the required information that can be associated with the occurrence. It is not obvious how the event timestamp or the identifier of the source component which generates the event can be accessed (if at all). Also, no temporal events can be specified, which are important for management purposes and their absence reduces the necessary monitoring capabilities. The definition of events and monitoring configurations is done statically through a script file. Changes to these definitions involve halting and recompilation of the system. This makes dynamic changes to the monitored behaviour impossible.

3.4 OSI Management Standards 3.4.1 OSI Management Approach The International Standards Organisation (ISO) have defined a series of standards for the management of the communication system for Open Systems Interconnection (OSI). These define managed objects as a representation of a managed resource, but managers do not directly invoke operations on managed objects as we have assumed in figure 1.1. Instead, managers interact with a management agent which is local to the managed objects, using the Common Management Information Protocol (CMIP) as shown in figure 3.6. CMIP provides the primitives for supporting management operations to permit the managers to control remote managed objects and query state, and notifications can be used by the agent to send status and event reports from the managed objects to the manager. Managed Open System Management Operations Manager

Agent Notifications Communications using CMIP Protocols

Performing Management Operations Notifications Emitted

Local System Environment

Managed Objects

Figure 3.6 OSI Interaction Model Managed objects may also be used to implement elements within a monitoring system to perform processing or dissemination functions as explained below. Managed objects may be created and deleted dynamically, in response to changes in the managed system or to change the monitoring system. Managed objects constitute the Management Information Base (MIB).

58

Chapter 3 Related Work The OSI standards define a set of management functions relating to configuration management, fault management, performance management, accounting and security management for communications systems, but monitoring is not a specific management function. However, the specifications relating to the above 5 functions and, in particular, fault and performance management, define elements which could be used as a generic monitoring service for distributed systems. A summary of monitoring and event reporting in OSI can be found in (LaBarre 1991). The OSI Guidelines for the Definition of Managed Objects (GDMO) define a set of techniques and a notation for specifying managed objects (ISO 1992b).

3.4.2 Generation of Monitoring Information The state variables which are made visible by objects are called attributes. A generic set of states and the changes which can take place between those states are defined in (ISO 1990a). These states are reflected in a set of attributes which are common to all managed objects. Additional attributes may be provided to represent the application specific status vector of the object. The typical information which would be generated in a state change event report sent to a manager includes: • Managed object class & instance identifiers • Event time • Old and new values for all state attributes • Application specific status attributes Objects often generate notifications to their local agent as event or status reports. The agent performs filtering and dissemination of this monitoring information using event discriminator objects, as explained later. An alternative, for very simple objects, is to use a metric object to periodically poll the managed object to read the attributes and then generate the notifications (ISO 1992a). The predefined event reports include: i) Object Management reports generated whenever a managed object is created / deleted, an operational / administrative state change occurs, or changes occur in non-state attributes e.g., names or important operational parameters (ISO 1990a). ii) Alarm Reports have been defined for the following alarm classes (ISO 1990b): • communication faults e.g., call set up errors, signal distortion • quality of service degradation e.g., problems with throughput or response time • processing errors e.g., buffer overflow, file access error, memory violation • equipment alarms e.g., cable cut detected, locked ports or power problems • environment problems e.g., high/low temperature, smoke detection or excess humidity. 59

Chapter 3 Related Work An alarm message can include probable cause, specific problem code, additional problem data, perceived severity, severity trend, back up status, back up object instance, threshold information, proposed repair, additional textual information as well as state attribute values.

3.4.3 Event Reporting Service The OSI Event Reporting Service performs both the processing and dissemination functions of our monitoring model. It is responsible for both filtering of event reports and dissemination of the selected reports to chosen destinations (ISO 1990c). The components which constitute the service are shown in figure 3.7. EFD Notifications Event Reports Potential Event Reports

Noifications

Event detection & processing

•••

Event Forwarding Discriminators (EFDs)

Managed Objects

Management Operations

Management Responses

Figure 3.7 OSI Event Reporting Service Notifications are received by a local event detection and processing component which adds the event time, originating object class and instance to the notification message to form a potential event report. These are conceptually sent to all local Event Forwarding Discriminators (EFDs). An EFD holds a discriminator construct which defines the criteria used to select which of the potential event reports are forwarded to the destination address stored in the EFD. An EFD sends reports to a single destination at any time. The EFD is a managed object and so can itself generate notifications which are forwarded by another local EFD. The EFDs specify the following information: • EFD Identifier: unique identifier for the EFD • Discriminator Construct: a logical expression on attributes within the potential event reports, for example, {{(dest=fred) and (count >50)} or {(source=xyz) and (size @y)

4

x

y

(@x>@y) && (|@x|@y) 5

x

(@x>@y) && (|@x@y) && (|@x==|@y) (@x==@y) && (|@x=[3*sec]) && (x.a>10) && (y.b == 20)

can be rewritten as: (b) z:((x:e1 when x.a > 10 ; e2) & (e3 ; y:e4 when y.b == 20)) when (@z - |@z) >= [3*sec]

In version (b) the guards associated with component events (e1 and e4) are evaluated earlier and therefore filtering can be performed at an earlier stage 9. A serious issue concerns the use of guards that are associated with event expressions constructed using | and ! event operators. The guard may refer to the attribute of an event which has not actually occurred. To cope with this situation the operator ? can be used to check and see if the event has occurred before referring to its attributes. The use of this operator is illustrated in section 4.4. As it will be discussed later event information is maintained as long as it can potentially contribute to new rule firings and is not considered out of date, as specified by the user. Temporal constraints are particularly important because they can determine when to discard this information.

4.3 GEM Scripts A GEM script may consist of event declarations, rule definitions and top level commands and can be down-loaded to a particular event monitor.

4.3.1 Event Declarations Event declaration is necessary to determine the explicit user-declared classes of events that can be manipulated by the monitor and the names and types of their attributes. An event declaration (for both internally and externally triggered events) has the following format:

9

Pushing guard subexpressions into the associated event expression in this way can be done automatically by a clever interpreter but in the current implementation, this is left to the programmer.

75

Chapter 4 GEM event [()]

where event-id uniquely identifies the class of the event. For externally triggered events it provides an association between the monitoring report and the event triggered by the monitor when that report is received. The optional formal-attribute-declarations provides the names and types of event dependent attributes associated with the named event. Event independent attributes do not require explicit declaration. The following types are supported in GEM: char, long, double, string, time, itime. The first four are similar to those in C++. The types time and itime are used for time point and time period expressions, respectively. E.g., event signal event warning(double temp, itime interval, string mesg)

Special operators have been provided for accessing event independent attributes. For an event instance e, @e returns its timestamp and $e returns its source id. Event dependent attribute values can be accessed by using the usual dot notation. Attribute a of an event instance e is accessed by e.a.

4.3.2 Rule Definitions As mentioned before, the monitor is passive until an event is triggered. A rule definition specifies a sequence of actions to be performed upon the occurrence of an event and has the following format: rule [] { ==> }

Where, rule-id detection-window event-expression action-sequence

is the unique identifier used to refer to the rule, specifies the time window in which the rule operates, specifies the composite event to be detected, and specifies the actions to be executed when the rule fires.

Each time an event is triggered the monitor tries to satisfy as many rules as possible (i.e., the event is "seen" by all rules which are dependent on it). When the event specified in the LHS of a rule is detected, the rule fires and its action sequence is executed. As we shall see, the firing of a rule can cause other rules to fire, by explicit triggering actions. Even for relatively small rule sets, rule firing and rule interaction can quickly become complex and unpredictable, so a precise execution semantics is essential. We have adopted a simple

76

Chapter 4 GEM execution semantics in which all the rules that can fire will fire and their action sequences are executed to the end. Other more sophisticated execution semantics exists in rule based languages such as OPS5 (Brownston et al. 1985). In OPS5 a conflict resolution algorithm is used based on factors such as specificity and condition counts in the LHS of the rules to decide which one of the rules must be chosen for execution.

i) Detection Window When the monitor is instantiated a default detection window (dw) from now-dw to now is specified that determines the sliding time window within which the monitor and, therefore, the rules may operate. The default can be overridden by defining a rule specific dw using a time period expression. Periodically, the dw of all rules are moved forward in time by a predefined amount — a stepping time period10, again set at monitor instantiation time. The occurrence time of an event that can potentially contribute to firing of the rule must fall within the rule's detection window, otherwise the event is ignored or discarded. In effect each rule can be viewed as operating on items (events) placed on a conveyor belt. These items fall off the edge at the end of the specified period of time, as shown in figure 4.3. Default detection window (dw) now rule r1 [15*hour] {...}

dw = 15 hours

Discarded rule r2 {...}

dw = 12 hours (Default)

rule r1 [9*hour] {...}

dw = 9 hours Time

... 4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20 21 ...

Figure 4.3 Conveyor belt analogy of a detection window The detection window has a dual purpose. It is used as a primary mechanism for dealing with delays by ignoring late arrivals and as a means of specifying persistency for events within the system. Each rule may maintain its own event history (as described in chapter

1 0 The clock resolution, the size of the detection window(s), the expected rate of event occurrences and the

available size of the memory may have to be taken into account in order to select the value of the stepping time period. Note that the detection window associated with each rule must be a multiple of the stepping time period otherwise, events may persist longer than they should.

77

Chapter 4 GEM 7). As the window moves forward, the portion of this history which falls outside dw is discarded. Intuitively dw indicates how long the programmer is interested in an event and enables the user to control the discarding of event information at rule level. This is particularly important when dealing with long delays and to avoid the constituents of an as yet incomplete composite event occurrence to linger on in the system forever.

ii) Event Expression Specification of events and the structure of event expressions were discussed at length in previous sections. Chapter 7 describes the algorithm and mechanism used for evaluation of such expressions.

iii) Action Sequence One or more actions can be specified in the RHS of a rule which are executed in the form of a procedural code. The following actions are possible: notify, forward, trigger, enable and disable. a) notify Action The notify action can be used to explicitly generate and report a new instance of a userdeclared event. It has the following format: notify [()]

where, event-id attribute-value-list

specifies the class of the event to be notified, and is the list of values for its event dependent attributes

The position, type and number of the supplied and expected attribute values must match exactly (c.f., parameter passing in C). These are checked when the rule is defined. Its timestamp is set to the occurrence time of the composite event in the LHS of the rule and its source id. is set to the monitor's own identifier. The newly generated event would not be visible internally (i.e., other rules will not see it). Only "the outside world" will detect its occurrence. For example, given the following event declaration: event e1 event e2 ( long l ) event e3 ( long l, double d, string s )

78

Chapter 4 GEM we can write: rule testrule {\ ==>\ notify e1;\ notify e2(10);\ notify e3(10, 3.4, "Hello World")\ }

As shown in this example notifications with varying levels of details can be generated for different managers. b) forward Action The forward action is used to report one or more user-declared primitive event occurrences which contributed to the rule firing and is particularly useful for filtering purposes. This differs from the notify action in that forward reports one or more existing primitive event instances whereas notify effectively causes a new event instance (with possibly new timestamp, source id. and user specified attributes) to be generated and reported. This action has the following format: forward (|)

An event class identifier can be used to directly refer to the corresponding primitive event occurrence. For example, given a predeclared event, link_failure, we can write: rule failure { link_failure when ... ==> forward link_failure }

An event variable can also be used to rewrite the failure rule in the following more compact form: rule failure { x:link_failure when ... ==> forward x }

An event variable is necessary to unambiguously distinguish between multiple instances of an event class which may have to be reported. It also provides local naming for a composite event occurrence and is used to report its constituent user-declared primitive events. For example, given the predeclared events e1 and e2, the following rules can be defined: rule R1 { e1 ; e2 ==> forward e2 }

R1 reports the occurrence of an e2 after an e1 has occurred. rule R2 { e1 ; e2 ; x:e2 ==> forward x }

79

Chapter 4 GEM R2 reports the second occurrence of an e2 after an e1 has occurred. rule R3 { e1 ; x:(e2 & e3) ==> forward x }

R3 reports the occurrences of an e2 and an e3 (in any order), after an e1 has occurred. Implicit temporal events cannot be directly reported in this way. They have to be defined in terms of user-declared events. For example, event time_ten rule R4 { at[10:00] ==> time_ten } rule R5 { x:((time_ten & e2) ; e3) ==> forward x }

Rule R4, defines the implicit time event at[10:00] in terms of the user-declared event time_ten. At the specified time, time_ten is triggered by the rule (event triggering is described in the next section). Rule R5 can detect its occurrence and report it as one of the constituent events when it fires. c) trigger Action A user-declared event can be triggered by simply "calling" it, much in the same way as a function call in conventional languages, with the following format: [()]

where, event-id attribute-value-list

specifies the class of the event to be triggered, and is the list of values for its event dependent attributes

This is similar to the notify with respect to setting attributes. For example, given the following event declaration: event signal_loss(long a, string m)

We can write: rule r1 { \ ==> signal_loss(20, "Link Failure")

The newly triggered event is visible to all the other enabled rules within the monitor. The "outside world" cannot detect its occurrence unless explicitly reported by another rule using the notify or forward action. The trigger action provides a mechanism for easily naming and referring to the same composite event in different rules. E.g., rule r2 { ...; signal_loss;... ==> ...}

80

Chapter 4 GEM This provides a convenient abstraction mechanism. An internal event can be defined, triggered and notified in terms of other internal and external events and would typically inherit attributes from the events referenced in the LHS of the rule. An event triggering could in turn cause a number of rules to fire and other events to be triggered. The order in which the rules are fired is implementation dependent, but all the rules which can fire, will fire. A rule cannot directly "see" the events that it triggers. E.g., rule r2 { ... ; signal_loss ; ... ==>

...; signal_loss ; ... }

This prevents recursive firing of r2, given that no other rule makes signal_loss visible to it. Some event dependency checks may have to be performed to identify and avoid recursive and infinite triggering of events, and firing of rules 11. d) enable & disable Actions GEM rules may be enabled or disabled using the following actions, respectively: enable [] disable []

where, rule-id specifies the identity of the rule. If it is omitted, the identity of the current rule is used. When an event is triggered, only the enabled rules are examined. All rules are disabled by default and have to be enabled explicitly. Once enabled, a rule will remain so until it is specifically disabled. As we shall see, the event evaluation tree of each rule may maintain historical data which may ultimately contribute to its firing. This data is automatically discarded when the rule is disabled. These actions provide a powerful mechanism for concentrating on specific aspects of system activity under certain conditions and filtering unwanted information. For example, enabling all the rules related to configuration and performance monitoring and disabling all the fault monitoring rules during a specific period. In the following example, rules A and B refer to the user-declared events e1 and e2.

1 1 Circular event dependencies may span several rules and can be automatically checked. In the current

implementation, however, the programmer is responsible for making these checks.

81

Chapter 4 GEM rule A { e1 ==> forward e1; e2; disable } rule B { e2 + [5*sec] ==> enable A } enable A enable B

The effect of these rules and the top level enable command (as described later) is to report the occurrence of an event of type e1 and to ignore all other occurrences (at least) for the next five seconds. e2 is triggered internally to allow enabling of rule A, 5 seconds after an e1 is forwarded. This is particularly useful in fault management area and in particular the alarm correlation application in which a large number of similar alarms may be generated as a result of the same fault. All the duplicates can be removed for a specified length of time.

4.3.3 Top level Commands GEM provides a number of top level commands which are independent of rules and are used for controlling and querying the state and operation of the monitor. Some of these commands are similar to actions that can be performed in the RHS of the rules.

i) triggering events As we described in section 3.2 an event can be triggered in the RHS of a rule. A similar effect can be produced at the top level by simply "calling" the event with appropriate attributes. The format is similar to that of an event report described earlier: event-id [] [] [()]

This is useful for simulation purposes, checking event dependencies, or initialising the state of the event evaluation tree. For example, given these event declarations: event e1 ( long l, string s, itime it ) event e2

the following can be used as valid top level event triggering commands: e2 e2 [8:30 15/5/95] e1 (3, "some string", [5*hour+30*min]) e1 [10:30:0.0 Mon 15/5/95] (3, "some string", [5*hour+30*min]) e1 "s10" [10:30 15/5/95] (3, "some string", [5*hour+30*min])

82

Chapter 4 GEM

ii) enable & disable commands The enable and disable are similar to the corresponding actions described in section 4.3.2(iii). They can be used to activate or deactivate rules. The format is the same with the exception that the wild card character '*' can only be used at the top level to enable or disable all the existing rules. e.g., disable *

iii) stop command The stop command is used to halt the operation of the monitor. It results in the discarding of all the event information held by the monitor.

iv) ~ (remove) command As monitoring requirements change, the set of declared event classes and defined rules must change. The remove command '~' allows us to delete those which are no longer needed. An event class can be removed using a command of the following form: ~

The specified event class can only be removed (undeclared) if there are no rules which specifically refer to it. If there exist rules dependent on the event class, an error message is displayed and the command is ignored. A rule can be removed using the following command: ~ rule

As a result of this command, the event information maintained by the rule is discarded and the monitor removes the rule. Note that the removed rule may have been explicitly referenced by others. It is the user's responsibility to make sure that those rules are removed or changed accordingly.

v) load command The load command can be used to instruct the monitor to read GEM commands from a specified file. E.g., load "testfile.mi".

vi) ? (query) command Table 4.3 shows the available commands used to query the state of the monitor.

83

Chapter 4 GEM

Commands

Descriptions

?

Display all the user-declared events

?

Display the declaration for the specified event

? rule

Display all the defined rules. Note that some rules may have been referenced but not defined yet.

? rule

Display the definition of the specified rule

? notify

Display the declaration of all the events that are notified by rules

? notify

Display the definition of all rules notifying the specified event

? trigger

Display the declaration of all the events triggered by all rules

? trigger

Display the definition of all the rules triggering the given event

? enable

Display all the rules which are enabled

? enable

Display all the rules which may enable the specified rule

? disable

Display all the rules which are disabled

? disable

Display all the rules which may disable the specified rule

Table 4.3 Available query commands in GEM

4.3.4 Selective Consumption of Events When a rule is fired and the actions in its RHS are executed, by default all the constituent event instances which participated in its firing are consumed which means that they cannot take part in any future firing of that rule. GEM allows the user to override this default consumption by annotating an event using the ^ operator as one which persists within the rule's detection window through different firings. The following examples show the use of ^ operator. (i) Every day at 10:30 am the following rule reports all the daily login event occurrences. rule report_logins [day] { login ; [10:30]^

==> forward login }

Without the ^ operator only the first occurrence of login event would have been reported.

(ii) The exits rule reports all those who left the building up to one hour after a fire alarm. rule exits [hour] { fire^ ; left ==> forward left }

84

Chapter 4 GEM The annotation of an event expression using the '^' operator, applies to it and all its subexpressions.

4.3.5 GEM Error Messages Appendix C lists the error messages generated by the GEM interpreter.

4.4 Examples of GEM Scripts Example 1 event power_cut event disaster

the following rule causes a disaster report to be generated when the occurrence of a power_cut event is detected whose timestamp is between 24th December 1995 and 2nd January 1996 and whose source is the main generator. rule disaster [10*min] {\ x:power_cut\ when ([24/Dec/95] \

85

Chapter 4 GEM notify shut_down("Friday 13th")} enable protection

Example 3 Rule A reports the occurrence of the first user-declared event which occurs within 5 seconds of an event e1. The operator '*' is used as a wild card to refer to any user-declared event. Note that only event independent attributes (timestamp and source identifier) of such an event can be referenced in a guard. No detection window is specified for the rules. In this case it automatically defaults to the detection window specified at monitor start up time. event e1 event e2 ( ... ) ... rule A { ( e1 ; y:*) when (@y-@e1) forward y }

Example 4 The following script specifies that a gauge threshold notification must be generated when the value of the val attribute of a gauge_changed event crosses certain low or high threshold as shown in figure 4.4. Gauge Value

notification upper threshold notification

notification

lower threshold

time

Figure 4.4

Gauge threshold operation

The upper and lower thresholds are 10 and 5, respectively. The interval between these thresholds is called the hysteresis interval. The gauge value may oscillate about either of the thresholds and consequently multiple notifications may be generated. The enable and

86

Chapter 4 GEM disable actions in the rules are used to force such notifications to be generated alternately when the value crosses the thresholds. Here we assume that the initial value is less than 10. event gauge_changed( double val) event upper_exceeded event lower_exceeded rule gauge_rule1 {\ x:gauge_changed when x.val >= 10 ==> \ notify upper_exceeded; enable gauge_rule2; disable \ } rule gauge_rule2 {\ x:gauge_changed when x.val \ notify lower_exceeded; enable gauge_rule1; disable \ } enable gauge_rule1

Example 5 The following backup rule specifies that a backup notification must be generated every hour between 8 am and 5 pm, on working days, and every two hours between 10 am and 4 pm during weekends. Such an event can be used, for example, by a system administrator to know when to make backups of all the system files. The back_up event depends purely on temporal events and guards, not on any external events. event backup rule backup {\ ( x:every[hour]\ when ([8:00] < @x) && (@x < [17:00]) &&\ ([Mon] warning($x, x.t, y.p)\ } enable warning

The detection window of 15 seconds takes into account the maximum delay and the temporal constraint specified in the guard. Although the events specified with variables x and y may occur in any order, the interval specified by @x-@y is always positive, no matter which one occurs first. The scheduled time event can be viewed as the period of time that the rule must wait after the detection of the sub-event before it can fire. During this interval if an event arrives which chronologically precedes one of the existing pairs of events and a successful match can be found, it is automatically picked up instead by the algorithm. Therefore, the older matching pair is correctly recognised as a composite event given a maximum delay of 5 seconds. The event dependent attributes of warning event are set using the attributes of the subevents in the LHS (i.e., the source identifier, the t attribute of the temp_alarm event and the p attribute of the press_alarm). Example 7 The following script is based on a slightly altered version of an example given in (Jakobson & Weissman 1995). The rule EXPECTED_ALARM_RULE detects that a carrier group

88

Chapter 4 GEM alarm type "A" on a network element occurred and during the following 1 minute interval an expected carrier group alarm type "B" did not occur at the same network element. On detecting such an event, it is forwarded by the monitor. event ALARM_A event ALARM_B rule EXPECTED_ALARM_RULE {\ ({x:ALARM_A ; z:(y:ALARM_A + [1*min])} ! w:ALARM_B)\ when (?z && (x == y)) || (?w && ($x == $w))\ ==> forward x \ } enable EXPECTED_ALARM_RULE

The equality between event variables can be used to see if they refer to the same event occurrence(s). A guard associated with event expressions constructed using | or ! event operators may refer to the attribute of an event which has not actually occurred. To cope with this situation the operator ? can be used to check and see if the event has actually occurred before referring to its attributes. In the above example, events z or w may occur after x. This is in concept similar to checking that a pointer is not zero before using it in C/C++. Example 8 The indanger rule can detect and report the names of all the people who entered the building before a fire alarm but failed to leave the building 2 minutes after the alarm was raised. In this example we assume that nobody stays in the building more than 24 hours, hence the detection window is set to one day. The detection window has to be adjusted appropriately if this assumption cannot be made. event entered ( string name ) event left ( string name ) event fire event indanger ( string name ) rule indanger [day] {\ ({ e:entered ; (f:fire + [2*min] )^} ! l:left)\ when (?l && (e.name == l.name)) || (?f && (@e < @f))\

89

Chapter 4 GEM ==> notify indanger ( e.name )\ } enable indanger

The annotation of an event expression using the ^ operator applies to it and all its subexpressions. On each rule firing, after the notification of an indanger event, the entered event instance which participated in the rule firing is consumed. The event records representing the fire and the corresponding scheduled time event (fire + [2*min]) are maintained and can participate in successive rule firings, allowing multiple notifications to be generated as a result of the same fire alarm.

4.5 Conclusion GEM is a declarative, interpreted and Generalised Event Monitoring language. GEM scripts may include event declarations, rule definitions and top level commands. These scripts are interpreted by event monitors which perform common event processing activities (e.g., filtering, composition & notification) on event reports generated in a distributed system. The examples in this chapter illustrated that composite events of arbitrary complexity can be specified in the LHS of the rules. The notion of real-time has been tightly integrated into the language and various temporal constraints may be specified without making any assumptions about the order of occurrence and arrival of events. Event expressions may include specification of scheduled (relative), periodic and absolute times, the occurrence interval of events, the order of event occurrences and the time intervals between event occurrences. Concurrent composite events are detected on-the-fly and in the presence of communication delays and unordered delivery of messages. Built-in features of the language can be used to deal with delays in a flexible and user-defined manner, as described in detail in the next chapter. GEM allows dynamic: • addition and removal of rule definitions and event class declarations with arbitrary number of event specific attributes, • explicit triggering of event instances, • enabling or disabling of specific rules, • loading of specified GEM scripts, and • querying the event class declarations and rule definitions. Event monitors interpreting GEM scripts can be created dynamically close to the sources of event reports, using the capabilities of the underlying Regis/Darwin environment. These

90

Chapter 4 GEM features allow a scalable, dynamic and distributed event processing service to be constructed as a configuration of multiple event disseminators, event monitors and other generic monitoring components, as described in chapter 6. The initial version of GEM was used as a basis for the specification of GMSL (Generalised Monitoring Service Language) in the Esprit SysMan project (7026) (Debski & Janas 1994). It also was used in Hewlett-Packard Laboratories (Bristol), as the basis for an event correlation language for the Telecommunication Management Network. Currently GEM is being used to detect composite events and convert these into simple events which trigger obligation policies within managers which then perform the management actions specified by these policies (Sloman 1994).

91



5.1 Introduction In the previous chapter we described various features of the GEM language which can be used to specify, filter, compose and notify events. This chapter discusses the effects of delays and out-of-order event arrivals on composite event detection and presents a flexible solution using built-in features of GEM. A detailed example is followed to demonstrate our approach.

5.2 Problem of Delays Variable communication delays in a distributed environment means that the occurrence and the detection times of an event may not be the same. In addition, events may not be received and observed in the order of occurrence. These make the task of detecting composite events particularly difficult. A composite event is defined in the context of a sequence of primitive events over a time period called the occurrence interval. It is defined as the period of time during which the event is in the process of occurring. The first and the last primitive events in the interval are called the initiator and the terminator, respectively, as shown in the figure 5.1. The terminator of a composite event is particularly significant as its timestamp is considered to be the occurrence timestamp of the composite event as a whole. occurrence interval initiator occurrence

detection

• •

terminator

•

•

• •

•

•

• •

Time

Figure 5.1 Occurrence interval, initiator & terminator of composite events

92

Chapter 5 Dealing with Delays Due to the effects of delays, event composition techniques, commonly used in centralised systems, may not recognise certain composite events or may result in erroneous and invalid detections and consequently unwanted rule firings. This is particularly true for composite events whose occurrence intervals overlap, possibly causing the constituents of one occurrence to be matched with those of another occurrence. Figure 5.2 shows examples of invalid detections caused by delayed events: rule r1 {{a ; b} ! c ==> ... } rule r2 {(e ; f) ==> ... } e f 20 23 a b 1 4

e f 22 23 e f 20 21

a c

b

.........

Occurrence Arrival & Detection

e

f e

f

......... ......... 0

1

2

3

4

5

6

7

8 .... 20 21 22 23 24 25 26 Time

Erroneous Detections

Expected Detections

Figure 5.2 Examples of erroneous event detection due to delays Let us assume that an event instance of class e with occurrence timestamp t is represented as et. The figure shows a sequence of such instances which are received by an event monitor after some delay. It also shows the definitions of two rules r1 and r2, which depend on these events and illustrates the results of the expected and erroneous detections. Given the delayed sequence (a1, b4, c2) the rule r1 will erroneously fire on b4, its action sequence is executed and the constituent events (a1 b4) are consumed before the cancellation event c2 arrives. Similarly given the delayed sequence ( e20, e22, f23, f21) the rule r2 fires when the constituent events (e20 f23) are detected. However the expected behaviour should have been for r2 to fire twice on the sequences (e20 f21) and (e22 f23). We call a detection valid when no late-arriving event could invalidate or alter the detected composite event. A valid detection is very difficult in the absence of explicit guarantees, provided by the underlying communication system, about the orderings or the maximum

93

Chapter 5 Dealing with Delays possible delays involved12. In the absence of such guarantees, the best that can be done is to assume a maximum tolerated delay and to discard late-arriving events (c.f., timeouts).

5.3 Uniform Delaying Technique Given a well-synchronised global clock and a known maximum communication delay D, one obvious approach which has been adopted by (Shim & Ramamoorthy 1990) is to use a separate ordering stage which uniformly delays every event by D time units before it is passed to a detection stage. This is illustrated in figure 5.3. Event reports

Ordered history segment

Clock

Detection

Ordering

Event Specifications

Maximum Delay value (D)

Figure 5.3 Uniform Delaying Technique rule r2 {(e ; f) ==> ... }

rule r1 {{a ; b} ! c ==> ... }

e f 20 21 e f 22 23

a c

b

....... e f e f

Occurrence Arrival

.......

Detection

....... 0

1

2

3

4

5

6

7

8

9

10 ... 20 21 22 23 24 25 26 27 28 29 30 Time

Figure 5.4 Application of the uniform delaying technique During the ordering stage, every newly arrived event report e, with occurrence timestamp t, is inserted in its correct place in a totally ordered event history whose elements satisfy the following constraint: (t + D) < C, where C is the current time. On every clock update the

1 2 Even given such guarantees, failures may prevent a valid detection, but we shall not consider the effects

of failures here.

94

Chapter 5 Dealing with Delays segment of the history which has been delayed enough is forwarded to the detection stage where an existing composite event detection algorithm (e.g., based on petri-nets, evaluation trees or state machines) can be used to detect the specified event. The application of this technique to our earlier example is shown in figure 5.4. In this example D, is assumed to be 6 seconds. The ordered sequence can now be used for a valid event detection. This approach has several disadvantages, as it is: pessimistic – it introduces possibly unnecessary additional delays for all events, inefficient – every event is ordered with respect to all the others in the maintained history even when its ordering is not significant in the ultimate detection stage, inflexible – in some circumstances no explicit ordering may be necessary e.g. for events generated locally or in a local area network that provides ordering guarantees. This knowledge about specific events cannot be used.

5.4 Dealing with Delays in GEM The approach that we have adopted is to push the knowledge and the capability of dealing with delays into the detection stage itself. This is done using the following built-in features of GEM: i) Detection Windows: As described before, each rule operates within a sliding detection window (dw) which can be viewed as an absolute timeout facility. The occurrence of a composite event, represented by the rule's event expression is detected if all its constituent events are within this time window. Events arriving outside this window are ignored by the rule. Also, as the window slides forward in time, old event information maintained by the rule is discarded. The length of the window specified by the user determines the maximum delay that the rule can cope with. ii) Ordering of Events in a Hierarchical History: GEM's event detection algorithm makes no assumptions about communication delays and the order of arrival of events. Every event record is inserted in chronological order in an internal hierarchical history maintained in a tree-based structure. Not every record is ordered with respect to all others in the history as only the necessary orderings are preserved. The structure of the evaluation mechanism and its maintained history is described in chapter 7. iii) Incremental Detections & Cancellations: Event detection is performed in an incremental fashion. On every new event arrival, more than one event record may be generated and added to the history associated with a composite event expression E, with each new record corresponding to one of E's event operators or sub-expressions. The detection algorithm always preserves the consistency of the maintained history with the associated event specification. A late-arriving new event may invalidate one or more

95

Chapter 5 Dealing with Delays higher level and intermediate event records which violate this consistency and cause them to be cancelled and discarded. A rule fires when the arrival of a new event results in the generation of an event record for its event expression at the highest level. At this point the rule's actions are executed and the constituent event records participating in the firing are consumed13. Event consumptions may also result in cancellation of other higher level intermediate records which are dependent on the consumed ones. The outline of the detection algorithm and a detailed example are presented in chapter 7. iii) Local Detection of Temporal Events: Using a synchronised global clock, every event monitor relies on a local timer component to detect temporal events. The delay in detecting local events is considered negligible. iv) Scheduled Time Events: As we have seen, a scheduled time event in GEM can be specified as E+[I], where E is an expression representing any primitive or composite event and I is the schedule time interval. Given a known maximum delay D in detection of E and the features described above, a scheduled time event can be used for a valid detection of E. To do this a rule of the form: rule R1 [DW] { E ==> ... }

must be rewritten as: rule R2 [DW+I] { E + I ==> ... }

where I>=D. The scheduled time event can be viewed as a dummy terminator event which is detected locally and is added to the original composite event (specified by E) to provide a correction interval I. The algorithm guarantees that during this interval it will cancel and discard all erroneous and invalid event records in the history maintained for E or its subexpressions. It can also be viewed as a mechanism for deliberately delaying the firing of the rule until all its late-arriving constituent events are received. Note that R2's detection window must also be extended by I to achieve the original result (hence [DW+I]). Figure 5.5 shows rules r 1 and r 2 which are replaced with rules r 1.1 and r 2.1 14 , according to this approach.

1 3 The ^ operator can be used to override such automatic consumptions, as described in section 4.3.4. 1 4 Both rules' detection windows must be set appropriately. For example if the (default) detection window

of r1 was 20 seconds, r1.1's detection window must now be 26 seconds.

96


rule r2.1 ... { (e ; f) + [6*sec] ==> ... } e f T 22 23 29 e f T 20 21 27

e f 20 21 rule r1.1 ... {({a ; b} ! c ) + [6*sec] ==> ... } e f 22 23 a b 1 4 a c

e f 20 23

C

b

T....... e f

Occurrence Arrival & Detection

e f

C T

T

....... 0

1

2

3

4

5

6

7

8

9

10 ... 20 21 22 23 24 25 26 27 28 29

subevents

T

time event

Top level event

C cancel subevent

Time

Schedule time event

Figure 5.5 An example of event specific delaying technique Consider the rule r 1.1. At time 5 on the arrival of b 4 the event specified by the subexpression ({a ; b} ! c) is detected by the algorithm and a corresponding event record is inserted in the history. This causes a time event to be scheduled for time 10 ( b 4 's occurrence timestamp + delay). The event record representing the sub-event is maintained in the history while we wait for the time event. When the cancellation event c2 arrives (at time 7) the algorithm makes sure that the event record is automatically cancelled so that the rule is not fired erroneously. The figure also shows the incremental and concurrent detections of composite events specified by rule r2.1's event expression, as the sequence (e20, e22, f23, f21) arrives at the monitor. It shows how the event record with "wrongly matched" constituent events (e20 f23) is cancelled at time 27. Rule r2.1 is fired twice with composite event instances which now include a constituent dummy time event T. This approach has several advantages. A scheduled time event can be associated with any event specified in GEM and, therefore, a similar control over delays can be exercised at both primitive and composite event level. It provides the user with unlimited flexibility and

97

Chapter 5 Dealing with Delays a finer grain of control in dealing with delays 15. This is particularly useful when events referenced in a rule may arrive from sources with different expected delays or where there is a known temporal relationship between event occurrences (e.g., events e1 and e2 will never occur more than 2 minutes apart). This information can be directly incorporated into the rule definitions. Furthermore the detection time is not affected by the external buffering strategy employed at a separate ordering stage. The disadvantage is that the detection algorithm and the associated mechanism is now more complex as it now should deal with delays as well.

5.5 Calculating D and I In general there is always some delay in the detection of an event which may include both communication and processing times. In practice, delays introduced by processing will be a very small fraction of the total delays. In a distributed monitoring system each subexpression of E may be detected at different stages and by monitors running on different machines. Therefore in determining the value of D the maximum cumulative communication and processing times associated with all E's sub-expressions would have to be taken into account. This includes the delays introduced at successive stages of correlation and filtering (e.g., previous forwarding actions by other monitors). Table 5.1 shows how the minimum value of I can be calculated for GEM event expressions with respect to expected delays associated with E's sub-expressions. Event Expression (E)

Correction Interval (I)

e

D

User declared (basic) events.

at[...]

0

By assumption these events are detected locally with negligible delays.

every[...]

Comment

E1 ; E2 E1 & E2

max(D1,D2)

E1 | E2 E1 + [i]

{E1 ; E2} ! E3

if (i=D1), i already covers D1 therefore

else 0

the required correction interval is zero.

max(D1,D2,D3)

Table 5.1 Calculating the correction interval

1 5 It is not always possible to determine a maximum value for delays therefore the specified time period

may represent the tolerated delay interval rather than an expected one.

98


5.6 An Example The following detailed example is based on the Mobile Biff service which is described in (Bacon et al. 1995). It illustrates the problems that delayed arrival of events may cause in event composition in a distributed environment and shows how the features of GEM can be used to deal with them. The Mobile Biff service allows an active badge holder to be notified of the arrival of a new email when he/she is anywhere but his/her office. This is done by triggering the beeping of the badge. Figure 5.6 shows the GEM script that can be downloaded to a monitor in order to detect the necessary composite event and to generate the notifications. event left ( string n ) event entered ( string n ) event has_mail ( string n ) event beep ( string n ) rule beep [day] {\ ({ l:left ; m:has_mail } ! e:entered )\ when ( ?m && ( m.n == l.n )) || ( ?e && ( l.n == e.n ))\ ==> notify beep ( l.n )\ } Mobile biff GEM Script enable beep

left entered

monitor

beep

has_mail

Figure 5.6 An overview of the monitor's operation and its GEM script The composite event to be detected has three primitive events associated with it. The events entered and left representing the user entering or leaving their office, respectively. These are generated by a location monitor (locmon). The has_mail event is generated by the mail server and represents the arrival of a new email for a person. All these events have one event dependent attribute — the name of the individual concerned. In this example, we assume that employees will not stay in their offices for more than 24 hours, hence the detection window of one day. When the composite event on the LHS of the rule is detected, the rule fires and a beep notification is generated which can be used to sound the beep of the person's active badge. Obviously this is not necessary or desirable when the person is in his/her office as it may disturb the people working in the same office. But considering that the locmon, mail_server and the monitor executing the GEM script are distributed and events may be subject to variable communication delays, this is exactly what may happen. Figure 5.7 (a-b) show a 99

Chapter 5 Dealing with Delays particular sequence of such events arriving at the monitor (on the left) and the resulting notifications generated by it (on the right). They show the effects of delays on messages and the resulting reorderings. In particular note the following sequence of messages: ... left "locmon" [15:25:16.454 Tue 18/7/95] ("John") ... entered "locmon" [15:31:16.618 Tue 18/7/95] ("John") has_mail "mail_server" [15:33:16.701 Tue 18/7/95] ("John") ...

The arrival of the event report has_mail("John") caused the rule to fire and an unnecessary beep("John") notification to be generated before the event report entered("John") arrived.

100


entered "locmon" 15:3 [15:3:15.853 Tue 18/7/95] ("Smith") 15:4 15:5 15:6 left "locmon" [15:7:16.51 Tue 18/7/95] ("Smith") has_mail "mail_server" [15:8:16.82 Tue 18/7/95] ("Smith")

15:7 15:8 15:9 15:10 15:11

beep "monitor" [15:8:16.82 Tue 18/7/95] ("Smith")

15:12 has_mail "mail_server" [15:12:16.172 Tue 18/7/95] ("Smith") 15:13 15:14 15:15 15:16 entered "locmon" 15:17 [15:17:16.234 Tue 18/7/95] ("John") 15:18 15:19 entered "locmon" [15:20:16.329 Tue 18/7/95] ("Tim")

15:20 15:21 15:22 15:23

has_mail "mail_server" 15:24 [15:24:16.380 Tue 18/7/95] ("John") 15:25 left "locmon" [15:25:16.454 Tue 18/7/95] ("John") 15:26 has_mail "mail_server" [15:27:16.521 Tue 18/7/95] ("Tim")

15:27 15:28 15:29 15:30

Figure 5.7 (a) Event reports received & generated by the monitor

101


entered "locmon" 15:31 [15:31:16.618 Tue 18/7/95] ("John") 15:32 15:33 has_mail "mail_server" [15:33:16.701 Tue 18/7/95] ("John") 15:34

* Erroneous notification of a beep event by the monitor due to unordered arrivals of entered and has_mail events. beep "monitor" [15:33:16.701 Tue 18/7/95] ("John")

15:35 left "locmon" 15:36 [15:36:16.759 Tue 18/7/95] ("Tim") 15:37 has_mail "mail_server" [15:38:16.823 Tue 18/7/95] ("Tim")

15:38 15:39 15:40 15:41 15:42

beep "monitor" [15:38:16.823 Tue 18/7/95] ("Tim")

15:43 left "locmon" 15:44 [15:44:16.907 Tue 18/7/95] ("John") 15:45 15:46 15:47 has_mail "mail_server" 15:48 [15:48:16.985 Tue 18/7/95] ("John") 15:49

beep "monitor" [15:48:16.985 Tue 18/7/95] ("John")

15:50 entered "locmon" [15:52:17.55 Tue 18/7/95] ("Tim") left "locmon" [15:53:17.137 Tue 18/7/95] ("Tim")

15:51 15:52 15:53 15:54

has_mail "mail_server" [15:54:17.213 Tue 18/7/95] ("Tim")

15:55

beep "monitor" [15:54:17.213 Tue 18/7/95] ("Tim")

15:56 15:57

Figure 5.7 (b) Continued from figure 5.7 (a)

102

Chapter 5 Dealing with Delays Assuming that all the incoming events may be subject to a maximum delay of 5 minutes16 this problem can be solved by using a scheduled time event (and a corresponding extension to the rule's detection window), to deliberately delay rule firing until late-arriving events associated with the original composite event are received. The beep rule can be rewritten as: rule beep [day+5*min] {\ (({ l:left ; m:has_mail } ! e:entered )\ when ( ?m && ( m.n == l.n )) ||\ ( ?e && ( l.n == e.n ))) + [5*min] \ ==> notify beep ( l.n )\ }

The algorithm makes sure that the erroneous scheduled beep("John") notification is cancelled when entered("John") event arrives. This is illustrated in figure 5.8 (a-b) which shows a similar sequence of events arriving at the monitor with the same length and pattern of delays. With the new version of the rule, in addition to this sequence, there is a 5 minute scheduled time event relative to every has_mail event. In figure 5.8 (b) consider a corresponding sequence of messages which caused the erroneous rule firing in figure 5.7 (b): ... left "locmonitor" [16:25:34.390 Tue 18/7/95] ("John") ... entered "locmonitor" [16:31:34.545 Tue 18/7/95] ("John") has_mail "mail_server" [16:33:34.610 Tue 18/7/95] ("John") ...

The arrival of has_mail("John") event caused the monitor to request its timer for a time event scheduled for 5 minutes after has_mail's occurrence time (i.e., at 16:38:34). Arrival of entered("John") event at the monitor at around 16:35:30 causes the scheduled notification of beep event to be cancelled and hence prevents the undesirable behaviour. In addition to dealing with delays the example shows how concurrent composite events (i.e., with overlapping occurrence intervals) are detected with all the possible reorderings.

1 6 In this case a 5 minute delay is probably too high but it is used here just for illustration.

103


entered "locmon" [16:3:33.859 Tue 18/7/95] ("Smith")

16:3 16:4 16:5 16:6

left "locmon" [16:7:33.936 Tue 18/7/95] ("Smith") has_mail "mail_server" [16:8:34.10 Tue 18/7/95] ("Smith")

16:7 16:8 16:9 16:10 16:11

Schedule time event at 16:13:34.10 (latest has_mail's time + [5*min]) when the rule can fire

16:12 has_mail "mail_server" [16:12:34.990 Tue 18/7/95] ("Smith") 16:13 16:14

Scheduled time event beep "monitor" [16:13:34.10 Tue 18/7/95] ("Smith")

16:15 16:16 16:17 entered "locmon" [16:17:34.182 Tue 18/7/95] ("John") 16:18 16:19 entered "locmon" [16:20:34.274 Tue 18/7/95] ("Tim")

16:20 16:21 16:22 16:23

has_mail "mail_server" [16:24:34.358 Tue 18/7/95] ("John")

16:24 16:25

left "locmon" 16:26 [16:25:34.390 Tue 18/7/95] ("John") 16:27 has_mail "mail_server" [16:27:34.454 Tue 18/7/95] ("Tim") 16:28 16:29 16:30

Figure 5.8 (a) Event reports received & generated by the monitor

104


16:31 entered "locmon" [16:31:34.545 Tue 18/7/95] ("John") 16:32 16:33 has_mail "mail_server" [16:33:34.610 Tue 18/7/95] ("John") 16:34 16:35 left "locmon" [16:36:34.697 Tue 18/7/95] ("Tim")

16:36

Schedule time event at 16:38:34.610 (has_mail's time + [5*min]) Cancels intermediate subevent and stops rule firing at 16:38:34.610

16:37 16:38

has_mail "mail_server" [16:38:34.731 Tue 18/7/95] ("Tim")

16:39 16:40 16:41 16:42

Schedule time event at 16:43:34.731 (has_mail's time + [5*min])

16:43 left "locmon" [16:44:34.827 Tue 18/7/95] ("John")

16:44

Scheduled time event beep "monitor" [16:43:34.731 Tue 18/7/95] ("Tim")

16:45 16:46 16:47

16:48 has_mail "mail_server" [16:48:34.869 Tue 18/7/95] ("John") 16:49


16:50 16:51 entered "locmon" [16:52:34.939 Tue 18/7/95] ("Tim") left "locmon" [16:53:35.23 Tue 18/7/95] ("Tim") has_mail "mail_server" [16:54:35.98 Tue 18/7/95] ("Tim")

16:52 16:53 16:54

Scheduled time event beep "monitor" [16:53:34.869 Tue 18/7/95] ("John")

16:55 16:56 16:57


16:58 16:59 17:00

Scheduled time event beep "monitor" [16:59:35.98 Tue 18/7/95] ("Tim")

17:1

Figure 5.8 (b) Continued from figure 5.8 (a)

105


5.7 Conclusion Communication delays complicate the task of composite event detection in a distributed environment as events may be detected some time after their occurrence and possibly in the wrong order. As a result, event composition techniques commonly used in centralised systems, may not recognise certain events or may cause invalid detections (e.g., constituents of two composite events with overlapping occurrence intervals may be matched incorrectly). GEM allows this problem to be dealt with in a user-defined and flexible manner. Default and user-specified detection windows can be associated with monitoring rules, as a means of providing a flexible ageing mechanism for events and allowing late-arriving events to be processed or discarded, as required. Also, new event records arriving within such a window are inserted in their right place in an ordered hierarchical history. On-the-fly and concurrent detection of composite events proceeds in an incremental fashion as new events arrive. Late-arriving event records may result in automatic cancellation of invalid intermediate records (the detection mechanism and algorithm are described in chapter 7). Based on the above features, a scheduled time event can be used to provide a correction interval (within the corresponding detection window) which guarantees the valid detection of composite events in the presence of given communication delays. Chapters 4 and 5 showed how events could be specified, filtered, correlated and notified in the presence of delays. The next chapter shows how event monitors and other generic monitoring components can be used to provide a distributed event monitoring service.

106



6.1 Introduction In this chapter we will describe how a distributed event monitoring service can be constructed from a set of generic components (event monitors, timers, disseminators, etc.) using the Regis/Darwin environment (Magee et al. 1994). Figure 6.1 shows the general overview of an event service which is an integral part of a generalised distributed monitoring service (Mansouri-Samani & Sloman 1994). Event generators may be instrumented monitored objects or other components responsible for polling and observing the activity and changes in the status of monitored objects. Event monitors process event reports from different nodes in the system by performing filtering, composition and notification on these reports based on a given specification. Event disseminators send event reports to clients who subscribe to receive them. Note that event monitors can themselves act as generators for other monitors. Primitive & Composite Events

Primitive Events Event Generator

Databases

Event Monitor

Event Disseminator

Specification

Subscribe

Managers

Display

Figure 6.1 Components of an event monitoring service It is shown how these components can be distributed and configured onto physical nodes to perform the required monitoring functionalities close to where they occur and thus reduce network traffic. The significant features of the Regis/Darwin environment are

107

Chapter 6 Distributed Monitoring Service described by showing the overall operation, structure and interaction of these components. A detailed example is presented to demonstrate the operation of a constructed distributed monitoring service.

6.2 Regis/Darwin environment Regis is a programming environment for development and execution of distributed programs. It separates program structure from communication and computation. Regis programs consist of multiple, concurrently executing and interacting C++ computational components which execute as lightweight threads or processes. These components interact via communication objects also programmed in C++. Darwin is the configuration language used to define the structure of a distributed system as a composite component type which defines internal primitive or composite component instances and interface bindings. Composite component types are constructed from the basic computational components and these, in turn, can be configured into more complex composite types in a hierarchical fashion.

6.2.1 The monit Component Figure 6.2 shows the monit component and its Darwin interface, which interprets GEM scripts and provides the basic filtering, composition and notification functionalities of the monitoring service as described before. component monit ( char *n, double d, double r ) { provide command ; inrep ; command require outrep outrep ; tcancel ; trequest inrep treq ; } monit tcancel

Figure 6.2 An overview of the monit component It is implemented as a primitive Regis component. Components (primitive or composite) are viewed in terms of services they provide and/or require. A component's provided services allow other components to interact with it whereas its required services allow it to interact with others. The monit component provides two services (depicted by filled-in circles) and requires three external services to support those services (the empty circles).

108

Chapter 6 Distributed Monitoring Service The Darwin component interface specifies the parameters of the monit component, the set of services required and provided by it together with the types of these services. This component type has three parameters. The parameter n is the monitor's name, used as default source identifier for notified or internally triggered event reports generated by the monitor. It is also used to find a default GEM initialisation file at start-up time, as described later. The other two parameters specify the monitor's default detection window and stepping time interval, respectively. These parameters were described in chapter 4. The angle brackets enclose the types of the provided and required services. The first word of the type specification is the interaction mechanism used to implement the service. In this case port is used to receive messages of the specified type. Other interaction mechanisms may be used in Regis, as described later. GEM commands are received by monit through its command port which among other things allows new events to be declared and monitoring rules to be defined. These commands are null-terminated strings of characters which are interpreted by the monitor. String-formatted event reports are received from the monitored or other monit components through inrep. These events are filtered and/or composed as specified by GEM rules. The component may forward or generate new event reports through outrep. GEM rules may refer to temporal events. To detect such events, monit requires the services provided by a timer component. This requirement is represented by treq which is used to register with the timer by sending to it appropriate messages of type timer_reqT. When a rule is disabled or removed from the monitor's rule set, all its requests to the timer component must be cancelled. This is done by sending an appropriate cancellation message of type tmembase to the timer using tcancel.

6.2.2 Composite Components A timer component may provide its services to two or more monit components. Furthermore a monit component must be co-resident with the timer component which is satisfying its service requirements. This implies that there has to be at least one timer component on every machine on which a monit is running. Figure 6.3 shows the composite component monitor which is defined by declaring its sub-component instances (monit and timer) and the bindings between their service requirements and provisions, using the Darwin inst and bind statements respectively.

109


component timer(double r) { provide request ; cancel ; } component monitor(char *n, double d, double r, double tr) { provide command ; inrep ; monitor command require outrep outrep ; inst inrep request trequest t:timer( tr ); timer m:monit( n, d, r ); bind cancel monit tcancel inrep -- m.inrep; command -- m.command; m.treq -- t.request; m.tcancel -- t.cancel; m.outrep -- outrep; }

Figure 6.3 Darwin descriptions of the timer and monitor components The timer component has a single parameter, r, which determines the required resolution in seconds. Binding is only possible between required and provided services with compatible interaction mechanisms and types. For example the monit's treq is bound to the timer's request by the statement: m.treq -- t.request

Services which are provided internally and are required externally can be bound to interface service provisions. For example: command -- m.command;

Similarly, requirements which cannot be satisfied internally can be made visible at a higher level by binding them to interface requirements. For example: m.outrep -- outrep;

In a similar fashion multiple instances of the monitor component can be used to constructed a distributed event monitoring service, as shown later.

110


6.2.3 Communication In Regis, service provisions and requirements are realised by instances of interaction classes and remote references to these instances, respectively. Regis has a library of interaction template classes which currently includes port (one-to-one), entry (for bidirectional request-reply), and event for one to many communication. An interaction object of type port, mentioned before, is really a queue of typed messages and can remotely be accessed by an object of type portref which is bound to it. Figure 6.4 shows various methods used to place/remove messages in/from a port. portref: out(msg) outv(msgs,n) send(msg) sendv(msgs,n)

port: in(msg) inv(msgs, n) out(msg) outv(msgs,n) send(msg) sendv(msgs,n)

// // // // // //

receive message of type T in msg receive max. n messages of type T in msgs send msg of type T synchronously synchronously send n elements of msgs send msg of type T asynchronously asynchronously send n elements of msgs

Figure 6.4 Methods of port and portref communication mechanisms GEM event reports and commands are string-formatted messages and therefore communication objects of type portref and port can be used to send and receive them.

6.2.4 Component Placement At execution time, Darwin's hierarchical specification is flattened to produce a network of intercommunicating active objects. Each component instance can be placed on a different machine by using @ in Darwin's inst statement, where integer expression denotes a logical processor. For example: inst L : monitor ( "locmon", 60, 5, 0.5 ) @ 1; T : monitor ( "tempmon", 60, 5, 0.5 ) @ 2;

locates each instance of the monitor component on a separate machine. The integer machine identifiers are mapped to physical workstations by the Regis execution environment. By default, a component is mapped to the same location as its enclosing composite component

111

Chapter 6 Distributed Monitoring Service and the top level component is mapped to processor 0 — the processor location at which the program is initiated. A detailed mapping can be specified in a mapping file or it may be left to the environment to select the right workstations.

6.3 Downloading GEM Scripts to Monitors GEM scripts can be downloaded to a monitor through its command port or from a file. Figure 6.5 shows the kbmonitor component which allows GEM commands to be entered by the user through the standard input. kboard17 component is used to redirect the standard input to a port, in this case command. kboard command

monitor command outrep outrep

inrep inrep

kbmonitor Figure 6.5 The structure of kbmonitor component A monitor may also read the GEM scripts from a file. At start-up, by default, each monitor attempts to read an initialisation file (if it exists) whose file name has the form: .gem

where monitor-name is the first parameter passed to the monitor component when it is instantiated. GEM scripts may also be downloaded from arbitrary files by using the load command. For example, issuing the following command to a monitor, load "netload.gem"

causes the script in file netload.gem to be loaded into it.

6.4 Encoding and Decoding Event Reports To allow programmers to easily generate events and access or manipulate event attributes in their applications, GEM event declarations can be used to automatically generate appropriate C++ structures which provide the necessary methods for encoding and decoding each event to / from the format used by event monitors, as shown in figure 6.6. GEM event attribute types are mapped to similar C++ types.

1 7 Kboard is a composite component which uses Regis's standard Keyboard component.

112

Chapter 6 Distributed Monitoring Service Predefined C++ event base structure

GEM event declaration event tempchanged(double val)

struct evebase { char *sid; // source name time ts; // timestamp evebase(); evebase(char *s,time t); char *encode(); void decode(char *e); ... };

Generated C++ structure

struct tempchanged:public evebase{ double val; tempchanged(); tempchanged(char *s,time t,double val); char *encode(); void decode(char *e); ... };

Figure 6.6 Generating the required encoding and decoding routines The following Regis code segment shows how an event of type tempchanged can be reported by the "Sensor1" component through its outtemp port reference object (portref): ... // create an instance of event class tempchanged // with given attributes. now represents the current time tempchanged e ("Sensor1", now, 105.5); char *s=e.encode(); outtemp.sendv( s, size );

// send the string

...

The following lines show how the event can be received through a corresponding intemp port object (port): ... tempchanged e; intemp.inv(buff, size); e.decode(buff); ...

// e's attributes can be accessed in the application

6.5 Event Dissemination Clients of the event monitoring service may be interested in different classes of events. Figure 6.7 shows three different clients (controller, safety manager and display

113

Chapter 6 Distributed Monitoring Service components), who are interested in one or more classes of events generated by different monitors. Clients Event reports

controller tempalarm firealarm

firealarm tempalarm

safety manager indanger

dissem

monitor indanger

display any event

pressalarm

Figure 6.7 An example of event dissemination The dissem component is used to receive event reports from different monitors and to send them to those who are interested. In other words, it allows many-to-many communication by using interaction objects inrep (of type port — supporting many-to-one) and notify (of type eport18— supporting one-to-many). As shown in figure 6.8, each client receives the required event reports through one or more communication reference objects of type eportref, which are bound to the appropriate object of type eport. The figure also shows the main methods that these communication objects provide for this task.

eportref

t

enable(event class name) wait(event report, size) disable()

p

controller

f

dissem inrep notify

i

safety manager

d

display

announce(event report)

eport

Figure 6.8 Methods provided by eport and eportref communication objects

18 eport is very similar to Regis's event interaction class and is used to disseminate GEM event reports.

114

Chapter 6 Distributed Monitoring Service The outline Regis code for the clients are given in figure 6.9 which demonstrates how each one registers its interest in events and receive those events. controller: ... char buff[BUFF_SIZE]; t.enable("tempalarm"); f.enable("firealarm"); for (;;) { t.arm(1); f.arm(2); switch(sel()){ case 1: t.wait(buff, BUFF_SIZE); cool_temp(); break; case 2: f.wait(buff, BUFF_SIZE); unlock_doors(); sprinkler_on (); } }

safety manager: ... char buff[BUFF_SIZE]; i.enable("indanger"); indanger e; for (;;) { i.wait(buff, BUFF_SIZE); e.decode(buff); rescue(e.name); } display: ... char buff[BUFF_SIZE]; d.enable(); for (;;) { d.wait(buff, BUFF_SIZE); cout notify left ( l.name )\ } enable entering Gem Script for enable leaving

command locchanged

inrep

locmon

entered

outrep

locmon

left

Figure 6.13 An overview of the locmon's operation and its GEM script The following example sequence shows the events generated by the badgeman component and entered at locmon, during a particular period. locchanged "badgeman" [20:14:11.910 Fri 11/8/95] ("John","outside","R201") locchanged "badgeman" [20:18:11.330 Fri 11/8/95] ("Bob","outside","R253") locchanged "badgeman" [20:20:11.526 Fri 11/8/95] ("John","R201","R207") locchanged "badgeman" [20:23:11.352 Fri 11/8/95] ("Smith","outside","R101") locchanged "badgeman" [20:25:11.566 Fri 11/8/95] ("John","R207","R201") locchanged "badgeman" [20:27:11.781 Fri 11/8/95] ("Bob","R253","outside") locchanged "badgeman" [20:29:11.997 Fri 11/8/95] ("Smith","R101","R108") locchanged "badgeman" [20:31:12.211 Fri 11/8/95] ("John","R201","R209") locchanged "badgeman" [20:33:12.404 Fri 11/8/95] ("Tim","outside","R251") locchanged "badgeman" [20:35:11.591 Fri 11/8/95] ("Smith","R108","R101") locchanged "badgeman" [20:39:12.0 Fri 11/8/95]

("John","R209","outside")

locchanged "badgeman" [20:43:12.378 Fri 11/8/95] ("Bob","outside","R301") locchanged "badgeman" [20:47:12.778 Fri 11/8/95] ("Tim","R251","R255") locchanged "badgeman" [20:48:11.962 Fri 11/8/95] ("Smith","R101","outside")

119

Chapter 6 Distributed Monitoring Service From these 14 messages only 8 of them are significant in this example. The following sequence is the 8 corresponding messages which are generated by locmon and sent to safemon component: entered "locmon" [20:14:11.910 Fri 11/8/95] ("John") entered "locmon" [20:18:11.330 Fri 11/8/95] ("Bob") entered "locmon" [20:23:11.352 Fri 11/8/95] ("Smith") left

"locmon" [20:27:11.781 Fri 11/8/95] ("Bob")

entered "locmon" [20:33:12.404 Fri 11/8/95] ("Tim") left

"locmon" [20:39:12.0 Fri 11/8/95]

("John")

entered "locmon" [20:43:12.378 Fri 11/8/95] ("Bob") left

"locmon" [20:48:11.962 Fri 11/8/95] ("Smith")

(ii) The temperature monitor (tempmon) receives tempchanged event reports from the reactor system indicating changes in the reactor temperature. Such an event has an attribute (val) representing the temperature value. It generates a temp_alarm notification when the reactor's temperature exceeds a maximum threshold value of 100 degrees and does not fall below a minimum threshold value of 70 degrees within 5 minutes. This excess and drop in temperature is signalled by triggering exceeded and reduced events respectively. The temperature value may oscillate about either of the thresholds and consequently multiple such events may be triggered. The enable and disable actions in the rules are used to reduce the number of these events and force them to be triggered alternately. The interval between these thresholds is called the hysteresis interval and is used to ignore transient changes. Figure 6.14 shows the GEM script which is downloaded to tempmon to generate temp_alarm reports.

120


event event event event

tempchanged ( double val ) exceeded reduced temp_alarm

temperature value

tempalarm

rule upper {\ exceeded exceeded x:tempchanged when x.val > 100 \ ==> exceeded;\ reduced 100 enable lower;\ disable\ 70 } reduced rule lower {\ time x:tempchanged when x.val < 70\ event time ==> reduced;\ enable upper;\ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 disable\ notify trigger } rule excess_temp [5*min] {\ ({ x:exceeded ; ( y:exceeded + [5*min] )} ! z:reduced ) \ when ( ?y && ( x == y )) || ?z \ ==> notify temp_alarm\ } enable upper enable excess_temp Gem Script for tempmon

command tempchanged

inrep

outrep

temp_alarm

tempmon

Figure 6.14 An overview of the tempmon's operation and its GEM script The following sequence shows tempchanged events reported by the reactor component and the resulting exceeded, reduced and scheduled time events which are triggered internally within tempmon during almost the same time period as the previous sequence: tempchanged "reactor"

[20:14:11.477 Fri 11/8/95] (45.9)

tempchanged "reactor"

[20:16:11.711 Fri 11/8/95] (50.3)


[20:18:11.926 Fri 11/8/95] (67.4)


[20:20:11.178 Fri 11/8/95] (100.0)


[20:22:11.402 Fri 11/8/95] (110.8)

• exceeded

"tempmon"

[20:22:11.402 Fri 11/8/95]


[20:24:11.645 Fri 11/8/95] (95.7)


[20:26:11.854 Fri 11/8/95] (68.5)

121

Chapter 6 Distributed Monitoring Service • reduced

"tempmon"

[20:26:11.854 Fri 11/8/95]

• [20:27:11.402 Fri 11/8/95] tempchanged "reactor"

[20:28:12.73 Fri 11/8/95]


[20:30:12.317 Fri 11/8/95] (79.8)


[20:32:11.560 Fri 11/8/95] (81.7)


[20:34:11.764 Fri 11/8/95] (93.4)


[20:36:11.998 Fri 11/8/95] (101.2)

• exceeded

"tempmon"

(69.7)

[20:36:11.998 Fri 11/8/95]


[20:38:12.211 Fri 11/8/95] (74.3)


[20:40:12.455 Fri 11/8/95] (66.1)

• reduced

"tempmon"

[20:40:12.455 Fri 11/8/95]

• [20:41:11.998 Fri 11/8/95] tempchanged "reactor" • exceeded

"tempmon"

[20:42:11.648 Fri 11/8/95] (108.8) [20:42:11.648 Fri 11/8/95]


[20:44:11.872 Fri 11/8/95] (109.2)


[20:46:12.126 Fri 11/8/95] (98.7)

• [20:47:11.648 Fri 11/8/95] tempchanged "reactor"

[20:48:12.339 Fri 11/8/95] (106.1)

Internally triggered events and scheduled time events are marked with •. As you can see the first two excesses in temperature do not result in a temperature alarm because the temperature is reduced before the corresponding 5 minute time interval has elapsed. But after the last excess and at the corresponding scheduled time (20:47), the rule fires and the following event is notified by tempmon, which is sent to safemon component. temp_alarm "tempmon" [20:47:11.648 Fri 11/8/95]

(iii) The safety monitor (safemon) receives entered and left events from locmon, fire events from fire system and temp_alarm events from tempmon and notifies indanger events for all those who entered the building but failed to leave it 2 minutes after a fire or a temperature alarm. Figure 6.15 shows the GEM script downloaded to safemon to generate indanger event reports which have the name of the person in question as an attribute.

122


event event event event event

entered ( string name ) left ( string name ) fire temp_alarm indanger ( string name )

rule indanger [day] {\ ({ e:entered ; ( r:( fire | temp_alarm) + [2*min])^ } ! l:left )\ when ( ?l && ( e.name == l.name )) || ( ?r && ( @e < @r ))\ ==> notify indanger ( e.name )\ } enable indanger Gem Script for safemon entered command left inrep fire

outrep

indanger

safemon

temp_alarm

Figure 6.15 An overview of the safemon's operation and its GEM script In this example we assume that nobody stays in the building more than 24 hours, hence the detection window is set to one day. The detection window has to be adjusted appropriately if this assumption cannot be made. The following events are received by safemon from the other monitoring components: entered

"locmon"

[20:14:11.910 Fri 11/8/95] ("John")

entered

"locmon"

[20:18:11.330 Fri 11/8/95] ("Bob")

entered

"locmon"

[20:23:11.352 Fri 11/8/95] ("Smith")

left

"locmon"

[20:27:11.781 Fri 11/8/95] ("Bob")

entered

"locmon"

[20:33:12.404 Fri 11/8/95] ("Tim")

left

"locmon"

[20:39:12.0 Fri 11/8/95]

entered

"locmon"

[20:43:12.378 Fri 11/8/95] ("Bob")

("John")

temp_alarm "tempmon"

[20:47:11.648 Fri 11/8/95]

left

[20:48:11.962 Fri 11/8/95] ("Smith")

"locmon"

[20:49:11.647 Fri 11/8/95]

These events cover different possible cases: ° John enters and leaves before a temp_alarm, ° Tim enters before the alarm and does not leave,

123

Chapter 6 Distributed Monitoring Service ° Bob enters, leaves and enters again before the alarm and does not leave, ° Smith enters before the alarm and leaves before the 2 minute interval is over. On the occurrence / triggering of the scheduled time event (the last in the sequence), the following two events are notified by the monitor, indicating the individuals who are considered to be in danger: indanger "safemon" [20:49:11.647 Fri 11/8/95] ("Tim") indanger "safemon" [20:49:11.647 Fri 11/8/95] ("Bob")

6.7 Configuration of an Event Monitoring Service An event monitoring service can be configured at two levels: i) GEM scripts are interpreted by event monitors. New event class declarations can be added or removed when necessary. Also, individual rules within these monitors can be dynamically enabled, disabled, added and removed by invoking appropriate operations to change the composite events being recognised by particular event monitors. This allows the functionalities of a monitor to be changed at run time. In the previous example, in order to change the upper temperature limit for a tempalarm event, notified by tempmon monitor from 100 to 110 the following script can be downloaded to the monitor at runtime. ~rule upper

// removes the rule from the rule set

rule upper {\ x:tempchanged when x.val > 110 \ ==> exceeded;\ enable lower;\ disable\ }

Reallocation of monitoring functionalities between monitors is possible by removing GEM rules and relevant event classes from one monitor and downloading them to another one. Also clients of the event service can dynamically subscribe to event disseminator components to receive events which are already being generated. ii) The Darwin configuration facilities allow users to dynamically create new event monitor and disseminator components on distributed nodes within the system in order to extend and redimension the monitoring service. This allows us to adapt the monitoring service to the changes in the configuration of the monitored system.

124


6.8 Conclusion A dynamic distributed event monitoring service can be constructed from a set of generic monitoring components using the Regis/Darwin environment. Event monitors interpret GEM scripts and perform event filtering, composition and notification. Disseminators are used to send event reports to those who require them based on a subscription mechanism. Timer components are provided to support the specification of temporal events and constraints. As demonstrated in the case study, instances of these components and others can be distributed and configured onto physical nodes to perform the required monitoring functionalities on events close to where they occur. The significant features of the Regis/Darwin environment were described by showing the overall operation, structure and interaction of these components. The monitoring service can be configured at two different levels. The interpreted GEM scripts can be changed and replaced at run time allowing dynamic changes to the monitoring functionalities and responsibilities of event monitors. Also, the configuration capabilities of Regis/Darwin environments allows dynamic changes in the structure and placement of the monitoring components. These allow us to adapt the service to the changes in the monitored system and the monitoring requirements.

125



7.1 Introduction In this chapter, we will describe the logical structure of the event monitor and the implementation of the mechanism used for event detection in GEM and will present the outline of the event detection algorithm. A detailed example is used to illustrate the operation of the algorithm.

7.2 Overall Structure of an Event Monitor Figure 7.1 shows the overall logical structure of an event monitor. The event monitor consists of four major components: GEM interpreter, event class manager, rule manager and timer.

client

User Interface GEM Scripts Event Reports

GEM Interpreter

Rule Manager

Event Class Manager

Event Monitor

Event Reports

Timer

Figure 7.1 Structure of an event monitor

126


7.2.1 GEM Interpreter The functionalities of the GEM interpreter can be summarised as follows: • Receiving string-formatted event reports and GEM scripts and converting them to internal representations 19 . It reports all the syntactic errors by displaying appropriate messages. • Sending new event class declarations and the commands to remove or query the existing declarations to the event class manager. • Sending new rule definitions and the commands to remove, query, enable or disable the existing rules to the rule manager. • Performing other top level commands (e.g., for loading new GEM scripts from specific files, or stopping the monitor) The interpreter is implemented by using Flex++ and Bison++ tools. The specification of the grammar of GEM is contained in appendix B.

7.2.2 Event Class Manager The event class manager maintains the set of known event classes and provides facilities for addition, removal and querying of event class information. It can check the number and types of event attributes. Duplicate event class or attribute declarations are ignored and appropriate error messages are displayed.

7.2.3 Rule Manager The rule manager maintains the set of GEM rules which determines the main operations of the monitor. Its functionalities can be summarised as follows: • Receiving requests to add, remove, query, enable and disable rules. Duplicate rule definitions are ignored and references to undefined rules or undeclared event variables are detected and appropriate error messages are displayed. • Receiving new event records representing external events (arrival of new event reports) or those generated internally by rule trigger actions (not shown in the figure). For each such record, all the interested rules are visited and their event expressions are evaluated in an incremental fashion, as described later. Out of date records are ignored.

1 9 Note that the formats of incoming event reports and the top level event triggering commands are the

same, so the same interpreter is used for both.

127

Chapter 7 Monitor Implementation • Interacting with the event class manager to detect references to undeclared event classes or attributes and to check the number and types of attributes for each incoming event. • Subscribing with the timer component for user-specified time events and receiving the corresponding event records. Invalid time specifications result in appropriate error messages being displayed. • Generating event notifications with the same format as the incoming event reports. This is done through explicit notify actions in the RHS of the rules.

7.2.4 Timer The timer component receives requests for specific point, periodic or scheduled time events from the rule manager and generates the corresponding event records when the given actual times occur. The timer receives each request and, if it can be satisfied at some point in the future, it is inserted in an internal delay queue. When a requested time occurs, the timer uses the corresponding request to send an appropriate event record to the rule manager 20, calculate the next event occurrence and update the position of the request in the delay queue if future occurrences are possible. As a result, a single request may cause one or more time event records to be generated (e.g., for every[5*sec] event specification). This is repeated as long as new event records can be generated and the rule manager is still interested in the event, otherwise the request is discarded. The timer also updates the current time which is accessed in event specifications through the global variable now. Timer resolution can be set at configuration time and is bound by the resolution of the physical clock. To perform these tasks the timer relies on xntp (Network Time Protocol) to synchronise the clocks on different machines. The necessary C++ classes are provided to support the specification and manipulation of time intervals and both partial and exact time points.

7.3 Event Detection — Mechanism & Algorithm In this section, we shall concentrate on the mechanism and algorithm used for composite event detection.

2 0 Note that the timer component has been designed so that it can serve several event monitors and

therefore rule managers.

128


7.3.1 Event Evaluation Tree We have adopted a tree-based mechanism for composite event detection which enables us to easily deal with out-of-order event arrivals. event evaluation tree

guard evaluation tree

event node type guard

event records

history ...

•••

lower level event records

Figure 7.2 The structure of the event evaluation tree When a rule is interpreted, the event expression in its LHS is used to create an event evaluation tree, as shown in figure 7.2. This is a C++ object whose structure closely matches that of GEM's event expressions.

7.3.2 Event Evaluation Nodes The leaves and internal nodes of the event evaluation tree represent guarded primitive and composite event expressions, respectively. Each node is a C++ structure whose data members include the following: • type identifies the type of the node (primitive, ; , &, |, +, etc.) • guard represents the predicate to be evaluated when the event is detected • history an ordered history of event records • others e.g., values for time point or periodic events, pointers to parent node (if any) and one or more child nodes as shown in figure 7.3.

129


Event Evaluation Nodes

Internal nodes ;

&

+

left right left right

Leaf nodes at every user-declared

|

child

!

left right left middle right

Figure 7.3 Event Evaluation Nodes Each node also provides the following main methods: eval_event(), cancel_event(), consume_event(), and istrue_guard(). These are described below. Leaf nodes represent primitive event classes which may be user-declared (basic), time period (every) or time point (at) events. Every time a primitive event e is triggered for a rule r, if e's timestamp is within r's detection window, every interested leaf node of the event evaluation tree associated with r is informed of the occurrence. If e satisfies the node's guard (i.e. istrue_guard(e) returns true), a corresponding event record is entered in its history, and the parent node (if any) is informed of the existence of the new record by calling its eval_event() method and passing a pointer to the new record as a parameter. Parent / internal nodes represent composite events. Depending on its semantics, each parent in turn uses the new record from its child, its associated guard, and possibly the records in the histories of its other children (if any) to generate as many new event records as possible. The parent may generate and add to its history several records corresponding to possible combinations or matching of events. The process continues until the rule fires or no new records can be generated.

7.3.3 Guard Evaluation Tree A guard evaluation tree is associated with each node of the event evaluation tree to allow incremental evaluation of the guards and early filtering of events to reduce unnecessary computations. When a new event record is created for a node, the guard associated with that node is evaluated to determine if it can be inserted in the node's history. Guards have no side effects on event records and are implemented in the usual tree structured fashion with operands as leaves and operators as internal nodes. The interpreter automatically simplifies the guard(s) and consequently the structure of the tree(s) by performing as much of the evaluation as possible when the corresponding rule definition is entered. For example, the guard: "...when ((1 + 3) > @x)"

130

Chapter 7 Monitor Implementation is automatically changed to, "... when (4 > @x)"

7.3.4 Event Records In contrast to event evaluation nodes which represent the classes of event occurrences, event records represent instances of such occurrences. The overall structure of each event record closely matches that of its corresponding node as shown in figure 7.4, except for records for ‘!’ and ‘|’ nodes.

Internal records

Event Records

;

&

+

left right left right

child

Leaf records at every user-declared

|

!

child

left right

Figure 7.4 Event Records Leaf event records represent primitive event occurrences. In order to reduce duplications, and rather than inserting a copy of every primitive event information directly in every interested leaf node's history, an event record pointing to it is inserted and a reference count is maintained. More than one leaf event record may represent (and point to) the same primitive event instance. These records may belong to the event nodes of the same rule or different rules. Internal event records also use pointers and reference counts rather than copies and may refer to one or more other records at lower level nodes. As a result, each record can be viewed as representing an event instance tree, the leaves of which refer to the participating primitive event instances.

Event Record Type

Event Timestamp

Comment

leaf |

ct

; !

rt

lt < rt

&

rt

lt = (now - R's detection window)) && (R != r)) { for ( each leaf node n of rule's event evaluation tree interested in e && while R not fired ) { call n's eval_event (e); // "inject" e to leaf node } } } }

Every event evaluation node provides three main methods, eval_event (), cancel_event () and consume_event (). What follows is a pseudo-code description of these methods for various node types.

142

Appendix A — Event Detection Algorithm

leaf node eval_event ( e ) { ne = new record referring to e if ( not istrue_guard ( ne )) delete ne else { increment e's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event ( ne ) else fire ( ne ) } } cancel_event ( e ) { set ce to the oldest event record in this nodes history whose time is equal to e. for ( every ce whose timestamp == e's timestamp ) if ( e is ce's child ) { if (( this node has a parent ) && ( e's reference count > 0 )) call parents cancel_event (ce) mark ce as consumed return;

// only one such event record

} } } /* end of cancel_event() method of a leaf node */ consume_event ( e ) { // inform any interested parent records and delete the record if ( this node has a parent ) && ( e's reference count > 0 ) call parent's cancel_event(e) mark e as consumed ce = e's child // Other nodes of this rule may be interested in the event if ( ce's reference count > 1 ) { for ( all other nodes n of this rule which have seen ce ) call n's cancel_event ( ce ); } } /* end of consume_event() method of a leaf node */

143

Appendix A — Event Detection Algorithm ';' node eval_event ( e ) { if ( e belongs to left child ) { er = oldest event record in right child's history while ( e is not consumed && er exists ) { if ( e's timestamp < er's timestamp ) { ne = new event record with children e and er if ( not istrue_guard ( ne )) delete ne else { increment e and er's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } } er = next event record in right child's history } } else if ( e belongs to the right child ) { el = oldest event record in left child's history while ( e is not consumed && el exists ) { if ( el's timestamp < e timestamp ) { ne = new event record with children el and e if ( not istrue_guard ( ne )) delete ne else { increment el and e's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } } el = next event record in left child's history } } } /* end of eval_event() method of a ‘;’ node */

144

Appendix A — Event Detection Algorithm cancel_event ( e ) {

// ‘;’ node

ce = oldest event record in this node's history while ( ce exists && ( e's reference count > 0 )) { if ( e is ce's left or right child ) { decrement reference counts of ce's left & right children if (( this node has a parent ) && ( ce's reference count > 0 )) call parent's cancel_event( ce ) mark ce as consumed } ce = next event record in this node's history } } consume_event ( e ) {

// ‘;’ node

el = e's left child er = e's right child decrement reference counts of el and er if ( this node has a parent ) && ( e's reference count > 0 ) call parent's cancel_event(e) mark e as consumed call left child's consume_event (el) call right child's consume_event (er) }

'&' node eval_event ( e ) {

// ‘&’ node

if ( e belongs to left child ) { er = oldest event record in right child's history while ( e is not consumed && er exists ) { ne = new event record with children e and er if ( not istrue_guard ( ne )) delete ne else { increment e and er's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } er = next event record in right child's history

145

Appendix A — Event Detection Algorithm } } else if ( e belongs to the right child ) { el = oldest event record in left child's history while ( e is not consumed && el exists ) { ne = new event record with children el and e if ( not istrue_guard ( ne )) delete ne else { increment el and e's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } el = next event record in left child's history } } } cancel_event ( e )

{ ... // same as ';' node }

consume_event ( e ) {

// ‘&’ node

el = e's left child er = e's right child decrement reference counts of el and er if ( this node has a parent ) && ( e's reference count > 0 ) call parent's cancel_event(e) mark e as consumed c1 = the child node to which el belongs call c1's consume_event (el) c2 = the child node to which er belongs call c2's consume_event (er) }

'|' node eval_event ( e ) { ne = new event record with child e if ( not istrue_guard ( ne )) delete ne else { increment e's reference count insert ne in history in its correct place

146

Appendix A — Event Detection Algorithm if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } } cancel_event ( e ) {

// ‘|’ node

ce = oldest event record in this node's history while ( ce exists && ( e's reference count > 0 )) { if ( e is ce's child ) { /* e must be removed */ decrement e's reference count if (( this node has a parent ) && ( ce's reference count > 0 )) call parent's cancel_event( ce ) mark ce as consumed return;

// only one such event record exits

} ce = next event record in this node's history } } consume_event (e) {

// ‘|’ node

ce = e's child decrement ce's reference count if (this node has a parent && e's reference count > 0) call parent node's cancel_event(e); mark e as consumed n = the child node to which ce belongs // left or right child call n's consume_event(ce); }

'!' node eval_event ( e ) { if ( e belongs to left child ) { em = oldest event record in middle child's history

// event records of right child older than e cannot // affect the evaluation er = oldest event record in right child's history whose timestamp is greater than e's while ( e is not consumed && er exists && e's timestamp < er's timestamp ) {

147

Appendix A — Event Detection Algorithm en = new event record with children e and er if ( not istrue_guard ( ne )) delete ne else { // check if any of middle child's records invalidates ne

// event records of middle child older than e cannot // affect the evaluation em = event record in middle child's history whose timestamp is greater than e's timestamp m = em; while (( m exists ) && ( ne not invalidated ) && ( m's timestamp < er's timestamp ) { // m cancels ne if the guard evaluates to true n = new event record with children e and m if ( not istrue_guard ( n )) // ne is invalidated delete ne m = next event record in middle child's history } if ( ne still exists ) {

// ne is not invalidated

increment e and er's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } } er = next event record in right child's history } } else if ( e belongs to middle child ) { // cancellation event for ( every event record ce in this node's history ) { if ( e's timestamp is between timestamps of ce's children ) { // e cancels ce if the guard evaluates to true n = new event record with children ce's left child and e if ( istrue_guard ( n )) { // ce is invalidated decrement reference counts of ce's left & right children if ( this node has a parent ) call parent's cancel_child ( ce ) mark ce as consumed

148

Appendix A — Event Detection Algorithm } } } } else if ( e belongs to right child ) { el = oldest event record in left child's history em = oldest event record in middle child's history while ( e is not consumed && el exists && el's timestamp < e's timestamp ) { en = new event record with children el and e if ( not istrue_guard ( ne )) delete ne else { // check if any of middle child's records invalidates ne // middle child's records older than el cannot // affect the evaluation em = event record in middle child's history whose timestamp is greater than el's timestamp m = em; while (( m exists ) && ( ne not invalidated ) && ( m's timestamp < e's timestamp ) { // m cancels ne if the guard evaluates to true n = new event record with children el and m if ( not istrue_guard ( n )) // ne is invalidated delete ne m = next event record in middle child's history } if ( ne still exists ) {

// ne is not invalidated

increment el and e's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } } el = next event record in left child's history } } } // end of eval_event() method of a ‘!’ node cancel_event ( e ) {

// ‘!’ node

if ( e belongs to the left child ) {

149

Appendix A — Event Detection Algorithm ce = oldest event record in this node's history while ( ce exists && e's reference count > 0 ) { if ( e is ce's left child ) { decrement reference count of ce's left and right children if ( this node has a parent) && ce's reference count >0 )) calls parent's cancel_event(ce) mark ce as consumed } ce = next event record in the history } } else if ( e belongs to the middle child ) { // must generate new record which may have been cancelled by e p = first event record in the right child's history whose timestamp > e's timestamp if (p does not exist) return for ( each event record (le) in left child's event history whose timestamp is less than e's timestamp ) { for ( each event record (re) in right child's event history starting from p and while le is not consumed by rule ) { ne = new event record referring to le and re; if ( istrue_guard ( ne ) && ( no event record in this node's history refers to both le and re )) { increment reference counts of le and re insert ne in this node's history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) if ( p was consumed by the last firing ) { p = next event record in the right's history if ( p does not exist ) return } } else delete ne } } } else if ( e belongs to the right child ) { ce = oldest event record in this node's history while ( ce exists && e's reference count > 0 ) {

150

Appendix A — Event Detection Algorithm if ( e is ce's right child ) { decrement reference count of ce's left and right children if ( this node has a parent) && ce's reference count >0 )) calls parent's cancel_event(ce) mark ce as consumed } ce = next event record in the history } } } //

'!' node's cancel_event()

consume_event ( e ) // same as ';' node's consume_event

'+' node eval_event ( e ) { if ( e is a time event ) {

// a scheduled time occurred

for ( every event record ce in child's history ) { if (( ce's reference count == 0 ) && (( ce's time + scheduled time period ) == e's time )) { ne = new record with children ce if ( not istrue_guard ( ne )) delete ne else { increment ce's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } return // only one such record exists for every time event } } } else {

// user-declared event

t = e's timestamp + scheduled time period if ( t > now )

//

time period has not passed

send a request to timer for scheduled time event for time t else { ne = new record with child e if ( not istrue_guard ( ne )) delete ne else {

151

Appendix A — Event Detection Algorithm increment e's reference count insert ne in history in its correct place if ( this node has a parent ) call parent's eval_event( ne ) else fire ( ne ) } } } } //

'+' node's eval_event()

cancel_event ( e ) // similar to '|' node's cancel_event() consume_event (e)

// same as '|' node's consume_event()

Appendix B — GEM Grammar This appendix contains the grammar of GEM specified using bison++ tool. %union { long lval; char cval; char sval[128]; double dval; struct gem_tm *gtm;

// time points (possibly partial)

struct gem_itm *gitm;

// time periods

struct enode *ev_tree;

// event tree

struct gnode *gu_tree;

// guard tree

struct attrtyp_entry *attrdeclp;

// event attribute element

} %token AT TRIGGER EVENT RULE EVERY FORWARD %token NOTIFY STOP DISABLE ENABLE WHEN LOAD

// gem’s basic types %token CHAR LONG DOUBLE STRING ITIME TIME %token ID

/* system attribute tokens */ %token NOW SELF

/* current time & name of the monitor */

/* gem's basic values */ %token LONGconst %token CHARconst

152

Appendix C — GEM Error Messages %token DOUBLEconst %token STRliteral

/* Time related Tokens */ %token JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC %token MSEC SEC MIN HOUR DAY WEEK MONTH YEAR %token MON TUE WED THU FRI SAT SUN

%token OROR ANDAND

/* ||, && */

%token BIGARROW STARTAT /* ::, ==>, |@ */ %token '!' '+' '-' '/' '%' '*' '@' '?' '^' %token EQ NE '' LE GE

%type type_spec weekday parameter_list opt_keep %type event_id opt_event_id source_id opt_source_id %type event_var attribute_name %type time_period opt_time_period time_period_expr period %type time_point opt_time_point time_point_expr %type time_of_day date %type rule_id opt_rule_id %type system_attributes

event_attributes

%type logical_expr expression constant %type attr_decl %type comp_eve_exp guarded_prim_eve_exp prim_eve_exp %type prim_eve %type day_val month_val year_val

/* operator precedences - lowest to highest */ %left OROR ANDAND '|' '&' ';' '^' %left EQ NE %left '' LE GE %left '+' '-' %left '*' '/' '%' %left '!' '~'

WHEN UMINUS UPLUS

%start program %%

/*beginning of rules section */

program : /* empty */ | gem_statements '\n' ; gem_statements:

153

Appendix C — GEM Error Messages /* empty */ | declarations | rule_definitions | top_level_actions ; declarations: EVENT event_id opt_attr_decls; event_id: ID; opt_attr_decls: /* empty */ | '(' attr_decls ')' ; attr_decls : attr_decl | attr_decls ',' attr_decl ; attr_decl : type_spec attribute_name; type_spec: LONG | CHAR | DOUBLE | STRING | TIME | ITIME ; attribute_name : ID; rule_id: ID; rule_definitions: RULE rule_id opt_time_period '{' rule_body '}'; opt_event_id: /* empty */ | event_id ; opt_rule_id: /* empty */ | rule_id ; top_level_actions: event_id opt_source_id opt_time_point opt_parameter_list | ENABLE '*' | ENABLE rule_id | DISABLE '*' | DISABLE rule_id

154

Appendix C — GEM Error Messages | STOP | LOAD STRliteral | '?' queries | '~' removals ; queries: opt_event_id | EVENT opt_event_id | RULE opt_rule_id | ENABLE opt_rule_id | DISABLE opt_rule_id | NOTIFY opt_event_id | TRIGGER opt_event_id ; removals: | event_id | EVENT event_id | RULE rule_id ; opt_source_id: /* empty */ | source_id ; source_id : STRliteral ; opt_time_period: /*empty */ | time_period ; time_period : '[' time_period_expr ']' time_period_expr : period | LONGconst '*' period | time_period_expr '+' time_period_expr | '(' time_period_expr ')' ; period : MSEC | SEC | MIN | HOUR | DAY | WEEK | MONTH | YEAR; opt_time_point : /* empty */ | time_point ; time_point : '[' time_point_expr ']'

155

Appendix C — GEM Error Messages time_point_expr : time_of_day | time_of_day weekday | time_of_day date | time_of_day weekday date | weekday | weekday date | date ; time_of_day : LONGconst ':' LONGconst |

LONGconst ':' LONGconst ':' LONGconst

|

LONGconst ':' LONGconst ':' DOUBLEconst

; weekday : MON | TUE | WED | THU | FRI | SAT | SUN; date : day_val '/' month_val '/' year_val; day_val : '-' | LONGconst ; month_val : '-' | LONGconst | JAN | FEB | MAR | APR | MAY | JUN | JUL | AUG | SEP | OCT | NOV | DEC ; year_val : '-' | LONGconst ; rule_body : comp_eve_exp BIGARROW action_list ; comp_eve_exp : guarded_prim_eve_exp opt_keep | comp_eve_exp ';' comp_eve_exp | comp_eve_exp '|' comp_eve_exp | comp_eve_exp '&' comp_eve_exp | comp_eve_exp '+' time_period | '{' comp_eve_exp ';' comp_eve_exp '}' '!' comp_eve_exp ;

guarded_prim_eve_exp : prim_eve_exp | prim_eve_exp WHEN logical_expr ; opt_keep : /* empty */ | '^'; event_var: ID; prim_eve_exp :

156

Appendix C — GEM Error Messages prim_eve | event_var ':' prim_eve | '(' comp_eve_exp ')' | event_var ':' '(' comp_eve_exp ')' ; prim_eve : event_id | time_point | AT time_point | EVERY time_period | '*' ; action_list :

action | action_list ';' action;

action: FORWARD event_var | NOTIFY event_id | event_id opt_parameter_list | ENABLE opt_rule_id | DISABLE opt_rule_id ; opt_parameter_list : /* empty */ | '(' parameter_list ')' ; parameter_list : logical_expr | parameter_list ',' logical_expr ; logical_expr : expression | logical_expr OROR logical_expr | logical_expr ANDAND logical_expr | '!' logical_expr ; expression : '(' logical_expr ')' | expression '+' expression | expression '-' expression | expression '/' expression | expression '%' expression | expression '*' expression | expression EQ expression

157

Appendix C — GEM Error Messages | expression '' expression | expression GE expression | expression LE expression | expression NE expression | '-' expression | '+' expression | system_attributes | event_attributes | constant ; constant : LONGconst | DOUBLEconst | CHARconst | STRliteral | time_point | time_period ; system_attributes : NOW | SELF; event_attributes : event_var |

'?' event_var

|

'@' event_var

|

STARTAT event_var

|

'$' event_var

|

event_var '.' attribute_name

;

Appendix C — GEM Error Messages The following error messages may be displayed by the monitor if erroneous GEM scripts are entered. We shall give a brief explanation of those which are not self explanatory:

Error code

Displayed message

CONST_VAL_EXPECTED

Constant value expected

ATTR_TYPE_MISMATCH

Attribute type mismatch

TOO_FEW_ATTRS

Too few attributes

158

Comment long, double, char, string, time or itime expected.

Appendix C — GEM Error Messages

TOO_MANY_ATTRS

Too many attributes

DUPLICATE_EVENT_DECL

Duplicate event declaration

DUPLICATE_RULE_DEF

Duplicate rule definition

UNDECLARED_EVENT

Undeclared event

UNDEFINED_RULE

Undefined rule

EXACT_TIME_EXPECTED

Exact time expected

INVALID_TIME_POINT

Invalid time point

DUPLICATE_ATTR_NAME

Duplicate attribute name

UNDECLARED_EVENT_VAR

Undeclared event variable

Event variables must be declared in the scope of a rule before it is used.

BASIC_EVENT_EXPECTED

Basic event expected

The event is not a userdeclared event.

EVENT_VAR_EXPECTED

Event variable expected

DUPLICATE_VAR_DECL

Duplicate variable declarat'n

Event variable previously declared in this rule.

INVALID_SCOPE_FOR_EXP

Invalid scope for expression

Reference made to nonexistent / invalid event expression.

UNACCESSIBLE_ATTR

Can't access event attribute

Only timestamp and source id. of events specified by the wild-card character (*) can be accessed.

UNDECLARED_ATTR

Undeclared attribute

No such attribute exists in the event attribute declaration.

NEGATIVE_EVENT_USED

Can't access negative event

Attempt to reference cancellation event e3 in {e1;e2}!e3 expression.

OPTIONAL_EVENT_USED

Can't access optional event

Attempt to reference e1 or e2 in (e1|e2) expression.

REMOVING_USED_EVENT

Can't remove referenced event All rules depending on an event class must be removed before the event declaration can be removed.

159

Timestamp of incoming event reports is not exact

References

References

Bacon, J., Bates, J., Richard, H. & Moody, K. (1995) Using Events to Build Distributed Applications, In Proceedings of the 2nd International Workshop on Services in Distributed and Network Environments, Whistler, British Colombia, pp. 148—155. Bates, P. (1995) Debugging Heterogeneous Distributed Systems Using Event-Based Models of Behaviour, ACM Transactions on Computer Systems, Vol. 13, No. 1, pp. 1—31. Bemmerl, T. & Bode, A. (1991) An Integrated Environment for Programming Distributed Memory Multiprocessors, In Proceedings of the 2nd European Dist. Memory Comp. Conf., pp. 130—142. Bemmerl, T., Lindhof, R. & Treml, T. (1990) The Distributed Monitor System of TOPSYS, In Proceedings of the CONPAR 1990 - VAPP IV, Springer-Verlag, pp. 756—765. Birman, K. P., Joseph, T. A., Kane, K. & Schmuck, F. (1988) ISIS — A Distributed Programming Environment, User’s Guide and Reference Manual, second edition , Department of Computer Science, Cornell University. Brownston, L., Farrell, R., Kant, E. & Martin, N. (1985) Programming Expert Systems in OPS5: An Introduction to Rule-Based Programming: Addison Wesley. Chakravarthy, S., Krishnaprasad, V., Anwar, E. & Kim, S.-K. (1993) Anatomy of a Composite Event Detector, Technical Report UF-CIS-TR-93-039, University of Florida, Department of Computer and Information Sciences. Chakravarthy, S. & Mishra, D. (1993) Snoop: An Expressive Event Specification Language For Active Databases, Technical report UF-CIS-TR-93-007, University of Florida, Department of Computer and Information Sciences. Chandy, K. M. & Lamport, L. (1985) Distributed Snapshots: Determining Global States of Distributed Systems, ACM Trans. on Comp. Syst., Vol. 3, No. 7, pp. 63—75. Christian, F. (1989) Probabilistic Clock Synchronisation, Distributed Computing, Vol. 3, pp. 146—158.

160

References Dasgupta, P. (1986) A Probe-based Monitoring Scheme for an Object-oriented, Distributed Operating System, ACM Proceedings of the Conference on Object Oriented Programming Systems, Languages and Applications, pp. 57—66. Dayal, U. (1988) Active Database Management Systems, In Proceedings of the 3rd International Conference on Data and Knowledge Bases, Jerusalem, pp. 150—169. Debski, A. & Janas, E. (1994) SysMan Project (7026) — Generalised Monitoring Service Specification, SysMan Deliverable ATM.016.1994, ATM Computer. Duda, A., Harrus, H., Haddad, Y. & Bernard, G. (1987) Estimating Global Time in Distributed Systems, In Proceedings of the 7th ICDCS, Berlin, IEEE, pp. 299—306. Feldkuhn, L. & Erickson, J. (1989) Event Management as a Common Functional Area of Open Systems Management, In Proceedings of the IFIP Sym. on Integrated Network Management, Boston, North-Holland, pp. 365—376. Fidge, C. J. (1988) Partial Orders for Parallel Debugging, In Proceedings of the Workshop on Parallel and Distributed Debugging, ACM, pp. 183—194. Gatziu, S. & Dittrich, K. R. (1994) Detecting Composite Events in Active Databases using Petri Nets, In Proceedings of the 14th International Workshop on Research Issues in Data Engineering: Active Database Systems, Houston, Texas. Gehani, N., Jagadish, H. V. & Shmueli, O. (1992) Composite Event Specification in an Active Databases: Model & Implementation, In Proceedings of the 18th VLDB Conference, Vancouver, British Colombia, Canada. Haban, D. & Wybranietz, D. (1990) A Hybrid Monitor for Behaviour and Performance Analysis of Distributed Systems, IEEE Transactions on Software Engineering, Vol. 16, No. 2. Harter, P. K., Heimbigner, D. M. & King, R. (1985) IDD: An Interactive Distributed Debuggers, In Proceedings of the 5th Int. Conf. on Distributed Computing System, Denver, IEEE, pp. 498—506. Hofmann, R., Klar, R., Mohr, B., Quick, A. & Siegle, M. (1992) Distributed Performance Monitoring: Methods, Tools, and Applications, University of ErlangenNurnberg, IMMD VII, Martensstrabe 3, D-8520 Erlangen, Germany. Hoft, M. (1994) Managination — A Graphical User Interface Toolkit for Management Applications, In Proceedings of the Fifth IFIP/IEEE International Workshop on Distributed Systems: Operations & Management, Toulouse, France. Holden, D. (1989) Predictive Languages for Management, In Proceedings of the IFIP Sym. on Integrated Network Management, Boston, North-Holland, pp. 585—596. Holden, D. & Langsford, A. (1989) MANDIS: Management of Distributed Systems, Lecture Notes in Computer Science, Progress in Distributed Operating Systems and Distributed Systems Management, Vol. 433, pp. 162—173. Holden, D. B. (1991) A Tutorial to Writing Programs in SESL, Internal Report DMP/55, Sys. & S/W Eng. Grp., AEA Industrial Technology, Harwell. 161

References ISO (1990a) ISO/IEC DIS 10164-2 Information Technology - Open Systems Interconnection - Systems Management Part 2: State Management Function. ISO (1990b) ISO/IEC DIS 10164-4 Information Technology - Open Systems Interconnection - Systems Management Part 4: Alarm Reporting Function. ISO (1990c) ISO/IEC DIS 10164-5 Information Technology - Open Systems Interconnection - Systems Management Part 5: Event Report Management Function. ISO (1992a) ISO/IEC DIS 10164-11 Information Technology - Open Systems Interconnection - Systems Management Part 11: Workload Monitoring Function. ISO (1992b) ISO/IEC DIS 10165-4 Information Technology - Open Systems Interconnection - Structure of Management Information: Guidelines for the Definition of Managed Objects. Jakobson, G. & Weissman, M. (1995) Real-time Telecommunication Network Management: Extending Event Correlation with Temporal Constraints, In Proceedings of the Fourth Symposium on Integrated Network Management, Santa Barbara, California, USA, Chapman & Hall, pp. 290—301. Joyce, J., Lomow, G., Slind, K. & Unger, B. (1987) Monitoring Distributed Systems, ACM Transactions on Computer Systems, Vol. 5, No. 2, pp. 121—150. Klar, R., Quick, A. & Soetz, F. (1992) Tools for a Model-driven Instrumentation for Monitoring, In Proceedings of the 5th Int. Conf. on Modelling Techniques and Tools for Computer Performance Evaluation, Torino, Italy, Elsevier Science Publisher B.V., pp. 165—180. Klinger, S., Yemini, S., Yemini, Y., Ohsie, D. & Stolfo, S. (1995) A Coding Approach to Event Correlation, In Proceedings of the Fourth Symposium on Integrated Network Management, Santa Barbara, California, USA, Chapman & Hall, pp. 266—277. LaBarre, L. (1991) Management By Exception: OSI Event Generation, Reporting, and Logging, In Proceedings of the 2nd IFIP Symposium on Integrated Network Management, Washington. Lamport, L. (1978) Time, Clocks and the Ordering of Events in Distributed Systems, CACM, Vol. 21, pp. 558—564. LeBlanc, R. J. & Robbins, A. D. (1985) Event-Driven Monitoring of Distributed Programs, In Proceedings of the 5th International Conference on Distributed Computing Systems, pp. 515—522. Lumpp, J. E., Jr., Casavant, T. L., Seigle, H. J. & Marinescu, D. C. (1990) Specification and Identification of Events for Debugging and Performance Monitoring of Distributed Multiprocessor Systems, In Proceedings of the 10th International Conference on Distributed Systems, pp. 476—483. Magee, J., Kramer, J., Sloman, M. (1989) Constructing Distributed Systems in Conic, IEEE Transactions on Software Engineering, Vol. 15, No. 6, pp. 663—675.

162

References Magee, J., Dulay, N. & Kramer, J. (1994) REGIS: A Constructive Development Environment for Distributed Programs, IOP/IEE/BCS Distributed Systems Engineering, Vol. 1, No. 5, pp. 304—312. Malony, A. D., Reed, D. A., Arendt, J. W., Grabas, D., Aydt, R. A. & Totty, B. K. (1989) An Integrated Performance Data Collection Analysis and Visualisation System, In Proceedings of the 4th Conference on Hypercube Concurrent Computers and Applications, pp. 229—236. Manning, C. R. (1987) Traveler: The Apiary Observatory, In Proceedings of the European Conference on Object Oriented Programming, pp. 97—105. Mansouri-Samani, M. & Sloman, M. (1993) Monitoring Distributed Systems, IEEE Network, Vol. 7, No. 6, pp. 20—30. Mansouri-Samani, M. & Sloman, M. (1994) Monitoring Distributed Systems, In M. Sloman (Ed.), Network and Distributed Systems Management, Addison Wesley, pp. 303—347. Mansouri-Samani, M. & Sloman, M. S. (1995) GEM — A Generalised Event Monitoring Language for Distributed Systems, Technical Report DoC 95/8, Imperial College, London University. Marinescu, D., C., Lump, J. E., Casavant, T. L. & Siegel, H. J. (1990) Models for Monitoring and Debugging Tools for Parallel and Distributed Software, Journal of Parallel Distributed Computing, Vol. 9, No. 2, pp. 171—184. Marriott, D. A., Sloman, M. S. & Yialelis, N. (1995) Management Policy Service for Distributed Systems, Research Report DoC 95/10, Department of Computing, Imperial College. Marzullo, K., Cooper, R., Wood, M. D. & Birman, K. P. (1991) Tools for Distributed Application Management, IEEE Computer, Vol. 24, No. 8, pp. 42—51. Marzullo, K. & Wood, M. D. (1992) Tools for Distributed Application Management, Department of Computer Science, Cornel University. McDowell, C. E. & Helmbold, D. P. (1989) Debugging Concurrent Programs, ACM Computing Surveys, Vol. 21, No. 4, pp. 593—622. Mohr, B. (1990) Performance Evaluation of Parallel Programs in Parallel and Distributed System, In Proceedings of the CONPAR 1990 - VAPP IV, Springer-Verlag, pp. 176—187. Mohr, B. (1991) SIMPLE: a Performance Evaluation Tool Environment for Parallel and Distributed Systems, In Proceedings of the 2nd European Distributed Memory Computing Conference, Munich, Germany, Springer Berlin, LNCS 487, pp. 80—89. Ng, K. & Kramer, J. (1995) Automated Support for Distributed Software Design, In Proceedings of the 7th International Workshop on Computer-Aided Software Engineering, Toronto, Canada.

163

References Nygate, Y. A. (1995) Event Correlation using Rule and Object Based Techniques, In Proceedings of the Fourth Symposium on Integrated Network Management, Santa Barbara, California, USA, Chapman & Hall, pp. 290—301. OMG (1995) CORBAservices: Common Object Services Specification, Revised Edition, OMG Document Number 95-3-31. Schwarz, R. & Mattern, F. (1992) Detecting Causal Relationships in Distributed Computations: In Search of the Holy Grail, In Proceedings of the Advanced Course on Distributed Systems, Estoril, Portugal. Shim, Y. C. & Ramamoorthy, C. V. (1990) Monitoring and Control of Distributed Systems, In Proceedings of the First International Conference on Systems Integration, Morristown, NJ, IEEE Computing Press, pp. 672—681. Sloman, M. (1987) Distributed Systems Management, Research Report DOC 87/6, Imperial College. Sloman, M. (1994) Policy Driven Management for Distributed Systems, Plenum Press Journal of Network and Systems Management, Vol. 2, No. 4, pp. 333—360. Snodgrass, R. (1988) A relational Approach to Monitoring Complex Systems, ACM Trans. on Computer Systems, Vol. 6, No. 2, pp. 157—196. Spezialetti, M. & Bernberg, S. (1995) EVEREST: An Event Recognition Testbed, In Proceedings of the 15th International Conference on Distributed Computing Systems, Vancouver, IEEE Computing Society, pp. 377—385. Spezialetti, M. & Kearns, J. P. (1988) A General Approach to Recognising Event Occurrences in Distributed Computations, In Proceedings of the 8th International Conference on Distributed Computing Systems, pp. 300—307. Spezialetti, M. & Kearns, J. P. (1989) Simultaneous Regions: A Framework for Consistent Monitoring of Distributed Systems, In Proceedings of the 9th Intl. Conference on Distributed Computing Systems, pp. 61—68. Stone, J. M. (1988) A Graphical Representation of Concurrent Processes, Workshop on Parallel and Distributed Debugging, Jan 1989, SIGPLAN Notices, Vol. 24, No. 1, pp. 226—235. Tsai, J. J.-P., Fang, K.-Y. & Chen, H.-Y. (1990) A Non-intvasive Architecture to Monitor Real-Time Distributed Systems, IEEE Computer, Vol. 23, No. 3, pp. 11—23. Van Riek, M. & Tourancheau, B. (1991) A General Approach to the Monitoring of Distributed Memory Machines - A Survey, Research Report 91/28, Laboratoire de l’Informatique du Parallelisme, Ecole Normale Superieure de Lyon, Institut des Sciences de la Matiere de l’Universite Claude Bernard de Lyon, Institut IMAG, Unite de Recherche Associee au CNRS No. 1398. Wolfson, O., Sengupta, S. & Yemini, Y. (1991) Managing Communication Networks by Monitoring Databases, IEEE Transactions on Software Engineering, Vol. 17, No. 9, pp. 944—953.

164

References Wybranietz, D., Haban, D. (1990) Monitoring and Measuring Distributed Systems, Performance Instrumentation and Visualisation, ACM Press, pp. 27—45.

165