On-line Avoidance of the Intrusive E ects of Monitoring on ... - CiteSeerX

On-line Avoidance of the Intrusive Eects of Monitoring on Runtime Scheduling Decisions  Wanqing Wu Madalene Spezialetti Rajiv Gupta Dept. of Comp. Science Math., Comp. and Eng. Center Dept. of Comp. Science University of Pittsburgh Trinity College University of Pittsburgh Pittsburgh, PA 15260 Hartford, CT 06106 Pittsburgh, PA 15260

Abstract

Distributed programs are often instrumented for collecting information to assist in analyzing the behavior of an application. However, the act of monitoring a process introduces intrusive overhead that delays the times at which actions occur in a monitored execution in comparison to the times at which they occur in an unmonitored execution. These delays can alter the actions of individual processes and the behavior of the system as a whole. The overall behavior of process scheduling is determined by the choices a scheduler makes in terms of when particular processes will be given access to the processor. Monitoring introduced delays can result in dierent scheduling choices being made which can in turn lead to changes in the behavior of processes throughout the distributed system. This paper presents intrusion removal techniques which are designed to accommodate monitoring delays in an attempt to minimize the intrusion on the scheduling and execution behavior of the monitored computation. Keywords - monitoring actions, intrusion, distributed programs, message passing, local schedulers.

1 Introduction

Application programs written in distributed programming languages are composed of multiple processes which are assigned for execution to various processing sites of a distributed system. Each processing site is typically assigned multiple application processes whose execution is scheduled using a particular scheduling paradigm. An attempt to monitor the runtime behavior of such an application through code instrumentation introduces delays, or intrusion, into the application which would not have been present in an unmonitored execution. The monitoring actions which are taken may be complex, potentially requiring time consuming activity such as accessing, updating or analyzing extensive data logs. Even if the monitoring activity is relatively simple, such as storing a single data value, the cumulative eect of repeated actions over the lifetime of the computation can be signi cant. Not only can these delays potentially alter the behavior of the processes being monitored, but they can also alter the behavior of other unmonitored processes and the overall behavior exhibited by the system. In so doing, the act of monitoring a distributed computation  Supported in part by the National Science Foundation through Grant CCR-9212020 and a Presidential Young Investigator Award CCR-9157371.

can even change the characteristics of the behavior being monitored. The impact of these changes can be particularly signi cant when the behavior being studied is related to the interaction of processes, such as in race detection, resource allocation and the fairness and performance of scheduling techniques. At a minimum, monitoring delays the execution progress of the monitored application process. The implication of this delay is that events happen later, in terms of actual time, than they would have in an original, or unmonitored execution of the program. These delays change the relative ordering of events at communicating processes and hence can alter the outcome of the processes' execution. Further, the outcomes of scheduling decisions can be altered by the changes in the points at which process scheduling decisions are made and the number and status of processes located at a site when scheduling is performed. Thus, by altering the timing characteristics of an application, monitoring actions aect the process scheduling and execution behavior. An approach for minimizing the intrusive eects of monitoring is to use software techniques for estimating the true execution timing of a computation by compensating for intrusion-induced eects. The compensation can be performed o-line by postprocessing execution traces collected by running the instrumented program [5, 2]. A drawback of o-line compensation is that the activity captured by the trace re ects behavior which may have been altered due to monitoring intrusion. Thus, while the trace-based approach suf ces if the user is interested in collecting certain types of performance data for any particular execution of the program, it is not suitable for studying behavioral characteristics that capture the likelihoods of various legal executions of a non-deterministic application. Fairness of a scheduling algorithm is an example of such a characteristic. Previously we have considered two approaches for addressing the limitations of the o-line compensation approach: static analysis techniques were developed to identify program points at which monitoring actions can be carried out non-intrusively [6] and an on-line compensation approach was developed to reduce the impact of monitoring [4]. The on-line approach in [4] is limited to the situation in which no processes can be created dynamically and at most one process is mapped to each processing site. In this paper we relax the above assumptions.

2 Intrusive Eects of Monitoring

In this section we rst describe the distributed execution model assumed in this paper and then identify the intrusive eects of monitoring. We assume that a distributed computation is a multiprocess computation where a subset of processes that are mapped onto a processing site are scheduled using a round robin scheduler. An application process can spawn additional application processes at a given site. Processes communicate via message passing and a queue of pending messages is maintained for each process. The reception of a message by a process is deterministic if the receive identi es a unique sender and non-deterministic if the receive speci es a set of valid senders and during its execution the receiver may non-deterministically select any one of the available messages from the speci ed senders. We assume that the code to carry out the monitoring actions is embedded in the application processes and clearly distinguishable from the application code. Each user process at a site is represented by an entry in a single process queue at the site. The entry holds information regarding the current state of the process. New processes are added to the back of the queue and processes are removed from the queue upon termination. The user processes at a given site are scheduled in a round robin fashion. A quantum de nes the maximum execution time assigned to a process each time it is placed on the processor. If a process does not block before the expiration of its quantum, it is preempted by the scheduler. The scheduler then searches the queue for a ready process, which is chosen to execute next. The model assumes that the processor is either executing a user process, a system-related process or the scheduler. Thus, if the processor is available for allocation, the scheduler continually searches for a runnable process until one is located. We assume that the execution of a a system action, such as a message receive, is an atomic action. Next, we consider a number of scenarios which illustrate the eects of monitoring intrusion on the overall behavior of a distributed system.

Process Preemption

Under round robin scheduling, a process is allocated a xed maximum quantum in which to execute. During that period the process will make some amount of progress in its execution. If the process executes monitoring code during its quantum, no progress will be made in the execution of the original application code while the monitoring code is being executed. If the process is then preempted due to the expiration of its quantum, the preemption would occur at an earlier point in the application's execution than if monitoring code had not executed during the quantum. As a result, actions which may otherwise have been taken by the application during that quantum would not have been reached due to the monitoring induced delay. Depending on the nature of these actions, the monitoring intrusion may also eect the ability of other processes to run, thus potentially altering the outcome of future scheduling decisions. It must therefore be insured that the execution of monitoring code does not alter the points in the application's execution at which preemp-

tions take place. Site S Proc P1 ....

Send(P2) * .... P1 currently executing

* process P1’s preemption point + process P2’s blocking point.

Proc P2 +Recv(P1)

Proc P3 .... .... ....

.... ....

P2 P3 blocked on ready local receive from P1

selected next. (a) P1 preempted after sending a message to P2.

Site S

Proc P1 Proc P2 mon. code +Recv(P ) .... 1 .... * Send( P2) .... ....

P1 currently executing

Proc P3 .... .... ....

P2 P3 blocked on ready local receive from P1 selected next.

(b) P1 preempted before sending a message to P2.

Figure 1: Intrusion in Scheduling Time. As an example, consider the behavior in Figure 1 in which P1 is the currently executing process. Figure 1(a) illustrates the execution of P1 if no monitoring had taken place during the quantum. In that case, P1 is preempted after it has sent a message to P2, which is blocked waiting to receive the message. Consequently, when P1 is preempted, P2 will be able to run and thus will be selected by the scheduler. However if, due to a monitoring action, the quantum expired before P1 sent the message to P2, as shown in Figure 1(b), P2 would be blocked at the time of P1's preemption and therefore would not be chosen to run by the scheduler.

Process Scheduling

Alterations in the outcomes of scheduling decisions can result even when correct preemption points are maintained. This situation arises from the fact that the execution of monitoring code at one processor can cause the activity at that processor, as a whole, to run behind that of another processor at which less monitoring has occurred. If processes on these processors interact via message passing, delays at one processor can eect the state of processes, and therefore the outcome of scheduling decisions, at the other. Thus, while in isolation the preemption points may be maintained, the interaction of processes across sites can lead to alterations in scheduling choices. * Process P4’s preemption point. Proc P4 Site S2

....

Send(P2)

Site S2

*....

Site S1 Proc P1 .... .... ....

P1 currently executing

Site S1 Proc P2 Recv(P4)

Proc P3

.... ....

.... .... ....

P2 P3 blocked on ready remote receive from P4 selected next

(a) Message available at P2.

Proc P1 .... .... ....

Proc P4 Mon. Code .... Send( P2) *....

Proc P2 Recv( P4) .... ....

Proc P3 .... .... ....

P2 P3 P1 currently blocked on ready receive executing remote from P4 selected next (b) Message not available at P2.

Figure 2: Intrusion in Scheduling Decision. For example, consider Figure 2, which illustrates the activity at two sites, S1 and S2 . In both the original execution, shown in Figure 2 (a), and the monitored execution, shown in Figure 2(b), P4 is preempted following the send of a message to P2, which is located at S2 . In Figure 2(a), the message is sent suciently

early so that it is available for P2 . Thus, if P1 were immediately preempted, P2 would be chosen as the next process to run. However, in Figure 2(b) the sending of the message is delayed due to monitoring intrusion at P4, so that the message is not yet available at P2. If P1 were preempted, P2 would still be blocked waiting for the message and thus P3 would be selected as the next process to run.

Process Execution

Even if a process does not execute monitoring code itself, its internal behavior can be eected by monitoring intrusion. This phenomenon occurs as a result of interprocess communication with processes which have been directly or indirectly eected by monitoring intrusion. In particular, delays to processes which are sending messages can result in the alteration in the pool of available messages at the receiver. Since the actions of the receiver may vary depending on the contents of a particular message, the selection of a message at a point that it would not otherwise have been chosen can alter the course of the receiving process's execution. Thus, care must be taken to insure that message pools available at the point of a receive are the same during a monitored execution as they would have been during an original execution. Site S2

Site S2

Proc P2

Proc P2 Site S1

....

Send( P1) Proc P1 Recv ( P2 or P3)

Site S1

....

Send( P1) *....

*....

Proc P1 Site S3

....

Proc P3 ....

Send(P1) *....

(a) Two messages available.

Recv (P2 or P3)

....

Site S3 Proc P3 Mon. Code .... *

Send( P1) .... (b) One message available.

Figure 3: Intrusion in Non-deterministic Receive. As an example, consider Figure 3, which shows the activity of processes P1, P2 and P3, which are located at processors S1 , S2 and S3 respectively. In the original execution shown in Figure 3(a), Process P1 is executing a non-deterministic receive from P2 or P3. At that point, messages are available from both sources for selection. However, in Figure 3(b), P3's execution is delayed, so that it has not yet sent its message at the point that P1 reached its receive. Thus, P1 will accept P2's message. If, in an original execution, P1 would have selected the message from P3, the behavior exhibited by P1 after accepting P2's message may not correspond to the behavior it would have exhibited in the original execution.

3 System Overview

The intrusion removal system at each site consists of several modules which are brie y described here. Detailed descriptions of the intrusion removal algorithms are presented in subsequent sections. Timing Module. The Timing Module at a site provides local modules access to the local time of the site. It also maintains the local intrusion time, which is the amount of time which has been spent on activities related to monitoring or intrusion removal at that

site. Given these values, it is possible to derive an estimate of the actual time at which an activity would have occurred in the absence of monitoring. In order to maintain intrusion time a processor requires knowledge of the intrusion times at remote processors. In the absence of a global clock, there will be a variance between the perceptions of the actual time at various processors. Clock synchronization algorithms are used to reduce this variance [1]. Process Scheduling Module. This module handles the maintenance of the process queue and the scheduling and preemption of processes. It consists of three components: the Process Queue, the Round Robin Scheduler and the Interrupt Timer. When the processor is available the Round Robin Scheduler will continually search the Process Queue for a ready process. When a process is selected it will be placed on the processor and the interrupt timer will be set to the value of the quantum. The process will run until it terminates, blocks or is preempted at the expiration of the interrupt timer. The interrupt timer, however, will not tick constantly. Rather, time spent on monitoring activity or intrusion removal will not be charged to the application process. Message Handling Module. This module is responsible for handling both the transmission and reception of messages for local processes. It is responsible for removing the intrusive eects of monitoring on the interprocess communication. The module maintains a Message Pool consisting of all messages which have arrived for local processes but which have not yet been consumed by the processes. A message queue is associated with each process and is maintained in the order in which the messages would have originally been received in a non-monitored execution. The Communicator and the Message Selector form an interface to the message pool. The Communicator accepts messages which are transmitted to or from local processes. The messages are timestamped with the local time. The Message Selector accepts requests from an application process which is executing a deterministic or non-deterministic receive, examines the process's message queue and returns an appropriate message if one is found. It waits for delayed messages and also prevents premature delievery of messages. Failure Detection Module. When a process executes a receive, messages which may have been available in an original execution may not be available under a monitored execution due to delays at the sender. In order to restore the message pools, a process may have to wait for messages from other sites. The waiting time cannot be accurately predicted. Our approach addresses this issue by assuming an upper bound on the delay of message arrival and dynamically recognizing if insucient waiting was performed for a past non-deterministic message reception. If such a situation is encountered, the current trace can be discarded, the upper bound on the waiting time increased and the program re-executed. The Failure Detection Module is responsible for determining if insucient waiting was performed in response to a non-deterministic receive. Application Process. The executing application process will consist of both the original application code

and any additional monitoring code. The sends executed by the process will be handled through the Communicator and the receives will be handled through the Message Selector.

4 Algorithms for Intrusion Removal

In this section we rst describe the maintenance of intrusion time. Next detailed algorithms for process scheduling and message handling will be discussed.

4.1 Maintaining Timing Information

In order to remove intrusion, a notion of current time as well of the advancement of time must be available at each site. The local time is used by the process itself to represent the current point in time in relation to an original execution. It is also used to timestamp outgoing messages, so that messages can be ordered at the receiving process's site. De nition: The local time at processing site Si during the execution of an instrumented version of a process, denoted by LTiin , is given by the value pair (CTi ;
i), where CTi is the clock time of Si obtained from Si 's internal clock and
i is the current intrusion time of Si . Thus, the estimate of the clock time at Si during the execution ofo the original uninstrumented program, denoted by LTi , is given by CTi ?
i . We assume that the timing module makes available the two functions LocalTime() and ClockTime(), which return the site's current local time and the clock component of the local time, respectively. The values of the local time are updated as various activity occurs in the system according to whether or not the activity would have been present in an original execution. When activity is occurring which also would have occurred in an original execution, only the clock time will advance. The occurrence of activity which would not have been present in an original execution is counted as pure intrusion and advances both the clock and the intrusion time.

4.2 Process Scheduling

The Process Scheduling Module is responsible for determining which process will be given access to the processor, when that access will occur and for how long a process, which does not voluntarily surrender the processor, will be permitted to run. Three components of the Process Scheduling Module interact to implement to accomplish these tasks: the Interrupt Timer, the Process Queue and the Scheduler.

Interrupt Timer

To implement the round robin scheduling accurately, the intrusion removal system provides an InterruptTimer for each site. The InterruptTimer contains routines which insure that each process receives a full quantum to dedicate to execution of the application process. In so doing, it is insured that the preemption points of the original execution are maintained in the monitored execution. When a process is selected to run, the Scheduler will call InitQuantum to set the preemption interrupt to the size of the quantum. As mentioned above, an application experiences intrusion to its execution time in two main ways: through the execution of monitoring code and through the invocation of the MessageSelector

upon the execution of a receive statement. The intrusive eect of the execution of these actions is removed by adjusting the time until preemption and the local intrusion time. The intrusive eects of monitoring code execution are removed by the routines StartMonitoring and EndMonitoring. StartMonitoring, which is invoked directly before the execution of monitoring code, disables the preemption interrupt and sets the values for the clock and intrusion times at which monitoring began from the local time. EndMonitoring removes the monitoring execution intrusion by adjusting the local intrusion and the preemption interrupt time. First, the new local intrusion is calculated by incrementing the local intrusion time by the execution time of the monitoring code. EndMonitoring then determines if there is any time remaining in the quantum, that is, whether or not the execution of the application before the monitoring code began had used the entire quantum. If the application did not use the entire quantum, the preemption interrupt is set to allow the application to execute for the balance of the quantum. If no time remains, the application is preempted and the Scheduler is invoked to choose the next process. InterruptTimer (Site S) f

InitQuantum() f

StartTime, (ts;
s ) = LocalTime(); /* Quantum - size of the execution timeslice */ SetPreemptInterrupt(Quantum); g

StartMonitoring() f

DisablePreemptInterrupt(); BeginMonitor, (tbm;
bm ) = LocalTime(); g

EndMonitoring() f

EndMonitor, tem = ClockTime();
em = (tem ? tbm ) +
bm ; Intrusion time of site S =
em ; QuantumLeft = Quantum ? ((tem ?
em ) ? (ts ?
s )); if QuantumLeft > 0 f SetPreemptInterrupt(QuantumLeft) g else f Scheduler(Ready,) g g

StartMsgSelection() f

/* called when appl. calls message selector */ DisablePreemptInterrupt(); BeginMsgSelection, (tbs;
bs) = LocalTime(); g

EndMsgSelection() f

/* only called if successful in nding message; if message not found, message selector calls the scheduler */ EndMsgSelection, (tes;
es ) = LocalTime(); QuantumLeft = Quantum ? ((tes ?
es ) ? (tbs ?
bs )); if QuantumLeft > 0 f SetPreemptInterrupt(QuantumLeft) g else f Scheduler(Ready,) g g g

Figure 4: Quantum Timer for Timeslicing. The execution intrusion caused by the message selection routines is removed by the routines StartMsgSelection and EndMsgSelection. StartMsgSelection, which is called by the MessageSelector when it begins execution, will disable the preemption interrupt and save the values of the clock and local intrusion time at that point when the MessageSelector began execution. If a message is found to ful ll a receive request, EndMsgSelection is called to adjust the quantum and intrusion times. Based on the beginning and ending times of

the message selection, it is determined if there is time remaining in the current quantum for use by the application. If so, the preemption interrupt will be reset, otherwise the Scheduler is invoked to choose the next process to run. It should be noted that EndMsgSelection is not invoked if no message was found to ful ll the receive. In that case, the process would block, as it would in the original execution, and the issue of resetting the preemption timer is moot.

Process Queue

Each active process is represented by an entry in the process queue, which is maintained as a single queue. When a new process is created, it is placed at the back of the queue. The current running process is at the front of the queue. Each entry in the process queue contains the process id, the status of the process, and if the process is blocked on a receive, the identities of the acceptable senders. The status of a process is one of the following: (i) Ready: the process is ready to run; it is either a new process or a process that was preempted due to quantum expiration during its last opportunity to run; (ii) Local Deterministic: blocked on a deterministic local receive; (iii) Local Non-Deterministic: blocked on a non-deterministic receive with only local senders; (iv) Non-Local Deterministic: blocked on a deterministic non-local receive with at least one non-local sender; or (v) Non-Local Non-Deterministic: blocked on a non-deterministic receive with at least one nonlocal sender. It should be noted that a process whose status indicates that it is blocked on a receive is runnable if a message is available for it to receive. The availability of messages is determined when the process's status is checked by the Scheduler as it traverses the Process Queue to select the next process to run.

Scheduler

When the processor becomes available, the Scheduler will continually scan the process queue until it

nds a process which is runnable either because it had previously been preempted or because it is blocked on a receive and a message has become available. The relationships between the times at which messages become available to blocked processes and the times at which the Scheduler examines processes for runnability, are crucial to determining which process will be selected by the Scheduler to run. Thus, the eects of intrusion on communication can lead to alterations in scheduling choices during a monitored computation. Two main situations may arise which would cause a process to have an incorrect status when checked by the Scheduler. First, a message which would have been available to be received in an original execution may be delayed due to intrusion at its sender, causing the receiving process's status to be incorrectly set to unrunnable when checked by the Scheduler. Secondly, the Scheduler may check a process's status later in the monitored computation than in the original, causing a message which would not have been present in the original computation to be incorrectly treated as available in the monitored computation. By receiving a message prematurely, a process whose status should have been unrunnable will be set to runnable and incorrectly chosen by the Scheduler.

First, consider the situation of a message which has been delayed due to intrusion at its sender such that it is not available to be received when the process is checked by the Scheduler. A solution to this situation would be for the Scheduler to wait for a bounded time, MAXWAIT, before each traversal of the process queue. The wait would provide time for delayed messages to arrive and their receiving processes would have the appropriate status of runnable. However, the delay may result in the status of blocked processes being changed to runnable prematurely. If such a process appears in the queue before the process which would originally have been chosen, it will be incorrectly selected. Thus, when messages arrive, it must be insured that they are not considered as available before the (relative) point in the computation at which they would have originally arrived. As an illustration, consider the activity shown in Figure 5, in which all processes are currently blocked. It is assumed that process P4 has just become blocked, so that P1 will be the rst process checked in the process queue. In the scenario shown in Figure 5(a), no monitoring is done. At time 10, the Scheduler begins cyclicly scanning the process queue to locate a runnable process. On the rst pass, no runnable process is found. During the second pass, a message arrives for P1 after the Scheduler has checked its status. A message then arrives for P4 before P4 is checked by the Scheduler. Thus, when the Scheduler checks the status of P4 , it will determine that a message is available and P4 will be chosen to run. However, consider what the outcome would be if, due to intrusion at the sender, the message for P4 had arrived after its status had been checked in the second pass. As a result, a third pass would have been undertaken, and P1 would have been found to be runnable and chosen to execute. Now consider the behavior under monitoring and intrusion removal which is shown in Figure 5(b). To accommodate possible delayed messages, the Scheduler rst performs a MAXWAIT, assumed here to be 10, before scanning the process queue. However, as a result of this delay, the message which arrived for P1 during the second pass of the original execution, now arrives during the rst pass. If that message were assumed to be available when it arrives, P1 will be judged to be runnable when it is checked in the second pass and would be incorrectly chosen to run. This error would have been caused by the premature acceptance of P1's message relative to the progress of the Scheduler as exhibited in the original execution. To prevent such errors, the intrusion removal algorithms must insure that messages are judged to be available in light of adjusted clock times. The routine MsgAvail, shown in Figure 6, determines if any of the messages in a process's message queue should be considered available, where available indicates that a message would have been received at or before the current point in an original execution. The module accepts the identity of the receiving process, the type of the receive (local deterministic, local non-deterministic, non-local deterministic or non-local non-deterministic) and a list of valid senders. If the receive is local, a test only needs to be made to determine if an acceptable message is in the pool, since

Site S P1 blocked on nonlocal receive blocked P2 on local receive blocked P3 on local receive P4 blocked on nonlocal receive Process Queue

(Clock time, Intrusion time) Schedulers’s Actions (15,5) MAXWAIT (10) (25,15) (26,15)

examine P1

examine P2 message for P1 (27,15) sent at remote examine P3 time (24,11) (28,15) Schedulers’s Actions arrives at local examine P4 Clock time 10 time (26,15) (29,15) examine P1 11 examine P2 MAXWAIT message for P1 12 examine P3 sent at remote (10) time 13 13 examine P4 arrives at local (39,25) examine P1 message for P4 time 15 14 examine P1 sent at remote (40,25) examine P2 15 time (37,23) examine P2 arrives at local (41,25) examine P3 16 time (39,25) examine P3 message for P4 (42,25) examine P4 sent at remote 17 examine P4 time 14 (43,25) arrives at local 18 time 16 (44,25) Schedule P4 Schedule P4 19 (b) With Monitoring. (a) Without Monitoring.

Figure 5: Intrusion Removal in Scheduling Decisions. all local processes maintain the same adjusted view of the local time and are therefore correctly synchronized. However, if the receive is non-local and messages are present in the process's queue, it is necessary to ascertain when the message would have originally arrived. To determine this value, each incoming remote message is timestamped with the sender's local time at the time it was sent. From this value, the original arrival time is calculated by adding the original sending time, (tsend ?
send), to the communication time. A message will only be considered as available if its arrival time is less than or equal to the adjusted current time, thus assuring that messages are not made available prematurely. MsgAvail is called by both the Scheduler and the MessageSelector. The Scheduler, shown in Figure 7, is invoked when either a process is preempted due to the expiration of its quantum or when a process is blocking due to the execution of an unsuccessful receive. In the former case, the Senders parameter is empty, while in the latter case it will contain a list of valid senders. After saving the status information of the process being removed from the processor, the Scheduler begins to check the entries in the process queue. It will select the rst process it encounters which it determines to be runnable. Such a process may be runnable either because it was preempted during its last use of the processor or because it was blocked on a receive and a message is now available. The Scheduler will MAXWAIT each traversal of the process queue only if at least one process is blocked on a non-local receive. After each MAXWAIT the local intrusion time is incremented by the value of MAXWAIT. After a process has been selected to run, the quantum timer will be reset and the process will begin to run. Let us return to the example in Figure 5(b) to examine the complete behavior of the intrusion removal

MsgAvail(Precv ,RecvType,Senders) f /* return true if message is already available, otherwise return false */ case RecvType of f Local Deterministic, Local Non-Deterministic: if message pool contains a message for Precv from a process in Senders f return(true) g else f return(false) g Non-Local Deterministic, Non-Local Non-Deterministic: avail = true; for each process Psender 2 Senders f if message pool contains a message for Precv from Psender such that (tsend ?
send ) + comm  (tcurr ?
curr ), where message was sent at time (tsend;
send ), current time (tcurr ;
curr ) = LocalTime(), and the time between message transmission and receiving is comm f /* message from Psender found */ g else f avail = false; /* not found */ g g return(avail) g g

Figure 6: Checking Availability of Messages.

algorithms. In the adjusted clock time, the Scheduler begins execution at (15,5), which indicates that the current clock time at site S is 15, of which 5 units is intrusion. Since the rst process to be examined, P1, is blocked on a non-local receive, the Scheduler will rst MAXWAIT and the intrusion time will be adjusted accordingly. The Scheduler will then proceed to examine each process in order, invoking MsgAvail to test if its status has become runnable. During the rst pass over the process queue, a message arrives for P1 after P1 has been checked. The message is timestamped (24,11) from its sender. Upon arrival, this message is placed by the Communicator into the message pool for P1, and its availability will be determined when P1's status is once again checked by the Scheduler. Since no runnable process is encountered in the rst pass, the Scheduler will again cycle over the process queue. Before starting its second pass, the Scheduler performs another MAXWAIT to accommodate the late arrival of messages for the current pass. When it has completed the wait and adjusted the local intrusion time, it will call MsgAvail to check the status of P1 's messages. MsgAvail will determine that the original arrival time for the message currently in P1's pool is 15, based on the sender's timestamp of (24,11) and a communication time of 2. However, based on a current local time of (39,25), the original local time is 14. Thus, it will be determined that the message originally arrived after the current point in the execution and therefore should not be considered available. Subsequently, a message for P4 arrives, timestamped (37,23) from its sender, and it is stored in P4's message queue. When the Scheduler invokes MsgAvail to test P4's status, it will be determined that, based on the sending timestamp of (37,23) and communication time of 2, the original arrival time of the message was 20. Since the original current local time from the clock of (42,25) is 17, the message would have arrived before the current point in time and thus would have been available to P4 originally. Thus, P4 is determined to be runnable and correctly selected for

execution by the Scheduler. It is important to note that the Scheduler MAXWAITs once for each pass over the process queue. In so doing, it is waiting for late arriving messages for that particular pass. As shown in the example, even though this wait may inadvertently result in messages arriving prematurely relative to the Scheduler's testing, it will not cause an incorrect selection since such messages will not be treated as available until the local adjusted time passes their true arrival time. Further, both the current time and the intrusion time of the local site advance during the MAXWAIT. Thus, while a delay is actually occurring during the MAXWAIT period, the adjusted time is not advancing and so intrusion from MAXWAIT is removed.

Scheduler(CurrentStatus;Senders) f /* CurrentProc points to process in the scheduler's queue that has just been taken o the processor */ CurrentProc.Status = CurrentStatus; CurrentProc.Senders = Senders; MAXWAIT PROC = unde ned; found = false; while not found do f CurrentProc = NextQueue(CurrentProc); case CurrentProc.Status of f Ready: found = true; Local Deterministic, Local Non-Deterministic: found = MsgAvail([CurrentProc]Id,Status,Senders); Non-Local Deterministic, Non-Local Non-Deterministic: Partition CurrentProc.Senders into Senderslocal and Sendersnonlocal ; if not MsgAvail([CurrentProc]Id,Status,Sendersnonlocal ) f if (MAXWAIT PROC = unde ned or CurrentProc) f Wait(MAXWAIT); MAXWAIT PROC = CurrentProc; Update intrusion time of site,
=
+ MAXWAIT ; g g found = MsgAvail(CurrentProc.Id,Local Non-Deterministic, Senderslocal ) or MsgAvail(CurrentProc.Id,Non-Local Non-Deterministic, Sendersnonlocal ) g g /* when found = true */ CurrentProc.Status = READY; InitQuantum(); run(CurrentProc) g

Figure 7: Scheduling Algorithm.

4.3 Message Handling and Selection

The components of the Message Handling Module, the Communicator, the Message Pool and the Message selector, attempt to insure that the communication behavior exhibited by the monitored applications re ects the original behavior. The Message Pool contains all messages which have arrived for any process at its site. The messages are held in queues associated with each local process. When the Communicator receives a message destined for a local process, it will place the message into the destination process's queue in the Message Pool. The MessageSelector forms an interface between the user processes and the Message Pool. Its primary task is the servicing of message requests from processes when they execute a receive. In so doing, the Selector must insure that the correct message is returned to a requesting process and the execution behavior of the process, in relation to the success or failure of the original receive, is maintained.

As an illustration of the possible scenarios which can arise, consider Figure 8(a). In the monitored computation, a message arrives for the process before it has executed the receive, and thus would be present at the time of the receive request. If the message also would have been available in an original execution, the Selector should return the message to the process and the process should continue execution. However, although a message is present in the monitored computation, it may have arrived after the receive's execution in an original execution. Since the process would have blocked in an original execution, the Selector must also insure that the requesting process blocks on the receive, permitting another process to run. Without Intrusion

S With Intrusion Proc P S

R

R continue P’s execution

S

R force process P to deschedule (a) With monitoring, message arrives before receive. Without Intrusion

With Intrusion

S

Proc P

R

S R deschedule proc P

S R prevent P from descheduling

(b) With monitoring, message arrives after receive.

Figure 8: Switching vs Continued Execution. Conversely, consider Figure 8(b), in which a message is not present at the time of a receive request in the monitored computation. If no message would have been available in an original execution, the requesting process must also block in the monitored execution. However, if the message would originally have been present, the process would not have blocked. Thus, in the monitored execution, the requesting process must also not block. Rather, it will be delayed until the appropriate message becomes available. At that point, the Selector will return the message to the requesting process which will then continue execution. It is important to note that the receiving process should not be be taken o the processor in this case, since originally no other process would have run between process's execution of the receive statement and the reception of the message.

It should also be noted that for non-deterministic receives with at least one non-local sender, the presence of acceptable messages at the time of a receive request is not sucient to insure that the correct message will be chosen by the MessageSelector. Rather, the complete original message pool must be present before a message selection can occur. By waiting a bounded amount of time before selecting a message, the Selector attempts to accommodate the late arrival of non-local messages which would have been present in the original execution, but which were delayed at their senders in the monitored execution. MessageSelector(Precv,Recvtype,Senders) f StartMsgSelection(); /* inform quantum timer */ case Recvtype of f /* All senders Local */ Local Deterministic, Local Non-Deterministic: if MsgAvail(Precv ,RecvType,Senders) f EndMsgSelection(); Deliver() g else f Scheduler(Recvtype,Senders) g /* At least one sender Nonlocal - need to MAXWAIT */ Non-Local Deterministic, Non-Local Non-Deterministic: Wait(MAXWAIT); Update intrusion time,
=
+ MAXWAIT ; Partition Senders into Senderslocal and Sendersnonlocal ; if MsgAvail(Precv ,Local Non-Deterministic,Senderslocal) or MsgAvail(Precv ,Non-Local Non-Deterministic, Sendersnonlocal ) f EndMsgSelection(); Deliver() g else f Scheduler(Recvtype,Senders) g g g g

Figure 9: Message Selection. The code for the MessageSelector is shown in Figure 9. When invoked by a process executing a receive, it will be passed the id of the process, the type of receive and the list of acceptable senders. The type of the receive can be local deterministic or local nondeterministic, in which all senders must be local, or a non-local deterministic or non-local non-deterministic in which at least one sender is located at a remote processor. The InterruptTimer is rst informed of the beginning of the Selector's execution, so that its execution time can be excluded from the requesting processes quantum time. Next, a test is made to determine whether or not all sending processes are local. If they are all local, the sending processes would be operating with the same view of delayed time as the receiver, since any intrusion which eects the senders would also eect the receiver. It is therefore not necessary to wait before making a selection and and so a check is made immediately to determine if an acceptable message is available. If one is found, the clock handler is informed that the Selector's execution is completing and the Selector returns the message to the requesting process. However, if even one of the senders is nonlocal, it is necessary to wait before checking for an available message. Thus, the Selector waits for MAXWAIT (and appropriately increments the intrusion time) before checking message availability. If an appropriate message is found, the InterruptTimer is informed of the end of the message selection and the message is returned to the requestor. During the MAXWAIT, the requesting process is not removed from the processor. Thus, if a message is returned to the process, it will

continue execution and no other process would have run between the execution of the receive and the message reception. In either the local or non-local case, if no message is available, the Scheduler will be called to block the requestor and select a new process to run.

4.4 Failure Detection

The intrusion removal system continuously examines the actions of the system in order to detect failures to remove intrusion. Failures occur when it is determined that insucient waiting was performed for the arrival of a message. When such a failure is detected, the user is informed of its occurrence. The user can then either continue the execution in spite of the failure or restart execution with a larger MAXWAIT. Insuf cient waiting for a message can result in a failure of the intrusion removal system in a number of ways: (i) Scheduling of a Blocked Process: A blocked process was checked by the Scheduler and passed over because no message was available to it. If the newly arriving message should have been available then the process would have been scheduled; (ii) Descheduling of an Executing Process: The message should have been delivered when a process executed a receive. The delivery of the message would have prevented the descheduling of the process before its quantum expired; and (iii) Outcome of a Non-deterministic Receive: The message would have aected the outcome of a successful non-deterministic receive.

5 Concluding Remarks

The act of monitoring a distributed computation introduces delays that can alter the actions of individual processes and as well as the behavior of the system as a whole. The intrusion removal techniques presented in this paper are designed to make accommodations for monitoring delays in the context of a round robin scheduling strategy. Currently, we are applying this technique to other scheduling strategies. Techniques are also being investigated which can make productive use of the waiting periods.

References

[1] F. Cristain and C. Fetzer, \Probabilistic Internal Clock Synchronization," 13th IEEE SRDS, Oct. 1994. [2] J.A. Gannon, K.J. Williams, M.S. Andersland, T.L. Casavant, and J.E. Lummp, \Trace Recovery in MultiProcessing Systems: Architectural Considerations," Int'l Conf. on Par. Proc., Vol. II, pp.99-102, 1994. [3] L. Lamport, \Time, clocks and the ordering of events in distributed systems," CACM, 21(7):558-565, 1978. [4] R. Gupta and M. Spezialetti, \Dynamic Techniques for Minimizing the Intrusive Aects of Monitoring Actions," IEEE-CS 15th ICDCS, pp.368-376, June 1995. [5] A. Malony and D. Reed, \Models for Performance Perturbation Analysis," ACM/ONR Workshop on Parallel and Distributed Debugging, pp.15-25, 1991. [6] M. Spezialetti and R. Gupta, \Perturbation Analysis: A Static Analysis Approach for the Non-Intrusive Monitoring of Parallel Programs," Int'l Conf. on Parallel Processing, Vol. II, pp.81-88, August 1994.