Dynamic Techniques for Minimizing the Intrusive E ect of Monitoring ...

Dynamic Techniques for Minimizing the Intrusive Eect of Monitoring Actions  Rajiv Gupta Madalene Spezialetti Dept. of Computer Science EECS Dept., Packard Lab University of Pittsburgh Lehigh University Pittsburgh, PA 15260 Bethlehem, PA 18015

Abstract

Programs written in distributed programming languages are often non-deterministic in nature and hence the execution of such programs is not only dependent upon the program input, but also on the timing of the execution. Thus, an attempt to monitor the runtime behavior of a non-deterministic program through code instrumentation, such as during debugging, can potentially alter the program's behavior. In this paper we present techniques for dynamically minimizing the intrusive eects of monitoring by attempting to ensure that the likelihoods of various outcomes of the nondeterministic events are the same for uninstrumented and instrumented programs. This goal is achieved by minimizing the intrusive eects of monitoring on message passing between processes located at dierent sites of a distributed system. Monitoring actions alter the message pool at the time of non-deterministic message selection and the order in which the messages arrive at a processor. The dynamic techniques presented in this paper minimize intrusion by restoring the message pool and message ordering.

1 Introduction

Programs written in parallel and distributed programming languages are often non-deterministic in nature and hence the execution of such programs is not only dependent upon the program input, but also on the timing of the execution. Thus, an attempt to monitor the run-time behavior of a non-deterministic program through code instrumentation, such as during debugging, can potentially alter the program's behavior. The intrusive eects of monitoring actions can be broadly classi ed into the following two categories: Type 1 - Execution Selection: A non-deterministic distributed computation has multiple legal executions for a given input. The selection of a particular execution is made at run-time and typically this selection is in uenced by the timing of various events. Since the monitoring actions alter the execution timing, the likelihoods of the selection of various legal executions is altered. Type 2 - Execution Timing: The timing characteristics for the particular execution that is selected are also aected by monitoring actions. The delays in the execution of processes alter the relative timings of their execution. Thus, not only is the overall execution times of the processes aected, but so are other timing characteristics such waiting for messages, processor idling etc.  Supported in part by the National Science Foundation through Grant CCR-9212020 to Lehigh University and a Presidential Young Investigator Award CCR-9157371 to the Univ. of Pittsburgh. Contact Author: [email protected].

Techniques for removing the intrusive eects of for handling this kind of intrusion is to provide architecture speci c monitoring hardware so that the intrusive eects of monitoring actions are minimized [11]. Another less expensive approach is to use software techniques for estimating the true execution timing of a computation by compensating for intrusioninduced eects. Execution traces are collected by running the instrumented program. Postprocessing of the traces is carried out to compensate for intrusive eects of monitoring. Such techniques have been explored by Malony et al. [7, 9] and Casavant et al. [4]. Models of perturbation in execution have been developed and used to process traces for predicting true performance. The above research is based upon an assumption that the user is interested in analyzing the performance of the program under a speci c program execution. Furthermore, it is assumed that there is no interest in evaluating the likelihoods of various legal executions. There are certain behavioral characteristics which are directly in uenced by the changes in the likelihoods of selecting various legal executions. For example, the goal of monitoring may be to access the fairness of a mutual exclusion algorithm, to study the load balancing characteristics of a scheduler, or to identify starvation of processes. In each of these situations the distributed algorithms are expected to be non-deterministic in nature and the likelihood of selection for various legal executions will determine the fairness characteristics, load balancing abilities, and the degree of starvation respectively. Thus, for monitoring such applications we must develop techniques for minimizing the extent of Type 1 intrusion. The detection of the potential for Type 1 intrusion using static analysis techniques has been considered in our previous work [10]. The synchronization structure of the program is analyzed in conjunction with the monitoring actions to determine whether the relative times of the execution of the participants of a non-deterministic interaction are unaected. If the potential for intrusion is detected, the user may attempt to avoid it by moving the monitoring actions to another point in the program. However, if this approach is not successful, no solution is suggested for the removal of the intrusive eects of Type 1. In this paper we develop techniques for minimizing the eects of Type 1 intrusion. One of the major dierences between the two kinds of intrusion is that although techniques that postprocess traces can be used to minimize the eects of Type 2 intrusion, a similar approach is not feasible for Type 1 intrusion. Since the trace based approach collects a trace for a speci c program execution, if during the postprocessing of the trace it is determined that at some Type 2 have been extensively studied. One approach

point in execution the monitoring actions altered the selection of execution then, based on the remainder of the trace, the behavior of the alternate execution cannot be determined. Thus, in order to handle Type 1 intrusion we must develop dynamic techniques that are able to minimize the eects of monitoring at runtime and lead the program to make non-deterministic choices that are likely to be made in the absence of monitoring activity. We assume an execution model in which a distributed computation is composed of multiple processes located at dierent processing sites. The processes communicate via message passing and nondeterminism arises due to the order in which the messages arriving at a process from dierent sources are handled. Monitoring actions alter the message pool at the time of non-deterministic message selection and the order in which the messages arrive at a processor. By restoring the message pool and message ordering, we can undo the intrusive eects of monitoring at runtime. The basis of our approach is to appropriately stall the execution of a process so that other processes that may be delayed due to monitoring actions are able to catch up. Since to achieve the above goals we must rely upon synchronized clocks, the scheme cannot entirely eliminate intrusion. However, the degree of intrusiveness is bounded by the errors in the clock. If communication delays caused by monitoring actions are unbounded, then any scheme based upon limited waiting cannot completely eliminate the eects of intrusion. Our approach is to perform limited waiting. However, we are able to detect if intrusion has occurred due to insucient waiting and therefore by increasing the amount of waiting and then restarting the execution of the program, we can collect more accurate monitoring information. In section 2 we discuss the distributed execution model and assumptions made in this work. In section 3 we discuss in greater detail the intrusive eects of monitoring on message passing. Section 4 provides detailed algorithms for dynamically reducing the intrusive eects of monitoring on message passing. In section 5 we present our concluding remarks and future work on some optimizations that can be employed to reduce the overhead of our dynamic protocol by limiting its use to a subset of processes belonging to a distributed computation.

2 Distributed Execution Model

We assume that a distributed computation is a multiprocess computation in which each process is mapped to a dierent site of a distributed system. The processes communicate via message passing and a queue of pending messages is maintained for each process. A monitoring action is merely a code segment used to record information and is clearly distinguishable from the application code. The execution of a monitoring action takes time which slows down the execution of the monitored process and potentially slows down the execution of other processes waiting to receive messages from the monitored process. The reception of a message by a process is either deterministic or non-deterministic in nature. Message reception is deterministic if the receive identi es a unique sender. A non-deterministic message reception is supported using the any-way branch shown in Figure 1a. If messages from dierent senders have arrived at a receiver, the receiver may non-deterministically select a message by choosing the appropriate path

of an any-way branch in a non-deterministic fashion. The select statements supported in languages such as CSP [6] and Ada provide this form of nondeterminism. Although the criteria for message selection are not expressed in the application by the user, the actual implementation may perform this selection based upon criteria such as the order in which the messages arrive, priorities of the senders, a fairness criteria such as selecting a message from a process that has been least recently considered, or a purely random selection may be made. If no messages are pending at the receiver, we assume that the receiver selects the path along the any-way branch which corresponds to the sender whose message is the rst to arrive at the receiver. Even in this situation the reception of a message can be non-deterministic because multiple senders may simultaneously send messages to the receiver (see Figure 1b). The order in which these messages arrive may vary from one execution to another. Thus, depending upon which message arrives rst, a dierent execution path is taken. S

S R

R

(a)

S

S R

R

(b)

Figure 1: Non-determinism Message Reception: (a) Multiple Pending Messages at an Any-way Branch; and (b) Message Race due to Multiple Messages in Transit to an Any-way Branch.

3 Intrusive Aects of Monitoring on Messages

The monitoring actions cause the timings at which messages are sent to be altered. The consequence of an alteration in message timing is its potential aect on the results of a non-deterministic message selection by a process. A dierent message may be selected due to the following reasons: Message Pool: At the time of message selection, the selecting process examines the pool of messages that have arrived and are waiting to be processed. Due to the monitoring delays in the selecting process and the message sending processes, the pool of messages from which the selection is to be made is altered. The pool of messages may contain messages that should not be considered. Further, messages that should be in the pool may not have arrived yet. Message Ordering: Due to the changes in the timings at which messages are sent, the order in which the messages arrive at the receiving process is changed. The message ordering is often an important selection criteria at the receiving process and therefore it can aect the outcome of an any-way branch, as shown in Figure 1. The intrusive eects of monitoring discussed above are illustrated in Figure 2. In this example the receiving process Pr is processing messages m1 and m2 sent by processes P1 and P2 respectively. In Figures 2a and 2b we illustrate the changes in the message pool at the

m1

(a) Pr tr

m2

m1

m1

P1

Pr

t1

t2

tr’ P2

m1 m2

Without Intrusion (b)

m1 Pr

m2

t2 Without Intrusion

(c)

m1 Pr tr m1 m2

Pr tr’

P2

t2 Without Intrusion

t2’ With Intrusion

t2’ With Intrusion

m1

P1

Pr

P2

P1

t1’ m2

m1

t1 m2

m2

m1

P1

t1

tr m1 m2

P1

t1’

P2

P1

t1’

tr’ P2

m2 m1

m2 t2’ With Intrusion

P2

Figure 2: (a) Additional Messages in Pool; (b) Fewer Messages in Pool; and (c) Messages Reordered. time of message selection by Pr that may result due to monitoring actions. In the rst situation an additional message (m2 ) has arrived while in the second situation a message (m1 ) which should have arrived has not arrived yet. Finally in Figure 2c the message pool is unchanged; however the order in which the messages should have arrived is reversed.

4 Minimizing Message Intrusion

Next we develop a dynamic execution strategy for minimizing the eect of intrusion by undoing the modi cations in the message pool and message ordering at each processing site. The basic idea behind our approach is to maintain estimates of times spent on monitoring activities and from these estimates to infer the times at which various actions would have occurred if no monitoring had been performed. The estimates form the basis for restoring message pools and message ordering. No dynamic technique can entirely eliminate intrusion caused by monitoring due to the following reasons. Global Clock: The determination of the true times of various actions would require a global clock. Since a global clock is not available in a distributed system, we maintain synchronized clocks and minimize the intrusion through these clocks. A bounded amount of error between synchronized clocks is unavoidable and thus the intrusion cannot be entirely removed. We do not consider this to be a serious problem in practice. If the time spent on monitoring activities is orders of magnitude greater than the errors in the clocks, then the variations in the likelihood of various non-deterministic events due to these errors is going to be negligible. On the other hand, if the monitoring times are of the same magnitude as errors in the clocks, then the eect of monitoring will cause variations that could have as easily been caused due to variations in the communication times among processing sites. This is because errors in the clocks are a function of variations in communication delays between processors [2]. Bounded Delays: In order to accurately restore the

message pools a processor may have to wait for messages from other sites. The delay in the arrival of a message depends upon the monitoring activities at the sender prior to the sending of the message and the communication time. An upper bound on the delay between a pair of processes is required to avoid inde nite delay. By choosing a very high bound on this delay, we can reduce the likelihood that a message pool will not be entirely restored. However, the possibility that this bound would be exceeded at some point during execution cannot be entirely eliminated. Our approach addresses this issue as follows. We assume an upper bound on the delay of message arrival for the purpose of program execution and in addition dynamically recognize if insucient waiting was performed for a past non-deterministic message reception. If such a situation is encountered, we can discard the trace generated, increase the upper bound on communication delays and reexecute the program. Processing Site Failure Detection & Restart

Timing Local Time Clock Time

Communicator

Intrusion Time Corrections for RemoteClocks

Interconnection Network

Message Selection Instrumented Application Process

Figure 3: Overview of Intrusion Elimination. Next we brie y outline the main components of the intrusion removal system shown in Figure 3. In subsequent subsections we describe these components in detail. Application process includes the monitoring actions in it. When these actions are encountered the relevant monitoring information is saved and the estimate of local intrusion time is also updated. Messages to be sent to other processes are sent directly to the communicator module while messages received from other processes are received via the message selection module. Timing module at each site provides access to the local time which is composed of the local clock time and local intrusion time. In addition, the timing modules at dierent sites interact to maintain synchronized clocks. The timing information is accessible to all other modules at a processing site. Communicator module is responsible for handling messages. It sends out messages from the local application process to other sites. The application messages received from other sites are deposited into a local message queue maintained at each site. The messages are deposited at an appropriate position in the queue so as to eliminate message ordering intrusion. In addition to application messages, other messages, such as those generated by the clock synchronization protocol, are also handled by this module. Message selection module is responsible for providing messages to the application process in response to deterministic and non-deterministic message reception requests. The module is responsible for removing the intrusive eects of monitoring on the message pool. Thus, the module must determine whether cer-

tain messages must be ignored and/or additional waiting is required for all relevant messages to arrive before a response to a non-deterministic message selection can be made. The local value of intrusion time must also be updated by this module. Failure detection and restart module keeps track of the recent non-deterministic message receptions and, by comparing the times at which these actions took place with the timing of future messages, situations under which insucient waiting was performed by the message selection module are identi ed. The detection of a failure to perform sucient waiting implies potential intrusion that has not been removed. If such an intrusion is unacceptable, the waiting time is increased and the computation is restarted to collect fresh monitoring data.

clock available at the site. Finally, when the application process executes a message reception command, the intrusion time may increase, decrease, or remain unchanged, depending upon the possible changes in the waiting associated with the receiving of the message. The changes in intrusion time for various situations that may arise are illustrated in Figure 4. Monitoring

(a)

(b)

0 2 4

∆ =0

12

∆ =0

10

4.1 Maintaining Time

In order to undo the reordering of messages arriving at a processor and the alterations in the message pool which are caused due to monitoring activities, we must estimate the timing at which the messages would have been sent and received if no monitoring actions were performed. On the basis of these estimates each processor can restore the message pool by waiting for late messages or ignoring messages that have become available early. In addition, the messages can also be appropriately ordered. The timing module at each processor maintains its local time and estimates of corrections required to reconcile the dierence between its clock time and clock times of other processors. Next we de ne the above timing information and brie y describe how these times are computed. De nition: The local time at processor Pi during the execution of an instrumented version of a process, denoted by LTiin , is given by the value pair (CTi ;
i), where CTi is the clock time of Pi obtained from Pi's internal clock and
i is the current intrusion time of Pi. Thus, the estimate of the clock time at Pi during the execution ofo the original uninstrumented program, denoted by LTi , is given by CTi ?
i. De nition: The correction in processor Pj 's clock time required to match the clock of processor Pi, denoted as Cji, is the value pair (dji; Eji). The value dji is Pi 's estimate for the deviation between its local clock and Pj 's local clock and Eji is the range of error in the estimate of dji. Thus, if an event took place when Pj 's clock time was CTj , then processor Pi can compute its adjusted clock time (ACTji) at which the event took place as CTj + dji. The minimum and maximum values of adjusted clock time are given by ACTjimin = ACTji ? Eji and ACTjimax = ACTji + Eji respectively. At any given processor the local value of intrusion time indicates the delay in the execution of the activity being currently performed by the application process due to monitoring activities. As the execution of an application process proceeds, the intrusion time at that site may increase, decrease, or remain unchanged. During the execution of statements belonging to the original application, the intrusion time remains unchanged. If the action underway represents a monitoring command introduced to collect information or an activity that is part of the intrusion minimization protocol, then the intrusion time increases. The change in the intrusion time is computed using the internal

0

S 15 17

R

15 17

ct Without Intrusion - With Wait (d)

0

Executing

Waiting

(e)

15 17

S

∆ =0

9

∆ =7

12

S

15 17 19

R

R

∆ =2

ct With Intrusion ∆ =0

(f) 0

∆ =2

3

S

∆ =0

S

R 17 19

ct ct Without Intrusion - Without Wait

∆ =7

10

10

10

0 2

∆ =2

ct

0 2 4

(c)

R

15 17

∆ =0 ∆ =2

S

R

ct With Intrusion

Figure 4: Updating Intrusion Time due to Message Reception. Let us rst consider the situation in which a process has to wait for the arrival of a message at a receive (see Figure 4a). The introduction of a monitoring action prior to the receive either reduces the amount of waiting (Figure 4b) or entirely eliminates the waiting (Figure 4c) at the receive. In the former case the intrusion time for the actions starting from the receive is zero while in the latter case it is a positive amount. However, the intrusion time for execution actions after the monitoring and before the receive is non-zero in both cases. Next, let us consider the situation in Figure 4d where the message arrives before the receiver is ready to accept the message and thus there is no waiting involved. The introduction of monitoring in the receiver, as shown in Figure 4e, causes all actions after the monitoring to have non-zero intrusion time. Let us assume that monitoring is introduced in the sender causing a delay in the arrival of the message. Further assume that this delay causes the receiver to wait for the message as shown in Figure 4f. In this case the intrusion time of actions following the receive is increased. We have illustrated the changes in intrusion times through examples. The general treatment of this issue will be presented along with the discussion of the message selection module. In Figure 5 we summarize the timing module. As shown, the timing module provides access to the local time and estimates of corrections between local clock and remote clocks. The utilities provide access to the timing information, allow updating of timing information, and comparison of timestamps from different processing sites. Let LTjin and LTkin denote

TimingModuleAtPi f

Information: Local Time: LTi = n(CTi ;
i ) 6= i Corrections: Cji = ((0d;ji0); Eji ) jj = i Utilities: ClockTime() f nd CTi from internal clock; return( CTi ) g LocalTime() f return( (ClockTime();
i ) ) g Correction(Pj) f return( (dji ; Eji ) ) g Wait(t) f (CTi ;
i ) = LocalTime() repeat (CTi0;
0i ) = LocalTime() until CTi0  CTi + t

g

g MaintainCorrections() f Cristain and Fetzer algorithm g LTj OP LTk f Let LTj = (CTj ;
j ) and LTk = (CTk ;
k ) (dji ; Eji ) = Correction(Pj); (dki ; Eki ) = Correction(Pk) case OP of < : return (ACTjimax ?
j < ACTkimin ?
k ) > : return (ACTjimin ?
j > ACTkimax ?
k )  : return (:(LTj < LTk _ LTj > LTk ))  : return (LTj < LTk _ LTj  LTk )  : return (LTj > LTk _ LTj  LTk ) endcase g

Figure 5: The Timing Module. two timestamps collected during the execution of an instrumented program and LTjo and LTko denote the corresponding execution times during the execution of the uninstrumented program. The comparison of local timestamps LTjin and LTkin using operators >, , Send msg to Pj g HandleReceive (M: Message) f Let M = < Sender = Pj ; SendTime = (CTj ;
j ); MsgContents = msgc > StartTime = ClockTime(); DetectFailure( M ); done = false; PTR = TPTR - 1;

repeat if HPTR=TPTR *** MQ was empty f MQ[HPTR] = M g elseif PTRM.SendTime

g

*** continue scanning MQ f MQ[PTR+1]=MQ[PTR]; PTR?? g else *** found position in MQ to insert M f MQ[PTR+1]=M; done=true g until done TPTR++; Intrusion due to HandleReceive,
recv = ClockTime() ? StartTime; Update intrusion time,
i =
i +
recv g

Figure 7: Communicator's Handling of Application Messages.

is encountered). This timing information, along with the time at which the message was received (MRT in), are required for updating the intrusion time. We assume that the communication time of a message is given by comm . Without Intrusion (tj,0) (ti,0) (a) M R To

≤

R T R To S MRTo

∆j

∆j

− ∆i ≤ x

g

Ready to receive time, RTRT = LocalTime()(= (CTi ;
i )) Observed waiting time, OWT = 0; while 6 9 message from Pi in MQ f Wait(1); OWT ++ g Get rst message from Pj in MQ say M and let M = < Sender = Pj ; SendTime = (CTj ;
j ); MsgContents = msgc > Message sending time, MST in = (CTj ;
j ) Message receiving time, MRT in = (CTj + comm ;
j ) Let intrusion time due to message selection routine be
det if MRT in  RTRT in f Fig 9a: No waiting would have occurred in original execution Updated intrusion time,
i =
i + OWT +
det g else f Fig 9b: Waiting for message would have been encountered in original execution Real wait time, RWT = MRT o - RTRT o = (ACTji + comm ?
j ) - (CTi ?
i ) Updated intrusion time,
i =
i + (OWT ? RWT ) +
det g return(msgc)

Figure 8: Deterministic Message Selection.

− ∆i > x

With Intrusion (tj, ∆ j )

(ti, ∆ i)

(tj, ∆ j )

(ti, ∆ i)

MSTin

MSTin S

S

RTRTin

MRTin RTRT

R

Intrusion Time = ∆ + ∆ i det

(b) M R To

>

R T R To

Intrusion Time = ∆ + OWT i

(tj, ∆ j ) MSTin S

det

RWT ∆i

RWT

With Intrusion

(ti, ∆ i)

− ∆j >

(tj, ∆ j )

R WT

(ti, ∆ i)

MSTin S

RTRTin R

∆

RTRTo MRTo R

− ∆j ≤

+

Without Intrusion (ti,0) (tj,0) S

∆i

OWT

MRTin R

in

MRTin

OWT

MRTin

Intrusion Time = ∆ − RWT + OWT i

DetReceiveAtPi(Pj : Remote Site) f in

x

RTRTo R

R

+

∆

det

Intrusion Time = ∆ − RWT i

RTRTin

+

∆

det

Figure 9: Updating Intrusion Time at Receiving Process. Let us discuss the method for updating intrusion time in greater detail. The various situations that may arise during message reception are shown in Figure 9. In Figure 9a the situation is considered in which the original execution of the receiving process would not have encountered any waiting at message receive, that is, MRT o is less than or equal to RTRT o . In this case, if no waiting is observed (that is, OWT is zero) the intrusion time of the receiver remains unchanged; otherwise it is increased by OWT since this waiting was caused by the delay in the message's arrival in the instrumented version. The situation shown in Figure 9b is one in which, in the original execution, the receiver would have encountered waiting at the receive, that is, MRT o is greater than RTRT o . The intrusion time of the receiver is reduced by this real waiting time (RWT). However, it must be increased by the observed waiting time (OWT) that is caused by the delay in the sending of the message to the receiver. In addition, we add an additional amount of time,
det, since the actions of the deterministic message reception routine take time and hence introduce

additional intrusion in the receiver. It should be noted that the intrusion time computed also has an error associated with it. This error arises because of variations in communication time (which was assumed to be comm ) and the error in ACTji (which was used to compute RWT) due to the lack of a global clock.

NonDetReceiveAtPi(P ) f

Ready to receive time, RTRT in = LocalTime() (=(CTi ;
i )) if 9 Pj 2 P such that there is no message from Pj in MQ f Wait(MAXWAIT) g WT = 0; done = false;

repeat ReadPosition( (CTi + WT;
i ) ); if 9 a message from some Pj 2 P in MQ f done = true g else f WT++ g until done

4.3.2 Non-Deterministic Message Reception

When a non-deterministic message reception statement is encountered by the application process, the non-deterministic message reception routine is invoked. The routine is provided with the subset of remote sites (P ) from which a message is to be selected non-deterministically. The message reception routine has to perform the following three tasks and is summarized in Figure 10: Construct message pool: Intrusion removal is performed to identify the appropriate pool of messages that should be considered. The message reception routine examines the message queue to determine if messages from each of the relevant remote sites are pending in the message queue. If there is a process from which no message is currently available, we wait for a time period of MAXWAIT allowing for any delayed messages to arrive. We assume that no message is delayed by more than MAXWAIT due to intrusion at this time. Next, we examine the messages in the queue and identify the ones which would have arrived before the non-deterministic reception statement would have been encountered in the original execution. These messages are ones whose receiving times are less than the time at which the non-deterministic message reception is executed, that is, MRT in < RTRT in. The routine ReadPosition identi es these messages. If messages from none of the senders is available, then we increment the time and check if any messages are now available. Finally, we will encounter a time at which at least one message would have been available and therefore the message reception can be performed. The above waiting that may be encountered is measured and is denoted by WT in the algorithm. Select a message and update failure detection information: When message selection is performed, if the message pool contains a single message then that message is returned to the application process. On the other hand, if multiple messages are in the pool then some heuristic is used to select the message which is returned to the application process. The time at which the non-deterministic selection is made is given by (CTi + WT;
i), where (CTi;
i) is the time at which the non-deterministic receiving routine was invoked and WT is the amount of waiting that is encountered. We will remember this time by calling the routine UpdateFailureDetectionInfo to later detect if the intrusion removal performed in the preceding step by waiting for MAXWAIT time was sucient or not. We will discuss failure detection in greater detail in the next subsection. Update intrusion time: The updating of intrusion time is based upon the waiting encountered at the non-deterministic receive. As was shown in the preceding subsection, waiting reduces the intrusion that was encountered before reaching the non-deterministic receive statement. After appropriately reducing the intrusion, we must increment it by the amount of time that is spent in the execution of the message reception routine which is denoted by
ndet in the algorithm.

Select a message M from MQ[HPTR..RPTR] using a heuristic UpdateFailureDetectionInfo(P ; (CTi + WT;
i )); Intrusion due to message selection routine,
ndet = ClockTime() ? (CTi + WT ) Update intrusion time: if
i < WT f
i =
ndet g else f
i =
i ? WT +
ndet g return(M.MsgContents) g

ReadPosition(RTRT in = (CTi ;
i )) f { setup RPTR according to RTRT in RPTR = nil; PTR = HPTR; done = false; while (PTR 6= TPTR) and (: done) f Message send time, MST in = MQ[PTR].SendTime = (CTj ,
j ) Message receive time, MRT in = (CTj + comm ,
j ) if MRT in < RTRT in f RPTR = PTR; PTR++ g else f done = true g g

g

Figure 10: Non-Deterministic Message Selection.

4.4 Failure Detection

In the previous section the non-deterministic message reception routine was constructed assuming that the arrival of a message from a remote site is delayed by no more than MAXWAIT due to intrusion. It is possible that the delay is greater and therefore insuf cient for the removal of intrusion. In order to handle this situation, we save timing information associated with non-deterministic actions and when a new message arrives at a site, we check to see whether this message would have been part of the message pool during a previous non-deterministic action. If this is the case, then we know that some of the intrusion wasn't removed. In such a situation we can increase the value of MAXWAIT with an appropriate amount and either reexecute the program or output an indicator that the monitoring information collected so far may be inaccurate. If a trace of monitoring information contains only occasional indications of failure to remove intrusion, we may regard this trace to be close to the real execution trace. However, if the failures are detected frequently we may reexecute the program and collected fresh traces. The algorithm for failure detection is summarized in Figure 11. The routine UpdateFailureDetectionInfo is invoked following a non-deterministic action. Each processing site, corresponding to each remote site Pj , maintains a variable Possiblej and time LRTjin. The variable Possiblej is true if it is possible that a future message from Pj could have aected the outcome

of a previously executed non-deterministic message. Thus, it is set to true if Pj was a valid candidate for message reception and no message was available from it during non-deterministic message selection. The variable LRTjin is the local time when the last nondeterministic action for which Pj was a valid candidate took place. The routine DetectFailure is invoked on a message reception to determine if the message received should have been waited upon during an earlier non-deterministic message selection. By comparing the time LRTjin with the time at which the newly arrived message was received (MATjin) we can determine if in the original execution the message would have arrived prior to the non-deterministic action and may have been selected. Once failure to remove intrusion is detected, the routine Restart is invoked to take appropriate action. FailureRestartModuleAtPi f

Information: Waiting Time: MAXTIME; Intrusion Detection Info: 8 Remote sites Pj f Possiblej ; LRTjin = (lrtj ; lr
j ) g Utilities: UpdateFailureDetectionInfo(P ; (CTi ;
i )) f for each process Pj 2 P f if 9 message from Pj in MQ finPossiblej = false g else f Possiblej = true; LRTj = (CTi;
i ) g

gg

DetectFailure ( M: message ) f Let M = < Sender = Pj ; SendTime = (CTj ;
j ); MsgContents = msgc > if Possiblej f Message arrival time, MATjin = (CTj + comm ;
j ) if LRTjin > MATjin f { intrusion has been detected AddWaitReqd = LRTjin ? MATjin = (lrtj ? lr
j ) ? (ACTji + comm ?
j ) MAXWAIT = MAXWAIT + AddWaitReqd Restart()

gg

g

Restart() f if the program is long running f output indicator to invalidate trace; continue program execution g elseif the program is short running f terminate processes; restart execution g g

Figure 11: Detection of Failure to Remove Intrusion and Restart Module.

4.5 An Example

We illustrate our technique by considering interactions between a server process (Ps) and two client processes (P1 and P2). The server receives requests from the clients and responds to them one at a time. We assume that the server contains an any-way branch to receive requests from the two clients and further we assume that the implementation chosen services the requests in the order they are received. This kind of

interaction arises in a number of situations. For example, the server may implement a token based mutual exclusion algorithm or the server may be a scheduler which distributes load among the clients. The purpose of monitoring such an interaction may be to assess the fairness of the server in its treatment of the requests from the two clients. In Figure 12a, the execution of the program without instrumentation is shown. We assume that the clients generate requests at the same rate and therefore the server alternately processes requests from the two clients. In Figure 12b, monitoring code is introduced in client P2. As a result the rate at which requests from P2 arrive is lowered. The execution under this situation is altered such that two requests from client P1 are processed for every request from client P2 that is processed. In Figure 12c, the execution following the removal of intrusion is shown. The server processes a request from client P1 rst. Next the server Ps waits for time MAXWAIT during which it rst receives a message from P1 and then from P2. By examining the timestamps on the received messages, Ps determines that if the intrusion time in P2 is accounted for then the message from P2 would have arrived before the message from P1. Therefore the messages are reordered and the request from client P2 is served next. As the remainder of the execution shows, the messages from the two clients are processed alternately as was the case in Figure 12a. Thus, our techniques are able to achieve an execution in presence of monitoring which is the same as the execution that would have occurred in absence of monitoring. In the preceding example the intrusion was successfully removed. However, if sucient waiting for delayed messages is not performed we may fail to remove intrusion. In the example in Figure 12d, the server processes a request from P2 rst and then from P1. After monitoring is introduced, as shown in Figure 12e, the message from P1 is processed rst because even after waiting the delayed message from P2 has not arrived. Finally when the message does arrive the failure is detected. The amount of waiting can now be increased and program can be executed again resulting in the execution with intrusion removal shown in Figure 12f.

5 Conclusion and Future Work

In this paper we described run-time techniques for minimizing the intrusive eects of monitoring on the likelihoods of various outcomes of a non-deterministic event. Previous non-intrusive monitoring techniques have not considered this form of intrusion since they focus on analyzing the characteristics of a speci c program execution. However, the monitoring of certain program characteristics, such as fairness in access to a shared resource and load distribution characteristics of a scheduler, relate to the likelihoods of selecting various legal executions rather than the timing characteristics of a speci c program execution. We developed algorithms for maintaining the estimates of intrusion times at all sites. Using these estimates the timings of various message passing events during the execution of the uninstrumented version of the program were obtained. By restoring the message pool and message ordering using these estimates, the intrusive eects of monitoring were minimized. We are currently exploring language support and static analysis techniques for reducing the overhead of intrusion removal techniques. As an example, consider the implementation of token based mutual ex-

P1

Ps

P2

S R

S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

R S R

P1

Ps

P2

S R

R S R

R S R

S

M O N I T O R

R S R

R S R

R S R

S

R S R

R S R

R

R S R

R S R

S R

S R

S R

S R S M O N I T O R

R S R

R S R

R S R

R S R

R S R

S R

(a)

Ps

Ps

P2

P2 R S R

R S R R

S R

R S R

R S R

S R M O N I T O R

M A X W A I T

S

R

(b) P1

P1

S

(d) P1

Ps

P2 M A X W A I T

S

S R S R

R S R

R S R

R S R

R

R

M O N I T O R

R S R

R S R

S failure

(e)

M A X W A I T

P1

M O N I T O R

R S R

R S R R

Ps

P2 M O N I T O R

M A X W A I T

S

S R S R

R S R

S

R M A X W A I T

S

R S R

R S R

M A X W A I T

R

(c) R

R S R

R S R (f)

Figure 12: Example-1: (a) Uninstrumented Execution; (b) Instrumented Execution without Intrusion Removal; (c) Instrumented Execution with Intrusion Removal. Example-2: (d) Uninstrumented Execution; (e) Failure in Intrusion Removal; (f) Successful Intrusion Removal.

clusion algorithm in a distributed system. The token manager can be implemented by creating two any-way branches and nesting them within a loop. The rst any-way branch non-deterministically assigns the token to any one of the requesting processes. The second any-way branch can be used to accept the token being returned. Even though the second any-way branch may appear to be non-deterministic, its actions are always deterministic since the token is assigned to a single process at any given time. By identifying anyway branches which are composed of mutually exclusive senders, the user can allow the overhead associated with non-deterministic message reception to be reduced. A language construct can be provided to indicate such cases. Static analysis techniques described in [10] can be used to identify subsets of processes and portions of processes during which the intrusion removal techniques are required. This approach can further reduce the overhead of intrusion removal techniques.

References

[1] K.M. Chandy and L. Lamport, \Distributed Snapshots: Determining Global States of Distributed Systems," ACM Trans. on Comp. Sys., 3:63-75, 1985. [2] F. Cristain and C. Fetzer, \Probabilistic Internal Clock Synchronization," 13th IEEE Symposium on Reliable Distributed Systems, Oct. 1994. [3] F. Cristain, \Probablistic Clock Synchronization," Distributed Computing, 3:146-158, 1989. [4] J.A. Gannon, K.J. Williams, M.S. Andersland, J.E. Lummp, and T.L. Casavant, \Using Perturbation Tracking to Compensate for Intrusion Propagation in Message Passing Systems," 14th IEEE-CS Int'l Conf. on Dist. Computing Sys., Poland, 1994. [5] L. Lamport, \Time, clocks and the ordering of events in distributed systems," Comm. of the ACM, 21(7):558-565, 1978. [6] C.A.R. Hoare, \Communicating Sequential Processes," Comm. of the ACM, 21(8):666-677, 1978. [7] A. Malony and D. Reed, \Models for performance perturbation analysis," ACM/ONR Workshop on Parallel and Distributed Debugging, pages 15-25, 1991. [8] A. Olson and K.G. Shin, \Fault-Tolerant Clock Synchronization in Large Multicomputer Systems," IEEE Trans. on Par. and Dist. Sys., 5(9):912-923, Sept. 1994. [9] S.R. Sarukkai and A. Malony, \Perturbation analysis of high level instrumentation for SPMD programs," 4th ACM SIGPLAN Symp. on Principles and Practices of Parallel Programming, pages 44-53, 1993. [10] M. Spezialetti and R. Gupta, \Perturbation Analysis: A Static Analysis Approach for the Non-Intrusive Monitoring of Parallel Programs," Int'l Conf. on Parallel Processing, Vol. II, pages 81-88, August 1994. [11] J.P. Tsai, K.F. Fang, and H. Chen, \A Noninvasive Architecture to Monitor Real-Time Distributed Systems," IEEE Computer, 23(3):11-23, 1990.