Optimism: Not Just For Event Execution Anymore - CiteSeerX

Published in the 13th Workshop on Parallel and Distributed Simulation, PADS-1999. c 1999, IEEE. Personal use of this material is permitted. However, permission to reprint or republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

Optimism: Not Just For Event Execution Anymore Christopher H. Young, Radharamanan Radhakrishnan, and Philip A. Wilsey Dept of ECECS, University of Cincinnati, PO Box 210030, Cincinnati, OH 45221{0030 (cyoung,ramanan,paw)@ececs.uc.edu

Abstract Optimism is a technique used by the Time Warp paradigm to make decisions about event execution under uncertainty. While the bene ts of throttling the optimism of Time Warp has been studied, the bene ts of extending optimism to operations besides event execution is presented in this paper. Speci cally, we discuss how optimism can be mapped to fossil collection which has traditionally been assumed to be a non-recoverable operation that must be performed under global control. Optimistic Fossil Collection (OFC) is a technique for fossil collection that does not require global control. However, as states are fossil collected optimistically, a recovery mechanism is required to recover from errors. Performance results show that Time Warp using OFC can improve the performance of simulations on a network of workstations. Another bene t of OFC is that the simulation checkpoints provide for fault tolerance in more than just fossil collection.

1 Introduction In a Time Warp simulation [11], each process ful lls the requirement of event execution in nondecreasing timestamp order through a rollback based recovery system. Each LP in a Time Warp simulation assumes that the execution of an event will not cause a causality error. A causality error occurs when an event with a lower timestamp arrives at an LP after it has executed an event with a higher timestamp. Time Warp allows for causality errors (i.e., it is optimistic ) because it uses a rollback based recovery scheme. Thus, optimistic simulation techniques are those techniques that are able to continue processing in spite of the uncertainty that another event could arrive with a timestamp less  Support for this work was provided in part by the Advanced Research Projects Agency under contracts DABT63{96{C{0055 and J{FBI{93{116.

than the next event to execute. This contrasts with the conservative mechanism [3, 7] that require blocking in the face of uncertainty until the uncertainty is removed by the arrival of more information either by waiting or by information collection. The null messaging optimization of conservative schemes is one method of collecting information to resolve uncertainty. Purely optimistic techniques, such as Time Warp, resolve this uncertainty using only currently available information. This paper shows how the optimistic paradigm can be extended to more than event execution. The optimism of Time Warp can be mapped to other operations as long as there is a state that can be stored/recreated and a method for removing side eects. This paper demonstrates this by mapping optimism to fossil collection in a Time Warp simulation. The bene ts of this is that optimism can be applied whenever it is desirable to reduce reliance on global knowledge, and provide localized control. This paper also shows that this is a viable alternative that can give good performance even in the presence of errors or faults due to incorrect decisions. The remainder of this paper is organized as follows. Section 2 describes the local and global control mechanisms in Time Warp. Section 3 formally describes the construction of the decision functions that implement the OFC scheme. One method of checkpointing and recovery for OFC is described in Section 4. Performance results for simulation models described in Section 5 are presented in Section 6. Finally, Section 7 presents some concluding remarks.

2 Time Warp's Control Mechanisms The Time Warp simulation protocol as de ned by Jeerson [11] is divided into a local control mechanism and a global control mechanism. Jeerson de nes them as follows: (a) the local control mechanism (LCM) deals with ensuring the proper ordering of event execution and message reception; and (b) the global control mechanism (GCM) is concerned with global issues

such as memory management, ow control, I/O, error handling and termination detection (all contributing to its \strong correctness"). This division of the protocol into two separate mechanisms is based upon the need to divide simulation operations into groups that \allow errors" (recoverable ) and those that \do not allow errors" (non-recoverable ). Operations that are not recoverable are performed using global knowledge while operations that can be check-pointed and recovered can be done optimistically using local knowledge. Thus, Time Warp allows event execution decisions to be based on local information exclusively because of the ability to recover from erroneous decisions. In this section, an investigation of the properties of the LCM that allow us to extend optimism to fossil collection is detailed. The LCM can be thought of as a decision function (which determines if the next event should be executed) and a rollbackbased recovery mechanism. The decision function always returns a \yes" unless optimism is being throttled. If optimism is being throttled then it is usually done based upon predictions that another message could arrive. These predictions tend to be based upon locally gathered information such as average timestamp increment and arrival time [6]. The rollbackbased recovery system is responsible for restoring the process state and removing all side eects from the system. This is done through the use of anti-messages to cancel the erroneous message(s) at each recipient and possibly rollback the recipient's computation. The GCM is a mechanism for ensuring that nonrecoverable operations are performed safely. Global knowledge is required because these operations must be done in an error free fashion (this can be considered the \conservative" portion of Time Warp because it operates similarly to event execution in a conservative simulation). Global knowledge comes from the estimation of the Global Virtual Time (GVT) which is the lower bound on the earliest timestamp at which an event can be executed in the system. Using this knowledge, we show how optimism can be applied to fossil collection. Fossil collection is an operation that normally requires global control because once memory has been identi ed as being fossils and fossil collected then there is no way of undoing this operation without check-pointing these fossils somewhere. The addition of LP check-pointing allows each LP to fossil collect without the requirement of a GVT estimation (thus allowing fossil collection to be managed locally instead of globally). The next section describes the mapping of optimism to fossil collection using a decision function and a rollback-based recovery scheme.

3 Optimism in Fossil Collection Optimistic Fossil Collection (OFC) is a fully distributed fossil collection model for Time Warp simulations [23]. Under OFC, each LP is responsible for making its own fossil collection decisions, without the bene t of global simulation information. Viewing the histories as a queue of entries ordered by simulation time-stamps, GVT represents a time-line that separates fossils from active histories. Traditional Time Warp identi es fossils by estimating GVT (using GVT estimation algorithms that calculate a virtual time that true GVT is guaranteed to have passed) and fossil collecting all fossils older than GVT. In contrast, OFC uses a local decision function to estimate the active history line. For example, each LP may examine event arrival times and create a statistical model of the expected variance from LVT. The model is then used to determine an estimate of the active history size X , such that the probability of a future rollback of a distance larger than X (an OFC fault) is some pre-speci ed risk factor, . It was demonstrated that such a decision function is successful in bounding the rollback distance for several example simulation models [23]. Accordingly, history entries with a time-stamp smaller than LV T ? X can be optimistically reclaimed. If a smaller  (risk factor) is chosen, the predicted active-history size X , becomes larger and OFC reclaims histories less aggressively. Other decision functions that use less rigorous heuristics can also be constructed [20]. The rollback behavior of each LP cannot be predicted exactly because it requires information about the states of other LPs; GVT is a global property that cannot be predicted precisely based only on the information available to the LP. Therefore, there is a nonzero probability that a history entry that was predicted to be fossilized (and was subsequently collected) will be needed in the future (an OFC fault). When this situation arises, the simulation cannot overcome the causality error using rollback because the required history information is not available. In order for OFC to be feasible, a recovery mechanism that allows the reconstruction of the erroneously collected history items must be implemented. The recovery mechanism restarts the simulation from a globally consistent state (a legally reachable state with correct message behavior [2]). The simplest possible recovery mechanism is to restart the simulation from the initial state. Thus, an OFC implementation consists of two largely independent mechanisms: the decision function (the model used to identify fossils), and the recovery mechanism. With the additional level of recovery, OFC-Time Warp becomes a two level recovery system; it incorporates:

(i) rollbacks to recover from erroneous optimism in computation, and (ii) a recovery mechanism to overcome erroneous optimism in fossil collection. The primary advantages of OFC are as follows:

 OFC eliminates the cost of GVT estimation for

fossil collection1. Depending on the implementation, GVT estimation overhead can be signi cant because it involves several rounds of communication to determine the lowest LVT time (allowing for messages in transit) and all the LPs in the simulation need to be noti ed of the newly estimated GVT [20].  OFC allows fossil collection to occur with no global implications in the presence of fault free behavior. In particular, in GVT-Time Warp (Time Warp using the traditional GVT estimation algorithms for fossil collection) the performance of all the LPs is aected by an LP that has poor memory behavior. If a single LP is memory constrained, it initiates GVT calculation frequently, causing a drop in the performance of all LPs as they participate in the GVT estimation. In contrast, OFC-Time Warp is fully distributed; each LP manages its own memory space independently of the other LPs, and can tailor its decision function to optimize fossil collection locally.

is greater freedom in implementing the decision function. In the remainder of this section, we identify two general classes of decision functions: (a) one that predicts future rollback distances by statistical analysis of previous rollback behavior; and (b) one that converges heuristically to an active history estimate.

3.1.1 Bounding the Rollback Distance Statistically

A purely optimistic decision function always assumes that it is all right to perform the operation in question. This is similar to event execution where the purely optimistic decision function assumes that it is always correct to execute the next event. Bounded time windows [1,15,19], throttled Time Warp [4,9,12] or any other method of bounding the optimism in event execution makes the decision function more complex. i.e., the decision function must now consider event timestamps and arrival times which may make the process block for some time. The method is still considered optimistic as there is the possibility of an erroneous decision by the decision function. There are a multitude of possible decision functions for OFC. The purely optimistic decision function (it is always safe to reclaim memory) is only viable in rollback free simulation. General purpose decision functions could adapt the optimism of the process based upon predicted behavior. It has been shown [20] that OFC is safe and live for any decision function that increases its estimate after an OFC fault. Thus, there

This type of decision function requires that each LP sample event arrival times and create a statistical model of the rollback behavior. Note that, at any given point, the longest rollback distance that can occur from the current LVT de nes the active history. The sampled rollback behavior is used to estimate a bound on the maximum rollback distance (the active history size)2 . More precisely, an estimated bound X , can be obtained such that the probability of a future rollback of a distance larger than X (an OFC fault) is some pre-speci ed risk factor, . Accordingly, history entries with a time-stamp smaller than LV T ? X can be optimistically reclaimed. If a smaller  (risk factor) is chosen, the predicted active-history size X , becomes larger and OFC reclaims histories less aggressively. The value of X can be expressed as a function of  if an underlying assumption on the rollback behavior is assumed. Consider the case where the length of a rollback is represented by a geometric distribution. Then the probability of a rollback of length L is given by P (X = L) = p(1 ? p)L?1, where p is the average rollback distance. Assuming a geometric distribution is reasonable since studies have shown that short rollbacks occur more frequently than long rollbacks [7]. The probability that a rollback of length X exceeds some distance L is given by P (X > L) = (1 ? p)L+1 : Thus, for a given p (obtained from sampling the rollback distances) and , solve for L such that P (X > L) <  and P (X > L ? 1) > . This analysis can be adapted for other distributions. Note that it is not cost eective to verify that the sampled rollback distances conform to the presumed distribution at run time. The discrepancy between the distribution and the actual behavior of rollbacks adversely aects the risk factor. However, it is possible to use the Chebyshev inequality [17] to provide an upper bound on X for a given  that is independent of the underlying distribution. The Chebyshev inequality provides an upper bound on the probability that a random variable will deviate a given distance from its mean . The bound is computed in terms of the

1 Real world interactions such as those in war-game simulations may still require GVT calculations.

2 This diers from GVT which de nes the bound for rollbacks that might occur.

3.1 Decision Functions in OFC

variance 2 and holds regardless of the original distribution F (provided the variance and mean are nite). The Chebyshev inequality gives the probability of the change in LVT exceeding L as 2 P fjX ? j  Lg  L 2

(1)

The bound is a function of both the variance and the distance from the mean. Once an independent sample of the rollbacks has been gathered, a con dence interval for the mean can be determined via the central limit theorem [17]. Empirical results have shown that the statistical bounds are highly successful for the models that were studied [20].

3.1.2 Converging on the Active History The liveliness proof [20] for OFC requires only that the decision function increase its estimate of the active history X when an OFC fault occurs. Starting with an initial estimate of X , heuristic decision functions increase X to f (X ), where f (X ) is monotonic. An example of such a decision function is to simply double the current estimate of X when an OFC fault occurs. A conservative initial estimate of X will eventually converge on an upper limit for the active history size as it suers OFC faults.

4 Check-pointing and Recovery As mentioned earlier in Section 3, a recovery mechanism is required to restart the simulation from a globally consistent state. A globally consistent state is a state that may occur in a valid execution of the simulation [2]. The initial state of the simulation is consistent, but recovery from the initial state requires throwing away all computation previous to the fault. In addition, guarantees on the liveliness of OFC can only be provided for this method of recovery if the active history size is allowed to grow arbitrarily large. Checkpointing the simulation and recovering from an intermediate checkpoint reduces the amount of lost computation. Check-pointing a distributed application requires saving the state of all processes and the state of all communication channels between processes. It is further required that the combination of all of these states form a meaningful and consistent checkpoint. A full de nition of checkpoint consistency is given by Chandy and Lamport [2]. Processes can coordinate during check-pointing to ensure that a checkpoint is consistent (a process called coordinated check-pointing [5])

or each process can take its local checkpoints independently of the other processes without synchronization (a process called uncoordinated or independent checkpointing [5]). The problem with independent checkpointing is that the consistency of the checkpoint is determined only during recovery. It is possible for all checkpoints to be inconsistent and the simulation is forced to restart from the initial state (this is called the domino eect [5]). In this section, we describe a check-pointing algorithm that produces globally consistent checkpoints using virtual time (the viability of this approach was recognized previously [13]). There are two main advantages for this algorithm: (i) it produces consistent checkpoints without coordination (i.e., it is dominoeect-free), and (ii) the size of the saved state is small. While the local state necessary to be consistent in real time includes all of the input, output, and state queues, only a minimal subset of the entries is required for our algorithm. Thus, the algorithm is fully-distributed, lightweight and domino-eect-free. The algorithm is lazy because it does not enforce consistency of the checkpoints as they are created; instead, the last consistent checkpoint is detected when a failure occurs. The algorithm is described in two parts: (i) check-pointing, and (ii) recovery. 4.1 Check-pointing

As each LP advances without an OFC fault, it checkpoints itself at pre-negotiated simulation times (similar to target virtual times [18]). This feature is the key to the algorithm's superiority to coordinated checkpointing algorithms; instead of coordinating in real time at a considerable overhead to realize a consistent state, static coordination in virtual time is established. The check-pointing steps are as follows:  LPs negotiate a simulation checkpoint interval,  ; this step could be carried out o-line, or infrequently.  A checkpoint is taken independently by each LP as it reaches the pre-approved checkpoint simulation time t; the checkpoint is taken at the last state before simulation time t. Note that if an LP rolls back past t, it will be check-pointed again when its simulation time reaches t again.  At a checkpoint, each LP saves the state before t and all messages with a sendtime < t and a receivetime  t. Note that no communication occurs other than in the rst step (the algorithm is distributed), and that only one state entry and a subset of the messages in the

output queue are saved at each checkpoint (i.e., the algorithm is lightweight). 4.2 Recovery

In the event of an OFC fault, the recovery mechanism is invoked. Because check-pointing is carried out in a distributed fashion, it is necessary rst to determine which checkpoint should be used for recovery. The recovery algorithm constructs a consistent checkpoint from the individual checkpoints by using the latest checkpoint that has been reached by all LPs (accounting for messages in transit). Some LPs may have surged ahead with their computation and may have taken checkpoints for virtual times that other LPs have not yet reached. It has been shown that recovery from this checkpoint is consistent if the messages in transit are drained, and the messages saved at each LP for this checkpoint are resent [23]. Determining the correct recovery checkpoint is the rst function of the recovery mechanism; it requires consensus among the LPs, in a manner similar to GVT estimation algorithms. The recovery method detailed in [22, 24] is brie y described here. A circulating token is used to stop the simulation and discard all state and event histories. The channels between the processes are drained using special tokens to ensure all messages in transit have been received3. A second round of token passing determines the checkpoint with the maximum timestamp that all LPs share in common. All processes recover from this checkpoint, re-send the checkpoint messages, and the simulation continues. 4.3 Checkpoint Consistency

The section details consistency proofs for the checkpoints made by the algorithm. Safety and liveliness properties of an OFC-Time Warp simulation have also been derived [20] but are not presented here.

Lemma 4.1 No messages are lost during checkpointing and recovery.

Proof: The checkpoints occur at the same simulation time t across all LPs. The checkpoint information taken consists of the last state before t and all messages with a sendtime < t and a receivetime  t. All messages with receivetime < t have already been delivered 3 The messages in transit can alternatively be detected and discarded without ushing the network. This scheme requires that each message be stamped with an incarnation number that is incremented every time the simulation is restarted after an OFC fault.

(necessary for the checkpoint to be consistent and follows from the ushing of the network). Messages with receivetime  t will be restored from the checkpoint if sendtime < t. Messages with sendtime  t will be recreated (the simulation is restarted at time t). Since all messages are accounted for, the lemma follows.

Lemma 4.2 No duplicate messages will be generated. Proof: Because the simulation at each LP is restarted from a point in its history, there is the danger of a message being recreated that are still in the network. These messages would be received multiple times. This behavior cannot occur because messages (from the previous incarnation) are drained out of the system before the simulation is restarted. The lemma follows.

Lemma 4.3 Checkpoints constructed by the algorithm are consistent. Proof: A checkpoint is consistent if the following two conditions hold: (i) the union of the individual LP states represents a valid global state; and (ii) messaging behavior is preserved; no messages are lost or duplicated. The proof that the rst condition holds follows from sequential simulation. More precisely, the Time Warp state recorded by each checkpoint is a sequential simulation state; the LP states are recorded at a single simulation time. Time Warp enforces only a partial order on events whereas sequential simulation enforces a total order; thus, every sequential state is a valid Time Warp state. It is now sucient to show that message behavior is correct. More precisely, no messages are lost or duplicated. These conditions are proven by Lemma 4.1 and Lemma 4.2 respectively.

5 Simulation Models PHOLD is a simulation benchmark developed by Fujimoto [8] that is based upon the Hold model used to test queue implementations. The PHOLD model consists of n processes and e events. The model used in this paper simulates 80 processes with 80 events for 10000 simulation time units. Events start uniformly distributed among the processes. The receiving process is chosen by a uniformly distributed random variable in the range from (0 : : : n ? 1) and the timestamp increment comes from a exponentially distributed random variable with a mean of 1. SMMP is a simulation of a queuing model of a 40 processor shared memory multiprocessor. The model is made up of 40 sub-models of processors with onchip caches that service a pre-de ned percentage of the memory requests. If a request is not serviced by the

Model

GVT Time (s) Memory (Kb) Period Avg. Max RAID 100 810 681 8046 RAID 1000 759 3160 8214 PHOLD 100 599 73 160 PHOLD 1000 578 175 414 SMMP 100 276 325 2351 SMMP 1000 216 328 2351 Table 1. Performance of GVT-TW

on-chip cache then the request is sent to another submodel of main memory. The simulation divides these 41 (40 processors cache modules + 1 main memory module) sub-models among 4 processors. The sources make requests that have a xed probability (60%) of being serviced by the cache or by the main memory. Cache requests are serviced locally by the server object on the same processor while main memory requests are serviced by a single server on last processor. RAID is a simulation of a Redundant Array of Inexpensive Disks implementing RAID level 5. Each processor of the simulation has 2 user processes that generate a read or write request for a strip of random length and location. The user processes will wait for the responses to this request to arrive before making a new requests. The requests from the user processes are fed into a RAID controller (which acts like a fork object in queuing simulations). The controller implements RAID level 5 using left symmetric parity placement. The controller splits this request into individual requests for speci c disks (Parity requests will be added if the request is for a write). The disks will service these requests and return the time the request was serviced.

6 Performance Results Five runs of each application with each fossil collector was run on network of Sun workstations using the warped [16] simulator. Memory usage and execution time was tracked for each run of each application and each fossil collector. These runs were averaged together to determine the results of interest. The values of interest were as follows: (a) time: the average time required to complete the simulation; (b) average memory usage: this is an average of the average memory usage for every processor (this single value represents how memory was utilized across all processors in the simulation); and (c)maximum memory usage: this represents the average of the maximum memory used for each run of the simulation.

Model

Chkpnt. Faults Time Mem. (Kb) Inter. (secs) Avg. Max RAID 20000 4.6 749 1178 3795 RAID 25000 4.6 769 1208 3699 RAID 50000 4 756 1052 3650 PHOLD 10 22 542 111 243 PHOLD 20 18 571 99 250 PHOLD 100 9.8 587 134 333 SMMP 10000 2.2 276 131 628 SMMP 20000 2.7 309 137 588 SMMP 30000 1.7 334 139 680 Table 2. Performance of OFC-TW (risk = 0.1)

Performance results for GVT were gathered using Mattern's algorithm [14] for two dierent GVT periods (100 and 1000). Mattern's algorithm was chosen for comparison because it is the fastest GVT estimation algorithm available in warped. In addition, in warped [16], GVT estimates were required only for fossil collection. The number of cycles between GVT estimations is a tunable parameter of warped. The GVT period used during the simulation in uences the performance of the simulator. A GVT period of 100 simulation cycles causes frequent cycles of GVT estimations in an attempt to reclaim memory aggressively. This leads to increased messaging overhead and decreases simulator performance. A larger GVT period allows the memory usage of the simulation to increase and possibly incur a performance penalty due to memory eects at the cost of reduced communication. Reducing the overhead of communication is desirable on a network of workstations where message sends can cost hundreds of microseconds. The results shown in Table 1 are as expected. Increasing GVT periods decrease the runtime of the simulation at the expense of increased memory size. A more complete analysis of the communication overhead for these simulations is found in the literature [20]. The total fossil collection overhead for GVT-Time Warp with a period of 100 is over three times the overhead for OFC-Time Warp (which includes the check-pointing overhead of OFC) for these simulations. Increasing the GVT period to 1000 reduces the overhead of GVT-Time Warp to twice the overhead of OFC-Time Warp. Furthermore, the latency of fossil collection for GVT-Time Warp (the time from invocation until fossil collection) ranged from 10 times to 151 times the latency in OFC-Time Warp. Both of these results are due to the lightweight nature of check-pointing/fossil collection of OFC-Time Warp. The Chebyshev model was run with two dierent levels of aggressiveness (0.1 and 0.01) and a range

of three checkpoint intervals (see Tables 2 and 3) to compare the performance of OFC-Time Warp against GVT-Time Warp. Parameters for the decision function were determined by sampling. A sample size of 32 rollbacks was used to determine the sample mean and variance of the rollback length. Once the rollback length (L) was determined dynamically then memory was fossil collected whenever the next event to process on the processor was more than L  1:5 time units from the virtual time of the last fossil collection. This allowed memory to accumulate and allowed more memory to be fossil collected in larger chunks instead of fossil collecting small chunks of memory frequently. Fossil collection decreased the average memory usage but increased the overhead. Hence, a length of 1:5L was used as a heuristic [20]. The memory requirements of OFC-Time Warp are determined by the decision function and the level of aggressiveness. Aggressive fossil collection by the Chebyshev model in OFC-Time Warp (represented by a level of aggressiveness  = 0:1) gives an average and maximum memory bound less then GVT-Time Warp for a GVT period of 1000. Lower memory bounds are achieved with GVT-Time Warp using a GVT period of 100 but at the cost of GVT thrashing. It is possible for OFC-Time Warp to bound memory more tightly than GVT by using local information to determine the memory bound. Young [24] describes how the communication properties of the simulation allow some processes to reclaim more memory in the simulation than GVT would allow (e.g., rollback free processes do not require any history information regardless of the current GVT value). This explains how SMMP achieves a better bound under OFC-Time Warp than GVT-Time Warp. An increased execution time occurs because more frequent calls are made to fossil collection. The number of fossil collections for OFC-Time Warp is similar to the number of calls for GVT-Time Warp with the more aggressive GVT period. If OFC-Time Warp were triggered by a memory bound instead of the virtual time length then the number of fossil collections would be reduced and its performance would improve. A side-eect of tighter memory bounds (by being more aggressive) is the increased probability of faults due to over aggressiveness. Decreasing the level of optimism (from  = 0:1 to  = 0:01) leads to less faults in the simulation and increased memory usage (see Table 3). The expense of lost computation due to OFC faults in the simulation is determined by the checkpoint interval of the simulation. Smaller checkpoint intervals give better performance for frequently faulting processes; while larger checkpoint intervals are favored by infrequently faulting processes. In addition, the

Model

Chkpt. Faults Time Mem. (Kb) Inter. (secs) Avg. Max RAID 20000 0 737 2740 4858 RAID 25000 0 734 3163 5078 RAID 50000 0 749 3035 4914 PHOLD 10 1 574 275 391 PHOLD 20 0.6 555 229 339 PHOLD 100 0.8 547 242 347 SMMP 10000 0.5 233 228 618 SMMP 20000 0 233 236 680 SMMP 30000 0 229 254 721 Table 3. Performance of OFC-TW (risk = 0.01)

checkpoint interval aects the number of erroneously processed events in the simulation. For example, the RAID application using  = 0:01 suered a three percent increase in the number of erroneously executed events when the checkpoint interval was raised from 20000 to 50000. The current implementation of OFCTime Warp ignores this eect and check-points/fossil collects based solely upon virtual time.

7 Conclusions This paper presented a method for extending the optimistic protocol of Time Warp to fossil collection. The result is a distributed technique for fossil collection that operates in spite of uncertainty over the true state of the history items (the consensus of GVT is not required). Each LP decides which history items are likely to be fossilized using a local decision function and proceeds to collect these items optimistically. A perfect decision function would converge on the active history for each LP. The active history is the minimum subset of history items that is sucient to recover from rollbacks that occur during the simulation interval (a more precise de nition of active history is given by Young [20]). The rollback based recovery method of Time Warp is used to recover from over optimism in fossil collection. Lightweight consistent checkpoints were created on-the- y. These checkpoints, consistent though uncoordinated in real time, signi cantly reduce the check-pointing overhead by only saving state and event information that would exist in a sequential simulation at a speci c virtual time. The performance of OFC-Time Warp was shown to be on par with GVTTime Warp for several simulations and dierent GVT periods on a network of workstations. We anticipate that further re nement to our decision and recovery algorithms will enhance the performance of OFC [20,21]. We also anticipate performance improvements from the

application of adaptive protocols. Most importantly, optimism should now be seen as a general purpose technique to establish local control over simulator operations that normally require global information. Hybinette [10] has recently presented another example of extending optimism by using optimistic I/O to precompute scenes in virtual environments. The only requirement for optimism is the ability to recover from errors by the decision function. The algorithms in this paper should be seen only as one possible approach for applying optimism to simulator functions other than event execution. The bene ts of optimism using a good decision function will be decreased overhead and blocking that can occur while acquiring global knowledge to resolve uncertainty.

References [1] D. Ball and S. Hoyt. The adaptive Time-Warp concurrency control algorithm. In Distributed Simulation, pages 174{177. Society for Computer Simulation, Jan. 1990. [2] K. M. Chandy and L. Lamport. Distributed snapshots: Determining global states of distributed systems. ACM Transactions on Computer Systems, 3(1):63{75, Feb. 1985. [3] K. M. Chandy and J. Misra. Asynchronous distributed simulation via a sequence of parallel computations. Communications of the ACM, 24(11):198{206, Apr. 1981. [4] S. R. Das and R. M. Fujimoto. An adaptive memory management protocol for Time Warp parallel simulation. In Sigmetrics, pages 201{210, May 1994. [5] E. Elnozahy, D. Johnson, and Y. Wang. A survey of rollback-recovery protocols in message-passing systems. Technical Report Tech. Rept. CMU-CS-96-181, School of Computer Science, Carnegie Mellon University, oct 1996. [6] A. Ferscha. Probabilistic adaptive direct optimism control in time warp. In Proc. of the 9th Workshop on Parallel and Distributed Simulation (PADS 95), pages 120{129, June 1995. [7] R. Fujimoto. Parallel discrete event simulation. Communications of the ACM, 33(10):30{53, Oct. 1990. [8] R. Fujimoto. Performance of time warp under synthetic workloads. Proceedings of the SCS Multiconference on Distributed Simulation, 22(1):23{28, Jan. 1990. [9] D. O. Hamnes and A. Tripathi. Investigations in adaptive distributed simulation. In Proc. of the 8th Workshop on Parallel and Distributed Simulation (PADS 94), pages 20{23. Society for Computer Simulation, July 1994. [10] M. Hybinette and R. Fujimoto. Optimistic computations in virtual environments. In Proceedings of the Virtual Worlds and Simulation Conference (VWSIM'99), pages 39{44, January 1999.

[11] D. Jeerson. Virtual time. ACM Transactions on Programming Languages and Systems, 7(3):405{425, July 1985. [12] Y.-B. Lin and B. R. Preiss. Optimal memory management for Time Warp parallel simulation. ACM Transactions on Modeling and Computer Simulation, 1(4):283{307, Oct. 1991. [13] F. Mattern. Virtual time and global states in distributed systems. In Proceedings of the Workshop on Parallel and Distributed Algorithms, pages 215{226, Oct. 1989. [14] F. Mattern, H. Mehl, A. A. Schoone, and G. Tel. Global virtual time approximation with distributed termination detection algorithms. Tech. Rep. RUU{ CS{91{32, Dept. of Computer Science, University of Utrecht, The Netherlands, 1991. [15] A. Palaniswamy and P. A. Wilsey. Adaptive bounded time windows in an optimistically synchronized simulator. In Third Great Lakes Symposium on VLSI, pages 114{118, 1993. [16] R. Radhakrishnan, D. E. Martin, M. Chetlur, D. M. Rao, and P. A. Wilsey. An Object-Oriented Time Warp Simulation Kernel. In Proceedings of the International Symposium on Computing in Object-Oriented Parallel Environments (ISCOPE'98), volume LNCS 1505, pages 13{23. Springer-Verlag, Dec. 1998. [17] S. M. Ross. Introduction to Probability Models. Academic Press, San Diego, CA, 4th edition, 1989. [18] A. I. Tomlinson and V. K. Garg. An algorithm for minimally latent global virtual time. In Proc of the 7th Workshop on Parallel and Distributed Simulation (PADS), pages 35{42. Society for Computer Simulation, July 1993. [19] S. J. Turner and M. Q. Xu. Performance evaluation of the bounded Time Warp algorithm. Proceedings of the SCS Multiconference on Parallel and Distributed Simulation, 24(3):117{126, 1992. [20] C. H. Young. Methods for Optimistic Reclamation of Fossils in Time Warp Simulation. PhD thesis, University of Cincinnati, February 1999. (expected). [21] C. H. Young, N. B. Abu-Ghazaleh, R. Radhakrishnan, and P. A. Wilsey. Performance bene ts of optimism in fossil collection. In J. Ralph H. Sprague, editor, 32nd Hawaii International Conference on System Sciences (HICSS-32), Jan. 1999. [22] C. H. Young, N. B. Abu-Ghazaleh, and P. A. Wilsey. OFC: A Distributed Fossil-Collection Algorithm for Time-Warp. In 12th International Symposium on Distributed Computing, DISC'98 (formerly WDAG), Sept. 1998. [23] C. H. Young and P. A. Wilsey. A distributed method to bound rollback lengths for fossil collection in Time Warp simulators. Information Processing Letters, 59(4):191{196, Aug. 1996. [24] C. H. Young and P. A. Wilsey. Optimistic fossil collection for Time Warp simulation. In H. El-Rewini and B. D. Shriver, editors, 29th Hawaii International Conference on System Sciences (HICSS-29), volume Volume I, pages 364{372, Jan. 1996.