External Adjustment of Runtime Parameters in Time Warp ... - CiteSeerX

Published in the Proceedings of the International Parallel Processing Symposium, IPPS-1997. c 1997, IEEE. Personal use of this material is permitted. However permission to reprint or republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

External Adjustment of Runtime Parameters in Time Warp Synchronized Parallel Simulators Radharamanan Radhakrishnan, Lantz Moore, and Philip A. Wilsey Dept. of ECECS, PO Box 210030, Cincinnati, OH 45221–0030

Abstract Several optimizations to the Time Warp synchronization protocol for parallel discrete event simulation have been proposed and studied. Many of these optimizations have included some form of dynamic adjustment (or control) of the operating parameters of the simulation (e.g., checkpoint interval, cancellation strategy). Traditionally dynamic parameter adjustment has been performed at the simulation object level; each simulation object collects measures of its operating behaviors (e.g., rollback frequency, rollback length, etc) and uses them to adjust its operating parameters. The performance data collection functions and parameter adjustment are overhead costs that are incurred in the expectation of higher throughput. This paper presents a method of eliminating some of these overheads through the use of an external object to adjust the control parameters. That is, instead of inserting code for adjusting simulation parameters in the simulation object, an external control object is defined to periodically analyze each simulation object's performance data and revise that object's operating parameters. An implementation of an external control object in the WARPED Time Warp simulation kernel has been completed. The simulation parameters updated by the implemented control system are: checkpoint interval, and cancellation strategy (lazy or aggressive). A comparative analysis of three test cases shows that the external control mechanism provides speedups between 5%-17% over the best performing embedded dynamic adjustment algorithms.

1 Introduction The Time Warp parallel synchronization protocol has been the topic of research for several years [12]. However, the successful utilization of the Time Warp mechanism has been plagued by the time and space overheads of rollback such as: state saving, state restoration and event re-processing. Numerous modifications/optimizations to Time Warp have been proposed and analyzed [1, 4, 8, 10, 15, 19, 23, 29]. Many of these studies have focused on techniques to dynamically adjust a simulator's runtime parameters. It has been established that dynamic cancellation [26] and dynamic checkpointing [22] have resulted in better performance than their static counterparts [11, 28, 26]. However, these dynamic optimizations come with a cost and they have only suc Support for this work was provided in part by the Advanced Research Projects Agency under contract J–FBI–93–116, monitored by the Department of Justice.

ceeded in marginally lowering the overheads of Time Warp (improvements on the order of 10% are common). Part of the problem is the fact that the logic needed in dynamically determining which optimization to use is part of the code executed by a simulation object. This logic manifests itself in the source code in two parts: (i) a performance data collection phase that generally executes with each action performed by the simulation object; and (ii) a parameter adjustment phase that occurs less often (for example after processing k events). If not properly controlled, the overhead due to statistics collection and parameter adjustment can cause an overall slow down in a simulation [6, 11, 25]. Attempts to reduce the overhead of dynamic parameter adjustment fall into two categories: (i) a reduction in the frequency or duration of adjustment [17, 26] or (ii) a reduction in the complexity of the performance data and control functions [11, 28]. In this paper, we present and analyze a technique for dynamic control that removes the dynamic adjustment phase from the core simulation object module and instead executes it as an agent external to the simulation object. Thus each simulation object only collects performance data — parameter adjustment occurs in a separate object. The motivation for this removal is overhead reduction; in the simple case we have only removed an if test (if time to adjust then perform analysis and update parameters). While this may at first seem a trivial reduction, the replication of the code across each simulation object and the small computational grain size generally found in simulation objects magnify the effects. Furthermore, this restructuring allows a much broader spectrum of possibilities of optimizations; such as enabling a dynamically altering frequency at which the parameter adjustment phase itself is conducted. The implementation of external control described in this paper is made in the WARPED Time Warp simulation kernel [18]. In the implementation presented herein, the adjustment of two parameters is studied, namely: (i) the frequency of checkpointing [11, 28], and (ii) the cancellation strategy (lazy or aggressive) [26]. The external control function is triggered by a single token that is circulated through the simulation objects (in much the same manner as the token in Mattern's GVT estimation algorithm [19]). Performance data is collected for three different simulation test cases to compare the runtime performance. The configurations that are compared include: (i) static parameter settings, (ii) traditional dynamic parameter adjustment, and (iii) dynamic parameter adjustment by an external object. The results on a 4 processor SMP SUN workstation show that the external control mechanism provides speedups between 5%-17% over the best performing embedded dynamic adjustment algorithms.

The remainder of this paper is organized as follows. Section 2 reviews the basic operation of the Time Warp mechanism and describes the optimizations of periodic checkpointing, lazy and aggressive cancellation, and the dynamic configurations of each. Section 3 presents a high level overview of the WARPED Time Warp simulation kernel [18] and applications used in these experiments. Section 4 highlights the costs of dynamic parameter adjustment using profiled data from the test environment. Section 5 reviews the the issues of external control of dynamic parameters. Performance results are shown in Section 6. Finally, Section 8 contains some concluding remarks.

2 Background A parallel simulation with distributed synchronization is generally organized as a set of simulation objects interacting with each other by exchanging time-stamped event messages [12]. These communicating objects are referred to as Logical Processes or LPs. The Time Warp mechanism is an optimistic synchronization protocol based on the virtual time paradigm [12]. In a Time Warp simulation, no explicit synchronization occurs between the logical processes. The lack of explicit synchronization permits the parallel simulators to advance their local simulation times at different rates. Consequently, the possibility exists for incoming event messages to arrive with time-stamps in the simulated past of the receiving simulator. Such messages are called stragglers (or straggler messages) and receipt of a straggler message forces the simulator to rollback to an earlier time to process the straggler in its proper order. In a Time Warp simulator, each LP operates as a distinct discrete event simulator, maintaining input and output event lists, a state queue, and a local simulation time (called Local Virtual Time or LVT). The state and output queue are needed to support rollback processing. That is, on receipt of a straggler message, the LP must rollback to undo some work that has been done. Rollback involves two steps: (i) restoring the state to a time preceding the time-stamp of the straggler and (ii) canceling any output event messages that were erroneously sent (by sending anti-messages). After rollback, the events are then re-executed in their proper order. One important overhead associated with checkpointing state and event information for rollback is the memory space required for the saved data. This space can be freed only when global progress of the simulation advances beyond the (simulation) time at which the saved information is needed. The process of identifying and reclaiming this space is called fossil collection. The global time against which fossil collection algorithms operate is called the Global Virtual Time (or GVT) and several algorithms for GVT estimation have been proposed [5, 9, 16, 19]. In addition to its use for fossil collection, GVT is also useful for deciding when irrevocable operations can be performed and, in some instances, when the simulation has completed.

2.1

Periodic Checkpointing

Memory consumption is a potentially onerous problem for Time Warp simulations. Time Warp objects with large states require considerable memory space as well as CPU cycles for state saving. In general, states are saved after every event execution in a Time Warp simulation. However, it is possible to save state pe-

riodically, after a certain number of fixed event executions. Thus using a periodic state saving policy, the arrival of a straggler message may require the system to rollback to an earlier state than necessary and coast forward, reconstructing the state required to correctly execute the straggler event in its proper order [12]. While coasting forward, no messages are sent out to the other processes in the system. The difficulty of periodic state saving is determining an appropriate fixed frequency for checkpointing. Some applications operate best with a fairly small value; while others require much larger values [6, 25]. Currently, no practical techniques for statically analyzing simulation applications to decide the checkpoint frequency are known [14]; hence motivating investigations into dynamically adjusting the checkpoint interval.

2.2

Dynamic Checkpointing

There have been a variety of proposals to dynamically adjust the checkpoint interval [11, 17, 21, 28]. These techniques all employ an adaptive control mechanism [3] to dynamically establish values for the checkpoint interval. As with any control system, dynamically adjusting the simulators control parameters requires that several output values be monitored [3]. While the particular output values monitored by each proposed method vary, several of them are shared [11]. The WARPED simulation kernel can be configured to execute three of the control strategies for dynamic checkpointing: Lin's model [17], Palaniswamy's model [21] and Fleischmann's model [11]. In the comparisons performed in this paper, Fleischmann's model of control is always used. Fleischmann's model is selected because it has given us the best overall performance for all the applications that we have studied.

2.3

Lazy and Aggressive Cancellation

The performance of a Time Warp simulator depends highly on the efficiency of the cancellation strategy employed to undo the effects of erroneous computation (the sending of antimessages). Two known cancellation strategies exist [12], namely: aggressive cancellation, and lazy cancellation. Under aggressive cancellation the arrival of a straggler message and the subsequent rollback forces the immediate generation of anti-messages for all output messages that were sent prematurely. In contrast, a lazy cancellation policy delays sending anti-messages until forward processing demonstrates, by comparison of old and new output, that the original output messages were incorrect. Thus, there is a potential reduction in communication as well as a decrease in wasted optimistic computation. Lazy cancellation relies on the regeneration of the same output messages for its performance. The performance under lazy cancellation deteriorates if the probability of the regenerated output messages being different from the original messages is high. In contrast, aggressive cancellation performs poorly if the same messages are generated before and after a rollback. Independent studies have shown that lazy cancellation sometimes performs better than aggressive cancellation, but that, even within the same application domain, some executions perform better under aggressive cancellation [4, 27]. Unfortunately, practical techniques to statically analyze an application for selecting cancellation strategies have yet to be developed [14]. Consequently, most Time Warp simulators implement both strategies and leave it to the user to select which cancellation strategy to use in their particular simula-

cancellation. Upper Threshold Dead Zone Lower Threshold

Figure 1. Thresholding for Cancellation tions. Recently, however, a technique for dynamically selecting the cancellation strategy has been investigated [26].

2.4

Dynamic Cancellation

Dynamic cancellation is a technique where each Time Warp object decides for itself which cancellation strategy to employ [26]. That is, each Time Warp object analyzes its recent history for evidence of success in lazy cancellation. A lazy hit/miss ratio is established and used to decide which cancellation strategy to employ (thus lazy comparisons during aggressive cancellation are also required). The decision to switch is made at the Time Warp object level and is made by an analysis of the hit/miss ratio using various control functions [26]. The hit/miss ratio will here on be referred to as the lazy hit/miss ratio. When a rollback occurs and a new event generated, it is first compared to a copy of the erroneously sent event. If they are the same, then a lazy hit is said to have occurred. If they are not the same, then a lazy miss is said to have occurred. The ratio of lazy hits to lazy misses is denoted by the lazy hit/miss ratio. While Rajan explored several different control strategies [26], one of the more successful used a thresholding function with a dead zone to select the cancellation strategy (Figure 1, where lazy hit/miss ratio is the y axis and time is the x axis). More precisely, all Time Warp objects begin by using aggressive cancellation, collecting lazy hit/miss data. After a sufficiently long measurement cycle, the lazy hit/miss ratio is used to select cancellation strategies; if the value falls below a lower threshold, aggressive cancellation is used; changes from aggressive to lazy occur when the ratio crosses an upper boundary; and values in the dead zone remain unchanged [26]. The setting of the upper and lower bounds requires some tuning based on the specific time warp simulator and its implementation [26]. Alternative techniques are also investigated by Rajan [26]. In particular three basic approaches for dynamic control were identified:

 Various configurations with one, two, or three thresholds: (i) one threshold strictly decides the cancellation strategy, (ii) two thresholds provide a buffer or dead zone as described above, and (iii) three thresholds allow the establishment of very low threshold to enable a permanent switch to aggressive cancellation (thereby eliminating the overhead of lazy comparisons in aggressive cancellation).  After k lazy comparisons, a permanent binding to a cancellation strategy is made. This strategy is similar Lin's approach for dynamic cancellation [17].  Rollback using lazy cancellation and switch to aggressive after p consecutive misses. Each new rollback begins with lazy

In the comparisons performed in this paper, the two threshold model with a lower threshold value of .4 and an upper threshold value of .6 is always used. This model and the specific thresholds were used because they give the best performance for all the applications studied.

3 The Experimentation Environment The WARPED simulation kernel provides the functionality to develop applications modeled as discrete event simulations [18]. Considerable effort has been made to define a standard programming interface to hide the details of Time Warp from the application interface. All Time Warp specific activities such as state saving, rollback, and so on, are performed automatically by the kernel without intervention from the application. Consequently, an implementation of the WARPED interface can be constructed using either conservative [20] or optimistic [7, 12] parallel synchronization techniques; furthermore, the simulation kernel can also exist as a sequential kernel. In fact, the current software distribution of WARPED includes both sequential and parallel (Time Warp) implementations. As previously indicated, this independence is achieved through a standard application programming interface (API). In the WARPED Time Warp simulation kernel objects are grouped into entities called logical processes, or LPs (Figure 2). Processor parallelism occurs at the LP level and each LP is responsible for GVT management, communication management, and scheduling for the simulation objects that it contains. In addition, communication between simulation objects within the same LP is performed by direct insertion into the input queue of the receiving object. Since the parallelism occurs at the LP level, simulation objects that execute relatively independently of each other can be placed on separate LPs to maximize parallelism. Conversely, simulation objects that frequently communicate with each other should be placed on the same LP to benefit from fast intra-LP communication. The WARPED system is composed of a set of C++ class and template libraries which the user accesses in several ways. Where the kernel needs information about data structures within the application, they are passed into kernel template classes. When kernel data or functions need to be made available to the user, they can be accessed by one of two mechanisms: inheritance and method invocation. A more detailed description of the internal structure and organization of the WARPED kernel is available on the www at http://www.ececs.uc.edu/˜ paw/warped.

3.1

Applications

To observe the effect of the various optimizations studied in this paper, a series of tests are conducted on two application domains: (i) queuing model simulations, and (ii) simulation of digital systems described in the hardware description language VHDL [2, 24]. A description of these two applications are given below.

3.2

Application: Queuing Model Simulation

Included in the WARPED distribution is a simulation library for building queuing models. This library is called KUE. In addition

Simulation Object

Simulation Object

Simulation Object

Simulation Object Simulation Object

Simulation Object Logical Process

Logical Process

Simulation Object

Simulation Object Simulation Object

Simulation Object

Simulation Object

Simulation Object

Simulation Object

Simulation Object

Simulation Object

Simulation Object Simulation Object

Simulation Object Logical Process

Logical Process

Simulation Object

Simulation Object Simulation Object

Simulation Object

Simulation Object

Simulation Object

MPI Communication Direct Communication

Figure 2. Structure of LPs and Simulation Objects in the to the library, the distribution includes three instances of use of KUE. For purposes of this paper, we consider two of the three models. The two example queuing models are:

SMMP: modeling a shared memory multiprocessor. Each processor is assumed to have a local cache with access to a common global memory. For the experiments in this paper, a 16 processor machine with settings as follows: a 10ns cache, a 100ns main memory, and a cache hit ratio of 90% was chosen for simulation. As the modeling was done in terms of queuing model objects, there was 100 queuing objects in this simulation. Each event in this simulation represents a request for memory access from a processor to a common global memory.

RAID: model of a nine disk RAID level 5 disk array of IBM 0661 3.5” 320Mb SCSI disk drives with flat-Left symmetric parity placement policy. Sixty processes send requests for stripes of random lengths and location to forks which split each the requests into individual requests for each disk according to the placement policy. Thirty servers process the requests in a FCFS fashion and route the appropriate requests back to the source process. This simulation had a total of 94 queuing objects to model the RAID disk array. Every event in this simulation carries information about stripe lengths and location so that the servers may carry out the appropriate stripe retrieval.

3.3

Application: VHDL Simulation

The second application studied is digital system simulation. In particular, we have developed a VHDL simulation kernel [13] that operates on top of the WARPED kernel. A VHDL code generator is currently under development. Consequently, we needed a simple technique for translating moderately sized VHDL examples into executable code to link with the VHDL simulation kernel. The ISCAS benchmarks provide a good example of moderately

WARPED

System

sized VHDL codes that are regular and easy to translate with a perl script. In particular, the c499 benchmark from the ISCAS 85 benchmark suite will be used in this study. The c499 benchmark was chosen because it has the largest model that is currently operating on the WARPED kernel(165 time warp objects). Each gate in the benchmark is modeled as a time warp object and communication between these objects take place through the sending and receiving of events. Events in this simulation model signals which are sent and received between gates in the circuit. All the tests with c499 were run with a common set of 10,000 input vectors.

4 Costs of Dynamic Adjustment Adaptive control techniques and methods have been a topic of research with simulator implementors. Adaptive control theory presents us with a host of solutions to work with and utilize. Adaptive or dynamic control traditionally had been used in checkpointing algorithms. Recently, it has also been used for dynamically switching between lazy and aggressive cancellation strategies. Dynamic parameter adjustment is an attempt to add overhead for the expectation of gaining a more optimal configuration and ultimately higher throughput. In order to study the effect of these dynamic optimizations, we instrumented the three example simulation applications described above. The aim of these experiments is to quantify the cost of dynamic optimizations. The quantified (normalized) results for the dynamic optimizations are presented in Figure 3. There are three possible dynamic optimizations possible, namely: Dynamic Checkpointing, Dynamic Cancellation and both Dynamic Cancellation and Dynamic Checkpointing together. From the figure it is apparent that each optimization has a fixed overhead and they are additive in nature. The more number of dynamic optimizations you have, the more the overhead. Also this overhead is constant throughout the simulation.

Cost of Dynamic Optimizations

1.8

Performance Results

Dynamic Checkpointing Dynamic Cancellation Dynamic Cancellation & Checkpointing

1.6

1.5

1.7

1.4 Normalized Performance

1.6

Normalized Cost

Aggressive Cancellation & Periodic Checkpointing Lazy Cancellation & Periodic Checkpointing Dynamic Cancellation & Checkpointing(internal control) Dynamic Cancellation & Checkpointing(external control)

1.5

1.4

1.3

1.3

1.2

1.1

1.2 1.0 1.1 0.9

1.0

0.9

RAID

SMMP Applications

ISCAS2

0.8

RAID

SMMP Applications

ISCAS2

Figure 3. Cost of Dynamic Optimizations(normalized against Dynamic Checkpointing)

Figure 4. Performance Results (normalized against Aggressive Cancellation & Periodic Checkpointing

Hence it is imperative that all dynamic optimizations are carefully tested and quantified to minimize the overhead when applications are accessing these optimizations frequently, during the course of the simulation. Also it is important to keep track of these optimizations and ensure that they are not being called too frequently. Ideally the presence of a central parameter adjustment process which sets the parameters dynamically for each object with minimal overhead is desirable.

ject examines that performance data gathered by each Time Warp object, adjusts the Time Warp parameters (checkpoint interval and cancellation strategy), and resets the performance data indicators. After processing 1000 events, the token is forwarded to the next LP for processing. Currently a round robin scheme of forwarding the token has been implemented. Further analysis and experimentation may lead to different forwarding schemes.

5 External Parameter Adjustment As mentioned in the previous section, there is a certain fixed amount of overhead present in the dynamic optimizations to Time Warp. These overheads are avoidable. This section proposes one such strategy to improve the performance of dynamic optimization to Time Warp. Associated with each optimization is a cost for performance analysis and parameter adjustment. This cost cannot be eliminated, but it can be reduced. External control achieves this reduction. It removes some code from each Time Warp object and adds an additional processing object (control object) to each LP. This control object can be triggered in many ways (Section 7). The technique described in this paper uses a token to trigger the control object. Initially one of the LPs is selected at random to be the initiator. This LP initiates an adjustment phase by dispatching a token to its nearest neighbor.1 The neighboring LP is triggered by the incoming token to invoke the control object. The control ob1 Note

that the frequency with which this initiator LP sends this token is a parameter which is decided dynamically based on past adjustments. In the experiments presented in this paper, waiting for 1000 events to be processed, before forwarding the token, gave the best results.

6 Performance Results This section details the experiments conducted to characterize the performance and effectiveness of the optimization techniques in Section 2. To see the effect of the various optimizations presented earlier, a series of tests were conducted on two application domains. All the studies have been conducted using a network of SUN workstations and the data provided in this paper has been collected by running the simulations on a 4 processor Sun SparcCenter 1000 (running Solaris 2.5). In the following studies, the algorithms were tested with fixed configurations of 4 LPs (one for each of the processors in the multiprocessor). The results have been normalized to reflect the considerable gain achieved by introducing the simple external control object. The results are summarized in Figure 4. The SMMP benchmark processed 11,300 committed events per second when no dynamic optimizations were used. The RAID model processed 10,917 committed events per second when no dynamic optimizations were used. The ISCAS2 model processed 4680 committed events per second when no dynamic optimizations were used. The no dynamic optimizations case was used as the base value of the normalization (i.e 1.0 in the graph represents 11,300 committed events per

second for SMMP).

8 Conclusions

From the graph, it is clear that the dynamic optimizations, dynamic cancellation and dynamic checkpointing, result in a 50% improvement in performance. Both SMMP and RAID report improved performance when lazy cancellation is used, but ISCAS2, which usually prefers aggressive cancellation, reports a degradation in performance due to the unnecessary comparisons performed by the lazy cancellation algorithm. When both dynamic optimizations are turned on, all the applications report improved performance. But when the dynamic optimizations are invoked with the external control object, all the applications perform even better. On average, all applications report a performance improvement of 15%. This clearly supports our notion that by removing the overhead of parameter adjustment from the simulation object's code and by the use of an external control object, total overhead of dynamic parameter adjustment is reduced. This reduction in overhead results in the 15% performance improvement.

An approach to remove the parameter adjustment phase from within Time Warp objects was presented. This removal is an attempt to moderate the overhead of dynamic parameter adjustment in a Time Warp simulation and to enable greater freedom of the parameter control mechanism. The parameter adjustment phase is instead configured as an additional component of each logical process. The public domain WARPED Time Warp simulation kernel was modified to implement this optimization. Empirical performance data was collected for two application domains: queuing model simulation and VHDL simulation. A comparative analysis with static parameter and traditional dynamic parameter adjustment shows that the external control mechanism provides speedups between 5%-17% over the best performing alternatives.

7 Future Enhancements By employing a circulating token, a decidedly simple external control mechanism, we have gained modest speedup of Time Warp simulations. We envision several enhancements to this dynamic optimization control mechanism. Such enhancements to the token passing mechanism include variable speed token passing, and passing multiple tokens; a token could include more data, allowing the objects to make better informed decisions; replace the tokens with intelligent autonomous agents. Variable speed token passing is different than passing a token at random intervals. One variable speed policy would be to pass the token very quickly at the beginning and reduce the speed over time. A second policy would have the speed in direct proportion to the stability of the entire system. These policies would help to alleviate the overhead due to switching dynamic optimizations. During any given simulation, some dynamic optimizations may evolve quite differently than others. We currently use a single token to trigger the evaluation of all dynamic optimizations. However, using variable speed token passing and employing a one-toone mapping of tokens to dynamic optimizations would allow the various optimizations to evolve at their own pace. In addition to controlling the migrations of tokens, another enhancement to the token passing scheme involves adding data to the token. If data about each simulation object was integrated into the token, then each simulation object could conceivably make better decisions about dynamic optimizations. For example, instead of making local greedy changes to the dynamic algorithms, a simulation object could make use of its neighbors data to find a regionally or globally optimal setting. Replacing the token passing mechanism with intelligent autonomous agents is an attractive alternative. The agents could passively monitor the system and instruct the Time Warp objects to change parameters when needed. In addition, maintaining profile data from previous runs would allow the agents to learn about the dynamics of a simulation so that they could make better decisions in the future. User definable agents would allow programmers to give the simulation system “hints” about a simulation.

Acknowledgments The authors would gratefully like to acknowledge the suggestions and contributions of Dale E. Martin, Timothy J. McBrayer, Raghunandan Rajan, and Christopher H. Young. The authors would also like to acknowledge the time and effort spent by the anonymous referees for their insight and their suggestions & comments.

References [1] A KYILDIZ, I. F., C HEN , L., DAS , S. R., F UJIMOTO , R. M., AND S ERFOZO , R. F. The effect of memory capacity on time warp performance. Journal of Parallel and Distributed Computing 18, 4 (Aug. 1993), 411–422. [2] A SHENDEN , P. The Designers Guide to VHDL. Morgan Kaufmann Publishers, Inc, San Mateo, CA, 1996. [3] A STROM , K. J., AND W ITTENMARK , B. Adaptive Control. Addison Wesley, Reading, MA, 1989. [4] BALL , D., AND H OYT, S. The adaptive time-warp concurrency control algorithm. In Distributed Simulation (Jan. 1990), Society for Computer Simulation, pp. 174–177. [5] BAUER , H., AND S PORRER , C. Distributed logic simulation and an approach to asynchronous GVT-calculation. In 6th Workshop on Parallel and Distributed Simulation (Jan. 1992), Society for Computer Simulation, pp. 205–208. [6] B ELLENOT, S. State skipping performance with the time warp operating system. In 6th Workshop on Parallel and Distributed Simulation (Jan. 1992), Society for Computer Simulation, pp. 53–61. [7] C HANDY, K. M., AND S HERMAN , R. Space-time and simulation. In Distributed Simulation (1989), Society for Computer Simulation, pp. 53–57. [8] DAS , S. R., AND F UJIMOTO , R. M. An adaptive memory management protocol for time warp parallel simulation. In Sigmetrics (May 1994), pp. 201–210. [9] D' S OUZA , L. M., FAN , X., AND W ILSEY, P. A. pGVT: An algorithm for accurate GVT estimation. In Proc. of the 8th Workshop on Parallel and Distributed Simulation (PADS 94) (July 1994), Society for Computer Simulation, pp. 102–109. [10] F ERSCHA , A. Probabilistic adaptive direct optimism control in time warp. In Proc. of the 9th Workshop on Parallel and Distributed Simulation (PADS 95) (June 1995), pp. 120–129. [11] F LEISCHMANN , J., AND W ILSEY, P. A. Comparative analysis of periodic state saving techniques in time warp simulators. In Proc. of

the 9th Workshop on Parallel and Distributed Simulation (PADS 95) (June 1995), pp. 50–58. [12] F UJIMOTO , R. Parallel discrete event simulation. Communications of the ACM 33, 10 (Oct. 1990), 30–53. [13] IEEE Standard VHDL Language Reference Manual. New York, NY, 1993. [14] L IN , Y. Estimating the likelihood of success of lazy cancellation in time warp simulations. International Journal in Computer Simulation (1996). [15] L IN , Y., AND L AZOWSKA , E. D. A study of time warp rollback mechanism. ACM Transactions on Modeling and Computer Simulation 1, 1 (Jan. 1991), 51–72. [16] L IN , Y.-B., AND L AZOWSKA , E. Determining the global virtual time in a distributed simulation. In 1990 International Conference on Parallel Processing (1990), pp. III–201–III–209. [17] L IN , Y.-B., P REISS , B. R., L OUCKS , W. M., AND L AZOWSKA , E. D. Selecting the checkpoint interval in time warp simulation. In Proc of the 7th Workshop on Parallel and Distributed Simulation (PADS) (July 1993), Society for Computer Simulation, pp. 3–10. [18] M ARTIN , D. E., M C B RAYER , T. J., AND W ILSEY, P. A. WARPED: A time warp simulation kernel for analysis and application development. In 29th Hawaii International Conference on System Sciences (HICSS-29) (Jan. 1996), H. El-Rewini and B. D. Shriver, Eds., vol. Volume I, pp. 383–386. [19] M ATTERN , F. Efficient algorithms for distributed snapshots and global virtual time approximation. Journal of Parallel and Distributed Computing 18, 4 (Aug. 1993), 423–434. [20] M ISRA , J. Distributed discrete-event simulation. Computing Surveys 18, 1 (Mar. 1986), 39–65. [21] PALANISWAMY, A., AND W ILSEY, P. A. Adaptive checkpoint intervals in an optimistically synchronized parallel digital system simulator. In VLSI 93 (Sept. 1993), pp. 353–362. [22] PALANISWAMY, A., AND W ILSEY, P. A. An analytical comparison of periodic checkpointing and incremental state saving. In Proc. of the 7th Workshop on Parallel and Distributed Simulation (PADS 93) (July 1993), Society for Computer Simulation, pp. 127–134. [23] PALANISWAMY, A., AND W ILSEY, P. A. Parameterized time warp: An integrated adaptive solution to optimistic pdes. Journal of Parallel and Distributed Computing 37, 2 (Sept. 1996), 134–145. [24] P ERRY, D. L. VHDL, 2nd ed. McGraw–Hill, New York, NY, 1994. [25] P REISS , B. R., M AC I NTRYE , I. D., AND L OUCKS , W. M. On the trade-off between time and space in optimistic parallel discrete-event simulation. In 6th Workshop on Parallel and Distributed Simulation (Jan. 1992), Society for Computer Simulation, pp. 33–42. [26] R AJAN , R., R ADHAKRISHNAN, R., AND W ILSEY, P. A. Dynamic cancellation: Selecting time warp cancellation strategies at runtime. International Journal in Computer Simulation (1997). (forthcoming). [27] R EIHER , P. L., W IELAND , F., AND J EFFERSON , D. R. Limitation of optimism in the time warp operating system. In Winter Simulation Conference (Dec. 1989), Society for Computer Simulation, pp. 765– 770. ¨ [28] R ONNGREN , R., AND AYANI , R. Adaptive checkpointing in time warp. In Proc. of the 8th Workshop on Parallel and Distributed Simulation (PADS 94) (July 1994), Society for Computer Simulation, pp. 110–117. [29] YOUNG , C., AND W ILSEY, P. A. A distributed method to bound rollback lengths for fossil collection in time warp simulators. Information Processing Letters 59, 4 (Aug. 1996), 191–196.