Flexible State Update Mechanism for Large-Scale Distributed

SIMULATION http://sim.sagepub.com

Flexible State Update Mechanism for Large-Scale Distributed Wargame Simulations Suiping Zhou, Wentong Cai, Stephen J. Turner, Junhu Wei and Wenbo Zong SIMULATION 2007; 83; 707 DOI: 10.1177/0037549707085541 The online version of this article can be found at: http://sim.sagepub.com/cgi/content/abstract/83/10/707

Published by: http://www.sagepublications.com

On behalf of:

Society for Modeling and Simulation International (SCS)

Additional services and information for SIMULATION can be found at: Email Alerts: http://sim.sagepub.com/cgi/alerts Subscriptions: http://sim.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav

Downloaded from http://sim.sagepub.com at PENNSYLVANIA STATE UNIV on April 16, 2008 © 2007 Simulation Councils Inc.. All rights reserved. Not for commercial use or unauthorized distribution.

Flexible State Update Mechanism for Large-Scale Distributed Wargame Simulations Suiping Zhou Wentong Cai Stephen J. Turner School of Computer Engineering Nanyang Technological University Nanyang Avenue, 639798 Singapore [email protected] Junhu Wei Institute of System Engineering Xi’an Jiaotong University Xi’an, China Wenbo Zong STMicroelectronics Asia Pacific Singapore Large-scale wargame simulations may involve tens of thousands of entities and generally run on distributed simulation nodes. An important issue in constructing these applications is to reduce the amount of data transmitted among the distributed simulation nodes so that the network and the simulation nodes will not be flooded by the state update messages. To this end, we first propose a timeliness condition on how frequently the state updates of a local simulation entity need to be sent to other remote simulation nodes. A flexible state update mechanism is then proposed to ensure the update intervals of the entities meet the required timeliness update condition and, in the meantime, balance the number of entities that need to be updated in each simulation time step (referred to here as the frame). Experiments with a distributed wargame simulation indicate that the proposed mechanism can significantly reduce the number of messages sent over the network while still maintaining the simulation fidelity. Keywords: Wargame simulation, distributed simulation, update interval

1. Introduction In large-scale distributed wargame simulations, the state updates about tens of thousands of simulation entities which need to be exchanged in a timely manner. This may incur large processing overheads on the simulation nodes and also large communication overheads on the network. Due to the constraint of the limited processing power of a simulation node and the limited bandwidth of the net-

SIMULATION, Vol. 83, Issue 10, October 2007 707–719 c 2007 The Society for Modeling and Simulation International 1

DOI: 10.1177/0037549707085541 Figure 3 appears in color online: http://sim.sagepub.com

work, some updates may not be processed in time at a remote node, which may cause some anomalies to the simulation [1–4]. How to reduce the amount of data exchanges among the simulation nodes is therefore an important issue in such applications. Some work has been done on how to reduce the amount of data exchanged over the network in large-scale distributed simulations. In Bassiouni et al. [5], the authors provide an excellent review on the network bandwidth requirement reduction techniques. The two most widelyused techniques are Dead-Reckoning (DR) and relevance filtering. DR is quite successful in reducing the overall network traffic for relatively small sizes (in terms of the number of simulation entities involved) of distributed simulations such as SIMulator NETworking (SIMNET) [6] Volume 83, Number 10 SIMULATION


707

Zhou, Cai, Turner, Wei, and Zong

and the first generation of Distributed Interactive Simulation (DIS) [7]. However, for large-scale wargame simulations with tens of thousands of entities, the network traffic may still be very high [5]. The DR technique also requires each node to maintain the DR models for each moving entity in the simulation, which may become a considerable overhead for a node as the size of the simulation increases. In a more recent study, a smart update rate reduction mechanism was proposed [8]. With this mechanism, simulators with different levels of fidelity are allowed to specify different requirements on the maximum update rates that they can handle. The mechanism aims to enhance the interoperability among various existing simulators. Although the implementation complications and the computational overhead of the mechanism have not been evaluated, it has the potential to reduce the requirement of network bandwidth for distributed simulations. Relevance filtering is currently the dominant network bandwidth requirement reduction technique for largescale distributed wargame simulations [5, 9–14]. Instead of controlling the frequency at which local updates are sent to remote nodes, the relevance filtering technique reduces network traffic by attempting to transmit information only to interested entities. For instance, let e1 , e2 and e3 be three entities simulated at node 1, node 2 and node 3, respectively. Suppose that in the simulated virtual environment, e1 and e2 are separated by a very large distance, while e1 and e3 are close to each other. Then the state updates of e1 only need to be sent to node 3. In this example, whether an entity is interested in the state of another entity depends on their distance in the virtual environment. It should be noted that an entity may express its interests in other different forms. For example, a ship may want to know about all ships within a radius of 30 nautical miles, while it may have no interest in the state of groundbased vehicles. Here, the ship’s interest is represented by the spatial distance and the type of entity. The relevance filtering technique is exemplified as the Data Distribution Management (DDM) services in the High Level Architecture (HLA) [9]. The main objective of the HLA is to foster the interoperability and the reusability of simulation components. The HLA is now an IEEE standard [10] for linking together distributed simulation components (known as federates), and is widely adopted in distributed simulations. The HLA’s DDM adopts a publish/subscription model of data distribution. A federate may declare its update regions and subscription regions, where a region is defined as a set of multidimensional routing spaces [12, 13]. For example, an update region may represent the area where an entity is locating, and a subscription region may represent the area of interest of an entity. A data connection will be established between a pair of federates only when an update region and a subscription region overlap. The goal of the HLA’s DDM services is to limit the messages received by federates in large distributed federations to those messages of interest in order to reduce (1) the data set required to be processed by 708 SIMULATION

the receiving federate and (2) the message traffic over the network. While DDM could significantly reduce the data set received by a federate, its capability in reducing the network traffic is limited by the computational overhead for matching update/subscription regions and the communication overhead for updating these regions due to the movement of the entities [15]. In this paper, we investigate how to reduce the network bandwidth requirement in large-scale distributed wargame simulations. We assume that the distributed simulation nodes are not separated by large geographical distances so that the major concern is the network traffic rather than the physical message transmission delays between different simulation nodes. We propose a novel state update mechanism to reduce the network bandwidth requirement of such applications. In essence, we investigate the network bandwidth reduction problem by asking the fundamental question: how frequently do the state updates of a local entity need to be sent to other remote nodes? To this end, we first proposed a timeliness condition on the update frequency (the frequency that the state of the entity is sent to other remote nodes1 at the local node, the state of the entity is still updated in every frame) of an entity. With this condition, the update frequency of an entity is determined by its velocity and its utility in the application. In our utility model, the utility of an entity is determined by the number of entities on which this entity may have influence and the distances between this entity and those entities which are within its area of influence. Thus, the utility model reflects how heavily an entity is ‘used’ by other entities. To ensure the update frequency as required by the timeliness condition, a flexible update mechanism is proposed. Our approach to the network bandwidth reduction problem is different from the relevance filtering techniques, such as the HLA’s DDM. Basically, relevance filtering determines where to send an update, while the proposed flexible update mechanism determines how frequently to send an update. Therefore, our approach can be used either separately or together with the relevance filtering techniques in large-scale wargame simulations. With our approach, an entity’s update frequency is determined by its potential impact on the simulation, i.e. by its moving speed, its distance to other entities and the number of entities around it rather than by its kinematics alone as in DR. In addition, for large-scale distributed wargame simulations, the number of updates may still be very large even with DR. Thus, our approach can also be used as a scheduling technique to enhance the DR so that the priority of sending an update is determined by the relative importance of the update. The remainder of this paper is organized as follows. In Section 2, we propose a timeliness condition on the state updating for distributed wargame simulations. To determine the relative importance of an entity, a utility model is described in Section 3. In Section 4, based on the timeliness condition and the utility model, a flexible

Volume 83, Number 10


FLEXIBLE STATE UPDATE MECHANISM FOR LARGE-SCALE DISTRIBUTED WARGAME SIMULATIONS

be sent to other simulation nodes frequently to avoid or minimize GTAs. To determine the update frequency of a moving entity in a distributed simulation, we have the following definition. Definition 2.1 Strong Timeliness Condition: A simulation entity ei should be updated within the time interval 1 i that is computed as 2i 1i 2 (1) 3i where 2 i is the threshold of the collision detection algorithm adopted at the node where ei is simulated and 3 i is the velocity of entity ei .

Figure 1. Go-Through-Accident

update mechanism is proposed to efficiently utilize the network bandwidth. We have implemented the flexible update mechanism in a Distributed Simulation Engine (DSE) provided by a third party. The DSE is described in Section 5 together with the integration of the update mechanism with DSE. The performance of the implementation was evaluated using a battlefield benchmark. The benchmark and the performance results are discussed in Section 6. Conclusions are drawn in Section 7. 2. A Timeliness Condition In distributed wargame simulations, the states of simulation entities need to be sent to other simulation nodes in a timely manner1 otherwise, some anomalies may occur in the simulation. For instance, as demonstrated in Figure 1, if the state updates of an entity (e.g. aircraft) cannot be sent to another entity (e.g. missile) within the required time interval, they may pass through each other although they should actually collide. We call this phenomenon GoThrough-Accident (GTA) [3]. More generally, a GTA occurs when two entities that should collide go through each other, or when two entities that should go through each other collide. Apart from GTAs, other types of anomalies may also occur if state updates are not sent to other simulation nodes frequently. For example, the searching radar of a surface-to-air missile (SAM) system may be turned on quickly for some time, and then turned off to avoid being attacked by a hostile aircraft. If a state update is not sent by the search radar to the aircraft while the radar is on, the aircraft may incorrectly fail to detect the radar and warn the pilot of the aircraft. Unlike the frequent positional updates of simulation entities, this kind of state updates will not contribute much to the network traffic. Thus, they can be treated as events, and sent to other simulation nodes whenever they occur. In this paper, we focus on the position updates of simulation entities in large-scale distributed wargame simulations. The position update of a moving entity needs to

It is obvious that 1 i in Definition 2.1 determines the maximum update interval of entity ei that can guarantee no GTA in the simulation. It should be noted that Definition 2.1 may be overly strict. In some cases, updating within the time interval 1 i is unnecessary. For example, if an entity is not close to other entities, it can be updated within a longer interval than 1 i . Therefore, we provide the following relaxed timeliness condition. Definition 2.2 Relaxed Timeliness Condition: A simulation entity ei should be updated within the time interval 4i that is computed as 4i 2

1 1i U 5ei 6

(2)

where U 5ei 6 is the utility value of entity ei which is between 0 and 1.

Remark 2.1 As will be described in the next section, the utility value of an entity is determined by the number of entities on which this entity may have influence and the distances between this entity and those entities that are within its area of influence. It can be seen from equation (2) that if the utility value of entity ei is less than 1, then its update interval 4i will be larger than 1 i , and the larger U 5ei 6 is, the closer 4i is to 1 i . Therefore, if an entity is far from other entities, the relaxed timeliness condition allows the entity to be updated less frequently. 3. A Utility Model 3.1 A General Utility Model With the relaxed timeliness condition, the question now is how to determine the utility value of a simulation entity. For an entity e in a distributed wargame simulation, we have the following observations: Volume 83, Number 10 SIMULATION


709


3 An entity e has an area of influence on other entities, i.e. entities falling into this area will be affected by the state of entity e. If entity e has influence on a large number of entities, then the state update of entity e may have a great impact on the whole simulation. On the other hand, if entity e only has influence on a small number of entities, then the state update of entity e may have little impact on the whole simulation. Thus, the number of entities on which an entity may have influence is an important factor in deciding its relative importance. 3 If entity e is close to some other entities, then the state update of entity e may have great impact on other entities (e.g. for collision detection). On the other hand, if entity e is far from other entities, then the state update of entity e may have little impact on other entities. Thus, the distance between an entity and other entities is another important factor in deciding its relative importance. To evaluate the relative importance of a simulation entity, we first propose a general utility model. In the following two subsections we will refine this utility model to take into consideration the above two factors in deciding the relative importance of an entity. Suppose that there are n entities in the simulation. These entities are classified into m different types denoted by C1 7 C2 7 8887 Cm (such as tank, aircraft, etc.), and there 1m are n j entities of type C j , where j 4 17 m and j21 n j 2 n. We then have the following definition. Definition 3.1 The utility of a simulation entity ei , where i 4 17 n, is defined as U 5ei 6 2

m 2

9 j U j 5ei 6

(3)

j21

where U j 5ei 6 4 07 1 is the relative utility 1 of entity ei with respect to entity type C j , 9 j 0 and mj21 9 j 2 1. Remark 3.1 In the above definition, 9 j is the weight given to entity type C j to reflect its relative importance in the simulation, which is generally application dependent. In this paper, for simplicity, we treat all entity types equally. In the remainder of the paper, we therefore choose 9 j 2 1m. As can be seen from Definition 3.1, the relative utility U j 5ei 6 plays an important role in deciding the overall utility of entity ei . In the following sections, we provide the definition of the relative utility of an entity based on the relative importance of the entity with respect to other entities. 710 SIMULATION

Figure 2. Area of entity influence

3.2 Relative Utility To determine the relative importance of a simulation entity, we need to know the number of entities on which this entity may have influence. Let r j7k denote the radius of the Area of Interest (AOI) (the position of a simulation entity is the focus here and we assume that all entities of the same type have the same AOI size) [12] for entity type C j with respect to entity type Ck , for j 4 17 m, k 4 17 m. Note that different types of entities may have different sizes of AOIs. We now define the area of influence of an entity as follows. Definition 3.2 The area of influence of an entity ei of type Ck with respect to entity type C j , denoted as 1 j 5ei 6, is the circular area centered around the position of ei and with a radius of r j7k , where i 4 17 n k , j 4 17 m, k 4 17 m. As an example, the area of influence of entity e1 of type C4 with respect to entity type C5 is shown in Figure 2, where e2 and e3 are entities of type C5 . Since the distance between e1 and e2 is less than r574 , i.e. e1 is within the area of interest of e2 , the state updates of e1 will have an influence on e2 . On the other hand, the distance between e1 and e3 is greater than r574 , therefore the state updates of e1 have no influence on e3 . Let N j 5ei 6 denote the number of entities of type C j which are within 1 j 5ei 6, for i 2 17 27 8887 n7 j 2 17 27 8887 m. We have the following definition. Definition 3.3 The relative utility of an entity ei with respect to an entity type C j is U j 5ei 6 2

N j 5ei 6 nj

(4)

where i 2 17 27 8887 n7 j 2 17 27 8887 m. In the above definition, the relative utility of an entity ei with respect to an entity type C j is decided by the number of entities of type C j that are within the area of influence of entity ei with respect to entity type C j .




Figure 3. Multiple levels of area of influence

3.3 A Refined Utility Model In Definition 3.3, we treat equally all the entities of type C j that are within the area of influence of entity ei of type Ck with respect to type C j . To consider the effect of the distance between ei and other entities of type C j in deciding the relative importance of entity ei with respect to type C j , we further divide 1 j 5ei 6 into multiple levels as shown in Figure 3. Note that in Figure 3, a level is defined as a ring surrounding the entity. Suppose that 1 j 5ei 6 is divided into l j levels, and there 5l j 6 526 are N 516 j 5ei 6, N j 5ei 6, . . . , N j 5ei 6 entities of type C j within each of these levels, respectively. Considering the different distances between these entities and entity ei , a refined utility model is given as follows. Definition 3.4 The utility of a simulation entity ei of type Ck , where i 4 17 n, is defined as U 5ei 6 2

M 2

9 j U j 5ei 6

(5)

j21

where U j 5ei 6 2 with and

1l j

lj 1 2 5s6 5 N 5s6 j 5ei 6 n j s21 j

(6) 5l 6

s21

516 526 j 5s6

1, j 2 1, 0 j j 5 5 5 j

s21

N 5s6 j 5ei 6 2 N j 5ei 6.

1l j

Remark 3.2 In the above definition, different weights are given to different levels to reflect how close an entity is to other entities. The closer an entity is to other entities, the more important the state updates of this entity are to other entities. 4. Flexible Update Mechanism With the relaxed timeliness condition and the utility model, the network bandwidth requirement could be reduced since entities with lower importance can be updated

less frequently. However, the timeliness condition also requires the state of an entity to be updated to the remote nodes within the update interval 4i as defined by equation (2). To this end, we have developed a flexible update mechanism. In the following discussion, we assume that the state update of an entity needs to be sent to other remote nodes at least once in every l frames (e.g. the ‘heart-beat’ mechanism is widely used in distributed simulations to keep the remote nodes updated). Therefore, the update intervals of the entities are re-evaluated every l frames. Although the utility value of an entity may change with time due to the movements of the entities, we do not expect the update intervals in the flexible update mechanism to change frequently. This is because the sizes of the different levels of the area of influence of an entity are usually large enough so that it takes some time (in terms of number of simulation frames) for an entity to pass from one level to another level. 4.1 A Simple Update Mechanism First, we propose a simple update mechanism. With the simple update mechanism, the update times of local entities are directly determined by their update intervals as computed by equation (2). Suppose that the frame time of the simulation is 4T . The state update frequency of an entity can be determined as follows. Definition 4.1 If k 5 4T 6 4i 5k 7 16 5 4T , then ei will be updated every k frames, for k 2 17 27 8 8 8 7 l 8 1. If 4i 4T , then ei will be updated every frame. If 4i 9 l 5 4T , then ei will be updated every l frames. To update the entities according to their required update intervals, we propose the following Simple Update Mechanism. 1. Put the entities into l different lists (as shown in Figure 4) in a circular manner, so that the entities which need to be updated in the current frame are put into the current list L c and the entities that need to be updated j frames later are put into list L 5c7 j6 mod l . 2. Update all entities in the current list at the current frame. 3. Move an entity into a new list according to step 1 after updating the entity. With the simple update mechanism, instead of updating all the entities in each frame, only the entities in the current list are updated in the current frame. Therefore, we expect that the number of entities to be updated in each frame will be reduced. However, since different entities may have different utility values and different velocities, the number of entities that need to be updated in each Volume 83, Number 10 SIMULATION


711


Figure 5. Flexible update mechanism Figure 4. A simple update mechanism

frame may be different. For example, in one frame, there may be a large number of entities to be updated, while in another frame, there may be just a few entities to be updated. Therefore, a flexible update mechanism that could balance the number of entities that need to be updated in each frame and also ensure the timely update of entities is desirable. 4.2 Flexible Update Mechanism To balance the number of entities that need to be updated in each frame, we first estimate the average number of entities updated per frame. Let ki denote the number of entities to be updated every i frames, where 1 6 i 6 l. Then the average number of entities updated per frame can be estimated as 3 l 4 2 ki 2 8 (7) i i21 Note that the value of changes with the utility values and hence the update intervals of entities change. Since the update intervals of the entities are re-evaluated every l frames, is also calculated every l frames. The Flexible Update Mechanism is described as follows. 1. Put the entities into l different lists according to their update intervals as determined by Definition 4.1. As shown in Figure 5, list L i (1 6 i 6 l) contains the entities that are required to be updated every i frames, which is evenly divided into i partitions. 2. In each frame, one partition in each list is selected in a circular manner for updating. For example, if partition 1 ( pi71 ) of list i is updated at the current frame, then in the next frame, partition 2 ( pi72 ) of this list will be updated. If the update intervals of entities do not change, the number of entities to be updated in each frame will correspond to as calculated by equation (7). 712 SIMULATION

3. When the update interval of an entity changes, the entity is deleted from its present list, and inserted into an appropriate partition of a new list according to its new update interval and its position in the old list. For example, suppose that the next update of an entity in the ith list is scheduled at the jth frame. If the update interval of the entity is increased by 1 frame, then this entity is moved to a partition of the 5i 7 16th list that will be updated at the 5 j 7 16th frame. 4. After the entities are redistributed among the lists, re-balance the partitions in a list L i (1 6 i 6 l) in the following way: 3 Calculate the average size of the partitions in L i given by s 2 ki i, where ki is the number of entities in the list. 3 Move some entities in list L i from partitions whose sizes are greater than s to those partitions whose sizes are less than s. Remark 4.1 In balancing, when an entity is moved from one partition to a new partition, if the new partition will be updated later than the partition from which the entity is moved, the entity is added into an extra list L for updating. This is required to ensure that the moved entities are updated within their update intervals. The union of the entities in the list L and the entities selected from the list partitions are updated, and then the entities in the list L are deleted. In the frames in which the partitions are rebalanced, the number of entities updated in each frame is plus the number of entities in the extra list L. Usually the number of entities in L is quite small. Remark 4.2 The flexible update mechanism ensures that the entities are updated within their update intervals according to equation (2). 4.3 Simulation Study To compare the performance of the flexible update mechanism with the simple update mechanism, we have de-




Figure 6. Simulation results

veloped some experimental scenarios to simulate the state updates on one simulation node in a large-scale wargame simulation. The simulation was run on a personal computer, which simulates 1000 entities. We assume that an entity needs to be updated at least once every 10 frames (i.e. l 2 10). In the experiments, the update intervals of entities were re-calculated every 10 frames. When the update interval of an entity changes, we assume that it is either increased or decreased by 1 frame randomly. Two different scenarios were used in the experiments to simulate different levels of interactivity of the wargame simulation. Scenario 1: The update intervals of 50% of the entities (randomly chosen) change every 10 frames. Scenario 2: The update intervals of all the entities change every 10 frames. At the beginning of the simulation, the update intervals of the 1000 entities were randomly set to values between 1 and 10 according to a uniform distribution. For each scenario, the two update mechanisms are tested individually. To compare the performances of the update mechanisms, we focus on their capabilities in reducing and balancing

the number of entities to be updated in each frame. To avoid warm-up transient, simulation results were collected from the 50th frame. Simulation results are shown in Figure 6. The number of entities that need to be updated in 1 frame is 1000 if all the entities are to be updated in each frame. It can be seen from the experimental results that both of the proposed update mechanisms greatly reduce the number of entities to be updated in each frame. With the simple update mechanism, however, the number of entities to be updated in each frame varies considerably. The results show that the flexible update mechanism performs better than the simple update mechanism both in reducing the number of updates in each frame and in balancing the network traffic. We have therefore decided to integrate the flexible update mechanism with a simulation engine provided by a third party. 5. Integration of the Flexible Update Mechanism with DSE Given the timeliness condition, the utility model and the flexible update mechanism, we now query the quality of their performance in terms of the number of GTAs in the Volume 83, Number 10 SIMULATION


713


application and the amount of data exchanged over the network. To determine performance quality, the models are implemented in a DSE [16] provided by the Defence Science and Technology Agency (DSTA) of Singapore. The DSE is a model-driven distributed simulation engine. It provides a set of base classes that represent the common concepts in wargame simulations, a simulation kernel that keeps track of time and orchestrates the execution of all application components and distributed database services for storing running instances for all concepts. With DSE, local updates are broadcast to other remote nodes in every frame. The underlying communication between distributed DSE components is handled by a SoftwareBus which is also developed by DSTA. One of the major issues in implementing the update mechanism is the utility computation. Once the utility value of an entity is computed, the update interval of the entity can be easily decided according to the timeliness condition. In computing the utility value of a local entity, we only consider remote entities (i.e. entities in a different simulation process) as interactions between two entities in the same process take place without communication of data. The utility computation therefore consists of counting the number of remote entities within different levels of a local entity’s area of influence. One obvious way to make the computation is to scan all local entities according to the utility formula. The computation complexity is therefore O5n 1 n 2 6, where n 1 and n 2 are the number of local and remote entities, respectively. A better way to compute the utility values is to use a grid map strategy1 that is, the battlefield is divided into cubes of a specified size. The levels of the area of influence of an entity are therefore translated into a number of cubes. For ease of illustration, Figure 7 shows a 2D view of such a translation, although a 3D grid is used in our implementation. There is a counter associated with each entity type for each cube. These counters are initialized at the beginning of the utility computation, simply by going through all remote entities and incrementing the corresponding counter. Hence, the complexity for initializing the map is O5n 2 6. With the map initialized, the utility of a local entity is computed by locating the cube where the entity lies and then summing the corresponding counters associated with the cubes in the appropriate levels surrounding that entity. This takes a constant time for each local entity, dependent upon the (maximum) number of cubes in an area of influence i.e. A. The time for all local entities is therefore O5A n 1 6. Assuming that A

n 2 , which is normally the case, this is a significant improvement compared to the previous method with complexity O5n 1 n 2 6. Note that the grid size should be large enough so that it takes some time (in terms of number of simulation frames) for an entity to pass through. Since there is a set of counters (one for each entity type) associated with each cube, a large grid size reduces the number of counters that a simulation node needs to maintain and the overhead of 714 SIMULATION

Figure 7. Levels of area of influence translated into 2D grids

maintaining the counters will be less. For the utility computation, we do not need the exact positions of the entities in these grids. Instead, what matters in the utility computation is the number of entities in these grids. Thus, we do not expect the utility value of an entity to change frequently. As we mentioned in Section 4, we assume that the update intervals of entities are re-evaluated every l frames. Thus, in our implementation, these counters are updated every l frames as is the utility value of an entity. The computational overhead of the utility calculation is therefore not high. We have conducted some experiments to evaluate the overhead of the utility computation. The results show that the overhead of the utility computation is small [17]. In addition, to further reduce the overhead of the flexible update mechanism, a type-based filtering scheme can be used. For example, if an entity ei of type Ck has no influence on entities of type C j , then we can simply set U j 5ei 6 to 0 and discard entity type C j when computing the utility value of entity ei . The utility values are used to determine entities’ update intervals using equation (2). To implement the flexible update mechanism described in Section 4.2, certain builtin models in DSE have to be modified. The modified DSE with the flexible update mechanism is referred to as DSE II. 6. Experiments In this section, we describe the experiments conducted to evaluate the performance of the flexible update mechanism in large-scale distributed real-time simulations. 6.1 Battlefield Benchmark and Experimental Setup A battlefield is designed for benchmarking. The battlefield consists of a blue army and a red army. The red




Figure 8. Experimental setup

army defends a military base with anti-aircraft artillery troops. The artillery troops are deployed in clusters, each of which consists of multiple artilleries. The blue army consists of several waves of bombers, and aims to destroy the military base of the red army. When the bombers fly over the base, they drop bombs on the ground targets. In the meantime, the ground artilleries strike back on the invaders. In the simulation, the planes fly at a constant horizontal speed and the positions of artilleries are fixed. The initial state of a bomb is the same as the plane that drops it. The subsequent states of bombs are subject to the movement equations. Similarly, shells get their initial position from the artilleries that fire them. They possess initial speed and their state depends upon the movement equations. Up to 8 DELL computers each with two 288 GHz processors and 1 GB main memory were used in the benchmark experiments. These computers were connected through a Gigabit Ethernet switch. The experimental setup is shown in Figure 8 for two simulation processes. The SoftwareBus occupies one computer and every other computer hosts a pair of processes – one for the blue army and one for the red army. To evaluate the performance and the scalability, i.e. how the number of processes impacts on the performance of the proposed update mechanism, a suite of experiments was conducted for the same scenario described above. Experimental scenarios are listed in Table 1. In order to isolate other factors that affect the performance and only explore the impact of the flexible update mechanism, the size of the simulation remains the same when the number of processes increases. In each simulation, we use a different number of processes. Odd-numbered processes are used to simulate the blue army. Similarly, even-numbered processes simulate the red army. The planes are equally allocated to the odd-numbered processes, and the artilleries

Table 1. Experimental scenarios Number of planes

60

Bombing fequency (seconds)

1.1

Number of artilleries

120

Firing frequency (seconds)

0.7

Length of simulation(seconds)

120

Frame time (millisecond)

50

are equally allocated to the even-numbered processes. Processes 1 and 2 run on one computer, processes 3 and 4 run on another computer, and so on. All processes that simulate the blue army therefore have the same number of planes, and all processes that simulate the red army have the same number of artilleries. The process Display in Figure 8 is an optional process. It is responsible for capturing the states of entities and all activities which occur in the simulation and reporting them to the User Front End. The User Front End is a process that graphically displays the simulation. Table 2 lists the configuration parameters for the utility computation. The area of influence of a moving entity with respect to an entity type with which it can collide (e.g. shell–plane) is divided into 3 levels with ranges 100, 300 and 500. The weights of these levels are set to 0.6, 0.3 and 0.1. A sequential version of the battlefield simulation was also developed. Because all computation of the models is performed in one process, there is no communication and synchronization problem with the sequential simulation. The number of collisions of planes and shells, and collisions of bombs and artilleries in the sequential simulation can be viewed as the theoretical value. The number Volume 83, Number 10 SIMULATION


715


Table 2. Utility computation parameters Description

Value

Battlefield minimum coordinates 5x7 y7 z6 Battlefield maximum coordinates 5x7 y7 z6 Threshold of the collision detection algorithm 2 Size of cube in 3D grid for utility computation Frequency of utility computation (frames) Number of entity types (m)

(–10, –10, –10) (11 000, 1500, 2000) 40 200 200 200 4 4

of GTAs occuring in DSE and DSE II can then be calculated as the difference between the theoretical value and the actual number of collisions. This is only an approximation, but is useful in situations where the major concern is that the overall statistics are valid. To obtain an accurate value of the number of GTAs, a log file of the actual collisions could be kept for both the sequential simulation and DSE or DSE II and a comparison of corresponding values made. 6.2 Results and Analysis The following measures are collected from the experiments to demonstrate the performance of the flexible update mechanism. All measures in the figures shown in this subsection are the average of multiple runs. 3 Data Exchange: This refers to the total amount of data sent among the processes. This measure represents the communication overhead of a distributed simulation. The flexible update mechanism is expected to reduce the number of updates and the amount of data exchange. For the same simulation scenario: the smaller the amount of data exchange, the more efficiently the bandwidth is utilized. This measure is collected by the DSE/DSE II. It includes the exchange of state updates, as well as the messages that DSE/DSE II uses to undertake the synchronization, coordination and control between processes. 3 Skipped Frames: The DSE/DSE II maintains the synchronization between the simulation time and the real-time during simulation execution. If a model takes less than the allocated frame time to perform the computation, the DSE/DSE II sleeps until the next frame can be initiated. Otherwise, if a model takes more than the frame time, the DSE/DSE II has to skip some frames to save computation time to maintain synchronization with realtime. In real-time simulation, this measure reflects the computation overhead. 716 SIMULATION

3 GTA: GTA is a very important measure in real-time wargame simulations and virtual environment applications. As we mentioned in Section 2, a GTA happens when two entities that should collide go through each other, or when two entities that should go through each other collide. GTA greatly impacts the accuracy of a simulation. The following factors can possibly cause GTAs. (1) The latest states of entities fail to be processed in time at a destination process because of the communication and processing delay. (2) In real-time simulation, when the computation of models consumes more than the simulation frame time, the frame will be skipped to maintain the synchronization between the simulation time and the real time. Any collision that should happen at the skipped frame will therefore be missed. The number of GTAs is calculated using the difference between the number of collisions detected in the simulation and the theoretical value (from the sequential simulation). Figure 9 shows the amount of data exchange between processes during the simulation. From Figure 9 we can see that the amount of data exchange with DSE increases faster than with DSE II as the number of processes increases. It is noted that in Figure 9 the results of DSE are not available when the number of simulation processes is larger than 8. This is because the communication that the SoftwareBus undertakes increases as the number of processes increases. In these experiments, which require large amount of data to be exchanged, the SoftwareBus is unable to handle more than 8 processes as the communication overhead exceeds its capability. The DSE II reduces the date exchange, so the SoftwareBus can still work well even for a large number of processes. In real-time simulations, in order to maintain the synchronization between the simulation time and the realtime, when the computation consumes more time than that allocated to the frame, the simulation engine has to give up one or more subsequent frames to finish the computation of the current frame. The heavier the computation burden, the more frames are skipped and the less accurate the simulation. Figure 10 is the comparison between DSE and DSE II in terms of frame skipping. Because the flexible update mechanism reduces the number of state updates and therefore the amount of data exchange, the time used for processing the data exchange is reduced accordingly. Therefore, DSE II has fewer frames skipped and is more capable of keeping the simulation synchronized with realtime, thus making the simulation more accurate. In real-time simulations, whether the updates can be processed by other processes on time is determined by many factors, such as the amount of data exchanged among these processes. Figure 11 shows the comparison between DSE and DSE II in terms of GTA. Because the DSE II consumes less bandwidth and causes less frame skipping, it always outperforms the DSE in this measure.




Figure 9. Data exchange

Figure 10. Average skipped frames

7. Conclusions Large-scale distributed wargame simulations may cause a high communication overhead on the network. How to efficiently utilize the limited network bandwidth is therefore an important issue in such applications. To reduce the amount of data exchanged over the network, a relaxed timeliness condition is proposed to determine the update interval of a simulation entity based on the utility concept. A utility model is defined and used to determine the relative importance of a simulation entity so that entities with higher importance will be updated more frequently and those with lower importance will be updated less frequently. By controlling the update interval of entities, the bandwidth requirement of the distributed simulation can be reduced. To balance the number of updates per frame

and to ensure the required update intervals, a flexible state update mechanism is developed. The update mechanism has been implemented in a distributed simulation engine which is provided by a third party. The performance of the flexible update mechanism has been evaluated using a distributed real-time battlefield simulation. The experimental results show that the flexible update mechanism is able to reduce the data exchange during the simulation. Compared to the original DSE, the proposed mechanism also results in fewer skipped frames and a smaller number of Go-Through-Accidents. Finally, although the focus of this paper is on distributed wargame simulations, we certainly hope to see the application of the proposed mechanism in other nonmilitary systems which involve interactions among a large number of moving entities. Volume 83, Number 10 SIMULATION


717


Figure 11. GTA

8. Acknowledgements This research was supported by a grant from the Defence Science and Technology Agency of Singapore under the Defence Innovative Research Programme. 9. References [1] Krishnawamy, V., M. Raynal, D. Bakken, and M. Ahamad. 2001. Shared State Consistency for Time-Sensitive Distributed Applications. In Proceedings of the International Conference on Distributed Computing Systems (ICDCS 2001), April 2001, Phoenix, Arizona, USA, pp. 606–614. [2] Torres-Rojas, F., M. Ahamad, and M. Raynal. 1999. Timed Consistency for Shared Distributed Objects. In Proceedings of the 18th Annual ACM Symposium on Principles of Distributed Computing, May 1999, Atlanta, GA, USA, pp. 163–172. [3] Ozaki, A., M. Furuichi, K. Takahashi, H. Matsukawa. 2001. Design and Implementation of Parallel and Distributed Wargame Simulation System. IEICE/IEEE Joint Special Issue on Autonomous Decentralized Systems and Systems’ Assurance, E84D(10), 1376–1384. [4] Zhou, S., W. Cai, B. S. Lee, and S. J. Turner. 2004. Time-Space Consistency In Large-Scale Distributed Virtual Environments. ACM Transactions on Modeling and Computer Simulation, 14(1), 31– 47. [5] Bassiouni, M., M.-H. Chiu, M. Loper and M. Garnsey. 1997. Performance and Reliability Analysis of Relevance Filtering for Scalable Distributed Interactive Simulation. ACM Transactions on Modeling and Computer Simulation, 7(3), 293–331. [6] Miller, D.C., and J.A. Thorpe. 1995. SIMNET: The Advent of Simulator Networking. In Proceedings of the IEEE, 83(8), 1114–1123. [7] IEEE. 1993. IEEE Std. 1278: Standard for Information Technology – Protocols for Distributed Interactive Simulation Applications. [8] Möller, B. and M. Karlsson. 2005. Developing Well-Balanced Federations Using the HLA Evolved Smart Update Rate Reduction. In Proceedings of the 2005 Fall Simulation Interoperability Workshop, Orlando, FL. [9] U.S. Department of Defense. 1998. High Level Architecture Interface Specification. Version 1.3. [10] IEEE. 2001. IEEE Std. 1516: High Level Architecture. [11] Rak, S. and D. Van Hook. 1996. Evaluation of Grid-based Relevance Filtering for Multicasting Group Assignment. In Proceedings of 718 SIMULATION

the 14th DIS Workshop on Standards for the Interoperability of Distributed Simulation, Institute for Simulation and Training, Orlando, FL, USA, pp. 739–747. [12] Morse, K.L. and J. Steinman. 1997. Data Distribution Management in the HLA: Multidimentional Regions and Physically Correct Filtering. In Proceedings of the 1997 Spring Software Interoperability Workshop, Orlando, FL, USA. [13] Morse, K.L., L. Bic, and M. Dillencourt. 2000. Interest Management in Large-Scale Virtual Environments. Presence: Teleoperators and Virtual Environments, MIT Press, 9(1), 52–68. [14] Morse, K.L. and M. Zyda. 2002. Multicast Grouping for Data Distribution Management. Simulation Practice and Theory, 9(3–5), 121–141. [15] Petty, M.D. and K.L. Morse. 2004. The Computational Complexity of the High Level Architecture Data Distribution Management Matching and Connecting Processes. Simulation Modelling Practice and Theory, 12(3–4), 217–237. [16] Defence Science and Technology Agency of Singapore. 2002. Distributed Simulation Engine Programmer’s Guide, Version 2.6. [17] Zhou, S., S.J. Turner, W. Cai, H. Zhao and X. Pang. 2004. A Utility Model for Timely State Update in Distributed Wargame Simulations. In Proceedings of the 18th Workshop on Parallel and Distributed Simulation, Kufstein, Austria.

Suiping Zhou is currently an Assistant Professor in the School of Computer Engineering at Nanyang Technological University (Singapore). Previously, he worked as an engineer in Beijing Simulation Center, China Aerospace Corporation, and then joined Weizmann Institute of Science (Israel) as a Postdoctoral fellow. He received his B.Eng., M.Eng. and Ph.D in Electrical Engineering from Beijing University of Aeronautics and Astronautics (P.R. China). He is a member of IEEE and his current research interests include: large-scale distributed interactive applications (e.g. MMOGs), parallel/distributed systems, and human behavior representation in modeling and simulation. He is currently an associate editor of International Journal of Computer Games Technology. He has served as technical program committee member of several international conferences and workshops in computer games and virtual environments.




Wentong Cai is an Associate Professor with the School of Computer Engineering at Nanyang Technological University (NTU), Singapore, and head of the Computer Science Division. He received his B.Sc. in Computer Science from Nankai University (P.R. China) and Ph.D., also in Computer Science, from University of Exeter (UK). He was a Postdoctoral Research Fellow at Queen’s University (Canada) before joining NTU in February 1993. Dr Cai’s research interests include Parallel and Distributed Simulation, Grid and Cluster Computing, and Parallel and Distributed Programming Environments. He has authored or coauthored over 180 journal and conference papers in the above areas. Dr Cai is a member of the IEEE. He is currently an associate editor of ACM Transactions on Modeling and Computer Simulation (TOMACS), editorial board member of Multiagents and Grid Systems and editorial board member of International Journal of Computers and Applications.

Computer Science from Manchester University (UK). His current research interests include: parallel and distributed simulation, distributed virtual environments, parallel algorithms and languages, grid computing and multi-agent systems.

Stephen John Turner joined Nanyang Technological University (Singapore) in 1999 and is currently an Associate Professor in the School of Computer Engineering and Director of the Parallel and Distributed Computing Center. Previously, he was a Senior Lecturer in Computer Science at Exeter University (UK). He received his M.A. in Mathematics and Computer Science from Cambridge University (UK) and his M.Sc. and Ph.D. in

Wenbo Zong is currently with STMicroelectronics Asia Pacific, Singapore, working on the design and implementation of multimedia signal processing algorithms. Previously, he worked as a Research Assistant at Nanyang Technological University (NTU), Singapore, on various research projects in the areas of distributed simulation and digital library. He obtained the B.Eng. degree from NTU in 2004.

Junhu Wei is an Associate Professor in Systems Engineering Institute, School of Electronic and Information Engineering at Xi’an Jiaotong University (P.R. China). Previously, he joined the School of Computer Engineering at Nanyang Technological University as a Research Fellow. He received his B.Eng. in Automatic Control (1988), M.Eng. in Systems Engineering (1991) and Ph.D. in Control Science and Engineering (2001) from Xi’an JiaoTong University. His research interests include: modeling and simulation of complex systems, planning and scheduling of manufacturing systems, parallel and distributed simulation.

Volume 83, Number 10 SIMULATION


719