Supporting Large-Scale Distributed Simulation Using HLA - CiteSeerX

Supporting Large-Scale Distributed Simulation Using HLA TAINCHI LU, CHUNGNAN LEE, and WENYANG HSIA National Sun Yat-Sen University and MINGTANG LIN Chin Min College

This article describes the design of a Web-based environment to support large-scale distributed simulation using Java and IEEE standard P1516 high level architecture (HLA) framework and rules. Based on the run-time infrastructure (RTI) services within the HLA and Java application programmer’s interfaces (APIs) of the RTI, the proposed HLA-based environment provides an architectural foundation to enhance interactivity, portability, and interoperability for Web-based simulations. In addition, the proposed architectural design not only provides a client/server mechanism for simulation on the Web, but also supports a distributed federation execution over the network. A 3-level control mechanism (3LCM) was implemented to HLA-based middleware federateServer, in order to adaptively maintain information consistency and minimize message traffic for distributing information among client hosts and the federateServers. A dynamic filtering strategy (DFS), associated with the data distribution management (DDM) in the HLA RTI, is proposed to minimize false positive updates and enhance the filtering efficiency of subscription regions within an HLA federation. To verify the feasibility of this prototype, a distributed discrete event simulation application in Java was developed and performance of the proposed modeling design and Java RMI’s distributed object model presented. From the experimental results, we show that the proposed environment based on HLA using 3LCM and DFS is workable and practical for supporting a large-scale distributed simulation. Categories and Subject Descriptors: I.6.8 [Simulation and Modeling]: Types of Simulation— Distributed; B.4.4 [Input/Output and Data Communications]: Performance Analysis and Design Aids—Simulation General Terms: Algorithm, Performance Additional Key Words and Phrases: Data distribution management (DDM), distributed interactive simulation (DIS), high level architecture (HLA), modeling and simulation, networked virtual environment, run-time infrastructure (RTI)

Authors’ addresses: T. Lu, C. Lee, and W. Hsia, Institute of Computer and Information Engineering, National Sun Yat-Sen University, Kaohsiung, Taiwan; M. Lin, Department of Management Information Systems, Chin Min College, Miaoli, Taiwan. Permission to make digital / hard copy of part or all of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and / or a fee. © 2001 ACM 1049-3301/00/0700 –0268 $5.00 ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000, Pages 268 –294.

Supporting Large-Scale Distributed Simulation

•

269

1. INTRODUCTION Web-based simulation [Page 1998], which is a grand challenge technology, allows multiple users connected to a Web-based networked synthetic environment to interact simultaneously with one another and facilitate interoperability among simulations. This has become increasingly important for military training, in entertainment, and industrial and academic sectors. A variety of relevant research areas have been studied within this domain [Zyda and Sheehan 1997], such as network architecture design [Macedonia et al. 1995]; distributed virtual environments [Chim et al. 1998; Broll 1997]; simulation using Web technologies [Evans and Rogers 1997]; and Web-based modeling design [Page et al. 1997]. According to Fishwick et al. [1996], the potential impact of simulations is primarily in education and training, publications, and simulation programs. However, one of the bottlenecks for developing a workable networked simulation environment is the limited network bandwidth. Attempts to overcome bandwidth limitations have focused on increasing network speed and reducing unnecessary information transmissions over the network. To attack these problems, several common technical frameworks for modeling and simulation (M&S) have been successfully identified to date. The U.S. Department of Defense M&S master plan [U.S. DOD 1996] developed a common technical framework for M&S, and the high level architecture (HLA) is one of subobjectives within this master plan. Currently, the HLA is defined by three components, namely federation rules [U.S. DOD 1998a], the HLA interface specification [U.S. DOD 1998b], and the object model template (OMT) [U.S. DOD 1998c]. Federation rules are used to govern the interaction of simulations in a federation. Interface specifications define the run-time infrastructure (RTI) services. Object model templates provide a common method for recording information. Within the HLA, federations comprise federates that exchange information in the form of objects and interactions through the network. The HLA RTI provides a set of services used to interconnect simulation during a federation execution. These RTI services are grouped into the six categories: federation management, declaration management, object management, ownership management, time management, and data distribution management. This article focuses on the functionality of the HLA data distribution management (DDM) services. We use the data distribution management [Morse 1997] in the HLA to reduce message traffic over the network and data sets required to be processed by a receiving host. The fundamental concept to support value-based filtering of DDM services is the routing space. Based on the routing space in the HLA DDM, federates can either express an interest in receiving or sending data via subscription and update regions. The subscription and update regions can dynamically change in both size and location over time as an object’s state in the routing space changes. So far the RTI does not automatically provide information filtering based on known functions, it only determines which federates should discover which objects by matching subscription and update regions. ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

270

•

T. Lu et al.

Unfortunately, when a subscribing federate extends its subscription region in a routing space only, false positive updates can occur explicitly. As a result, the filtering efficiency of a subscription region deteriorates if the attribute updates fall into the extended section. Hence, in this article, we propose a dynamic filtering strategy (DFS) to solve the false positive updates and add more accurate filtering ability to the RTI applications. This scheme depends upon filtering efficiency to dynamically change the position of the subscription regions or reduce the regions in order to avoid unreasonable extension of the subscription regions. A reduction in the extension of the regions can reduce false positive updates. In addition to the above, we present a Web-based distributed architectural design in which a 3-Level Control Mechanism (3LCM) [Lu et al. 1999] is integrated into an HLA-based simulation modeling. The 3LCM is a technique to support a large number of clients and maintain as much information consistency as possible. It comprises three different kinds of methods, including the shared repository technique, information interest management, and predictive simulation scheme, to dynamically maintain a reasonable turnaround time and the information consistency of shared objects within the Web-based environment. However, the original design is limited in that the system using this approach must be constructed as a centralized model. Therefore, in order to provide a more flexible and practical Web-based distributed simulation platform, we incorporate the 3LCM with an HLA federation execution. The incorporation of 3LCM with federateServer can greatly increase the number of client hosts. In terms of federation management in HLA, we let an individual host execute the 3LCM control mechanism (i.e., we designate this host as a federateServer) as a federate participating in a federation. However, all of the interactions, communications, and regulations of federates must rely upon RTI services. The federateServer (i.e., an HLA federate) can join the federation by applying the services of the RTI federation management. Based on a common high level simulation architecture within the HLA and the proposed architectural design, it can support and facilitate the interoperability of all types of models and simulations on a large-scale Web-based distributed simulation environment. The remainder of this article is organized as follows. Section 2 briefly describes an overview of contemporary methodologies for networked simulation models. In Section 3, we give the fundamental definition of the HLA DDM terms and describe the design and implementation issues of the proposed HLA-based architectural design. Section 4 describes the testing simulation environment using the HLA federate interface and Java RMI’s distributed object model, respectively, and presents the simulation results. In the last section some conclusions are summarized. 2. METHODOLOGIES FOR NETWORKED SIMULATION MODEL In the past decade, the area of distributed simulation has been studied intensively by many research agencies, such as the Simulation Interoperability ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

•

271

Standards Organization (SISO), MITRE Corporation, and the NPSNET Research Group. The U.S. Department of Defense (DOD) in particular has driven the evolution of standards to support the interoperability of simulations in order to reduce costs and improve quality. In this section we briefly review two IEEE standards: the DIS and HLA. 2.1 The Distributed Interactive Simulation (DIS): IEEE 1278 Distributed interactive simulation (DIS) [IEEE 1995] is a standard set of messages and rules, entitled the Protocol Data Unit (PDU), used for sending and receiving information through the network. IEEE Standard 1278 defines the protocol data units (PDUs) of DIS. The DIS protocol uses 27 different kinds of messages (i.e., PDUs) to exchange information among simulation nodes. Particularly, the DIS uses a predictive algorithm, known as the dead reckoning model (DRM). It infrequently transmits state update packets and approximates the real state on the basis of previously received update information. A DRM consists of two major components, a prediction technique and a convergence technique. The prediction technique is used to estimate the future value of the shared state by referring to past and current update information. The convergence technique is used to correct the predicted state without incurring visual distortion. DRM notation consists of three elements: the first specifies whether the coordinate system of the entity is fixed (F) or rotating (R); the second element specifies whether the rate is constant (P) or varying (V); the third specifies which coordinate system is to be used, world (W) or body (B). In less than a decade, DIS has matured into a new generation of modeling and simulation technology, known as the HLA. In contrast to DIS, which uses a full broadcast distribution approach, HLA supports the efficient routing of data among simulations. 2.2 High Level Architecture (HLA): IEEE P1516 HLA is an integrated architecture, which is developed to provide a common architecture for modeling and simulation. The HLA consists of three components as shown in Table I, including federation rules [U.S. DOD 1998a]; interface specification [U.S. DOD 1998b]; and the object model template (OMT) [U.S. DOD 1998c]. Federation rules describe the responsibilities of federates and their relationships with the run-time infrastructure. The interface specification identifies how federates will interact with the federation via the RTI. The OMT provides a common framework for all objects and interactions managed by a federate. The RTI also provides sufficient services for simulation systems and specifies the federate interface using the following languages: CORBA IDL, C11, Ada 95, and Java. Moreover, the RTI services are classified into six categories that describe the interface between federates and the RTI. For example, federation management services allow the creation and destruction of a federation execution and data distribution management services provide information to the RTI for supporting efficient ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

272

•

T. Lu et al. Table I.

HLA Components

HLA components

Descriptions

Federation rules

There are ten rules. Five related to the federation and five related to the federate.

Interface specification

Six categories in the RTI: Federation Management (FM) Declaration Management (DM) Object Management (OM) Ownership Management (OWM) Time Management (TM) Data Distribution Management (DDM)

Object Model Template (OMT)

Federation Object Model (FOM) Simulation Object Model (SOM) Management Object Model (MOM)

routing of data [Tacic and Fujimoto 1997]. For detailed descriptions of HLA DDM, consult Cohen and Kemkes [1997]. 3. DESIGN OF WEB-BASED DISTRIBUTED SIMULATION In this section we first give the fundamental DDM definition in HLA, then outline the design issues for the Web-based large-scale distributed simulation. It consists of three major components: the HLA federation, the RTI console, and the federateServer with 3LCM. We also give a detailed discussion of the RTI console, the federateServer with 3LCM, and DDF. Finally, we present the implementation issues. 3.1 Fundamental DDM Definition In HLA In this section we briefly introduce DDM in HLA. Conceptually, DDM provides a flexible and extensive mechanism for isolating publication and subscription interests further. DDM services are used by federates to reduce the transmission and the reception of irrelevant data. The basic terms in the DDM are given as follows: ● Federate: All applications participating in a federation are called federates. ● Federation: A set of interacting federates that are used as a whole to achieve some specific objectives. ● Dimension: A dimension is a named coordinate axis segment. All dimensions must be based on the same coordinate axis and have the same lower and upper bounds. ● Routing space: A routing space is a multidimensional coordinate system through which federates can declare their intention by area of interest to send or receive objects’ attribute values. The notation of a routing space is expressed as RS d 5 ~alower, aupper! where d is the dimension of a routing space, a lower is called axis lower bound, and a upper is called axis upper bound. ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

3D Routing Space RS

•

273

3

Y-Axis Upper Bound

SR1 UR2

F3

UR1 Overlap Region

F1 Y

Z

X-Axis Lower Bound

Z-Axis Upper Bound

X

SR Subscription Region X-Axis Upper Bound

UR Update Region

F F2 Fig. 1.

Federate

An example of a 3D routing space.

● Range: A range is a continuous interval on a dimension defined by an ordered pair of values. The notation of a range is expressed as r 5 ~rlower, rupper!, where rlower is called range lower bound and rupper is called range upper bound. ● Extent: An extent is a sequence of ranges, one for each dimension in a routing space. ● Region: A region is a set of extents bounded to the same routing space, and is defined as a subspace within the routing space. Basically, the RTI provides a default region for every routing space and the default region covers the entire routing space. In general, a region used for the update of instant attributes or sending interactions is called an update region. The notation of an update region is expressed as UR. Similarly, a region used for subscription of either attributes or interactions is called a subscription region. Notation of a subscription region is expressed as SR. Figure 1 shows an example of 3D routing space usage in which it comprises three federates. The subscribing federate F3 specifies its subscription region SR1, to discover objects of interest and to receive the objects’ attribute values. The publishing federate F2 publishes the objects’ attributed values associated with update regions from the update region UR2 to the subscription region SR1. In contrast, the update region UR1 associated with a publishing federate F1 does not overlap with the subscription region SR1. Therefore, objects’ attribute values of the update region UR1 will not be routed from the federate F1 to the federate F3. ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

274

•

T. Lu et al.

3.2 Design Issues We illustrate our Web-based distributed simulation model in Figure 2. There are three major components: the HLA federation, the RTI console, and the federateServer. A federate that runs the 3LCM is called the federateServer. In the federateServer, it uses the 3LCM to reduce information consistency updates when the number of participants is large. The major principle of 3LCM is to use shared repository techniques for a small number of clients, information interest management for a medium number of clients, and a predictive simulation scheme for a large number of clients, to maintain well-defined view consistency within a simulation environment. We use the HLA to establish the underlying platform for distributed simulation. During a federation execution, all federates must rely on the RTI to interact and exchange data. Up to this point, the federateServer represents itself as a federate to join or leave the federation. The federation via the RTI responds asynchronously to federates’ requests. In other words, the communication and interaction among federateServers is through the RTI interface. Each federateServer manages a federation execution and must be responsible for regulating information consistency among client hosts on the Web. The communication and interaction among the federateServer and clients hosts is via socket connection. In order to get fewer false positive updates and reduce the redundant attribute updates when subscription regions overlap with update regions in routing space, we proposed a dynamic filtering strategy (DFS). The detailed description of the RTI console, the design of a federateServer, and the DFS are presented in the following sections. 3.3 The RTI Console We adopted the RTI 1.3 Version 5, published by the software distribution center of the Defense Modeling and Simulation Office (DMSO), to build the RTI console and implement the HLA interface specification. The RTI software can be executed on a standalone workstation or executed over an arbitrarily complex network (e.g., Fast-ethernet, ATM, FDDI, and so on). The purpose of the RTI console is to manage the creation and destruction of federation executions, listen to a well-known port, and provide a console interface for manual operations. Federates must communicate and interact with one another via the RTI. In order to understand the performance of distributed simulation executions in Java on the HLA-based design and other types of Web-based distributed simulation platforms, we use Java remote method invocation (RMI) [Orfali and Harkey 1997] as the other platform to support a Webbased distributed discrete event simulation. An RMI is the object-oriented analog of traditional remote procedure call (RPC) for distributed object invocation in Java. We built an RMI server based on a master-slave architecture and use bidirectional communication model by which the RMI server can also invoke the remote methods defined in client-side stubs. Hence, the RMI server can be any object whose methods are invoked ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

•

275

Federation Federate Join Federation

federateServer Three-Level Control

Client Host

RTI

Client Host

Inter-Process Communications

RTI Console Join Federation federateServer Three-Level Control

Join Federation

federateServer Three-Level Control

Client Host

Client Host

Client Host

Client Host

Client Host

Federate

Federate

Fig. 2.

Prototype of a Web-based distributed simulation model upon a Java platform.

remotely, so a client is an object that invokes a remote method. Thus, it becomes common practice to use RMI to design and implement distributed simulation. Figure 3 shows both the interaction model using the RTI and Java RMI. The RTI components with one RTI console execute the RtiExec process to manage the creation and destruction of the HLA federation executions and the participating federates. The simulation application uses libRTI, which is the library service specified in the HLA interface specification for federate developers, to communicate with the RtiExec and other federates. In contrast, an RMI object is a remote Java object whose methods can be invoked from another Java virtual machine through the network. In the following, we will present our middleware and the implementation issues towards the HLA-based design and the RMI remote object. 3.4 The Middleware: federateServer The primary consideration for a flexible networked virtual environment (NVE) in providing distributed simulations is to adapt to a variety of clients using an appropriate strategy and control for different levels of information consistency [Frécon and Nöu 1998]. When there are only a few clients in the NVE, high overhead and network bandwidth limitations are not major problems for the NVE. So it is possible to provide absolute consistency and to maintain reasonable turnaround time at the same time. When more and more clients join the virtual environment, it becomes ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

276

T. Lu et al.

•

Comparisons of the interaction model of RTI Components and Java RMI Remote Object

Simulation Application (libRTI)

federate

RTI Components Simulation Application (libRTI)

RtiExec (RTI Console)

federate

Simulation Application (libRTI)

federate

Simulation Application (libRTI)

Simulation Application (objectRMI)

federate

Client

Simulation Application (objectRMI)

Remote Interfaces (RMI Server)

Client

Simulation Application (objectRMI)

Client

Java RMI Components Simulation Application (objectRMI)

Client

Fig. 3.

RTI components vs. Java RMI components.

impractical to maintain absolute information consistency. Thus, some absolute information consistency can be sacrificed in order to not only maintain smooth interactions but also reduce update frequency. We specify three kinds of shared objects in the NVE: an active object, a passive object, and an interrelated object in which actual real-life behaviors are specified [Lu et al. 1999]. Different types of behaviors require different amounts of synchronization. An active object has autonomous behaviors and embedded problem-solving mechanisms. In contrast to an active object, a passive object has a completely deterministic or independent state, which can only be influenced by active objects. We use the concepts of area of interest (AOI) and shared objects to limit the data, and adopt the blind broadcasting approach to update the more important shared objects (i.e., active objects) in the NVE. In order to further reduce the frequency for transmitting updates and support a large-scale NVE, we may consider a predictive simulation scheme, which is used to predict the true state of virtual space in client hosts. Based on the considerations mentioned above, we let an individual host run the 3LCM (i.e., we specify this host as a federateServer) as an HLA federate within a federation. The 3LCM executed in a federateServer can be used to select an appropriate shared state maintenance technique for regulating the information consistency of shared objects among a federateServer and client hosts. In 3LCM management particularly, ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

•

277

we also divide an HLA federation execution into three steps. The event trace of an interaction diagram of client hosts, federateServers with 3LCM, and a federation is shown in Figure 4. If the specified federation does not exist initially, one of the federateServers creates a new federation and all federateServers join this federation; the client hosts can then connect to the federateServer, download all Java applets from the server, and obtain the mutual interactions within the federation via the federateServer. In the first step we use the RTI declaration management (DM) to control where the data are sent or received by a federate. We also follow the characteristics for attributes in the attribute table of the federation object model (FOM) to create, modify, and delete objects and interactions in client sites. As the federateServer receives the object attributes, it manages object identification, facilitates object registration and distribution, and coordinates attribute updates among federates via the RTI. In the second step, participants in client hosts must express their interest in objects to narrow down the data and send their requests to the connected federateServer. When this federateServer receives those messages, it invokes DDM services to tell the RTI to discover which objects fall within the extent of the region specified by another federate. In the third step, client hosts transmit nonregular updates and use a predictive simulation scheme, which is a second-order polynomial prediction, to manage states at remote client hosts. Moreover, the federateServer not only uses DDM services to limit the scope of interest in data, but also adopts a dynamic filtering strategy (DFS) to enhance the filtering efficiency of update and subscription regions. Finally, all client hosts disconnect the federateServer, and the federateServer terminates a federate’s participation in a federation. However, the federateServer with DFS and 3LCM switches the level on the basis of the number of connecting clients, and is not synchronized by logical time. 3.4.1 Three-Level Control Mechanism (3LCM). The fundamental goal of 3LCM is to maintain information consistency and control the dynamicshared state in a large-scale NVE. This approach consists of three major techniques for dynamic-shared state maintenance: the shared repository technique, information interest management, and the predictive simulation scheme. At the first level, the major requirement is to achieve absolute consistency of shared objects, including accurate dynamic-shared state, identical view, reasonable turnaround time, and high reliability. An accurate dynamic-shared state is particularly fundamental in creating realistic simulation environments. In order to provide accurate information and absolute consistency throughout the network, each client host must send update information to the server when the shared objects are modified, the server then notifies other clients immediately. Our implementation uses the Java language to build a shared repository in the server and spawn the Java server thread for each participating client in order to listen to the client’s requests. The shared repository contains the current value for the shared state used in the NVE and stores the NVE’s current state in a centralized database. Consequently, the shared repository guarantees that ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

278

T. Lu et al.

•

Client Hosts

federate Servers

Federation

Create and Join a Federation (FM)

Initialization

Connect and Download Java Applets

Define and Create Objects Publish the Desires for Object Attributes (DM)

The First Step

Publish Shared States of Objects Create and Register Objects (OM)

Reflect Objects' Attributes and Interactions Broadcast Object's Updated States

Define Interrelation Among Shared Objects Create Update and Subscription Regions (DDM) Publish Shared States of Objects

The Second Step

Discover Objects and Publish Attributes

Reflect Objects' Attributes and Interactions Blind Broadcast Objects' Updated States

Predictive Simulation Scheme Dynamic Filtering Strategy (DFS)

The Third Step

Reflect Objects' Attributes and Interactions Send Updated States and Correct Residuals

Leave federate Servers

Halt

Fig. 4.

Resign and Destroy the Federation

An interaction diagram among client hosts, federateServers, and an HLA federation.

each client can see the updates in the same order and maintain absolute consistency among the participating client hosts. When the Java thread in the server receives the update information from a client, it will keep the messages in the shared repository in order and immediately notify other client hosts by broadcasting the update information.The Shastra system, developed at Purdue University and used as a centralized server in a collaborative environment, is a typical example of using a shared repository to provide absolute consistency [Anupam and Bajaj 1994]. As more and more clients connect to the NVE server in order to join the shared virtual space, the communication overhead will increase dramatically. As a result, the NVE cannot afford to support absolute consistency and an identical view among all client hosts. This is the proper time to switch from the first to the second level in order to maintain the dynamic shared state. In the second level, we use blind broadcasting to send and receive the update information through the network for less important shared objects (i.e., passive objects). The blind broadcasting approach has been used to support many critical NVE applications. A typical example is found in the VEOS system, which was developed at the University of Washington and used an “epidemic” approach to reduce the broadcast message overhead [Bricken and Coco 1994]. Blind broadcasting can maintain ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

•

279

high interaction among a medium number of clients, but it sacrifices absolute consistency and faces the issue of network jitter. As a result of network jitter, attribute updates may not be received at a steady rate and the display may fluctuate. Hence, when the number of simultaneous clients increases over a predefined threshold at the second level, the predictive simulation scheme is adopted to accommodate even more simultaneous clients. The predictive simulation scheme is used to estimate the future value of the shared state by referring to the past and current update information. It transmits state update packets less frequently and approximates the real state on the basis of previously received update information. Generally speaking, due to the easy programming and fast computing in predicting object position, a second-order polynomial prediction is the most popular NVE prediction technique in use today. For example, each client predicts the next position of the shared objects using the equation, x9 5 x 0 1 v 0 *Dt 1 1 / 2aDt 2 , where x9 is the next position, x 0 is the initial position, v 0 is the velocity, and a is the acceleration. We can derive the initial position x 0 , velocity v 0 , and acceleration a from the pervious update information of this shared object. Because predictive simulation runs in the client hosts, the NVE server can support more simultaneous clients without increasing significant overhead. However, it does not guarantee that all client hosts will share identical states about each shared object. Thus, it requires the client hosts to tolerate and adapt to potential discrepancies. 3.4.2 Dynamic Filtering Strategy (DFS). In this section we propose a DFS on continuous coordinates to get fewer false positive updates and reduce the redundant attribute updates when subscription regions overlap with update regions [Morse and Steinman 1997]. Within the strategy, we use a predictive method to predict the object’s next state in which the object is within an update region but not within an overlap region. In general, for the sake of simplicity for the DDM implementation, the subscribing federate can receive attribute updates from the publishing federate, even though the object itself is not within the overlap region. However, when an object is within the update region but not in the overlap region, the precision of the routing space implementation incurs an update problem on the object discovery and attribute reflection through the data distribution management in HLA. Figure 5 shows an example of the false positive updates in a 3D routing space. A federation FN1 consists of three federates, named F1, F2, and F3, respectively. Based on the HLA DDM services, the federation begins the process by defining a routing space. Federates can either create subscription region to narrow the scope of their data requirements or express their intention to publish specific attributes of objects. The HLA RTI determines which federates should discover which objects by matching subscription and update regions. The publishing federate updates an object’s attributes to the RTI. The RTI notifies all subscribing federates that they should reflect the updated attributes. In Figure 5, a federate F1 publishes a ship O1 and associates with an update region UR1. Similarly, ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

280

T. Lu et al.

•

F2

Federation FN1= {F1, F2, F3} FN1 defines a 3D routing space

SR1

3D Routing Space

Overlap Region

O2

Y

S^n- 2

UR1

S^n+1

O1

S^n

Current Real State Vector

S^n- 1

O3

Previous Real State Vector

→

S n+1 Predicted State

False Positive Updates

Vector

UR2 X F3

F1

Fig. 5. Prediction of the next state in which false positive updates occur within the extended section.

federate F2 publishes a satellite O2 and an aircraft O3 within an update region UR2; then federate F3 subscribes for the satellite and the aircraft in a subscription region SR1 specified via the RTI. When the update region UR2 and the subscription region SR1 of different federates overlap (i.e., the gray box shown in Figure 5), the RTI establishes communications connectivity between the publishing and subscribing federates. Federate F3 will receive attribute updates from the object associated with update region UR2. Obviously, attribute updates of the satellite O2 will be published to federate F3. As for the aircraft O3, it is outside the overlap region. We strictly define that the objects that are outside the overlap region, their attribute updates, will not be routed to the subscribing federates through this 3D routing space. Hence, the federate F3 is forced to expand its subscription region to receive the attributes of the object O3; then a false positive update occurs within the subscription region SR1 and the efficiency of subscription decreases. Conceptually the filtering efficiency of update and subscription regions can be characterized based on their intersections and extensions. Therefore, we first define an overlap region in a 3D routing space by the following formula:

Vol~OR! 5 Vol~SR! ù Vol~UR!, ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

(1)

Supporting Large-Scale Distributed Simulation

•

281

where Vol ~ OR ! is the abstract volume of an overlap region, Vol ~ SR ! is the abstract volume of a subscription region, and Vol ~ UR ! is the abstract volume of an update region. In addition, the impact index of subscription and update region intersection is defined by Vol ~ OR ! / Vol ~ UR ! ; then we can further define a filtering efficiency « , which is a function of impact index, subscription region and incremental volume, as follows:

«5

Vol~OR! Vol~UR!

3

Vol~SR!

~SRx 1 Dx!~SRy 1 Dy!~SRz 1 Dz!

,

(2)

where Vol ~ SR ! 5 SR x 3 SR y 3 SR z . SR x , SR y , and SR z are the length, width, and height of the original subscription region SR , respectively. Dx , and Dy , and Dz are the increments toward the length, width, and height for the expanded subscription region SR9 . From the the filtering efficiency « formula above, we can see that the smaller the extension of the subscription region, the more efficient the subscription region. Furthermore, as shown in Figure 5, we denote that the n th real state vector of an ˆ n and its n 1 1 th predicted state vector as S Y n11 . To enable object as S predictive processing in a subscribing federate, object discovery is provided only once by the RTI to a federate when the subscription region overlaps a remote object’s update region. A subscribing federate can receive each object’s update attributes, including location, velocity, acceleration, direction, and force affiliation. However, when an object within the overlap region moves to a new location that is outside the overlap region but is inside the update region, the subscribing federate stops receiving all updates of the attribute values from this specified object. Meanwhile, the Y n11 , of subscribing federate attempts to predict the n 1 1 th state vector, S the object. Within the proposed algorithm, we also define the boundary objects in which their real state vectors consist of any of x-axis lower bound, y-axis lower bound, z-axis lower bound, x-axis upper bound, y-axis upper bound, and z-axis upper bound, confirming the specified subscription region. The notation of real state vectors for the boundary objects can be x ˆ min y ˆ min z ˆ max x ˆ max y z ˆ min ˆ max , S n , S n , S n , S n , and S , respectively. represented by S n n x ˆ min S means that the n th real state vector for a specified object contains the n minimum value on x -axis coordinate bounded to the subscription region. On the other hand, these six vectors define the external bound for the distribution of objects inside the subscription region. Similarly, the n 1 1 th min x Y min y Y n11 predicted state vectors of the boundary objects are defined as S , S n11 , min z Y max x Y max y max z min x min x min x Y n11 Y n11 Y n11 Y n11 Y n11 S , Sn11 , Sn11 , and S respectively. We use ~~ S ! x~ S ! y~ S ! z ! t to represent x, y, and z components for the boundary object’s real state vector, min x ˆ n11 S , where t means the transpose of a vector. In order to consider the sampling effects of subscription regions, the algorithm uses both real state vectors and predicted state vectors of the boundary objects to determine

ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

282

•

T. Lu et al.

whether to expand or move the subscription regions along each coordinate axis. As shown in Algorithm I, the dynamic filtering strategy checks the current filtering efficiency or default threshold to determine how to sample the subscription region continuously over time. If the filtering efficiency is larger than the threshold, the subscription region can be expanded or moved along the x-, y-, or z- axis, depending on the interval of the maximum and minimum objects along that axis. Otherwise, the subscription region can just be moved. In other words, the subscription region can either move to a new location or expand the region to receive the attribute updates of the specified objects, depending on the filtering efficiency. Y n11 Otherwise, the algorithm predicts the n 1 1 th predicted state vector S and publishes these predicted attributes of objects to the subscribing federate via RTI. Algorithm I. The Dynamic Filtering Strategy. // Main function dynamic_filtering_strategy (): // The subscription region can be expanded if ( « $ threshold ) x x min x x max x x ˆ max Y n11 Y n11 if ~~ S ! x 2 ~ Sˆ min !x 1 ?S 2 Sˆ min ? 1 ?S 2 Sˆ max ? , SR x ! n n n n move_along_x_axis(); else expand_along_x_axis(); y y min y y max y y ˆ max Y n11 Y n11 if ~~ S ! y 2 ~ Sˆ min !y 1 ?S 2 Sˆ min ? 1 ?S 2 Sˆ max ? , SR y ! n n n n move_along_y_axis(); else expand_along_y_axis(); z z min z z max z z ˆ max Y n11 ˆ min Y n11 if ~~ S ! z 2 ~ Sˆ min !z 1 ?S 2 S ? 1 ?S 2 Sˆ max ? , SR z ! n n n n move_along_z_axis(); else expand_along_z_axis(); // The subscription region can only be moved x x min x x max x x ˆ max Y n11 ˆ min Y n11 ! x 2 ~ Sˆ min !x 1 ?S 2 S ? 1 ?S 2 Sˆ max ? , SR x else if ( ~ S n n n n y y min y y max y y ˆ max Y n11 ˆ min Y n11 && ~ S ! y 2 ~ Sˆ min !y 1 ?S 2 S ? 1 ?S 2 Sˆ max ? , SR y n n n n max z min z min z min z max z max z ˆ n ! z 2 ~ Sˆ n ! z 1 ? S Y n11 2 Sˆ n ? 1 ? S Y n11 2 Sˆ n ? , SR z ) && ~ S move_along_x_axis(); move_along_y_axis(); move_along_z_axis(); // The subscription region cannot be expanded or moved else curr curr ˆ n11 Y n11 let S 5 S and stop routing this object’s attributes at next time interval; // Subroutines move_along_x_axis(): x min x x Y n11 SR shift to ~ Sˆ min !x 2 ?S 2 Sˆ min ? along x-axis; n n move_along_y_axis(): y min y y Y n11 SR shift to ~ Sˆ min !y 2 ?S 2 Sˆ min ? along y-axis; n n move_along_z_axis():

ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

•

283

z min z z Y n11 SR shift to ~ Sˆ min !z 2 ?S 2 Sˆ min ? along z-axis; n n expand_along_x_axis(): min x x max x x Y n11 Y n11 ˆ max SR x is increased by ? S 2 Sˆ min ? 1 ?S 2 S ?; n n expand_along_y_axis(): min y y max y y Y n11 Y n11 ˆ max SR y is increased by ? S 2 Sˆ min ? 1 ?S 2 S ?; n n expand_along_z_axis(): min z z max z z Y n11 Y n11 SR z is increased by ? S 2 Sˆ min ? 1 ?S 2 Sˆ max ?; n n

3.5 Implementation Issues In this section we describe distributed simulation implementation issues in HLA [U.S. DOD 1998d]. According to U.S. DOD [1998c], the HLA object model template (OMT) prescribes the format and syntax for recording information in HLA object models, in which it includes objects, attributes, interactions, and parameters. First, we follow the HLA OMT to define our object models. The attribute table, complex datatype table, and routing space table of a federation object model (FOM) are also identified to facilitate data exchange of all object attributes during the course of an HLA federation execution. Second, we list our Java pseudocodes to describe the conceptual architecture towards the proposed design in HLA-based distributed simulation environment. 3.5.1 Representation of Shared Objects. An FOM attribute table is designed to provide descriptive information about all object attributes represented in a federation. An HLA object model supports representation of the characteristics for attributes in the attribute table specified as a fixed set of attribute types: {,Object., ,Attribute., ,Datatype., ,Cardinality., ,Units., ,Resolution., ,Accuracy., ,Accuracy Condition., ,Update Condition., ,T/A., ,U/R., ,Routing Space.}. In the following we briefly review the rationale for the attribute table within HLA OMT and describe usage of our attribute table, complex datatype table, and routing space table, respectively. All object attributes whose values can be exchanged during the process of an HLA federation execution must be well documented in the attribute table of an FOM. In the attribute table, the Object field specifies the class of objects and the Attribute field identifies the attributes by which all defined attributes make up all the characteristics of a specific object. The Datatype field refers to the datatype of the attribute. The Cardinality field records the size of an array or a sequence. If the size of an array or a sequence is unbounded, we specify a designation of 11 in this field; otherwise we enter 1 in this field to represent a single instance of the datatype in the Datatype field. The Units field contains the units for each attribute. We can designate them as m, km, kg, and so on, in the Units field whenever such units exist; otherwise N/A (i.e., “not applicable”) is entered in this field. The Resolution field has different kinds of entries based on the kinds of attributes. The Accuracy field is used to capture the maximum deviation of the attribute value. Conceptually the value of this ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

284

•

Table II.

T. Lu et al. User-Defined Characteristics for Object Attributes in the Attribute Table Attribute Table

Object Vehicle

Attribute

Vehicle_ID Current_Position Destination_Position Creator_Name

CardiResoAccuracy Datatype nality Units lution Accuracy Condition Long Location Location String

1 1 1 1

N/A M M N/A

N/A 1 1 N/A

N/A Perfect Perfect N/A

Update Type

Update Condition T/A U/R

Routing Space

N/A Static N/A N UR N/A Always Conditional Voluntary TA UR ThreeDSpace Always Conditional Voluntary TA UR ThreeDSpace N/A Static N/A N UR N/A

field can be perfect for many discrete attributes. The Accuracy Condition field contains conditions required for the given accuracy in a simulation execution. Generally speaking, the value of this field can be a particular type of update algorithm, or its value can be an unconditional always. Both the Update Type and the Update Condition fields record the update policies for an attribute. The Update Type field can take any of the values from the set {static, periodic, conditional}. The Transferable/Acceptable (T/A) field designates whether an attribute is transferable (T) from a federate or acceptable (A) by some federates in the federation. In an FOM, the only valid values in this field are TA or N (i.e., “not transferable or acceptable”). The Updateable/Reflectable (U/R) field of the attribute table identifies the current capabilities of a federate with respect to an attribute, which is either updateable (U) or reflectable (R) or both. For a federation, the appropriate value of this field is UR, because all attributes in an FOM are both updateable and reflectable. The Routing Space field records the association of object attributes within a routing space if a federation is using DDM services. However, an HLA federation without DDM services must be set to N/A in the Routing Space field. In Table II, we list the FOM attribute table and define a Vehicle object, which is characterized according to the four attributes: Vehicle_ID, Current_Position, Destination_Position, and Creator_Name, respectively. Table III specifies the “Location” datatype, which matches a datatype field from the attribute table. In addition, a routing space table (shown in Table IV) also defines the 3D routing space, ThreeDSpace, in a federation. 3.5.2 Federation Execution with 3LCM. In Algorithm II we list the Java pseudocodes describing the conceptual operation of a federateServer. The federateServer initially applies federation management services to create and join the federation. Then declaration management services are specified to allow the federate to declare its desire to publish object attributes and interactions. In the next step, the federateServer must build a serverSocket and accept inbound client connections. When a new client host connects to the federateServer, the server will spawn a new thread to handle clients’ requests. Moreover, the server will dynamically adjust the control level to support an appropriate number of simultaneous clients at each level and maintain a reasonable system performance and turnaround time. When the degree of consistency reaches the third level (i.e., the level using the predictive simulation scheme), the client applet will be notified ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation Table III.

•

285

User-Defined Specification of Fields Matching a Datatype Entry from the Attribute Table Complex Datatype Table

Complex data type

Field name

Location

LocX LocY LocZ

Data type Cardinality Long Long Long

Units

1 1 1

Table IV.

Resolution Accuracy

M M M

1 1 1

Perfect Perfect Perfect

Accuracy condition Conditional Conditional Conditional

Routing Space Table

Routing Space Table Routing space

Dimension

Dimension type

ThreeDSpace

DimX DimY DimZ

Long Long Long

Dimension range/set [-2 [-2 [-2

31 31 31

–2 –2 –2

31 31 31

-1] -1] -1]

Range/set units

Normalization function

M M M

Linear(LocX) Linear(LocY) Linear(LocZ)

by the federateServer to spawn another new Java thread to predict the true state of other shared objects. Because the predictive simulation in the form of Java applet runs upon the client hosts, the load in the server can be kept within a reasonable range by the control mechanism, and distributed computing is possible. Algorithm II. // The federateServer class public class FederateServer extends NullFederateAmbassador { private ServerSocket serverSocket5new ServerSocket(); // Build the network connection private RTIambassador rtiamb5new RTIambassador(); // Build the RTI connection rtiamb.createFederationExecution(); // Create a federation execution rtiamb.joinFederationExecution(); // Join a federation execution rtiamb.publishObjectClass(); // Declare the attributes to the RTI rtiamb.subscribeObjectClassAttributes(); // Subscribe the attributes from the RTI while(true){ try { (new ClientServiceThread(serverSocket.accept())).start(); // Listen the connection from clients }catch(IOException e) rtiamb.resignFederationExecution(); // Leave the federation execution rtiamb.destroyFederationExecution();}}} // Terminate the federation execution // The ClientServiceThread class public class ClientServiceThread extends Thread { public static level51, clients50; public void run() { synchronized(getClass()) { // Synchronization control if(degreeOfLevel(clients11)) switchTo(level); } // Change to a specified control level ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

286

•

T. Lu et al. while(true){ Vehicle objectInstance5ReceiveClientObject(); switch(level) { case 1: SRTManagement(objectInstance); break; // Shared repository technique case 2: IIMManagement(objectInstance); break; // Information interest management case 3: PSSManagement(objectInstance); break; }}}} // Predictive simulation scheme

4. SIMULATION In this section we address three typical sets of experiments toward the distributed discrete event simulation. First, we used 3LCM to measure turnaround time, and compared it to the shared repository technique adopted by many client/server systems (e.g., the Shastra system) to maintain the dynamic shared state. An approximate Poisson discrete distribution was used to establish the performance evaluation model. The purpose of our first experiment was to study the performance of 3LCM on a centralized server when the number of client hosts increased from one to the maximum predefined threshold. In the second experiment, we used Java RMI’s distributed object model to both build an RMI server and to develop the HLA-based simulation environment on a Java platform, using HLA RTI services to run client/server discrete event simulation [van Halderen et al. 1998]. The same functional simulation program was run on these two different platforms to evaluate their performance. In the third experiment, a predefined number of client hosts were connected to the five federateServers, one at a time, in a sequential order. Then, turnaround time was measured. All federateServers with 3LCM were switched automatically to an appropriate control level, depending on the number of connected client hosts. More detailed descriptions regarding these three experiments appear below. 4.1 Experiment 1. Performance of a Centralized Server Using 3LCM on the Web This experiment presents the simulation results of the 3LCM running on a centralized server. In general, the network bandwidth, turnaround time, and consistency delay are the uncertain parameters in the network environment. In other words, it is difficult to use network bandwidth, turnaround time, or consistency delay to estimate how many simultaneous clients can be supported by a Web-based simulation environment. Consequently, we used an approximate Poisson discrete distribution to approximate the maximum number of simultaneous clients allowed, based on the mean client connection rate of the server, m , which is a known and well-defined parameter and can be controlled throughout the whole experiment. We assumed that clients arrive at a mean rate of nine per unit time, and derive the maximum number of simultaneous clients as 20 (i.e., we assumed that the maximum number of clients connected to the server is r ). ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation Table V.

•

287

Critical Parameters of the Three-Level Control Mechanism Predefined Client Hosts

O l e / k!>1f l 5E ~ X ! 59f r 5X520 x

F ~ x ! 5P ~ X#x ! 5 Control Level

Level 1: Shared repository technique Level 2: Information interest management Level 3: Predictive simulation scheme Amount

k 2g

k50

Maximum number of simultaneous clients for one centralized server is 20. Threshold

Clients

Cumulative clients

20% (1 / 5 r )

4

4

30% (3 / 10 r )

6

10

50% (1 / 2 r )

10

20

20

20

100% (r )

In Table V, we list the threshold of simultaneous and cumulative clients for each level. The threshold field means that a predefined percentage of simultaneous client hosts is allowed to join in the server; they are 20%, 30%, and 50%, respectively. The testing environment consisted of 20 Pentium II-200 MMX as clients and one UltraSparc 1 as the centralized server. We used the shared repository technique, information interest management, predictive simulation scheme, and our proposed 3LCM to run the experiments in the same testing experimental environment. To emulate information interest management using the blind broadcasting approach, we assumed that the updated rate will be delayed. In other words, we reduced the frequency of sending update information to clients. The reduction rate was 15% of the shared repository. We further reduced the updated rate to emulate predictive simulation by using the dead reckoning model. The reduction rate was also 15% of blind broadcasting. The number of clients kept increasing until 20 hosts were participating simultaneously. The experimental results are shown in Figure 6. The vertical axis is the turnaround time, which can be defined as follows:

Tclient2to2server~l, m! 5 MPT~l! 1 ST 1 Max1#i#m~CSCTi!,

(3)

where T client2to2server is the maximum turnaround time measured among m clients that sent requests to the server to execute a distributed simulation and receive the returned simulation results. l is the level of information consistency; MPT ~ l ! is the message processing time of the server on the control level l ; ST is the simulation time for a centralized server; and CSCT i is the client-to-server communication time (including sending and receiving times). The horizontal axis is the number of participating client hosts. From the results we can see that the turnaround time T client2to2server (1,20) for the shared repository technique increased very significantly as the number of participants increased. To the contrary, the 3LCM kept a ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

288

T. Lu et al.

•

Turnaround Time vs. Number of Client Hosts 30000

Turnaround Time (ms)

3LCM Shared Repository Tech.

25000

Info. Interest Management Predictive Sim. Scheme

20000 15000

Level 3

Level 2

Level 1 10000 5000 0 1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Number of Participating Client Hosts Fig. 6.

Turnaround time versus the number of simultaneous client hosts using 3LCM.

reasonable turnaround time, T client2to2server (0,20) (where l equals zero, meaning that we used 3LCM to measure the turnaround time) for a large number of clients. Although the predictive simulation scheme kept a significantly low turnaround time T client2to2server (3,20), it sacrificed information consistency throughout the entire simulation. As we can see, there was an obvious decrease in the curve of 3LCM where the number of client hosts was 4 and 10, respectively, due to level switching at the predefined thresholds. Figure 6 also shows the range for each level. 4.2 Experiment 2. HLA Distributed Model vs. the RMI Distributed Object Model In our second experiment we studied the HLA-based distributed model to support distributed simulations and we evaluated the simulation performance of the proposed DFS. We first measured HLA DDM performance in terms of the turnaround time using RTI 1.3V5 software provided by the HLA Software Distribution Center (SDC). Moreover, we incorporated the DFS function with the HLA DDM services. The simulation program we developed required a number of entities and an event loop to run for a fixed number of iterations. During the event loop, the location of the entities changed in a random way. There were at least two default regions for each federate: a transmission update region and a transmission subscription region used by an entity. These regions were used to provide the routing of data by federates when an overlap region occured. The value of the turnaround time, measured from an HLA distributed federation execution with DDM services, can be defined as follows: ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

Tfederate2to2federation~n! 5 Max1#j#~FRCTj 1 STj!,

•

289

(4)

where T federate2to2federation is the maximum turnaround time a federate f communicates with a logical group of n federates in a federation for executing a distributed simulation via the RTI; FRCT j is the federate-toRTI coordination time among a federate j , the other n 2 1 federates, and the RTI console; ST j is the simulation time of a federate j . In addition, we used the same distributed discrete event simulation to measure the turnaround time on a Java RMI platform. Based on the definition of the discrete event simulation model [Pooch and Wall 1993], the system events modeled by method invocations are called event methods. An event method invoked by an entity is randomly generated for the RMI distributed model, too. When an entity sends a synchronous event message to the RMI server for object state updates and interactions, the entity immediately waits for the reply back from the other client hosts. Therefore, we built an event queue in the RMI server to keep the messages from client hosts in an FCFS order. The turnaround time, measured in the RMI distributed object model, can be defined as follows:

Tclient2to2RMI~n! 5 Max1#k#n~CSCTk 1 STk! 1 MPT,

(5)

where T client2to2RMI is the maximum turnaround time among n clients that send requests to the RMI server for executing a distributed simulation and receive the returned simulation results; CSCT k is the client-to-server communication time (i.e., including the sending and receiving times) for a client host k ; ST k is the simulation time of a client host k ; and MPT is the message processing time in the RMI server. In this experiment we used five workstations, including one Sun Enterprise 250, one Sun UltraSparc 60, two UltraSparc 2, and one UltraSparc 1, to build a prototype of the simulation environment. We designated an UltraSparc 1 to be either the RTI console on the HLA-based distributed model or the RMI server on the RMI distributed object model, and allocated five workstations to execute the simulation program. We evaluated the performance of the common HLA DDM services with and without the proposed DFS function. In Figure 7, the vertical axis is the turnaround time and the horizontal axis is the number of federates. From the simulation results we can see that the federation with a DFS function can further reduce the turnaround time. To study the performance of the HLA-based distributed model further, we built an RMI server based on the master-slave architecture and used an FCFS event queue to handle the events from the client hosts. In Figure 8, we observed that the turnaround time for the RMI distributed object model increases dramatically, reflecting a scalability problem. But the HLA-based distributed model, as shown in Figure 7, can maintain a stable turnaround time suitable for supporting a large-scale distributed simulation. ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

290

T. Lu et al.

•

5000

Performance Comparisons of HLA Model with DFS and without DFS

Turnaround Time (ms)

HLA Model without DFS function 4000 HLA Model with DFS function 3000

2000

1000

0 1

2

3

4

5

Number of Federates Fig. 7.

The simulation results of the HLA model with and without DFS function.

In general, the HLA model with RTI DDM services provides a dynamic value-based filtering method to minimize overheads in terms of computational complexity for handling requests, message traffic for distributing information, and memory requirements for storing attribute information. Additionally, the DFS checks the current filtering efficiency or default threshold to determine how to dynamically sample the subscription region continuously over time. It avoids the unnecessary extension of subscription regions and reduces the false positive updates. The RMI server must be responsible for processing the exchange of all object attributes and interactions among the client hosts. So it keeps the clients’ events in an unbound queue and uses the blocking receiving model to receive the clients’ simulation results. As a result, its load will increase very fast as the number of client hosts increases. 4.3 Experiment 3. HLA with 3LCM for Web-Based Distributed Simulation Our last experiment uses a simulation environment similar to that described in Section 4.2. First, we designated a Sun Sparc 20 as the RTI console, and allocated five workstations to execute the 3LCM as five federateServers. The next step was to involve 20 clients connecting to any of five federateServers using a sequential order. The clients then executed a distributed discrete event simulation program in the federateServer to which they were connected. In addition to federation management via the RTI services, 3LCM could maintain information consistency between clients and a federateServer. The maximum number of simultaneous clients ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

•

291

Performance of RMI Object Model 15000

Turnaround Time (ms)

12000

9000

6000

3000

0 1

2

3

4

5

Number of Client Hosts Fig. 8.

The simulation results of the RMI object model.

for each federateServer was six. The testing simulation program was implemented using the Java Application Programmer’s Interfaces (APIs) of the HLA federate interface specification. The other federateServers also executed the testing simulation program simultaneously under the control of the RTI. The final step involved sending the simulation results to the client hosts. Figure 9 shows two experimental results, one adopts 3LCM to implement in a centralized server (i.e., the first experiment), and the other is built upon the proposed HLA-based design with the 3LCM. The turnaround time, which is measured upon the HLA-based design with 3LCM, can be defined as follows:

Tclient2to2federation~l, m, n! 5 Max1#j#m, 1#j#n~Max~CSCTi! 1 MPTj~l! 1 STj 1 FRCTj!,

(6)

where T client2to2federation is the maximum turnaround time among m clients that send requests to the n federateServers for executing a distributed simulation and receive the returned simulation results, CSCT i is the communication time (i.e., including the sending and receiving time) calculated from a client i to the federateServer j . MPT j ~ l ! is the message processing time of the federateServer j upon the control level l . ST j .is the simulation time of the federateServer j , and FRCT j is the coordination time among a federate j , the other n 2 1 federates and the RTI console. ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

292

T. Lu et al.

•

Turnaround Time vs. Number of Client Hosts Turnaround Time (ms)

15000 Five federateServers with 3LCM A Centralized Server with 3LCM

10000

Level 2

Level 1

5000

Level 1

Level 2

Level 3

Level 3

0 1

Fig. 9.

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Simulation results running on an HLA-based distributed simulation environment.

The horizontal axis represents the number of client hosts, and the interval for each control level is also indicated. The short vertical lines are used for a centralized server with 3LCM and the long vertical lines are used for the HLA-based model. The turnaround time for the central server with 3LCM increases dramatically when the number of participants exceeds 18 because the information updates produced by the client hosts overwhelm the capacity of the server, even when the predictive simulation scheme is used in the third level. However, the turnaround time for the five federateServers with 3LCM remains constant over all ranges of simulation because our simulation model, based on the HLA RTI, can have five distributed servers to share the load. At the same time, the value-based filtering strategy in the DDM of RTI also minimizes the overheads among federateServers. Although in this experiment we distributed the load to each federateServer evenly, better turnaround time is still obtained with an unbalanced load. Figure 9 shows that incorporating 3LCM with the HLA-based distributed simulation model indeed provides a low and stable turnaround time for a large-scale distributed simulation environment. 5. CONCLUSIONS In this article we explored the mechanism required to support a large-scale distributed simulation on the Web, and adopted high level architecture (HLA), developed by the Defense Modeling and Simulation Office (DMSO), to provide a more flexible distributed simulation environment. The HLA is a convenient and powerful modeling and simulation (M&S) platform to develop distributed simulation applications. Compared with the RMI distributed object model, whose turnaround time increases dramatically as the number of client hosts increases, the turnaround time for the RTI distributed ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

Supporting Large-Scale Distributed Simulation

•

293

model increases just slightly. Simulation results show that by incorporating the 3-level control mechanism (3LCM) with HLA-based middleware, federateServer can adaptively maintain information consistency and minimize message traffic for distributing information among client hosts and the federateServers. The result shows that the environment remains almost constant over the entire range of simulation, in other words, it can support an even larger number of clients and provide as much information consistency as possible for a large-scale Web-based distributed simulation. In general, when a simulation environment needs to support a large number of users, a federateServer should be used, otherwise we can connect the client host to the federation via the RTI directly. The current RTI does not automatically provide information filtering based on known functions, it only determines which federates should discover which objects by matching subscription and update regions. Therefore, we have proposed a dynamic filtering strategy (DFS) to minimize false-positive updates and enhance the filtering efficiency of subscription regions on continuous coordinates. Simulation results show that the turnaround time for the federation with a DFS function is smaller than the turnaround time for the federation without DFS function. Although the difference is not very significant in the experiment, we expect that it becomes more important when the number of objects and regions in the routing space is large. From the experimental results, we have shown that our design, using HLA, is a workable and practical simulation prototype. ACKNOWLEDGMENTS

The authors would like to thank the reviewers for their valuable comments. This research was partially supported by the National Science Council of Taiwan, under contracts NSC-88-2213-E-110-011. REFERENCES ANUPAM, V. AND BAJAJ, C. 1994. Shastra: An architecture for development of collaborative applications. Int. J. Intell. Coop. Inf. Syst. 3, 2, 155–172. BRICKEN, W. AND COCO, G. 1994. The VEOS project. Presence: Teleoper. Virtual Environ. 3, 2 (Spring), 30 –39. BROLL, W. 1997. Distributed virtual reality for everyone: A framework for networked VR on the Internet. In Proceedings of the 1997 IEEE Annual International Symposium on Virtual Reality (Albuquerque, NM, Mar. 1-5). IEEE Computer Society Press, Los Alamitos, CA, 121–128. CHIM, J. H. P., LAU, R. W. H., SI, A., VA LEONG, H., TO, D., GREEN, M., AND LAM, M. L. 1998. Multi-resolution model transmission in distributed virtual environments. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST ’98, Taipei, Taiwan, Nov. 2–5), J. M. Shieh and S.-N. Yang, Chairs. ACM Press, New York, NY, 25–34. COHEN, D. AND KEMKES, A. 1997. Using DDM: An application perspective. In Proceedings of the 1997 Spring Workshop on Simulation Interoperability (Mar. 3-7). Available on the Internet. EVANS, E. AND ROGERS, D. 1997. Using Java applets and CORBA for multi-user distributed application. IEEE Internet Comput. 1, 1 (May/June), 43–55. FISHWICK, P. A. 1996. Web-based simulation: Some personal observations. In Proceedings of the 1996 Winter Conference on Simulation (WSC ’96, Coronado, CA, Dec. 8 –11), J. M. ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.

294

•

T. Lu et al.

Charnes, D. J. Morrice, D. T. Brunner, and J. J. Swain, Eds. ACM Press, New York, NY, 772–779. FRÉCON, E. AND NÖU, A. A. 1998. Building distributed virtual environments to support collaborative work. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST ’98, Taipei, Taiwan, Nov. 2–5), J. M. Shieh and S.-N. Yang, Chairs. ACM Press, New York, NY, 105–113. IEEE. 1995. IEEE Standard for Distributed Interactive Simulation- Application Protocols: IEEE Std 1278.1. IEEE Press, Piscataway, NJ. LU, T. C., LIN, M. T., AND LEE, C. N. 1999. Control mechanism for large-scale virtual environments. J. Visual Lang. Comput. 10, 1, 69 – 85. MACEDONIA, M. R., ZYDA, M. J., PRATT, D. R., BRUTZMAN, D. P., AND BARHAM, P. T. 1995. Exploiting reality with multicast groups: A network architecture for large-scale virtual environments. IEEE Comput. Graph. Appl. 15, 3, 38 – 45. MORSE, K. L. 1997. HLA data distribution management: Design document, version 0.7. Available through the Internet. MORSE, K. L. AND STEINMAN, J. S. 1997. Data distribution management in the HLA: Multidimensional regions and physically correct filtering. In Proceedings of the 1997 Spring Workshop on Simulation Interoperability (Mar. 3-7). Available on the Internet ORFALI, R. AND HARKEY, D. 1997. Client/Server Programming with Java and CORBA. John Wiley and Sons, Inc., New York, NY. PAGE, E. H., GRIFFIN, S. P., AND ROTHER, L. S. 1998. Providing conceptual framework support for distributed web-based simulation within the high level architecture. In Proceedings of the SPIE Conference on Enabling Technologies for Simulation Science II (SPIE, Orlando, FL, Apr. 13-17). SPIE, Bellingham, WA, 287–292. PAGE, E. H. 1998. The rise of Web-based simulation: Implications for the high level architecture. In Proceedings of the Winter Conference on Simulation (WSC ’98, Washington D.C., Dec. 13–16), D. J. Medeiros, E. F. Watson, J. S. Carson, and M. S. Manivannan, Eds. IEEE Computer Society Press, Los Alamitos, CA, 1663–1668. POOCH, U. W. AND WALL, J. A. 1993. Discrete Event Simulation: A Practical Approach. CRC Press computer engineering series. CRC Press, Inc., Boca Raton, FL. TACIC, I. AND FUJIMOTO, R. M. 1997. Synchronized data distribution management in distributed simulations. In Proceedings of the 1997 Spring Workshop on Simulation Interoperability (Mar. 3-7). Available on the Internet. U.S. DEPARTMENT OF DEFENSE (DMSO). 1996. High level architecture for modeling and simulation master plan, Version 1.7. U.S. DEPARTMENT OF DEFENSE (DMSO). 1998a. High level architecture rules. Version 1.3. Available through the Internet. U.S. DEPARTMENT OF DEFENSE (DMSO). 1998b. High level architecture federate interface specification. Version 1.3. Available through the Internet. U.S. DEPARTMENT OF DEFENSE (DMSO). 1998c. High-level architecture object model template specification, Version 1.3. Available through the Internet. U.S. DEPARTMENT OF DEFENSE (DMSO). 1998d. High level architecture run-time infrastructure programmer’s guide, 1.3, Version 5. Available through the Internet VAN HALDEREN, B. A. W. AND OVEREINDER, B. J. 1998. Fornax: Web-based distributed discrete event simulation in Java. In Proceedings of the ACM Workshop on Java for HighPerformance Network Computing (Palo Alto, CA, Feb./Mar.). ACM Press, New York, NY. ZYDA, M. AND SHEEHAN, J. 1997. Modeling and Simulation: Linking Entertainment and Defense. National Academy Press, Washington, D.C. Received: April 1999;

revised: November 1999;

accepted: December 1999

ACM Transactions on Modeling and Computer Simulation, Vol. 10, No. 3, July 2000.