enables collaborative development of simulation applications, and the deployment of simulation components as services takes advantage of the SOA.
SOAr-DSGrid: Service-Oriented Architecture for Distributed Simulation on the Grid Xinjun Chen, Wentong Cai, Stephen J. Turner, and Yong Wang Parallel and Distributed Computing Center School of Computer Engineering Nanyang Technological University Singapore 639798 {chen0081, aswtcai, assjturner, wang0065}@ntu.edu.sg Abstract Simulation is a low cost alternative to experimentation on real-world physical systems. Grid technology enables coordinated use of and secure access to distributed computing resources and data sources. The service-oriented architecture (SOA) is an ideal paradigm for next generation computing. The loose coupling among services in the SOA relieves service consumers from detailed knowledge of implementation, implementation language, and execution platform of the services to be consumed. In this paper, we propose a framework for developing a component-based distributed simulation and executing the simulation in a service-oriented architecture on the Grid. This framework consists of the schemas for developing simulation components and simulation applications, and underlying base component service modules for constructing a simulation component as a service. The use of component interfaces and schemas enables collaborative development of simulation applications, and the deployment of simulation components as services takes advantage of the SOA (e.g., loose coupling, heterogeneity, and transport neutrality). This paper discusses the motivation for developing such a framework and describes the details of its development.
1. Introduction Simulation permeates many areas such as production, business, education, and science and engineering. It is a low cost alternative to experimentation on real-world physical systems. With the advance of computer networks and prevalence of low-cost commodity computers, distributed simulation becomes a favorable technology to reduce simulation
time, to enable the execution of simulation components at different geographical locations, and to increase fault tolerance [8]. The High Level Architecture (HLA) for Modeling and Simulation (M&S) [13], the standard for distributed simulation, has been approved by US DoD and standardized as IEEE 1516. It facilitates interoperability among simulations and promotes reuse of simulation components (i.e., federates in HLA’s terminology). Federates communicate with each other through the Runtime Infrastructure (RTI), which provides the implementation of the HLA standards. Grid computing was proposed by Ian Foster as secure and coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations [6]. Grid technology enables coordinated use of geographically distributed computing resources and facilitates access to geographically distributed data sources. The Globus Toolkit has become the de facto standard platform for Grid computing. The advantages of Globus middleware lie in its functionalities which are defined in Open Grid Services Architecture (OGSA) [7], such as resource management, data management, service discovery, security, service lifecycle, and notification. The latest Globus Toolkit version 4 (GT4) [11] implements Web Service Resource Framework (WSRF) [15] and Web Service Notification (WSN), and utilizes Web Service Addressing (WS-Addressing). The early work on executing HLA-based distributed simulation on the Grid [4, 17, 18] aims to take advantages of both Grid computing middleware (e.g., Globus Toolkit) and the HLA/RTI. The Grid middleware is employed to perform the task of resource management (including scheduling, loadbalancing and monitoring), coordination of simulation execution, and security. The HLA/RTI is used to perform simulation related tasks such as synchronization, time management and data
distribution management. However, the dependency on the HLA/RTI requires that the simulation component developers must have a profound knowledge and skill of HLA/RTI programming and that the execution of the simulation federation must follow the conventions of the HLA/RTI. In addition, the simple integration of the Grid middleware and the HLA/RTI cannot fully exploit the benefits brought by the Service Oriented Architecture (SOA). Under the SOA, simulation components are loosely coupled. They can be discovered and composed to form a simulation application. Components can be deployed to heterogeneous resources and communicate via standard SOAP messages . But, how they communicate (i.e., the underlying transport layer protocol) need not be fixed till deployment. To make natural progress over the work we have done and to address the problems mentioned above, in this paper we propose a Service Oriented Architecture for executing Distributed Simulation on the Grid: SOAr-DSGrid. In this architecture, different views are presented for the simulation development and simulation execution. The component-based view is used in the simulation development. To expose the operations implemented in a simulation component and to facilitate the composition of a simulation, schemas are defined for the component developers to publish component interfaces and for the simulation application developers to describe the constituent components and their interactions in a simulation. The service-oriented view is presented during the simulation execution. Each simulation component is implemented as a GT4 service by extending some base component service modules. To hide the details of service and simulation management implementation, the base component service modules provide the underlying simulation management such as event scheduling and time advancement in addition to the basic service implementations. Component services communicate with each other through pre-defined service interfaces and they are configured according to the component interactions specified in the simulation description. The separation of component interface from service interface makes the development of a component as a service easier. Component providers just need to describe the operations of a component using the component interface schema and implement the component operations accordingly. They do not need to worry about how a service is implemented, how a service interface is defined, and how services communicate with each other. Simulation application developers can then use the component interfaces and simulation description schema to compose a simulation application. They do not need to worry about how the
component services are located and how they are deployed and invoked. The use of component schema and the simulation description schema enables collaborative development of simulation applications, and the deployment of simulation components as services takes advantage of the SOA (e.g., loose coupling, heterogeneity, and transport neutrality). Using the component-based view, a simulation application developer can search for the required simulation components, and describe the composition of the components. Under our serviceoriented architecture, simulation component services can be dynamically deployed to the heterogeneous resources and communicate via various transport layer protocols. They work in a peer-to-peer manner: a component service can be a service provider and a service consumer at the same time. The rest of this paper is organized as follows: Section 2 describes related work on service-oriented architecture, composition, and distributed simulation on the Grid. In Section 3, we present the system architecture and components of SOAr-DSGrid. Section 4 details the interfaces and schemas in the framework. Section 5 explains the component development and application development, component runtime, and the organizer. We conclude the paper and look into future work in Section 6.
2. Background and related work The Service Oriented Architecture (SOA) evolves from component technology such as DCOM and CORBA. It reorganizes software applications into a set of interacting services, each accessible through a standard interface (e.g., WSDL) and messages (e.g., SOAP). The most significant advantage of the SOA is the loose coupling among services. The consumer of a service is not required to have a detailed knowledge of implementation, implementation language, or execution platform of the service. The only concern of the consumer is how a service can be invoked according to the service interface. Exposing applications as services enables reuse of applications. Standards (e.g., WSDL, SOAP, and WS-I basic profile) ensure interoperability between services. Low dependency between loosely coupled services improves the scalability of a system. However, the basic SOA lacks support for simulation management, service orchestration, and security. Although GT4 incorporates comprehensive security mechanisms, it does not solve the problem of service orchestration, nor does it provide simulation management. Composition can be categorized into composition in time and composition in space [14]. Composition in time, commonly known as workflow, specifies a series
of tasks to be executed in a logical sequence. Composition in space refers to the inherent relationship between components. In this kind of composition, the services are composed based on their intrinsic properties (e.g., compatible ports), and independent of execution order. In the web service community, various specifications are proposed to address the service composition (or orchestration). WSFL and XLANG are earlier work from IBM and Microsoft respectively. They are no longer active and are superseded by BPEL4WS [2]. These workflow specifications, in general, address only composition in time. Indiana University’s XCAT [1, 5, 12, 14] supports both composition in time and composition in space. Krakow’s Application Flow Composer (AFC) [3] is a component-based system for Grid workflow composition based on CCA technology [5]. The contribution of AFC lies in its capability of reading incomplete application information and generating a complete description of a Grid application. We cannot directly adopt BPEL4WS, XCAT or AFC for the following reasons. BPEL4WS was developed for stateless web services; whereas our simulation components are implemented as GT4 services. In addition, composition in space, rather than composition in time, is more appropriate to specify the composition of simulation components. Although XCAT and AFC provide support for composition in space and can be used to develop Grid workflow compositions, they do not address our requirements. They do not provide any simulation oriented services and they compose the components according to their service interfaces exposed. In our architecture, component interfaces are used in the composition. Simulation components are implemented as services. The simulation service interfaces are meant for communication between simulation component services. They are different from the component interfaces and are not exposed to the simulation application developers. All of our previous work on the execution of distributed simulation over the Grid are based on the HLA/RTI [4, 16, 17, 19]. In [4], we described a Load Management System (LMS), implemented on top of GT2, to manage and coordinate geographically distributed resources for HLA-based simulation. The LMS has a resource management subsystem to acquire resource information, and a job management subsystem to monitor job execution and to perform load balancing. A federate migration algorithm was
described in [17]. Our recent work based on GT3 was reported in [16] and [19]. We implemented a FederateProxy-RTI framework called HLAGrid [16]. Under this framework, simulation application developers develop simulation federates using the HLA/RTI interface as usual. But, the federate code is compiled with the HLAGrid library instead. Federates run on the resources at the client site, and proxies and the RTI, running on the resources available over the Grid, provide a backbone to support distributed simulation. There is one proxy for each federate and HLAGrid relays the federate initiated and RTI initiated services between federates and proxies. This framework provides the federate developer with a familiar interface and decouples the execution of simulation federates and the backbone that provides distributed simulation services. Apart from the benefits from the Grid, the framework also supports secure execution of simulation federate, flexibility and heterogeneity [16]. To support the execution of large-scale HLA-based distributed simulations over the Grid, we also developed federate factory services to dynamically map Grid services to computing resources and a centralized index service to support dynamic discovery of federation and federates [19]. The architecture described in this paper differs from our previous work in the following aspects (as discussed in Section 1): a component-based view is provided to the component and simulation application developers; whereas a service-oriented view is adopted in the execution of simulation components. To fully exploit the advantages of the SOA, instead of using the HLA/RTI, we integrate simulation execution (e.g., time management) as part of a component service.
3. Architecture overview As shown in Figure 1, there are five roles in the architecture: the component provider (i.e., simulation component developer), the client (i.e., the simulation application developer), the repository, the organizer, and the index service. The component provider develops simulation components. The client builds simulation applications. The repository stores the relevant files including component interface files, entity type files, and rule type files. Component interface files and entity type files which contain descriptive information about components and entities, are used during the development of a simulation application.
Query about component interface files, rule type files, and entity type files
Organizer entity type files, and rule type files
Repository nt ne po s, m e d co fil an ve ce s, l e files rie erfa i t f Re i nt ype pe t ty le ty r u enti
Retrieve rule type files
Query about rule type files
Index Service
Register resource instances
Deposit rule type files
Re tri Su in ev bm e file terf it s im u s, ace com la tio an f i p on nr d les equ en , en est t tity rul typ e ty e pe fi le s
Query about component services
Query about organizer service, resource instances, component interface files, rule type files, and entity type files
Client
Deposit component interface files
Register component interface files, rule type files, and entity type files
Register rule type files
AComponent BComponent CComponent
Retrieve rule type files
Component Provider
Figure 1. SOAr-DSGrid architecture overview A rule type file, which provides rule logic to determine the target simulation operation or next component, is retrieved and used at runtime. They will be further explained in Section 4. In this service-oriented architecture, the organizer, the index service, and every simulation component are implemented as GT4 services. The organizer acts as an agent to the client. It receives simulation description files submitted by the client, executes the simulation application, and returns the simulation results to the client. A centralized index service provides directory information on components, component interface files, entity type files, and rule type files. This directory information is required for component development and application development. Component Provider
Component Runtime
Schema for component interface file
Schema for entity type file
Client
Schema for simulation description file
Organizer
SOAr-DSGrid Runtime
Globus Toolkit Version 4
Figure 2. Layered structure of SOAr-DSGrid Figure 2 illustrates the layered structure of the entire framework. SOAr-DSGrid is a layer residing between the users and GT4. The parts contained in the dotted
box in the middle represent the schemas provided to the component providers and clients. The solid parts are used by the component providers, and the striped parts are used by the clients. The details about the interfaces and schemas are covered in Section 4. Other parts in the middle layer form the SOAr-DSGrid runtime. The component runtime includes base modules for generating simulation component services and tools for configuring and deploying component services. Section 5 elaborates the component runtime and addresses the issue of how the runtime links to the interfaces.
4. Interfaces and schemas A simulation scenario used in SOAr-DSGrid is depicted in Figure 3. A simulation component processes certain types of entities. An entity is associated with a set of attributes (e.g., a1). A component has a set of public variables (e.g., v2) and private variables. Public variables are also referred to as shared variables, as they can be read by other components. There are two types of private variables: one receives setup values at component initialization time (e.g., v3); the other is internal to the component with no setup values (e.g., v1). A simulation component (e.g., CA) starts simulating by receiving a processible entity (e.g., e1). It processes the entity by updating the attributes of the entity and/or variables of itself. It can also generate new entities (e.g., e2). It then routes the processed or generated
entity to one of the possible next components based on the routing rule (e.g., r1). private variable v3 (setup)
ow ne r
private variable v1
of
owner of
owner of
Component CB
ibe sub scr
attribute a1
public variable owner of v2
e1
e2 e2
Component CA process entity e1 and produce entity e2
Second, unlike conventional workflow which requires the client to define a static route, SOAr-DSGrid allows the client to specify multiple possible routes for an entity during application development, and the exact route is determined automatically at runtime according to the routing rules prescribed. Third, the routing of entities is determined by an individual component without the need of a centralized component.
rule r1
send entity e2 to CB or CC based on rule r1
e2
Component CC
Figure 3. Definitions and relationships The component interface file is defined by the component provider and is used during the development of a simulation application. The clients refer to these files to determine whether the corresponding component is suitable for building a specific simulation application. The component interface contains information about variables, operations, and the entity types involved. Figure 4 is the pictorial representation of the component interface schema. Clients develop simulation applications using the schema for simulation description files. As depicted in Figure 5, the schema requires clients to specify the participating component instances involved in a simulation application with relevant setup values. The entity flows, variable flows, operation rules and routing rules, and simulation results to be collected are also specified. An entity flow comprises a set of entity links. Each entity link denotes a set of possible routes from one source component to a set of target components. Routing rules are used by the source component to determine the specific target component at runtime. A variable flow specifies one publisher component and a set of subscriber components. Simulation results are values of specified variables of participating component instances returned at the end of a simulation session. The main features of the design are summarized as follows. First, a component does not invoke another component’s simulation operation directly. Components are linked with each other through entity and variable flows. For components with multiple operations, operation rules are used to determine which operation should be performed on a received entity.
5. Simulation development and runtime support The operations defined in the component interface will be implemented as component-specific operations in the component service. These operations are internal to a component service. In addition to the componentspecific operations, a component service also implements a set of common service operations. These are the operations defined in the corresponding WSDL file and are exposed to other component services.
5.1. Component development As described in Section 1, the component development and application development use the component-based view. Figure 6 shows the sequence diagram that illustrates the process of developing a simulation component. To develop a simulation component, a component provider first writes a component interface file using the component interface schema shown in Figure 4. New entity type files are defined if the simulation logic in the component involves new types of entities. If the simulation component contains multiple operations and the necessary rule type files are not available, new rule type files will be defined by the component provider. Subsequently, the component developer implements each operation defined in the component interface as a class. After that, the component provider requests the SOAr-DSGrid’s service generator to generate a GT4 service corresponding to the component specified. The service generator takes in the simulation operation classes and entity type files, combines them with some predefined base component service modules, and creates the code of a component service. The service generator also builds and deploys the service on behalf of the component provider. Finally, the component provider deposits the component interface files and entity type files into the repository, and registers them with the index service.
Figure 4. Component interface schema
Figure 5. Schema for simulation description files
Component Provider
Index Service
Repository
Create component interface file and entity type files if any Implement simulation logic Invoke service generator to generate, build, and deploy a component service deposit entity type files if any register entity types deposit component interface register component interface register component service
Figure 6. Implementing a simulation component During the entire process of component development, the component provider only deals with the component interface file definition, component operations implementation, entity type files definition, and rule type files definition. The advantages of this design are obvious. The component providers are relieved from not only simulation management (e.g., time management an event scheduling) but also the complexity of service development and deployment. As for the service generator, it combines the operations implementation with the base component service modules to generate a component service class. According to WSRF, a GT4 service also includes a resource class and a resource home class. The resource class includes both component-specific variables specified in the component interface file, and component-general variables that are common to all the component services. All component services have a similar resource home.
description file according to the schema shown in Figure 5. In the case where the simulation description file involves new rule types other than those available in the repository, the client creates new rule type files. The new rule type files will be deposited to the repository and registered with the index service (for reuse by other simulation application developers). The execution of a simulation application is initiated by submitting the simulation description file to the organizer. The detailed runtime operation of the organizer is described in subsection 5.4. As can be seen, if all the required simulation components could be found from the index service, the development of a simulation application is quite straightforward, as the client only needs to produce a correct simulation description file with necessary rule type files. The execution of a simulation application is handled by the organizer.
5.3. Component runtime The component runtime (or the base component service modules) implements some component-general elements including input entity queue, output entity queue, output queue manager, message queue, message manager, time manager, and rule engine, as shown in Figure 7. The component manager, output queue manager, and message manager are active threads. The rule engine and time manager are passive objects invoked on demand. Common service operations (i.e., service interfaces) are defined to manipulate the input entity queue and message queue. At runtime, the output queue manager and message manager take the responsibility for the communication among components. SOAr-DSGrid packages all these common elements into a base component from which every component extends.
5.2. Simulation application development The development of simulation applications is carried out by simulation application developers (or clients in SOAr-DSGrid terminology). As described in Section 1, a client builds an application in a component-based manner. He/She selects appropriate components by investigating published component interfaces. The directory information about the component interface files can be obtained through the index service. The client then builds a simulation
Figure 7. Component structure A component service is activated by the organizer. The activation of a component service involves starting the three abovementioned threads. The component manager removes the first entity from the queue, and consults the rule engine to find the appropriate
simulation operation to invoke. After processing the entity, the component manager invokes the time manager to advance simulation time. It also consults the rule engine to determine the next component for the processed entity. The processed entities are stored temporarily in the output entity queue, from which the output queue manager dispatches processed entities to their respective next components. Time management is very important to the correct execution of distributed simulations. It is a great challenge especially when we implement the time management in a distributed manner like that in SOArDSGrid. We utilize an algorithm similar to that presented in [9] for time management. However, the use of shared variables in SOAr-DSGrid renders the safe time algorithm insufficient. Shared variables can be updated at any unexpected time. So, it is difficult to determine a lookahead for a component that has a shared variable. To implement the shared variables, we adopt an approach described in [10] and introduce a history list for each shared variable. To obtain the value of a shared variable, the consumer components pull the shared variable from the provider component.
5.4. Organizer The organizer is the agent for the clients to execute simulation applications. It is a GT4 web service responsible for processing simulation description files. It initializes a simulation session, but does not intervene in the execution of simulation components. The simulation initialization process is shown in Figure 8. Clients locate the existing organizer service by querying the index service. They create a resource instance (according to WSRF) for the organizer and submit a simulation request. The organizer parses the simulation description file and creates a simulation session accordingly. The organizer queries location information of participating component services and instantiates a resource for each participating component service. Subsequently, the organizer initializes the participating component services by passing them the relevant information such as setup variables, linked components, and shared remote variables. Each participating component service subscribes its shared remote variables of the respective owners. If a rule type file is not available locally, it should be downloaded from the repository using the directory information obtained from the index service. After finishing all the setup procedures, the organizer activates all the participating component services to start simulation.
Figure 8. Simulation initialization
6. Conclusions and future work This paper presents the SOAr-DSGrid, a serviceoriented architecture for executing distributed simulation on the Grid. SOAr-DSGrid presents simulation component developers and simulation application developers with a component-based framework for ease of component development and simulation development. Its underlying runtime infrastructure uses a service-oriented architecture to ensure the interoperability between simulation components. In SOAr-DSGrid, every simulation component is exposed as a component conforming to a component interface schema, and implemented as a GT4 service. A simulation application is developed by creating a simulation description file using the simulation description schema. The execution of a simulation application is handled by the organizer. SOAr-DSGrid allows flexible dynamic service orchestration by employing rule type files and a rule engine. The runtime communication in SOAr-DSGrid is conducted in a peer-to-peer manner. Performance is always a critical success factor to an architecture. From the results reported in [19], we note that the initial simulation setup cost is high. The performance improves when the workload increases. This also applies to SOAr-DSGrid. As discussed in subsection 5.4, when a client submits a simulation request to the organizer, the organizer parses the simulation description file, creates a simulation session
for it, and initializes the simulation component services. As a result, the cost of the initialization process is comparatively high. However, the organizer does not intervene in the simulation execution after activating the simulation component services. The communications between simulation component services are in a fully distributed peer-to-peer manner. During runtime, there is no need to rediscover service endpoints and the runtime communication overhead will be mainly contributed by the time spent on generating and interpreting SOAP messages. Therefore, it is obvious that the greater the simulation workload, the better the performance will be. To evaluate the performance of SOAr-DSGrid, a prototype of distributed simulation of wafer manufacturing will be developed. With the increase of simulation scale, the index service may need to be implemented in a distributed manner (e.g., one index service for each Virtual Organization). The time management may also need to be refined to effectively tackle the zero-lookahead issue. In addition, mechanisms will be developed to support hierarchical composition of simulation applications. Multi-level time management protocols will be investigated accordingly for the hierarchical composition.
7. References [1] R. Armstrong, D. Gannon, A. Geist, K. Keahey, S. Kohn, L. McInnes, S. Parker, B. Smolinski, Toward a Common Component Architecture for High-Performance Scientific Computing, in Proc. of High Performance Distributed Computing Conference, 1999, pp.13-22. [2] BPEL4WS, Business Process Execution Language for Web Services Version 1.1, http://www128.ibm.com/developerworks/library/specification/ws-bpel/. [3] M. Bubak, K. Gorka, T. Gubala, M. Malawski, K. Zajac, Component-based System for Grid Application Workflow Composition, in Proc. of the 10th European PVM/MPI Users' Group Meeting, 2003, pp.611-618. [4] W. Cai, S. J. Turner, H. Zhao, A Load Management System for Running HLA-based Distributed Simulations over the Grid, in Proc. of the 6th IEEE Intl. Symposium on Distributed Simulation and Real Time Application, Oct 2002, pp.7-14. [5] CCA, The Common Component Architecture Forum, http://www.cca-forum.org/glossary/index.html. [6] I. Foster, C. Kesselman, S. Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations, in Intl. Journal of Supercomputing Applications, 15(3), 2002, pp.200-222. [7] I. Foster, H. Kishimoto, A. Savva, The Open Grid Service Architecture, Version 1.0, http://www.gridforum.org/documents/GWD-I-E/GFDI.030.pdf.
[8] R. M. Fujimoto, Background and Applications - Why Parallel/Distributed Simulation, in Parallel and Distributed Simulation Systems, Wiley Interscience, 1999, pp.4-5. [9] B. P. Gan, L. Liu, S. Jain, S. J. Turner, W. Cai, W. J. Hsu, Distributed Supply Chain Simulation Across Enterprise Boundaries, in Proc. of the 32th Winter Simulation Conference, 2000, pp.1245-1251. [10] B. P. Gan, Y. H. Low, J. Wei, X. Wang, S. J. Turner, W. Cai, Synchronization and Management of Shared State in HLA-based Distributed Simulation, in Proc. of the 35th Winter Simulation Conference, 2003, pp.847-854. [11] Globus, Globus Toolkit http://www.globus.org/toolkit/.
Version
4.0,
[12] M. Govindaraju, S. Krishnan, K. Chiu, A. Slominski, D. Gannon, R. Bramley, Merging the CCA Component Model with the OGSI Framework, in Proc. of 3rd Intl. Symposium on Cluster Computing and the Grid, 2003, pp.182-199. [13] IEEE, IEEE Standard 1516 (HLA Rules), 1516.1 (Interface Specification), and 1516.2 (Object Model Template), https://www.dmso.mil/public/transition/hla/. [14] S. Krishnan, D. Gannon, XCAT3: A Framework for CCA Components as OGSA Services, in Proc. of 9th Intl. Workshop on High-Level Parallel Programming Models and Supportive Environments, April 2004. [15] WSRF, The Web-Service Resource Framework Version 1.0, http://www.globus.org/wsrf/specs/ws-wsrf.pdf. [16] Y. Xie, Y. M. Teo, W. Cai, S. J. Turner, Service Provisioning for HLA-based Distributed Simulation on the Grid, in Proc. of the 19th Workshop on Principles of Advanced and Distributed Simulation, 2005, pp.282-291. [17] Z. Yuan, W. Cai, Y. H. Low, S. J. Turner, Federate Migration in HLA-based Distributed Simulation, in Proc. of 1st Workshop on HLA-based Distributed Simulation on the Grid (in conjunction with Intl. Conference of Computational Science), 2004, pp.856-864. [18] K. Zajac, M. Bubak, M. Ma-lawski, P. Sloot, Towards Grid Management System for HLA-based Interactive Simulations, in Proc. of the 7th IEEE Intl. Symposium on Distributed Simulation and Real Time Applications, 2003, pp.4-11. [19] W. Zong, Y. Wang, W. Cai, and Service Discovery for Simulation, in Proc. of the 9th Distributed Simulation - Real pp.116-124.
S. J. Turner, Grid Services HLA-based Distributed IEEE Intl. Symposium on Time Applications, 2004,
