FREE FOR PUBLIC RELEASE
VEVA – A Procedure Model for Distributed Simulation Experiments
Robert Siegfried Alexander Laux
Günter Herrmann
Johannes Lüthi
Universität der Bundeswehr München Werner-Heisenberg-Weg 39 85577 Neubiberg, Germany
ITIS GmbH Werner-Heisenberg-Weg 39 85577 Neubiberg, Germany
FH Kufstein Andreas Hofer-Straße 7 6330 Kufstein, Austria
[email protected] [email protected]
[email protected]
[email protected]
ABSTRACT In order to plan and execute high-quality distributed simulation experiments, a structured proceeding is required for all participating stakeholders. Whereas the Distributed Simulation Engineering and Execution Process (DSEEP) is considered a high-level framework which needs to be tailored to individual applications, the VEVA procedure model is very detailed and tries to guide the users as much as possible. The VEVA defines the phases and steps to be carried out for planning, executing and analyzing a distributed simulation experiment. Furthermore, the VEVA includes an extensive role model, defining indepth the responsibilities and duties of all involved parties. To ensure that all critical aspects are thought of (and documented), the VEVA provides documentation guidelines. These guidelines are provided in the form of templates which have to be filled out during the course of the simulation experiment. First applications of the VEVA procedure model (including the roles and documentation guidelines) demonstrate the applicability and feasibility of the VEVA.
1.0 INTRODUCTION Distributed simulation experiments are highly complex projects. On the one hand, this is due to the variety of stakeholders involved and on the other hand, the complexity stems from the various simulation systems (virtual and constructive) and real systems (live) to be interconnected. Inhibiting the typical project characteristics (aim focused, many persons, time/resource constraints, uniqueness), distributed simulation experiments have to be executed using a strict procedure model. Several procedure models have been proposed: The Distributed Interactive Simulation (DIS) standard defines recommended practices on exercise management [3], the High Level Architecture for Distributed Simulation (HLA) standard proposes the Federation Execution and Development Plan (FEDEP) [5]. Last but not least and in contrast to the standard-specific approaches, the Distributed Simulation Engineering and Execution Process (DSEEP) provides a generalized, high-level framework which has to be tailored to the individual needs [4]. With these approaches in mind, a detailed procedure model for distributed simulation experiments called VEVA (“Vorgehensmodell für den Einsatz der VIntEL-Architektur”, engl. “Procedure Model for Application of the VIntEL-Architecture”) was developed [1,6,2]. The VEVA procedure model is grounded in the VIntEL-development of the German Federal Office of Defense Technology and Procurement (BWB, P1.3, Koblenz). VIntEL (“Verteilte Integrierte Erprobungs-Landschaft”, engl. “Distributed
RTO-MP-MSG-076
015 - 1
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments Integrated Test Bed”) is a multi-year effort aiming at increasing the reliability and applicability of distributed simulations and strengthening the credibility of the simulation results [7]. Besides architectural and technical considerations, the need for a wide-ranging organizational support was quickly identified and is addressed by the VEVA procedure model.
2.0 DESCRIPTION OF THE VEVA PROCEDURE MODEL 2.1
Overview
Figure 1 gives an overview of the VEVA procedure model, the documentation templates defined by the VEVA, and the products as well as further documentation to be created during a distributed simulation experiment. The documentation according to VEVA guidelines contains all information necessary to describe the purpose and content of an experiment as well as organizational and technical issues related to this experiment. In contrast, the column “Products and further documentation” lists documentation which may be required by organizational regulations (e.g. an IT-Security specification according to some specific guideline). To put it another way, the VEVA defines only guidelines for documentation with regards to the content of an experiment, not considering any documentation required due to organizational regulations. Phases
Necessary documentation according to VEVA guidelines
Products and further documentation
Define objectives
Objectives and requirements Organization
IT-Security specification
Develop conceptual model
Conceptual model Data management plan
Scenario
Develop systemspecific model
Simulation model Data management plan
Federation Object Model (FOM) Federation Agreements
Prepare simulation
Configuration and execution plan Data management plan
Execute simulation
Configuration file for InitService Network plan Customized components Federation
Configuration and execution plan Results
Raw execution outputs (data)
Evaluate results
Results
Derived outputs
Wrap up experiment
Results
Reusable components
Figure 1: Overview of the VEVA procedure model (Phases, Documentation according to VEVA guidelines, Products and further documentation).
VEVA defines seven phases which in turn consist of several steps. Although the illustration in Figure 1 suggests a waterfall-like approach, the VEVA procedure model explicitly allows to go back to previous phases if necessary. The first (and supposedly most important) phase of the VEVA procedure model deals with defining the requirements and objectives. The user or subject matter expert (SME) has to define the objectives and the 015 - 2
RTO-MP-MSG-076
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments questions to be answered with the simulation experiment. The user and SMEs are not required to have any knowledge about (distributed) simulation. Indeed, the requirements and objectives should be specified in terms of the problem domain to be investigated. As indicated in Figure 2, each simulation experiment is usually driven by some question at hand and starts with some initial thoughts on objectives, scenarios, quality requirements etc. This initial phase is not part of the VEVA (and therefore displayed in grey in Figure 2). Once the decision is made to carry out the simulation experiment and to use the VEVA procedure model, the first step is to specify the requirements in detail.
Define objectives Initial thoughts... (Military) User Analysis Domain expert Configuration management Quality management Security management
Initial thoughts on: Objectives, scenarios, quality, time schedule, ...
Specify requirements
Objectives and requirements Intention and context Specification of problem and objectives Scenario description Organization Time schedule Organization plan
Figure 2: Phase 1 of the VEVA procedure model (“Define objectives”).
Afterwards, the requirements and objectives of the user are transformed into a conceptual model. The conceptual model contains all objects and units as well as the interactions between the objects and units. Modelling a detailed scenario is also part of constructing the conceptual model. Important to note is that the conceptual model does not make any assumptions about the simulation systems to be used. Within the third phase, the relation of objects of the conceptual model to actual simulation systems is defined. Furthermore, the simulation model (e.g. the FOM in an HLA-based simulation) is developed. Phase 4 (“Prepare simulation”) deals with all activities necessary to create the actual simulation environment and preparing the simulation runs. Especially, this includes integration and test of the simulation environment as well as working out the detailed experiment plan (specification of simulation runs, number of repetitions, input parameters, etc.). After finishing this phase, the simulation environment is fully functional. The fifth phase contains the actual execution of the simulation runs as defined in phase 4. If necessary, the simulation environment has to be modified between the simulation runs. Planning, preparing and testing these modifications is already done in the previous phase.
RTO-MP-MSG-076
015 - 3
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments After executing the simulation runs and collecting the data, the raw execution output has to be prepared for analysis (e.g. data fusion). These initial preparations go along with plausibility checks which are useful for a quick assessment of the validity of the data. Afterwards, the analysis and interpretation is done. The final phase of the VEVA procedure model deals with creating final reports for the original user, identifying possibly reusable components and documenting problems as well as best practices. The VEVA procedure model was developed with reusability in mind: Results from early phases may be reused in future simulation experiments. This way, much work and effort may be saved if parts of an earlier simulation experiment can be reused. Within this context, especially the conceptual model is of high importance. The high importance of the conceptual model is due to three aspects: •
Developing the conceptual model requires a deep understanding of the problem domain and the requirements as well as the objectives of the user. Whereas the requirements and the objectives are defined by the user and SMEs – ideally without reference to distributed simulation – the conceptual model is developed by the modellers and simulation experts. The actual development process requires all parties to work together closely, leading to a common, shared understanding of the objectives and requirements.
•
The conceptual model is usually much more detailed than the requirements and objectives initially defined by the user. Therefore, developing the conceptual model forces the involved parties to express their thoughts (requirements/objectives/constraints) in a very concise way. This process of clarification and specification helps to avoid misunderstandings and to focus on a common goal.
•
Developing a conceptual model is crucial for reusability. First, the conceptual model may technically be realized in various ways. For specific kinds of simulation experiments, the first two phases do not need to be executed again. Instead, the repeated execution of a simulation experiment may start with the third phase (i.e. development of the system-specific model). This is helpful, if a simulation experiment shall be executed again using different simulation systems (e.g. for cross-comparison purposes). Second, the conceptual model provides the information necessary for evaluation whether parts of a simulation experiment may be reused. The requirements and objectives are usually too coarse, whereas the technical realization usually does not allow any evaluation of the simulation pragmatics. The conceptual model is the missing link between the requirements and objectives on the one hand, and the technical realization on the other hand.
The high importance of the conceptual model is also acknowledged by Wang, Tolk and Wang [11], following the arguments of Robinson [8].
2.2
Roles
Contrary to the DSEEP, the VEVA procedure model explicitly defines 12 roles along with their duties and responsibilities. Figure 3 illustrates the roles in combination with the three institutions – Sponsor, Client, Verification, Validation and Accreditation (VV&A) – involved in the simulation experiment. We consider VV&A as a separate institution to stress their independence. Each role may be fulfilled by members of one or more institutions: Administration – Responsible for the whole IT infrastructure (network, clients, router, etc.). Analysis – Assists in defining target parameters and analysis requirements. Responsible for analyzing the simulation results and interpretation (jointly with the domain experts). Configuration management – Responsible for correct versioning, distribution and archiving of all documentation and products created during the experiment.
015 - 4
RTO-MP-MSG-076
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments Domain expert – Provides the required domain-specific knowledge. Implementation – Responsible for adapting simulation systems (e.g. to new gateways) and providing interfaces. Lead – The Lead is in charge of the project and is responsible for all planning and execution issues. Modelling – Responsible for creating the conceptual model based on the objectives, requirements and scenarios defined by the sponsor. Operation – Operates the simulation systems respectively participating real systems. Quality management – Responsible for all quality-related issues. This includes assistance in defining requirements unambiguously as well as reviewing documents and products. Security management – All issues related to security (IT security as well as military security) are handled by this role. Technology management – Responsible for setup, configuration and operation of the simulation systems and simulation-related systems (e.g. HLA runtime infrastructure). (Military) user – The actual user who originally defines the need for a simulation experiment. Within the VEVA process the user’s main task is to define the requirements and objectives (at the beginning) and to interpret the simulation results (jointly with the analysis and domain experts). Sponsor (Military) user Domain expert Lead Security management Configuration management Analysis Technology management Operation
Implementation Modeling
Quality management
Administration
Client
VV&A Figure 3: Roles defined within the VEVA procedure model.
All steps within the seven phases of the VEVA are carried out by one or multiple roles. As indicated in Figure 3, several roles may be staffed with members of different institutions (either jointly or exclusively by one institution). Depending on the actual simulation experiment and the available personnel several roles may be combined (e.g. Administration and Technology management).
RTO-MP-MSG-076
015 - 5
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments
2.3
Documentation
Besides describing the process of developing a distributed simulation, the VEVA procedure model contains detailed documentation templates. In total, the VEVA defines 7 documents. Document 1: Objectives and requirements First and foremost, the sponsor has to define the objectives and requirements of the simulation experiment. This document is the basis for planning and executing the whole simulation experiment. Therefore, the importance of this document can not be over-estimated. Figure 4 illustrates the structure of the documentation guidelines in more detail. As an example of the documentation aspects an excerpt from the document “Objectives and Requirements” is shown. The whole document “Objectives and Requirements” is divided into three topics (“Intention and context”, “Specification of problem and objectives”, “Scenario description”). Each topic is further refined into several aspects (e.g. topic “Intention and context” is split into eight aspects). All documentation guidelines defined by the VEVA follow a top-down outline. At the beginning, more general aspects are addressed and subsequently refined within the following documentation. This outline of the documentation helps to ease both creation and understanding of the documents. Document 2: Organization Whereas the first document focuses on describing the purpose and content of the simulation experiment, the second document groups all organizational aspects. This includes the advised time schedule, as well as assignment of roles and further organizational regulations (e.g. document management, way of collaboration, periodic meetings etc.) Document “Objectives and requirements“ Topic 1: “Intention and context“ OR 1.1 Context of the experiment OR 1.2 Purpose of the experiment OR 1.3 Type of experiment OR 1.4 Intended usage of the results OR 1.5 Importance of the results OR 1.6 Traceability OR 1.7 Reproducibility OR 1.8 Classification
Structure of the documentation templates:
Topic 2: “Specification of problem and objectives“ OR 2.1 Derivation of objectives OR 2.2 Definition of target parameters and requirements OR 2.3 Requirements on the analysis OR 2.4 Input parameters OR 2.5 Execution conditions OR 2.6 Abort criteria OR 2.7 Further constraints
Document Topic Aspects VEVA specifies and describes all aspects in detail!
Topic 3: “Scenario description“ OR 3.1 Military scenario OR 3.2 References OR 3.3 Units and objects OR 3.4 Geography OR 3.5 Environment OR 3.6 Initial state OR 3.7 Criteria for final states OR 3.8 Requirements on validity and fair-fight
Figure 4: Structure of the documentation guidelines (example is taken from the document “Objectives and Requirements”).
015 - 6
RTO-MP-MSG-076
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments Document 3: Conceptual model After defining purpose and content of the simulation experiment and consideration of organizational issues, the VEVA requires the development of a conceptual model. This conceptual model has to be described in detail within this document. Firstly, this document contains a detailed modelling of the scenarios (in contrast to the description by the sponsor in the first document). Secondly, all objects and units participating within the simulation experiment have to be defined. Each object and unit has to be uniquely identified as well as their capabilities and attributes. Thirdly, all communication and interaction relationships between the objects and units have to be defined. Document 4: Data management plan The purpose of the experiment (as defined in document 1) determines mainly the data to be recorded during the simulation runs. Further requirements on data recording may result from technical issues (e.g. for debugging purposes) as well as from quality management issues (e.g. monitoring of specific latencies). The data management plan explicitly lists all measures to be recorded, including the measuring device (e.g. log file, stop watch, screen video, etc.) and all measuring points (e.g. DVI video-output of system S in rack K in Room R). Furthermore, the data management plan contains all regulations regarding the handling of data. This includes regulations on the temporary storage of simulation results as well as security instructions and agreements on the long-term storage of data. Document 5: Simulation model The document regarding the actual simulation model is mainly technical in its nature. First of all, the systems actually used are documented. Secondly, the object model, interaction model and state model is described. In case of HLA-based simulations, this is equivalent to describing the Federation Object Model (FOM). Thirdly, time management issues are addressed, followed by detailed federation agreements. The federation agreements contain all information necessary for executing the simulation which is not contained in the object-, interaction- or state-model. Although, the notion of federation agreements is taken from HLA, similar kinds of agreements are generally always necessary. With the actual simulation model defined, it is possible to perform a first serious feasibility check. It is now possible to check whether all objectives and requirements (as defined by the sponsor in document 1) may be met with the advised simulation model (assuming otherwise perfect function of the systems). Therefore, this document contains checklists for performing such a feasibility check. Detected shortcomings or unmet requirements may either result in a change of the requirements or in the improvement of the simulation model. Document 6: Configuration and execution plan The actual configuration of the distributed simulation (network, routers, etc.) is part of the configuration and execution plan. Besides the actual configuration, this document contains also the integration- and test plan. The final part, the experiment plan, describes in detail the planning of the simulation runs (When? Who?) as well as their execution. Document 7: Results All results generated by the simulation experiment need to be documented. Besides the actual simulation results (raw data, processed data), this also includes the analysis and interpretation. Furthermore, the sponsor should draw conclusions whether the original requirements are met and whether the objectives are
RTO-MP-MSG-076
015 - 7
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments achieved. Finally, the lessons learned should be preserved for future experiments.
2.4
Products
Additionally to the documentation, the VEVA defines the products to be developed. Products are different from documentation: Whereas documentation is usually a description (text, figures, diagrams, etc.), the term “product” refers to work items which may be directly used within the simulation environment (e.g. the FOM, configuration files, etc.). Currently: Manual configuration of each system is time-consuming and error-prone!
Standardization and formalization allows modeling the scenario once and distributing it to all systems
System A System B
Do all systems operate with the same initial state?
System A
System C
System B System C
Without time-consuming manual checks, all systems operate with the same initial state
Figure 5: This example illustrates the possible benefits achievable by standardizing and formalizing the simulation scenario.
Figure 1 identifies possible products to be developed within a simulation experiment. Currently, standards exist only for few of the products shown in Figure 1. Figure 5 illustrates the possible benefits achievable when standardizing and formalizing the scenario, e.g. using the Military Scenario Description Language (MSDL) [9]. Assuming the scenario is not only available as documentation (as defined by the documentation guidelines described above), but also as a product following well-defined standards, multiple benefits may be achieved: •
First and foremost, the time-consuming and error-prone configuration of each system is obsolete. Instead, the scenario is developed once and distributed to all systems.
•
The formalized scenario is a necessary pre-requisite for reproducing simulation runs (possibly even after a long time).
•
The formalized scenario may be reused (in total or in parts) within future simulation experiments. The degree of reusability is much higher as if the scenario is only available as documentation.
Similar reasons hold for many other products to be developed within a simulation experiment. For example, if it is possible to standardize and formalize the federation agreements, sophisticated tools may be put in place to ensure that the federation agreements are automatically and continuously enforced/verified during the whole simulation run.
015 - 8
RTO-MP-MSG-076
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments
3.0 INCREASED SIMULATION INTEROPERABILITY WITH VEVA A structured proceeding, in this case according to the VEVA procedure model, is advantageous for many reasons. In the following, we will focus on the contributions of VEVA to improving simulation interoperability and the quality of the simulation results.
3.1
Levels of Conceptual Interoperability
Figure 6 illustrates the relation of the VEVA procedure model and the corresponding documentation to the levels of conceptual interoperability model (LCIM) as defined by Tolk et al. [10,11]. The LCIM is a valuable approach in order to systematically identify interoperability issues on several levels. This clear identification assists in fixing and preventing interoperability problems more efficiently. In general, the earlier phases of the VEVA deal with the “high” levels of the LCIM. In other words, the intended pragmatics of a simulation experiment is defined at the very beginning of each simulation experiment. This illustrates once again why the first phase (“Define objectives”) is of such high importance for the whole simulation experiment. Continuing the VEVA process, the pragmatic aspects are refined, and subsequently the semantics, syntactical and technical issues are addressed.
Figure 6: Relation of VEVA (and according documentation) to the levels of conceptual interoperability defined by Tolk [10].
By integrating the LCIM in the development and execution process, the VEVA procedure model also contributes to a higher degree of interoperability of simulation models.
3.2
Availability of Documentation related to Interoperability Issues
By providing documentation guidelines in form of detailed templates, each simulation experiment using VEVA documents the same aspects. Information related to interoperability issues is always found in the same place. This simplifies finding specific information regarding interoperability challenges and allows a well-founded interoperability analysis. As the phases and steps explicitly define which documents and products have to be created, establishing common standards becomes easier. By integrating best practices and experiences from practical applications, all users of the VEVA benefit from a common procedure model.
RTO-MP-MSG-076
015 - 9
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments
3.3
Practical Evaluation and Feedback
Evaluation and further development of the VEVA procedure model is based heavily on practical experiences: In a first step, the VEVA procedure model is applied to all experiments of the current VIntEL-development. Within this controlled environment, basic evaluation is carried out and lots of valuable feedback is built into the VEVA procedure model. Second, and perhaps most notably, the VEVA procedure model is continuously applied to distributed simulation experiments of various branches of the German Federal Armed Forces. These experiments cover live, virtual and constructive simulations, support multiple areas of application (procurement, training, analysis and decision support) and range from rather small experiments (involving a very homogenous infrastructure) to large experiments (involving multiple simulators possibly located in two or more countries). The user feedback from first applications of the VEVA is very promising [2]: •
All users (as well as additional participants) emphasize the need for a structured proceeding. From our experiences, the acceptance of using a procedure model like the VEVA is very high. However, we have to emphasize that adequate coaching and assistance of the users was provided during these first applications. In our opinion such assistance is necessary if the users are not very familiar with distributed simulation experiments or use the VEVA for the first time. Furthermore, the practical insight gained by accompanying such an experiment is the most valuable input for further development of the VEVA.
•
The documentation templates are regularly considered very important. Instead of criticizing the templates for restricting their freedom, the users usually point out the high importance of such templates. The users appreciate the guidelines defined by these templates and have trust in these templates to contain current best practices and up-to-date research results. By forcing the users to think about various aspects, the documentation templates help to ensure/improve the quality of an experiment. Last but not least, potential savings are quoted as these templates are defined only once.
•
The advantages of using VEVA do not come for free: Creating the documentation according to VEVA is a time- and resource expensive task. Especially in the first phases various participants (user, subject matter expert, modeler, etc.) have to work together to reach a common understanding. Defining unambiguous objectives and requirements is quite challenging and takes some time.
The early application of our proposed procedure model VEVA in various experiment shows that additional work related with its application is recognized, but in the long run the additional planning at the beginning pays off multiple times in the course of an experiment.
4.0 SUMMARY AND OUTLOOK 4.1
Summary
The VEVA procedure model provides a detailed (step-by-step) approach to planning, executing and analyzing a distributed simulation experiment. Besides the procedure model itself, the VEVA provides sophisticated documentation templates, guiding the user and leading to a standardized set of experiment documentation. Evaluation results of various applications of the VEVA are very promising, as the users regularly point out the benefits of a structured proceeding as well as the immense value of guidelines and templates for the actual documentation. Evaluating the interoperability of simulation systems benefits greatly from the standardized set of documentation defined by the VEVA for each simulation experiment. Assessing and knowing the possible
015 - 10
RTO-MP-MSG-076
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments level of interoperability along with limitations and pitfalls is a very important foundation for describing and documenting substantial quality aspects of a simulation experiment. Subsequently, the trustworthiness can be improved significantly and the military user can gain more confidence in the results delivered by the simulation experiment.
4.2
Outlook
Although the VEVA is quite mature at the current state, there are still many open issues. First and foremost, the VEVA needs to be applied to simulation experiments regularly. Practical application and evaluation is the only feasible way to improve the VEVA and guarantee the broad applicability. In addition, a critical mass of VEVA experienced users may reduce the overhead induced by the process in the long run. Regarding simulation interoperability the most important benefit is expected by a standardization and formalization of the products developed during simulation experiments. Besides reducing ambiguities and enabling reusability, the standardization and formalization of the products is a necessary pre-requisite for defining and establishing a holistic tool-chain supporting the military user.
5.0 REFERENCES [1]
Günter Herrmann, Alexander Laux, Axel Lehmann, Susann Schleupner, Robert Siegfried, Johannes Lüthi, “Verifikation, Validierung & Akkreditierung im Rahmen des Systemdemonstrators Verteilte Integrierte Erprobungs-Landschaft (SD VIntEL)“ (in German), ITIS Institut für Technik Intelligenter Systeme e.V., Final study report, BWB contract E/UR2F/9A054/7F193, 2009.
[2]
Günter Herrmann, Alexandros Karagkasidis, Axel Lehmann, Junlan Qian, Robert Siegfried, Johannes Lüthi, “Verifikation, Validierung & Akkreditierung im Rahmen des Systemdemonstrators Verteilte Integrierte Erprobungslandschaft (SD VIntEL), Phase 2“ (in German), ITIS GmbH, Final study report, BWB contract E/UR2F/0B002/7F193, 2010.
[3]
IEEE: IEEE Recommended Practice for Distributed Interactive Simulation – Exercise Management and Feedback. 1996. – IEEE Std 1278.3-1996
[4]
IEEE: IEEE Draft Recommended Practice for Distributed Simulation Engineering and Execution Process (DSEEP). 2008. – IEEE P1730/Dv3.0
[5]
IEEE: IEEE Recommended Practice for High Level Architecture (HLA) – Federation Development and Execution Process (FEDEP). 2003. – IEEE Std 1516.3-2003
[6]
Alexander Laux, “Konzeption eines ganzheitlichen Vorgehensmodells zur Durchführung verteilter Simulationsexperimente“ (in German), Universität der Bundeswehr München, Bachelor Thesis, 2009
[7]
Eckehard Neugebauer, Daniel Nitsch, Oliver Henne, “Architecture for a Distributed Integrated Test Bed“, NATO RTO Modelling and Simulation Group Symposium (MSG-069), Brussels, Belgium, October 2009, RTO-MP-MSG-069 paper 19
[8]
Stewart Robinson, "Conceptual modelling for simulation – Part I: Definition and Requirements", Journal of the Operational Research Society, 2008, 59:278-290, doi:10.1057/palgrave.jors.2602368
[9]
Simulation Interoperability Standards Organization (SISO): Military Scenario Definition Language (MSDL). 14.10.2008. – SISO-STD-007-2008
RTO-MP-MSG-076
015 - 11
FREE FOR PUBLIC RELEASE
FREE FOR PUBLIC RELEASE
VEVA - A Procedure Model for Distributed Simulation Experiments [10] Andreas Tolk, “The Levels of Conceptual Interoperability Model”, Proceedings of the 2003 Fall Simulation Interoperability Workshop, Paper 03F-SIW-007, Orlando, USA, September 2003. [11] Wenguang Wang, Andreas Tolk, Weiping Wang, “The Levels of Conceptual Interoperability Model: Applying Systems Engineering Principles to M&S”, Proceedings of the 2009 Spring Simulation Multiconference, San Diego, USA, April 2009.
015 - 12
RTO-MP-MSG-076
FREE FOR PUBLIC RELEASE
